Method and apparatus for detecting inter-neighbor space

Abstract

Disclosed is an apparatus (system, apparatus, method, etc.) for refining a three-dimensional (3D) model, in particular a 3D model of an ivory column. An initial 3D model and a plurality of two-dimensional (2D) images corresponding to the 3D model are received or generated. The 3D model is refined using an edge boundary of a space around or between two or more objects in the 3D model identified from the 2D image.

Description

Priority declaration

[0001] This patent application claims priority to U.S. Provisional Patent Application No. 63 / 337,590, filed on May 2, 2022, entitled “Interproximal Space Detection with Three-Dimensional Models,” the entire contents of which are incorporated herein by reference. CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] All publications and patent applications mentioned in this specification are herein incorporated by reference in their entirety to the same extent as if each individual publication or patent application was specifically and individually indicated to be incorporated by reference. Technical Field

[0003] The present disclosure relates generally to methods and apparatus for orthodontic treatment planning and, more particularly, to methods and apparatus for accurately modeling a patient's dentition from intraoral scan data, including interproximal regions that are difficult to resolve. Background Art

[0004] Orthodontic and dental treatment using a series of patient-removable appliances (e.g., "aligners") are very useful for treating patients. Treatment planning is typically performed in conjunction with a dental professional (e.g., dentist, orthodontist, dental technician, etc.) by generating a model of the patient's teeth in the final configuration and then breaking the treatment plan down into multiple intermediate stages (steps) corresponding to the individual aligners to be worn in sequence. The process can be interactive, with the stage divisions, and in some cases the final target position, being adjusted based on the constraints on tooth movement and the preferences of the dental professional. Once the final treatment plan is finalized, a series of aligners can be manufactured corresponding to the treatment plan.

[0005] Treatment planning for a patient may start with a three-dimensional (3D) model of the patient's teeth. Traditionally, the 3D model can be determined using a non-contact optical scanner. However, errors in the 3D model caused by incomplete or erroneous scan data in some cases may lead to errors in the treatment plan for the patient concerned.

[0006] Therefore, new and useful apparatuses (devices, methods, etc.) are needed to obtain high-quality 3D models. Summary of the invention

[0007] The methods and apparatus (e.g., systems and devices) described herein can refine an initial or preliminary 3D model of a patient's dentition to include accurate representations of difficult-to-image areas, including interproximal areas. These methods and apparatus can be part of an orthodontic treatment method and / or apparatus. In some examples, projection data associated with camera positions that have been used to capture optical scans can be used to determine the presence of areas within the dentition that may be difficult or impossible to accurately image using an intraoral scanner (particularly including interproximal spaces). In some examples, these methods and apparatus can be configured to confirm that interproximal spaces exist or may exist, and then the boundaries of the teeth in these areas can be refined, thereby refining the digital 3D model of the patient's dentition.

[0008] As used herein, a digital model of a patient's dentition may be equivalently referred to herein as a digital 3D model of the patient's dentition, or simply a 3D model. In general, an initial or preliminary 3D model of a patient's dentition may be generated or received by a conventional optical scanner, including an intraoral scanner. Any suitable intraoral scanner may be used. In addition, any suitable digital representation of the subject's dentition (all or part of the dentition) may be used. For example, a digital 3D model of a patient's dentition may be configured as a "point cloud" of data points. A point cloud may refer to a plurality of data points that may define a surface of a 3D model.

[0009] In some examples, a 3D model of a patient's dentition can be processed to identify surfaces of the 3D model (e.g., segmentation) and / or can be processed to find or determine interproximal spaces or other areas of the dentition associated with or adjacent to surfaces, and in some cases overlapping surfaces. These areas are generally referred to herein as "interproximal areas" and may be difficult to visualize with an intraoral scanner because imaging from any external angle (e.g., buccal, lingual, and / or occlusal) may be at least partially blurred or obscured, making it difficult to resolve these areas with millimeter or sub-millimeter accuracy. Current methods of defining tooth surfaces (including interproximal spaces) may be based solely on 3D capture (e.g., using an intraoral scanner), are typically limited to a resolution of more than 300 μm, and are unable to accurately mark each interproximal space.

[0010] In general, the methods and apparatus described herein can be used during scanning of a patient's dentition (e.g., in real time or semi-real time) or after scanning of the dentition to improve the quality (e.g., resolution) of a 3D digital model of the patient's dentition. These methods can also or alternatively be used to help identify one or more structures of the patient's dentition, such as for segmenting the patient's dentition (including accurately distinguishing between teeth, teeth and gums, etc.).

[0011] In any of these methods, a 3D digital model of a patient's dentition can be inspected or analyzed to determine the location of an interproximal area ("IP space"). Any appropriate (e.g., manual, automatic, or semi-automatic) technique can be used to identify the presence of an interproximal space. For example, a machine agent comprising a trained network (e.g., a neural network) can be used. Any of the methods and devices described herein may include using a trained neural network to identify interproximal areas. After the interproximal space is located, the methods and devices described herein can remove point cloud data associated with the interproximal space to refine the 3D model.

[0012] For example, methods and apparatus for refining one or more 3D models are described herein. For example, the method and apparatus may include: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; refining the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from multiple two-dimensional (2D) images of the subject's dentition corresponding to the 3D model, thereby generating a refined 3D model; and, outputting information associated with the refined 3D model. 2D images may be received together with 3D model data, or separately from 3D model data. For example, 2D images may be taken while collecting 3D imaging data (e.g., structured light, confocal, etc.). 2D images may be visible light images, fluorescent images, white light images, near infrared images, etc.

[0013] In some examples, refining the 3D model may include identifying interproximal regions and identifying edge boundaries around interproximal spaces from a 2D image of the subject's dentition, and generating potential new surface points for the 3D model based on the identified edge boundaries for multiple 2D images. In some examples, refining the 3D model may also include removing at least a subset of new surface points from the 3D model that fall within one or more edge boundaries around the space from the 2D image. The remaining points (e.g., in a point cloud) may be smoothed or otherwise used to generate a new external tooth surface. In some examples, identifying edge boundaries of a space around one or more teeth or between one or more teeth may include: for each subset of at least one subset of the multiple 2D images, using machine learning to identify the boundaries of a space around one or more teeth or between one or more teeth from each 2D image of the subject's dentition in the subset. In some examples, refining the 3D model may include smoothing the edges of each identified boundary.

[0014] In some examples, generating potential new surface points of the 3D model based on the identified edge boundaries can include using a camera position corresponding to each of the multiple 2D images. Generating potential new surface points can also include mapping the identified edge boundaries to the 3D model, and determining rays formed between multiple points on each edge boundary and the position of the camera relative to the surface of the subject's dentition during capture of the 2D images of the subject's dentition, and generating new points of the 3D model between points on the surface of the 3D model at the location where each ray enters and leaves the surface of the 3D model. In some examples, each point can have an associated normal vector in a direction perpendicular to the ray and perpendicular to a tangent to the identified space.

[0015] In some examples, a method for refining one or more 3D models may include using edge boundaries of a space around one or more teeth or between one or more teeth in a 3D model, wherein the space includes an interproximal space between two teeth of the subject's dentition. The modified or improved 3D model may be used for further treatment planning and may provide more precise position and other information about the patient's teeth, which may be critical in designing and / or tracking dental treatment. Alternatively or additionally, the methods and devices described herein may be configured to output information about interproximal tooth areas. In some examples, the method may include outputting information including an indication of whether two teeth are in contact. In some examples, the method may include outputting information, and outputting the information includes displaying the information to a user. The displayed information may include quantitative information (e.g., how large the interproximal region / area is, including size information, area, volume, etc.).

[0016] In some examples, a method for refining one or more 3D models may include outputting information to a segmentation module, where the segmentation module (which may include a segmentation algorithm) separates the refined 3D model into individual teeth.

[0017] Any suitable 2D image may be used as part of the process. For example, multiple 2D images of a subject's dentition may include white light images, near infrared images, and / or fluorescent images. The 2D images may be associated with a 3D model, and in particular a 3D model including the position and / or orientation of a camera (e.g., an intraoral scanner) relative to the dentition may be associated with each 2D image. Position information may be included with the 3D data as part of a data set. For example, a 3D data scan may collect dentition images interleaved with a 2D data collection so that the camera for each dentition may be located in approximately the same position, or may be determined by extrapolation from the 3D data. The 2D images may come directly from structured light images. In structured light images, there is typically enough stray light and penetration to clearly resolve tooth boundaries. This can give more viewing directions.

[0018] In any of these examples, the method and apparatus may include receiving or generating a three-dimensional (3D) model of the subject's dentition, wherein the 3D model is based on a scan of the subject's dentition. The step of receiving or generating a three-dimensional (3D) model of the subject's dentition may include receiving and / or generating a 2D image. The 3D model is based on a scan of the subject's dentition. Any of these methods may also include refining the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth of the 3D model identified from multiple 2D images of the subject's dentition, thereby generating a refined 3D model, wherein refining the 3D model includes identifying edge boundaries around the space from the 2D images of the subject's dentition, and generating potential new surface points of the 3D model based on the edge boundaries identified for the multiple 2D images, and outputting information associated with the refined 3D model.

[0019] In some examples, a method may include: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; and refining the 3D model to generate the refined 3D model by: identifying an edge boundary around one or more teeth or around a space between one or more teeth from the 2D image of the subject's dentition to thereby identify the space, and smoothing an edge of the identified space to thereby form an edge boundary around the space, mapping the identified edge boundary to the 3D model and determining a plurality of points on the edge boundary that correspond to a camera image; generating potential new surface points of the 3D model based on the identified edge boundaries using the camera position corresponding to the 2D image, and generating one or more new points of the 3D model between points on the 3D model surface at the position where each ray enters and leaves the 3D model surface, repeating the steps of identifying edge boundaries around the space from the 2D image and generating potential new surface points of the 3D model based on the edge boundaries identified for multiple 2D images taken from different positions, and removing new surface points that fall within the one or more edge boundaries around the space from the 2D image from the 3D model, and outputting information associated with the refined 3D model.

[0020] In some examples, a method may include: capturing three-dimensional (3D) information of a surface of a subject's dentition; generating a 3D model of the surface of the subject's dentition based on the 3D information; identifying edge information of the surface from a captured 2D image, wherein the edge information is tangent to the surface of the subject's dentition; refining the generated 3D model using the edge information of the surface identified from the captured 2D image, thereby generating a refined 3D model; and outputting information associated with the refined 3D model.

[0021] In some examples, the surface of the subject's dentition can include an interproximal space between two teeth of the subject's dentition, and wherein the edge information includes an edge of the tooth in the interproximal space. Additionally, the output information can include an indication of whether the two teeth are in contact. In some examples, the output information can include a measurement of the interproximal space between the two teeth.

[0022] In some examples, outputting the information may include outputting to a user display. Outputting the information may additionally or alternatively include outputting to a segmentation algorithm, and wherein the segmentation algorithm separates the refined 3D model into individual teeth.

[0023] In some examples, identifying edge information may include analyzing the captured 2D image using a trained neural network.In some examples, the 2D image of the subject's dentition may include a white light image, a near infrared image, and / or a fluorescent image.

[0024] Optionally, in some embodiments, the methods described herein may include capturing three-dimensional (3D) information of a surface of a subject's dentition by projecting a structured light pattern onto the surface, and capturing the projected structured light pattern with one or more cameras, and generating a 3D model of the surface of the subject's dentition based on the 3D information using a corresponding algorithm or a triangulation algorithm. The methods and apparatus described herein are not limited to the use of structured light, but may be used with other imaging techniques, including confocal scanning or other 3D scanning techniques. In some examples, the methods may also include identifying at least some edge information of a surface from a two-dimensional (2D) image from one or more cameras (which may be obtained from 3D scan data), for example based on penetration and stray light. Penetration is a technique that may be used for edge lighting or used with edge lighting; other techniques that may be used include unfocused light and leakage light from a DOE (stray light may typically be used). For example, based on the penetration of the structured light pattern into the teeth, at least some edge information of a surface may be obtained from a two-dimensional (2D) image captured by one or more cameras, where the edge information is tangent to the surface of the subject's dentition. Any of these methods may further include refining the generated 3D model using surface edge information identified from the respective captured 2D images, thereby generating a refined 3D model, and outputting information associated with the refined 3D model.

[0025] Described herein is a system that can be used to refine one or more 3D models. In some examples, the system may include one or more processors, and a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receive or generate a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; refine the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from multiple 2D images of the subject's dentition, thereby generating a refined 3D model; and output information associated with the refined 3D model. Multiple 2D images may be received or generated together with the 3D model; typically, multiple two-dimensional (2D) images of the subject's dentition correspond to the 3D model. The 3D model is typically not generated from multiple 2D images, but may include 2D images taken simultaneously. Alternatively, the 2D images may be taken separately from the 3D imaging.

[0026] In some examples, executing instructions to refine the 3D model can enable the system to further identify edge boundaries around the space from a 2D image of the subject's dentition, and generate potential new surface points of the 3D model based on the identified edge boundaries for multiple 2D images. In addition, execution of the instructions can enable the system to remove new surface points that fall within one or more edge boundaries around the space from the 2D image from the 3D model. In other examples, executing instructions to identify edge boundaries of a space around one or more teeth or between one or more teeth can enable the system to use machine learning to identify the boundaries of a space around one or more teeth or between one or more teeth from each 2D image of the subject's dentition in the subset for each subset of at least one subset of the multiple 2D images. In some examples, execution of the instructions can enable the system to smooth the edges of each identified boundary.

[0027] In some examples, executing instructions to generate potential new surface points of the 3D model based on the identified edge boundaries can enable the system to use the camera position corresponding to each of the multiple 2D images. In addition, in some examples, execution of the instructions can enable the system to map the identified edge boundaries to the 3D model and determine rays formed between multiple points on each edge boundary and the position of the camera relative to the surface of the subject's dentition during capture of the 2D image of the subject's dentition, and generate new points of the 3D model between points on the 3D model surface at the location where each ray enters and leaves the 3D model surface. Each point can have an associated normal vector in a direction perpendicular to the ray and perpendicular to a tangent to the identified space.

[0028] In some examples, execution of the instructions may enable the system to generate a refined 3D model using edge boundaries of a space, wherein the space includes an interproximal space between two teeth of a subject's dentition. In some examples, the output information may include an indication of whether the two teeth are in contact. In some examples, the output information includes a measurement of the space between the two teeth.

[0029] In some examples, executing the instructions to output information associated with the refined 3D model can enable the system to display the information to a user. Additionally or alternatively, executing the instructions to output information associated with the refined 3D model can enable the system to output to a segmentation algorithm, wherein the segmentation algorithm is configured to separate the refined 3D model into individual teeth.

[0030] In some examples, the multiple 2D images can be white light images, near infrared images, and / or fluorescent images. In any of these examples, the structured light images themselves can be used in place of the 2D images with the methods and apparatus described herein. Thus, while structured light images can be used to find structured light pattern locations (and generate 3D surface data), 2D images as described herein can also be used in place of or in addition to structured light images.

[0031] In some examples, a system may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receive or generate a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receive or generate multiple two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; refine the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from the multiple 2D images of the subject's dentition, thereby generating a refined 3D model, wherein refining the 3D model includes identifying edge boundaries around the space from the 2D images of the subject's dentition, generating potential new surface points of the 3D model based on the identified edge boundaries of the multiple 2D images, and outputting information associated with the refined 3D model.

[0032] In some other examples, a system may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receive or generate a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receive or generate a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; refine the 3D model to generate a refined 3D model, wherein the refinement enables the system to: use machine learning to identify edge boundaries around one or more teeth or around spaces between one or more teeth from the 2D image of the subject's dentition to identify the space, and smooth the edges of the identified space to form edge boundaries around the space, by mapping the identified edge boundaries onto the 2D image of the subject's dentition; The method comprises: projecting rays onto the 3D model and determining rays formed between a plurality of points on an edge boundary and a position of the camera relative to a surface of the subject's dentition during capture of a 2D image of the subject's dentition, generating potential new surface points of the 3D model based on the identified edge boundaries using the camera position corresponding to the 2D image, and generating one or more new points of the 3D model between points on the surface of the 3D model at the position where each ray enters and exits the 3D model surface, repeating the steps of identifying edge boundaries around the space from the 2D image and generating potential new surface points of the 3D model based on the edge boundaries identified for a plurality of 2D images taken from different positions, and removing new surface points from the 3D model that fall within the one or more edge boundaries around the space from the 2D image, and outputting information associated with the refined 3D model.

[0033] In some examples, a system may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receive or generate a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receive or generate a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; refine the 3D model to generate a refined 3D model, wherein the refinement enables the system to: use machine learning to identify edge boundaries around one or more teeth or around spaces between one or more teeth from the 2D image of the subject's dentition to identify the space, and smooth the edges of the identified space to form edge boundaries around the space, by mapping the identified edge boundaries onto the 2D image of the subject's dentition; The method comprises: projecting rays onto the 3D model and determining rays formed between a plurality of points on an edge boundary and a position of the camera relative to a surface of the subject's dentition during capture of a 2D image of the subject's dentition, generating potential new surface points of the 3D model based on the identified edge boundaries using the camera position corresponding to the 2D image, and generating one or more new points of the 3D model between points on the surface of the 3D model at the position where each ray enters and exits the 3D model surface, repeating the steps of identifying edge boundaries around the space from the 2D image and generating potential new surface points of the 3D model based on the edge boundaries identified for a plurality of 2D images taken from different positions, and removing new surface points from the 3D model that fall within the one or more edge boundaries around the space from the 2D image, and outputting information associated with the refined 3D model.

[0034] In some examples, a system may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, enable the system to: capture three-dimensional (3D) information of a surface of a subject's dentition; generate a 3D model of the surface of the subject's dentition based on the 3D information; identify edge information of the surface from a captured 2D image, wherein the edge information is tangent to the surface of the subject's dentition; refine the generated 3D model using the edge information of the surface identified from the captured 2D image, thereby generating a refined 3D model; and, output information associated with the refined 3D model.

[0035] In some examples, for some systems, the surface of the subject's dentition can include an interproximal space between two teeth of the subject's dentition, and wherein the edge information includes an edge of the tooth in the interproximal space. Furthermore, the output information can include an indication of whether the two teeth are in contact. Additionally or alternatively, the output information can include a measurement of the interproximal space between the two teeth. In some examples, the output information can be output to a user display.

[0036] In some examples, the output information can be output to a segmentation algorithm, and wherein the segmentation algorithm separates the refined 3D model into individual teeth. In some examples, executing the instructions to identify edge information can enable the system to analyze the captured 2D image using a trained neural network. The 2D image can be a white light image, a near infrared image, and / or a fluorescent image.

[0037] In some examples, a system may include one or more processors and a memory configured to store instructions that, when executed by the one or more processors, enable the system to: capture three-dimensional (3D) information of a surface of a subject's dentition by projecting a structured light pattern onto the surface and capturing the projected structured light pattern using one or more cameras, generate a 3D model of the surface of the subject's dentition based on the 3D information using a correspondence algorithm or a triangulation algorithm, identify at least some edge information of the surface from captured two-dimensional (2D) images from the one or more cameras based on penetration of the structured light pattern into the teeth, wherein the edge information is tangent to the surface of the subject's dentition, refine the generated 3D model using the edge information identified from the surfaces of the respective captured 2D images, thereby generating a refined 3D model, and output information associated with the refined 3D model.

[0038] Any of the methods described herein may be configured as a non-transitory computer-readable storage medium configured to perform any of the methods. For example, the non-transitory computer-readable storage medium may include instructions that, when executed by one or more processors of a device, cause the device to perform operations including: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition, receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model, refining the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from the plurality of 2D images of the subject's dentition to generate a refined 3D model, and outputting information associated with the refined 3D model.

[0039] In some examples, executing instructions to refine the 3D model can enable the device to perform operations including identifying edge boundaries around a space from a 2D image of the subject's dentition, and generating potential new surface points for the 3D model based on the identified edge boundaries for multiple 2D images. In some examples, execution of the instructions can enable the device to perform operations including removing new surface points from the 3D model that fall within one or more edge boundaries around the space from the 2D image. In some examples, execution of the instructions can enable the device to perform operations including identifying edge boundaries of a space around one or more teeth or between one or more teeth, enabling the device to use machine learning to identify, for each subset of at least one subset of the multiple 2D images, the boundaries of the space around one or more teeth or between one or more teeth for each 2D image of the subject's dentition in the subset. Further execution of the instructions can cause the device to perform operations including smoothing the edges of each identified boundary.

[0040] In some examples, executing instructions to generate potential new surface points of the 3D model based on the identified edge boundaries may enable the device to perform operations using a camera position corresponding to each of the plurality of 2D images.

[0041] In some examples, execution of the instructions may enable the device to perform operations including: mapping the identified edge boundaries to the 3D model, determining rays formed between a plurality of points on each edge boundary and a position of the camera relative to a surface of the subject's dentition during capture of a 2D image of the subject's dentition, and generating new points of the 3D model between points on the surface of the 3D model at the location where each ray enters and exits the surface of the 3D model. In some examples, each point may have an associated normal vector in a direction perpendicular to the ray and perpendicular to a tangent to the identified space.

[0042] In some examples, the identified space may include an interproximal space between two teeth of the subject's dentition. In some examples, the outputted information may include an indication of whether the two teeth are in contact. In some examples, the outputted information may include a measurement of the space between the two teeth.

[0043] In some examples, executing the instructions to output information associated with the refined 3D model can enable the device to perform operations including displaying the information to a user. In addition, executing the instructions to output information associated with the refined 3D model can enable the device to perform operations including outputting to a segmentation algorithm, and wherein the segmentation algorithm is configured to separate the refined 3D model into individual teeth. In some examples, the multiple 2D images can be white light images, near infrared images, and / or fluorescent images.

[0044] In some examples, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors of the device, enable the device to perform operations including: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating multiple two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; refining the 3D model using edge boundaries of spaces around one or more teeth or between one or more teeth in the 3D model identified from the multiple 2D images of the subject's dentition, thereby generating a refined 3D model, wherein refining the 3D model includes identifying edge boundaries around spaces from the 2D images of the subject's dentition, generating potential new surface points of the 3D model based on the edge boundaries identified for the multiple 2D images, and outputting information associated with the refined 3D model.

[0045] In some examples, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors of a device, cause the device to perform operations, the operations including: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; refining the 3D model to generate a refined 3D model, wherein the refinement enables the system to: use machine learning to identify edge boundaries around one or more teeth or around a space between one or more teeth from a 2D image of the subject's dentition to identify the space, and smooth the edges of the identified space to form an edge boundary around the space; The method comprises mapping the identified edge boundary to the 3D model and determining rays formed between multiple points on the edge boundary and the position of the camera relative to the surface of the subject's dentition during the capture of the 2D image of the subject's dentition, generating potential new surface points of the 3D model according to the identified edge boundary using the camera position corresponding to the 2D image; and generating one or more new points of the 3D model between points on the surface of the 3D model at the position where each ray enters and leaves the surface of the 3D model; repeating the steps of identifying edge boundaries around the space from the 2D image and generating potential new surface points of the 3D model according to the edge boundaries identified for multiple 2D images taken from different positions; and removing the new surface points that fall within the one or more edge boundaries around the space from the 2D image from the 3D model, and outputting information associated with the refined 3D model.

[0046] In some examples, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors of the device, enable the device to perform operations including: capturing three-dimensional (3D) information of a surface of a subject's dentition; generating a 3D model of the surface of the subject's dentition based on the 3D information; capturing a two-dimensional (2D) image of the surface of the subject's dentition; identifying edge information of the surface from the captured 2D image, wherein the edge information is tangent to the surface of the subject's dentition; refining the generated 3D model using the surface edge information identified from the captured 2D image to generate a refined 3D model; and outputting information associated with the refined 3D model.

[0047] In some examples, the surface of the subject's dentition may include an interproximal space between two teeth of the subject's dentition, and the edge information may include an edge of the tooth in the interproximal space. In addition, the output information may include an indication of whether the two teeth are in contact. In some examples, the output information may include a measurement of the interproximal space between the two teeth. In some examples, the output information may be output to a user display.

[0048] In some examples, the output information may be output to a segmentation algorithm, and the segmentation algorithm may separate the refined 3D model into individual teeth.

[0049] In some examples, executing the instructions to identify edge information can enable the device to perform operations including analyzing a captured 2D image using a trained neural network. The 2D image can be a white light image, a near infrared image, and / or a fluorescent image.

[0050] In some examples, a non-transitory computer-readable storage medium includes instructions that, when executed by one or more processors of the device, enable the device to perform operations including: capturing three-dimensional (3D) information of a surface of a subject's dentition by projecting a structured light pattern onto the surface; and, capturing the projected structured light pattern with one or more cameras; generating a 3D model of the surface of the subject's dentition from the 3D information using a correspondence algorithm or a triangulation algorithm; identifying at least some edge information of the surface from captured two-dimensional (2D) images from one or more cameras based on penetration of the structured light pattern into the teeth, wherein the edge information is tangent to the surface of the subject's dentition; refining the generated 3D model using the edge information of the surface identified from each captured 2D image, thereby generating a refined 3D model; and, outputting information associated with the refined 3D model.

[0051] For example, the methods described herein include: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; generating a refined 3D model based on the 3D model of the subject's dentition using one or more edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from multiple 2D images of the subject's dentition, wherein the multiple two-dimensional (2D) images of the subject's dentition correspond to the 3D model; and, outputting the refined 3D model and / or information associated with the refined 3D model.

[0052] The 3D model or data for the 3D model and the 2D image may be received as part of the same data stream.

[0053] Generating the refined 3D model may include identifying edge boundaries around the space from the 2D images of the subject's dentition; and generating potential new surface points of the 3D model based on identifying the edge boundaries identified for the plurality of 2D images.

[0054] Any of these methods can include removing new surface points from the 3D model that fall within one or more edge boundaries around the space from the 2D image. In some examples, identifying edge boundaries of the space around one or more teeth or between one or more teeth includes: for each subset of at least one subset of the plurality of 2D images, identifying, using a trained neural network, boundaries of the space around one or more teeth or between one or more teeth from each 2D image of the dentition of the subject in the subset.

[0055] Any of these methods may include smoothing the edges of each identified boundary.

[0056] Generating potential new surface points of the 3D model based on the identified edge boundaries may include using a camera position corresponding to each of the plurality of 2D images.

[0057] As described above, any of these methods may include: mapping the identified edge boundary to the 3D model, and determining rays formed between a plurality of points on each edge boundary and the position of the camera relative to the surface of the subject's dentition during the capture of a 2D image of the subject's dentition; and generating new points of the 3D model between points on the surface of the 3D model at the location where each ray enters and leaves the surface of the 3D model. Each point may have an associated normal vector in a direction perpendicular to the ray and perpendicular to a tangent to the identified space.

[0058] The space may include an interproximal space between two teeth of the subject's dentition. The outputted information may include an indication of whether the two teeth are in contact. The outputted information may include a measurement of the space between the two teeth. In any of these examples, outputting the information may include displaying the information to a user.

[0059] Outputting the information may include outputting to a segmentation algorithm, and wherein the segmentation algorithm divides the refined 3D model into individual teeth.

[0060] The multiple 2D images may be white light images, near infrared images and / or fluorescence images.

[0061] For example, the methods described herein may include: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating multiple two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of the same scan of the subject's dentition; generating a refined 3D model based on the 3D model of the subject's dentition using edge boundaries of spaces around one or more teeth or between one or more teeth in the 3D model identified from the multiple 2D images of the subject's dentition, wherein generating the refined 3D model includes identifying edge boundaries around the spaces from the 2D images of the subject's dentition and generating new surface points of the refined 3D model based on the identified edge boundaries; and, outputting the refined 3D model and / or information associated with the refined 3D model.

[0062] In some examples, the method includes: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition that are taken as part of the same scan of the subject's dentition and that correspond to the 3D model; generating a refined 3D model based on the 3D model of the subject's dentition by: identifying an edge boundary around one or more teeth or around a space between one or more teeth from the 2D image of the subject's dentition, identifying the space using a trained neural network, and smoothing edges of the identified space to form an edge boundary around the space, mapping the identified edge boundary to the 3D model and aligning the edge boundary at a plurality of points on the edge boundary with a camera captured while capturing the subject's dentition; casting rays between positions of a surface of a subject's dentition during a 2D image relative to the 2D image, generating potential new surface points of a refined 3D model according to the identified edge boundaries using a camera position corresponding to the 2D image, and generating one or more new points of the 3D model between points on the surface of the 3D model at the position where each ray enters and leaves the 3D model surface, repeating the steps of identifying edge boundaries around the space from the 2D image and generating potential new surface points of the refined 3D model according to the edge boundaries identified for multiple 2D images taken from different positions, and removing new surface points that fall within the one or more edge boundaries around the space in the 2D image from the refined 3D model; and outputting the refined 3D model and / or information associated with the refined 3D model.

[0063] All methods and apparatus described herein, in any combination, are contemplated herein and may be used to achieve the benefits described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] A better understanding of the features and advantages of the methods and apparatus described herein will be obtained by reference to the following detailed description and accompanying drawings which set forth illustrative embodiments, in which:

[0065] Figure 1 A block diagram of an example three-dimensional (3D) model generation system is shown.

[0066] Figure 2A Example views of a portion of a subject's dentition including interproximal spaces are shown.

[0067] Figure 2B An example 3D model is shown, in which the corresponding Figure 2A The characteristics of the interproximal space are missing.

[0068] Figure 3 is a flow chart illustrating an exemplary method for modifying a 3D model of a subject's dentition.

[0069] Figure 4is shown for use based on Figure 3 Flowchart of an exemplary method for refining a preliminary 3D model using a two-dimensional (2D) image as mentioned in .

[0070] Figure 5 It is shown that the Figure 4 A flow chart of an exemplary method of identifying edge boundaries to generate new surface points of a 3D model.

[0071] Fig. 6A An example of identifying interproximal spaces from a 2D image of a patient's dentition is shown.

[0072] Figure 6B An example of identifying interproximal spaces from a 2D image of a patient's dentition is shown.

[0073] Fig. 7A An example of a plan view of a 3D model of an object is shown.

[0074] Figure 7B Another example of a plan view of a 3D model of an object is shown.

[0075] Figure 8 An example refined 3D model is shown, which may be Figure 3 -The result of the operation described in 7.

[0076] Fig. 9 shows a block diagram of a device which may be Figure 1 An example of a 3D model generation device. DETAILED DESCRIPTION

[0077] A conventional three-dimensional (3D) model of a patient's teeth can be generated based on one or more optical scans. For example, data from an optical scan can be processed and filtered to generate an initial 3D model. Unfortunately, some of the data from the optical scan may not capture all of the features of the patient's teeth, particularly the interproximal spaces between the teeth. As a result, the associated 3D model may lack interproximal spaces.

[0078] In the present disclosure, methods and apparatus (e.g., systems, devices, etc.) are described that can refine an initial 3D model to include interproximal space data that may be missing or incomplete. In some examples, projection data associated with a camera position that has been used to capture an optical scan can be used to determine the presence of an interproximal space. If it is determined that an interproximal space exists, the interproximal space data is removed to refine the 3D model.

[0079] Figure 1A block diagram of an example 3D model generation apparatus 100 is shown. Although described herein as an apparatus, the 3D model generation apparatus 100 may be implemented with any feasible apparatus, system, or device. The 3D model generation apparatus 100 may include a processing node 110, an image capture device 120, a display device 130, and a data storage device 140. As shown, the image capture device 120, the display device 130, and the data storage device 140 may each be coupled to the processing node 110.

[0080] Image capture device 120 may include one or more separate image capture devices that capture optical images of a patient's teeth (sometimes referred to herein as a subject's dentition). In some examples, image capture device 120 may include white light, near infrared light, ultraviolet light, and / or fluorescent light sources and sensors. In some other examples, image capture device 120 may include unstructured or structured light sources. In other examples, image capture device 120 may include any number of feasible light sources and sensors.

[0081] Thus, image capture device 120 may capture multiple images of the patient's teeth. In some examples, image capture device 120 may simultaneously capture images of the patient's teeth using multiple light sources and sensors.

[0082] The display device 130 may be any feasible image display device. In some examples, the display device 130 may be an integral part of the 3D model generation apparatus 100 and integrated into a housing or chassis. In other examples, the display device 130 may be communicatively coupled to the 3D model generation apparatus 100, for example, by a wired or wireless connection. In some cases, the display device 130 may be a computer monitor, a tablet device, a mobile phone, etc. The display device 130 may be used to display image data (e.g., image data collected by the image capture device 120) and 3D model data that may be determined (calculated) by the processing node 110.

[0083] Data storage device 140 may be any feasible data storage device, including random access memory, solid-state memory, disk-based memory, non-volatile memory, etc. Data storage device 140 may store image data, including image data captured by one or more image capture devices 120. Data storage device 140 may also store 3D model data, including 3D model data determined and / or rendered by processing node 110.

[0084] Data storage device 140 may also include a non-transitory computer-readable storage medium that may store instructions that may be executed by processing node 110. For example, processing node 110 may include one or more processors (not shown) that may execute instructions stored in data storage device 140 to perform any number of operations including processing image data from image capture device 120 and generating a 3D model of a patient's teeth.

[0085] The methods described herein can be performed by, for example, Figure 1 The device 100 shown in the figure may be implemented, or may be implemented using a device such as Figure 1 The method may be performed based on the data (e.g., 3D and / or 2D data) provided by the apparatus shown. For example, the processing node 110 of the apparatus 100 may include software, firmware, and / or hardware configured to perform the method described herein, the method comprising: identifying one or more interproximal regions, determining the boundaries of the interproximal regions in a 2D image corresponding to the 3D model from one or more (preferably, multiple different) camera angles, and / or using the identified boundaries and camera positions to refine the surface of the 3D digital model.

[0086] As described above, a 3D model of a subject's dentition may not include accurate interproximal spaces. Interproximal spaces include gaps and / or overlapping areas between adjacent teeth that can impact a treatment plan for providing dental treatment, including orthodontic treatment, to the patient. Therefore, if the interproximal spaces are incorrect or missing in a 3D model of the patient, the associated treatment plan may be incorrect.

[0087] Figure 2A An exemplary view of a portion of a subject dentition 200 is shown including an interproximal space 201. In this view, the interproximal space clearly shows that the teeth adjacent to the interproximal space 201 are separated. Figure 2B An example of a digital 3D model 210 of the same area of ​​the dentition is shown, where Figure 2A The interproximal space 201 is missing and appears as a continuous connection between the teeth. Figure 2B , the 3D model 210 shows that adjacent teeth are connected, rather than separated.

[0088] Figure 3 is an example of a method 300 for modifying a 3D model of a subject's dentition as described herein. Some examples may perform the operations described herein with additional operations, fewer operations, operations in a different order, operations in parallel, and some different operations. Figure 1 The method 300 is described with reference to the 3D model generating apparatus 100; however, the method 300 may be performed by any other suitable system, apparatus, or device.

[0089] At 302, a preliminary 3D model of a subject's dentition (also referred to as an initial 3D model) may be received or generated. The preliminary 3D model may be based on a scan of the subject's dentition. For example, processing node 110 may receive a preliminary 3D model that may have been created by another device, apparatus, or method. In another example, 3D model generation apparatus 100 may generate the preliminary 3D model. The preliminary 3D model may include, for example, Figure 2B The interproximal space shown is incorrectly represented.

[0090] like Figure 3 As shown, 2D images of the subject's dentition can be included with the received or generated 3D model, or they can be received or generated separately. For example, the 2D image can be part of the 3D data, for example, the 2D image can be interlaced with the 3D data (for example, four 3D frames can be interlaced with one 2D frame, or other combinations). If the 2D image is included in the 3D data, the device can know where the 2D image was taken from (including the camera position when it was taken). This can be done by interpolating between the positions of the 3D capture on both sides of the 2D capture (for example, in time). Optionally, IMU data from an intraoral scanner can be used. Therefore, typically, at 304, the 3D model generation device 100 can receive or generate multiple two-dimensional (2D) images of the subject's dentition. For example, the image capture device 120 can capture an image 2D image of the subject's dentition. In another example, the 3D model generation device 100 can receive multiple 2D images of the subject's dentition from another device. In particular, in some examples, the 2D image can be included with the 3D data. For example, 2D images may be interleaved with 3D data (e.g., 4 3D frames and 1 2D frame, or other combinations). Thus, a 3D model and 2D images may be received together (simultaneously), and the 3D model may be generated by separating the 3D data from the same data stream as the 2D images; the 2D images may be generated by separating from the same data stream. This example may be particularly helpful because the same data comprising the 2D images and the 3D data may include position and / or orientation information indicating where the image was taken, including the camera position at the time of the capture. This may be accomplished by interpolating between the positions of the 3D capture on either side (in time) of the 2D capture (which may include IMU data).

[0091] The multiple 2D images may be from buccal, lingual and / or occlusal directions. The 2D images may be based on any suitable imaging modality, such as white light (WL), near infrared light, structured light, fluorescent light or any other feasible light, including stray light or other edge light effects.

[0092] In some examples, the 2D images may be captured by an intraoral camera that can be moved throughout the patient's mouth. Some intraoral cameras may include one or more inertial measurement units (IMUs) that can help provide positional information for the camera when capturing the images. In some examples, the 2D images may correspond to a preliminary 3D model and be captured (photographed) as part of the same scan used to determine the preliminary 3D model.

[0093] Next, in box 306, the 3D model generation device 100 can refine the preliminary 3D model (received or generated in box 302) based on the multiple 2D images (received or generated in box 304). In some examples, information from the 2D images can be used to determine rays that intersect various areas of the preliminary 3D model. For example, the 2D images can be analyzed to identify the boundaries of the interproximal regions; different 2D images corresponding to different relative camera positions can be checked for each identified interproximal region. The identified interproximal region can be divided into multiple points surrounding the boundary, and these points can be used to project rays that pass through the interproximal region boundary into and outside the boundary of the interproximal region in 3D space. Multiple rays that pass through the boundary are initially used to add points to the point cloud, and information from the additional rays can be used to determine which added points can be removed later. In this way, information associated with the intersection of rays can be used to refine the preliminary 3D model to include more accurate interproximal space information. The following is combined with Figure 4-7B Describe this process in more detail.

[0094] Back to Figure 3 , the method may include output 308, for example, the 3D model generation device 100 may output information associated with the refined 3D model. For example, the refined 3D model may be rendered and displayed on the display device 130 for viewing by a user. In another example, the refined 3D model may be output and stored in the data storage device 140 for later retrieval. As described above, the refined 3D model may include interproximal space information. In some examples, the output information may include measurements of the interproximal spaces between teeth. In some examples, the refined 3D model may be output to a segmentation algorithm. The segmentation algorithm may separate the refined 3D model into individual teeth.

[0095] Figure 4 It is shown that for at least Figure 3 An example of a method 400 for refining a preliminary 3D model based on a 2D image as mentioned in block 306 of FIG. Figure 4In , method 400 may include, at 402, analyzing the 3D model by identifying edge boundaries of a 2D image of the subject's dentition. The edge boundary may be the edge of one or more teeth around the interproximal region in one or (preferably) multiple 2D images associated with the 3D model. As described above, this step may be performed manually, automatically, or semi-manually. In some examples, the edge (or boundary) of the interproximal region may be identified for the same interproximal region from multiple different 2D images, each 2D image having a different associated camera position (e.g., angle) relative to the patient's teeth. In addition, in some examples, the 3D model generation device 100 may use a machine learning (ML) training model to help identify the edge boundary. The boundary may be divided into a finite number of points around its periphery, and rays extending through these points from a putative camera position may be visualized and used to generate additional points in a point cloud of the 3D model so that the points may be added to the 3D model. In some examples, after the edge boundary is identified, the 3D model generation device 100 may smooth the identified edge boundary. For example, the 3D model generating apparatus 100 may use a smoothing spline-based method (algorithm) to smooth the identified edge boundaries.

[0096] In summary, boundary detection can be done with any classical boundary detection method. In some examples, a deep learning network can be used. For example, the training of the network can come from labeling or can be learned automatically. This may be because larger inter-proximal spaces (e.g., more than 500 μm or 1 mm) can be detected and used as the basis for automatic teaching of the system. For example, a convolutional neural network (such as but not limited to U-Net) can be used.

[0097] In some examples, identifying edge boundaries (sometimes referred to as identifying edge information) can include determining the surface and surface tangent information of the 3D model. For each ray that passes through the 3D model from the identified camera to the boundary point, a new point can be added to the original point cloud, for example, at a constant interval along the ray and between the intersections of the outer surface of the original 3D model. Each of these points is associated with a normal in a direction perpendicular to the ray, and is tangent to the IP space boundary after being projected into 3D at the position of the point. Conflicting points added in this way from different 2D images of the same interproximal region can then be removed, thereby generating a new surface representing the identified tooth (or possible gum) surface that defines the interproximal region. For example, multiple 2D images of the same interproximal space can be processed as described herein, and rays extending from the identified camera position through the periphery can be used to add new points, which can be removed later if these new points conflict.

[0098] Therefore, at 404, the 3D model generation apparatus 100 may generate new surface points of the 3D model based on the identified edge boundary. A point cloud may be a collection of points associated with one or more surfaces of a 3D model. A point cloud may define one or more surfaces of a 3D model of a patient's teeth.

[0099] New surface points can be found along the ray, which can be associated with the identified edge boundary from block 402. Figure 5 - Fig. 7 describes the determination of new surface points in more detail.

[0100] As described above, at 406, at least some of the new surface points may be removed from the preliminary 3D model. The new surface points may be determined to be associated with the interproximal space. Thus, removing the surface points from the preliminary 3D model may remove points from the point cloud associated with the interproximal space. By removing the new surface points, the interproximal space may be included in the preliminary 3D model. In this manner, the preliminary 3D model may be modified or updated to generate a refined 3D model.

[0101] For example, Figure 5 It is shown that for at least Figure 4 The method 500 for generating new surface points of a 3D model based on the identified edge boundaries mentioned in block 404 of the present invention. The 3D model generating apparatus 100 may determine at 502 the position of a camera for generating one or more 2D images. The 2D images may include Figure 3 The 2D image of the subject's dentition is received or generated in connection with block 304. The camera position may be determined from the 3D data and / or based on one or more IMUs included in the camera used to generate the one or more 2D images. In some examples, the camera position information may be stored with the 2D image in a Figure 1 data storage device 140. Optionally, the camera position can be determined relative to the earth instead of (or in addition to) being determined relative to the patient. For example, the camera position can be determined by interpolation during a 3D scan (e.g., by structured light capture, confocal, etc.). In some examples, imaging can alternate between the capture of structured light images and the capture of other images (e.g., white light, fluorescence, etc.). Thus, the camera position can be determined by stitching of structured light; the camera position can be determined based on stitching using structured light, and the camera position during acquisition of other imaging (e.g., white light, fluorescence, etc.) can be estimated during this interpolation interval (e.g., in some examples within a 10ms interval). In some cases, the use of an IMU can improve or enhance such interpolation techniques.

[0102] The 3D model generation apparatus 100 may determine a plurality of rays associated with the determined camera position at 504. A ray refers to a path of light that may be traced or directed back to a camera that has captured a 2D image or is otherwise associated with the 2D image. Therefore, for each possible camera position and image, there may be one or more associated rays.

[0103] The 3D model generation apparatus 100 may determine new points of the 3D model between the points where the ray enters and leaves the 3D model at 506. Each ray determined in block 504 may be projected. The projected ray may intersect the 3D model (e.g., between the outer surfaces of the 3D model). For example, the projected ray may enter and leave the 3D model. To determine the new points, the 3D model generation apparatus 100 may project, associate, and / or attach a plurality of points to each ray. In some examples, each new point may be evenly spaced from the other new points.

[0104] The 3D model generation device 100 can then determine at 508 which of the new points associated with the normal vector in the direction perpendicular to the ray is also tangent to the identified space. Any point identification that is tangent to the identified space (e.g., tangent to the tooth surface determined by the identified interproximal space) can be associated with the interproximal space. In some examples, the point tangent to the identified space can be the initial point of the point cloud associated with the interproximal space. Therefore, points along the ray starting from the point tangent to the identified space should be removed from the point cloud associated with the 3D model. These points can be as described above with reference to FIG. Figure 4 is removed as described in block 406 of FIG. Figure 3-Figure 5 The method is below Figure 6A-6B and Figure 7A-7B is shown in the example shown.

[0105] Fig. 6A and 6B The identification of an interproximal region 608 between two teeth is shown (e.g., by a trained machine learning agent). The region can be extracted from the 2D image, and the boundaries of the interproximal region 606 can be determined, such as Fig. 6A (right side). Figure 6B Another example of a tooth region including an interproximal space 608' is shown, which can be automatically or semi-automatically identified, for example, by a machine learning agent so that the boundaries of the interproximal region 606' can be determined. As described above, the bounded region can be closed, and the surface can be smoothed, extended, or otherwise processed (including as described herein).

[0106] In some examples, after the IP region has been identified (and marked) on the 2D image and the boundaries of the region have been identified, smoothing (e.g., smoothing splines) can be applied to obtain a smooth representation of the boundaries. For each 2D image associated with the 3D model from which the interproximal region has been identified, the camera position corresponding to the position of the camera relative to the 3D data during image capture can be known or determined. In some examples, the 3D captures and WL images can be taken in interlaced scanning cycles. For example, an intraoral scanner can alternate between acquiring a color image and a 3D capture; in some examples, some 3D captures (e.g., structured light data captures) and a color image (e.g., a white light 2D image) can be collected alternately by the intraoral scanner and included with the 3D data, and the method or device can use the 3D data to determine the camera position for each 2D image.

[0107] For example, for each 2D image, the camera position may be refined by interpolating the position of the 3D capture and / or by interpolating the position and adding the IMU information to determine the camera position. In either of these examples, filtering the coarse acceleration may change when the camera position is not accurate enough. In the case of multiple camera views, other methods of determining or refining the camera position may use the images themselves for registration.

[0108] In any of these methods, a certain number of points along the boundary can be chosen and each point used to define a ray in 3D. This may be determined by the camera model. This is illustrated for two different images (taken at two different camera positions). For example, Fig. 7A An example of a plan view of a 3D model of a subject's dentition 600 is shown. The example subject's 3D model 600 can be a complete or partial model, for example, showing a portion of a patient's dental arch (e.g., an upper dental arch). Although shown in terms of an upper dental arch, the procedures and methods described herein can be applied to any portion of a subject's dental arch using a complete or partial 3D model. The 3D model 600 can include a plurality of teeth 610. The 3D model 600 can also include an outer boundary 611 and an inner boundary 612.

[0109] exist Fig. 7A In the example shown, camera 620 is shown at a position associated with a previously captured 2D image. Ray 630 is shown leaving camera 620 and intersecting 3D model 600. Fig. 7A, only one ray 630 is shown for simplicity. Any number of rays may leave the camera 620 at any location. As described above, the rays may be directed to points identified on the interproximal region boundary. Ray 630 enters and intersects the 3D model 600 through the outer boundary 611 (e.g., buccal) and exits through the inner boundary 612 (e.g., lingual). Multiple points 640 are shown on ray 630. Points 640 may be evenly (regularly) spaced and located within the 3D model 600. Each point may be associated with a normal vector perpendicular to ray 630.

[0110] A system such as the 3D model generation device 100 can determine which of the points 640 may be tangent to the tooth. As shown, point 641 is tangent to the surface of tooth 615. As will be described in more detail below, since at least one point 640 is tangent to the tooth surface, points 640 associated with ray 630 may be removed from the point cloud constituting the 3D model 600. Removing these points facilitates the insertion and definition of the interproximal space between two teeth in the 3D model 600.

[0111] Figure 7B Shown with Fig. 7A , showing a second camera position. As described above, in this example, for simplicity, the 3D model 700 of the object can be a partial model showing a portion of the patient's upper dental arch. Although shown in terms of the upper dental arch, the processes and methods described herein can be applied to any portion of the 3D model of the object. The 3D model 700 can include a plurality of teeth 710. The 3D model 700 can also include an outer boundary 711 and an inner boundary 712.

[0112] Camera 720 is shown in a position associated with a previously captured 2D image. Rays 730 are shown that leave camera 720 and intersect 3D model 700. For clarity, only one ray 730 is shown. Any number of rays may leave camera 720 at any location. Ray 730 enters and intersects 3D model 700 through inner boundary 712 (e.g., lingual side) and exits through outer boundary 711 (e.g., buccal side). Multiple points 740 are shown on ray 730. Points 740 may be evenly (regularly) spaced and located within 3D model 600. Each point may be associated with a normal vector perpendicular to ray 730.

[0113] A system such as the 3D model generation device 100 can determine which of the points 740 may be tangent to the tooth. As shown, point 741 is tangent to the surface of tooth 715. Since at least one point 740 is determined to be tangent to the surface of the tooth, the points 740 associated with the ray 730 can be removed from the point cloud constituting the 3D model 700. Removing these points facilitates the insertion and definition of the interproximal space between the two teeth in the 3D model 700.

[0114] In any of these methods and devices, points can be removed by determining where a ray (or point / area of ​​a ray) intersects with each ray previously found using a 3D Poisson reconstruction model. Since the model is (generally speaking) impermeable and the starting position of the ray is outside the model, each ray either does not intersect with the 3D model at all, or may intersect with the 3D model at least twice, the first intersection being with the front face (when entering the model) and the second intersection being with the back face (when leaving the model). As described above, new points with normals along the ray between the intersections can be added. For example, for each ray, new points can be added to the original point cloud (e.g., the original 3D model) at a constant spacing along the ray and between the intersections (between the outer surfaces of the model). Each of these points can be associated with a normal in a direction perpendicular to the ray and tangent to the IP space boundary after being projected into 3D at the position of the point. Conflicting points can then be removed. For example, the above steps (generating rays and adding new points) can be repeated for any number of 2D images including identified interproximal region views. This process adds points to the original point cloud (3D model). The 3D model can then be checked to determine whether the added point falls within the inter-neighboring spatial region of all other 2D images. If so, the new point can be removed from the point cloud (e.g., from the 3D model).

[0115] Figure 8 An example of a refined 3D model is shown, which may be Figure 3 - The result of the operation described in Figure 7. As shown, the resulting 3D model includes the interproximal space 808 as described above.

[0116] Fig. 9 A block diagram of a device 800 is shown, which may be Figure 1 The 3D model generation device 100 is an example of the 3D model generation device 100. Although described as a device in this article, the functions of the device 900 can be performed by any feasible device, system or method. The device 900 may include a device interface 920, a processor 930 and a memory 940.

[0117] The device interface 920 coupled to the processor 930 can be used to interface with any feasible input and / or output device. For example, the device interface 920 can be coupled to and interface with one or more image capture devices 950. Example image capture devices may include 2D image capture devices that are sensitive to white light, near infrared light, fluorescence, or any other feasible light. Some image capture devices 950 may include 3D image capture devices. In another example, the device interface 920 can be coupled to and interface with a display device 960. Through the display device 960, the processor 930 can display a 2D image, a 3D model, a refined 3D model, etc.

[0118] The coupling between the image capture device 950 and the device interface 920 and between the display device 960 and the device interface can be a wired or wireless interface. Example wireless interfaces include Bluetooth, Wi-Fi, cellular long term evolution (LTE), and the like.

[0119] The processor 930 , which is also coupled to the memory 940 , may be any suitable processor or processors capable of executing scripts or instructions of one or more software programs stored in the device 900 , such as within the memory 940 .

[0120] The memory 940 may include 3D model data 942 that has been generated or received by the device 900. For example, preliminary 3D model data may be received from another device via a data interface (not shown). In another example, the 3D model data 942 may also include refined 3D model data. The processor 930 may display any feasible 3D model data on the display device 960.

[0121] The memory 940 may include 2D image data 944. For example, the 2D image data 944 may be captured from any feasible 2D image capture device (e.g., a camera) included in the image capture device 950. In another example, the 2D image data may be received via a data interface (not shown). The 2D image data 944 may include camera position information associated with the 2D image.

[0122] The memory 940 may also include a non-transitory computer-readable storage medium, such as one or more non-volatile storage elements, such as EPROM, EEPROM, flash memory, hard disk drive, etc., which may store the 3D model refinement software module 946. The 3D model refinement software module 946 includes program instructions that, when executed by the processor 930, may cause the device 900 to perform corresponding functions. Therefore, the non-transitory computer-readable storage medium of the memory 940 may include instructions for performing all or part of the operations described herein.

[0123] The processor 930 may execute the 3D model refinement software module 946 to determine the temperature of one or more body locations of the patient. For example, in some cases, the execution of the 3D model refinement software module 946 may refine an existing (preliminary) 3D model of the subject's dentition by adding or refining interproximal space information. In some cases, 2D image information is used to determine the interproximal spaces. Using the 2D image information, a trained neural network may be used to locate the interproximal spaces. In another example, the execution of the 3D model refinement software module 946 may determine one or more rays associated with a camera. The rays may intersect the 3D model and be tangent to the teeth of the 3D model. The execution of the 3D model refinement software module 946 removes portions of the 3D model that may be associated with rays that intersect the 3D model and are tangent to the tooth surface. Thus, the processor 930 may execute the 3D model refinement software module 946 to perform the Figure 3-Figure 5 The associated operation.

[0124] As described above, these methods are not strictly limited to the spaces between teeth; interproximal distances can also refer to raised structures (e.g., tooth surfaces) that are not easily captured in 3D intraoral scans. For example, the posterior wall of the last molar may sometimes be missing in a captured 3D model. The methods and devices described herein can be used to determine the tooth shape in this area, and the wall can be seen in a 2D image (e.g., a white light image), but the 3D capture may not be able to see the wall.

[0125] In addition to refining a 3D model as described herein, the methods can also be configured to output data about the refined interproximal regions. For example, numerical values ​​for interproximal distances can be calculated and added as auxiliary information to any 3D model processed as described herein. For example, the above method can be used to refine the interproximal regions in a 3D model, and the refined model can be segmented at points added between two adjacent teeth; the minimum distance between these teeth can then be determined.

[0126] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are considered to be part of the inventive subject matter disclosed herein and can be used to achieve the benefits described herein.

[0127] The process parameters and order of steps described and / or illustrated herein are provided as examples only and may be changed as desired. For example, although the steps illustrated and / or described herein may be shown or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed. The various exemplary methods described and / or illustrated herein may also omit one or more steps described or illustrated herein, or include additional steps in addition to those steps disclosed.

[0128] Any of the methods described herein (including user interfaces) may be implemented as software, hardware, or firmware, and may be described as a non-transitory computer-readable storage medium storing a set of instructions executable by a processor (e.g., a computer, tablet, smart phone, etc.), which, when executed by the processor, causes the processor to control the execution of any steps, including but not limited to: display, communication with a user, analysis, modification of parameters (including timing, frequency, intensity, etc.), determination, alarm, etc. For example, any of the methods described herein may be performed at least in part by a device including one or more processors having a memory storing a non-transitory computer-readable storage medium storing a set of instructions for the process of the method.

[0129] Although various embodiments have been described and / or illustrated herein in the context of a fully functional computing system, one or more of these example embodiments may be distributed as a program product in various forms, regardless of the specific type of computer-readable medium used to actually perform the distribution. The embodiments disclosed herein may also be implemented using software modules that perform certain tasks. These software modules may include scripts, batches, or other executable files that may be stored on a computer-readable storage medium or in a computing system. In some embodiments, these software modules may configure a computing system to perform one or more of the example embodiments disclosed herein.

[0130] As described herein, the computing devices and systems described and / or illustrated herein broadly represent any type or form of computing device or system capable of executing computer-readable instructions, such as those contained within the modules described herein. In their most basic configuration, these computing devices can each include at least one memory device and at least one physical processor.

[0131] As used herein, the term "memory" or "memory device" generally refers to any type or form of volatile or non-volatile storage device or medium capable of storing data and / or computer-readable instructions. In one example, a memory device can store, load, and / or maintain one or more of the modules described herein. Examples of memory devices include, but are not limited to, random access memory (RAM), read-only memory (ROM), flash memory, hard disk drive (HDD), solid-state drive (SSD), optical drive, cache, variations or combinations of one or more of them, or any other suitable storage memory.

[0132] In addition, as used herein, the term "processor" or "physical processor" generally refers to any type or form of hardware-implemented processing unit capable of interpreting and / or executing computer-readable instructions. In one example, a physical processor can access and / or modify one or more modules stored in the above-mentioned memory device. Examples of physical processors include, but are not limited to, a microprocessor, a microcontroller, a central processing unit (CPU), a field programmable gate array (FPGA) application specific integrated circuit (ASIC) implementing a soft-core processor, a portion of one or more of them, a variation or combination of one or more of them, or any other suitable physical processor.

[0133] Although shown as separate elements, the method steps described and / or illustrated herein may represent portions of a single application. Additionally, in some embodiments, one or more of the steps may represent or correspond to one or more software applications or programs that, when executed by a computing device, may cause the computing device to perform one or more tasks, such as method steps.

[0134] In addition, one or more devices described herein can convert data, physical devices, and / or representations of physical devices from one form to another. Additionally or alternatively, one or more of the modules described herein can transform a processor, volatile memory, non-volatile memory, and / or any other portion of a physical computing device from one form of computing device to another form of computing device by executing on the computing device, storing data on the computing device, and / or otherwise interacting with the computing device.

[0135] As used herein, the term "computer-readable medium" generally refers to any form of device, carrier, or medium capable of storing or carrying computer-readable instructions. Examples of computer-readable media include, but are not limited to, transmission-type media (such as carrier waves) and non-transitory media (such as magnetic storage media (e.g., hard drives, tape drives, and floppy disks), optical storage media (e.g., compact disks (CDs), digital video disks (DVDs), and Blu-ray disks), electronic storage media (e.g., solid-state drives and flash memory media)) and other distribution systems.

[0136] Those of ordinary skill in the art will recognize that any process or method disclosed herein may be modified in many ways. The process parameters and order of steps described and / or illustrated herein are given as examples only and may be changed as desired. For example, although the steps illustrated and / or described herein may be illustrated or discussed in a particular order, the steps do not necessarily need to be performed in the order illustrated or discussed.

[0137] The various exemplary methods described and / or illustrated herein may also omit one or more steps described or illustrated herein, or may include additional steps in addition to those steps disclosed. In addition, the steps of any method disclosed herein may be combined with any one or more steps of any other method disclosed herein.

[0138] A processor as described herein may be configured to perform one or more steps of any method disclosed herein.Alternatively or in combination, a processor may be configured to combine one or more steps of one or more methods as disclosed herein.

[0139] When a feature or element is referred to as being "on" another feature or element in this article, it can be directly on another feature or element, or there can also be intermediate features and / or elements. On the contrary, when a feature or element is referred to as being "directly on" another feature or element "on", there are no intermediate features or elements. It should also be understood that when a feature or element is referred to as being "connected", "attached" or "coupled" to another feature or element, it can be directly connected, attached or coupled to another feature or element, or there can be intermediate features or elements. On the contrary, when a feature or element is referred to as being "directly connected", "directly attached" or "directly coupled" to another feature or element, there are no intermediate features or elements. Although described or shown with respect to one embodiment, the features and elements described or shown in this way can be applied to other embodiments. Those skilled in the art will also understand that the reference to the structure or feature set "adjacent" to another feature can have a part overlapping with the adjacent feature or located below the adjacent feature.

[0140] The terms used herein are only used for the purpose of describing specific embodiments and are not intended to limit the present invention. For example, as used herein, the singular forms "one", "an" and "the" are intended to also include plural forms, unless the context clearly indicates otherwise. It will be further understood that when used in this specification, the terms "include" and / or "comprise" specify the presence of the features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more associated listed items and can be abbreviated as " / ".

[0141] For ease of description, spatial relative terms such as "below", "below", "lower", "above", "upper", etc. may be used herein to describe the relationship between an element or feature as shown in the figure and other elements or features. It should be understood that, in addition to the orientation depicted in the figure, spatial relative terms are intended to cover different orientations of the device in use or operation. For example, if the device in the figure is inverted, the element described as "below" or "below" other elements or features will be oriented to be "above" other elements or features. Therefore, the exemplary term "below" can cover the orientation above and below. The device can be oriented in other ways (rotated 90 degrees or in other orientations), and the spatial relative descriptors used herein are interpreted accordingly. Similarly, the terms "upward", "downward", "vertical", "horizontal", etc. are used herein only for the purpose of explanation unless otherwise explicitly stated.

[0142] Although the terms "first" and "second" may be used herein to describe various features / elements (including steps), these features / elements should not be limited by these terms unless the context indicates otherwise. These terms can be used to distinguish one feature / element from another feature / element. Therefore, without departing from the teachings of the present invention, the first feature / element discussed below can be referred to as the second feature / element, and similarly, the second feature / element discussed below can be referred to as the first feature / element.

[0143] Throughout the specification and the claims that follow, unless the context requires otherwise, the word "comprise" and variations such as "include" and "comprising" mean that various components can be used together in methods and articles (e.g., compositions and devices including apparatus and methods). For example, the term "comprising" will be understood to imply the inclusion of any stated elements or steps but not the exclusion of any other elements or steps.

[0144] Generally, any apparatus and method described herein should be understood to be inclusive, but all or a subset of components and / or steps may alternatively be exclusive and may be expressed as "consisting of various components, steps, sub-components or sub-steps" or alternatively "consisting essentially of various components, steps, sub-components or sub-steps."

[0145] As used herein in the specification and claims, including those used in the examples and unless otherwise expressly stated, all numbers can be interpreted as beginning with the word "about" or "approximately", even if the term does not appear explicitly. When describing a magnitude and / or position to indicate that the described value and / or position is within the reasonable expected range of the value and / or position, the expression "about" or "approximately" can be used. For example, a numerical value can have a value that is + / -0.1% of the value (or range of values), + / -1% of the value (or range of values), + / -2% of the value (or range of values), + / -5% of the value (or range of values), + / -10% of the value (or range of values), etc. Unless the context indicates otherwise, any numerical value given herein should also be understood to include approximately or approximate the value. For example, if the value "10" is disclosed, "about 10" is also disclosed. Any numerical range described herein is intended to include all subranges contained therein. It should also be understood that when a value is disclosed, "less than or equal to" the value, "greater than or equal to the value," and possible ranges between values ​​are also disclosed, as appropriately understood by those skilled in the art. For example, if the value "X" is disclosed, "less than or equal to X" and "greater than or equal to X" (e.g., where X is a numerical value) are also disclosed. It should also be understood that throughout the application, data is provided in a variety of different formats, and that the data represents endpoints and starting points, as well as ranges for any combination of data points. For example, if a specific data point "10" and a specific data point "15" are disclosed, it should be understood that greater than, greater than or equal to, less than, less than or equal to, and equal to 10 and 15, and between 10 and 15 are considered disclosed. It should also be understood that every unit between two specific units is also disclosed. For example, if 10 and 15 are disclosed, 11, 12, 13, and 14 are also disclosed.

[0146] Although various illustrative embodiments are described above, any of a number of changes may be made to the various embodiments without departing from the scope of the invention as described in the claims. For example, in alternative embodiments, the order in which the various described method steps are performed may generally be changed, and in other alternative embodiments, one or more method steps may be skipped entirely. Optional features of the various device and system embodiments may be included in some embodiments and not included in other embodiments. Therefore, the foregoing description is provided primarily for exemplary purposes and should not be construed as limiting the scope of the invention as set forth in the claims.

[0147] The examples and illustrations included herein show by way of illustration and not limitation the specific embodiments in which the subject matter can be practiced. As described above, other embodiments can be utilized and derived therefrom so that structural and logical replacements and changes can be made without departing from the scope of the present disclosure. These embodiments of the subject matter of the present invention may be referred to herein individually or collectively by the term "invention", which is merely for convenience and is not intended to voluntarily limit the scope of the present application to any single invention or inventive concept (if more than one is actually disclosed). Therefore, although specific embodiments have been illustrated and described herein, any arrangement calculated to achieve the same purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all adaptations or changes of various embodiments. After reading the above description, the combination of the above-mentioned embodiments and other embodiments not specifically described herein will be apparent to those skilled in the art.

Claims

1. A method comprising: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; generating a refined 3D model from the 3D model of the subject's dentition using one or more edge boundaries of spaces around one or more teeth or between one or more teeth in the 3D model identified from a plurality of two-dimensional (2D) images of the subject's dentition to which the plurality of 2D images correspond; and The refined 3D model and / or information associated with the refined 3D model is output.

2. The method according to claim 1, wherein: The 3D model or data of the 3D model and the 2D image are received as part of the same data stream.

3. The method according to claim 1, wherein: Generating the refined 3D model includes: identifying a marginal boundary around the space from the 2D image of the subject's dentition; and Potential new surface points of the 3D model are generated based on the identified edge boundaries for the plurality of 2D images.

4. The method according to claim 3, further comprising: New surface points falling within one or more of the edge boundaries surrounding the space from the 2D image are removed from the 3D model.

5. The method according to claim 3, wherein: Identifying the edge boundary of the space around the one or more teeth or between the one or more teeth includes: for each subset of at least one subset of the multiple 2D images, using a trained neural network to identify the boundary of the space around the one or more teeth or between the one or more teeth from each 2D image of the 2D images of the subject's dentition in the subset.

6. The method according to claim 5, further comprising: Smooth the edges of each identified boundary.

7. The method according to claim 3, wherein: Generating potential new surface points of the 3D model based on the identified edge boundaries includes using a position of a camera corresponding to each of the plurality of 2D images.

8. The method according to claim 7, further comprising: mapping the identified edge boundaries to the 3D model and determining rays formed between a plurality of points on each of the edge boundaries and a position of the camera relative to a surface of the subject's dentition during capture of a 2D image of the subject's dentition; as well as New points of the 3D model are generated between points on the surface of the 3D model at the location where each ray enters and leaves the surface of the 3D model.

9. The method according to claim 8, wherein: Each point has an associated normal vector in a direction perpendicular to the ray and to the identified tangent to the space.

10. The method according to claim 1, wherein: The space includes an interproximal space between two teeth in the subject's dentition.

11. The method according to claim 10, wherein: The information output includes an indication of whether the two teeth are in contact.

12. The method according to claim 10, wherein: The information output includes a measurement of the space between the two teeth.

13. The method according to claim 1, wherein: Outputting the information includes displaying the information to a user.

14. The method according to claim 1, wherein: Outputting the information includes outputting to a segmentation algorithm, and wherein the segmentation algorithm separates the refined 3D model into individual teeth.

15. The method according to claim 1, wherein: The plurality of 2D images are white light images.

16. The method according to claim 1, wherein: The plurality of 2D images are near infrared images.

17. The method according to claim 1, wherein: The plurality of 2D images are fluorescence images.

18. A method comprising: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of a same scan of the subject's dentition; generating a refined 3D model from the 3D model of the subject's dentition using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from the plurality of 2D images of the subject's dentition, wherein generating the refined 3D model comprises identifying edge boundaries around the space from the 2D images of the subject's dentition and generating new surface points of the refined 3D model based on the identified edge boundaries; and The refined 3D model and / or information associated with the refined 3D model is output.

19. A method comprising: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of a same scan of the subject's dentition; A refined 3D model is generated based on the 3D model of the subject's dentition by the following steps: identifying edge boundaries around one or more teeth or around spaces between one or more teeth from a 2D image of the subject's dentition using a trained neural network to thereby identify the spaces, and smoothing edges of the identified spaces to thereby form the edge boundaries around the spaces, generating potential new surface points of the refined 3D model according to the identified edge boundary by mapping the identified edge boundary to the 3D model and casting rays between a plurality of points on the edge boundary and a position of a camera relative to a surface of the subject's dentition during capture of the 2D image of the subject's dentition, using a camera position corresponding to the 2D image, and generating one or more new points of the 3D model between points on the surface of the 3D model at a position where each ray enters and exits the surface of the 3D model, repeating the steps of identifying the edge boundaries around the space from a 2D image and generating potential new surface points of the refined 3D model based on the identified edge boundaries for a plurality of 2D images taken from different locations, and removing new surface points from the refined 3D model that fall within one or more of the edge boundaries surrounding the space from the 2D image; and The refined 3D model and / or information associated with the refined 3D model is output.

20. A system comprising: one or more processors; as well as a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model; Refining the 3D model using edge boundaries of spaces around one or more teeth or between one or more teeth in the 3D model identified from the plurality of 2D images of the subject's dentition to generate a refined 3D model; and Information associated with the refined 3D model is output.

21. The system of claim 20, wherein: Executing the instructions to refine the 3D model enables the system to further: identifying a marginal boundary around the space from the 2D image of the subject's dentition; and Potential new surface points of the 3D model are generated based on the identified edge boundaries for the plurality of 2D images.

22. The system of claim 21, wherein: Execution of the instructions enables the system to remove new surface points from the 3D model that fall within one or more of the edge boundaries surrounding the space from the 2D image.

23. The system of claim 21, wherein: Executing the instructions to identify the edge boundaries of the space around the one or more teeth or between the one or more teeth enables the system to use machine learning to identify, for each subset of at least one subset of the multiple 2D images, the boundaries of the space around one or more teeth or between one or more teeth in each of the 2D images of the subject's dentition in the subset.

24. The system of claim 23, wherein: Execution of the instructions enables the system to smooth the edges of each identified boundary.

25. The system of claim 21, wherein: Executing the instructions to generate potential new surface points of the 3D model based on the identified edge boundaries enables the system to use a position of a camera corresponding to each of the plurality of 2D images.

26. The system of claim 25, wherein: Execution of the instructions enables the system to further: mapping the identified edge boundaries to the 3D model and determining rays formed between a plurality of points on each of the edge boundaries and a position of the camera relative to a surface of the subject's dentition during capture of the 2D image of the subject's dentition; and New points of the 3D model are generated between points on the surface of the 3D model at the location where each ray enters and leaves the surface of the 3D model.

27. The system of claim 26, wherein: Each point has an associated normal vector in a direction perpendicular to the ray and to the identified tangent to the space.

28. The system of claim 20, wherein: The space includes the interproximal space between two teeth of the subject's dentition.

29. The system of claim 28, wherein: The information output includes an indication of whether two teeth are in contact.

30. The system of claim 28, wherein: The information output includes a measurement of the space between two teeth.

31. The system of claim 20, wherein: Executing the instructions to output information associated with the refined 3D model enables the system to display the information to a user.

32. The system of claim 20, wherein: Execution of the instructions to output information associated with the refined 3D model enables the system to output to a segmentation algorithm, and wherein the segmentation algorithm is configured to separate the refined 3D model into individual teeth.

33. The system of claim 20, wherein: The plurality of 2D images are white light images.

34. The system of claim 20, wherein: The plurality of 2D images are near infrared images.

35. The system of claim 20, wherein: The plurality of 2D images are fluorescence images.

36. A system comprising: one or more processors; as well as a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of a same scan of the subject's dentition; Refining the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from the plurality of 2D images of the subject's dentition to generate a refined 3D model, wherein refining the 3D model comprises identifying edge boundaries around the space from the 2D images of the subject's dentition and generating potential new surface points of the 3D model based on the edge boundaries identified for the plurality of 2D images; and Information associated with the refined 3D model is output.

37. A system comprising: one or more processors; as well as a memory configured to store instructions that, when executed by the one or more processors, enable the system to: receiving or generating a three-dimensional (3D) model of the subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of a same scan of the subject's dentition; Refining the 3D model to generate a refined 3D model, wherein the refinement enables the system to: identifying edge boundaries around one or more teeth or around a space between one or more teeth from a 2D image of the subject's dentition using machine learning to identify the space, and smoothing edges of the identified space to form the edge boundaries around the space, generating potential new surface points of the 3D model from the identified edge boundary using the position of the camera corresponding to the 2D image by mapping the identified edge boundary to the 3D model and determining rays formed between a plurality of points on the edge boundary and a position of a camera relative to a surface of the subject's dentition during capture of the 2D image of the subject's dentition, and generating one or more new points of the 3D model between points on the surface of the 3D model at a position where each ray enters and exits the surface of the 3D model, repeating the steps of identifying the edge boundaries around the space from the 2D image and generating potential new surface points of the 3D model based on the identified edge boundaries for a plurality of 2D images taken from different locations, and removing from the 3D model new surface points that fall within one or more of the edge boundaries surrounding the space from the 2D image; and Information associated with the refined 3D model is output.

38. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, enable the device to perform operations comprising: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model; Refining the 3D model using edge boundaries of spaces around one or more teeth or between one or more teeth in the 3D model identified from the plurality of 2D images of the subject's dentition to generate a refined 3D model; and Information associated with the refined 3D model is output.

39. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, enable the device to perform operations comprising: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of a same scan of the subject's dentition; Refining the 3D model using edge boundaries of a space around one or more teeth or between one or more teeth in the 3D model identified from the plurality of 2D images of the subject's dentition to generate a refined 3D model, wherein refining the 3D model comprises identifying edge boundaries around the space from the 2D images of the subject's dentition and generating potential new surface points for the 3D model based on the edge boundaries identified for the plurality of 2D images; and Information associated with the refined 3D model is output.

40. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, enable a device to perform operations comprising: receiving or generating a three-dimensional (3D) model of a subject's dentition, wherein the 3D model is based on a scan of the subject's dentition; receiving or generating a plurality of two-dimensional (2D) images of the subject's dentition corresponding to the 3D model taken as part of a same scan of the subject's dentition; Refining the 3D model to generate a refined 3D model, wherein the refinement enables the system to: identifying edge boundaries around one or more teeth or around spaces between one or more teeth from a 2D image of the subject's dentition using machine learning, thereby identifying the spaces, and smoothing edges of the identified spaces to form the edge boundaries around the spaces, generating potential new surface points of the 3D model from the identified edge boundary using the position of the camera corresponding to the 2D image by mapping the identified edge boundary to the 3D model and determining rays formed between a plurality of points on the edge boundary and a position of a camera relative to a surface of the subject's dentition during capture of the 2D image of the subject's dentition, and generating one or more new points of the 3D model between points on the surface of the 3D model at a position where each ray enters and exits the surface of the 3D model, repeating the steps of identifying the edge boundaries around the space from the 2D image and generating potential new surface points of the 3D model based on the identified edge boundaries for a plurality of 2D images taken from different locations, and removing from the 3D model new surface points that fall within one or more of the edge boundaries surrounding the space from the 2D image; and Information associated with the refined 3D model is output.

41. A method comprising: Capturing three-dimensional (3D) information of the surface of the subject's dentition; generating a 3D model of the surface of the subject's dentition based on the 3D information; capturing a two-dimensional (2D) image of a surface of the subject's dentition; identifying edge information of the surface from the captured 2D image, wherein the edge information is tangent to the surface of the subject's dentition; Refining the generated 3D model using the edge information of the surface identified from the captured 2D image, thereby generating a refined 3D model; and Information associated with the refined 3D model is output.

42. The method according to claim 41, wherein: The surface of the subject dentition includes an interproximal space between two teeth of the subject dentition, and wherein the edge information includes edges of the teeth in the interproximal space.

43. The method of claim 41, wherein: The information output includes an indication of whether two teeth are in contact.

44. The method of claim 41, wherein: The information output includes measurements of the interproximal space between two teeth.

45. The method of claim 41, wherein: Outputting the information includes outputting to a user display.

46. ​​The method of claim 41, wherein: Outputting the information includes outputting to a segmentation algorithm, and wherein the segmentation algorithm separates the refined 3D model into individual teeth.

47. The method of claim 41, wherein: Identifying the edge information includes analyzing the captured 2D image using a trained neural network.

48. The method of claim 41, wherein: The 2D image is a white light image.

49. The method of claim 41, wherein: The 2D image is a near infrared image.

50. The method of claim 41, wherein: The 2D image is a fluorescence image.

51. A method comprising: capturing three-dimensional (3D) information of a surface of a subject's dentition by projecting a structured light pattern onto the surface of the subject's dentition and capturing the projected structured light pattern with one or more cameras; generating a 3D model of the surface of the subject's dentition based on the 3D information using a correspondence algorithm or a triangulation algorithm; identifying at least partial edge information of the surface from the captured two-dimensional (2D) image from the one or more cameras based on penetration of the structured light pattern in the tooth, wherein the edge information is tangential to the surface of the subject's dentition; Refining the generated 3D model using the edge information of the surface identified from the respective captured 2D images, thereby generating a refined 3D model; and Information associated with the refined 3D model is output.

52. A system comprising: one or more processors; as well as a memory configured to store instructions that, when executed by the one or more processors, enable the system to: Capturing three-dimensional (3D) information of the surface of the subject's dentition; generating a 3D model of the surface of the subject's dentition based on the 3D information; capturing a two-dimensional (2D) image of a surface of the subject's dentition; identifying edge information of the surface from the captured 2D image, wherein the edge information is tangent to the surface of the subject's dentition; Refining the generated 3D model using the edge information of the surface identified from the captured 2D image, thereby generating a refined 3D model; and Information associated with the refined 3D model is output.

53. The system of claim 52, wherein: The surface of the subject dentition includes an interproximal space between two teeth of the subject dentition, and wherein the edge information includes edges of the teeth in the interproximal space.

54. The system of claim 52, wherein: The information output includes an indication of whether two teeth are in contact.

55. The system of claim 52, wherein: The information output includes a measurement of the interproximal distance between two teeth.

56. The system of claim 52, wherein: The outputted information is outputted to a user display.

57. The system of claim 52, wherein: The output information is output to a segmentation algorithm, and wherein the segmentation algorithm separates the refined 3D model into individual teeth.

58. The system of claim 52, wherein: Executing the instructions to identify the edge information enables the system to analyze the captured 2D image using a trained neural network.

59. The system of claim 52, wherein: The 2D image is a white light image.

60. The system of claim 52, wherein: The 2D image is a near infrared image.

61. The system of claim 52, wherein: The 2D image is a fluorescence image.

62. A system comprising: one or more processors; as well as a memory configured to store instructions that, when executed by the one or more processors, enable the system to: capturing three-dimensional (3D) information of a surface of a subject's dentition by projecting a structured light pattern onto the surface of the subject's dentition and capturing the projected structured light pattern with one or more cameras; generating a 3D model of the surface of the subject's dentition based on the 3D information using a correspondence algorithm or a triangulation algorithm; identifying at least a portion of edge information of the surface from the captured two-dimensional (2D) image from the one or more cameras, wherein the edge information is tangential to a surface of the subject's dentition; Refining the generated 3D model using the edge information of the surface identified from the respective captured 2D images, thereby generating a refined 3D model; and Information associated with the refined 3D model is output.

63. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, enable the device to perform operations comprising: Capturing three-dimensional (3D) information of the surface of the subject's dentition; generating a 3D model of the surface of the subject's dentition based on the 3D information; capturing a two-dimensional (2D) image of a surface of the subject's dentition; identifying edge information of the surface from the captured 2D image, wherein the edge information is tangent to the surface of the subject's dentition; Refining the generated 3D model using the edge information of the surface identified from the captured 2D image, thereby generating a refined 3D model; and Information associated with the refined 3D model is output.

64. A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, enable the device to perform operations comprising: capturing three-dimensional (3D) information of a surface of a subject's dentition by projecting a structured light pattern onto the surface of the subject's dentition and capturing the projected structured light pattern with one or more cameras; generating a 3D model of the surface of the subject's dentition based on the 3D information using a correspondence algorithm or a triangulation algorithm; identifying at least partial edge information of the surface from the captured two-dimensional (2D) image from the one or more cameras based on penetration of the structured light pattern in the tooth, wherein the edge information is tangential to the surface of the subject's dentition; Refining the generated 3D model using the edge information of the surface identified from the respective captured 2D images, thereby generating a refined 3D model; and Information associated with the refined 3D model is output.

Images