Systems and methods for scanning teeth for dental restorations

Abstract

An intraoral scanning method includes receiving a plurality of intraoral scans of a dental point during an intraoral scanning session, generating a three-dimensional (3D) surface of the dental point from the plurality of intraoral scans, identifying hard tissue and soft tissue in at least one of a) the plurality of intraoral scans of the dental point or b) the 3D surface of the dental point, and displaying a view of the 3D surface, the first visualization is used to display a first portion of the 3D surface identified as hard tissue, and the second visualization is used to display a second portion of the 3D surface identified as soft tissue.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of dentistry, and in particular, to techniques for processing intraoral scans of preparation teeth and visualizing other dental points generated from the intraoral scans and a three-dimensional (3D) surface of the preparation teeth. Background Art

[0002] For dental restorative work such as crowns and bridges, one or more intraoral scans can be generated for the prepared teeth and / or surrounding teeth on the patient's dental arch using an intraoral scanner. In the case of subgingival preparation, the gums cover at least part of the margin line (also referred to herein as the finish line) and are retracted to fully expose the margin line. Therefore, the intraoral scan is usually generated after the doctor wraps the tooth retraction line (also referred to as the packaging line or retraction line) under the gums (gum) around the prepared teeth, then pulls out the retraction line, and briefly exposes the subgingival margin line. The process of wrapping the retraction line between the preparation body and the gums is long, and each preparation body may take about 10 minutes to complete. In addition, the process is painful for the patient and can damage the gums. After the retraction line has been wrapped around the prepared teeth and then the retraction line is pulled out, the intraoral scan performed must be performed within a narrow time window when the gums shrink back above the margin line. If enough intraoral scans are not generated before the gums shrink, the process needs to be repeated. Once enough intraoral scans are generated, these scans are used to generate a virtual three-dimensional (3D) model of the dental points including the prepared teeth and the surrounding teeth and gums. For example, a virtual 3D model of a patient's dental arch can be generated. The virtual 3D model can then be sent to a laboratory. Summary of the invention

[0003] Several example implementations of the present disclosure are described herein.

[0004] In a first embodiment, a method includes receiving a plurality of intraoral scans of a dental site during an intraoral scanning session, generating a three-dimensional (3D) surface of the dental site based on the plurality of intraoral scans, identifying hard tissue and soft tissue in at least one of a) the plurality of intraoral scans of the dental site or b) the 3D surface of the dental site, and displaying a view of the 3D surface wherein at least one of a first visualization or a first transparency level is used for a first portion of the 3D surface identified as hard tissue and at least one of a second visualization or a second transparency level is used for a second portion of the 3D surface identified as soft tissue.

[0005] A second embodiment may further expand upon the first embodiment. In the second embodiment, identifying hard tissue and soft tissue includes: processing at least one of a) a plurality of intraoral scans or b) data from a 3D surface using a trained machine learning model that has been trained to identify hard tissue and soft tissue, wherein the trained machine learning model outputs a first classification indicating hard tissue or a second classification indicating soft tissue for each location in the plurality of intraoral scans or the 3D surface.

[0006] The third embodiment can further expand the second embodiment. In the third embodiment, the trained machine learning model or the second trained machine learning model further outputs, for each position, a third classification identifying the position as part of an edge line, or a fourth classification identifying the position as not part of an edge line.

[0007] The fourth embodiment may extend any one of the first embodiment to the third embodiment. In the fourth embodiment, the first visualization includes an opaque visualization, and the second visualization includes a semi-transparent visualization.

[0008] A fifth embodiment may extend the fourth embodiment. In a fifth embodiment, the hard tissue includes teeth and the soft tissue includes gums, and wherein a scanned portion of the teeth below the gum line is visible through a translucent visualization for the gums.

[0009] The sixth embodiment can extend any one of the first embodiment to the fifth embodiment. In the sixth embodiment, the dental point includes a prepared tooth, and the method further includes: identifying a margin line around at least a portion of the prepared tooth in at least one of a) one or more intraoral scans of a plurality of intraoral scans or b) data from a 3D surface, and displaying the margin line on the 3D surface using one or more additional visualizations.

[0010] A seventh embodiment may extend the sixth embodiment. In the seventh embodiment, the method further comprises: grading the plurality of portions of the edge line, wherein the one or more additional visualizations comprises a plurality of visualizations, wherein each visualization in the plurality of visualizations is associated with a different edge line grade, and wherein each portion of the plurality of portions of the edge line is displayed using a corresponding one of the plurality of visualizations associated with the edge line grade of the portion.

[0011] An eighth embodiment may extend the seventh embodiment. In an eighth embodiment, a margin line is identified by processing at least one of a) a plurality of intraoral scans or b) data from a 3D surface using a trained machine learning model that has been trained to identify margin lines, wherein for each location, the trained machine learning model outputs a confidence rating and an indication of whether the location depicts a margin line rating, wherein locations identified as depicting a margin line and having a confidence rating that meets or exceeds a confidence threshold are assigned a higher margin line rating than locations identified as depicting a margin line and having a confidence rating that does not meet the confidence threshold.

[0012] A ninth embodiment may extend the eighth embodiment. In the ninth embodiment, the method further comprises: for each position, determining whether the position is associated with at least one of the following: a) a curvature sharpness below a curvature sharpness threshold, b) an s-shaped curve, c) a gap in the edge line, or d) less than a threshold value of the number of intraoral scans; and, downgrading those positions associated with at least one of the following: a) a curvature sharpness below a curvature sharpness threshold, b) an s-shaped curve, c) a gap in the edge line, or d) less than a threshold value of the number of intraoral scans.

[0013] The tenth embodiment can extend any one of the first to ninth embodiments. In the tenth embodiment, one or more intraoral scans of the plurality of intraoral scans include a representation of a dental tool, and the method further includes: identifying the dental tool in at least one of a) the one or more intraoral scans including the representation of the dental tool or b) the 3D surface; and removing the dental tool from at least one of a) the one or more intraoral scans or b) the 3D surface.

[0014] The eleventh embodiment can extend any one of the first to tenth embodiments. In the eleventh embodiment, the method further includes: receiving an additional intraoral scan of a dental site; adding data from the additional intraoral scan to the 3D surface; updating the view of the 3D surface, wherein the data from the additional intraoral scan is semi-transparent in the updated view of the 3D surface; then segmenting the data from the additional intraoral scan into hard tissue and soft tissue; and then updating the view of the 3D surface so that the data from the additional intraoral scan associated with the hard tissue is not transparent, and the data from the additional intraoral scan associated with the soft tissue remains semi-transparent.

[0015] The twelfth embodiment can extend any one of the first to eleventh embodiments. In the twelfth embodiment, the method further includes: receiving an additional intraoral scan of a dental site; adding data from the additional intraoral scan to the 3D surface; updating the view of the 3D surface, wherein the data from the additional intraoral scan is semi-transparent in the updated view of the 3D surface; then identifying at least one of a moving tissue or a dental tool in the data from the additional intraoral scan; removing at least one of the moving tissue or the dental tool from the 3D surface; and then updating the view of the 3D surface to reflect the removed moving tissue or the removed dental tool.

[0016] A thirteenth embodiment may extend the twelfth embodiment. In the thirteenth embodiment, the method further comprises locking one or more regions identified as hard tissue in at least one of a) the plurality of intraoral scans or b) the 3D surface, wherein those regions identified as moving tissue or dental tools and those locked regions are not removed from the 3D surface.

[0017] The fourteenth embodiment can extend any one of the first embodiment to the thirteenth embodiment. In the fourteenth embodiment, the method further includes: determining whether the dental site includes a prepared tooth; in response to determining that the prepared tooth is not detected, using a first moving tissue detection algorithm to identify and remove moving tissue from at least one of the multiple intraoral scans or 3D surfaces; and, in response to determining that the prepared tooth is detected, using a second moving tissue detection algorithm to identify and remove moving tissue from at least one of the multiple intraoral scans or 3D surfaces, wherein the second moving tissue detection algorithm is more aggressive than the first moving tissue detection algorithm in identifying the moving tissue.

[0018] The fifteenth embodiment may expand on the fourteenth embodiment. In the fifteenth embodiment, the method further comprises: receiving a user input indicating that the dental point includes preparing the tooth.

[0019] The sixteenth embodiment may extend the fourteenth embodiment or the fifteenth embodiment. In the sixteenth embodiment, the method further comprises: analyzing at least one of the plurality of intraoral scans or 3D surfaces to determine the dental location comprises preparing the tooth.

[0020] The seventeenth embodiment may expand any one of the first to sixteenth embodiments. In the seventeenth embodiment, the first transparency level includes 0-50% transparency, and wherein the second transparency level includes 51-100% transparency.

[0021] The eighteenth embodiment may extend any one of the first to seventeenth embodiments. In the eighteenth embodiment, the second transparency level includes 100% transparency, and the second portion of the 3D surface identified as the soft tissue is invisible due to the 100% transparency.

[0022] The nineteenth embodiment can extend any one of the first to eighteenth embodiments. In the nineteenth embodiment, the 3D surface is a 3D surface of a prepared tooth, and the method further comprises overlaying the 3D surface onto a second 3D surface of a dental arch including the prepared tooth, wherein the gums from the second 3D surface are displayed using a semi-transparent visualization.

[0023] The twentieth embodiment can extend any one of the first embodiment to the nineteenth embodiment. In the twentieth embodiment, the method further includes: receiving a first user input specifying at least one of the first visualization or the first transparency level; and receiving a second user input specifying at least one of the second visualization or the second transparency level.

[0024] The twenty-first embodiment can extend any one of the first to twentieth embodiments. In the twenty-first embodiment, the method further includes: receiving coordinates input by a user; determining a first tooth closest to the coordinates; displaying the first tooth closest to the coordinates using at least one of a first visualization or a first transparency level; and displaying a second tooth using at least one of a second visualization or a second transparency level.

[0025] The twenty-second embodiment can extend the twenty-first embodiment. In the twenty-second embodiment, the method further includes: receiving new coordinates input by a new user; determining that the second tooth is the tooth closest to the new coordinates; displaying the second tooth closest to the new coordinates using at least one of a first visualization or a first transparency level; and displaying the first tooth using at least one of a second visualization or a second transparency level.

[0026] The twenty-third embodiment may expand the twenty-first embodiment or the twenty-second embodiment. In the twenty-third embodiment, at least one of the proximal surface or the distal surface of the first tooth is visible by displaying the second tooth using at least one of the second visualization or the second transparency level.

[0027] The twenty-fourth embodiment may extend any one of the twenty-first embodiment to the twenty-third embodiment. In the twenty-fourth embodiment, receiving the coordinates input by the user includes receiving a user input of dragging the prompt feature to the coordinates.

[0028] The twenty-fifth embodiment can extend any one of the twenty-first embodiment to the twenty-fourth embodiment. In the twenty-fifth embodiment, the first tooth is a prepared tooth having a margin line, and wherein determining the first tooth closest to the coordinate comprises: identifying the margin line of the prepared tooth; determining that the point on the 3D surface closest to the coordinate is within the margin line in the plane; and classifying the point on the prepared tooth within the margin line as the prepared tooth.

[0029] The twenty-sixth embodiment can extend any one of the twenty-first embodiment to the twenty-fourth embodiment. In the twenty-fifth embodiment, determining the first tooth closest to the coordinates includes: performing one or more morphological operations to divide the 3D surface into a plurality of parts corresponding to different teeth; finding the point on the 3D surface closest to the coordinates; and selecting the part associated with the first tooth, the part including the point on the 3D surface closest to the coordinates.

[0030] In a twenty-seventh embodiment, a computing device including a memory and a processing device performs the operations of any one of the first to twenty-sixth embodiments.

[0031] In a twenty-eighth embodiment, a non-transitory computer-readable medium includes instructions, which, when executed by a processing device, causes the processing device to perform the method of any one of the first to twenty-sixth embodiments.

[0032] In a twenty-ninth embodiment, a non-transitory computer-readable medium comprises instructions that, when executed by a processing device, cause the processing device to perform operations, including: displaying a view of a 3D surface of a dental site during an intraoral scan, the 3D surface generated based on one or more previously received intraoral scans captured during the intraoral scan, wherein a first visualization is used to display a first portion of the 3D surface representing hard tissue and a second visualization is used to display a second portion of the 3D surface identified as soft tissue; receiving a new intraoral scan of the dental site during the intraoral scanning session; adding data from the new intraoral scan to the 3D surface; and, updating the view of the 3D surface, wherein data from the new intraoral scan is semi-transparent in the updated view of the 3D surface until the data from the new intraoral scan is analyzed.

[0033] The 30th embodiment can extend the 29th embodiment. In the 30th embodiment, the operation also includes: analyzing the data from the new intraoral scan to identify hard tissue and soft tissue in the new intraoral scan of the dental point; and then updating the view of the 3D surface so that the data from the new intraoral scan associated with the hard tissue is opaque and the data from the new intraoral scan associated with the soft tissue remains semi-transparent.

[0034] The thirty-first implementation may extend the thirtieth implementation. In the thirty-first implementation, analysis is performed by processing data from a new intraoral scan using a trained machine learning model that has been trained to identify hard tissue and soft tissue, wherein the trained machine learning model outputs a first classification indicating hard tissue or a second classification indicating soft tissue for each location in the data from the new intraoral scan.

[0035] The thirty-second embodiment may expand on the thirtieth embodiment or the thirty-first embodiment. In the thirty-second embodiment, the hard tissue includes teeth, and the soft tissue includes gums, and wherein a scanned portion of the teeth below the gum line is visible through the translucent gums.

[0036] The thirty-third embodiment can extend any one of the twenty-ninth embodiment to the thirty-second embodiment. In the thirty-third embodiment, the dental point includes preparing a tooth, and the operation further includes: identifying a margin line in a 3D surface surrounding at least a portion of the prepared tooth; and using one or more visualizations to emphasize the margin line on the 3D surface.

[0037] The thirty-fourth embodiment can further extend the thirty-third embodiment. In the thirty-fourth embodiment, the operation also includes: grading the plurality of portions of the edge line, wherein the one or more visualizations include a plurality of visualizations, wherein each of the plurality of visualizations is associated with a different edge line grade, and wherein each of the plurality of portions of the edge line is displayed using a corresponding one of the plurality of visualizations associated with the edge line grade of the portion.

[0038] The thirty-fifth embodiment can further expand the thirty-fourth embodiment. In the thirty-fifth embodiment, edge lines are identified by processing data from a 3D surface using a trained machine learning model that has been trained to identify edge lines, wherein for each position, the trained machine learning model outputs a confidence level and an indication of whether the position depicts an edge line level, wherein positions identified as depicting edge lines and having a confidence level that meets or exceeds a confidence threshold are assigned a higher edge line level than positions identified as depicting edge lines and having a confidence level that does not meet the confidence threshold.

[0039] The thirty-sixth embodiment can further expand the thirty-fifth embodiment. In the thirty-sixth embodiment, the operation also includes: for each position, determining whether the position is associated with at least one of the following: a) a curvature sharpness below a curvature sharpness threshold, b) an s-shaped curve, c) a gap in the edge line, or d) a threshold value less than the number of intraoral scans; and, reducing the level of those positions associated with at least one of the following: a) a curvature sharpness below a curvature sharpness threshold, b) an s-shaped curve, c) a gap in the edge line, or d) a threshold value less than the number of intraoral scans.

[0040] The thirty-seventh embodiment can further extend any one of the twenty-ninth embodiment to the thirty-sixth embodiment. In the thirty-seventh embodiment, the operation further includes: identifying at least one of a moving tissue or a dental tool in data from a new intraoral scan; removing at least one of the moving tissue or the dental tool from the 3D surface; and then updating the view of the 3D surface to reflect at least one of the removed moving tissue or the removed dental tool.

[0041] The thirty-eighth embodiment can further extend the thirty-seventh embodiment. In the thirty-eighth embodiment, the operation further includes: locking one or more regions identified as hard tissue in at least one of the new intraoral scan or the 3D surface, wherein those regions identified as moving tissue or dental tools and those locked regions are not removed from the 3D surface.

[0042] In a thirty-ninth embodiment, the non-transitory computer-readable medium according to any one of the twenty-ninth to thirty-eighth embodiments is a component of a computing device, which includes the non-transitory computer-readable medium and a processing device that performs an operation.

[0043] In a fortieth embodiment, a system includes: an intraoral scanner configured to generate a plurality of intraoral scans of a dental site, and a computing device including a memory and a processor. The computing device is configured to: receive a plurality of intraoral scans of the dental site; generate a three-dimensional (3D) surface of the dental site based on the plurality of intraoral scans; display a view of the 3D surface of the dental site; determine whether the dental site includes a prepared tooth; in response to determining that the prepared tooth is not detected, use a first moving tissue detection algorithm to identify and remove moving tissue from at least one of the plurality of intraoral scans or the 3D surface; and, in response to determining that the prepared tooth is detected, use a second moving tissue detection algorithm to identify and remove moving tissue from at least one of a) the plurality of intraoral scans or b) the 3D surface, wherein the second moving tissue detection algorithm identifies some locations identified by the first moving tissue detection algorithm as not moving tissue as moving tissue.

[0044] The forty-first embodiment may expand on the fortieth embodiment. In the forty-first embodiment, the computing device further receives a user input indicating that the dental point includes preparing the tooth.

[0045] The 42nd embodiment may expand on the 40th embodiment or the 41st embodiment. In the 42nd embodiment, the computing device is further configured to analyze at least one of a plurality of intraoral scans or 3D surfaces to determine that the dental location includes a prepared tooth.

[0046] The forty-third embodiment can expand any one of the fortieth embodiment to the forty-second embodiment. In the forty-third embodiment, the computing device is further used to analyze data from multiple intraoral scans to identify hard tissue and soft tissue, wherein the first visualization is used to display a first portion of the 3D surface representing the hard tissue, and the second visualization is used to display a second portion of the 3D surface identified as soft tissue.

[0047] The forty-fourth embodiment can extend the forty-third embodiment. In the forty-fourth embodiment, the data is analyzed by processing the data using a trained machine learning model that has been trained to identify hard tissue and soft tissue, wherein the trained machine learning model outputs a first classification indicating hard tissue or a second classification indicating soft tissue for each location in the data.

[0048] The forty-fifth embodiment may extend the forty-third embodiment or the forty-fourth embodiment. In the forty-fifth embodiment, the first visualization includes an opaque visualization, wherein the second visualization includes a translucent visualization, wherein the hard tissue includes teeth, wherein the soft tissue includes gums, and wherein a scanned portion of the teeth below the gum line is visible through the translucent visualization of the gums.

[0049] In a forty-sixth embodiment, a method includes: receiving multiple intraoral scans of a dental location during an intraoral scanning session; generating a three-dimensional (3D) surface of the dental location based on the multiple intraoral scans, the dental location including multiple teeth; receiving coordinates; determining a first tooth among the multiple teeth that is closest to the coordinates; using a first visualization to display the first tooth closest to the coordinates; and, using a second visualization to display the remaining teeth among the multiple teeth.

[0050] The forty-seventh embodiment may further extend the forty-sixth embodiment. In the forty-seventh embodiment, the method further comprises: performing one or more morphological operations to divide the 3D surface into a plurality of teeth.

[0051] The forty-eighth embodiment may further extend the forty-seventh embodiment. In the forty-eighth embodiment, the one or more morphological operations include at least one of an erode operation or a dilate operation.

[0052] The forty-ninth embodiment can further expand any one of the forty-sixth embodiment to the forty-eighth embodiment. In the forty-ninth embodiment, the method further includes: receiving new coordinates; determining that a second tooth of the plurality of teeth is a tooth closest to the new coordinates; using the first visualization to display the second tooth closest to the new coordinates; and, using the second visualization to display the first tooth.

[0053] The fiftieth embodiment may further extend any one of the forty-sixth embodiment to the forty-ninth embodiment. In the fiftieth embodiment, at least one of the proximal surface or the distal surface of the first tooth is visible by using the remaining teeth of the plurality of teeth displayed using at least one second visualization.

[0054] The fifty-first embodiment may further expand any one of the forty-sixth embodiment to the fiftieth embodiment. In the fifty-first embodiment, the coordinates are received via user input.

[0055] The fifty-second embodiment may further extend the fifty-first embodiment. In the fifty-second embodiment, receiving the coordinates input by the user includes receiving a user input of dragging the prompt feature to the coordinates.

[0056] The fifty-third embodiment can further expand any one of the forty-sixth embodiment to the fifty-second embodiment. In the fifty-third embodiment, the first tooth is a prepared tooth having a margin line, and wherein determining the first tooth closest to the coordinate comprises: identifying the margin line of the prepared tooth; and determining that the point on the 3D surface closest to the coordinate is within the margin line; and classifying the point on the prepared tooth within the margin line as the prepared tooth.

[0057] The fifty-fourth embodiment may further expand any one of the forty-sixth embodiment to the fifty-third embodiment. In the fifty-fourth embodiment, the first visualization includes an opaque visualization, and wherein the second visualization includes a semi-transparent visualization.

[0058] In a fifty-fifth embodiment, a computing device including a memory and a processing device performs the operations of any one of the forty-sixth to fifty-fourth embodiments.

[0059] In a fifty-sixth embodiment, a non-transitory computer-readable medium includes instructions, which, when executed by a processing device, cause the processing device to execute the method of any one of the forty-sixth to fifty-fourth embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Embodiments of the disclosure are illustrated by way of example and not limitation in the figures of the accompanying drawings.

[0061] Figure 1 One embodiment of a system for performing intraoral scanning and / or generating a virtual three-dimensional model of an intraoral location is shown.

[0062] Figure 2A A flow chart of a method for scanning a dental site according to an embodiment is shown.

[0063] Figure 2B A flow chart of a method for displaying a 3D surface of a dental site according to an embodiment is shown.

[0064] Figure 3A-3D A view of a 3D surface of a first dental site generated from intraoral scan data during an intraoral scanning session is shown in accordance with an embodiment.

[0065] Figure 3E-3H A view of a 3D surface of a second dental site generated from intraoral scan data during an intraoral scanning session is shown in accordance with an embodiment.

[0066] Figure 4A-4G A graphical user interface showing a view of a 3D surface of dental sites generated from intraoral scan data during an intraoral scanning session is shown according to an embodiment.

[0067] Figure 5A A flow chart of a method of scanning and preparing teeth according to an embodiment is shown.

[0068] Figure 5B A flow chart showing a method of scanning different parts of an oral cavity using two different scanning modes according to an embodiment.

[0069] Fig. 6A A flow chart of a method for resolving conflicting scan data of dental sites according to an embodiment is shown.

[0070] Figure 6B A resolution of conflicting scan data for dental sites according to an embodiment is shown.

[0071] Fig. 7A A flow chart showing a partial shrinking method of a scan-prepared tooth according to an embodiment.

[0072] Figure 7B Another flow chart of a partial shrinking method of a scan-prepared tooth according to an embodiment is shown.

[0073] Figure 7C-7G A partial shrinking method of a scan-prepared tooth according to an embodiment is shown.

[0074] Figure 8 A flow chart illustrating a method of identifying and grading margin lines of a prepared tooth according to an embodiment.

[0075] Fig. 9 An example workflow of generating an accurate virtual 3D model of a dental site and manufacturing a dental prosthesis based on the virtual 3D model according to an embodiment of the present disclosure is shown.

[0076] Fig.10 A flowchart of a method for segmenting intraoral scan data into various classifications and updating a 3D surface based on the classifications according to an embodiment is shown.

[0077] Fig.11 A workflow for training a machine learning model and applying the trained machine learning model during intraoral scanning according to an embodiment of the present disclosure is shown.

[0078] Fig.12 A flowchart of a method of training a machine learning model to perform segmentation of an intraoral scan and / or a 3D surface generated by an intraoral scan is shown according to an embodiment.

[0079] Fig.13 A block diagram of an example computing device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0080] Described herein are methods and systems for generating 3D surfaces and virtual 3D models of dental sites based on intraoral scan data in accordance with embodiments of the present disclosure. Certain embodiments are directed to a user interface for an intraoral scanning system and / or an intraoral scanning application. The user interface may present different portions of the 3D surface generated by the intraoral scan data in different ways, depending on the classification of those different portions. In some embodiments, processing logic performs point-level (e.g., pixel-level) or plaque-level classification of the 3D surface of the dental site and / or the intraoral scan and / or projection of the 3D surface of the dental site. For example, a point or pixel may be classified as soft tissue (such as gums) or hard tissue (such as teeth). Different visualizations may then be used to display the soft tissue and hard tissue. For example, an opaque visualization may be used to present hard tissue, and a transparent or translucent visualization may be used to present soft tissue. In one embodiment, an opaque visualization is used to present hard tissue, while the soft tissue is completely transparent (e.g., so that the soft tissue is not visible in the display screen)

[0081] By using transparent or semi-transparent visualization to display or present soft tissue, and using opaque visualization to display hard tissue, the user interface for the intraoral scanning system enables the dental practitioner to see the patient's gums, and to see and understand the hard tissue below the gum line. Therefore, the dental practitioner is able to observe the subgingival portion of the teeth, including the subgingival margin line on the 3D surface of the prepared teeth, while also observing how the gums cover the subgingival portion of the teeth. In an embodiment where the soft tissue is hidden (e.g., 100% transparent), the dental practitioner can observe the teeth (including the subgingival portion of the teeth) without any part of the scanned teeth being obscured by the gums. Therefore, the user interface of the intraoral scanning system provides a new user experience, allowing dental practitioners to generate 3D models of prepared teeth faster and more accurately.

[0082] Some embodiments can obtain accurate intraoral scan data of the prepared teeth, including the margin lines of the prepared teeth. For example, embodiments cover techniques for exposing only a portion of the margin line at a time and generating an intraoral scan of the exposed portion of the margin line without using a shrink line (exposing all of the margin line at once). Some embodiments provide multiple scanning modes, one of which is used to scan the prepared teeth and another scanning mode is used to scan teeth other than the prepared teeth. In some embodiments, tools such as dental probes, scrapers, airflow, etc. can be used to expose a small area of ​​the margin line when scanning. The process can be repeated for all areas of the margin line until the entire margin line is scanned.

[0083] For many restorative procedures (e.g., creating crowns, bridges, veneers, etc.), the patient's existing teeth are ground down to residual teeth. The ground teeth are referred to herein as prepared teeth, or simply preparations. The prepared teeth have margin lines (also referred to as margin lines), which are the boundaries between the natural (unground) portion of the prepared tooth and the prepared (ground) portion of the prepared tooth. The prepared teeth are typically created so that a crown or other prosthesis can be mounted or placed on the prepared tooth. In many cases, the margin lines of the prepared teeth are subgingival (below the gum line). Although the term preparation generally refers to the residual teeth of the prepared teeth, including the margin lines and shoulders of the residual teeth, the term preparation herein also includes artificial residual teeth, pivots, cores and posts, or other devices that can be implanted in an intraoral cavity to receive a crown or other prosthesis. The embodiments described herein with reference to the prepared teeth are also applicable to other types of preparations, such as the aforementioned artificial residual teeth, pivots, etc.

[0084] After creating the tooth, the practitioner performs operations to prepare the prepared tooth for scanning. Preparing the prepared tooth for scanning may include wiping the prepared tooth of blood, saliva, etc. and / or separating the patient's gums from the prepared tooth to expose the margin line. In some cases, the practitioner will insert material (e.g., shrink material, such as shrink wire) around the prepared tooth between the prepared tooth and the patient's gums. The practitioner will then remove the wire before generating an intraoral scan of the prepared tooth. The soft tissue of the gums will then return to its natural position and, in many cases, collapse back over the margin line after a short period of time. Therefore, the practitioner uses an intraoral scanner to scan the prepared tooth and generate a set of intraoral images of the prepared tooth before the soft tissue returns to its natural position. For this process, the intraoral scanner can be used in a first scanning mode, which is referred to as a standard preparation or fully shrink scanning mode.

[0085] In one embodiment, the intraoral scanner is used in the preparation scanning mode. Alternatively, the scanner can be used in the standard scanning mode. When the scanner is in the preparation scanning mode, the user can use a partial retraction scanning technique or a standard preparation scanning technique (where the retraction line is used to expose the edge line). For the partial retraction scanning technique, the practitioner (e.g., a dentist or doctor) can use a tool such as a dental probe, a dental scraper, a triple syringe, a tool that outputs air or water flow, etc. to partially expose the edge line around the prepared tooth being scanned. When a portion of the edge line is exposed, the intraoral scanner generates a scan of the prepared tooth area with the exposed portion of the edge line. Then, the practitioner uses the tool to expose another portion of the edge line, which is also imaged. This process continues until all the edge lines are exposed and scanned. For the standard scanning mode and the preparation scanning mode, different algorithms, settings, rules and standards can be used to stitch images together. Compared to standard techniques such as using retraction lines, the partial retraction scanning technique can be a more effective technique for scanning subgingival preparations. The partial retraction scanning technique can be performed faster (e.g., about 1-2 minutes, or even less than a minute), and the patient is minimally uncomfortable. Additionally, practitioners can perform a partial retraction sweep technique without the need to rush to avoid gingival collapse back above the margin line.

[0086] The intraoral points where the prosthesis is implanted should generally be accurately measured and carefully studied so that prostheses such as crowns, dentures or bridges can, for example, be appropriately designed and sized to fit in place. A good fit enables mechanical stresses to be properly transferred between the prosthesis and the jaw, and can prevent gum infections and tooth decay, for example, that occur via the interface between the prosthesis and the intraoral point. After the intraoral points have been scanned, a virtual 3D model (also referred to herein as a 3D model) of the dental points can be generated, and the 3D model can be used to manufacture a dental prosthesis. However, if the area of ​​the prepared tooth containing the margin line lacks definition, the margin line may not be properly determined, and therefore the edge of the prosthesis may not be properly designed.

[0087] Therefore, embodiments disclosed herein provide automated systems and methods for analyzing, marking, and / or updating edge lines in a 3D surface of a prepared tooth generated from intraoral scan data. A 3D surface (or an image generated from a 3D surface or a scan for generating a 3D surface) can be analyzed to identify edge lines. In some embodiments, a trained machine learning model trained to determine edge lines on a prepared tooth is used to analyze a 3D surface, an intraoral scan, and / or an image generated by projecting the 3D surface onto a 2D surface. Edge segments can be graded or scored, and edge segments can be marked or drawn on a 3D surface according to their grades or scores. For example, edge segments with grades or scores above an upper threshold can be displayed with a first visualization (e.g., a green visualization), indicating that these edge segments are acceptable. Edge segments with grades or scores below a lower threshold can be displayed with a second visualization (e.g., a red visualization), indicating that these edge segments are unacceptable. Edge segments with grades or scores above a lower threshold and below an upper threshold can be displayed with a third visualization (e.g., a yellow visualization), indicating that these edge segments may be acceptable. Margin line segments may be identified and graded during intraoral scanning and before a final 3D virtual model is generated (e.g., before the scan is complete). Margin line segments may then be marked on a view of the 3D model during intraoral scanning to indicate to the practitioner performing the intraoral scan that further intraoral scans should be generated for those areas with low margin line scores / grades.

[0088] Other embodiments are also described for automatically selecting which intraoral scans to use when generating a 3D model of a dental arch depicting multiple dental locations (e.g., multiple teeth). Traditionally, a physician selects a preparation tooth in the scanning software, then scans the selected preparation tooth, selects another preparation tooth in the scanning software (if multiple preparation teeth are to be scanned), then scans the other preparation tooth, selects a specific dental arch to be scanned, and then scans the rest of the one or more other teeth on the dental arch with one or more preparation teeth. This informs the scanning software which intraoral scan to use for each tooth on the dental arch when generating the 3D model. However, the process of individually selecting and then scanning each preparation tooth and dental arch may be cumbersome for the physician. In an embodiment, the scanning software is capable of automatically identifying which intraoral scan to use for each tooth and / or which scan mode setting to use (e.g., preparation scan mode or standard scan mode) without the physician manually identifying the scan associated with a specific preparation tooth and / or the scan mode setting for a specific tooth. Thus, the physician can scan the preparation teeth and other teeth in any desired order and using any desired technique without manually identifying when to scan the preparation teeth.

[0089] Also described are embodiments of automatically identifying teeth represented in a scan and / or 3D surface, identifying gums in a scan and / or 3D surface, identifying excess material in an intraoral scan and / or 3D surface, identifying moving tissue and / or dental tools in an intraoral scan and / or 3D surface, and / or identifying edge segments in an intraoral scan and / or 3D surface.

[0090] Various embodiments are described herein. It should be understood that these various embodiments may be implemented as independent solutions and / or may be combined. Therefore, references to an embodiment or an embodiment may refer to the same embodiment and / or different embodiments. In addition, some embodiments are discussed with reference to restorative dentistry, particularly preparing teeth and margin lines. However, it should be understood that the embodiments discussed with reference to restorative dentistry (e.g., restorations) may also be applied to orthodontic dentistry (e.g., orthodontics). In addition, the embodiments discussed with reference to preparing teeth may also be applied to teeth in general, not just preparing teeth. In addition, the embodiments discussed with reference to margin lines may also be applied to other tooth features, such as cracks, chips, gum lines, caries, etc.

[0091] Some embodiments are discussed herein with reference to intraoral scanning. However, it should be understood that the embodiments described with reference to intraoral scanning are also applicable to laboratory scanning or model / impression scanning. Laboratory scanning or model / impression scanning may include one or more scans of a dental site or a model or impression of a dental site, which may or may not include a height map, and may or may not include a color image.

[0092] Figure 1An embodiment of a system 100 for performing intraoral scanning and / or generating a virtual three-dimensional (3D) model of a dental site is shown. In one embodiment, one or more components of the system 100 perform the following steps with reference to FIG. Fig.12 Describes one or more operations.

[0093] The system 100 may include a dental clinic 108 and / or a dental laboratory 110. The dental clinic 108 and the dental laboratory 110 each include computing devices 105, 106, wherein the computing devices 105, 106 may be connected to each other via a network 180. The network 180 may be a local area network (LAN), a public wide area network (WAN) (e.g., the Internet), a private WAN (e.g., an intranet), or a combination thereof.

[0094] The computing device 105 may be coupled to an intraoral scanner 150 (also referred to as a scanner) and / or a data storage device 125. The computing device 106 may also be connected to a data storage device (not shown). The data storage device may be a local data storage device and / or a remote data storage device. The computing device 105 and the computing device 106 may each include one or more processing devices, memory, secondary storage, one or more input devices (e.g., such as a keyboard, mouse, tablet computer, etc.), one or more output devices (e.g., a display, printer, etc.), and / or other hardware components.

[0095] The intraoral scanner 150 may include a probe (e.g., a handheld probe) for optically capturing a three-dimensional structure by generating an intraoral scan. The intraoral scanner 150 may be used to perform an intraoral scan of a patient's oral cavity. The intraoral scanning application 115 running on the computing device 105 may communicate with the scanner 150 to implement the intraoral scanning procedure. The result of the intraoral scan may be intraoral scan data 135A, 135B to 135N, which may include one or more sets of intraoral scans and / or intraoral images. Each intraoral scan may be a two-dimensional (2D) or 3D point cloud or image including x, y, and z information. Some intraoral scans (such as scans generated by a confocal scanner) include a 2D height map. In one embodiment, the intraoral scanner 150 generates many discrete (i.e., separate) intraoral scans. Multiple sets of discrete intraoral scans may be merged into a set of smaller hybrid intraoral scans, wherein each hybrid intraoral scan is a combination of multiple discrete intraoral scans. The scanner 150 may transmit the intraoral scan data 135A, 135B to 135N to the computing device 105. The computing device 105 may store the intraoral scan data 135A- 135N in the data storage device 125 .

[0096] The intraoral scan data 135A-135N may selectively include one or more color images (e.g., color 2D images) and / or images generated under specific lighting conditions (e.g., ultraviolet (UV) images, infrared (IR) images, and / or near infrared (NIRI) images).

[0097] According to an example, a user (e.g., a practitioner) can perform an intraoral scan of a patient. In this way, the user can apply the scanner 150 to one or more intraoral locations of the patient. The scan can be divided into one or more segments. As an example, these segments can include an upper dental arch segment, a lower dental arch segment, an occlusal segment, and optionally one or more prepared tooth segments. As another example, these segments can include a lower cheek area of ​​the patient, a lower tongue area of ​​the patient, an upper cheek area of ​​the patient, an upper tongue area of ​​the patient, one or more prepared teeth of the patient (e.g., teeth of the patient to which a dental device such as a crown or other dental prosthesis will be applied), one or more teeth in contact with the prepared teeth (e.g., teeth that are not affected by the dental device itself but are located next to one or more such teeth or in contact with one or more such teeth when the mouth is closed), and / or a patient's occlusion (e.g., a scan is performed when the patient's mouth is closed, and the scan is directed to the interface area of ​​the patient's upper and lower teeth). Through this scanner application, the scanner 150 can provide intraoral scan data 135A-135N to the computing device 105. The intraoral scan data 135A-135N may be provided in the form of multiple sets of intraoral scan data, each of which may include 3D point clouds, 2D images, and / or 3D images of specific teeth and / or intraoral locations. In one embodiment, separate data sets are created for the maxillary arch, mandibular arch, patient bite, and each prepared tooth. Alternatively, a single large data set is generated (e.g., for the mandibular and / or maxillary arches). Such scans may be provided to the computing device 105 from the scanner 150 in the form of one or more points (e.g., one or more point clouds).

[0098] The manner in which the patient's mouth is scanned may depend on the procedure being applied thereto. For example, if an upper or lower denture is to be made, a full scan of the mandibular or maxillary edentulous arch may be performed. Conversely, if a dental bridge is to be created, only a portion of the entire arch may be scanned, including the edentulous area, adjacent prepared teeth (e.g., abutting teeth), and the opposing arch and dentition. Furthermore, the manner in which the mouth is scanned may depend on the physician's scanning preferences and / or the patient's condition.

[0099] By way of non-limiting example, dental procedures can be roughly divided into oral rehabilitation (restorative) and orthodontic procedures, which are then further subdivided into specific forms of these procedures. In addition, dental procedures can include the identification and treatment of gum disease, sleep apnea, and intraoral diseases. The term restorative procedure refers in particular to any procedure involving the oral cavity and for designing, manufacturing, or installing a dental prosthesis or its real or virtual model at a dental point (intraoral point) in the oral cavity, or for the design and preparation of an intraoral point to receive such a prosthesis. A prosthesis may include any restoration, such as a crown, veneer, inlay, onlay, implant, and bridge, as well as any other artificial partial or complete denture. The term orthodontic procedure refers in particular to any procedure involving the oral cavity and for designing, manufacturing, or installing an orthodontic element or its real or virtual model at an intraoral point in the oral cavity, or for designing and preparing an intraoral point to receive such an orthodontic element. These elements may be appliances, including but not limited to brackets and wires, retainers, transparent aligners, or functional appliances.

[0100] For many restorative procedures (e.g., creating crowns, bridges, veneers, etc.), a prepared tooth is created (e.g., by grinding a portion of a tooth into a stump). The prepared tooth has a margin line (also called a finish line), which may be important for a fitting fit of a dental prosthesis. After the prepared tooth is made, a practitioner may perform an operation to prepare the prepared tooth for scanning. Preparing the prepared tooth for scanning may include wiping blood, saliva, etc. from the prepared tooth and / or separating the patient's gums from the prepared tooth to expose the margin line.

[0101] In some cases, the practitioner will perform a standard preparation (full retraction) technique to expose the entire margin line at one time by inserting a wire around the prepared tooth between the prepared tooth and the patient's gums, and then removing the wire before generating a set of intraoral scans of the prepared teeth. The soft tissue of the gums will then return to its natural position and, in many cases, collapse back over the margin line after a brief period of time. Therefore, some of the intraoral scan data 135A-135N may include images taken before the gums collapse back over the margin line, while other intraoral scan data 135A-135N may include images taken after the gums collapse over the margin line.

[0102] In some cases, the dentist performs a partial retraction scanning technique. For the partial retraction scanning technique, the gums are pushed aside with a tool to expose a small section of the margin line of the subgingival preparation. This small section is scanned, and the tool is moved to allow this small section of the gums to collapse back above the margin line and expose another small section of the margin line. Therefore, preparing the prepared tooth for scanning can include using the tool to expose only a portion of the margin line, and then scanning it when it is exposed. Then, preparing the prepared tooth can include using the tool to expose another portion of the margin line, scanning it when it is exposed. This process can continue until all of the margin lines have been scanned.

[0103] Examples of tools that can be used to expose a portion of the margin line at a time include a dental probe, a dental scraper, a triple syringe, an air gun, dental floss, a water gun, etc. In some embodiments, specific tools are developed for exposing one or more portions of the margin line around one or more teeth (e.g., a first tool for exposing the interproximal portion of the margin line, a second tool for exposing the lingual portion of the margin line, etc.). Different tools developed for exposing different portions of the tooth margin line can have protrusions, lengths, probes, scrapers, etc. with different lengths, widths, angles, etc.

[0104] During an intraoral scan, the intraoral scanning application 115 may register and stitch together two or more intraoral scan data (e.g., intraoral scan data 135A and intraoral scan data 135B) generated so far by the intraoral scanning session. In one embodiment, performing the registration includes capturing 3D data of surface points in multiple scans and registering the scans by calculating transformations between the scans. One or more 3D surfaces may be generated based on the intraoral scans that were registered and stitched together during the intraoral scan. The one or more 3D surfaces may be output to a display so that a physician or technician can view their scan progress so far.

[0105] As each new intraoral scan is captured and registered with the previous intraoral scan and / or 3D surface, one or more 3D surfaces can be updated, and the updated 3D surfaces can be output to a display. In an embodiment, segmentation is performed on the intraoral scan and / or 3D surface to segment points and / or slices on the intraoral scan and / or 3D surface into one or more classifications. In one embodiment, the intraoral scanning application 115 classifies the points as hard tissue or soft tissue. The 3D surface can then be displayed using the classification information. For example, hard tissue can be displayed using a first visualization (e.g., an opaque visualization), and soft tissue can be displayed using a second visualization (e.g., a transparent or semi-transparent visualization).

[0106] In an embodiment, separate 3D surfaces are generated for the upper and lower jaws. This process can be performed in real time or near real time to provide updated views of the 3D surfaces captured during the intra-oral scan.

[0107] When a scanning session or a portion of a scanning session associated with a particular scan role or segment (e.g., a maxillary role, a mandibular role, an occlusal role, etc.) is completed (e.g., all scans of intraoral points or dental points have been captured), the intraoral scanning application 115 can automatically generate a virtual 3D model of one or more scanned dental points (e.g., maxillary and mandibular). The final 3D model can be a set of 3D points and their connections to each other (i.e., a mesh). In order to generate the virtual 3D model, the intraoral scanning application 115 can register and stitch together the intraoral scans generated from the intraoral scanning sessions associated with the particular scan role or segment. The registration performed at this stage may be more accurate than the registration performed during the capture of the intraoral scans, and may take more time to complete than the registration performed during the capture of the intraoral scans. In one embodiment, performing scan registration includes capturing 3D data of each point of the surface in multiple scans and registering the scans by calculating the transformation between the scans. The 3D data can be projected into the 3D space of the 3D model to form a portion of the 3D model. The intraoral scans can be integrated into a common reference frame by applying appropriate transformations to the points of each registered scan and projecting each scan into 3D space.

[0108] In one embodiment, registration is performed on adjacent or overlapping intraoral scans (e.g., each consecutive frame of an intraoral video). In one embodiment, registration is performed using hybrid scans. A registration algorithm is performed to register two adjacent or overlapping intraoral scans (e.g., two adjacent hybrid intraoral scans) and / or to register an intraoral scan with a 3D model, which primarily involves determining a transformation that aligns one scan with another scan and / or with a 3D model. Registration can involve identifying multiple points in each scan (e.g., a point cloud) of a scan pair (or a scan and a 3D model), performing surface fitting on the points, and matching points of the two scans (or scans and a 3D model) using a local search around the points. For example, the intraoral scanning application 115 can match points of one scan with the nearest points interpolated on the surface of another scan, and iteratively minimize the distance between matching points. Other registration techniques may also be used.

[0109] The intraoral scanning application 115 can repeat the registration for all intraoral scans of the intraoral scanning sequence to obtain a transformation of each intraoral scan, registering each intraoral scan with the previous (multiple) intraoral scans and / or a common reference frame (e.g., with the 3D model). The intraoral scanning application 115 can integrate the intraoral scans into a single virtual 3D model by applying appropriately determined transformations to each intraoral scan. Each transformation can include rotations about one to three axes and translations in one to three planes.

[0110] In many cases, data from one or more intraoral scans does not perfectly correspond to data from one or more other intraoral scans. Therefore, in an embodiment, the intraoral scanning application 115 can process the intraoral scans (e.g., which can be a mixture of intraoral scans) to determine which intraoral scans (or which portions of the intraoral scans) to use for portions of the 3D model (e.g., for portions representing a particular dental site). The intraoral scanning application 115 can use data (such as geometric data represented in the scans and / or timestamps associated with the intraoral scans) to select the best intraoral scan for depicting the dental site or a portion of the dental site (e.g., for depicting the margin line of a prepared tooth). In one embodiment, the images are input into a machine learning model that has been trained to select and / or grade scans of dental sites. In one embodiment, one or more scores are assigned to each scan, wherein each score can be associated with a particular dental site and indicate the quality of the representation of that dental site in the intraoral scan.

[0111] Additionally or alternatively, weights may be assigned to the intraoral scans based on the scores assigned to the scans. The assigned weights may be associated with different dental sites. In one embodiment, weights may be assigned to each scan (e.g., each hybrid scan) of a dental site (or multiple dental sites). During model generation, conflicting data from multiple intraoral scans may be combined using a weighted average to depict the dental sites. The weights applied may be those weights assigned based on the quality scores of the dental sites. For example, processing logic may determine that data from a particular overlapping region of a first set of intraoral scans is superior in quality to data from a particular overlapping region of a second set of intraoral scans. Then when the differences between the intraoral scan data sets are averaged, the first set of intraoral scan data may be weighted more than the second set of intraoral scan data. For example, a first intraoral scan assigned a higher grade may be assigned a weight of 70%, while a second intraoral scan may be assigned a weight of 30%. Therefore, when the data is averaged, the combined result looks more like a depiction from the first set of intraoral scan data than a depiction from the second set of intraoral scan data.

[0112] The intraoral scanning application 115 can generate one or more 3D surfaces and / or 3D models based on the intraoral scan, and can display the 3D surfaces and / or 3D models to a user (e.g., a doctor) via a user interface. The doctor can then visually inspect the 3D surfaces and / or 3D models. The doctor can virtually manipulate the 3D surfaces and / or 3D models in up to six degrees of freedom (i.e., translation and / or rotation relative to one or more of three mutually orthogonal axes) via the user interface using appropriate user controls (hardware and / or virtual) so that the 3D models can be viewed from any desired direction. The doctor can view (e.g., visually inspect) the generated 3D surfaces and / or 3D models of the intraoral points and determine whether the 3D surfaces and / or 3D models are acceptable (e.g., whether the margin lines of the prepared teeth are accurately represented in the 3D model).

[0113] In one embodiment, the intraoral scanning application 115 includes standard scanning logic 119 and preparation scanning logic 118. The standard scanning logic 119 can provide a standard scanning mode in which one or more first algorithms are used to process the intraoral scan data. The preparation scanning logic 118 can provide a preparation scanning mode in which one or more second algorithms are used to process the intraoral scan data. The first and second algorithms can use different rules, settings, thresholds, etc. to select which images and which parts of the images are used to build parts of the virtual 3D model and / or classify and / or segment the intraoral scan and / or 3D surface.

[0114] The standard scan logic 119 can provide a standard scan mode in which one or more first algorithms are used to process intraoral scan data. The first algorithm of the standard scan mode can be optimized to generate a virtual 3D model of teeth other than prepared teeth. The first algorithm may include, for example, a moving tissue detection algorithm, an excess material removal algorithm, a blending algorithm, a stitching and / or registration algorithm, etc. Such an algorithm can be configured based on the assumption that the scanned tooth area is static (e.g., not moving). Therefore, if there are interference or rapid changes (e.g., features that are only displayed for a short time), the first algorithm can operate to minimize or filter out the data under the assumption that the data is not part of the scanned object. For example, the first algorithm can classify such data as a depiction of the tongue, cheek, doctor's finger, tool, etc., and such data may not be used when generating a 3D model of the teeth. However, if used on scan data showing prepared teeth, such an algorithm can also remove edge line data.

[0115] In one embodiment, in standard scanning mode, a raw intraoral scan is received from the intraoral scanner 150 and a preliminary registration is performed using an initial registration algorithm. The blending algorithm is then applied to the raw scan. If the scans are similar (e.g., both have timestamps that differ by less than a time difference threshold, and have a surface difference that is less than a surface difference threshold and / or a position / orientation difference that is less than a position / orientation threshold), the scans are blended together using the blending algorithm. For example, up to 10-20 consecutive scans that are separated by a few seconds or microseconds from each other can be blended together. This includes averaging the scan data being blended and generating a single blended scan based on the averaged data. The generation of a blended scan reduces the total number of scans processed in a later scan processing stage.

[0116] The hybrid scan is then processed using an excess material tissue algorithm that identifies moving tissue of a size greater than a size threshold (e.g., greater than 100 or 200 microns), and such excess material is then erased from the hybrid scan. In one embodiment, the excess material identification algorithm is a trained machine learning model (e.g., a neural network such as a convolutional neural network) that has been trained to identify such excess material. An embodiment of a machine learning model for excess material removal is described in U.S. Patent No. 11,238,586, issued on February 1, 2022, which is incorporated herein by reference.

[0117] Another registration algorithm then registers the hybrid scans together. A moving tissue algorithm then identifies moving tissue based on the differences between the hybrid scans. The moving tissue algorithm can identify moving tissue that is larger than some threshold value (e.g., 100 or 200 microns) in size and can erase such moving tissue from the hybrid scan. A merging algorithm can then merge all remaining image data from the hybrid scans together to generate a virtual 3D model of the prepared body. The merging algorithm can average the data differences between the scans. Such differences can have a difference that is less than a threshold difference value, e.g., less than 0.5 millimeters.

[0118] The preparation scan logic 118 may provide a preparation scan mode in which one or more second algorithms are used to process the intraoral scan data. The second algorithms of the preparation scan mode may be optimized to generate a virtual 3D model of the prepared teeth, depicting the edge lines as clearly and accurately as possible given the scan data. The second algorithms may include, for example, moving tissue detection algorithms, excess material removal algorithms, blending algorithms, registration algorithms, stitching or merging algorithms, etc. In some embodiments, the second algorithms include excess material removal algorithms, registration algorithms, and stitching or merging algorithms. In one embodiment, the preparation scan mode may be optimized for use with a partial shrinkage scanning technique.

[0119] When using the partial retraction scanning technique, the size of the exposed area of ​​the edge line is on the order of tens of microns. This exposed area of ​​the edge line can change between scans and can be removed by the excess material removal algorithm and / or the moving tissue removal algorithm of the standard preparation scanning mode. In addition, for partial retraction scanning, the scans may be interrupted due to the doctor moving the retraction tool between scans, while for standard scanning, the scans can be continuous without interruption. This can increase the time and difficulty of alignment between scans, which can be minimized in an embodiment of the second algorithm.

[0120] In some embodiments, the second algorithm includes a different version of a moving tissue algorithm and / or a different version of an excess material removal algorithm than the first algorithm. For example, the criteria that constitute excess material and / or moving tissue may vary for the excess material removal algorithm and / or the moving tissue detection algorithm.

[0121] In addition, the first algorithm may blend scan data from multiple scans together to generate a blended scan, while in some embodiments, the second algorithm may not be used to blend the scans. Alternatively, the criteria for which scans can be blended together (and / or which portions of scans can be blended together) in the second algorithm may be more stringent than the criteria for which scans (and / or which portions of scans) can be blended together in the first algorithm. In one embodiment, the first algorithm averages the blended image at least for regions that meet some criteria (e.g., size of matching region criteria). In one embodiment, the second algorithm omits blending of all or part of the scans.

[0122] In one embodiment, in the preparation scan mode, raw intraoral scans are received from the intraoral scanner 150. The raw scans may or may not be preliminarily registered with each other using an initial registration algorithm. In some cases, a hybrid scan is generated by blending the raw scans together. In one embodiment, the hybrid scan is generated based on multiple raw scans generated while the same area of ​​the margin line is exposed. Therefore, the hybrid scan may not include different scans in which different portions of the margin line are exposed. As described above, if a hybrid algorithm is used, the hybrid algorithm may have stricter criteria for which data can be blended together than the hybrid algorithm used for the standard preparation scan mode.

[0123] In one embodiment, the excess material identification algorithm is a trained machine learning model (e.g., a neural network, such as a convolutional neural network) that has been trained to identify excess material. The machine learning model may have been specifically trained with training data that includes margin line delineation so that it does not identify margin line areas that vary between scans as excess material. The machine learning model may have been trained using a training data set that includes scans where the gum is above the margin line and scans where the gum is not covering the exposed margin line. In one embodiment, such a machine learning model may be trained to remove the gum and leave an exposed margin line. In one embodiment, a specific excess gum removal algorithm (e.g., a trained machine learning model) is used instead of a general excess material removal algorithm. In one embodiment, two excess material removal algorithms are used, one of which is used to remove excess gum and the other is used to remove other excess material. In one embodiment, the input to the trained machine learning model that has been trained to remove excess gum is multiple sets of scans. The trained machine learning model may determine which scan data represents excess gum and should be removed for multiple sets of scans.

[0124] One embodiment of a machine learning model for removing excess material and / or removing excess gum is described in U.S. Patent No. 11,238,586, but the training data set used to train the machine learning model may be different from the training data set described in the referenced patent application. Specifically, the training data set used to train the machine learning model may include: a scan showing gums above a margin line, wherein the gum area above the margin line is marked as excess material with a point-level marker, and a scan showing an exposed margin line with retracted gums that do not cover portions of the margin line that are marked with a point-level marker identifying gum areas that are not classified as excess material. For example, the machine learning model may be trained to perform segmentation in a manner that so classifies pixels representing excess gums above the margin line.

[0125] In one embodiment, an excess gingiva removal algorithm is used to identify and remove excess gingiva overlying the preparation margin line. The excess gingiva removal algorithm may be similar to the excess material removal algorithm, but may be specifically trained to identify excess gingiva overlying the margin line for removal.

[0126] The registration algorithm registers the original scans and / or the hybrid scans (which may have been processed by the excess material removal and / or excess gingiva removal algorithms) to each other.

[0127] A merging algorithm (also called a stitching algorithm) may then merge all of the remaining image data from the scans (which may be original scans or hybrid scans) together to generate a virtual 3D model of the prepared volume. For any multiple scans that are merged to generate a 3D model, there will inevitably be some differences between the scans that need to be resolved in some way. Such differences may be determined by identifying conflicting surfaces between overlapping regions of the scans and then determining whether the conflicting surfaces meet one or more criteria. The merging algorithm may average some of the data differences between the scans, and for other differences, some of the data may be discarded. Criteria for determining whether to average data between scans or discard data from some scans include the size of the conflicting surfaces, the distance differences (e.g., height or depth) between the conflicting surfaces in the scans, the average or Gaussian curvature differences between the conflicting surfaces in the scans, and the like.

[0128] The first algorithm may include a first merge algorithm with a first size threshold and a first similarity threshold for determining whether to average together the data from the conflicting surfaces. The second algorithm may include a second merge algorithm with a second size threshold for determining whether to average together the data from the conflicting surfaces, wherein the second size threshold may be greater than or less than the first size threshold. Additionally, the second merge algorithm may include a second similarity threshold that is higher than the first similarity threshold.

[0129] The size of the conflicting region can be used to determine whether to perform merging, where the size of the conflicting region must exceed a certain size threshold or be averaged. Therefore, in an embodiment for the second algorithm, averaging can be performed on regions smaller than the threshold size, while averaging is not performed on regions greater than or equal to the threshold size.

[0130] In addition, the merging algorithm of the second algorithm can determine which image data is used for a specific overlapping area based on criteria such as the distance (depth or height) and / or curvature from the scanner. For example, points from the first scan with a larger distance probe and a larger curvature (e.g., a larger average curvature or a larger Gaussian curvature) can be selected, and conflicting points from the second scan with a smaller distance and / or a lower curvature can be omitted or removed. In an example, the scan data can include a height map, wherein each height map includes different values ​​representing the height (or the opposite depth) of each pixel of the scan. The scans can be registered together and can be determined to include overlapping pixels. The heights of these overlapping pixels can be compared, and the smaller height value (i.e., the larger depth value) of one of the scans can be retained, while the larger height value (i.e., the smaller depth value) of the other scan can be discarded. Additionally or alternatively, for the conflicting surfaces from each scan, the average curvature or Gaussian curvature can be calculated. A scan with a higher average curvature or a higher Gaussian curvature can be selected for depicting the surface area, while a scan with a lower average curvature data or a lower Gaussian curvature for the conflicting surface can not be used for the surface area.

[0131] In one embodiment, the difference in height values ​​of the two scans is determined, and if the height value difference of the overlapping pixels exceeds a threshold, the smaller height value is selected. If the height value difference of the overlapping pixels is less than the threshold, the height values ​​can be averaged (e.g., using a weighted average or a non-weighted average). In an embodiment, this calculation can be performed based on the average depth value of some or all pixels within the conflicting surface.

[0132] Thus, portions of the scan data may be selected for use based on one or more criteria, such as the size of the overlapping region (e.g., the number of adjacent overlapping pixels in question), the difference in height values ​​of the overlapping pixels, and / or the difference in mean or Gaussian curvature. For regions larger than a size threshold, data from a region of the scan having a smaller height value and / or a larger mean curvature value may be selected, and data from a region of another scan having a larger height value and / or a smaller mean curvature value may be discarded or erased. This prevents data representing the gingiva from being averaged with data representing the margin line, since the margin line is associated with a high curvature value, whereas the gingiva is associated with a much lower curvature value.

[0133] In some embodiments, data from the first set of scans is discarded and data from the second set of scans is averaged together. For example, 7 scans may have an overlapping region whose size meets or exceeds a size threshold. Data from 4 of the scans depicting the region may be discarded, while data from the remaining 3 scans depicting the region may be averaged. In one embodiment, percentiles are calculated for scans with overlapping regions, and scans with height values ​​within a certain percentile value (e.g., the 80th percentile) may be selected for removal.

[0134] The intraoral scanning application 115 may include logic for automatically identifying (e.g., highlighting) margin lines in an intraoral scan and / or 3D surface of the prepared tooth. This may make it easier for the physician to detect the accuracy of the margin lines. Such marking of margin lines may be performed during an intraoral scan to enable the physician to identify areas around the prepared tooth that would be useful for further scanning. For example, the intraoral scanning application 115 may mark and / or highlight specific segments of margin lines that are unclear, ambiguous, and / or uncertain. For example, segments of acceptable margin lines may be displayed in a first color (e.g., green), while segments of unacceptable margin lines may be displayed in a second color (e.g., red). In one embodiment, a first trained machine learning model is used to identify margin lines in the prepared tooth.

[0135] Once a physician (e.g., a dentist) has determined that the 3D model of the dental site is acceptable, the physician may instruct the computing device 105 to send the 3D model to the computing device 106 of the dental laboratory 110. The computing device 106 may include a dental modeling application 120 that may analyze the 3D model to determine whether it is suitable for manufacturing a dental prosthesis. The dental modeling application 120 may include logic for identifying margin lines and / or modifying the surface of one or more dental sites and / or modifying margin lines, as discussed with reference to the intraoral scanning application 115. If the 3D model is deemed suitable (or is capable of being modified to be in a state that is deemed suitable), a dental prosthesis may be manufactured based on the 3D model. If the 3D model cannot be placed in a suitable condition, instructions may be sent back to the dental clinic 108 to generate one or more additional intraoral images of one or more areas of the dental site.

[0136] Figure 2A-Figure 12 Methods related to intraoral scanning and generation and manipulation of virtual 3D models of dental sites are shown. These methods can be performed by processing logic, which can include hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, at least some operations of the method are performed by a computing device executing an intraoral scanning application, such as Fig.13 In an embodiment, the intra-oral scanning application 1350 may correspond to Figure 1 Intraoral scanning application 115.

[0137] For simplicity of explanation, these methods are depicted and described as a series of actions. However, actions according to the present disclosure can occur in various orders and / or simultaneously, and can be sent together with other actions not presented and described herein. In addition, not all actions shown are necessary for implementing the methods according to the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that these methods can alternatively be represented as a series of interrelated states via state diagrams or events.

[0138] Figure 2A A flow chart of a method 200 for scanning a dental site is shown according to an embodiment. At block 205 of the method 200, processing logic receives first intraoral scan data (e.g., a first set of intraoral scans) of a dental site. At block 210, processing logic processes the first intraoral scan data to generate a 3D surface of the dental site. Processing the intraoral scan data may include registering and stitching the intraoral scans together to generate an initial 3D surface, and subsequently registering the intraoral scans to the 3D surface to extend and / or add details to the 3D surface.

[0139] In some embodiments, at box 212, processing logic displays a view of the 3D surface before the 3D surface and / or the intraoral scan used to generate the 3D surface has been processed to identify moving tissue, identify excess tissue, identify dental tools, identify hard and soft tissue, etc. The initially displayed 3D surface can be shown using a first visualization, which in an embodiment can be a semi-transparent visualization. Alternatively, box 212 can be skipped and the 3D surface can be not displayed until further processing is performed on the intraoral scan data and / or the 3D surface.

[0140] Figure 3A An initial view of a 3D surface 300 generated from one or more intraoral scans is shown before further processing is performed on these intraoral scans and / or 3D surfaces to identify moving tissue, identify excess tissue, identify dental tools, and / or identify hard and soft tissue.

[0141] Back to Figure 2AAt box 215, the first intraoral scan data (e.g., an intraoral scan used to generate a 3D surface) and / or data from the 3D surface are processed to identify hard tissue and soft tissue. The data from the 3D surface may include 3D data (e.g., a 3D point cloud) or a 2D projection of the 3D surface. In addition, the first intraoral scan data may be processed to identify points associated with a margin line and points not associated with a margin line, to identify points associated with moving tissue and / or dental tools, to identify points associated with excess material, etc. Such processing may include inputting the intraoral scan data or the data from the 3D surface into one or more trained machine learning models, which may output segmentation information indicating, for each point, a probability that the point has one or more classifications (e.g., hard tissue classification, soft tissue classification, margin line classification, dental tool classification, moving tissue classification, excess material classification, etc.). Such processing may additionally or alternatively include performing image processing and / or data processing that is independent of the trained machine learning model. In one embodiment, processing logic performs segmentation and classifies each point (eg, each 3D point or pixel) as soft tissue or hard tissue.

[0142] At box 220, processing logic displays a view of the 3D surface (or updates the view of the 3D surface if the 3D surface has been displayed at box 212). In the view of the 3D surface (or updated view), a first portion or region of the 3D surface identified as hard tissue may be shown using a first visualization and / or a first transparency level, and a second portion or region of the 3D surface identified as soft tissue may be shown using a second visualization and / or a second transparency level. The first visualization may include a first color (e.g., white or off-white), and the second visualization may include a second color (e.g., pink). Additionally or alternatively, the first visualization may include a first transparency level (e.g., 0% transparent or opaque), and the second visualization may include a second transparency level (e.g., 2-1000% transparent, or semi-transparent, or fully transparent, or invisible). In some embodiments, the second transparency level may be user-adjustable, and may be, for example, 2%, 5%, 10%, 20%, 25%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, or 100% transparent. In one embodiment, the first transparency level includes a transparency of 0-50% and the second transparency level includes a transparency of 51-100%. In one embodiment, the first transparency level includes 0% transparency and the second transparency level includes 100% transparency, and due to the 100% transparency, the portion of the 3D surface identified as soft tissue is not visible. In an embodiment, the first visualization includes a first color and a first transparency and the second visualization includes a second color and / or a second transparency. In some embodiments, the user can select the first visualization and / or the first transparency level and the second visualization and / or the second transparency level. For example, the user can select whether they want soft tissue (e.g., gums) to be visible. If the user does not want the soft tissue to be visible, a transparency level of 100% can be set for the soft tissue, and if the user wants the soft tissue to be at least partially visible, a lower transparency level (e.g., 30-50%) can be selected.

[0143] In some embodiments, a 3D surface is generated based on a scan of the prepared tooth, which may include 3D surface information of adjacent teeth. A previous 3D surface of a dental arch including the prepared tooth may have been generated. In such an embodiment, the 3D surface of the prepared tooth may be registered with the 3D surface of the dental arch and may be overlapped on the 3D surface of the dental arch. The 3D surface of the dental arch may then be displayed using a second visualization and / or a second transparency level or a third visualization and / or a third transparency level. Therefore, in some embodiments, even if the gums of the prepared tooth from the 3D surface are not visible (due to a 100% transparency level setting), the gums or gums from the 3D surface of the dental arch are still visible (e.g., may be shown with a semi-transparent visualization).

[0144] Figure 3B A view of an updated 3D surface 310 generated by one or more intraoral scans is shown after further processing of the intraoral scans and / or 3D surfaces is performed to identify hard tissue 312, 313 and soft tissue 314, 315. For example, soft tissue (e.g., gums or gums) 314, 315 can be shown in pink. In addition, the soft tissue 314, 315 is shown as translucent. In contrast, the hard tissue 312, 313 is shown as opaque. Therefore, even the hard tissue 313 is visible below the soft tissue 315. Some portions 316 of the 3D surface 310 can be added after the 3D surface is processed to identify hard tissue and soft tissue, and / or can be based on an intraoral scan that has not been processed to identify hard tissue and soft tissue. In an embodiment, these portions 316 can be shown using a translucent visualization.

[0145] Back to Figure 2A At box 225, processing logic receives additional intraoral scan data (e.g., one or more new intraoral scans) for the dental site. At box 230, processing logic updates the 3D surface based on the additional intraoral scan data. This may include registering and stitching the one or more new intraoral scans to the 3D surface. At box 235, processing logic updates a view of the 3D surface based on the additional intraoral scan data. Portions of the updated 3D surface based on the additional intraoral scan data may be shown with a semi-transparent visualization. For example, in Figure 3B Portion 316 of 3D surface 310 is shown using semi-transparent visualization.

[0146] At box 240, processing logic processes the additional intraoral scan data to identify hard tissue and soft tissue. Processing logic may additionally or alternatively process the 3D surface (e.g., data from the 3D surface), the intraoral scan data, and / or the additional intraoral scan data to identify margin lines. Processing logic may determine confidence values ​​for segments of margin lines, and may grade or score the segments of margin lines based on the confidence values ​​and / or other criteria. Processing logic may additionally or alternatively process the 3D surface, the intraoral scan data, and / or the additional intraoral scan data to identify moving tissue and / or dental tools, and / or to identify excess material.

[0147] At box 245, the processing logic updates the view of the 3D surface so that those portions of the updated 3D surface that are based on the additional intraoral scan data and represent hard tissue are displayed using the first visualization (e.g., off-white and / or opaque). The updated view of the 3D surface may additionally include markings of edge lines (e.g., outlines or highlights of edge lines). The markings of each segment of the edge line may be shown using a visualization associated with a grade or score assigned to the segment of the edge line. For example, edge segments with high scores (e.g., above an upper threshold limit) may be shown with a third visualization (e.g., bright green), edge segments with medium scores (e.g., below an upper threshold limit and above a lower threshold limit) may be shown with a fourth visualization (e.g., bright yellow), and edge segments with low scores (e.g., below a lower threshold limit) may be shown with a fifth visualization (e.g., bright red).

[0148] Figure 3C An updated view of an updated 3D surface 320 generated by adding information from a new intraoral scan to a previously generated 3D surface is shown after further processing of the new intraoral scan and / or 3D surface is performed to identify hard tissue 312, 313 and soft tissue 314, 315. As shown, a portion 316 of the 3D surface has now been classified as hard tissue and is therefore displayed using a first visualization in the updated view of the updated 3D surface 320. Also shown are marked margin lines 321. The marked margin lines 321 include a first visualization 322 showing a first margin line segment that is well defined, a second visualization 324 showing a second margin line segment that is moderately defined, and a third visualization 326 showing a third margin line segment that is poorly defined.

[0149] Back to Figure 2A At block 250, processing logic may determine whether the intraoral scan is complete. This determination may be made based on a quality assessment of the 3D surface, based on user input, and / or based on user activity detection (e.g., detecting that the scanner has been placed in a holder or on a surface and is no longer in the mouth). If the intraoral scan is not complete, the method may return to block 225, and the process may continue by receiving and processing additional intraoral scan data.

[0150] For example, Figure 3DAn updated view of an updated 3D surface 330 is shown, which is generated by adding information from a further intraoral scan to the previously generated 3D surface after further intraoral scans and / or further processing of the 3D surface is performed to identify hard tissue 312, 313 and soft tissue 314, 315. As shown, after the additional intraoral scan data has been received and processed, the marked margin line 321 includes larger and / or better defined segments with a first visualization 322, and fewer and / or smaller poorly defined segments with a third visualization 326.

[0151] Back to Figure 2A If it is determined at block 250 that the intraoral scan is complete, the method continues to block 255 where a virtual 3D model may be generated for the dental site. The virtual 3D model may be similar to a 3D surface, but may be processed using algorithms that produce higher quality surfaces and use more processing resources and take more time than those algorithms used to generate the 3D surface.

[0152] Figure 3E-3H A view of a 3D surface of a dental site generated from intraoral scan data during an intraoral scanning session is shown according to one embodiment. Figure 3E A first view 340 of the 3D surface is shown after the first set of intra-oral scans have been processed. Figure 3F A second view 360 of the 3D surface is shown after a second set of intra-oral scans have been processed and added to the 3D surface. Figure 3G A third view 370 of the 3D surface is shown after a third set of intra-oral scans have been processed and added to the 3D surface. Figure 3H A fourth view 380 of the 3D surface is shown after a fourth set of intraoral scans have been processed and added to the 3D surface. As shown, each of the views 340, 360, 370, 380 uses a first opaque visualization 342 to represent hard tissue such as teeth and a second semi-transparent visualization 344 to represent soft tissue such as gums. However, in an alternative embodiment, the soft tissue can be completely transparent and therefore not visible in the display. In addition, a third visualization 346 is used to show acceptable edge segments, a fourth visualization 348 is used to show edge segments that are at the acceptable boundary, and a fifth visualization 352 is used to show unacceptable edge segments. Figure 3E The view 340 includes a first region 350 that at least partially represents the moving tissue. The first region 350 is based on a recent intraoral scan and is displayed using a semi-transparent visualization. Figure 3FIn view 360 of FIG. 1 , the first region has been updated to remove tissue classified as mobile tissue. In addition, view 360 includes a second region 354 that at least partially represents additional mobile tissue. The second region 354 is based on a recent intraoral scan and is shown using a semi-transparent visualization. Figure 3H In view 380 , second region 354 has been further updated to remove additional tissue that is classified as mobile tissue.

[0153] Figure 2B FIG. 2 is a flow chart showing a method 260 for displaying a 3D surface view of a dental site during scanning of the dental site according to one embodiment. In an embodiment, the method 260 may be used in conjunction with Figure 2A The method 200 is performed together. At block 262 of the method 260, processing logic receives intraoral scan data (e.g., a set of intraoral scans) of a dental site. At block 264, processing logic processes the intraoral scan data to generate a 3D surface of the dental site. Processing the intraoral scan data may include registering and stitching the intraoral scans together to generate an initial 3D surface, and then registering the intraoral scans to the 3D surface to extend and / or add details to the 3D surface.

[0154] In some embodiments, the processing logic displays a view of the 3D surface before the 3D surface and / or the intraoral scan used to generate the 3D surface has been processed to identify moving tissue, identify excess tissue, identify dental tools, identify hard tissue and soft tissue, etc. The initially displayed 3D surface can be shown using a first visualization, which in embodiments can be a semi-transparent visualization. Alternatively, the 3D surface may not be displayed until further processing is performed on the intraoral scan data and / or the 3D surface.

[0155] At box 266, the processing logic may receive the coordinates. The processing logic may have previously received or identified (e.g., through the application of machine learning) the tooth number of the prepared tooth being scanned. In an embodiment, the tooth number of the prepared tooth being scanned may be used to automatically generate the coordinates. The coordinates may be coordinates on or adjacent to the prepared tooth being scanned in a 2D or 3D coordinate system. In some embodiments, the graphical user interface may include a prompt feature, which may be a dot or point that the user can click and move (e.g., via a drag and drop procedure) to select the coordinates. In some embodiments, the coordinates of the prompt feature (e.g., a point on the screen) may be the received coordinates.

[0156] At box 268, the processing logic performs one or more morphological manipulations on the 3D surface to divide the 3D surface into multiple parts. In most cases, the multiple parts correspond to multiple different teeth. This is especially true if the teeth have been segmented in space (e.g., by a machine learning model trained to segment teeth from an image and / or 3D surface). In some embodiments, before performing one or more morphological parts, the 3D surface and / or data from an intraoral scan used to generate the 3D surface are input into a trained machine learning model that outputs tooth segmentation information. For example, a machine learning model can generate an output including multiple classifications, one of which can correspond to soft tissue, while the other classifications can each correspond to a different tooth. The tooth segmentation information, together with the results of one or more morphological manipulations, can accurately divide the 3D surface into different teeth. Examples of morphological operations that can be performed include erosion operations and / or expansion operations. Morphological operations can easily distinguish, for example, adjacent teeth separated by a thin "bridge" of connective soft tissue and / or hard tissue.

[0157] At box 270, processing logic determines a first tooth among the multiple teeth that is closest to the received coordinates (e.g., in a 3D coordinate space of the 3D surface). This can include finding the point on the 3D surface that is closest to the coordinates, and determining that the closest point is a point of the portion associated with the first tooth. In some embodiments, the shape of the first tooth can be compared to one or more shape criteria, and if the shape does not meet the one or more shape criteria, the shape of the tooth can be modified. For example, if it is determined that the final surface of the tooth is too elongated (e.g., an elliptical shape where the ratio between the first axis and the second axis of the ellipse exceeds a threshold), excess surface can be removed from the side of the tooth (e.g., the long side of the ellipse, causing the ratio between the first axis and the second axis to decrease).

[0158] At box 272, processing logic displays a view of the 3D surface. In an embodiment, processing logic uses a first visualization to determine at least a portion of a first tooth to be displayed. If the tooth closest to the coordinate is a prepared tooth, processing logic may use the identified edge line as a boundary of the prepared tooth. Processing logic may determine that the coordinate is within the edge line in a plane (e.g., an occlusal plane), and may classify points on the prepared tooth that are within the edge line as prepared teeth. The tooth area within the edge line may be shown with the first visualization, while the tooth area outside the edge line and other teeth and / or gums may be shown using a second visualization and / or one or more other visualizations. In one embodiment, the first visualization is used to display the first tooth at box 274, and the second visualization is used to display the remaining teeth at box 276. The first visualization may be an opaque (e.g., 0% transparency) visualization, and the second visualization may be a semi-transparent (e.g., 50-99% transparency) visualization.

[0159] By displaying only the selected teeth using the first visualization (eg, making only the selected teeth opaque and other teeth semi-transparent), proximal surfaces and distal surfaces close to the adjacent teeth are visible through the adjacent teeth due to the semi-transparent visualization of the adjacent teeth.

[0160] Processing logic may receive the new coordinates at block 278. For example, a user may select a prompt feature displayed in a graphical user interface and may drag the prompt feature to a new location (causing the prompt feature to have new coordinates).

[0161] At block 280, processing logic determines a second tooth of the plurality of teeth that is closest to the new coordinates. At block 282, processing logic updates the view of the 3D surface. In particular, processing logic displays the first tooth using the second visualization because it is no longer the selected tooth, and displays the second tooth using the first visualization because it is now the selected tooth.

[0162] If the selected tooth is a prepared tooth, the processing logic may also display one or more segments of a margin line around the selected prepared tooth. The margin line segments may be displayed according to the grade or score they are assigned. If the selected tooth is not a prepared tooth, the margin line may not be shown.

[0163] Figure 4A-4G A diagram of a graphical user interface (GUI) according to an embodiment is shown, which displays a 3D surface 400 of dental sites generated from intraoral scan data during an intraoral scanning session. The GUI also includes a viewfinder image 415 (e.g., a color viewfinder image) showing the current field of view (FOV) of the intraoral scanner, and a scan segment indicator 430.

[0164] The 3D surface 400 is generated by registering and stitching multiple intraoral scans captured during intraoral scanning. As each new intraoral scan is generated, the scan is registered to the 3D surface and then stitched to the 3D surface. Thus, with each intraoral scan, the 3D surface becomes more and more accurate until the 3D surface is complete. A 3D model can then be generated based on the intraoral scans.

[0165] In one embodiment, as shown, the scan segment indicator 430 may include an upper dental arch segment indicator 432, a lower dental arch segment indicator 434, and an occlusal segment indicator 436. When scanning the upper dental arch, the upper dental arch segment indicator 432 may be active (e.g., highlighted). Similarly, when scanning the lower dental arch, the lower dental arch segment indicator 434 may be active, and when scanning the patient's bite, the occlusal segment indicator 436 may be active. In addition, the user can select a specific tooth for preparing the teeth scan segment 438. The GUI shows that tooth 19 has been selected in the teeth indicator 418 for preparing the teeth scan segment 438. The user can select a specific segment indicator 432, 434, 436, 438 to cause the 3D surface associated with the selected segment to be displayed. The user can also select a specific segment indicator 432, 434, 436, 438 to indicate that a scan of the specific segment is to be performed. Alternatively, the processing logic can automatically determine the segment being scanned and can automatically select the segment to make it active.

[0166] The GUI may further include a task bar with multiple operating modes or intraoral scanning stages. The selection of the patient selection mode enables the doctor to enter patient information and / or select a patient that has been entered into the system. The selection of the scanning mode enables an intraoral scan of the patient's mouth. After the scan is completed, selecting the post-processing mode can prompt the intraoral scanning application to generate one or more 3D models based on the intraoral scan and / or the 2D images generated during the intraoral scan, and optionally perform analysis of (multiple) 3D models. Examples of analyses that can be performed include detecting areas of interest, evaluating the quality of (multiple) 3D models, and the like. Once the doctor is satisfied with the 3D model, they can generate an orthodontic and / or restorative prescription. The selection of the prescription fulfillment mode can cause the generated orthodontic and / or restorative dental prescription to be sent to a laboratory or other institution to generate a restorative device (e.g., crown, bridge, denture, etc.) or an orthodontic device (e.g., orthodontic braces).

[0167] In an embodiment, the GUI may include a button and / or indicator 440 showing whether the preparation scan mode and / or the partial retraction scan technique is selected. The user may select or click the button and / or indicator 440 to manually enable or disable the preparation scan mode and / or the partial retraction scan mode. Additionally or alternatively, the GUI may include a button and / or indicator 441 showing whether margin line feedback is active. The user may enable or disable margin line feedback (e.g., where margin line segments are graded and shown in the GUI) by selecting or clicking the button and / or indicator 441.

[0168] like Figure 4AAs shown, when the scan is initially generated (e.g., preparation scan segment), gums 406, first teeth 404 (e.g., preparation teeth), second teeth 405, and third teeth 407 can all be shown using semi-transparent visualization. Hint features 408 are also shown, which can be points or other shapes that the user can drag to any desired position relative to the 3D surface 400. Hint features 408 can be used to determine the current tooth under consideration.

[0169] Figure 4B The 3D surface 400 is shown after processing has been performed to identify hard and soft tissues and determine that the first tooth 404 is the tooth closest to the cue feature 408. The first tooth 404 is shown in an opaque visualization after determining that the first tooth 404 is the tooth closest to the cue feature 408. The second tooth 405, the third tooth 407, and the gums 406 are still shown using a semi-transparent visualization.

[0170] The user may choose to make soft tissue (eg, gums) visible (eg, semi-transparent) or invisible (eg, 100% transparent). Figure 4C The 3D surface 400 is shown after the soft tissue has been classified and then made invisible. Figure 4C Also shown is edge feedback 410 in the form of a plurality of graded and color-coded edge segments.

[0171] The physician can visually inspect the 3D surface 400. The physician can virtually manipulate the 3D surface in up to six degrees of freedom (i.e., translation and / or rotation about one or more of three mutually orthogonal axes) via a user interface using appropriate user controls (hardware and / or virtual) to enable viewing of the 3D surface from any desired direction. This can include, for example, zooming in, out, translating, rotating, and the like. Thus, the physician can view (e.g., visually inspect) the generated 3D surface of the dental site and determine whether the 3D surface is acceptable (e.g., whether the margin lines of the prepared teeth are accurately represented in the 3D surface).

[0172] Figure 4D The 3D surface 400 is shown after the user rotates and zooms in on the 3D surface 400. As shown, because the first tooth 404 is selected (e.g., based on being the tooth closest to the cue feature 408), the first tooth is shown in an opaque visualization, and the second tooth 405 and the third tooth 407 are shown in a semi-transparent visualization. In this example, the gums are not visible. Therefore, the user is able to observe the proximal and distal portions of the first tooth 404, including the proximal and distal regions of the edge line 410, through the semi-transparent views of the first tooth 405 and / or the second tooth 407.

[0173] Figure 4EThe 3D surface 400 is shown after the view is further rotated and the cue feature 408 is dragged to the third tooth 407. Therefore, the third tooth 407 is now the tooth closest to the cue feature 408. Therefore, the third tooth 407 is shown as an opaque visualization and the first tooth 404 is shown as a semi-transparent visualization. Because the third tooth 407 is not a prepared tooth, the edge line of the third tooth 407 is not shown.

[0174] Figure 4F-4G The same scaling and orientation of the 3D surface 400 is shown, but Figure 4F The first tooth 408 is the tooth closest to the hint feature 408, Figure 4G The third tooth 407 is the tooth closest to the hint feature 408. Figure 4F The first tooth 404 is opaque, while Figure 4G The middle third tooth 407 is opaque.

[0175] Figure 5A A flow chart of a method 500 for scanning dental sites according to an embodiment is shown. At block 515 of the method 500, processing logic activates a first intraoral scanning mode. The first intraoral scanning mode can be, for example, a default intraoral scanning mode that is automatically activated. For example, the standard scanning mode can be the default intraoral scanning mode and can be automatically activated. Alternatively, user input selecting the first intraoral scanning mode can be received.

[0176] At block 516, processing logic receives first intraoral scan data. The first intraoral scan data may include an intraoral scan sequence, which may be a sequential and / or raw intraoral scan. In an embodiment, the first intraoral scan data may be received during an intraoral scanning session shortly after the first intraoral scan data is generated according to the intraoral scanner, while the intraoral scanner continues to generate additional intraoral scan data.

[0177] At block 518, processing logic processes the first intraoral scan data using the hybrid algorithm to generate a lower number of hybrid intraoral scans. At block 519, processing logic processes the hybrid intraoral scan using one or more first algorithms to generate a 3D surface of a static dental site. The dental site may be static if no clinically significant changes occur at the dental site within a minimum time period (e.g., within a 30 second time period, within a 1 minute time period, within a 2 minute time period, etc.). The one or more first algorithms may correspond to, for example, a reference Figure 1The first algorithm of the standard scanning logic 119 described above. The first dental site can be or include, for example, an upper dental arch and / or a lower dental arch of a patient. The first dental site (e.g., one of a plurality of dental arches) can include natural teeth thereon (e.g., teeth other than prepared teeth). For example, a physician may have scanned all dental sites (e.g., dental arches) other than prepared teeth.

[0178] At box 520, processing logic receives an indication that the prepared tooth is to be scanned, and / or processing logic otherwise determines that the prepared tooth is to be scanned or is being scanned. The indication may or may not identify a specific prepared tooth to be scanned. If the practitioner does not identify the specific prepared tooth, the processing logic may later automatically identify the prepared tooth based on the shape of the prepared tooth and / or the surrounding surfaces around the prepared tooth (e.g., adjacent teeth). In an alternative embodiment, the indication that the prepared tooth is to be scanned may be received after receiving the second intra-oral scan data. In a further embodiment, the processing logic may automatically determine that the prepared tooth is to be scanned based on an analysis of the second intra-oral scan data received at box 530 (and the preparation scan mode may be activated after receiving and analyzing the second intra-oral scan data).

[0179] At box 525, processing logic activates the preparation scan mode. At box 530, processing logic receives second intraoral scan data. The second intraoral scan data can be an intraoral scan sequence of the prepared tooth. The intraoral scan sequence can be generated when the doctor performs the partial retraction intraoral scanning technique as described above. For example, the doctor can use a tool to retract a small area of ​​gingiva around the prepared tooth, exposing only a small portion of the margin line of the prepared tooth at a time, and can generate one or more scans of the small portion of the margin line and the surrounding surface while exposing the small portion of the margin line. The doctor then moves the tool to expose a new portion of the margin line by retracting the new area of ​​gingiva, and generates one or more additional scans of the newly exposed portion of the margin line. The doctor continues this process until scans are generated for all portions of the margin line.

[0180] At box 532, processing logic processes the second intraoral scan data using one or more second algorithms to generate a second 3D surface of a non-static dental point. In this example, the non-static dental point is a prepared tooth with exposed margin lines and gingiva surrounding the prepared tooth. In one embodiment, in each set of intraoral scans included in the second intraoral scan data, different portions of the margin lines are exposed and different portions of the gingiva are retracted, and thus the prepared teeth are considered non-static dental points. In one embodiment, the prepared tooth includes collapsed gingiva that collapses around the margin lines of the prepared teeth and can be considered a non-static dental point. Alternatively, if the gingiva collapses slowly enough, this situation can be considered a static dental point. The first algorithm used to process the first intraoral scan data at box 519 may not generate an accurate depiction of the margin lines due in part to the non-static nature of the gingiva and margin lines received at box 530. Therefore, one or more second algorithms are used at box 532, which can be configured to operate on scan data of the non-static dental point. The one or more second algorithms can correspond to, for example, reference Figure 1 A second algorithm of the preparation scan logic 118 is described.

[0181] In one embodiment, a hybrid scan is generated for the preparation scan mode without using a hybrid algorithm. Alternatively, a hybrid scan can be generated for a partially contracted intraoral scan mode. In an embodiment, the moving tissue detection algorithm used is more aggressive than the moving tissue detection algorithm of the standard scan mode. This can enable the moving tissue detection algorithm of the preparation scan mode to classify areas that would not be classified as moving tissue by the moving tissue detection algorithm of the standard scan mode as moving tissue. The moving tissue detection algorithm of the standard scan mode is generally not suitable for detecting moving tissue of prepared teeth because prepared teeth are generally associated with changed gums, excessive bleeding, excessive saliva, etc. compared to standard teeth, and the moving tissue of standard teeth is generally limited to, for example, the tongue, dental tools, and cheeks. Therefore, a unique improved moving tissue detection algorithm has been found to better clean up intraoral scans of prepared teeth. The unique improved moving tissue detection algorithm has different thresholds for identifying moving tissue compared to the standard moving tissue detection algorithm. For example, compared to the standard moving tissue detection algorithm, the unique moving tissue detection algorithm can have a lower moving tissue size threshold and can identify smaller areas as moving tissue.

[0182] In an embodiment, a unique moving tissue detection algorithm for preparing the scan mode uses protection based on region classification (eg, determined by a neural network). In an embodiment, regions classified as hard tissue (eg, teeth) are not removed by the moving tissue detection algorithm.

[0183] In an embodiment, a trained machine learning model classifies margin line regions, as discussed in more detail elsewhere herein. The regions classified as margin line regions are much smaller than the entire surface of the scanned and / or 3D surface being processed, and allow for the use of algorithms that are more accurate and / or less time consuming than algorithms (e.g., machine learning models) trained to classify hard and soft tissue.

[0184] In some embodiments, artificial directions are used to find conflicts. Examples of directions that can be used include the direction of the center of mass, the direction orthogonal to the insertion path, etc. For example, the center of mass can be determined, and the vector between the point or pixel and the center of mass can be determined. The vector direction can then be used to find conflicts. In another example, an insertion path can be calculated for inserting a cap, bridge, or other prosthesis onto the prepared tooth. Then, a direction orthogonal to the insertion path can be determined and used to find conflicts. In an embodiment, surface points can be projected along the determined directions, and if there are other parts of the surface along these directions on the projection rays, the surface points can be considered to be moving tissue.

[0185] In one embodiment, processing the second intraoral scan data (which may include multiple intraoral scans) using one or more second algorithms includes determining a conflicting surface from a pair of intraoral scans of the second intraoral scan data. This can be performed as part of a merging algorithm. A first intraoral scan in a pair of intraoral scans is at a first distance from the probe of the intraoral scanner for the conflicting surface, and a second intraoral scan in a pair of intraoral scans is at a second distance from the probe of the intraoral scanner for the conflicting surface. The processing logic can then determine which distance is greater. For example, the processing logic can determine that the first distance is greater than the second distance. The processing logic can also determine the difference between the two distances, and determine whether the difference between the first distance and the second distance is greater than a difference threshold. In response to determining that the difference is greater than the difference threshold, the processing logic discards the representation of the conflicting surface from the intraoral scan with the smaller distance (e.g., from the first intraoral scan in the above example). Then, a second 3D surface of a non-static dental point (e.g., a prepared tooth) can be determined by combining data from the first intraoral scan and the second intraoral scan, wherein the discarded representation of the conflicting surface from the first intraoral scan may not be used for the surface. If the difference is less than the difference threshold, the data of the conflicting surfaces from the two intraoral scans can be averaged together. The difference threshold for the moving tissue detection algorithm of the preparation scan mode can be lower than the difference threshold for the standard scan mode. Therefore, the moving tissue detection algorithm of the preparation scan mode can identify scan areas with certain differences as moving tissue, and these differences are too small for the moving tissue detection algorithm of the standard scan mode to identify them as moving tissue.

[0186] In one embodiment, processing the second intraoral scan data using one or more algorithms configured to determine a three-dimensional surface of a non-static dental point also includes determining a conflicting surface of a pair of intraoral scans based on the second intraoral scan data. This can be performed as part of a merging algorithm. The processing logic can remove a first average curvature or Gaussian curvature of the conflicting surface from the first intraoral scan in the pair. The processing logic can also determine a second average curvature or Gaussian curvature of the conflicting surface from the second intraoral scan in the pair. The processing logic can then determine which average curvature is larger (e.g., the processing logic can determine that the second average curvature is smaller than the first average curvature). The processing logic can also determine the difference between the two average curvatures or Gaussian curvatures, and determine whether the difference between the first average curvature or Gaussian curvature and the second average curvature or Gaussian curvature is greater than a difference threshold. In response to determining that the difference is greater than the difference threshold, the processing logic discards the representation of the conflicting surface from the intraoral scan with the smaller average curvature or Gaussian curvature (e.g., from the first intraoral scan in the above example). A second 3D surface of a non-static dental point (e.g., a prepared tooth) can then be determined by combining data from the first intraoral scan and the second intraoral scan, wherein the discarded representation of the conflicting surface from the first intraoral scan is not used to determine the surface. If the difference is less than a difference threshold, the data of the conflicting surfaces from the two intraoral scans can be averaged together.

[0187] In some embodiments, for the conflicting surfaces, both the distance and the mean or Gaussian curvature are determined, and these values ​​are used together to determine which scan data is to be discarded. In one embodiment, if the difference between the distances is below a distance difference threshold and the difference between the mean or Gaussian curvatures is below a curvature difference threshold, the conflicting surface data is averaged together. If the distance difference is greater than the distance difference threshold, or the mean or Gaussian curvature is greater than the curvature difference threshold, the relevant data from one of the two scans is discarded as described above.

[0188] It should be noted that although areas identified as moving tissue are discarded in the discussion, the data associated with these areas may not actually be discarded. The data may be filtered out so that it is not used in the 3D surface. However, the data may still be used for future processing, such as if later processing indicates that there are gaps or holes in the area where the data was previously discarded. In this case, the "discarded" data can be added back and used in the surface, such as to fill the gaps.

[0189] At block 540, processing logic generates a virtual 3D model using a first 3D surface of a static dental site determined from the first intra-oral scan data and a second 3D surface of a non-static dental site determined from the second intra-oral scan data. In the event that the static dental site includes a non-static dental site (e.g., the static dental site is a dental arch and the non-static dental site is a prepared tooth on the dental arch), portions of the 3D surface of the dental site depicting the non-static dental site may be erased or omitted and replaced by the 3D surface of the non-static dental site.

[0190] Figure 5B A flow chart of a method 550 for scanning prepared teeth using two different scan modes according to an embodiment is shown. At box 555 of method 550, processing logic receives first intraoral scan data. At box 560, processing logic automatically determines that a first scan mode will be used to process the first intraoral scan data. For example, processing logic may determine based on an analysis of the first intraoral scan data that it does not represent prepared teeth, and determines that a standard intraoral scan mode will be used to process the first intraoral scan data. At box 565, processing logic processes the first intraoral scan data using one or more first algorithms to generate a 3D surface of a static dental point (e.g., a dental arch or a portion of a dental arch) according to the first scan mode.

[0191] At box 570, processing logic receives second intraoral scan data. At box 575, processing logic automatically determines that a second scan mode (e.g., a preparation scan mode) is to be used to process the second intraoral scan data. For example, processing logic may determine based on an analysis of the second intraoral scan data that it was generated using a partial retraction scanning technique, and determine that the preparation scan mode is to be used to process the second intraoral scan data. For example, such a determination may be made based on a comparison of intraoral images (e.g., sequentially generated images) from the second intraoral scan data and determining that the difference between them exceeds a threshold. These differences may be compared to one or more thresholds, and if a threshold percentage of the difference exceeds a difference threshold, processing logic may determine that the second intraoral scan mode is to be used. At box 580, processing logic may automatically activate the second scan mode (e.g., the preparation scan mode). At box 585, processing logic processes the second intraoral scan data using one or more second algorithms to generate a 3D surface of a non-static dental point (e.g., a prepared tooth) according to the second scan mode (e.g., the partial retraction scan mode). At box 590, processing logic generates a virtual 3D model using the first 3D surface and the second 3D surface.

[0192] Fig. 6A FIG. 6 is a flow chart showing a method 600 for resolving conflicting scan data of dental sites according to an embodiment. The method 600 may be performed by Figure 1One or more second algorithms of the preparation scan logic 118 are executed to select which scan data to retain and which scan data to discard for conflicting surfaces. In one embodiment, the preparation scan logic 118 executes method 600 as part of a stitching and / or merging algorithm for generating a 3D surface based on multiple intraoral scans. In one embodiment, the processing logic determines that a partial retraction scan of a first prepared tooth will be performed or has been performed, wherein the partial retraction scan includes an intraoral scan of the prepared tooth that is not wrapped with a gingival retraction line. The processing logic receives multiple intraoral scans generated by an intraoral scanner (before or after determining that a partial retraction scanning technique has been performed) and processes the multiple intraoral scans using a stitching or merging algorithm to stitch the multiple intraoral scans together according to a partial retraction intraoral scan pattern. In one embodiment, a stitching algorithm (also called a merging algorithm) executes method 600.

[0193] At block 605 of method 600, processing logic determines a conflict surface for a pair of intraoral scans from intraoral scan data generated by an intraoral scanner. At block 610, for the conflict surface of the first intraoral scan in the pair of intraoral scans, processing logic determines a first distance (also referred to as a first depth and / or a first height) from the probe of the intraoral scanner. The first depth may be a combined depth value (e.g., an average depth or a media depth) based on the depths of some or all pixels included in the conflict surface for the first intraoral scan. At block 615, processing logic determines a first average curvature (or a first Gaussian curvature) of the conflict surface for the first intraoral scan. At block 620, processing logic determines a second distance (also referred to as a second depth and / or a second height) from the probe of the intraoral scanner for the conflict surface of the second intraoral scan in the pair of intraoral scans. The second depth may be a combined depth value (e.g., an average depth or a media depth) based on the depths of some or all pixels included in the conflict surface for the second intraoral scan. At block 625, processing logic determines a second average curvature (or a second Gaussian curvature) of the conflict surface for the second intraoral scan.

[0194] At block 630, processing logic compares the first distance and / or the first mean curvature (or the first Gaussian curvature) to the second distance and / or the second mean curvature (or the second Gaussian curvature). At block 635, processing logic determines a) a first difference between the first distance and the second distance, and / or b) a second difference between the first mean curvature (or the second Gaussian curvature) and the second mean curvature (or the second Gaussian curvature). At block 640, processing logic determines the size of the conflicting surface.

[0195] At block 645, processing logic determines one or more of the following: a) whether the first difference is greater than a first difference threshold, b) whether the second difference is greater than a second difference threshold, c) whether the size of the conflicting surface is less than an upper size threshold. Processing logic may also determine whether the size is greater than a lower size threshold. If the first difference is less than the first difference threshold, the second difference is less than the second difference threshold, the size is less than the upper size threshold, and / or the size is greater than the lower size threshold, the method proceeds to block 675. In one embodiment, if the first difference is greater than the first difference threshold and the size of the conflicting surface is within a particular size range (e.g., less than the upper size threshold and / or greater than the lower size threshold), the method proceeds to block 675. In one embodiment, if the first difference is greater than the first difference threshold, the second difference is greater than the second difference threshold, and the size of the conflicting surface is within a particular size range, the method proceeds to block 675. Otherwise, the method continues to block 650.

[0196] At box 675, processing logic uses a combination of the first intraoral scan data and the second intraoral scan data of the conflicting surface. This can include, for example, averaging the first intraoral scan data and the second intraoral scan data of the conflicting surface. The first intraoral scan data and the second intraoral scan data can be averaged using a weighted average or a non-weighted average. For example, an intraoral scan with a larger distance measurement (e.g., a larger height measurement or a smaller depth measurement) can be assigned a higher weight than an intraoral scan with a smaller distance measurement (e.g., a smaller height measurement or a larger depth measurement).

[0197] At block 650, processing logic determines which intraoral scan has a greater distance and / or a greater average curvature (or a greater Gaussian curvature) for the conflicting surface. If the first intraoral scan has a greater distance and / or a greater average curvature than the second intraoral scan for the conflicting surface, the method continues to block 655. If the second intraoral scan has a greater distance and / or a greater average curvature than the first intraoral scan for the conflicting surface, the method continues to block 665.

[0198] At block 655, processing logic discards and / or ignores the second intraoral scan data of the conflicting surface. At block 660, processing logic uses the first intraoral scan data of the conflicting surface instead of the second intraoral scan data when generating a 3D surface of the virtual 3D model of the scanned dental site.

[0199] At block 665, processing logic discards and / or ignores the first intraoral scan data of the conflicting surface. At block 670, processing logic uses the second intraoral scan data of the conflicting surface instead of the first intraoral scan data when generating a 3D surface of the virtual 3D model of the scanned dental site.

[0200] Figure 6BA solution to conflicting scan data including dental site preparation of teeth and surrounding gums according to an embodiment is shown. Figure 6B , a dental point 680 is scanned by an intraoral scanner. The conflict scan data includes data from a first scan taken when the edge line 694 is exposed and data from a second scan taken when the edge line 694 is covered by gums. The first scan shows surfaces 682, 696 including the exposed edge line 694. The second scan shows surfaces 696, 684 including gums covering the edge line. The edge line is not detected in the second scan. A conflict surface 681 can be determined based on a comparison between the two scans.

[0201] For the first scan, a first distance 690 from the probe is determined for the collision surface 681. In one embodiment, the first distance may be the average distance of the surface 682 from the probe. However, for clarity, the first distance 690 for a specific point on the surface 682 is shown. For the second scan, a second distance 688 from the probe is determined for the surface 684. The second distance may be the average distance of the second scan of the surface 684. However, for clarity, the second distance 688 for a specific point is shown. As shown in FIG. Fig. 6A As described, the first distance and the second distance can be compared, and the difference between these distances can be calculated. The processing logic can determine that the first difference is greater than the difference threshold, and the first distance 690 is greater than the second distance 688. The second scan data for the conflicting surface 681 can then be discarded, so that the generated 3D model depicts the edge line 694.

[0202] A first average curvature may also be calculated for a first surface 682 of the conflict surface 681, and a second average curvature may be calculated for a second surface 684 of the conflict surface 681. As shown, the first surface 682 will have a greater average curvature than the second surface 684. The first average curvature and the second average curvature may be compared, and the results of the comparison may be used as an additional data point to determine which scan should be used to characterize the conflict surface, as referenced. Fig. 6A described.

[0203] Fig. 7AA flow chart of a partial retraction method 700 for scanning a prepared tooth according to an embodiment is shown. At box 705 of method 700, processing logic receives a first intra-oral scan of the prepared tooth after a gum retraction tool has temporarily retracted a first portion of gingiva surrounding the prepared tooth to partially expose a margin line. The first portion of the margin line is exposed in the first intra-oral scan. At box 710, processing logic receives a second intra-oral scan of the prepared tooth after receiving the first intra-oral scan. In the second intra-oral scan, the first portion of the margin line is obscured by the first portion of the gingiva. For example, the gum retraction tool may have been moved to expose a different portion of the margin line so that when the second intra-oral scan is generated, the gingiva collapses back over the first portion of the margin line.

[0204] At block 715, processing logic compares the first intra-oral scan to the second intra-oral scan. At block 720, processing logic identifies a conflicting surface between the first intra-oral scan and the second intra-oral scan at an area of ​​the prepared tooth corresponding to the first portion of the margin line. At block 722, processing logic determines that the conflicting surface satisfies a scan selection criterion. The scan selection criterion may include, for example, a reference to Figure 6A-6B The scan selection criteria may include any criteria described herein. For example, the scan selection criteria may include a first criterion that the average distance difference between scans of the conflicting surface regions is greater than a distance difference threshold. The scan selection criteria may further include a second criterion for selecting scans with a larger average distance. The scan selection criteria may further include a third criterion for selecting scans with a larger average curvature. Other criteria may also be used.

[0205] At box 725, processing logic discards or marks data of the area of ​​the prepared tooth associated with the conflicting surface of the second intra-oral scan. At box 730, processing logic stitches the first intra-oral scan and the second intra-oral scan together to generate a virtual model of the prepared tooth. Data of the area of ​​the prepared tooth from the first intra-oral scan is used to generate the virtual model of the prepared tooth, and data of the area of ​​the prepared tooth from the second intra-oral scan is not used to generate the virtual model of the prepared tooth.

[0206] Figure 7B Another flow chart of a partial retraction method 750 for scanning a prepared tooth according to an embodiment is shown. In one embodiment, the method 750 is performed after receiving an indication that a partial retraction scan is to be performed and activating a partial retraction intraoral scan mode.

[0207] At box 755 of method 750, after the gum retraction tool temporarily retracts a first portion of gum around the prepared tooth to partially expose the margin line, processing logic receives a first intra-oral scan of the prepared tooth, wherein the first portion of the margin line is exposed in the first intra-oral scan, and wherein a second portion of the margin line is obscured by the gum in the first intra-oral scan. At box 760, after the gum retraction tool has temporarily retracted a second portion of gum around the prepared tooth to partially expose the margin line, processing logic receives a second intra-oral scan of the prepared tooth, wherein the second portion of the margin line is exposed in the second intra-oral scan, and wherein the first portion of the margin line is obscured by the gum in the second intra-oral scan. At box 765, processing logic generates a virtual model of the prepared tooth using the first intra-oral scan and the second intra-oral scan, wherein the first intra-oral scan is used to generate a first region of the virtual model representing the first portion of the margin line, and wherein the second intra-oral scan is used to generate a second region of the virtual model representing the second portion of the margin line. In one embodiment, a third portion of the margin line is exposed in the first intraoral scan and the second intraoral scan, and both the first intraoral scan and the second intraoral scan are used to generate a third region of the virtual model representing the third portion of the margin line.

[0208] Figure 7C-7G A partial shrinking method of a scan-prepared tooth according to an embodiment is shown. Figure 7C A first view of the prepared tooth depicted in first intra-oral scan data 770 generated when a first region 774 of the margin line is exposed is shown. Fig.7D A second view of the prepared tooth depicted in second intraoral scan data 778 generated when a second region 776 of the margin line is exposed is shown. Fig. 7E A third view of the prepared tooth depicted in third intraoral scan data 784 generated when a third region 778 of the margin line is exposed is shown. Figure 7F A fourth view of the prepared tooth depicted in fourth intraoral scan data 790 generated when a fourth region 780 of the margin line is exposed is shown. Figure 7G A 3D model 795 of a prepared tooth generated using selected portions of the first intra-oral scan data 770, the second intra-oral scan data 778, the third intra-oral scan data 784, and the fourth intra-oral scan data 790 is shown. The selected portions are those portions of each of the first intra-oral scan data 770, the second intra-oral scan data 778, the third intra-oral scan data 784, and the fourth intra-oral scan data 790 that show exposed edge lines.

[0209] Figure 8A flow chart of a method 800 of identifying and grading margin lines on a 3D surface of a prepared tooth is shown according to an embodiment. At block 805 of method 800, a 3D surface of a prepared tooth is generated from intra-oral scan data including a plurality of intra-oral scans of the prepared tooth.

[0210] At box 810, the intraoral scan data and / or data from the 3D surface are processed to identify possible edge lines of the prepared teeth on the 3D surface. In one embodiment, the data from the 3D surface is processed by projecting the 3D surface onto one or more 2D planes to generate an image of the 3D surface. In one embodiment, the intraoral scan data and / or data from the 3D surface are processed by a machine learning model that has been trained to identify edge lines on the prepared teeth. The machine learning model can output a probability map that indicates the probability that the surface point represents an edge line for each surface point or slice (e.g., a group of surface points) of the data input into the machine learning model. In the case of scanning or projecting onto a 2D surface, the probability map can then be projected back onto the 3D model or 3D surface to assign probability values ​​to points on the 3D model or 3D surface. In an embodiment, for each surface point, the probability that the surface point is classified as an edge line and the confidence that the surface point is classified as an edge line are determined.

[0211] In one embodiment, at block 812, a cost function may be applied to find edge lines using probability values ​​and / or confidence values ​​assigned to points on a 3D model or 3D surface. In one embodiment, processing logic generates a matrix that identifies, for each point (e.g., an edge, vertex, voxel, etc. on a surface of a 3D model), the probability that the point represents an edge line. For example, entries in the matrix that have no chance of representing an edge line have an assigned probability of 0%.

[0212] Processing logic can use a cost function to create a closest contour that passes through points with a high probability of representing an edge line. In one embodiment, the total cost of the contour drawn for the edge line is the sum of all edges (e.g., vertices) included in the edge line, adjusted by the weight associated with each vertex. The weight of each vertex can be a function of the probability assigned to the vertex. The cost of a vertex included in an edge line can be approximately 1 / (A+P), where A is a small constant and P is the probability that the vertex represents an edge line. The smaller the probability of a vertex, the greater the cost of including the vertex in the edge line. The cost of a segment of the edge line can also be calculated based on the sum of the costs of the vertices included in those segments. When the probability is close to 100%, the cost is approximately 1, adjusted by length.

[0213] In an embodiment, a cue point may be used to assist in calculating a margin line. A cue point may be a user-selected point. For example, a user may drag a point or other feature on a tooth of interest (e.g., a prepared tooth under consideration). In an embodiment, a sequence of operations can be used to locate an area of ​​interest. Processing logic may perform a morphological erosion of a tooth surface (e.g., an area classified as hard tissue or a 3D surface of a tooth). This may cause the neighboring teeth of the tooth associated with the cue point (e.g., onto which the cue point has been dragged) to be disconnected by the operation. The processing logic may then take the tooth closest to the cue point and perform a morphological dilation to restore the tooth surface of the tooth close to the cue point. If after this process, there are still two or more neighboring teeth instead of one tooth under consideration, the processing logic may cut off the surface portion that does not fit into a specific bounding box determined by the typical size of the average tooth. Subsequently, the processing logic may take only those vertices of the tooth surface found that have a high probability of being a margin line (according to the neural network prediction). Now, for each connected component of the obtained surface, the processing logic is able to find the corresponding margin line. In an embodiment, each such component is a stripe or ring around the prepared tooth. In the case of a ring, the ring can be cut and further considered as a stripe or a series of stripes. The processing logic can find the two ends of each stripe, for example by running the wave from its interior point and taking the two points that are farthest away. The final edge line will be the path between the two ends of one or more stripes.

[0214] In one embodiment, a path finding operation or algorithm is applied to a 3D model or 3D surface using the values ​​from the matrix as a cost basis. Any path finding algorithm can be used. Some examples of path finding algorithms that can be used include dynamic programming, Dijkstra's algorithm, A* search algorithm, incremental heuristic search algorithm, etc. The path finding algorithm can apply a cost function to determine the path of the edge line.

[0215] The path finding algorithm using the probability of representing edge lines in the matrix as a cost basis can search for the path with maximum cost or the path with minimum cost. The above cost function uses a function based on the inverse of probability to search for minimum cost. Alternatively, a cost function directly based on probability can be used, wherein the maximum cost is searched. If the path finding algorithm is run to maximize the cost, the path between the vertices will be determined, which results in the maximum set of probability values. The probability scores of the vertices can be input into the path finding algorithm to find the path with maximum cost for the probability scores. The path finding algorithm can be used to define the outline representing the edge lines. In some embodiments, the edge lines can be adjusted to be close to those areas with maximum curvature.

[0216] Other techniques may also be used to calculate edge lines based on assigned probability values ​​and / or confidence values.

[0217] At block 815, processing logic ranks or scores the possible edge lines. In an embodiment, processing logic determines a rank or score for each surface point identified as an edge line based on at least one of the determined confidence values ​​and / or cost functions. In an embodiment, processing logic determines groups of surface points identified as edge lines that are close to each other and have similar ranks / scores in edge line segments (also referred to as edge line portions). Each edge line segment may be assigned a score / rank based on the scores / ranks of its constituent surface points.

[0218] In one embodiment, at block 820, for each surface point identified as a margin line and / or for each margin line segment, processing logic determines a) the sharpness of curvature, b) whether the margin line portion includes an s-shaped curve, c) whether the margin line portion is located in or adjacent to a gap in the margin line, and / or d) the number of intraoral scans associated with the margin line portion. At block 825, processing logic determines one or more of the following for each surface point identified as a margin line and / or for each margin line segment: a) Whether the curvature sharpness is lower than the curvature sharpness threshold; b) whether the surface point or edge segment includes an S-shaped curve; c) whether the surface point or edge line segment is located in or adjacent to a gap in the edge line (eg, whether the distance between the surface point and the gap is below a distance threshold); and d) Whether the number of intra-oral scans associated with the surface point or edge segment is below a threshold number of intra-oral scans (eg, used to determine whether the number of intra-oral scans of a surface point on a 3D surface is below a threshold).

[0219] At block 830, processing logic determines whether the answer to any of a, b, c, or d above is "yes". If so, the method continues to block 835, and processing logic lowers the rank / score of those positions for which the answer to a, b, c, and / or d is yes. If the answer to a, b, c, and d is all "no", the method proceeds to block 840.

[0220] At box 840, processing logic marks edge lines on the 3D surface. Visualization based on the score / grade associated with each edge line segment can be used to mark the edge line segment. In an embodiment, the edge line segment is divided into three separate score ranges, each score range has an assigned visualization. For example, in an embodiment, a low score range can be marked with red, a medium score range can be marked with yellow, and a high score range can be marked with green. In some embodiments, for better visual perception, segments with sizes below a size threshold can be absorbed into adjacent segments. In some embodiments, processing logic stores the change history of the segments to perform time-based processing on the edge line. This can include average, non-decreasing confidence, etc. of the points.

[0221] Fig. 9 An example workflow of a method 900 for generating an accurate virtual 3D model of a dental site and manufacturing a dental prosthesis based on the virtual 3D model according to an embodiment of the present invention is shown. The operations of the workflow can be performed at a dental clinic 105 or a dental laboratory 110. Those operations performed at a dental clinic 105 can be performed during a single patient visit or during multiple patient visits. The operations listed under the dental clinic 105 can be performed, for example, by an intraoral scanning application 115. The operations listed under the dental laboratory 110 can be performed, for example, by a dental modeling application 120.

[0222] The method 900 may begin at box 915, where processing logic executing on a computing device associated with the dental clinic 105 receives intraoral scan data. The intraoral scan data may have been generated by the intraoral scanner 150 during an intraoral scan process. As described above, the intraoral scan data may have been generated according to a standard preparation scan procedure or according to a partial shrinkage scan procedure. At box 918, the processing logic generates a virtual 3D model and / or 3D surface of one or more dental locations based on the intraoral scan data, as described above. The virtual 3D model and / or 3D surface may be an entire dental arch or a portion of a dental arch (e.g., including a portion of prepared teeth and adjacent teeth).

[0223] At box 920, processing logic performs automatic margin line marking on the 3D model and / or 3D surface. In one embodiment, automatic margin line marking is performed by first generating appropriate data inputs based on the 3D model and / or 3D surface (e.g., one or more images or height maps of the 3D model). These inputs include any information useful for margin line detection generated during scanning. The input may include image data, such as a 2D height map that provides depth values ​​at each pixel location, and / or a color image of the actual or estimated color of a given 2D model projection. 3D inputs may also be used, and the 3D inputs may include connections between Cartesian positions and vertices (i.e., a mesh). Each image may be a 2D or 3D image generated by projecting a portion of a 3D model or 3D surface representing a particular tooth onto a 2D surface. Different images may be generated by projecting a 3D model or 3D surface onto different 2D surfaces. In one embodiment, one or more generated images may include a height map that provides depth values ​​for each pixel of the image. Alternatively or additionally, an intraoral image for generating a 3D model or 3D surface may be used. The generated images and / or received intraoral scans can be processed by a machine learning model that has been trained to identify margin lines on prepared teeth. The machine learning model can output a probability map that indicates, for each pixel of an image or surface point of 3D data input into the machine learning model, the probability that the pixel or surface point represents a margin line. In the case of an image, the probability map can then be projected back onto the 3D model or 3D surface to assign probability values ​​to points on the 3D model or 3D surface. A cost function can then be applied to use the probability values ​​assigned to points on the 3D model or 3D surface to find margin lines. Other techniques can also be used to calculate margin lines based on the assigned probability values.

[0224] At block 925, processing logic calculates one or more edge line quality scores (also referred to as grades). Each edge line quality score may be based on a cost value for the edge line (or edge line segment) calculated using a cost function. In one embodiment, an edge line quality score is determined for the entire edge line. In one embodiment, multiple additional edge line quality scores are calculated, where each edge line quality score is for a specific segment of the edge line. Figure 8 As described above, the edge line quality score may be adjusted according to one or more adjustment criteria.

[0225] At box 930, processing logic can mark edge segments on a 3D model or 3D surface based on quality scores. For example, the edge line quality scores of one or more edge segments can be compared with one or more quality thresholds. Any score representing cost exceeds maximum cost may fail to meet the upper quality threshold limit. Those segments that fail to meet the upper quality threshold limit can be marked with a mark that distinguishes them from the remainder of the edge line. For example, red or other visualizations can be used to highlight low-quality edge segments on a 3D model or 3D surface. Other segments with other quality levels can be represented using other colors and / or visualizations.

[0226] At block 935, the doctor may provide feedback indicating that the 3D model is acceptable or that the 3D model should be updated. If the doctor indicates that the 3D model is acceptable, the 3D model is sent to the dental laboratory 110 for review, and the method continues to block 945. If the doctor indicates that the 3D model is not acceptable, the method continues to block 940.

[0227] At box 940, the doctor can use the user interface to indicate one or more areas of the 3D model to be rescanned. For example, the user interface may include an eraser function that enables the doctor to draw or circle a portion of the 3D model. The drawn area or the area inside the circle can be erased, and the remaining part of the 3D model can be locked. The locked area of ​​the 3D model will not be modified by the new intraoral scan data. Alternatively, a one-way lock can be applied, and the locked area can be modified under certain conditions. Alternatively, the processing logic can automatically select an area depicting an edge segment with a low quality score to erase, and the remaining part of the 3D model can be automatically locked. The processing logic can then indicate to the doctor with a graphic where to place the intraoral scanner 150 to generate replacement image data. The method can then return to box 915, and new intraoral image data depicting the erased area can be received. New intraoral image data can be generated using a standard scanning procedure or a partial contraction scanning procedure.

[0228] At block 918, the 3D model may be updated based on the new image data. In one embodiment, the unlocked portions of the 3D model are updated based on the new image data, but the locked regions are not updated. In one embodiment, a one-way lock is used to lock one or more regions. For a one-way lock, if adjusting the regions using the new image data would result in a degradation of the representation of those regions, the locked regions are not affected by the new image data. However, if the representation of those regions would be improved using the new image data, the new image data may be used to update those regions. In one embodiment, processing logic processes the new image data (e.g., using a trained machine learning model) to determine a quality score for the new image data. In some embodiments, multiple quality scores are determined for the new image data, wherein each quality score may be associated with a different region of the dental site. Additionally, quality scores may be determined for one or more regions of the 3D model. If the score for a region from the new image data is higher than the score for the same region from the 3D model, the image data may be used to update the region of the 3D model. If the score for a region of the new image data is less than or equal to the score for the same region from the 3D model, the new image data may not be used to update the region of the 3D model.

[0229] The operations of blocks 920 - 935 may then be repeated based on the updated 3D model.

[0230] At box 945, the laboratory technician can view the margin lines in the 3D model (e.g., using the dental modeling application 120). Alternatively or additionally, the processing logic (e.g., the processing logic of the dental modeling application 120) can process the 3D model to automatically determine and / or grade the margin lines. In one embodiment, viewing the margin lines at box 945 includes performing operations 920-930. At box 950, the processing logic determines whether to continue to use the 3D model to make dentistry or return the 3D model to the dental clinic 105. If the margin line meets the minimum quality threshold, the method proceeds to box 960. If the margin line does not meet the minimum quality threshold, the method continues to box 955, and the 3D model is returned to the dental clinic 105 to enable the doctor to generate further intraoral scans of the dental points. At box 955, the laboratory technician can manually mark unclear segments of the margin line. Alternatively, at box 955, the unclear segments can be automatically marked by the processing logic, or may have been marked at box 945. A message is then sent to the doctor requesting that additional intraoral images be generated. The message may provide a copy of the 3D model showing the area that should be re-imaged.

[0231] At box 960, the margin line can be automatically adjusted. In some cases, at box 950, the processing logic may determine that the margin line quality is insufficient, but for some reason the physician may not be able to collect new images of the dental site. In this case, the processing logic may proceed to box 960 even if the margin line has an unacceptable level of quality. In this case, the margin line can be automatically adjusted at box 960. Alternatively, the margin line can be manually adjusted using, for example, a CAD tool. In one embodiment, the margin line is adjusted by generating an image of the 3D model (e.g., by projecting the 3D model onto a 2D surface) and processing the image using a trained machine learning model that has been trained to correct margin lines in images of prepared teeth. In one embodiment, one or more operations of method 2000 of Figure 20 are performed at box 960.

[0232] At box 965, processing logic generates a dental prosthesis using the virtual 3D model of the dental site. In one embodiment, the virtual 3D model is input into a rapid prototyping machine (e.g., a 3D printer), and a physical model of the (multiple) dental sites (e.g., prepared teeth and adjacent teeth) is generated. The physical 3D model can then be used to generate the dental prosthesis. Alternatively, a virtual 3D model of the dental prosthesis can be generated based on the virtual 3D model of the (multiple) dental sites, and the virtual 3D model of the dental prosthesis can be used to directly manufacture the dental prosthesis using 3D printing. At box 970, the dental prosthesis can then be transported to the dental clinic 105.

[0233] Fig.10 A flowchart of a method 1000 for segmenting intraoral scan data and / or a 3D surface into various classifications and updating the 3D surface based on the classifications is shown according to an embodiment. At box 1005 of method 1000, processing logic receives an intraoral scan. At box 1010, processing logic processes the intraoral scan to identify moving tissue and / or dental tools in the intraoral scan. In one embodiment, the intraoral scan (or data from the intraoral scan) is input into a trained machine learning model. The machine learning model can then output a probability map that, for each point or pixel in the intraoral scan, includes the probability that the point or pixel is moving tissue and / or a dental tool. In some embodiments, the intraoral scan is registered to the 3D surface, and the 3D surface is updated based on the intraoral scan.

[0234] At box 1015, processing logic processes the intraoral scan data and / or the 3D surface to classify hard tissue and soft tissue in the intraoral scan and / or the 3D surface. In one embodiment, the intraoral scan (or data from the intraoral scan) or data from the 3D surface is input into a trained machine learning model. The machine learning model can then output a probability map that, for each point or pixel in the intraoral scan and / or the 3D surface, includes a first probability that the point belongs to a first class associated with hard tissue and a second probability that the point belongs to a second class associated with soft tissue. In an embodiment, a single trained machine learning model performs the operations of boxes 1010 and 1015.

[0235] At block 1020, processing logic determines an area of ​​the intraoral scan and / or 3D surface that is classified as hard tissue (e.g., points or pixels with a probability of representing hard tissue above a threshold, such as 70%, 80%, 90%, etc.). Processing logic may determine a group of contiguous points or pixels classified as hard tissue as a region. Processing logic may then determine an offset inward into the region to determine an updated region of hard tissue.

[0236] At box 1025, processing logic locks the determined updated area of ​​the intraoral scan and / or 3D surface that is classified as hard tissue. Algorithms such as a moving tissue detection and removal algorithm and / or an excess material detection and removal algorithm do not remove the lock point. At box 1030, processing logic updates the intraoral scan and / or 3D surface by removing the identified moving tissue and / or dental tools from the intraoral scan and / or 3D surface. However, even if the lock area is also classified as moving tissue or dental tools, it is not removed.

[0237] Fig.11 105 and a model application workflow 1117 for an intraoral scanning application according to an embodiment of the present disclosure are shown. In an embodiment, the model training workflow 1105 may be executed at a server that may or may not include an intraoral scanning application, and the trained model is provided to the intraoral scanning application (e.g., Figure 1 The model training workflow 1105 and the model application workflow 1117 may be executed by processing logic executed by a processor of the computing device. For example, one or more of these workflows 1105, 1117 may be implemented by one or more machine learning modules or other software implemented in the intraoral scanning application 115 and / or in a Fig.13 The illustrated computing device 1300 is implemented by firmware executing on a processing device.

[0238] The model training workflow 1105 is used to train one or more machine learning models (e.g., deep learning models) to perform one or more classification, segmentation, detection, recognition, etc. tasks on intraoral scan data (e.g., 3D scans, height maps, 2D color images, NIRI images, etc.) and / or 3D surfaces generated based on the intraoral scan data. The model application workflow 1117 applies one or more trained machine learning models to perform classification, segmentation, detection, recognition, etc. tasks on intraoral scan data (e.g., 3D scans, height maps, 2D color images, NIRI images, etc.) and / or 3D surfaces generated based on the intraoral scan data. One or more machine learning models can receive and process 3D data (e.g., 3D point clouds, 3D surfaces, parts of 3D models, etc.). One or more machine learning models can receive and process 2D data (e.g., 2D images, height maps, projections of 3D surfaces on planes, etc.).

[0239] A variety of different machine learning outputs are described herein. A specific number and arrangement of machine learning models are described and illustrated. However, it should be understood that the number and type of machine learning models used, as well as the arrangement of such machine learning models, can be modified to achieve the same or similar end results. Therefore, the arrangement of machine learning models described and illustrated are merely examples and should not be construed as limiting.

[0240] In an embodiment, one or more machine learning models are trained to perform one or more of the following tasks. Each task can be performed by a separate machine learning model. Alternatively, a single machine learning model can perform each task or a subset of tasks. Additionally or alternatively, different machine learning models can be trained to perform different combinations of tasks. In one example, one or several machine learning models can be trained, where the trained ML model is a single shared neural network with multiple shared layers and multiple higher-level different output layers, where each output layer outputs a different prediction, classification, recognition, etc. The tasks that one or more trained machine learning models can be trained to perform are as follows: I) Dental tissue segmentation - This may include performing point-level classification (e.g., pixel-level classification, voxel-level classification, and / or surface point-level classification) of different types of dental tissue from an intraoral scan, a group of intraoral scans, a 3D surface generated from multiple intraoral scans, a 3D model generated from multiple intraoral scans, etc. The different types of dental tissue may include, for example, hard tissue (e.g., teeth) and soft tissue (e.g., gums or gums). II) Moving tissue segmentation - this can include performing point-level classification (e.g., pixel-level classification, voxel-level classification, and / or surface point-level classification) of moving tissue, dental tools, and / or non-moving tissue from a set of intraoral scans, a 3D surface generated from multiple intraoral scans, a 3D model generated from multiple intraoral scans, etc. III) Margin Line Identification / Marking - This can include pixel-level or point-level identification / classification of margin lines around the prepared teeth based on an intraoral scan, a set of intraoral scans, a 3D surface generated from multiple intraoral scans, a 3D model generated from multiple intraoral scans, etc. This can also include marking the identified margin lines. Margin line identification and marking are described in U.S. Publication No. 2021 / 0059796, published on March 4, 2021, which is incorporated herein by reference. IV) Excess Material Segmentation - This may include segmenting excess material from a set of intraoral scans, a 3D surface generated from multiple intraoral scans, a 3D model generated from multiple intraoral scans, etc. Perform point-level classification (e.g., pixel-level classification, voxel-level classification, and / or surface point-level classification)

[0241] It should be noted that for any of the above tasks associated with intraoral scans / 3D surfaces / 3D models, although they are described as being performed based on inputs of intraoral scans, 3D surfaces, and / or 3D models, it should be understood that these tasks may also be performed based on 2D images, such as color images, NIRI images, etc. Any of these tasks may be performed using an ML model with multiple input layers or channels, where the first layer may include an intraoral scan / 3D surface (or a projection of a 3D surface) / 3D model (or a projection of a 3D model), the second layer may include a 2D color image, the third layer may include a 2D NIRI image, etc. In another example, the first layer or channel may include a first 3D scan, the second layer or channel may include a second 3D scan, and so on.

[0242] A machine learning model that can be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network. Artificial neural networks typically include a feature representation component with a classifier or regression layer that maps features to a desired output space. For example, a convolutional neural network (CNN) carries multiple layers of convolution filters. Pooling is performed at the lower layers, and nonlinear problems can be solved. Multilayer perceptrons are usually attached above the lower layers, and the top features extracted by the convolution layer are mapped to the decision layer (e.g., classification output). Deep learning is a class of machine learning algorithms that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each subsequent layer uses the output of the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks include a hierarchical structure in which different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in an image recognition application, the raw input may be a matrix of pixels; the first representation layer may extract pixels and encode edges; the second layer may compose and encode the arrangement of edges; the third layer may encode higher-level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer may recognize the scanned character. It is noteworthy that the deep learning process is able to learn by itself which features are best placed at which level. The "depth" in "deep learning" refers to how many layers of transformations the data undergoes. More precisely, deep learning systems have considerable credit assignment path (CAP) depth. CAP is the chain of transformations from input to output. CAP describes the potential causal relationship between input and output. For a feedforward neural network, the depth of CAP can be the depth of the network and can be the number of hidden layers plus 1. For a recurrent neural network where the signal may propagate through a layer more than once, the CAP depth may be infinite.

[0243] Training of a neural network can be accomplished in a supervised learning fashion, which involves feeding a training dataset consisting of labeled inputs through the network, observing its output, defining an error (by measuring the difference between the output and the labeled value), and adjusting the network weights across all layers and nodes to minimize the error using techniques such as deep gradient descent and backpropagation. In many applications, repeating this process for many labeled inputs in the training dataset results in a network that is able to produce the correct output when presented with inputs different from those in the training dataset. In high-dimensional settings, such as large images, this generalization is achieved when a sufficiently large and diverse training dataset is available.

[0244] For the model training workflow 1105, a training dataset containing hundreds, thousands, tens of thousands, hundreds of thousands, or more intraoral scans 1110A, images, and / or 3D models / surfaces 1110B should be used to generate a training dataset at box 1135. In an embodiment, up to millions of patient dentition cases that may have undergone prosthetic procedures and / or orthodontic procedures may be used to form a training dataset, wherein each case may include various indicia of one or more types of useful information. For example, each patient record may include data showing a 3D model, intraoral scan, height map, color image, NIRI image, etc. of one or more dental locations, data showing pixel-level segmentation of the data (e.g., 3D model, intraoral scan, height map, color image, NIRI image, etc.) into various dental classifications (e.g., teeth, restorations, gingiva, mobile tissue, maxillary, etc.), data showing one or more classifications assigned to the data (e.g., scan role, intraoral, non-intraoral, lingual view, buccal view, occlusal view, anterior view, left side view, right side view, etc.), and the like. The data can be processed to generate one or more training data sets at box 1135 for training one or more machine learning models at box 1140.

[0245] In one embodiment, generating one or more training data sets 1136 includes collecting one or more intraoral scans with labels 1110A and / or one or more 3D models / surfaces 1110B with labels. The labels used can depend on what the specific machine learning model is to be trained to do. For example, to train a machine learning model to perform classification of dental sites (e.g., tissue classifier 1152), the training data set can include pixel-level or point-level labels of hard and soft tissues of dental sites.

[0246] The processing logic may collect a training data set comprising 2D or 3D images, intraoral scans, 3D surfaces, 3D models, height maps, etc. of dental points (e.g., dental arches) with one or more associated labels (e.g., pixel-level labeled dental classifications in the form of maps (e.g., probability maps), image-level labels of dental tissues, etc.). In an embodiment, one or more images, scans, surfaces, and / or models in the training data set and optionally associated probability maps may be resized. For example, a machine learning model may be used for images with a certain pixel size range, and if one or more images fall outside of these pixel size ranges, their sizes may be resized. For example, methods such as nearest neighbor interpolation or box sampling may be used to resize the images. The training data set may be augmented additionally or alternatively. Training of large-scale neural networks typically uses tens of thousands of images, which are not easily available in many real-world applications. Data augmentation can be used to artificially increase the effective sample size. Common techniques include randomly rotating, shifting, shearing, flipping, etc., existing images to increase the sample size.

[0247] To implement the training, the processing logic inputs the training data set to one or more untrained machine learning models. Before the first input is input to the machine learning model, the machine learning model may be initialized. The processing logic trains the untrained machine learning model based on the training data set to generate one or more trained machine learning models that perform various operations as described above.

[0248] Training can be performed by inputting one or more images, scans, or 3D surfaces (or data from images, scans, or 3D surfaces) into the machine learning model one at a time. Each input can include data of an image, intraoral scan, or 3D surface from a training data item of a training data set. The training data item can include, for example, a height map and an associated probability map, which can be input into the machine learning model. As described above, the training data item can also include color images, images generated under specific lighting conditions (e.g., UV or IR radiation), etc. In addition, the pixels of the image can include height values, or can include both height values ​​and intensity values. The data input into the machine learning model can include a single layer (e.g., height values ​​from only a single image) or multiple layers. If multiple layers are used, one layer can include height values ​​from an image / scan / surface, and a second layer can include intensity values ​​from an image / scan / surface. Additionally or alternatively, the additional layers can include three layers for color values ​​(e.g., a separate layer for each color channel, such as an R layer, a G layer, and a B layer), a layer of pixel information of an image generated under specific lighting conditions, etc. In some embodiments, data from multiple images / scans / surfaces are input together into a machine learning model, where the multiple images / scans / surfaces may all be the same dental site. For example, a first layer may include height values ​​from a first scan of a dental site, a second layer may include height values ​​from a second scan of the dental site, a third layer may include height values ​​from a scan of the dental site, and so on.

[0249] The machine learning model processes inputs to generate outputs. The artificial neural network includes an input layer consisting of values ​​in the data points (e.g., intensity values ​​and / or height values ​​of pixels in the height map). The next layer is called a hidden layer, and each node of the hidden layer receives one or more input values. Each node contains parameters (e.g., weights) applied to the input values. Therefore, each node essentially inputs the input value into a multivariate function (e.g., a nonlinear mathematical transformation) to generate an output value. The next layer can be another hidden layer or an output layer. In either case, the nodes of the next layer receive output values ​​from the nodes of the previous layer, each node applies weights to these values, and then generates its own output value. This can be performed at each layer. The last layer is the output layer, in which there is a node for each class, prediction, and / or output that the machine learning model can generate. For example, for an artificial neural network trained to perform dental tissue classification, there may be a first classification (hard tissue) and a second classification (soft tissue). Other possible classifications may include a third classification (mobile tissue), a fourth classification (marginal line), and / or one or more additional dental classifications (e.g., a fifth classification for excess material). Moreover, the classification, prediction, etc. can be determined for each pixel or point in the image / scan / surface, the classification, prediction, etc. can be determined for the entire image / scan / surface, or the classification, prediction, etc. can be determined for each region or group of pixels in the image / scan / surface. For pixel-level or point-level segmentation, for each pixel / point in the image / scan / surface, the final layer applies the probability that the pixel / point of the image / scan / surface belongs to the first classification, the probability that the pixel / point belongs to the second classification, etc.

[0250] Thus, the output may include one or more predictions and / or one or more probability maps. For example, for each pixel / point in the input image / scan / surface, the output probability map may include a first probability that the pixel belongs to a first dental classification, a second probability that the pixel belongs to a second dental classification, etc. For example, the probability map may include the probability of a pixel belonging to a tissue classification representing teeth or a tissue classification representing gums.

[0251] The processing logic may then compare the generated probability map and / or other outputs with known probability maps and / or labels contained in the training data items. The processing logic determines an error (i.e., a classification error) based on the difference between the output probability map and / or label and the provided probability map and / or label. The processing logic adjusts the weights of one or more nodes in the machine learning model based on the error. An error term or increment may be determined for each node in the artificial neural network. Based on the error, the artificial neural network adjusts one or more parameters (the weights of one or more inputs of the node) for one or more of its nodes. The parameters may be updated in a back-propagation manner so that the nodes of the highest layer are updated first, followed by the nodes of the next layer, etc. The artificial neural network comprises multiple layers of "neurons", each of which receives input values ​​from the previous layer of neurons. The parameters of each neuron include weights associated with the values ​​received from each neuron of the previous layer. Therefore, adjusting the parameters may include adjusting the weights of each input of one or more neurons assigned to one or more layers in the artificial neural network.

[0252] Once the model parameters are optimized, model validation can be performed to determine whether the model has improved and to determine the current accuracy of the deep learning model. After one or more rounds of training, the processing logic can determine whether the stopping criteria are met. The stopping criteria can be a target level of accuracy, a target number of processed images from a training data set, a target change in parameters on one or more previous data points, a combination thereof, and / or other criteria. In one embodiment, the stopping criteria are met when at least a minimum number of data points have been processed and at least a threshold accuracy has been reached. The threshold accuracy can be, for example, 70%, 80%, or 90% accuracy. In one embodiment, the stopping criteria are met if the accuracy of the machine learning model has stopped improving. If the stopping criteria are not met, further training is performed. If the stopping criteria have been met, the training can be completed. Once the machine learning model has been trained, a retained portion of the training data set can be used to test the model.

[0253] For example, in one embodiment, a machine learning model (e.g., tissue classifier 1152) is trained to segment an intraoral image by classifying regions of the intraoral image into two or more classifications. A similar process can be performed to train a machine learning model to perform other tasks such as those described above. A set of many (e.g., thousands to millions) of intraoral scans of 3D models and / or dental arches with labeled dental classifications can be collected. In one example, each point in the 3D model can include a label with a first value representing a first label of a natural tooth, a second value representing a second label of a restoration, and a third value representing a third label of a gum / gingiva. For example, one of the three values ​​can be 1 and the other two values ​​can be 0.

[0254] The tissue classifier 1152 may include one or more machine learning models that operate on 3D data, or may include one or more machine learning models that operate on 2D data. If the tissue classifier 1152 includes a machine learning model that operates on 2D data, a set of images (e.g., height maps) may be generated for each 3D model with a labeled dental classification. Each image may be generated by projecting the 3D model (or a portion of the 3D model) onto a 2D surface or plane. In some embodiments, different images of the 3D model may be generated by projecting the 3D model onto different 2D surfaces or planes. For example, a first image of the 3D model may be generated by projecting the 3D model onto a 2D surface from a top-down perspective, a second image may be generated by projecting the 3D model onto a 2D surface from a first side perspective (e.g., an oral perspective), a third image may be generated by projecting the 3D model onto a 2D surface from a second side perspective (e.g., a lingual perspective), and the like. Each image may include a height map that includes a depth value associated with each pixel of the image. For each image, a probability map or mask can be generated based on the dental classifications labeled in the 3D model and the 2D surface onto which the 3D model is projected. The size of the probability map or mask can have a pixel size equal to the generated image. Each point or pixel in the probability map or mask can include a probability value indicating the probability that the point represents one or more dental classifications. For example, there can be two dental classifications, including a first dental classification representing hard tissue and a second dental classification representing soft tissue. A point with the first dental classification can have a value (1, 0) (the probability of the first dental classification is 100% and the probability of the second dental classification is 0%), and a point with the second dental classification can have a value (0, 1). If the machine learning model is trained to perform image-level classification / prediction as opposed to pixel-level classification / segmentation, then a single value or label can be associated with the generated image, as opposed to a schematic diagram with pixel-level values.

[0255] A training data set can be collected, where each data item in the training data set can include an image (e.g., an image including a height map) or a 3D surface and an associated probability map (which can be a 2D map if associated with an image, or a 3D map if associated with a 3D surface) and / or other labels. Additional data can also be included in the training data items. The accuracy of the segmentation can be improved through additional classifications, inputs, and multi-view support. Multiple sources of information can be incorporated into the model input and used jointly for prediction. Multiple dental classifications can be predicted simultaneously from a single model or using multiple models. Multiple problems can be solved simultaneously: tissue classification, moving tissue detection / classification, margin line detection / classification, excess material classification / detection, etc. The accuracy is higher than traditional image and signal processing methods.

[0256] The additional data may include color image data. For example, for each intraoral scan or image (which may be monochrome), there may also be a corresponding color image. Each data item may include a scan (e.g., a height map) as well as a color image. There may be two different types of color images available. One type of color image is a viewfinder image and the other type of color image is a scan texture. The scan texture may be a combination or blend of multiple different viewfinder images. Each intraoral scan may be associated with a corresponding viewfinder image generated at approximately the same time as the intraoral image was generated. If hybrid scans are used, each scan texture may be based on a combination of viewfinder images associated with the original scans used to produce the particular hybrid scan.

[0257] The default approach may be based solely on depth information and still allow for the differentiation of several dental classifications, such as teeth, gums, mobile tissue, excess material, etc. However, sometimes depth information is not enough to obtain good accuracy. For example, a partially scanned tooth may look like a gum, or even excess material in monochrome. In this case, color information can help. In one embodiment, color information is used as an additional 3 layers (e.g., RGB), resulting in a 4-layer input to the network. Two types of color information can be used, which may include viewfinder images and scanned textures. Viewfinder images are of better quality, but need to be aligned with the height map. Scanned textures are aligned with the height map, but may have color artifacts.

[0258] Another type of additional data may include images generated under specific lighting conditions (e.g., images generated under ultraviolet or infrared lighting conditions). The additional data may be 2D or 3D images and may or may not include a height map.

[0259] In some embodiments, the groups of data points are associated with the same dental site and are labeled sequentially. In some embodiments, a recursive neural network is used and the data points are input into the machine learning model in ascending order during training.

[0260] In some embodiments, each image or scan includes two values ​​for each pixel in the image, where the first value represents height (e.g., providing a height map) and the second value represents intensity. Both the height value and the intensity value can be used to train the machine learning model.

[0261] In one example, a confocal intraoral scanner can determine the height of a point on a surface (captured by pixels of an intraoral image) based on a focus setting of the intraoral scanner that results in a maximum intensity at that point on the surface. The focus setting provides a height or depth value for the point. Typically the intensity value (called a level) is discarded. However, the intensity value (level) associated with the height or depth value can be retained and can be included in the input data provided to the machine learning model.

[0262] Once one or more trained ML models are generated, they can be stored in the model storage 1145 and can be added to an intraoral scanning application (e.g., the intraoral scanning application 115). The intraoral scanning application 115 can then use the one or more trained ML models.

[0263] In one embodiment, the model application workflow 1117 includes one or more trained machine learning models, which are used as tissue classifiers 1152, mobile tissue / tool ​​detectors 1154 and / or edge line detectors 1170. In an embodiment, these logics can be implemented as separate machine learning models or a single combined machine learning model. For example, tissue classifiers 1152, mobile tissue / tool ​​detectors 1154 and / or edge line detectors 1170 can share one or more layers of a deep neural network. However, each of these logics can include different higher-level layers of a deep neural network trained to generate different types of outputs. For convenience, the examples shown only show some of the functions set forth in the above-mentioned task list. However, it should be understood that any other tasks can also be added to the model application workflow 1117.

[0264] According to one embodiment, for the model application workflow 1117, the intraoral scanner generates a sequence of intraoral scans 1148. The 3D surface generator 1155 can perform registration between these intraoral scans, stitch the intraoral scans together, and generate a 3D surface 1160 from the intraoral scans. As further intraoral scans are generated, these scans can be registered and stitched into the 3D surface 1160, thereby increasing the size of the 3D surface 1160 and the amount of data of the 3D surface 1160.

[0265] The intraoral scan data 1148 may be input to a tissue classifier 1152, which may include a trained neural network. Based on the intraoral scan data 1148, the tissue classifier 1152 outputs information about a dental point tissue classification 1162, which may be a point-level (e.g., pixel-level) classification of the input data. This may include outputting a set of classification probabilities for each pixel / point and / or a single classification for each pixel / point. The output dental point tissue classification 1162 may be, for example, a mask or map of classifications and / or classification probabilities. In one embodiment, the dental point tissue classifier 1152 identifies, for each pixel / point, whether it represents hard tissue or soft tissue.

[0266] In one embodiment, the moving tissue / tool ​​detector receives the intraoral scan data 1148 and classifies pixels / points representing moving tissue (excess tissue). The moving tissue / tool ​​detector may then output a moving tissue / tool ​​classification (eg, as a probability map or mask).

[0267] The edge line detector 1170 can receive data from the 3D surface 1160 and can process the data using a trained machine learning model to output a probability map or mask having pixel-level or point-level probabilities of a pixel / point belonging to a first classification associated with an edge line or a second classification not associated with an edge line. The edge line detector 1170 can then classify the classified edge line regions / pixels / points. Thus, an edge line classification 1175 can be output by the edge line detector 1170.

[0268] The user interface 1180 may receive a 3D surface and output a view of the 3D surface 1185. In an embodiment, the user interface 1180 also receives a tissue classification 1162, a moving tissue / tool ​​classification 1164, and / or a margin line classification 1175. The user interface 1180 may update the view of the 3D surface so that a first visualization (e.g., an opaque visualization) is used for hard tissue such as teeth, and a second visualization (e.g., a semi-transparent visualization) is used for soft tissue such as gums. The user interface 1180 may also remove moving tissue / tools from the 3D surface based on the moving tissue / tool ​​classification 1164, and may mark margin line segments according to scores based on the margin line classification 1175 in the view of the 3D surface 1185.

[0269] Fig.12 1 is a flow chart of an embodiment of a method 1200 illustrating a machine learning model trained to identify scan roles. At block 1202 of method 1200, processing logic collects a training data set, which may include intraoral scans of dental sites (e.g., height maps), 3D surfaces of dental sites, 2D images of dental sites, and / or projections of 3D surfaces of dental sites. Each data item (e.g., intraoral scan, image, 3D surface, etc.) may include one or more tags. The data items in the training data set may include pixel-level or point-level tags indicating one or more tooth classifications, such as hard tissue classification, soft tissue classification, mobile tissue classification, excess material classification, and / or margin line classification. Multiple other types of tags may also be associated with the data items in the training data set.

[0270] At box 1204, data items from the training data set are input into an untrained machine learning model. At box 1206, the machine learning model is trained based on the training data set to generate a trained machine learning model that classifies scanned characters from intraoral scans, images, and / or 3D surfaces (or projections of 3D surfaces). The machine learning model can also be trained to output one or more other types of predictions, image-level classifications, pixel-level classifications, patch-level classifications (where a patch is a group of pixels), decisions, etc. For example, the machine learning model can also be trained to perform pixel-level classification of intraoral scans, images, 3D surfaces, etc., into dental classifications.

[0271] In one embodiment, at box 1210, an input of a training data item is input into a machine learning model. The input may include data from an intraoral scan (e.g., a height map), a 3D surface, a 2D image, and / or a projection of a 3D surface. At box 1212, the machine learning model processes the input to generate an output. For each point or pixel, the output may include a first probability that the input belongs to a hard tissue classification, a second probability that the input belongs to a soft tissue classification, and a third probability that the input belongs to a mobile tissue or dental tool classification. For each pixel or point, the output may also include a probability that the pixel or point represents a margin line and / or a probability that the pixel or point represents excess material.

[0272] At block 1214, processing logic compares the output probability for each point or pixel to one or more classifications associated with the point or pixel in the input. At block 1216, processing logic determines an error based on the difference between the output and the label associated with the input. At block 1218, processing logic adjusts the weights of one or more nodes in the machine learning model based on the error.

[0273] At block 1220, processing logic determines whether the stopping criteria are met. If the stopping criteria are not met, the method returns to block 1210 and another training data item is input into the machine learning model. If the stopping criteria are met, the method proceeds to block 1225 and training of the machine learning model is complete.

[0274] Fig.13 A diagram of a machine in the example form of a computing device 1300 is shown within which a set of instructions may be executed for causing the machine to perform any one or more of the methodologies described herein. In alternative embodiments, the machine may be connected (e.g., networked) to other machines in a local area network (LAN), an intranet, an extranet, or the Internet. The computing device 1300 may correspond to, for example, Figure 1 The computing device 105 and / or computing device 106 of the present invention can be a computing device 105 and / or a computing device 106. The machine can operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), a tablet computer, a set-top box (STB), a personal digital assistant (PDA), a cellular phone, a global network device, a server, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by the machine. In addition, although only a single machine is shown, the term "machine" should also be understood to include any collection of machines (e.g., computers) that execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein, either individually or in combination.

[0275] The example computing device 1300 includes a processing device 1302, a main memory 1304 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (such as synchronous DRAM (SDRAM)), etc.), a static memory 1306 (e.g., flash memory, static random access memory (SRAM), etc.), and an auxiliary memory (e.g., a data storage device 1328), which communicate with each other via a bus 1308.

[0276] Processing device 1302 represents one or more general-purpose processors, such as a microprocessor, a central processing unit, etc. More specifically, processing device 1302 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 1302 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. Processing device 1302 is configured to execute processing logic (instructions 1326) for performing the operations and steps discussed herein.

[0277] The computing device 1300 may also include a network interface device 1322 for communicating with a network 1364. The computing device 1300 may also include a video display unit 1310 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 1312 (e.g., a keyboard), a cursor control device 1314 (e.g., a mouse), and a signal generating device 1320 (e.g., a speaker).

[0278] The data storage device 1328 may include a machine-readable storage medium (or more specifically, a non-transitory computer-readable storage medium) 1324 on which is stored one or more sets of instructions 1326 embodying any one or more of the methods or functions described herein, such as instructions for the intraoral scanning application 1350. Non-transitory storage media refers to storage media other than carrier waves. The instructions 1326 may also reside completely or at least partially in the main memory 1304 and / or the processing device 1302 during execution by the computer device 1300, and the main memory 1304 and the processing device 1302 also constitute computer-readable storage media.

[0279] The computer-readable storage medium 1324 may also be used to store an intraoral scanning application 1350, which may include one or more machine learning modules and may perform the operations described above. The computer-readable storage medium 1324 may also store a software library containing methods for the intraoral scanning application 1350. Although the computer-readable storage medium 1324 is shown as a single medium in the example embodiment, the term "computer-readable storage medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" should also be taken to include any medium other than a carrier wave that is capable of storing or encoding a set of instructions executed by a machine and causing the machine to perform any one or more methods of the present disclosure. Therefore, the term "computer-readable storage medium" should include, but is not limited to, solid-state memories and optical and magnetic media.

[0280] It should be understood that the above description is intended to be illustrative rather than restrictive. After reading and understanding the above description, many other embodiments will be apparent. Although embodiments of the present disclosure have been described with reference to specific example embodiments, it should be appreciated that the present disclosure is not limited to the described embodiments, but can be practiced by modification and change within the spirit and scope of the appended claims. Therefore, the description and drawings should be regarded as illustrative rather than restrictive. Therefore, the scope of the present disclosure should be determined with reference to the full scope of the appended claims and the equivalents of these claims.

Claims

1. A method comprising: receiving a plurality of intraoral scans of dental sites during an intraoral scanning session; generating a three-dimensional (3D) surface of the dental site based on the plurality of intraoral scans; identifying hard tissue and soft tissue in at least one of a) a plurality of intraoral scans of the dental site or b) the 3D surface of the dental site; as well as A view of the 3D surface is displayed wherein at least one of a first visualization or a first transparency level is used for a first portion of the 3D surface identified as hard tissue and at least one of a second visualization or a second transparency level is used for a second portion of the 3D surface identified as soft tissue.

2. The method according to claim 1, wherein: Identifying the hard tissue and the soft tissue includes: Using a trained machine learning model that has been trained to identify hard tissue and soft tissue to process at least one of a) the multiple intraoral scans or b) data from the 3D surface, wherein the trained machine learning model outputs a first classification indicating hard tissue or a second classification indicating soft tissue for each location in the multiple intraoral scans or the 3D surface.

3. The method according to claim 2, wherein: The trained machine learning model or the second trained machine learning model also outputs, for each location, a third classification identifying the location as part of the edge line, or a fourth classification identifying the location as not part of the edge line.

4. The method according to claim 1, wherein: The first visualization comprises an opaque visualization, and the second visualization comprises a semi-transparent visualization.

5. The method according to claim 4, wherein: The hard tissue comprises teeth and the soft tissue comprises gums and wherein a scanned portion of the teeth below a gum line is visible through the translucent visualization for the gums.

6. The method according to claim 1, wherein: The dental site comprises preparing a tooth, and the method further comprises: identifying a margin line surrounding at least a portion of the prepared tooth in at least one of a) one or more of the plurality of intraoral scans or b) data from the 3D surface; and The edge line is displayed on the 3D surface using one or more additional visualizations.

7. The method according to claim 1, further comprising: receiving an additional intraoral scan of the dental site; adding data from the additional intraoral scan to the 3D surface; updating the view of the 3D surface, wherein the data from the additional intraoral scan is semi-transparent in the updated view of the 3D surface; then segmenting the data from the additional intraoral scan into hard tissue and soft tissue; and The view of the 3D surface is then updated such that the data from the additional intra-oral scan associated with hard tissue is opaque and the data from the additional intra-oral scan associated with soft tissue remains semi-transparent.

8. The method according to claim 1, further comprising: receiving an additional intraoral scan of the dental site; adding data from the additional intraoral scan to the 3D surface; updating the view of the 3D surface, wherein the data from the additional intraoral scan is semi-transparent in the updated view of the 3D surface; subsequently identifying at least one of moving tissue or a dental tool in said data from said additional intraoral scan; removing at least one of the displaced tissue or the dental tool from the 3D surface; and The view of the 3D surface is then updated to reflect at least one of the removed moved tissue or the removed dental tool.

9. The method according to claim 8, further comprising: Locking one or more areas identified as hard tissue in a) the multiple intraoral scans or b) at least one of the 3D surfaces, wherein those areas identified as the moving tissue or the dental tool and those areas locked are not removed from the 3D surface.

10. The method according to claim 1, further comprising: determining whether the dental site includes preparing a tooth; In response to determining that the prepared tooth is not detected, using a first moving tissue detection algorithm to identify and remove moving tissue from at least one of the plurality of intraoral scans or the 3D surface; as well as In response to determining that a prepared tooth is detected, a second moving tissue detection algorithm is used to identify and remove moving tissue from at least one of the multiple intraoral scans or the 3D surface, wherein the second moving tissue detection algorithm is more aggressive in identifying moving tissue than the first moving tissue detection algorithm.

11. The method according to claim 1, wherein: The second transparency level comprises 100% transparency, and wherein the second portion of the 3D surface identified as soft tissue is not visible due to the 100% transparency.

12. The method according to claim 1, wherein: The 3D surface is a 3D surface of a prepared tooth, and the method further comprises: The 3D surface is overlaid onto a second 3D surface of a dental arch including the prepared teeth, wherein the gums from the second 3D surface are displayed using a semi-transparent visualization.

13. The method according to claim 1, further comprising: Receive coordinates input by the user; determining a first tooth that is closest to the coordinates; displaying the first tooth closest to the coordinate using at least one of the first visualization or the first transparency level; as well as The second tooth is displayed using at least one of the second visualization or the second transparency level.

14. The method according to claim 13, further comprising: Receive new coordinates input by new users; determining that the second tooth is the tooth closest to the new coordinates; displaying the second tooth closest to the new coordinates using at least one of the first visualization or the first transparency level; as well as The first tooth is displayed using at least one of the second visualization or the second transparency level.

15. The method according to claim 13, wherein: With the second tooth displayed using at least one of the second visualization or the second transparency level, at least one of a proximal surface or a distal surface of the first tooth is visible.

16. The method according to claim 13, wherein: Receiving the coordinates of the user input includes receiving user input of dragging a prompt feature to the coordinates.

17. The method according to claim 13, wherein: The first tooth is a prepared tooth having a margin line, and wherein determining the first tooth closest to the coordinate comprises: identifying a margin line of the prepared tooth; determining that the point on the 3D surface closest to the coordinate is within the edge line in the plane; and Points on the prepared tooth that are within the margin line are classified as the prepared tooth.

18. The method according to claim 13, wherein: Determining the first tooth closest to the coordinates comprises: performing one or more morphological operations to divide the 3D surface into a plurality of portions corresponding to different teeth; Find the point on the 3D surface closest to the coordinates; and A portion associated with the first tooth is selected, the portion comprising a point on the 3D surface that is closest to the coordinate.

19. A system comprising: an intraoral scanner for generating a plurality of intraoral scans; as well as A computing device for executing the method according to any one of claims 1 to 18.

20. A computer readable medium comprising instructions which, when executed by a processing device, cause the processing device to perform the method according to any one of claims 1 to 18.

Images