Automatic detection, generation, and / or correction of tooth features in digital models

Abstract

Methods and systems for labeling and / or correcting edge lines and / or other features of a tooth site are described. In one example, a 3D model of the tooth site, including a representation of a prepared tooth, is generated from intraoral scan data of the tooth site. An image of the prepared tooth, including a height map, is received or generated. The data from the image is processed using a trained machine learning model that has been trained to identify the edge lines of the prepared tooth, wherein the trained machine learning model outputs a probability map that includes the probability that each pixel in the image depicts an edge line. The 3D model of the tooth site is then updated by labeling the edge lines on the representation of the prepared tooth based on the probability map.

Description

[0001] This application is a divisional application of the patent application filed on September 4, 2020, with application number 202080077107X and invention title "Automatic detection, generation and / or correction of tooth features in digital models". Technical Field

[0002] Embodiments of this disclosure relate to the field of dentistry, and particularly to the use of machine learning and other processing techniques to identify, generate, and / or correct edge lines and / or other tooth features in digital models. Background Technology

[0003] For restorative dental work such as crowns and bridges, an intraoral scanner can be used to generate one or more intraoral scans of the prepared tooth and / or surrounding teeth on the patient's dental arch. In the case of subgingival preparations, the gingiva covers at least a portion of the margin line (also referred to herein as the finishing line) and is constricted to fully expose the margin line. Therefore, intraoral scans are typically created after the dentist has packed a tooth constriction line (also called a packing line or constriction line) under the gum around the prepared tooth, then removed the constriction line, briefly exposing the subgingival margin line. The process of packing the constriction line between the preparation and the gum is lengthy, taking approximately 10 minutes per preparation. Furthermore, this process is painful for the patient and can damage the gums. Intraoral scans performed after packing and removing the constriction line around the prepared tooth must be performed within a narrow time window during which the gum collapses back above the margin line. If insufficient intraoral scans are generated before the gum collapses, the process needs to be repeated. Once sufficient intraoral scans have been generated, these scans can be used to generate a virtual three-dimensional (3D) model of the tooth site, including the prepared tooth and surrounding teeth and gums. For example, a virtual 3D model of a patient's dental arch can be generated. This virtual 3D model can then be sent to the laboratory.

[0004] The laboratory can then perform a process called modeling, in which a virtual 3D model, or a physical 3D model generated from a virtual 3D model, is manually manipulated to create a 3D model that can be used to create crowns, bridges, or other dental prostheses. This may include, for example, manually marking edge lines in the virtual or physical 3D model. It may also include resculpting the virtual or physical 3D model to correct edge lines in areas that are unclear or covered by gums. This work of modifying the virtual and / or physical 3D models by the laboratory often results in an informed guess about the actual geometry of the patient's prepared teeth, including guesses about edge lines, guesses about tooth shape, and so on. The dental prosthesis can then be manufactured using the modified virtual or physical 3D model. If the guess about the true geometry of the patient's prepared teeth is incorrect, the process is repeated, resulting in additional work for the dentist and / or the laboratory. Furthermore, the process of manually modifying the virtual or physical 3D model is a time-consuming task performed by experienced laboratory technicians, which increases the overall cost of the dental prosthesis and adds to the time spent manufacturing it. Summary of the Invention

[0005] In a first aspect of this disclosure, a method includes: generating a three-dimensional model of a dental site from scan data of a dental site, the three-dimensional model including a representation of a prepared tooth. The method further includes: receiving or generating an image of the prepared tooth, the image including a height map. The method further includes: processing the data from the image using a trained machine learning model, the trained machine learning model being trained to identify edge lines of the prepared tooth, wherein the trained machine learning model outputs a probability map, the probability map including the probability that a pixel in the image depicts an edge line. The method further includes: updating the three-dimensional model of the dental site by labeling edge lines on the representation of the prepared tooth based on the probability map.

[0006] The second aspect of this disclosure may further extend the first aspect of this disclosure. In the second aspect of this disclosure, the image of the prepared tooth is generated by projecting at least a portion of a three-dimensional model onto a two-dimensional surface. The third aspect of this disclosure may further extend the first or second aspect of this disclosure. In the third aspect of this disclosure, the scanning data is intraoral scanning data, and the image of the prepared tooth is an intraoral image included in the intraoral scanning data.

[0007] The fourth aspect of this disclosure can further extend the first to third aspects of this disclosure. In the fourth aspect of this disclosure, the method further includes: for each point among a plurality of points on the 3D model mapped to pixels in an image of the prepared tooth, using a probability map to determine the probability that the point depicts an edge line. Additionally, the method includes: calculating the edge line by applying a cost function to the plurality of points on the 3D model, wherein the cost function selects points that together form a profile with a combined minimum cost, wherein for each point, the cost of the point is correlated with the inverse of the probability that the point depicts an edge line. The fifth aspect of this disclosure can further extend the fourth aspect of this disclosure. In the fifth aspect of this disclosure, the method further includes: determining whether the combined minimum cost exceeds a cost threshold; and in response to determining that the combined minimum cost exceeds the cost threshold, determining that the calculated edge line has an unacceptable level of uncertainty.

[0008] The sixth aspect of this disclosure may further extend the fourth aspect of this disclosure. In the sixth aspect of this disclosure, the method further includes: calculating separate costs for different segments of the calculated edge line; determining that a segment of the calculated edge line has a cost exceeding a cost threshold; and determining that a segment of the calculated edge line has an unacceptable level of uncertainty. The seventh aspect of this disclosure may further extend the sixth aspect of this disclosure. In the seventh aspect of this disclosure, the method further includes: highlighting segments of the calculated edge line with unacceptable levels of uncertainty in a three-dimensional model.

[0009] The eighth aspect of this disclosure may further extend the sixth aspect of this disclosure. In the eighth aspect of this disclosure, the method further includes: locking a region of a three-dimensional model, the region including segments of calculated edge lines with an acceptable level of uncertainty; receiving a new intraoral image, the new intraoral image depicting segments of calculated edge lines with an unacceptable level of uncertainty; and updating the three-dimensional model using the new intraoral image to output an updated three-dimensional model, wherein information from the new intraoral image is used to replace a first region, the first region including segments of calculated edge lines with an unacceptable level of uncertainty, and wherein the locked region of the three-dimensional model remains unchanged during the update, the locked region including segments of calculated edge lines with an acceptable level of uncertainty.

[0010] The ninth aspect of this disclosure can further extend the eighth aspect of this disclosure. In the ninth aspect of this disclosure, the scan data includes multiple blended images of the dental site, wherein each of the multiple blended images is based on a combination of multiple images. Additionally, receiving a new intraoral image includes: accessing multiple individual intraoral images used to generate at least some of the multiple blended images; identifying a subset of the multiple individual intraoral images depicting segments of calculated edge lines; and selecting a new intraoral image from the subset of the multiple individual intraoral images, wherein the new intraoral image includes an improved depiction of edge lines compared to an image of the prepared tooth. The tenth aspect of this disclosure can further extend the ninth aspect of this disclosure. In the tenth aspect of this disclosure, the method further includes: generating multiple different versions of an updated three-dimensional model, wherein each of the multiple different versions is based on a different individual intraoral image from a subset of the multiple individual intraoral images; and receiving a user selection of a specific version of the updated three-dimensional model corresponding to the new intraoral image.

[0011] The eleventh aspect of this disclosure may further extend the eighth aspect of this disclosure. In the eleventh aspect of this disclosure, the method further includes: generating a projected image of a first region by projecting at least a portion of the updated three-dimensional model onto an additional two-dimensional surface, the projected image including an additional height map; processing data from the projected image using a trained machine learning model, wherein the trained machine learning model outputs an additional probability map, the additional probability map including the probability that each pixel in the projected image depicts an edge line; and further updating the updated three-dimensional model of the tooth region by marking edge lines in the first region based on the additional probability map.

[0012] The twelfth aspect of this disclosure may further extend the first to eleventh aspects of this disclosure. In the twelfth aspect of this disclosure, the trained machine learning model, for at least one segment of an edge line, further outputs an indication as to whether the segment of the edge line depicted in the image is a high-quality edge line or a low-quality edge line. The thirteenth aspect of this disclosure may further extend the first to twelfth aspects of this disclosure. In the thirteenth aspect of this disclosure, the method further includes: determining that the edge line is indeterminate in at least one segment of the edge line associated with the image; processing data from the image or a new image generated from a three-dimensional model using a second trained machine learning model, the second trained machine learning model having been trained to modify the image of the teeth, wherein the output of the second trained machine learning model includes a modified image comprising a modified height map; and updating the three-dimensional model of the tooth region using the modified height map, wherein the updated three-dimensional model includes updated edge lines with an improved level of accuracy.

[0013] The fourteenth aspect of this disclosure may further extend the thirteenth aspect of this disclosure. In the fourteenth aspect of this disclosure, the modified image includes a plurality of pixels identified as part of an edge line. The fifteenth aspect of this disclosure may further extend the thirteenth aspect of this disclosure. In the fifteenth aspect of this disclosure, the image includes a depiction of an interfering surface that occludes the edge line, wherein at least a portion of the depiction of the interfering surface is removed in the modified image, and wherein the portion of the edge line occluded in the image is shown in the modified image. The sixteenth aspect of this disclosure may further extend the fifteenth aspect of this disclosure. In the sixteenth aspect of this disclosure, the interfering surface includes at least one of blood, saliva, soft tissue, or contractile material. The seventeenth aspect of this disclosure may further extend the first to sixteenth aspects of this disclosure. In the seventeenth aspect of this disclosure, the image is a monochrome image, and wherein the input to the machine learning model that initiates the processing of the image includes data from the image and additional data from a two-dimensional color image without a height map.

[0014] In an eighteenth aspect of this disclosure, a method includes: detecting edge lines in a three-dimensional model of a prepared tooth from one or more images, wherein each of the one or more images includes a height map; determining a quality score for each of a plurality of segments of the edge lines; determining whether any of the plurality of segments of the edge lines has a quality score below a quality threshold; and updating the three-dimensional model of the prepared tooth by replacing portions of the three-dimensional model associated with the segments of the edge lines with image data from new images in response to determining that a segment of the edge line has a quality score below the quality threshold.

[0015] The nineteenth aspect of this disclosure may further extend the eighteenth aspect of this disclosure. In the nineteenth aspect of this disclosure, the method further includes: prompting a user to generate a new intraoral image depicting a segment of an edge line; and receiving a new image, generated by an intraoral scanner, in response to the prompting. The twentieth aspect of this disclosure may further extend either the eighteenth or nineteenth aspect of this disclosure. In the twentieth aspect of this disclosure, the method further includes: locking a portion of the three-dimensional model comprising a segment of an edge line having a quality fraction that meets or exceeds a quality threshold; and erasing the portion of the three-dimensional model associated with the edge line segment before replacing it with image data from a new image. The locked portion of the three-dimensional model remains unchanged during the update, and the locked portion may include a segment of an edge line having a quality fraction that meets or exceeds a quality threshold.

[0016] The twenty-first aspect of this disclosure may further extend aspects eighteen to twentieth of this disclosure. In the twenty-first aspect of this disclosure, edge line detection includes: processing data from each of one or more images using a trained machine learning model, the trained machine learning model being trained to identify edge lines of a prepared tooth, wherein for each image, the trained machine learning model outputs a probability map, the probability map including the probability that a pixel in the image depicts an edge line; and for each of a plurality of points of a three-dimensional model, determining the probability that a point depicts an edge line using the probability map of one or more of the plurality of images. The twenty-second aspect of this disclosure may further extend the twenty-first aspect of this disclosure. In the twenty-second aspect of this disclosure, edge line detection further includes: calculating the edge line by applying a cost function to a plurality of points on the three-dimensional model, wherein the cost function selects points that together form a contour with the lowest combined cost, wherein for each point, the cost of the point is correlated with the inverse of the probability that the point depicts an edge line. The twenty-third aspect of this disclosure may further extend the twenty-second aspect of this disclosure. In the twenty-third aspect of this disclosure, determining a quality score for each segment includes: calculating a separate cost for each segment of the edge line using a cost function.

[0017] The twenty-fourth aspect of this disclosure may further extend aspects eighteen through twenty-three of this disclosure. In the twenty-fourth aspect of this disclosure, the method further includes: generating a three-dimensional model from scan data comprising a plurality of mixed images, wherein each of the plurality of mixed images is based on a combination of a plurality of individual intraoral images generated by an intraoral scanner; accessing the plurality of individual intraoral images used to generate at least some of the plurality of mixed images; identifying a subset of the plurality of individual intraoral images depicting segments of edge lines; and selecting a new image from the subset of the plurality of individual intraoral images, wherein the new image includes an improved depiction of the segments of edge lines compared to the depiction of the segments of edge lines from the scan data. The one hundred and twenty-fifth aspect of this disclosure may further extend the twenty-fourth aspect of this disclosure. In the twenty-fifth aspect of this disclosure, the method further includes: generating a plurality of different updated versions of the three-dimensional model, wherein each of the plurality of different updated versions is based on a different individual intraoral image from a subset of the plurality of individual intraoral images; and receiving a user selection of a specific updated version of the three-dimensional model corresponding to a new image.

[0018] The twenty-sixth aspect of this disclosure may further extend aspects eighteen to twenty-fifth of this disclosure. In the twenty-sixth aspect of this disclosure, the method further includes: projecting a portion of a three-dimensional model onto a two-dimensional surface to generate a projected image, the projected image depicting segments of edge lines having a quality score below a quality threshold, the projected image including an additional height map; and processing data from the projected image using a trained machine learning model, wherein the trained machine learning model has been trained to modify the image of the teeth, wherein the trained machine learning model outputs data for a new image, wherein the new image is a modified version of the projected image including the modified height map.

[0019] The twenty-seventh aspect of this disclosure may further extend the twenty-sixth aspect of this disclosure. In the twenty-seventh aspect of this disclosure, the image includes a depiction of an interfering surface that obscures the edge lines, wherein at least a portion of the depiction of the interfering surface is removed from the modified image, and wherein the obscured portion of the edge lines in the image is shown in the modified image. The twenty-eighth aspect of this disclosure may further extend the twenty-sixth aspect of this disclosure. In the twenty-eighth aspect of this disclosure, the new image includes a fabricated version of a segment of the edge lines.

[0020] The twenty-ninth aspect of this disclosure may further extend the eighty-sixth aspect of this disclosure. In the twenty-ninth aspect of this disclosure, the three-dimensional model is generated from scan data comprising a plurality of mixed images, wherein each of the plurality of mixed images is based on a combination of a plurality of individual intraoral images generated by an intraoral scanner. Additionally, the method further includes: accessing a plurality of individual intraoral images used to generate at least some of the plurality of mixed images; identifying a subset of the plurality of individual intraoral images depicting segments of edge lines; determining a specific image from the subset of the plurality of individual intraoral images, the specific image including a representation of the segments of the edge lines, representing the most similarity to a fabricated version of the segments of the edge lines; and updating the three-dimensional model of the prepared tooth by replacing portions of the three-dimensional model associated with the segments of the edge lines with image data from the specific image.

[0021] In a thirtieth aspect of this disclosure, a method includes: generating a three-dimensional model of a tooth region from scan data of a tooth region, the three-dimensional model including a representation of a tooth, wherein a portion of the three-dimensional model includes an interfering surface that occludes a portion of the tooth; receiving or generating an image of the tooth, the image including a height map, wherein the image depicts the interfering surface; processing the image to generate a modified image, wherein the modified image includes a modified height map, and wherein a portion of the tooth occluded by the interfering surface in the image is shown in the modified image; and updating the three-dimensional model of the tooth region by replacing the portion of the three-dimensional model including the interfering surface with the modified image including the modified height map, the interfering surface occluding a portion of the tooth, wherein the occluded portion of the tooth in the three-dimensional model is shown in the updated three-dimensional model.

[0022] The thirty-first aspect of this disclosure can further extend the thirty-first aspect of this disclosure. In the thirty-first aspect of this disclosure, processing the image includes: inputting data from the image into a trained machine learning model, the trained machine learning model being trained to modify the image of the teeth, wherein the trained machine learning model outputs data of the modified image. The thirty-second aspect of this disclosure can further extend the thirty-first aspect of this disclosure. In the thirty-second aspect of this disclosure, the input to the trained machine learning model includes data from the image and at least one of a first identifier of a dentist who generated the scan data or a second identifier of a laboratory that will manufacture dental prostheses from the updated three-dimensional model. The thirty-third aspect of this disclosure can further extend the thirty-first or thirty-second aspect of this disclosure. In the thirty-third aspect of this disclosure, the image is a monochrome image, and wherein the input to the trained machine learning model includes first data from the image and second data from a two-dimensional color image without a height map.

[0023] The thirty-fourth aspect of this disclosure may further extend aspects thirtieth to thirty-third of this disclosure. In the thirty-fourth aspect of this disclosure, the method further includes: displaying an updated three-dimensional model; receiving an indication that the updated three-dimensional model does not include an accurate depiction of the teeth; receiving one or more new intraoral images generated by an intraoral scanner; and updating the three-dimensional model using the one or more new intraoral images. The thirty-fifth aspect of this disclosure may further extend aspects thirtieth to thirty-fourth of this disclosure. In the thirty-fifth aspect of this disclosure, the teeth are prepared teeth including edge lines, interfering surfaces occlude segments of the edge lines, segments of the edge lines occluded by the interfering surfaces in the images are shown in the modified images, and segments of the edge lines occluded in the three-dimensional model are shown in the updated three-dimensional model.

[0024] In a thirty-sixth aspect of this disclosure, a method includes: receiving scan data including a plurality of images of at least a first dental site and a second dental site, wherein each of the plurality of images includes a timestamp and a height map; determining a first subset of the plurality of images for use of the first dental site, wherein the first subset is determined at least in part based on: a) timestamps of the images in the first subset, and b) geometric data of the images in the first subset; determining a second subset of the plurality of images for use of the second dental site, wherein the second subset is determined at least in part based on: a) timestamps of the images in the second subset, and b) geometric data of the images in the second subset; and generating a three-dimensional model of at least a portion of a dental arch, the three-dimensional model including a representation of the first dental site generated using the first subset and a representation of the second dental site generated using the second subset.

[0025] The thirty-seventh aspect of this disclosure may further extend the thirty-sixth aspect of this disclosure. In the thirty-seventh aspect of this disclosure, the method further includes: for each of a plurality of intraoral images, inputting data from the images and timestamps associated with the images into a machine learning model, the machine learning model being trained to select images for generating representations of three-dimensional models of tooth locations, wherein, for each image, the machine learning model outputs a first score associated with a first tooth location and a second score associated with a second tooth location. Each image in a first subset has a first score exceeding a threshold; and each image in a second subset has a second score exceeding a threshold.

[0026] The thirty-eighth aspect of this disclosure may further extend the thirty-sixth or thirty-seventh aspects of this disclosure. In the thirty-eighth aspect of this disclosure, each of the plurality of images is a blended image, the blended image being based on a combination of a plurality of individual intraoral images generated by an intraoral scanner. Additionally, the method further includes: identifying regions of a first tooth location that are unclear in the three-dimensional model; accessing a plurality of individual intraoral images used to generate at least some of the plurality of blended images; identifying a subset of the plurality of individual intraoral images, the subset depicting the unclear region; selecting a specific image from the subset of the plurality of individual intraoral images, wherein the specific image includes an improved depiction of the region; and updating the three-dimensional model using the specific image.

[0027] The thirty-ninth aspect of this disclosure may further extend the thirty-eighth aspect of this disclosure. In the thirty-ninth aspect of this disclosure, the method further includes: generating multiple different versions of an updated three-dimensional model, wherein each of the multiple different versions is based on a different single intraoral image from a subset of multiple single intraoral images; and receiving a user selection of a particular version of the updated three-dimensional model.

[0028] In a fortieth aspect of this disclosure, a method includes: receiving a first plurality of intraoral scans of a dental arch while an intraoral scanning application is in a first scanning mode; processing the first plurality of intraoral scans using one or more algorithms configured to determine a three-dimensional surface of a static tooth site while the intraoral scanning application is in the first scanning mode; determining by a processing device that a partial shrinkage scan of a first prepared tooth will be performed or has been performed, wherein the partial shrinkage scan includes an intraoral scan of the prepared tooth that has not yet been filled with retraction cord; activating a partial shrinkage intraoral scan mode; receiving a second plurality of intraoral scans; and processing the second plurality of intraoral scans using one or more algorithms configured to determine a three-dimensional surface of a non-static tooth site including collapsed gingiva while the intraoral scanning application is in a partial shrinkage intraoral scan mode.

[0029] The forty-first aspect of this disclosure may be further extended. In the forty-first aspect of this disclosure, determining that a partial contraction scan will be performed or has been performed includes receiving instructions based on user input.

[0030] The forty-second aspect of this disclosure may further extend the fortieth or forty-first aspect of this disclosure. In the forty-second aspect of this disclosure, receiving a second plurality of intraoral scans before determining that a partial contraction scan will be performed or has been performed includes automatically determining that a partial contraction scan has been performed based on analysis of data from one or more of the second plurality of intraoral scans.

[0031] The forty-third aspect of this disclosure may further extend aspects forty through forty-two of this disclosure. In the forty-third aspect of this disclosure, processing a second plurality of intraoral scans using one or more algorithms configured to determine three-dimensional surfaces of non-static tooth sites includes: determining conflicting surfaces from pairs of intraoral scans of the second plurality of intraoral scans, wherein, for the conflicting surfaces, a first intraoral scan of the pair of intraoral scans has a first distance from the probe of an intraoral scanner, and a second intraoral scan of the pair of intraoral scans has a second distance from the probe of an intraoral scanner; determining that the first distance is greater than the second distance; determining whether the difference between the first distance and the second distance is greater than a difference threshold; in response to determining that the difference is greater than the difference threshold, discarding representations of conflicting surfaces from the first intraoral scan; and determining the surface of the non-static tooth site by combining data from the first and second intraoral scans, wherein the discarded representations of conflicting surfaces from the first intraoral scan are not used to determine the surface.

[0032] The forty-fourth aspect of this disclosure may further extend the forty-third aspect of this disclosure. In the forty-fourth aspect of this disclosure, processing a second plurality of intraoral scans using one or more algorithms configured to determine three-dimensional surfaces of non-static tooth locations further includes: determining a first average curvature of the conflict surfaces from a first intraoral scan; determining a second average curvature of the conflict surfaces from a second intraoral scan; and determining that the second average curvature is less than the first average curvature.

[0033] The forty-fifth aspect of this disclosure may further extend aspects forty through forty-four of this disclosure. In the forty-fifth aspect of this disclosure, the method further includes: inputting a height map representing a surface of a non-static tooth location into a machine learning model, the machine learning model having been trained to identify portions of the gingival overlying edge, wherein the machine learning model outputs indications of portions of the gingival overlying edge; and hiding or removing data associated with portions of the gingival overlying edge from the height map.

[0034] The forty-sixth aspect of this disclosure may further extend the forty-fifth aspect of this disclosure. In the forty-sixth aspect of this disclosure, a machine learning model outputs a probability map, which for each pixel in the height map includes a first probability that the pixel belongs to a first dental category and a second probability that the pixel belongs to a second dental category, wherein the first dental category represents a portion of the gingival coverage edge line, and the method further includes: determining one or more pixels in the height map based on the probability map, wherein one or more pixels are classified as a portion of the gingival coverage edge line.

[0035] The forty-seventh aspect of this disclosure may further extend aspects forty through forty-six of this disclosure. In the forty-seventh aspect of this disclosure, processing a first plurality of intraoral scans using one or more algorithms configured to determine a three-dimensional surface of a static tooth location includes: processing the first plurality of intraoral scans using a hybrid algorithm to generate one or more hybrid intraoral scans; and processing the hybrid intraoral scans using a stitching algorithm to stitch the one or more hybrid intraoral scans together; wherein the hybrid algorithm is not used to generate the hybrid intraoral scans in a partially contracted intraoral scan mode.

[0036] The forty-eighth aspect of this disclosure may further extend aspects forty through forty-seven of this disclosure. In the forty-eighth aspect of this disclosure, the method further includes: generating a virtual three-dimensional model of the prepared teeth using a second plurality of intraoral scans; and generating a virtual three-dimensional model of the remainder of the dental arch from a first plurality of intraoral scans.

[0037] In a forty-ninth aspect of this disclosure, a method includes: determining that a partial shrinkage scan of a first prepared tooth will be performed or has been performed, wherein the partial shrinkage scan includes an intraoral scan of the prepared tooth that has not yet been filled with retraction cord; receiving a plurality of intraoral scans generated by an intraoral scanner; and processing the plurality of intraoral scans using a stitching algorithm according to a partial shrinkage intraoral scan pattern to stitch the plurality of intraoral scans together, the processing including: determining conflict surfaces of pairs of intraoral scans from the plurality of intraoral scans, wherein, for the conflict surface, a first intraoral scan of the pair of intraoral scans has a first distance from a probe of the intraoral scanner, and a second intraoral scan of the pair of intraoral scans has a second distance from the probe of the intraoral scanner; determining that the second distance is greater than the first distance; determining whether the difference between the first distance from the probe and the second distance from the probe is greater than a difference threshold; and in response to determining that the difference is greater than the difference threshold, discarding a representation of a conflict surface from the first intraoral scan, wherein a representation of a conflict surface from the second intraoral scan is used for the conflict surface.

[0038] The fiftieth aspect of this disclosure may further extend the forty-ninth aspect of this disclosure. In the fiftieth aspect of this disclosure, the process further includes: determining whether the size of the conflicting surface is less than a size threshold; and in response to determining that the difference is greater than the difference threshold and the size is less than the size threshold, discarding the representation of the conflicting surface from the first intraorbital scan.

[0039] The fifty-first aspect of this disclosure may further extend the forty-ninth or fiftieth aspect of this disclosure. In the fifty-first aspect of this disclosure, the method further includes performing the following operations before determining that a partial shrinkage scan of the first prepared tooth will be performed or has already been performed: receiving additional multiple intraoral scans of the dental arch; processing the additional multiple intraoral scans using a fusion algorithm to generate one or more fusion intraoral scans; and processing the fusion intraoral scans using a first stitching algorithm to stitch the one or more fusion intraoral scans together; wherein the fusion algorithm is not used to generate the fusion intraoral scan in a partial shrinkage intraoral scan mode.

[0040] The fifty-second aspect of this disclosure may further extend the fifty-first aspect of this disclosure. In the fifty-second aspect of this disclosure, the method further includes: generating a virtual model of the prepared tooth using multiple intraoral scans; and generating a virtual model of the dental arch using mixed intraoral scans, wherein the prepared tooth is part of the dental arch.

[0041] The fifty-third aspect of this disclosure can further extend aspects forty-nine to fifty-two of this disclosure. In the fifty-third aspect of this disclosure, the method further includes: determining by the processing device that a fully retracted intraoral scan of the second prepared tooth has been performed or will be performed; receiving a second plurality of intraoral scans; and processing the plurality of intraoral scans using an alternating stitching algorithm according to a fully retracted intraoral scan pattern to stitch the second plurality of intraoral scans together.

[0042] The fifty-fourth aspect of this disclosure may further extend aspects forty-nine through fifty-three of this disclosure. In the fifty-fourth aspect of this disclosure, determining that a partial contraction scan will be performed or has been performed includes receiving instructions based on user input.

[0043] The fifty-fifth aspect of this disclosure may further extend aspects forty-nine through fifty-four of this disclosure. In the fifty-fifth aspect of this disclosure, receiving multiple intraoral scans before determining that a partial contraction scan will be performed or has been performed includes automatically determining that a partial contraction scan has been performed based on analysis of data from one or more of the multiple intraoral scans.

[0044] In a fifty-sixth aspect of this disclosure, a method includes: receiving a first intraoral scan of the prepared tooth after a gingival retraction tool has temporarily retracted a first portion of the gingiva surrounding the prepared tooth to partially expose the margin line, wherein the first portion of the margin line is exposed in the first intraoral scan; receiving a second intraoral scan of the prepared tooth after receiving the first intraoral scan, wherein the first portion of the margin line is obscured by the first portion of the gingiva in the second intraoral scan; comparing the first intraoral scan and the second intraoral scan by a processing device; identifying conflict surfaces at regions of the prepared tooth corresponding to the first portion of the finish line between the first and second intraoral scans; discarding or marking data of regions of the prepared tooth from the second intraoral scan; and stitching the first and second intraoral scans together to generate a virtual model of the prepared tooth, wherein data of regions of the prepared tooth from the first intraoral scan is used to generate the virtual model of the prepared tooth, and wherein data of regions of the prepared tooth from the second intraoral scan is not used to generate the virtual model of the prepared tooth.

[0045] The fifty-seventh aspect of this disclosure may further extend the fifty-sixth aspect of this disclosure. In the fifty-seventh aspect of this disclosure, the method further includes performing the following operations before discarding data from the region of the prepared tooth in the second intraoral scan: for the region of the prepared tooth in the first intraoral scan, determining a distance from the probe of the intraoral scanner that generated the first intraoral scan; for the region of the prepared tooth in the second intraoral scan, determining a distance from the probe of the intraoral scanner that generated the second intraoral scan, wherein the second distance is less than the first distance; and determining that the difference between the first distance and the second distance is greater than a difference threshold.

[0046] The fifty-eighth aspect of this disclosure may further extend the fifty-seventh aspect of this disclosure. In the fifty-eighth aspect of this disclosure, the method further includes: determining the size of the conflict surface; determining whether the size of the conflict surface is less than a size threshold; and, in response to determining that the size of the conflict surface is less than the size threshold and the difference between a first distance and a second distance is greater than a difference threshold, discarding data from the region of the prepared tooth from a second intraoral scan.

[0047] The fifty-ninth aspect of this disclosure may further extend aspects fifty-six through fifty-eight of this disclosure. In the fifty-ninth aspect of this disclosure, the method further includes performing the following operations before receiving a first intraoral scan: receiving an instruction from the processing device to perform a partial shrinkage scan, wherein the partial shrinkage scan includes an intraoral scan of a prepared tooth that has not yet been filled with retraction cord; and activating a partial shrinkage intraoral scan mode.

[0048] The sixtieth aspect of this disclosure may further extend aspects fifty-six through fifty-nine of this disclosure. In the sixtieth aspect of this disclosure, the method further includes: receiving a third intraoral scan of the prepared tooth after a second portion of the gingiva surrounding the prepared tooth has been temporarily retracted by a gingival retraction tool, wherein a second portion of the margin line is exposed in the third intraoral scan, and wherein a first portion of the margin line is obscured by a first portion of the gingiva in the third intraoral scan; receiving a fourth intraoral scan of the prepared tooth after receiving the third intraoral scan, wherein a second portion of the margin line is obscured by a second portion of the gingiva in the second intraoral scan; comparing the third intraoral scan with the fourth intraoral scan; and marking the interval between the third and fourth intraoral scans. The conflict surface at the second region corresponding to the second part of the edge line of the prepared tooth; determining the third distance of the region of the prepared tooth from the probe in the third intraoral scan; determining the fourth distance of the region of the prepared tooth from the probe in the fourth intraoral scan, wherein the fourth distance is less than the third distance; determining that the difference between the third distance and the fourth distance is greater than a difference threshold; discarding the data of the second region of the prepared tooth from the fourth intraoral scan; and stitching the third intraoral scan and the fourth intraoral scan together to generate a virtual model of the prepared tooth, wherein the data of the second region of the prepared tooth from the third intraoral scan is used to generate the virtual model of the prepared tooth.

[0049] In a sixty-one aspect of this disclosure, a method includes: receiving intraoral scan data of a prepared tooth; generating a first surface of the prepared tooth using the intraoral scan data and a first one or more algorithms, wherein the first surface depicts the prepared tooth without gingival surface information; generating a second surface of the prepared tooth using the intraoral scan data and a second one or more algorithms, wherein the second surface depicts the prepared tooth with gingival surface information; selecting at least one of the first surface or the second surface; and displaying at least one of the selected first surface or the second surface.

[0050] The sixty-second aspect of this disclosure may further extend the sixty-first aspect of this disclosure. In the sixty-second aspect of this disclosure, displaying at least one of the selected first surface or second surface includes displaying a superposition of the first surface and the second surface.

[0051] The sixty-third aspect of this disclosure may further extend the sixty-first or sixty-second aspect of this disclosure. In the sixty-third aspect of this disclosure, the method further includes: receiving additional intraoral scan data of a dental arch including prepared teeth; generating a third surface of the dental arch, the third surface not including the prepared teeth; and generating a virtual three-dimensional model of the dental arch using the third surface and at least one of the first or second surface.

[0052] The sixty-fourth aspect of this disclosure may further extend aspects sixty-one through sixty-three of this disclosure. In the sixty-fourth aspect of this disclosure, generating a first surface of a prepared tooth using intraoral scan data and one or more algorithms includes: determining a conflict surface from the intraoral scan data, wherein, for the conflict surface, a first intraoral scan of the intraoral scan data has a first distance from the probe of an intraoral scanner, and a second intraoral scan of the intraoral scan data has a second distance from the probe; determining that the first distance is greater than the second distance; determining whether the difference between the first distance and the second distance is greater than a difference threshold; and, in response to determining that the difference is greater than the difference threshold, discarding a representation of the conflict surface from the second intraoral scan, wherein the representation of the conflict surface from the first intraoral scan is used for the conflict surface in the first surface.

[0053] The sixty-fifth aspect of this disclosure may further extend aspects sixty-one through sixty-four of this disclosure. In the sixty-fifth aspect of this disclosure, generating a second surface of a prepared tooth using intraoral scan data and one or more algorithms comprises: determining a conflict surface from the intraoral scan data, wherein, for the conflict surface, a first intraoral scan of the intraoral scan data has a first distance from the probe of an intraoral scanner, and a second intraoral scan of the intraoral scan data has a second distance from the probe of the intraoral scanner; and averaging the representations of the conflict surface from the first intraoral scan and the representations of the conflict surface from the second intraoral scan.

[0054] In a sixty-sixth aspect of this disclosure, a method includes: receiving a first intraoral scan of the prepared tooth after a gingival retraction tool has temporarily retracted a first portion of the gingiva surrounding the prepared tooth to partially expose the margin line, wherein the first portion of the margin line is exposed in the first intraoral scan, and wherein a second portion of the margin line is obscured by the gingiva in the first intraoral scan; receiving a second intraoral scan of the prepared tooth after the gingival retraction tool has temporarily retracted a second portion of the gingiva surrounding the prepared tooth to partially expose the margin line, wherein the second portion of the margin line is exposed in the second intraoral scan, and wherein the first portion of the margin line is obscured by the gingiva in the second intraoral scan; and generating a virtual model of the prepared tooth by a processing device using the first and second intraoral scans, wherein the first intraoral scan is used to generate a first region of the virtual model, the first region representing the first portion of the margin line, and wherein the second intraoral scan is used to generate a second region of the virtual model, the second region representing the second portion of the margin line.

[0055] The sixty-seventh aspect of this disclosure may further extend the sixty-sixth aspect of this disclosure. In the sixty-seventh aspect of this disclosure, the method further includes performing the following operations before receiving the first intraoral scan: receiving an instruction from the processing device to perform a partial shrinkage scan, wherein the partial shrinkage scan includes an intraoral scan of a prepared tooth that has not yet been filled with retraction cord; and activating a partial shrinkage intraoral scan mode.

[0056] The sixty-eighth aspect of this disclosure may further extend the sixty-sixth or sixty-seventh aspects of this disclosure. In the sixty-eighth aspect of this disclosure, a third portion of the edge line is exposed in a first intraoral scan and a second intraoral scan, and wherein both the first intraoral scan and the second intraoral scan are used to generate a third region of the virtual model, the third region representing the third portion of the edge line.

[0057] In a sixty-ninth aspect of this disclosure, a method includes: receiving first intraoral scan data of a prepared tooth, the first intraoral scan data having been generated after removing gingival retraction lines surrounding the prepared tooth to expose margin lines; generating a first surface of the prepared tooth using the first intraoral scan data and a first one or more algorithms; determining, for a portion of the first surface depicting a part of the prepared tooth, that margin lines are obscured by gingival tissue; generating a second surface of the margin-obscured portion of the prepared tooth using at least one of a) the first intraoral scan data or a second intraoral scan data and b) a second one or more algorithms; and replacing a portion of the first surface with the second surface.

[0058] The seventieth aspect of this disclosure may further extend the sixty-ninth aspect of this disclosure. In the seventieth aspect of this disclosure, the method further includes: receiving second intraoral scan data after the gingival retraction tool has temporarily retracted a portion of the gingiva on the prepared tooth portion to expose the edge line at the prepared tooth portion.

[0059] The seventy-first aspect of this disclosure may further extend the sixty-ninth or seventieth aspect of this disclosure. In the seventy-first aspect of this disclosure, generating a second surface of a portion of the prepared tooth using at least a) first intraoral scan data or second intraoral scan data and b) a second or more algorithms includes: determining a conflict surface at the portion of the prepared tooth from at least one of the first intraoral scan data or second intraoral scan data, wherein, for the conflict surface, the first intraoral scan of at least one of the first intraoral scan data or second intraoral scan data has a first distance from the probe of the intraoral scanner, and the second intraoral scan of at least one of the first intraoral scan data or second intraoral scan data has a second distance from the probe; determining that the first distance is greater than the second distance; determining whether the difference between the first distance and the second distance is greater than a difference threshold; and, in response to determining that the difference is greater than the difference threshold, discarding a representation of the conflict surface from the first intraoral scan, wherein the representation of the conflict surface from the second intraoral scan is used for the conflict surface in the first surface.

[0060] The seventy-second aspect of this disclosure may further extend aspects sixty-nine to seventy-one of this disclosure. In the seventy-second aspect of this disclosure, generating a first surface of a portion of a prepared tooth using first intraoral scan data and one or more algorithms includes: determining a conflict surface at the portion of the prepared tooth from the first intraoral scan data, wherein, for the conflict surface, a first intraoral scan of the first intraoral scan data has a first distance from the probe of an intraoral scanner, and a second intraoral scan of the first intraoral scan data has a second distance from the probe of the intraoral scanner; and averaging the representations of the conflict surface from the first intraoral scan and the representations of the conflict surface from the second intraoral scan.

[0061] In a seventy-third aspect of this disclosure, a computer-readable medium stores instructions that, when executed by a processing apparatus, cause the processing apparatus to perform a method according to any one of the first to seventy-second aspects of this disclosure.

[0062] In a seventy-fourth aspect of this disclosure, a computing device includes a memory and a processing means operatively coupled to the memory, wherein the processing means is configured to execute instructions from the memory that cause the processing means to perform a method according to any one of the first to seventy-second aspects of this disclosure.

[0063] In a seventy-fifth aspect of this disclosure, a system includes an intraoral scanner and a computing device operatively coupled to the intraoral scanner, wherein the intraoral scanner is used to generate scan data, and the computing device is used to perform the method of any one of the first to seventy-second aspects of this disclosure.

[0064] In a seventy-sixth aspect of this disclosure, a system includes an intraoral scanner and an accompanying computer-readable medium including instructions for performing the methods of any one of the first to seventy-second aspects of this disclosure. Attached Figure Description

[0065] Embodiments of this disclosure are shown in the accompanying drawings by way of example and not limitation.

[0066] Figure 1 An embodiment of a system for performing intraoral scanning and / or generating virtual three-dimensional models of intraoral morphology is shown.

[0067] Figure 2A A flowchart of a method for scanning and preparing teeth according to an embodiment is shown.

[0068] Figure 2B A flowchart is shown illustrating a method for scanning prepared teeth using two different scanning modes according to an embodiment.

[0069] Figure 3 A flowchart is shown of a method for processing intraoral scan data to generate a virtual 3D model of a prepared tooth, according to an embodiment.

[0070] Figure 4A A flowchart of a method for resolving conflicting scan data of a dental site according to an embodiment is shown.

[0071] Figure 4B A solution to conflict scan data of a tooth site according to an embodiment is shown.

[0072] Figure 5A A flowchart of a partial shrinkage method for scanning prepared teeth according to an embodiment is shown.

[0073] Figure 5B Another flowchart of a partial shrinkage method for scanning prepared teeth according to an embodiment is shown.

[0074] Figures 5C to 5G A partial shrinkage method for scanning prepared teeth according to an embodiment is shown.

[0075] Figure 6A A flowchart of a method for resolving the obstruction of the edge line of the prepared teeth according to an embodiment is shown.

[0076] Figure 6B A flowchart of a method for generating the surface of a prepared tooth according to an embodiment is shown.

[0077] Figure 7AA flowchart is shown illustrating a process for generating a virtual 3D model of a prepared tooth using intraoral scan data from an intraoral scanner, together with at least one of CBCT scan data, OCT scan data, or ultrasound scan data, according to an embodiment.

[0078] Figure 7B Another flowchart is shown, according to an embodiment, of generating a virtual 3D model of a prepared tooth using intraoral scan data from an intraoral scanner, together with at least one of CBCT scan data, OCT scan data, or ultrasound scan data.

[0079] Figure 7C The illustration shows the merging of intraoral scan data from an intraoral scanner with at least one of CBCT scan data, OCT scan data, or ultrasound scan data according to an embodiment.

[0080] Figure 8 A flowchart is shown of a method for resolving the obstruction margins of prepared teeth using at least one of CBCT scan data, OCT scan data, or ultrasonic scan data according to an embodiment.

[0081] Figure 9 An example workflow is shown for generating an accurate virtual 3D model of a tooth site and manufacturing a dental prosthesis from the virtual 3D model, according to an embodiment of the present disclosure.

[0082] Figure 10 Another example workflow is shown for generating an accurate virtual 3D model of a tooth site and manufacturing a dental prosthesis from that virtual 3D model, according to an embodiment of the present disclosure.

[0083] Figure 11 The workflow of training a machine learning model and applying the trained machine learning model to an image is illustrated according to an embodiment of the present disclosure.

[0084] Figure 12 A flowchart illustrating a method for training a machine learning model to determine edge lines in an image of a prepared tooth, according to an embodiment, is shown.

[0085] Figure 13 A flowchart illustrating a method for training a machine learning model to correct dental images according to an embodiment is shown.

[0086] Figure 14 A flowchart illustrating a method for training a machine learning model using image data according to an embodiment is shown.

[0087] Figure 15 A flowchart is shown of a method for identifying edge lines in a 3D model of a tooth region according to an embodiment.

[0088] Figure 16Another flowchart illustrates a method for identifying edge lines in a 3D model of a tooth region according to an embodiment.

[0089] Figure 17 A flowchart of a method for updating a 3D model of a tooth region according to an embodiment is shown.

[0090] Figure 18 Another flowchart of a method for updating a 3D model of a tooth region according to an embodiment is shown.

[0091] Figure 19 Another flowchart illustrates a method for identifying edge lines in a 3D model of a tooth region according to an embodiment.

[0092] Figure 20 A flowchart is shown of a method for correcting tooth representation in a 3D model of a tooth region according to an embodiment.

[0093] Figure 21 A flowchart is shown of a method for representing the edge lines of a prepared tooth in a 3D model of a corrected tooth region according to an embodiment.

[0094] Figure 22 A flowchart illustrating a method for generating 3D models of multiple tooth locations according to an embodiment is shown.

[0095] Figure 23 An example of the edge lines in a 3D model of a marked prepared tooth is shown according to an embodiment.

[0096] Figure 24A A first example of automatic correction of a 3D model of teeth according to an embodiment is shown.

[0097] Figure 24B A second example of automatic correction of a 3D model of teeth according to an embodiment is shown.

[0098] Figure 25 A block diagram of an example computing device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0099] This document describes methods and systems for accurately determining the shape, location, and orientation of the margin lines of prepared teeth and / or other accurate information for determining tooth locations. Some embodiments are capable of acquiring accurate intraoral scan data of the margin lines of prepared teeth. For example, embodiments cover techniques for generating an exposed portion of the margin line by exposing only a portion of it at a time without using a shrinkage line (which exposes the entire margin line at once). Other embodiments cover supplementing intraoral scan data with other image data (e.g., X-ray data, CBCT scan data, ultrasound data, etc.) to accurately define the margin lines. Other embodiments provide multiple scanning modes, wherein one scanning mode is used to scan prepared teeth whose margin lines have been exposed using a shrinkage line, while another scanning mode is used to scan prepared teeth whose margin lines are exposed only a portion at a time (e.g., using techniques other than shrinkage lines to expose the margin lines). In such embodiments, tools such as dental probes, trowels, airflow, etc., can be used to expose small areas of the margin line while the margin line is being scanned. This process can be repeated for all areas of the margin line until the entire margin line is scanned.

[0100] This document also describes methods and systems for identifying and / or correcting features in images and / or virtual 3D models of teeth. In some embodiments, the methods and systems identify and / or correct edge lines in images and / or virtual 3D models of prepared teeth. In other embodiments, other features of the teeth (which may or may not be prepared teeth) are identified and / or corrected. Examples of other features that can be identified and / or corrected include cracks, debris, gingival lines, worn areas of the teeth, cavities (also known as caries), etc. Furthermore, blood, saliva, poorly captured image areas, reflectivity, etc., can be identified and / or corrected. Additionally, insertion paths can be identified, model orientation can be determined, blurred areas or low image quality areas can be identified, and so on.

[0101] For many restorative procedures (e.g., creating crowns, bridges, veneers, etc.), a patient's existing teeth are ground down into a post. The ground-down tooth is referred to herein as a prepared tooth, or simply a preparation. A prepared tooth has a marginal line (also called a boundary line), which is the boundary between the natural (unground) portion of the prepared tooth and the prepared (ground) portion. Prepared teeth are typically produced in a manner that allows a crown or other prosthesis to be mounted or placed on the prepared tooth. In many cases, the marginal line of the prepared tooth is below the gingival margin (below the gingival line). While the term preparation generally refers to the post of a prepared tooth, including the marginal line and shoulder of the remaining tooth portion, the term preparation as used herein also includes artificial posts, pivots, cores and pillars, or other devices that can be implanted in the oral cavity to receive crowns or other prostheses. The embodiments of prepared teeth described herein also apply to other types of preparations, such as the aforementioned artificial posts, pivots, etc.

[0102] After the prepared tooth is created, the practitioner performs procedures to prepare it for scanning. Preparing the prepared tooth for scanning may include wiping away blood, saliva, etc., from the prepared tooth and / or separating the patient's gums from the prepared tooth to expose the margin line. In some cases, the practitioner inserts material (e.g., a shrinkage material such as a shrinkage suture) around the prepared tooth between the prepared tooth and the patient's gums. The practitioner then removes the suture before generating an intraoral scan set of the prepared tooth. The soft tissue of the gums then returns to its natural position and, in many cases, collapses back above the margin line after a short period of time. Therefore, the practitioner scans the prepared prepared tooth using an intraoral scanner and generates an intraoral image set of the prepared tooth before the soft tissue returns to its natural position. For this procedure, the intraoral scanner may be used in a first scanning mode (referred to as standard preparation or full shrinkage scan mode). In some embodiments, the margin line of the prepared tooth is exposed or largely exposed, and the margin line is scanned using the standard preparation scan mode, without the practitioner taking any steps to expose the margin line.

[0103] In one embodiment, an intraoral scanner is used in a second scanning mode (referred to as partial shrinkage scanning mode). In this second scanning mode, a practitioner (e.g., a dentist or physician) uses tools such as dental probes, dental scalers, triple injectors, or instruments for outputting air or water flow to partially expose the margin around the prepared tooth being scanned. While a portion of the margin is exposed, the intraoral scanner generates a scan of the area of ​​the prepared tooth with the exposed portion of the margin. The practitioner then uses the same tool to expose another portion of the margin, which is also imaged. This process continues until the entire margin is exposed and scanned. Different algorithms, settings, rules, and standards can be used to stitch the images together for full shrinkage scanning mode and partial shrinkage scanning mode. Partial shrinkage scanning can be a more efficient technique for scanning subgingival preparations compared to standard techniques such as using shrinkage lines. Partial shrinkage scanning can be performed more quickly (e.g., approximately 1-2 minutes, or even less than a minute) and with minimal patient discomfort. Furthermore, practitioners can perform partial shrinkage scanning without rushing to prevent the gingiva from collapsing back above the margin.

[0104] In some embodiments, the prepared tooth (including the margin line of the prepared tooth) is scanned using a standard preparation scan mode. It can then be determined that one or more portions of the margin line are unclear and / or obscured. Those portions of the margin line can then be rescanned using a partial shrinkage scan mode for a more efficient rescanning process.

[0105] The intraoral location where the prosthesis will be implanted should typically be accurately measured and carefully studied so that the prosthesis, such as a crown, denture, or bridge, can be properly designed and sized to fit properly. A good fit allows for proper transfer of mechanical stress between the prosthesis and the jaw and can prevent, for example, gingival infection and tooth decay through the interface between the prosthesis and the intraoral location. After the intraoral location has been scanned, a virtual 3D model of the tooth site (also referred to as the 3D model in this text) can be generated, and this 3D model can be used to fabricate the dental prosthesis. However, if the area containing the margin of the prepared tooth is not well defined, the margin may not be accurately determined, and the margin of the restoration may not be properly designed.

[0106] Therefore, the embodiments disclosed herein provide automated systems and methods for analyzing, labeling, and / or updating edge lines in a virtual 3D model of a prepared tooth generated from an intraoral scan. The virtual 3D model (or an image generated from the virtual 3D model or an image used to generate the virtual 3D model) is analyzed to identify edge lines. In some embodiments, a trained machine learning model is used to analyze images from an intraoral scan and / or images generated by projecting the virtual 3D model onto a 2D surface, the machine learning model being trained to determine edge lines on the prepared tooth. Edge lines can then be labeled or drawn on the virtual 3D model. The quality of the edge lines can be evaluated, and if an edge line has a low quality score (e.g., if the edge lines in the virtual 3D model are unclear, inaccurate, ambiguous, and / or uncertain), the dentist (also referred to herein as a physician) can be notified that the edge lines should be rescanned. Additionally or alternatively, quality scores can be calculated for different segments of the edge lines, and any segment of the edge line with a low quality score (e.g., a quality score below a quality threshold) can be highlighted or otherwise labeled in the 3D model.

[0107] Additionally or alternatively, a virtual 3D model (or an image generated from a virtual 3D model or an image used to generate a virtual 3D model) can be analyzed to identify and / or correct inaccurate and / or unclear representations of teeth (e.g., portions of teeth covered by interfering surfaces that obscure edge lines), and correct inaccurate and / or unclear representations of teeth. For example, blood, saliva, soft tissue (e.g., collapsed gingiva), and / or shrinkage material may obscure edge lines in the 3D model. In some embodiments, images from intraoral scans and / or images generated by projecting a virtual 3D model onto a 2D surface (e.g., height maps) are analyzed using a trained machine learning model trained to redraw or reshape the surface of the teeth to correct areas of inaccuracy and / or lack of clarity. Thus, a trained machine learning model can create correct edge lines in areas where clear edge lines are not displayed in an image of a prepared tooth, and can output a modified image with correct edge lines. The trained machine learning model can also reshape the surface of a prepared tooth (or other teeth) in an image and output a modified image with a reshaped surface. The modified image can be used to adjust the virtual 3D model of the teeth.

[0108] The embodiments provide improved 3D models of prepared teeth, generated with minimal or no manual manipulation of the 3D model. Traditionally, 3D models are calibrated in the laboratory to ensure clear and accurate margins. This may involve sending instructions back to the physician to perform additional scans of unclear areas, manually cutting the physical 3D model created from the virtual 3D model, manually manipulating the virtual 3D model (e.g., using computer-aided drafting (CAD) tools), and so on. Each of these manual operations is time-consuming and resource-intensive, increasing the amount of time spent manufacturing the prosthesis and its cost. Therefore, the automated methods and systems described herein for marking margins in the 3D model enable physicians to examine the margins in the 3D model before sending it to the laboratory and to rescan portions of the prepared teeth during the same patient visit, in which the original intraoral scan was performed. Furthermore, the automated methods and systems described herein for adjusting prepared teeth and / or margins can correct margins in the 3D model and / or add margins to the 3D model. Each of these systems and methods, which can be used individually or together, reduces the cost and time of manufacturing dental prostheses.

[0109] Additional embodiments are also described that automatically select which intraoral images from one or more intraoral scans to be used for generating a 3D model of a dental arch depicting multiple dental sites (e.g., multiple teeth). Traditionally, a dentist selects a preparatory tooth in the scanning software and then scans the selected preparatory tooth; selects another preparatory tooth in the scanning software (if multiple preparatory teeth are to be scanned) and then scans another preparatory tooth; selects a specific dental arch for scanning; and then scans the remainder of one or more other teeth on the dental arch with one or more preparatory teeth. This informs the scanning software which intraoral images should be used for each tooth on the dental arch when generating the 3D model. However, the process of individually selecting and then scanning each preparatory tooth and dental arch can be cumbersome for a dentist. In this embodiment, the scanning software can automatically identify which intraoral images are used for each tooth and / or which scanning mode setting (e.g., a partially contracted scanning mode or a standard preparatory scanning mode) is used, without requiring the dentist to manually identify the images to be associated with a specific preparatory tooth and / or the scanning mode setting to be used for a specific preparatory tooth. Therefore, doctors can scan prepared teeth and other teeth in any desired order and using any desired technique for exposing margins, and the scanning software can use the automation techniques described herein to select which scan mode settings and / or which images are used to depict the first prepared tooth, which scan mode settings and / or which other images are used to depict the second prepared tooth, and so on, thereby reducing the doctor's workload.

[0110] Embodiments are also described for automatically identifying teeth represented in a height map, identifying redundant gums in a height map (i.e., gums covering edge lines), identifying gums represented in a height map, identifying redundant material in a height map (e.g., material that is not gum or tooth), identifying low-quality surfaces in a height map (e.g., blurred surfaces), identifying model orientation from a height map, identifying insertion paths from a height map, and / or identifying edge lines in a height map.

[0111] This document describes various embodiments. It should be understood that these various embodiments can be implemented as independent solutions and / or can be combined. Therefore, reference to an embodiment or one embodiment can refer to the same embodiment and / or different embodiments. Furthermore, some embodiments are referenced to discussions of restorative dentistry, particularly to discussions of tooth preparation and margins. However, it should be understood that embodiments referenced to discussions of restorative dentistry (e.g., prosthodontics) can also be applied to orthodontic dentistry (e.g., orthodontics). Furthermore, embodiments referenced to discussions of tooth preparation can generally also be applied to teeth in general, not just tooth preparation. In addition, embodiments referenced to discussions of margins can also be applied to other dental features, such as cracks, debris, gingival lines, caries, etc. For example, embodiments discussed herein that can identify and correct margins can also identify and remove blood and / or saliva on the tooth surface, foreign bodies obscuring the tooth surface, poor data capture caused by reflection, capture areas with low resolution, etc.

[0112] Some embodiments are discussed herein with reference to intraoral scans and intraoral images. However, it should be understood that the embodiments described with reference to intraoral scans are also applicable to laboratory scans or model / impression scans. Laboratory scans or model / impression scans may include one or more images of a tooth site or a model or impression of a tooth site, which may or may not include height maps, and may or may not include color images. For example, in embodiments, a machine learning model may be trained to identify edge lines from images of laboratory scans of model / impression scans.

[0113] Figure 1 An embodiment of a system 100 for performing intraoral scanning and / or generating a virtual three-dimensional model of intraoral morphology is illustrated. In one embodiment, one or more components of system 100 perform the following actions with reference to Figures 2 to 3. Figure 24B Describe one or more operations.

[0114] System 100 includes a dental clinic 108 and a dental laboratory 110. Each of the dental clinic 108 and dental laboratory 110 includes computing devices 105 and 106, which can be interconnected via a network 180. Network 180 can be a local area network (LAN), a public wide area network (WAN) (e.g., the Internet), a private WAN (e.g., an intranet), or a combination thereof.

[0115] Computing device 105 may be coupled to intraoral scanner 150 (also referred to as a scanner) and / or data storage 125. Computing device 106 may also be connected to data storage (not shown). Data storage may be local data storage and / or remote data storage. Computing device 105 and computing device 106 may each include one or more processing devices, memory, secondary storage, one or more input devices (e.g., such as a keyboard, mouse, tablet computer, etc.), one or more output devices (e.g., a display, printer, etc.) and / or other hardware components.

[0116] Intraoral scanner 150 may include a probe (e.g., a handheld probe) for optically capturing three-dimensional structures. Intraoral scanner 150 can be used to perform intraoral scans of a patient's oral cavity. Intraoral scanning application 115 running on computing device 105 can communicate with scanner 150 to perform intraoral scans. The result of an intraoral scan may be intraoral scan data 135A, 135B to 135N, which may include one or more sets of intraoral images. Each intraoral image may be a two-dimensional (2D) or 3D image, including a height map of a portion of a tooth region, and may include x, y, and z information. In one embodiment, intraoral scanner 150 generates a plurality of discrete (i.e., single) intraoral images. The collection of discrete intraoral images may be merged into a smaller set of hybrid intraoral images, wherein each hybrid image is a combination of multiple discrete images. Scanner 150 may transmit intraoral scan data 135A, 135B to 135N to computing device 105. Computing device 105 may store intraoral scan data 135A-135N in data storage 125.

[0117] According to the example, a user (e.g., a practitioner) can subject a patient to an intraoral scan. In doing so, the user can apply scanner 150 to one or more intraoral locations of the patient. The scan can be divided into one or more segments. As an example, these segments may include the patient's lower cheek region, the patient's lower tongue region, the patient's upper cheek region, the patient's upper tongue region, one or more prepared teeth of the patient (e.g., teeth to which dental appliances such as crowns or other dental prostheses will be applied), one or more teeth in contact with the prepared teeth (e.g., teeth that are not affected by dental appliances but are located next to one or more such teeth or interfere with one or more such teeth when the mouth is closed), and / or the patient's occlusion (e.g., a scan performed with the patient's mouth closed, where the scan targets the interface area between the patient's upper and lower teeth). Through this scanner application, scanner 150 can provide intraoral scan data 135A-N to computing device 105. Intraoral scan The intraoral image data 135A-N can be provided in the form of intraoral image datasets, each of which may include 2D and / or 3D intraoral images of a specific tooth and / or region of the intraoral location. In one embodiment, separate image datasets are created for the maxillary arch, mandibular arch, patient occlusion, and each prepared tooth. Alternatively, a single large intraoral image dataset (e.g., for the mandibular arch and / or maxillary arch) is generated. Such images can be provided from a scanner to computing device 105 in the form of one or more points (e.g., one or more pixels and / or groups of pixels). For example, scanner 150 can provide such a 3D image as one or more point clouds. The intraoral images may each include a height map indicating the depth of each pixel.

[0118] The method of scanning a patient's oral cavity can depend on the procedure being performed. For example, if a maxillary or mandibular denture is being fabricated, the entire mandibular or maxillary edentulous arch may be scanned. Conversely, if a bridge is being fabricated, only a portion of the entire arch may be scanned, including the edentulous area, adjacent prepared teeth (e.g., abutment teeth), and the opposing arch and dentition. Furthermore, the method of scanning the oral cavity can depend on the dentist's scanning preferences and / or the patient's condition. For example, some dentists may perform an intraoral scan after exposing the margins of the prepared body using a shrinkage suture (e.g., in standard preparation scan mode). Other dentists may use partial shrinkage scanning techniques, where only a portion of the margins is exposed and scanned at a time (e.g., performing a scan in partial shrinkage scan mode).

[0119] As a non-limiting example, dental procedures can be broadly categorized into prosthodontic (restorative) procedures and orthodontic procedures, and then further subdivided into specific forms of these procedures. Furthermore, dental procedures can include the identification and treatment of gingival diseases, sleep apnea, and intraoral conditions. The term prosthodontic procedure specifically refers to any procedure involving the oral cavity and for the design, fabrication, or placement of dental prostheses or their real or virtual models at the intraoral sites (intraoral sites) within the oral cavity, or for the design and preparation of intraoral sites to receive such prostheses. For example, prostheses can include any restoration such as crowns, veneers, inlays, onlays, implants, and bridges, as well as any other artificial partial or complete dentures. The term orthodontic procedure specifically refers to any procedure involving the oral cavity and for the design, fabrication, or placement of orthodontic elements or their real or virtual models at the intraoral sites within the oral cavity, or for the design and preparation of intraoral sites to receive such orthodontic elements. These elements can be appliances, including but not limited to frameworks and wires, retainers, clear aligners, or functional appliances.

[0120] For many restorative procedures (e.g., making crowns, bridges, veneers, etc.), a prepared tooth is created (e.g., by grinding down a portion of a tooth into a post). The prepared tooth has a margin line that is important for proper fitting of the dental prosthesis. After the prepared tooth is created, the practitioner performs procedures to prepare it for scanning. Preparing the prepared tooth for scanning may include wiping away blood, saliva, etc., from the prepared tooth and / or separating the patient's gums from the prepared tooth to expose the margin line.

[0121] In some cases, practitioners will perform a standard preparation (full retraction) technique to expose the entire margin line at once by inserting a line around the tooth to be prepared between the tooth and the patient's gingiva, and then removing the line before generating an intraoral scan set of the prepared tooth. The soft tissue of the gingiva then returns to its natural position and, in many cases, collapses back above the margin line after a short period of time. Therefore, some of the intraoral scan data 135A-N may include images taken before the gingiva collapses above the margin line, while others may include images taken after the gingiva collapses above the margin line. Thus, some image data is superior to others in depicting the prepared tooth, particularly in depicting the margin line. In some embodiments, the dentist may provide instructions on which standard preparation technique will be used to prepare the tooth (e.g., by pressing a button or making a selection). Alternatively, the intraoral scanning application 115 may analyze the scan data and automatically determine (i.e., without user input) whether a standard preparation technique was performed.

[0122] In some cases, dentists perform partial retraction scanning. With partial retraction scanning, the gingiva is pushed aside with an instrument to expose a small segment of the margin line of the subgingival preparation. This small segment is scanned, and the instrument is moved, allowing this segment of the gingiva to collapse back beyond the margin line and expose another small segment of the margin line. Therefore, preparing a tooth for scanning may involve using an instrument to expose only a portion of the margin line, and then scanning it while it is exposed. Preparing a tooth for scanning may then involve using an instrument to expose another portion of the margin line, and scanning it while it is exposed. This process may continue until the entire margin line has been scanned.

[0123] Examples of tools that can be used to expose a portion of the marginal line at one time include dental probes, dental scrapers, triple injectors, air guns, dental floss, water guns, etc. In some embodiments, specific tools have been developed for exposing one or more portions of the marginal line around one or more teeth (e.g., a first tool for exposing the interproximal portion of the marginal line, a second tool for exposing the lingual portion of the marginal line, etc.). Different tools developed for exposing different portions of the tooth marginal line may have protrusions, lengths, probes, scrapers, etc., with different lengths, widths, angles, etc.

[0124] In some embodiments, a dentist may provide instructions on which partial retraction preparation techniques will be used to prepare the teeth (e.g., by pressing a button or making a selection). Alternatively, an intraoral scanning application 115 may analyze scan data and automatically determine, based on the scan data, whether a partial retraction preparation technique has been performed (i.e., without user input).

[0125] When a scanning session is complete (e.g., all images of intraoral sites or dental areas have been captured), the intraoral scanning application 115 can generate one or more virtual 3D models of the scanned dental areas. To generate the virtual 3D models, the intraoral scanning application 115 can register intraoral images generated from the intraoral scanning session and “stitch” or merge the intraoral images generated from the intraoral scanning session together. In one embodiment, performing image registration involves capturing 3D data of individual points on a surface in multiple images (views from a camera) and registering the images by calculating transformations between them. The 3D data can be in the form of multiple heightmaps that can be projected into the 3D space of the 3D model to form part of the 3D model. By applying appropriate transformations to points in each registered image and projecting each image into 3D space, the images can be integrated into a common reference frame.

[0126] In one embodiment, image registration is performed on adjacent or overlapping intraoral images (e.g., each consecutive frame of an intraoral video). In another embodiment, image registration is performed using a blended image. An image registration algorithm is performed to register two adjacent intraoral images (e.g., two adjacent blended intraoral images) and / or to register an intraoral image with a 3D model. This primarily involves determining a transformation to align one image with another image and / or with a 3D model. Image registration may involve identifying multiple points in each image (e.g., a point cloud) of an image pair (or image and 3D model), performing surface fitting on the points, and using a local search around the points to match the points in the two images (or image and 3D model). For example, an intraoral scanning application 115 may match points in one image with the nearest points interpolated on the surface of another image and iteratively minimize the distance between the matched points. Other image registration techniques may also be used.

[0127] Intraoral scanning applications can repeatedly register all images in an intraoral image sequence to obtain a transformation for each image, registering each image with the previous image and / or a common reference frame (e.g., with a 3D model). Intraoral scanning application 115 integrates all images into a single virtual 3D model by applying appropriately determined transformations to each image. Each transformation may include rotations about one to three axes and translations in one to three planes.

[0128] In many cases, data from one or more intraoral images do not perfectly correspond to data from one or more other intraoral images. Therefore, in an embodiment, the intraoral scanning application 115 can process the intraoral images (e.g., which may be composite intraoral images) to determine which intraoral images (and / or which portions of the intraoral images) are used for parts of the 3D model (e.g., for representing a specific tooth location). The intraoral scanning application 115 can use data such as geometric data represented in the images and / or timestamps associated with the images to select the optimal image for depicting a tooth location or a portion of a tooth location (e.g., for depicting the outline of a prepared tooth).

[0129] In one embodiment, images are fed into a machine learning model trained to select and / or grade images of dental sites. In another embodiment, one or more scores are assigned to each image, where each score may be associated with a specific dental site and indicate the quality of the representation of that dental site in the intraoral image. Once the selected set of images is used to generate a 3D model representing a portion of a specific dental site (e.g., a preparatory tooth), those images and / or portions of those images can be locked. The locked images or portions of the locked images selected for the dental site can be specifically used to create a particular region of the 3D model (e.g., for creating associated teeth in the 3D model).

[0130] Additionally or alternatively, intraoral images can be weighted based on scores assigned to those images. The assigned weights can be associated with different dental sites. In one embodiment, weights can be assigned to each image of a dental site (or multiple dental sites) (e.g., each blended image). During model generation, a weighted average can be used to combine conflicting data from multiple images to depict dental sites. The applied weights can be those assigned based on quality scores for the dental sites. For example, processing logic can determine that data from a specific overlapping region of a first set of intraoral images is of higher quality than data from a specific overlapping region of a second set of intraoral images. Then, when averaging the differences between the image datasets, the weight of the first image dataset can be greater than the weight of the second image dataset. For example, the first image, assigned a higher rating, could be assigned 70% of the weight, and the second image could be assigned 30%. Therefore, when the data is averaged, the merged result will look more like a depiction from the first image dataset and less like a depiction from the second image dataset.

[0131] In one embodiment, the intraoral scanning application includes a partial shrinkage scanning module 118 and a standard preparation (full shrinkage) scanning module 119. The standard preparation scanning module 119 can provide a standard preparation (full shrinkage) scanning mode, in which one or more first algorithms are used to process the intraoral scan data. The partial shrinkage scanning module 118 can provide a partial preparation scanning mode, in which one or more second algorithms are used to process the intraoral scan data. The first and second algorithms can use different rules, settings, thresholds, etc., to select which images and which parts of the images are used to construct portions of the virtual 3D model.

[0132] As described above, the standard preparatory scan module 119 can provide a standard preparatory (fully contracted) scan mode, in which one or more first algorithms are used to process intraoral scan data. The first algorithms of the standard preparatory scan mode can be optimized to depict a virtual 3D model of the prepared tooth as clearly and accurately as possible using the margins from scan data generated using a fully contracted technique for exposing margins. The first algorithms may include, for example, moving tissue detection algorithms, excess material removal algorithms, blending algorithms, stitching and / or registration algorithms, etc. Such algorithms can be configured under the assumption that the area of ​​the tooth being scanned is static (e.g., immobile). Therefore, if there are interfering or rapidly changing features (e.g., features that only show a very short time interval), the first algorithms can operate to minimize or filter out such data under the assumption that the interfering or rapidly changing features are not part of the scanned object. For example, the first algorithm can classify such data as descriptions of the tongue, cheek, doctor's fingers, tools, etc., and may not use such data when generating the 3D model of the prepared tooth. However, if such algorithms are used on scan data generated using a partially contracted scan technique, they may also remove margin data.

[0133] In one embodiment, in a standard pre-scan mode, raw intraoral scans are received from an intraoral scanner 150 and pre-registered with each other using an initial registration algorithm. A blending algorithm is then applied to the raw scans. If the scans are similar (e.g., all scans have timestamps that differ by less than a time difference threshold and have surface differences less than a surface difference threshold and / or position / or orientation differences less than a position / or orientation threshold), the scans are blended together using the blending algorithm. For example, up to 10-20 consecutive scans performed within seconds or microseconds can be blended together. This involves averaging the data from the scans being blended and generating a single blended scan from the averaged data. The generation of the blended scan reduces the total number of scans processed in later scan processing stages.

[0134] The mixed scan is then processed using a detrimental material identification algorithm that identifies mobile tissue larger than a size threshold (e.g., exceeding 100 or 200 micrometers) and removes this detrimental material from the mixed scan. In one embodiment, the detrimental material identification algorithm is a trained machine learning model (e.g., a neural network such as a convolutional neural network) that has been trained to identify such detrimental material. An embodiment of the machine learning model for detrimental material removal is described in U.S. Patent Application No. 16 / 865,162, filed May 1, 2020, which is incorporated herein by reference.

[0135] Another registration algorithm then registers the mixed scans together. A moving tissue algorithm then identifies moving tissue based on the differences between the mixed scans. This algorithm can identify moving tissue larger than a certain threshold (e.g., 100 or 200 micrometers) and can erase such moving tissue from the mixed scans. A merging algorithm then combines all the remaining image data from the mixed scans to generate a virtual 3D model of the prepared volume. The merging algorithm averages the data differences between the scans. This difference can be less than a threshold difference, for example, less than 0.5 mm.

[0136] The partial shrinkage scanning module 118 can provide a partial preparation scanning mode in which one or more second algorithms are used to process intraoral scan data. The second algorithms of the partial shrinkage scanning mode can be optimized to depict a virtual 3D model of the prepared tooth as clearly and accurately as possible using the margins, based on the scan data generated using partial shrinkage techniques for exposing margins. The second algorithms may include, for example, moving tissue detection algorithms, excess material removal algorithms, blending algorithms, registration algorithms, stitching or merging algorithms, etc. In some embodiments, the second algorithms include excess material removal algorithms, registration algorithms, and stitching or merging algorithms. The scan data from the two techniques will differ, and the differences in algorithms may be to account for the differences in scan data generated by different scanning techniques.

[0137] When using partial shrinkage scanning, the size of the exposed area of ​​the edge line can be on the order of tens of micrometers. This exposed area of ​​the edge line can vary between scans and can be removed by the excess material removal algorithm and / or moving tissue removal algorithm of the standard pre-scan mode. Furthermore, for partial shrinkage scanning, interruptions may occur between scans due to the physician moving the shrinkage tool between scans, whereas for standard scans, the scans can be continuous without interruption. This can increase the registration time and difficulty between scans, which is minimized in embodiments of the second algorithm. The second algorithm may omit the moving tissue and / or excess material removal algorithm so as not to identify and remove excess material (e.g., fingers, tongue, cheek, tools, etc.). In some embodiments, the second algorithm includes different versions of the moving tissue algorithm and / or different versions of the excess material removal algorithm compared to those included in the first algorithm. For example, the criteria constituting excess material and / or moving tissue detection algorithms may be changed for the excess material removal algorithm and / or moving tissue detection algorithm, such that portions of the edge line are not classified as excess material or moving tissue.

[0138] Additionally, the first algorithm can blend scan data from multiple scans to generate a mixed scan, while in some embodiments, the second algorithm may not be used to blend the scans. Alternatively, the criteria for which scans (and / or which parts of the scans) can be blended in the second algorithm may be more stringent than the criteria for which scans (and / or which parts of the scans) can be blended in the first algorithm. In one embodiment, the first algorithm averages the blended images at least for regions that meet a certain criterion (e.g., a size criterion for matching regions). In one embodiment, the second algorithm omits the blending of all or some scans.

[0139] In one embodiment, in a partially contracted scanning mode, a raw intraoral scan is received from an intraoral scanner 150. The raw scans may or may not be initially registered to each other using an initial registration algorithm. In some cases, a hybrid scan is generated by blending the raw scans together. In one embodiment, the hybrid scan is generated from multiple raw scans that are generated simultaneously in the same region of the edge line. Therefore, the hybrid scan may not include different scans in which different portions of the edge line are exposed. As mentioned above, if a blending algorithm is used, it may have stricter criteria for which data can be blended compared to a blending algorithm used for a standard pre-scanning mode.

[0140] In some embodiments, the partial shrinkage scanning mode does not use a redundant material removal algorithm. Alternatively, the partial shrinkage scanning mode may use a redundant material removal algorithm with stricter criteria than the criteria used to identify redundant material in the standard pre-scan mode. In one embodiment, the redundant material removal algorithm includes a size threshold greater than 200 micrometers, for example, 300 micrometers, 400 micrometers, or 500 micrometers, and areas exceeding this size threshold can be identified as redundant material and removed. This is to prevent the removal of edge line information from the scan.

[0141] In one embodiment, the excess material identification algorithm is a trained machine learning model (e.g., a neural network such as a convolutional neural network) trained to identify such excess material. The machine learning model can be specifically trained using training data including depictions of edge lines, so it does not identify regions of edge lines that vary between scans as excess material. The machine learning model may have been trained using a training dataset that includes scans of gingiva extending beyond the edge lines and scans of exposed edge lines not covered by gingiva. In one embodiment, such a machine learning model can be trained to remove gingiva and leave exposed edge lines. In one embodiment, a specific excess gingiva removal algorithm (e.g., a trained machine learning model) is used instead of a general excess material removal algorithm. In one embodiment, two excess material removal algorithms are used, one for removing excess gingiva and the other for removing other excess material. In one embodiment, the input to the trained machine learning model trained to remove excess gingiva is a set of scans. The trained machine learning model can determine, for the scan set, which scan data represent excess gingiva and should be removed.

[0142] U.S. Patent Application No. 16 / 865,162 describes an embodiment of a machine learning model for excess material removal and / or excess gingival removal, except that the training dataset used to train the machine learning model differs from the training dataset described in the referenced patent application. Specifically, the training dataset for training the machine learning model includes: scans showing gingival margins extending beyond the edge lines, wherein regions of gingival margins extending beyond the edge lines are labeled as excess material at the pixel level; and scans showing exposed edge lines with constricted gingival margins, the portions of which are not covered by the edge lines, and these portions are identified with pixel-level labels indicating gingival regions not classified as excess material. For example, the machine learning model can be trained to perform image segmentation in such a manner as to classify pixels representing excess gingival margins extending beyond the edge lines.

[0143] In one embodiment, a gingival excess removal algorithm is used to identify and remove excess gingiva that covers the edge line of the preparation body. The gingival excess removal algorithm can be similar to an excess material removal algorithm, but can be specifically trained to identify and remove excess gingiva covering the edge line.

[0144] One type of machine learning model that can be used in algorithms for removing excess material and / or excess gingiva is an artificial neural network, such as a deep neural network. Artificial neural networks typically include feature representation components with classifier or regression layers that map features to a desired output space. For example, a convolutional neural network (CNN) hosts multiple layers of convolutional filters. Pooling is performed, and nonlinear problems are solved at lower layers, typically topped by a multilayer perceptron that maps the top-level features extracted by the convolutional layers to a decision (e.g., classification output). Deep learning is a class of machine learning algorithms that use cascaded, multilayered nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure of layers, where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in image labeling applications, the raw input can be a pixel matrix; the first representation layer can abstract the pixels and encode edges; the second layer can combine and encode the arrangement of edges; the third layer can encode higher-level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer can identify bounding boxes around teeth in an image that contain a face. It's worth noting that the deep learning process can learn on its own which features are best placed at which level. The "depth" in "deep learning" refers to the number of layers through which data is transformed. More precisely, deep learning systems have a considerable credit allocation path (CAP) depth. A CAP is a chain of transformations from input to output. CAP describes the possible causal relationship between the input and output. For feedforward neural networks, the CAP depth can be the network depth or the number of hidden layers plus one. For recurrent neural networks (where signals can propagate through layers more than once), the CAP depth can be infinite.

[0145] In one embodiment, the U-net architecture is used. U-net is a type of deep neural network that combines an encoder and a decoder, with appropriate connections between them to capture both local and global features. The encoder is a series of convolutional layers that increase the number of channels while decreasing the height and width as they process the input to the output, while the decoder increases the height and width and decreases the number of channels. Layers from the encoder with the same image height and width can be connected to the output from the decoder. Any or all convolutional layers from the encoder and decoder can use conventional or depthwise separable convolutions.

[0146] In one embodiment, the machine learning model is a recurrent neural network (RNN). An RNN is a type of neural network that includes memory that enables the network to capture time-dependent information. An RNN learns an input-output mapping that depends on both the current input and past inputs. The RNN processes past and future scans and makes predictions based on this continuous scan information. An RNN can be trained using a training dataset to generate a fixed number of outputs (e.g., to classify time-varying data, such as video data, into a fixed number of classes). One type of RNN that can be used is a long short-term memory (LSTM) neural network.

[0147] A common architecture for this task is LSTM (Long Short-Term Memory). Unfortunately, LSTM is not well-suited for images because it cannot capture spatial information like convolutional networks. To address this, ConvLSTM—a variant of LSTM—can be utilized, which incorporates convolutional operations within LSTM units. ConvLSTM replaces matrix multiplication with convolution operations at each gate in an LSTM unit. By doing so, it captures underlying spatial features through convolutional operations in multidimensional data. The main difference between ConvLSTM and LSTM lies in the number of input dimensions. Since LSTM input data is one-dimensional, it is not suitable for spatial sequence data such as video, satellite, and radar image datasets. ConvLSTM is designed to take 3D data as its input. In one embodiment, a CNN-LSTM machine learning model is used. CNN-LSTM is an integration of CNN (convolutional layers) and LSTM. First, the CNN part of the model processes the data, and the one-dimensional result is fed into the LSTM model.

[0148] In one embodiment, a class of machine learning models called MobileNet is used. MobileNet is an efficient machine learning model based on a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks. MobileNets can be convolutional neural networks (CNNs) that can perform convolutions in both the spatial and channel domains. MobileNets can include a stack of separable convolutional modules, which consist of depthwise convolutions and pointwise convolutions (conv 1x1). Separable convolutions perform convolutions independently in the spatial and channel domains. This decomposition of convolutions can significantly reduce the computational cost from HWNK. 2 M to HWNK 2 (depthwise) plus HWNM(conv 1x1) (totaling HWN(K)) 2 The computational cost of +M), where N represents the number of input channels, K 2Here, M represents the size of the convolution kernel, H represents the number of output channels, and HxW represents the spatial size of the output feature map. This can reduce the bottleneck of computational cost for 1x1 convolutions.

[0149] In one embodiment, a Generative Adversarial Network (GAN) is used. A GAN is a class of artificial intelligence systems that uses two artificial neural networks competing against each other in a zero-sum game framework. A GAN includes a first artificial neural network that generates candidate networks and a second artificial neural network that evaluates the generated candidate networks. The GAN learns to map from a latent space to a specific data distribution of interest (a data distribution that varies from a photograph to an input image that is indistinguishable to the human eye), while the discriminator network distinguishes instances from those generated by the generator from those from the training dataset. The training objective of the generative network is to improve the error rate of the discriminator network (e.g., by generating new synthetic instances that appear to come from the training dataset to deceive the discriminator network). The generative and discriminator networks are trained together; the generative network learns to generate images that are increasingly difficult for the discriminator network to distinguish from real images (from the training dataset), while the discriminator network simultaneously learns to better distinguish synthetic images from images from the training dataset. The two networks of the GAN are trained once they reach equilibrium. A GAN may include a generator network that generates images inside an artificial mouth and a discriminator network that segments those images. In this embodiment, the discriminator network may be MobileNet.

[0150] In one embodiment, the machine learning model is a conditional generative adversarial (cGAN) network, such as pix2pix. These networks learn not only the mapping from the input image to the output image, but also a loss function used to train that mapping. A GAN is a generative model that learns a mapping from a random noise vector z to an output image y, G : z→y. In contrast, a conditional GAN ​​learns a mapping from the observed image x and the random noise vector z to y, G : {x, z}→y. The generator G is trained to produce an output that cannot be distinguished from a “real” image by an adversarially trained discriminator D, which is trained to detect “fake” images from the generator as well as possible. In an embodiment, the generator may include a U-net or encoder-decoder architecture. In an embodiment, the discriminator may include a MobileNet architecture. An example of a cGAN machine learning architecture that can be used is the pix2pix architecture described in “Image-to-image translation with conditional adversarial networks”, arXiv preprint (2017), by Isola, Phillip, et al.

[0151] Training a neural network can be achieved through supervised learning, which involves feeding the network a training dataset consisting of labeled inputs, observing its output, constraining the error (by measuring the difference between the output and the labeled value), and using techniques such as deep gradient descent and backpropagation to adjust the network's weights across all its layers and nodes to minimize the error. In many applications, this process is repeated among many labeled inputs in the training dataset to produce a network that can produce correct outputs when provided with inputs different from those present in the training dataset. This generalization is achieved in high-dimensional settings such as large images when sufficiently large and diverse training datasets are available.

[0152] The training of machine learning models and the use of trained machine learning models (e.g., for excess material removal algorithms and / or excess gingival removal algorithms) can be performed by processing logic executed by the processor of a computing device. For training machine learning models, a training dataset containing hundreds, thousands, tens of thousands, hundreds of thousands, or more images should be used to form the training dataset. In embodiments, up to millions of patient dentition cases undergoing restorative or orthodontic procedures can be used to form the training dataset, wherein each case may include a final virtual 3D model of a dental arch (or other dental site, e.g., a portion of a dental arch) without excess material and / or excess gingival tissue, and an initial virtual 3D model of a dental arch (or other dental site) including excess material and / or including excess gingival tissue. Additionally or alternatively, cases may include virtual 3D models of dental arches (or other dental sites) with labeled dental categories. For example, each case may include data showing initial 3D models of one or more dental sites generated from intraoral scans, data showing final 3D models corrected by laboratory technicians, data showing whether the physician accepted the modified 3D model, and so on. This data can be processed to generate training datasets for training one or more machine learning models. The machine learning models can be trained to automatically classify and / or segment intraoral scans during or after an intraoral scanning session, and the segmentation / classification can be used to automatically remove excess material and / or excess gingiva from the images. Such trained machine learning models can reduce the amount of post-processing that lab technicians spend cleaning virtual 3D models and can improve the accuracy of 3D models of dental arches or other dental sites generated from intraoral scans.

[0153] In one embodiment, a machine learning model is trained to segment intraoral images by classifying regions of those intraoral images into one or more dental categories. A collection of numerous (e.g., thousands to millions) 3D models of dental arches with labeled dental categories can be collected. Alternatively or additionally, numerous pairs of original 3D models and modified 3D models can be collected. Each pair of original 3D models including excess material and / or excess gingiva and corresponding modified 3D models without excess material and / or excess gingiva can be associated with a specific case and / or patient. Processing logic can compare the original 3D model with the corresponding modified 3D model to determine differences between them. Differences can represent excess material and / or excess gingiva removed from the original 3D model by software and / or laboratory technicians. The processing logic can automatically label each point on the original 3D model that is not present in the corresponding modified 3D model as excess material. Additionally, other points on the modified 3D model and / or the original 3D model can include labels (e.g., labeled as teeth or gingiva not covering the edge line). In an embodiment, labels from the modified 3D model can be transferred to the corresponding original 3D model. Therefore, the original 3D model can be modified to include at least a first label representing excess material and / or excess gingiva (i.e., gingiva covering the edge line) and a second label representing non-excess material and / or gingiva not covering the edge line. In the example, each point in the original 3D model can be modified to include a label having a first value for the first label representing excess gingiva, a second value for the second label representing other excess material, a third value for the third label representing teeth, and a fourth value for the fourth label representing non-excess gingiva. For example, one of the four values ​​could be 1, while the other three values ​​could be 0.

[0154] For each 3D model with labeled dental categories, a set of images (e.g., height maps) can be generated. Each image can be generated by projecting the 3D model (or a portion of the 3D model) onto a 2D surface or plane. In some embodiments, different images of the 3D model can be generated by projecting the 3D model onto different 2D surfaces or planes. For example, a first image of the 3D model can be generated by projecting the 3D model onto a 2D surface at a top-down viewpoint, a second image can be generated by projecting the 3D model onto a 2D surface at a first side viewpoint (e.g., buccal view), a third image can be generated by projecting the 3D model onto a 2D surface at a second side viewpoint (e.g., lingual view), and so on. Each image may include a height map that includes depth values ​​associated with each pixel of the image. For each image, a probability map or mask can be generated based on the labeled dental categories in the 3D model and the 2D surface onto which the 3D model is projected. The size of the probability map or mask can be equal to the pixel size of the generated image. Each point or pixel in the probability map or mask can include a probability value indicating the probability that the point represents one or more dental categories. For example, there can be four dental categories, including a first dental category representing excess gingiva, a second dental category representing other excess materials, a third dental category representing teeth, and a fourth dental category representing non-excess gingiva. Alternatively, a single dental category can be used for both excess gingiva and other excess materials. In such an embodiment, there can be three dental categories, including a first dental category representing excess materials (including excess gingiva), a second dental category representing teeth, and a third dental category representing non-excess gingiva. For example, in this example, the value of a point with the first dental category can be (1,0,0) (the probability of the first dental category is 100%, and the probabilities of the second and third dental categories are 0%), the value of a point with the second dental category can be (0,1,0), and the value of a point with the third dental category can be (0,0,1).

[0155] Training datasets can be collected, where each data item in the training dataset can include an image (e.g., an image including a height map) and an associated probability map. Additional data can also be included in the training data items. Segmentation accuracy can be improved by leveraging additional categories, inputs, and multi-view support. Multiple information sources can be merged into the model input and used collectively for prediction. Multiple dental categories can be predicted simultaneously from a single model. Multiple problems can be solved simultaneously: redundant material removal, tooth / gingival segmentation, splice conflict resolution, hole filling, etc. Accuracy is higher than traditional image and signal processing methods.

[0156] Additional data may include color images. For example, for each image (which may be monochrome), a corresponding color image may also exist. Each data item may include a height map and a color image. Two different types of color images may be available. One type of color image is a viewfinder image, and the other type is a scan texture. A scan texture may be a combination or blend of multiple different viewfinder images. Each intraoral scan may be associated with a corresponding viewfinder image generated approximately simultaneously with the generation of the intraoral image. If blended scanning is used, each scan texture may be based on a combination of viewfinder images associated with the original scan used to generate the specific blended scan.

[0157] The default method can be based solely on depth information and still allow differentiation of several dental categories: teeth, gums, and excess material (e.g., moved tissue). However, sometimes depth information is insufficient for good accuracy. For example, a partially scanned tooth might appear as gum, or even as monochromatic excess material. In such cases, color information can be helpful. In one embodiment, color information is used as an additional 3 layers (e.g., RGB), thus providing the network with 4 layers of input. Two types of color information can be used, which can include viewfinder images and scan textures. Viewfinder images offer better quality but require alignment relative to the heightmap. Scan textures are aligned to the heightmap but may have color artifacts.

[0158] Another type of additional data may include images generated under specific lighting conditions (e.g., images generated under ultraviolet or infrared illumination). The additional data may be 2D or 3D images and may or may not include height maps.

[0159] In some embodiments, the data point set is associated with the same tooth location and is labeled sequentially. In some embodiments, a recurrent neural network is used, and the data points are fed into the machine learning model in ascending order during training.

[0160] In some embodiments, each image includes two values ​​for each pixel in the image, wherein the first value represents the height (e.g., a height map is provided), and the second value represents the intensity. Both the height and intensity values ​​can be used to train a machine learning model.

[0161] In the example, a confocal intraoral scanner can determine the height of a point on the plane (captured by pixels of the intraoral image) based on the focus setting that produces the maximum intensity of the point on the plane. The focus setting provides either a height or depth value for the point. Typically, the intensity value (called the rank) is discarded. However, the intensity value (rank) associated with the height or depth value can be retained and included in the input data provided to a machine learning model.

[0162] A machine learning model can be trained using images generated from a 3D model with labeled dental categories. The machine learning model can be trained to classify pixels in an image as belonging to one or more dental categories. The result of this training is a function that can directly predict dental categories from a height map. Specifically, the machine learning model can be trained to generate a probability map where each point in the probability map corresponds to a pixel in the input image and indicates one or more of the following: a pixel representing a first probability of a first dental category, a pixel representing a second probability of a second dental category, a pixel representing a third probability of a third dental category, a pixel representing a fourth probability of a fourth dental category, a pixel representing a fifth probability of a fifth dental category, and so on. In the case of tooth / gingiva / excess material segmentation, three labels with values ​​are generated.

[0163] During the inference phase (i.e., using a trained machine learning model), one or more intraoral scans (and optionally other data) are fed into the trained model, which can be trained as described above. The trained machine learning model outputs a probability map, where each point in the probability map corresponds to a pixel in the image and indicates the probability that the pixel represents one or more dental categories. In the case of tooth / non-redundant gingiva / redundant material segmentation, three labels with values ​​are generated for each pixel. The corresponding predictions have probabilistic properties: for each pixel, there are three numbers whose sum is 1.0, and these can be interpreted as the probability that the pixel corresponds to one of the three categories.

[0164] In the case of three categories, storing this prediction of dental categories in RGB format is convenient. For example, the first value of the first dental category can be stored as a red intensity value, the second value of the second dental category as a green intensity value, and the third value of the third dental category as a blue intensity value. This makes visualization of the probability map very easy. Typically, high precision is not required, and characters can be used to replace floating-point numbers—that is, each channel of a pixel has 256 possible values. Further optimizations can be made to reduce the size and improve performance (e.g., using 16-value quantization instead of 256 values).

[0165] In one embodiment, a probability map is used to update intraoral images / scans to generate modified intraoral images / scans. The probability map can be used to determine pixels representing excess material, including excess gingiva. Data on pixels marked as excess material can then be removed from the intraoral images / scans or hidden within them. This can include actually removing pixels marked as excess material from the intraoral images / scans, applying filters to the intraoral images / scans, or modifying pixels marked as excess material in the intraoral images / scans to values ​​indicating no surface at the pixel (e.g., reducing the pixel's heightmap value to zero or another predetermined value).

[0166] The registration algorithm registers the raw and / or mixed scans (which may have already been processed by excess material removal and / or excess gingival removal algorithms) to each other. In some embodiments, the tissue movement algorithm is not used in the partial shrinkage scan mode to avoid accidental removal of edge line data. Alternatively, a tissue movement algorithm with more stringent standards than the one used in the standard pre-scan mode can be used.

[0167] The merging algorithm (also known as a stitching algorithm) then combines all the remaining image data from the scans (which can be original scans or mixed scans) to generate a virtual 3D model of the prepared body. For any merging of multiple scans to generate a 3D model, some differences will inevitably exist between these scans, resolved in some way. These differences can be determined by identifying conflicting surfaces between overlapping areas of the scans and then determining whether those conflicting surfaces meet one or more criteria. The merging algorithm can average some data differences between the scans and discard some data for other differences. Criteria used to determine whether to average data between scans or discard data from some scans include the size of the conflicting surfaces, the distance difference (e.g., height or depth) between conflicting surfaces in the scan, the average difference or Gaussian curvature difference between conflicting surfaces in the scan, and so on.

[0168] The first algorithm may include a first merging algorithm having a first size threshold and a first similarity threshold, for determining whether to average data from conflicting surfaces together. The second algorithm may include a second merging algorithm having a second size threshold, for determining whether to average data from conflicting surfaces together, wherein the second size threshold may be greater than the first size threshold. Additionally, the second merging algorithm may include a second similarity threshold higher than the first similarity threshold.

[0169] The size of the conflicting regions can be used to determine whether to perform merging, wherein the size of the conflicting regions must exceed a certain size threshold or be averaged. Therefore, in an embodiment used for the second algorithm, averaging can be performed on regions smaller than the threshold size, and averaging can be omitted on regions larger than or equal to the threshold size.

[0170] Additionally, the merging algorithm of the second algorithm can determine which image data to use for specific overlapping regions based on criteria such as distance from the scanner (depth or height) and / or curvature. For example, points from the first scan that are at a greater distance from the probe and have a greater curvature (e.g., a greater average curvature or a greater Gaussian curvature) can be selected, and conflicting points from the overlapping second scan that have a smaller distance and / or a smaller curvature can be omitted or removed. In the example, the scan data can include height maps, where each height map includes different values ​​representing the height (or conversely, the depth) of each pixel in the scan. Scans can be registered together and can be determined to include overlapping pixels. The heights of these overlapping pixels can be compared, and smaller height values ​​(i.e., larger depth values) from one scan can be retained, while larger height values ​​(i.e., smaller depth values) from the other scan can be discarded. Additionally or alternatively, for the conflicting surfaces from each scan, the average curvature or Gaussian curvature can be calculated. Scans with a larger average curvature or a larger Gaussian curvature can be selected to depict the surface region, while scans with smaller average curvature or smaller Gaussian curvature for conflict surfaces may not be used for that surface region.

[0171] In one embodiment, the difference between the height values ​​of two scans is determined, and if the difference in the height values ​​of overlapping pixels exceeds a threshold, the smaller height value is selected. If the difference in the height values ​​of overlapping pixels is less than the threshold, the height values ​​can be averaged (e.g., using a weighted average or an unweighted average). In an embodiment, this calculation can be based on the average depth value of some or all pixels within the conflict surface.

[0172] Therefore, portions of the scanned data can be selected for use based on one or more criteria, such as the size of the overlapping region (e.g., the number of adjacent overlapping pixels discussed), the difference in height values ​​of the overlapping pixels, and / or the difference in average curvature or Gaussian curvature. For regions larger than a size threshold, data from regions of scans with smaller height values ​​and / or larger average curvature values ​​can be selected, while data from regions of other scans with larger height values ​​and / or smaller average curvature values ​​can be discarded or erased. This prevents data representing gingiva from being averaged with data representing edge lines, since edge lines are associated with large curvature values, while gingiva is associated with much smaller curvature values.

[0173] In some embodiments, data from a first scan set is discarded, and data from a second scan set is averaged together. For example, seven scans may have overlapping regions whose size reaches or exceeds a size threshold. Data from four of the scans depicting the region may be discarded, while data from the remaining three scans depicting the region may be averaged. In one embodiment, percentiles are calculated for scans with overlapping regions, and scans with height values ​​within a certain percentile value (e.g., the 80th percentile) can be selected for removal.

[0174] Intraoral scanning application 115 can generate a 3D model from intraoral images and display the 3D model to a user (e.g., a physician) via a user interface. The physician can then visually examine the 3D model. The physician can virtually manipulate the 3D model relative to up to six degrees of freedom (i.e., translate and / or rotate it relative to one or more of three mutually orthogonal axes) via the user interface to view the 3D model from any desired orientation. The physician can review (e.g., visually examine) the generated 3D model of the intraoral location and determine whether the 3D model is acceptable (e.g., whether the edges of the prepared teeth are accurately represented in the 3D model).

[0175] Intraoral scanning application 115 may include logic for automatically identifying (e.g., highlighting) edge lines in images and / or 3D models of prepared teeth. This makes it easier for physicians to check the accuracy of edge lines. Additionally, intraoral scanning application 115 may mark and / or highlight specific segments of edge lines that are unclear, ambiguous, and / or uncertain. Additionally or alternatively, intraoral scanning application 115 may mark and / or highlight specific areas (e.g., surfaces) that are unclear, ambiguous, and / or uncertain. For example, acceptable segments of edge lines may be displayed in a first color (e.g., green), while unacceptable segments may be displayed in a second color (e.g., red). In one embodiment, a first trained machine learning model is used to identify edge lines in prepared teeth.

[0176] If a portion of the margin is determined to be unclear or covered by the gingiva, the intraoral scanning application 115 may suggest that the practitioner rescan those portions of the margin. The practitioner may have already generated the original intraoral scan dataset for generating a 3D model of the prepared tooth using partial shrinkage technology, and the intraoral scanning application 115 may have already generated the 3D model using a partial scan mode provided by the partial shrinkage scanning module 118. Alternatively, the practitioner may have already generated the original intraoral scan dataset for generating a 3D model of the prepared tooth using full shrinkage technology, and the intraoral scanning application 115 may have already generated the 3D model using a standard scan mode provided by the standard preparation scanning module 119. In either case, the practitioner may not want to subject the patient to additional scan data of unclear portions of the margin by filling the area around the prepared tooth with shrinkage lines. Therefore, the practitioner can use partial shrinkage technology to obtain an intraoral scan of the unclear portions of the margin, and the partial shrinkage scanning module 118 can execute a partial shrinkage scan mode to obtain and / or process such an intraoral scan. Therefore, the amount of time spent on rescanning prepared teeth and the patient discomfort associated with such rescanning can be minimized.

[0177] Additionally or alternatively, the intraoral scanning application 115 may include logic for automatically correcting the surface of the tooth in an image and / or 3D model of the tooth and for modifying unacceptable edge lines of the prepared tooth. This may be referred to as “virtual cleaning” or “sculpting” of the edge lines. In one embodiment, a second trained machine learning model is used to modify the image and / or 3D model of the prepared tooth to correct the edge lines of the prepared tooth (e.g., sculpting the edge lines or performing virtual cleaning of the edge lines). Updated edge lines (e.g., virtually cleaned or sculpted edge lines) may be indicated in the modified image and / or modified 3D model. A physician may examine the modified edge lines to determine their accuracy.

[0178] In the example, a portion of the actual edge line of the scanned prepared tooth may not be clearly defined in the 3D model. For example, during the initial 3D data collection step (e.g., via scanning) that causes the first 3D virtual model to be generated, a portion of the physical tooth surface may be covered by foreign matter (e.g., such as saliva, blood, or debris). A portion of the physical tooth surface may also be obscured by another element (e.g., such as a portion of the gum, cheek, tongue, dental instrument, artificial, etc.). Or, for example, during the initial 3D data collection step (e.g., via scanning) that causes the first virtual 3D model to be generated, the area may have been deformed or otherwise defective and may not correctly correspond to the physical tooth surface (e.g., due to some defects in the actual scanning process). Automatic correction can be performed to remove the representation of foreign matter and reveal the underlying tooth surface and / or edge line. If automatic correction of the tooth surface and / or edge line is performed, the obscured area can be created, and the obscured object can be removed in the 3D model.

[0179] A physician can examine a 3D model with marked and / or corrected margin lines. Based on the examination of the marked and / or corrected margin lines, the physician can determine that portions of the 3D model depicting the margin lines are inappropriate or undesirable, and that the remainder of the 3D model is acceptable. Alternatively or additionally, the intraoral scanning application 115 can automatically select portions of the 3D model that depict unclear or otherwise inappropriate areas of the teeth. For example, an unacceptable portion of the 3D model may correspond to a portion of the true margin line of the scanned prepared tooth that is not sufficiently clearly defined in the 3D model. The user can mark or otherwise delineate unacceptable portions of the 3D model via a user interface. Alternatively, the techniques described herein can be used to automatically delineate or mark unacceptable portions.

[0180] The intraoral scan application 115 can then apply eraser logic to remove, erase, or otherwise remove marked portions from the 3D model. All areas except the marked portions can be locked. For example, a dental procedure of interest may be providing a dental prosthesis, and the portion of the 3D model to be removed or eliminated could be a part of the edge line of the tooth preparation that exists on the surface of the actual tooth but is not clearly represented in the 3D model (or in the intraoral scan data 135A-N used to create the 3D model).

[0181] Intraoral scanning application 108 can instruct the user to generate one or more additional intraoral images (and corresponding sets of one or more intraoral images) of the tooth locations corresponding to the deleted or removed portions of the 3D model. The user can then use scanner 150 to generate one or more additional intraoral images (e.g., using partial or full shrinkage scanning techniques) that at least partially overlap with the previously generated intraoral images. One or more additional intraoral images can be registered with the 3D model (and / or with the intraoral image dataset used to create the 3D model) to provide a composite of the 3D model and one or more additional intraoral images. In this composite, the previously deleted / removed portions of the 3D model are at least partially replaced by the corresponding portions of one or more additional intraoral images. However, portions of the 3D model other than the deleted or removed portions may not be applied to the composite or updated 3D model.

[0182] Once a physician (e.g., a dentist) has determined that the 3D model is acceptable, the physician can instruct computing device 105 to send the 3D model to computing device 106 in dental laboratory 110. Computing device 106 may include a tooth modeling application 120, which can analyze the 3D model to determine if it is sufficient for fabricating a dental prosthesis. Tooth modeling application 120 may include logic for identifying edge lines and / or modifying the surfaces and / or edge lines of one or more tooth sites, as discussed with reference to intraoral scanning application 115. If the 3D model is deemed suitable (or can be modified to place it under deemed suitable conditions), a dental prosthesis can be fabricated from the 3D model. If the 3D model cannot be placed under suitable conditions, instructions can be sent back to dental clinic 108 to generate one or more additional intraoral images of one or more regions of the tooth sites.

[0183] Figure 2 to Figure 22 A method relating to intraoral scanning of dental sites and the generation and manipulation of virtual 3D models of dental sites is illustrated. This method can be executed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processing device to perform hardware simulation), or a combination thereof. In one embodiment, at least some operations of the method are performed by executing dental modeling logic (e.g., Figure 25 The dental modeling logic 2550 is executed by a computing device. The dental modeling logic 2550 may be a component of, for example, an intraoral scanning device including a handheld intraoral scanner and a computing device operatively coupled (e.g., via a wired or wireless connection) to the handheld intraoral scanner. Alternatively or additionally, the dental modeling logic may be executed on a computing device at a dental laboratory.

[0184] For simplicity of explanation, these methods are depicted and described as a series of actions. However, the actions according to this disclosure can occur in various orders and / or simultaneously, and together with other actions not presented and described herein. Furthermore, not all actions shown are necessary to implement the methods according to the subject matter of this disclosure. Moreover, those skilled in the art will understand and recognize that these methods can alternatively be represented as a series of interrelated states via state diagrams or events.

[0185] Figure 2A A flowchart of a method 200 for scanning and preparing teeth according to an embodiment is shown. At block 215 of method 200, processing logic activates a first intraoral scanning mode. For example, the first intraoral scanning mode may be an automatically activated default intraoral scanning mode. For example, a standard preparation scanning mode may be the default intraoral scanning mode and may be automatically activated. Alternatively, user input selecting the first intraoral scanning mode may be received.

[0186] At block 216, the processing logic receives first intraoral scan data. The first intraoral scan data may include a series of intraoral scans, which may be sequential and / or raw intraoral scans. In an embodiment, the first intraoral scan data may be received during an intraoral scanning session shortly after the first intraoral scan data is generated by the intraoral scanner, while the intraoral scanner continues to generate additional intraoral scan data.

[0187] At box 218, the processing logic uses a hybrid algorithm to process the first intraoral scan data to generate a smaller number of hybrid intraoral scans. At box 219, the processing logic uses one or more first algorithms to process the hybrid intraoral scans to generate a 3D surface of a static tooth site. A tooth site can be considered static in the sense that no clinically significant changes occur at the tooth site within a short time period (e.g., within a 30-second time period, a 1-minute time period, a 2-minute time period, etc.). A prepared tooth prepared by filling the gingiva around the tooth with filling material and then removing the filling material to expose the subgingival margin is considered a static tooth site due to the rate at which the gingiva collapses back above the gingiva after this procedure. For example, one or more first algorithms may correspond to a reference. Figure 1 The first algorithm of the standard preparatory scan module 119 is described. The first tooth site may be, for example, the patient's upper and / or lower dental arches. The first tooth site (e.g., one of the dental arches) may include a preparatory tooth thereon, wherein the preparatory tooth includes a subgingival margin. Alternatively, the dentist may have scanned the entire tooth site (e.g., the dental arch) except for the preparatory tooth. In either case, the detail of the margin may not be sufficient.

[0188] At box 220, the processing logic receives an instruction to perform a partial shrinkage scan of the prepared tooth. This instruction may or may not identify the specific prepared tooth to be scanned. If the dentist does not identify a specific prepared tooth, the processing logic may later automatically identify the prepared tooth based on its shape and / or the surrounding surfaces (e.g., adjacent teeth) of the prepared tooth. In an alternative embodiment, the instruction that a partial shrinkage scan of the prepared tooth has been performed may be received after the second intraoral scan data has been received. In other embodiments, the processing logic may automatically determine that a partial shrinkage scanning technique has been used to scan the prepared tooth based on analysis of the second intraoral scan data received at box 230 (and may activate the partial shrinkage intraoral scan mode after receiving and analyzing the second intraoral scan data).

[0189] At box 225, the processing logic activates the partial contraction intraoral scanning mode. At box 230, the processing logic receives second intraoral scan data. The second intraoral scan data may be an intraoral scan sequence of the prepared tooth. The intraoral scan sequence may have already been generated while the dentist is applying the partial contraction intraoral scanning technique as discussed above. For example, the dentist uses a tool to contract a small area of ​​the gingiva around the prepared tooth, exposing only a small portion of the edge line of the prepared tooth at a time, and generating one or more scans of that small portion of the edge line and the surrounding surface while exposing that small portion of the edge line. The dentist then moves the tool to expose a new portion of the edge line by contracting a new area of ​​the gingiva, and generates one or more additional scans of the newly exposed portion of the edge line. The dentist continues this process until scans of all portions of the edge line have been generated.

[0190] At box 232, the processing logic uses one or more second algorithms to process the second intraoral scan data to generate a second 3D surface for the non-static tooth region. In this example, the non-static tooth region is a prepared tooth with exposed edge lines and gingiva surrounding the prepared tooth. Since different portions of the edge lines are exposed and different portions of the gingiva are contracted in each intraoral scan set included in the second intraoral scan data, the prepared tooth is considered a non-static tooth region. Therefore, due to the non-static nature of the gingiva and edge lines received at box 230, the first algorithm used at box 219 to process the first intraoral scan data may not generate an accurate depiction of the edge lines. Therefore, at box 232, one or more second algorithms are used, which can be configured to operate on the scan data of the non-static tooth region. For example, one or more second algorithms may correspond to a reference... Figure 1The second algorithm for the partially contracted intraoral scanning module 118 is described. In one embodiment, a hybrid algorithm is not used to generate a hybrid scan for the partially contracted intraoral scanning pattern, and / or a moving tissue detection algorithm is used. Alternatively, a hybrid scan can be generated for the partially contracted intraoral scanning pattern, and / or a moving tissue detection algorithm can be used, which has been configured not to classify the changed gingiva as moving tissue.

[0191] In one embodiment, processing second intraoral scan data (which may include multiple intraoral scans) using one or more second algorithms includes identifying conflicting surfaces from pairs of intraoral scans of the second intraoral scan data. This can be performed as part of a merging algorithm. For a conflicting surface, the first intraoral scan of the pair of intraoral scans may have a first distance from the probe of the intraoral scanner, and the second intraoral scan of the pair of intraoral scans may have a second distance from the probe of the intraoral scanner. Processing logic then determines which distance is greater. For example, the processing logic may determine that the first distance is greater than the second distance. Additionally, the processing logic determines the difference between the two distances and determines whether the difference between the first and second distances is greater than a difference threshold. In response to determining that the difference is greater than the difference threshold, the processing logic discards the representation of the conflicting surface from the intraoral scan with the smaller distance (e.g., from the first intraoral scan in the example above). A second 3D surface of a non-static dental site (e.g., a prepared tooth) can then be determined by combining data from the first and second intraoral scans, wherein the discarded representation of the conflicting surface from the first intraoral scan is not used to determine the surface. If the difference is less than the difference threshold, the data from the conflicting surfaces of the two intraoral scans can be averaged together.

[0192] In one embodiment, processing the second intraoral scan data using one or more algorithms configured to determine the three-dimensional surfaces of non-static tooth sites also includes determining conflicting surfaces from pairs of intraoral scans of the second intraoral scan data. This can be performed as part of a merging algorithm. The processing logic then determines a first mean curvature or Gaussian curvature of the conflicting surfaces from the pair of first intraoral scans. Additionally, the processing logic determines a second mean curvature or Gaussian curvature of the conflicting surfaces from the pair of second intraoral scans. The processing logic then determines which mean curvature is larger (e.g., the processing logic may determine that the second mean curvature is smaller than the first mean curvature). Furthermore, the processing logic may determine the difference between the two mean curvatures or Gaussian curvatures and determine whether the difference between the first and second mean curvatures or Gaussian curvatures is greater than a difference threshold. In response to determining that the difference is greater than the difference threshold, the processing logic discards representations of conflicting surfaces from intraoral scans with a smaller mean curvature or Gaussian curvature (e.g., from the first intraoral scan in the example above). A second 3D surface of a non-static tooth location (e.g., a prepared tooth) can then be determined by combining data from the first and second intraoral scans, wherein representations of conflict surfaces discarded from the first intraoral scan are not used to determine the surface. If the difference is less than a difference threshold, the data from the conflict surfaces of the two intraoral scans can be averaged together.

[0193] In some embodiments, both distance and average curvature or Gaussian curvature are determined for the conflicting surfaces, and these values ​​are used together to determine which scan data to discard. In one embodiment, if the difference between the distances is less than a distance difference threshold and the difference between the average curvature or Gaussian curvature is less than a curvature difference threshold, the conflicting surface data are averaged together. If the distance difference is greater than a distance difference threshold or the average curvature or Gaussian curvature is greater than a curvature difference threshold, associated data from one of the two scans is discarded, as described above.

[0194] In one embodiment, processing the second intraoral scan data using one or more second algorithms includes inputting the intraoral scan of the second intraoral scan data into a machine learning model that has been trained to identify excess gingiva. For example, a height map representing a surface of a non-static tooth location can be input into the machine learning model, which has been trained to identify portions of gingival overlay edges, wherein the machine learning model outputs an indication of the portions of gingival overlay edges. This output can be in the form of a map or mask with the same resolution as the input height map, wherein each pixel of the map or mask includes an indication of whether the pixel represents excess gingiva. Processing logic can then hide or remove data associated with portions of gingival overlay edges (i.e., those pixels identified as excess gingiva) from the height map. In one embodiment, the machine learning model outputs a probability map that, for each pixel in the height map, includes a first probability that the pixel belongs to a first dental category and a second probability that the pixel belongs to a second dental category, wherein the first dental category represents portions of gingival overlay edges. Processing logic can then determine one or more pixels in the height map based on the probability map that are classified as portions of gingival overlay edges (i.e., excess gingiva).

[0195] At box 240, the processing logic generates a virtual 3D model using a first 3D surface of a static tooth region determined from first intraoral scan data and a second 3D surface of a non-static tooth region determined from second intraoral scan data. In cases where a static tooth region includes a non-static tooth region (e.g., the static tooth region is the dental arch and the non-static tooth region is a preparatory tooth on the dental arch), the portion of the 3D surface of that tooth region depicting the non-static tooth region can be erased or omitted and replaced by the 3D surface of the non-static tooth region.

[0196] Figure 2B A flowchart of a method 250 for scanning prepared teeth using two different scanning modes according to an embodiment is shown. At block 255 of method 250, processing logic receives first intraoral scan data. At block 260, the processing logic automatically determines that a first scanning mode will be used to process the first intraoral scan data. For example, the processing logic may determine, based on analysis of the first intraoral scan data, that it was generated using a standard scanning technique and determine that a standard intraoral scanning mode will be used to process the first intraoral scan data. For example, if the overlapping areas between sequentially generated scans are less than a threshold difference, it can be determined that a standard intraoral scanning mode should be used. At block 265, the processing logic processes the first intraoral scan data using one or more first algorithms to generate a 3D surface of a static tooth site (e.g., a dental arch or a portion thereof) according to the first scanning mode.

[0197] At box 270, the processing logic receives the second intraoral scan data. At box 275, the processing logic automatically determines that a second scan mode (e.g., a partial shrinkage scan mode) will be used to process the second intraoral scan data. For example, the processing logic may determine, based on analysis of the second intraoral scan data, that it was generated using a partial shrinkage scan technique and determine that a partial shrinkage intraoral scan mode will be used to process the second intraoral scan data. For example, this determination may be made based on a comparison of intraoral images (e.g., sequentially generated images) from the second intraoral scan data and determining that the difference between them exceeds a threshold. For example, height maps from the second intraoral scan data can be registered with each other, and conflict surfaces between them can be identified. The difference can be compared with one or more thresholds, and if the threshold percentage of the difference exceeds the difference threshold, the processing logic can determine that the second intraoral scan mode will be used. At box 280, the processing logic may automatically activate the second scan mode (e.g., a partial shrinkage scan mode). At box 285, the processing logic uses one or more second algorithms to process the second intraoral scan data to generate a 3D surface of a non-static tooth location (e.g., a prepared tooth) according to a second scan mode (e.g., a partially contracted scan mode). At box 290, the processing logic uses the first and second 3D surfaces to generate a virtual 3D model.

[0198] Figure 3 A flowchart of a method 300 for processing intraoral scan data to generate a virtual 3D model of a prepared tooth, according to an embodiment, is shown. At block 305 of method 300, processing logic receives first intraoral scan data of a dental arch. At block 310, the processing logic processes the first intraoral scan data using one or more first algorithms to generate a 3D surface of a static tooth location (e.g., a dental arch or a portion thereof) according to a first scan mode (e.g., a standard scan mode).

[0199] At box 315, the processing logic receives the second intraoral scan data. At box 320, the processing logic automatically determines that the second intraoral scan data depicts a prepared tooth. This determination can be based on processing the second intraoral scan data using a machine learning model that has been trained to identify prepared teeth. Alternatively, one or more rule-based algorithms can be used to process the second intraoral scan data, and the second intraoral scan data can be used to determine whether it depicts a prepared tooth based on criteria met by one or more rule-based algorithms. For example, a prepared tooth may have a shape that does not naturally appear in the mouth. Rule-based algorithms can be used to analyze the shape of the tooth location represented in the second intraoral scan data to determine whether it meets the criteria for a prepared tooth.

[0200] At box 325, the processing logic processes the second intraoral scan data using one or more first algorithms according to the first scan mode. The generated first 3D surface includes a depiction of the prepared tooth and a depiction of the surrounding gingival surface covering at least a portion of the edge line of the prepared tooth.

[0201] At box 330, the processing logic also uses one or more second algorithms of the second intraoral scan mode to process the second intraoral scan data to generate a second 3D surface of the prepared tooth. The second 3D surface may include a depiction of the prepared tooth and optionally a depiction of the surrounding gingiva, but the second 3D surface may not include a depiction of any excess gingiva covering the edge line of the prepared tooth.

[0202] At box 335, the processing logic generates a virtual 3D model of the tooth site. At box 340, a first 3D surface or a second 3D surface is selected for the prepared tooth. In one embodiment, a first option for using the first 3D surface and a second option for using the second 3D surface are presented to the user for selection. Each option may be accompanied by a version of the 3D model of the tooth site, showing what the 3D model will look like if that option is selected. The user (i.e., the doctor) can then select his or her preferred option. Alternatively, the processing logic may automatically determine which version of the virtual 3D model has the sharpest depiction of the edge lines, and that option can be selected. At box 345, the processing logic displays the virtual 3D model with the selected first 3D surface or the selected second 3D surface.

[0203] Figure 4A A flowchart of a method 400 for resolving conflicting scan data at a tooth location according to an embodiment is shown. Method 400 can be derived from... Figure 1 One or more second algorithms of the partial shrinkage scanning module 118 are executed to select which scan data to retain and which to discard for conflicting surfaces. In one embodiment, the partial shrinkage scanning module 118 executes method 400 as part of a stitching and / or merging algorithm for generating 3D surfaces from multiple intraoral scans. In one embodiment, the processing logic determines that a partial shrinkage scan of a first prepared tooth will be performed or has already been performed, wherein the partial shrinkage scan includes an intraoral scan of the prepared tooth that has not yet been filled with retraction cord. The processing logic receives multiple intraoral scans generated by the intraoral scanner (before or after determining that the partial shrinkage scanning technique has been performed) and processes the multiple intraoral scans using a stitching or merging algorithm to stitch the multiple intraoral scans together according to the partial shrinkage intraoral scan pattern. In one embodiment, the stitching algorithm (also referred to as the merging algorithm) executes method 400.

[0204] At block 405 of method 400, the processing logic determines the collision surfaces of paired intraoral scans from intraoral scan data generated by the intraoral scanner. At block 410, the processing logic determines a first distance (also referred to as a first depth and / or a first height) from the probe of the intraoral scanner for the collision surface of the first intraoral scan of the paired intraoral scans. The first depth may be a combination of depth values ​​(e.g., average depth or media depth) based on the depths of some or all pixels included in the collision surface of the first intraoral scan. At block 415, the processing logic determines a first average curvature (or a first Gaussian curvature) of the collision surface of the first intraoral scan. At block 420, the processing logic determines a second distance (also referred to as a second depth and / or a second height) from the probe of the intraoral scanner for the collision surface of the second intraoral scan of the paired intraoral scans. The second depth may be a combination of depth values ​​(e.g., average depth or media depth) based on the depths of some or all pixels included in the collision surface of the second intraoral scan. At block 425, the processing logic determines a second average curvature (or a second Gaussian curvature) of the collision surface of the second intraoral scan.

[0205] At box 430, the processing logic compares a first distance and / or a first mean curvature (or a first Gaussian curvature) with a second distance and / or a second mean curvature (or a second Gaussian curvature). At box 435, the processing logic determines a) a first difference between the first distance and the second distance, and / or b) a second difference between the first mean curvature (or the second Gaussian curvature) and the second mean curvature (or the second Gaussian curvature). At box 440, the processing logic determines the dimensions of the conflicting surfaces.

[0206] At block 445, the processing logic determines one or more of the following: a) whether a first difference is greater than a first difference threshold, b) whether a second difference is greater than a second difference threshold, and c) whether the size of the conflicting surface is less than an upper size threshold. The processing logic may also determine whether the size is greater than a lower size threshold. If the first difference is less than the first difference threshold, the second difference is less than the second difference threshold, the size is less than the upper size threshold, and / or the size is greater than the lower size threshold, the method proceeds to block 475. In one embodiment, if the first difference is greater than the first difference threshold and the size of the conflicting surface is within a specific size range (e.g., less than the upper size threshold and / or greater than the lower size threshold), the method proceeds to block 475. In another embodiment, if the first difference is greater than the first difference threshold, the second difference is greater than the second difference threshold, and the size of the conflicting surface is within a specific size range, the method proceeds to block 475. Otherwise, the method continues to block 450.

[0207] At box 475, the processing logic uses a combination of first and second intraoral scan data of the conflict surface. For example, this could include averaging the first and second intraoral scan data of the conflict surface. The first and second intraoral scan data can be averaged using either a weighted average or an unweighted average. For example, an intraoral scan with a larger distance measurement result (e.g., a larger height measurement result or a smaller depth measurement result) can be assigned a higher weight compared to an intraoral scan with a smaller distance measurement result (e.g., a smaller height measurement result or a larger depth measurement result).

[0208] At box 450, the processing logic determines which intra-orifice scan has a greater distance and / or a greater mean curvature (or a greater Gaussian curvature) for the conflicting surface. If, for the conflicting surface, the first intra-orifice scan has a greater distance and / or a greater mean curvature than the second intra-orifice scan, the method proceeds to box 455. If, for the conflicting surface, the second intra-orifice scan has a greater distance and / or a greater mean curvature than the first intra-orifice scan, the method proceeds to box 465.

[0209] At box 455, the processing logic discards and / or ignores the second intraoral scan data of the conflicting surfaces. At box 460, the processing logic uses the first intraoral scan data of the conflicting surfaces instead of the second intraoral scan data when generating the 3D surface of the virtual 3D model of the scanned tooth site.

[0210] At box 465, the processing logic discards and / or ignores the first intraoral scan data of the conflicting surfaces. At box 470, the processing logic uses the second intraoral scan data of the conflicting surfaces instead of the first intraoral scan data when generating the 3D surface of the virtual 3D model of the scanned tooth site.

[0211] Figure 4B A solution to conflict scan data of a tooth site, including the prepared tooth and surrounding gingiva, according to an embodiment is illustrated. Figure 4B In this process, an intraoral scanner scans the tooth region 480. Conflict scan data includes data from a first scan performed while the margin line 494 is exposed and data from a second scan performed while the margin line 494 is covered by the gingiva. The first scan shows surfaces 482, 496, which include the exposed margin line 494. The second scan shows surfaces 496, 484, which include the gingiva covering the margin line. The margin line was not detected in the second scan. Conflict surface 481 can be determined based on a comparison between the two scans.

[0212] For the first scan, a first distance 490 from the probe is determined for the collision surface 481. In one embodiment, the first distance may be the average distance of surface 482 from the probe. However, for clarity, the first distance 490 for a specific point on surface 482 is shown. For the second scan, a second distance 488 from the probe is determined for surface 484. The second distance may be the average distance of the second scan of surface 484. However, for clarity, the second distance 488 for a specific point is shown. (See reference...) Figure 4A As described, a first distance and a second distance can be compared, and the difference between these distances can be calculated. The processing logic can determine that the first difference is greater than a difference threshold, and that the first distance 490 is greater than the second distance 488. The second scan data of the conflict surface 481 can then be discarded, so that the generated 3D model depicts the edge line 494.

[0213] A first average curvature can be calculated for the first surface 482 of the conflict surface 481, and a second average curvature can be calculated for the second surface 484 of the conflict surface 481. As shown, the first surface 482 will have a larger average curvature than the second surface 484. The first and second average curvatures can be compared, and the result of the comparison can be used as additional data points to determine which scan should be used to depict the conflict surface, as referenced. Figure 4A As described.

[0214] Figure 5A A flowchart of a partial retraction method 500 for scanning prepared teeth according to an embodiment is shown. At block 505 of method 500, after a gingival retraction tool has temporarily retracted a first portion of the gingiva surrounding the prepared tooth to partially expose the margin line, the processing logic receives a first intraoral scan of the prepared tooth. The first portion of the margin line is exposed in the first intraoral scan. At block 510, the processing logic receives a second intraoral scan of the prepared tooth after receiving the first intraoral scan. In the second intraoral scan, the first portion of the margin line is obscured by a first portion of the gingiva. For example, the gingival retraction tool may have been moved to expose different portions of the margin line, causing the gingiva to collapse back to the first portion above the margin line when the second intraoral scan is generated.

[0215] At box 515, the processing logic compares the first intraoral scan with the second intraoral scan. At box 520, the processing logic identifies conflicting surfaces in the region corresponding to the first portion of the margin line of the prepared tooth between the first and second intraoral scans. At box 522, the processing logic determines that the conflicting surfaces meet scan selection criteria. Scan selection criteria may include, for example, references to… Figures 4A to 4BAny criteria described. For example, scan selection criteria may include a first criterion, namely, that the average distance difference between scans of conflicting surface regions is greater than a distance difference threshold. Scan selection criteria may also include a second criterion, namely, selecting scans with a larger average distance. Scan selection criteria may also include a third criterion, namely, selecting scans with a larger average curvature. Other criteria may also be used.

[0216] At box 525, the processing logic discards or marks data for the region of the prepared tooth associated with the conflicting surface from the second intraoral scan. At box 530, the processing logic stitches the first and second intraoral scans together to generate a virtual model of the prepared tooth. Data from the region of the prepared tooth from the first intraoral scan is used to generate the virtual model of the prepared tooth, while data from the region of the prepared tooth from the second intraoral scan is not used to generate the virtual model of the prepared tooth.

[0217] Figure 5B Another flowchart of a partial shrinkage method 550 for scanning prepared teeth according to an embodiment is shown. In one embodiment, method 550 is performed after receiving an instruction to perform a partial shrinkage scan and activating a partial shrinkage intraoral scanning mode.

[0218] At block 555 of method 550, after the gingival retraction tool has temporarily retracted a first portion of the gingiva surrounding the prepared tooth to partially expose the edge line, the processing logic receives a first intraoral scan of the prepared tooth, wherein the first portion of the edge line is exposed in the first intraoral scan, and wherein a second portion of the edge line is occluded by the gingiva in the first intraoral scan. At block 560, after the gingival retraction tool has temporarily retracted a second portion of the gingiva surrounding the prepared tooth to partially expose the edge line, the processing logic receives a second intraoral scan of the prepared tooth, wherein the second portion of the edge line is exposed in the second intraoral scan, and wherein the first portion of the edge line is occluded by the gingiva in the second intraoral scan. At block 565, the processing logic generates a virtual model of the prepared tooth using the first and second intraoral scans, wherein the first intraoral scan is used to generate a first region of the virtual model representing the first portion of the edge line, and wherein the second intraoral scan is used to generate a second region of the virtual model representing the second portion of the edge line. In one embodiment, a third portion of the edge line is exposed in both a first intraoral scan and a second intraoral scan, and both the first and second intraoral scans are used to generate a third region of the virtual model, which represents the third portion of the edge line.

[0219] Figures 5C to 5G A partial shrinkage method for scanning prepared teeth according to an embodiment is shown. Figure 5CA first view of the prepared tooth is shown as depicted in the first intraoral scan data 570 generated simultaneously with the first region of the edge line being exposed. Figure 5D A second view of the prepared tooth is shown, as depicted in the second intraoral scan data 578 generated simultaneously with the exposure of the second region at the edge line. Figure 5E A third view of the prepared tooth is shown, as depicted in the third intraoral scan data 584 generated simultaneously with the exposure of the third region at the edge line. Figure 5F The fourth view of the prepared tooth is shown as depicted in the fourth intraoral scan data 590 generated simultaneously with the fourth region of the edge line being exposed. Figure 5G A 3D model 595 of the prepared tooth is shown, generated using selected portions of the first intraoral scan data 570, the second intraoral scan data 578, the third intraoral scan data 584, and the fourth intraoral scan data 590. The selected portions are the corresponding portions of each of the first intraoral scan data 570, the second intraoral scan data 578, the third intraoral scan data 584, and the fourth intraoral scan data 590 that show the exposed edge lines.

[0220] Figure 6A A flowchart of a method 600 for resolving occluded edge lines of a prepared tooth according to an embodiment is shown. At block 605 of method 600, processing logic receives a first intraoral scan of the prepared tooth after removing shrinkage lines surrounding the filling to expose the edge lines. At block 610, the processing logic generates a first surface of the prepared tooth using the first intraoral scan data and one or more algorithms (e.g., according to a standard scan pattern). At block 615, the processing logic determines whether the edge lines are occluded by gingival tissue for a portion of the first surface depicting a part of the prepared tooth. At block 620, if the edge lines are occluded by gingival tissue, the method proceeds to block 630. Otherwise, the method continues to block 625 and accepts the surface.

[0221] At box 630, the processing logic uses first intraoral scan data and one or more algorithms (e.g., based on a partial shrinkage scan pattern) to generate a second surface for the portion of the prepared tooth whose margin is obscured by gingival tissue. At box 635, the processing logic determines whether the margin is obscured by gingival tissue for the second surface depicting the portion of the prepared tooth. At box 640, if the margin is obscured by gingival tissue, the method proceeds to box 650. Otherwise, the method continues to box 645, and at box 610, a portion of the first surface generated using the first or more algorithms is replaced with the second surface. In one embodiment, the operations of boxes 630-645 are omitted, and if it is determined at box 620 that the margin is obscured, the method proceeds directly from box 620 to box 650.

[0222] At box 650, the processing logic receives second intraoral scan data in which the margin line is exposed at a portion of the prepared tooth. The second intraoral scan data may be received after the gingival retraction tool has temporarily retracted the gingival portion above the portion of the prepared tooth to expose the margin line at that portion. At box 655, the processing logic uses the second intraoral scan data and one or more algorithms to generate a third surface of a portion of the prepared tooth. At box 660, the processing logic replaces a portion of the first surface with the third surface.

[0223] Figure 6B A flowchart of a method 662 for generating the surface of a prepared tooth according to an embodiment is shown. At block 664 of method 662, processing logic receives intraoral scan data of the prepared tooth. At block 666, the processing logic generates a first surface of the prepared tooth using the intraoral scan data and (e.g., a first algorithm of a standard intraoral scan mode). In one embodiment, generating a second surface of the prepared tooth using the intraoral scan data and the second algorithm includes determining a conflict surface from the intraoral scan data, wherein, for the conflict surface, the first intraoral scan of the intraoral scan data has a first distance from the probe of the intraoral scanner, and the second intraoral scan of the intraoral scan data has a second distance from the probe of the intraoral scanner. Generating the second surface also includes averaging the representation of the conflict surface from the first intraoral scan and the representation of the conflict surface from the second intraoral scan.

[0224] At block 668, the processing logic uses intraoral scan data and (e.g., a partially contracted scan mode) a second algorithm to generate a second surface of the prepared tooth. In one embodiment, generating a first surface of the prepared tooth using intraoral scan data and the first algorithm includes: determining a conflict surface from the intraoral scan data, wherein, for the conflict surface, a first intraoral scan of the intraoral scan data has a first distance from the probe of the intraoral scanner, and a second intraoral scan of the intraoral scan data has a second distance from the probe. The processing logic then determines whether the first distance is greater than the second distance and / or determines whether the difference between the first distance and the second distance is greater than a difference threshold. In response to determining that the difference is greater than the difference threshold and the first distance is greater than the second distance, the processing logic discards the representation of the conflict surface from the second intraoral scan, wherein the representation of the conflict surface from the first intraoral scan is used for the conflict surface in the first surface.

[0225] At box 670, the processing logic (or the user) selects a first surface or a second surface. Alternatively, in some embodiments, the processing logic and / or the user can select portions of the first surface and portions of the second surface. At box 672, the processing logic displays the selected surface (or a selected portion of the surface).

[0226] In some embodiments, a superposition of the first and second surfaces is shown. Users can view the superposition of the two surfaces to make an informed decision about which surface to choose.

[0227] It can also receive additional scan data of the dental arch, including prepared teeth. A third surface of the dental arch can be generated, wherein the prepared teeth are not included. A virtual 3D model of the dental arch can then be generated using the third surface and a selected first or second surface (or a selected portion of the first and second surfaces).

[0228] Figure 7A A flowchart of a method 700 for generating a virtual 3D model of a prepared tooth using intraoral scan data from an intraoral scanner, along with at least one of CBCT scan data, OCT scan data, or ultrasound scan data, according to an embodiment, is shown. At block 710 of method 700, processing logic receives multiple intraoral scans of the prepared tooth, which includes a marginal line located below the gingiva. At least a portion of the marginal line is not shown in the multiple intraoral scans. At block 715, processing logic receives at least one of a cone-beam computed tomography (CBCT) scan, an optical coherence tomography (OCT) scan, or an ultrasound scan of the prepared tooth, wherein the marginal line is shown in at least one of the CBCT scan, OCT scan, or ultrasound scan. At block 720, processing logic processes the CBCT scan, OCT scan, and / or ultrasound scan to identify a) the prepared tooth, b) the gingiva, and c) the marginal line. This processing can be performed using image processing techniques such as segmentation. The portion of the marginal line identified based on the CBCT, OCT, or ultrasound data can be labeled with a confidence score, which can be expressed in micrometers. In some cases, if the margin line or a portion of the margin line has a low confidence score, the doctor can manually correct the margin line, rescan certain areas of the margin line, or rescan the entire margin line.

[0229] In some embodiments, the machine learning model is trained to perform pixel-level classification of input CBCT, OCT, and / or ultrasound scans using techniques described herein with reference to height maps and intraoral scans. Pixel-level classification is performed to classify each pixel in the input CBCT, OCT, or ultrasound scan as one of a prepared tooth, gingiva, or margin.

[0230] When a dentist grinds a tooth to create a prepared tooth, multiple different types of margins can be produced, with varying axial reduction, margin placement, margin adaptation, margin geometry, and / or margin design. For example, margin placement can include supragingival and subgingival margins. The margins of a prepared tooth can have knife-shaped or feathered margins, chisel edge margins, shoulder margins, beveled shoulder margins, beveled edges, inclined shoulder margins, or chamfered edges, each with an associated margin line. Once the margin lines of a specific type of margin are identified at certain portions of the prepared tooth from intraoral scan data (including how the margin relates to or connects to the rest of the prepared tooth), this information can be used with data from additional imaging modalities to infer the location of margin lines for the rest of the prepared tooth. Furthermore, the tooth profile of the prepared tooth can be automatically calculated from additional scan data (e.g., from CBCT scans). This can be used to roughly estimate the shape of a subgingival tooth.

[0231] CBCT scan data typically have a much lower resolution than intraoral scan data. This reduced resolution can make it difficult to extract useful information about the edges. However, where some information about the edges is available from the intraoral scan data, this information can be used to improve the estimation of the shape and location of the edges depicted in a CBCT scan. A combination of high-resolution data of some portions of the edges from the intraoral scan and low-resolution data of all or the remainder of the edges from the CBCT scan can provide a more accurate estimate of the shape of the edges than either piece of information alone.

[0232] In some embodiments, intraoral scan data and CBCT scan data are combined to generate a multivariate embedding, which can be fed into a machine learning model trained to perform segmentation and edge identification using this combined intraoral and CBCT scan data.

[0233] At box 725, the processing logic uses multiple intraoral scans and at least one of CBCT, OCT, or ultrasonic scans to generate a virtual 3D model of the prepared tooth. At least one of the CBCT, OCT, or ultrasonic scans is used to delineate the edge lines (or portions thereof) in the virtual 3D model.

[0234] Figure 7B Another flowchart of a method 730 for generating a virtual 3D model of a prepared tooth using intraoral scan data from an intraoral scanner, along with at least one of CBCT scan data, OCT scan data, or ultrasound scan data, according to an embodiment, is shown. In one embodiment, method 730 is performed at block 725 of method 700.

[0235] At block 732 of method 730, the processing logic merges data from multiple intraoral scans to form a preliminary virtual 3D model of the prepared tooth, wherein the edge lines are covered by the gingiva in the preliminary virtual 3D model. At block 734, the processing logic then merges data from CBCT scans, OCT scans, or ultrasound scans with the preliminary virtual 3D model to generate a 3D virtual model. This may include registering the CBCT scans, OCT scans, or ultrasound scans with the virtual 3D model. The merging algorithm can then choose whether to use data only from intraoral scans, data only from CBCT scans, OCT scans, or ultrasound scans, or an average of data from both intraoral scans and other imaging modalities to depict one or more regions of the virtual 3D model.

[0236] In one embodiment, at block 736, processing logic determines the location of the margin line from a CBCT scan, OCT scan, or ultrasound scan. At block 738, processing logic then removes the gingival surface covering the margin line from a preliminary virtual 3D model. At block 740, additionally, processing logic replaces the removed gingival surface with the surface of the prepared tooth as depicted in the CBCT scan, OCT scan, or ultrasound scan.

[0237] Figure 7C The illustration shows the merging of intraoral scan data 750 from an intraoral scanner with additional scan data 760 generated using different imaging modalities (e.g., CBCT scan data, OCT scan data, or ultrasound scan data) to form a merged 3D surface 770, according to an embodiment. As shown, in the intraoral scan data 750, the edge lines are covered by the gingiva. In the additional scan data 760, the edge lines are also covered by the gingiva, but the imaging modality used to generate the additional scan data 760 depicts the object beneath the gingiva and the gingival surface. Therefore, the additional scan data 760 can be used to determine the shape and location of the edge lines, and this information can be used to remove the covered gingiva, thereby producing the merged 3D surface 770.

[0238] Figure 8A flowchart of a method 800 for resolving occluded edge lines of a prepared tooth using at least one of CBCT scan data, OCT scan data, or ultrasound scan data according to an embodiment is shown. At block 805 of method 800, processing logic receives multiple intraoral scans of the prepared tooth. At block 810, the processing logic generates a virtual 3D model of the prepared tooth using the intraoral scans. At block 815, the processing logic determines that a portion of the edge lines depicted in the virtual 3D model is unclear or occluded. At block 820, the processing logic receives at least one of a cone-beam computed tomography (CBCT) scan, an optical coherence tomography (OCT) scan, or an ultrasound scan of the prepared tooth, wherein the edge lines are shown in at least one of the CBCT scan, OCT scan, or ultrasound scan. At block 800, the processing logic then uses data from at least one of the CBCT scan, OCT scan, or ultrasound scan to correct the portion of the edge lines depicted in the virtual 3D model. In some cases, multiple types of additional scans are received, and combinations of CBCT and OCT scan data, CBCT and ultrasound scan data, OCT and ultrasound scan data, or CBCT, OCT and ultrasound scan data are used to enhance the virtual 3D model of the prepared teeth.

[0239] Figure 9 An example workflow of a method 900 for generating an accurate virtual 3D model of a tooth site and manufacturing a dental prosthesis from that virtual 3D model, according to an embodiment of this disclosure, is shown. The workflow can be performed at a dental clinic 105 or a dental laboratory 110. Those operations performed at the dental clinic 105 can be performed during a single patient visit or over multiple patient visits. The operations listed below at the dental clinic 105 can be performed, for example, by an intraoral scanning application 115. The operations listed below at the dental laboratory 110 can be performed, for example, by a tooth modeling application 120. In embodiments, the intraoral scanning application 115 and / or the tooth modeling application 120 can be combined. Figure 25 The tooth modeling logic is 2550.

[0240] Method 900 may begin at box 915, where processing logic executed on a computing device associated with dental clinic 105 receives intraoral scan data (or other scan data, such as CBCT scan data, OCT scan data, and / or ultrasound scan data). The intraoral scan data may have already been generated by intraoral scanner 150 during the intraoral scanning process. As mentioned above, the intraoral scan data may have been generated according to a standard preparatory scan procedure or according to a partial shrinkage scan procedure. At box 918, the processing logic generates virtual 3D models of one or more tooth sites based on the intraoral scan data, as discussed above. The virtual 3D model may be the entire dental arch or a portion of the dental arch (e.g., including the preparatory teeth and a portion of adjacent teeth).

[0241] At box 920, the processing logic performs automatic edge marking on the 3D model. In one embodiment, automatic edge marking is performed by first generating appropriate data input from the 3D model, such as one or more images or height maps of the 3D model. These inputs include any information useful for edge detection generated during scanning. Inputs may include image data, such as a 2D height map providing depth values ​​at each pixel location, and / or a color image of the actual or estimated colors of a given 2D model projection. 3D inputs may also be used, and 3D inputs include connections between Cartesian locations and vertices (i.e., a mesh). Each image may be a 2D or 3D image generated by projecting a portion of a 3D model representing a particular tooth onto a 2D surface. Different images may be generated by projecting the 3D model onto different 2D surfaces. In one embodiment, one or more generated images may include a height map providing depth values ​​for each pixel of the image. Alternatively or additionally, intraoral images used to generate the 3D model may be used. The generated images and / or received intraoral images may be processed by a machine learning model trained to identify edge lines on the prepared tooth. A machine learning model can output a probability map that indicates the probability that each pixel or surface represents an edge line for each pixel or 3D data of an image input into the model. In the case of an image, the probability map can be projected back onto the 3D model to assign probability values ​​to points on the 3D model. A cost function can then be applied to find the edge lines using the probability values ​​assigned to the points on the 3D model. Other techniques can also be used to compute the edge lines based on the assigned probability values. In one embodiment, Figure 15 The method described in section 1500 involves operations 1515-1525 and / or Figure 16 One or more of the operations in methods 1630-1640 described in the diagram are performed at box 920.

[0242] At box 925, the processing logic calculates one or more edge line quality scores. Each edge line quality score may be based on a cost value for the edge line (or a segment of the edge line) calculated using a cost function. In one embodiment, an edge line quality score is determined for the entire edge line. In another embodiment, multiple additional edge line quality scores are calculated, where each edge line quality score is for a specific segment of the edge line.

[0243] At box 930, the processing logic can mark segments of edge lines with low quality scores on the 3D model. For example, the edge line quality scores of one or more edge line segments can be compared to a quality threshold. Any score indicating a cost exceeding a maximum cost may fail to meet the quality threshold. Segments that fail to meet the quality threshold can be marked with a flag that distinguishes them from the rest of the edge line. For example, low-quality edge line segments can be highlighted on the 3D model. In one embodiment, Figure 16 One or more of the operations in methods 1600, 1645-1665, are performed at box 930.

[0244] In some embodiments, additionally or alternatively, the processing logic may determine the sharpness and / or quality values ​​of surfaces that do not include edge lines or are not associated with edge lines. The processing logic may mark such surfaces (or portions of surfaces) with low quality values ​​on the 3D model. For example, the surface quality score of one or more surface portions may be compared to a quality threshold. Any surface (or surface portion) with a surface quality score below the quality threshold may be marked or highlighted.

[0245] At box 932, optionally, the processing logic may adjust the surface of at least a portion of the 3D model of the tooth region and / or may suggest and / or generate segments of edge lines. In one embodiment, the processing logic inputs an image into a machine learning model trained to generate a modified version of the input image, wherein the modified version has corrected surfaces. In the example, the machine learning model may be trained to adjust the surface of the teeth in a tooth image and create segments of edge lines in regions where the edge lines are not shown. The modified image can be used to adjust the 3D model of the tooth region. For example, the processing logic may automatically select regions in the 3D model that depict indistinct segments of edge lines. Regions outside the selected regions in the 3D model may be locked so that these regions will not be modified. The modified image can then be registered with the 3D model, and a first version of the selected regions of the 3D model may be replaced with data from the modified image to generate a second version of the selected regions without affecting the locked regions of the 3D model. Alternatively, the modified 3D image may be projected onto the 3D model without first locking any parts of the 3D model, and a portion of the original surface of the 3D model may be replaced with a new surface using data from the modified image.

[0246] In some embodiments, the lock used to lock one or more regions (e.g., regions other than selected regions) is a one-way lock. With a one-way lock, if adjusting the regions using data from the modified image would degrade the representation of those regions, the locked regions are unaffected by the data from the modified image. However, if using data from the modified image would improve the representation of those regions, the data from the modified image can be used to update those regions. In one embodiment, the data from the modified image includes one or more scores associated with one or more regions. The scores can indicate the confidence and / or quality of the modified image at those regions. If the scores are above a threshold, the processing logic can determine that the data from the modified image will improve the quality of the representation of those regions. In one embodiment, each region is scored, and the current score of the region is used as a threshold.

[0247] In some embodiments, the operation of box 930 is performed while the operation of box 932 is skipped. In other embodiments, the operation of box 932 is performed while the operation of box 930 is skipped. In other embodiments, the operations of both boxes 930 and 932 are performed.

[0248] At box 935, the dentist can provide feedback indicating whether the 3D model is acceptable or should be updated. If the dentist indicates the 3D model is acceptable, it is sent to dental lab 110 for review, and the process continues to box 945. If the dentist indicates the 3D model is unacceptable, the process continues to box 940.

[0249] At box 940, the physician can use a user interface to indicate one or more areas of the 3D model to be rescanned. For example, the user interface may include an eraser function that allows the physician to draw or circle a portion of the 3D model. The drawn area or area inside the circle can be erased, and the rest of the 3D model can be locked. The locked area of ​​the 3D model may not be modified by the new intraoral scan data. Alternatively, a one-way lock can be applied, and the locked area can be modified under specific conditions. Alternatively, the processing logic can automatically select areas depicting edge segments with low quality scores for erasure, and can automatically lock the rest of the 3D model. The processing logic can then graphically instruct the physician where to position the intraoral scanner 150 to generate replacement image data. The method can then return to box 915, where new intraoral image data depicting the erased areas can be received. The new intraoral image data can be generated using a standard scanning procedure or a partial shrinkage scanning procedure.

[0250] At box 918, the 3D model can be updated based on new image data. In one embodiment, the unlocked portions of the 3D model are updated based on the new image data, but locked regions are not updated. In one embodiment, a one-way lock is used to lock one or more regions. With a one-way lock, if adjusting the regions with new image data would degrade the representation of those regions, the locked regions are unaffected by the new image data. However, if using the new image data would improve the representation of those regions, the new image data can be used to update those regions. In one embodiment, processing logic processes the new image data (e.g., using a trained machine learning model) to determine a quality score for the new image data. In some embodiments, multiple quality scores are determined for the new image data, where each quality score can be associated with a different region of the tooth site. Furthermore, quality scores can be determined for one or more regions of the 3D model. If the score of a region from the new image data is higher than the score of the same region from the 3D model, the image data can be used to update that region of the 3D model. If the score of a region from the new image data is less than or equal to the score of the same region from the 3D model, the new image data may not be used to update that region of the 3D model.

[0251] Then the operations of bounding box 920-935 can be repeated based on the updated 3D model.

[0252] At box 945, a lab technician can review the edge lines in the 3D model (e.g., using a dental modeling application 120). Alternatively or additionally, processing logic (e.g., the processing logic of the dental modeling application 120) can process the 3D model to automatically determine and / or grade the edge lines. In one embodiment, reviewing the edge lines at box 945 includes performing operations 920-930. At box 950, the processing logic determines whether to continue using the 3D model to fabricate a dental prosthesis or to return the 3D model to the dental clinic 105. If the edge lines meet a minimum quality threshold, the method proceeds to box 960. If the edge lines do not meet the minimum quality threshold, the method continues to box 955, and the 3D model is returned to the dental clinic 105 so that the dentist can generate additional intraoral scans of the dental site. At box 955, the lab technician can manually mark unclear segments of the edge lines. Alternatively, at box 955, the processing logic can automatically mark unclear segments, or at box 945, unclear segments have already been marked. A message is then sent to the dentist requesting the generation of additional intraoral images. This message can provide a copy of the 3D model, showing the areas that should be re-imaged.

[0253] At box 960, the edge line can be automatically adjusted. In some cases, at box 950, the processing logic may determine that the edge line quality is insufficient, but for some reason, the dentist may be unable to acquire new images of the tooth area. In this case, the processing logic can proceed to box 960 even if the edge line has an unacceptable quality level. In this case, the edge line can be automatically adjusted at box 960. Alternatively, the edge line can be manually adjusted using, for example, CAD tools. In one embodiment, the edge line is adjusted by generating an image of a 3D model (e.g., by projecting the 3D model onto a 2D surface) and processing the image using a trained machine learning model that has been trained to correct edge lines in images of prepared teeth. In one embodiment, Figure 20 One or more operations of method 2000 are performed at box 960.

[0254] At box 965, the processing logic uses a virtual 3D model of the dental site to generate a dental prosthesis. In one embodiment, the virtual 3D model is input into a rapid prototyping machine (e.g., a 3D printer) and a physical model of the dental site (e.g., prepared teeth and adjacent teeth) is generated. The physical 3D model can then be used to generate the dental prosthesis. Alternatively, a virtual 3D model of the dental prosthesis can be generated from the virtual 3D model of the dental site, and this virtual 3D model can be used to directly manufacture the dental prosthesis using 3D printing. At box 970, the dental prosthesis can then be transported to dental clinic 105.

[0255] Figure 10 Another example workflow of a method 1000 for generating an accurate virtual 3D model of a tooth site and manufacturing a dental prosthesis from that virtual 3D model, according to an embodiment of this disclosure, is shown. The workflow can be performed at a dental clinic 105 or a dental laboratory 110. Those operations performed at the dental clinic 105 can be performed during a single patient visit or over multiple patient visits. The operations listed below at the dental clinic 105 can be performed, for example, by an intraoral scanning application 115. The operations listed below at the dental laboratory 110 can be performed, for example, by a tooth modeling application 120. In embodiments, the intraoral scanning application 115 and / or the tooth modeling application 120 can be combined. Figure 25 The tooth modeling logic is 2550.

[0256] Method 1000 may begin at box 915, where processing logic executed on a computing device associated with dental clinic 105 receives intraoral scan data. As described above, the intraoral scan data may have been generated according to a standard (e.g., full shrinkage) scanning procedure or a partial shrinkage scanning procedure. At box 918, a 3D model of at least a portion of the dental arch (e.g., one or more tooth sites) is generated using the intraoral scan data. At box 920, the processing logic performs automatic edge marking on the 3D model, as discussed elsewhere herein. At box 922, the processing logic determines whether any suspicious areas of the 3D model are identified. For example, suspicious areas may be identified by identifying edge segments with cost values ​​exceeding a cost threshold or by identifying edge segments with quality values ​​below a quality threshold, where the quality values ​​may be based on cost values. For example, the processing logic may calculate one or more edge quality scores. Each edge quality score may be based on the cost value of the edge line (or a segment of the edge line) as calculated using a cost function, and the edge quality score may be compared to a quality threshold to determine whether a suspicious area is identified. If a suspicious area is identified, the method proceeds to box 924. If no suspicious area is identified, the method proceeds to box 945.

[0257] At box 924, the processing logic automatically locks areas of the 3D model except for the suspicious areas. This ensures that the locked areas, representing the areas of the teeth, will not be modified. In some embodiments, a one-way lock is used to lock areas of the 3D model except for the suspicious areas, as described above. At box 926, the suspicious areas can then be erased using the eraser tool.

[0258] The physician can then be instructed to generate one or more new intraoral images depicting the suspicious area that has been erased in the 3D model. Following such an instruction, the physician can perform additional intraoral scans, and at box 928, the processing logic can receive new scan data, including one or more new intraoral images depicting the suspicious area. The additional intraoral scans can be performed using either a standard scanning procedure or a partially condensed scanning procedure. The method can then return to box 920 and update the 3D model using the received new scan data.

[0259] Alternatively or additionally, in some cases, the scan data used to generate the 3D model includes mixed intraoral images, where each mixed intraoral image is based on a combination of multiple distinct intraoral images that may have been generated sequentially. In this case, the processing logic can access one or more image files of the mixed images and can determine which mixed images are used to generate the 3D model representing portions of the suspected region. The processing logic can then examine the distinct images used to generate one or more of the determined mixed images. The processing logic can analyze these different images to identify one or more distinct images that provide a superior representation of the suspected region, and can select one or more of the identified distinct images. The method can then return to box 920 and update the 3D model using the selected one or more distinct intraoral images.

[0260] In addition to or as an alternative to the operations described above in box 928, at box 929, the processing logic may also receive input from a physician who manually manipulates the 3D surface of the 3D model at suspicious areas. For example, the physician may manually draw surfaces, draw edge lines, etc. The physician may also manage soft tissue, for example, by manually removing a representation of a portion of soft tissue from the 3D model. This may enable the generation of one or more new images of the dental area from the 3D model, which can be fed into a machine learning model trained to identify edge lines. At box 920, the output of the machine learning model can then be used to update the 3D model.

[0261] At box 945, the lab technician can review the edge lines in the 3D model (e.g., using a dental modeling application 120). Alternatively or additionally, processing logic (e.g., the processing logic of the dental modeling application 120) can process the 3D model to automatically determine and / or grade the edge lines. At box 950, the processing logic determines whether to continue using the 3D model to fabricate the dental prosthesis or to return the 3D model to the dental clinic 105. If the edge lines meet a minimum quality threshold, the method proceeds to box 960. If the edge lines do not meet the minimum quality threshold, the method continues to box 955, and the 3D model is returned to the dental clinic 105 so that the dentist can generate additional intraoral scans of the dental site. At box 955, the lab technician can manually mark unclear segments of the edge lines. Alternatively, at box 955, the processing logic can automatically mark unclear segments, or at box 945, unclear segments have already been marked. A message is then sent to the dentist requesting the generation of additional intraoral images. This message can provide a copy of the 3D model showing the areas that should be re-imaged.

[0262] At box 960, the edge line can be automatically adjusted. In some cases, at box 950, the processing logic may determine that the edge line quality is insufficient, but for some reason, the dentist may be unable to acquire new images of the tooth area. In this case, the processing logic can proceed to box 960 even if the edge line has an unacceptable quality level. In this case, the edge line can be automatically adjusted at box 960. Alternatively, the edge line can be manually adjusted using, for example, CAD tools. In one embodiment, the edge line is adjusted by generating an image of a 3D model (e.g., by projecting the 3D model onto a 2D surface) and processing the image using a trained machine learning model that has been trained to correct edge lines in images of prepared teeth. In one embodiment, Figure 20 One or more operations of method 2000 are performed at box 960.

[0263] At box 965, the processing logic uses a virtual 3D model of the dental site to generate a dental prosthesis. In one embodiment, the virtual 3D model is input into a rapid prototyping machine (e.g., a 3D printer) and a physical model of the dental site (e.g., prepared teeth and adjacent teeth) is generated. The physical 3D model can then be used to generate the dental prosthesis. Alternatively, a virtual 3D model of the dental prosthesis can be generated from the virtual 3D model of the dental site, and this virtual 3D model can be used to directly manufacture the dental prosthesis using 3D printing. At box 970, the dental prosthesis can then be transported to dental clinic 105.

[0264] Figure 11 A workflow for training a machine learning model and applying the trained machine learning model to an image is illustrated according to embodiments of the present disclosure. The illustrated workflow includes a model training workflow 1105 and a model application workflow 1117. The model training workflow 1105 is used to train one or more machine learning models (e.g., deep learning models) to perform one or more image processing and / or labeling tasks on an image containing teeth. The model application workflow 1117 is used to apply one or more trained machine learning models to label one or more attributes and / or regions in the image of teeth and / or modify the image of teeth.

[0265] One type of machine learning model that can be used is an artificial neural network, such as a deep neural network. Artificial neural networks typically include feature representation components with classifier or regression layers that map features to a desired output space. For example, a convolutional neural network (CNN) hosts multiple layers of convolutional filters. Pooling is performed, and nonlinear problems are solved at lower layers, often topped by a multilayer perceptron that maps the top-level features extracted by the convolutional layers to a decision (e.g., a classification output). Deep learning is a class of machine learning algorithms that use cascaded, multi-layered nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as its input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure of layers, where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in image recognition applications, the raw input can be a pixel matrix; the first representation layer can abstract the pixels and encode edges; the second layer can combine and encode the arrangement of edges; the third layer can encode higher-level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer can identify bounding boxes around teeth in an image containing a face. It's worth noting that the deep learning process can learn on its own which features are best placed at which level. The "depth" in "deep learning" refers to the number of layers through which data is transformed. More precisely, deep learning systems have a considerable credit allocation path (CAP) depth. A CAP is a chain of transformations from input to output. CAP describes the possible causal relationship between the input and output. For feedforward neural networks, the CAP depth can be the network depth or the number of hidden layers plus one. For recurrent neural networks (where signals can propagate through layers more than once), the CAP depth can be infinite.

[0266] In one embodiment, a class of machine learning models called MobileNet is used. MobileNet is an efficient machine learning model based on a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks. MobileNets can be convolutional neural networks (CNNs) that can perform convolutions in both the spatial and channel domains. MobileNets can include a stack of separable convolutional modules, which consist of depthwise convolutions and pointwise convolutions (conv 1x1). Separable convolutions perform convolutions independently in the spatial and channel domains. This decomposition of convolutions can significantly reduce the computational cost from HWNK. 2 M to HWNK 2 (depthwise) plus HWNM(conv 1x1) (totaling HWN(K)) 2The computational cost of +M), where N represents the number of input channels, K 2 Here, M represents the size of the convolution kernel, H represents the number of output channels, and HxW represents the spatial size of the output feature map. This can reduce the bottleneck of computational cost for 1x1 convolutions.

[0267] Training a neural network can be achieved through supervised learning, which involves feeding the network a training dataset consisting of labeled inputs, observing its output, constraining the error (by measuring the difference between the output and the labeled value), and using techniques such as deep gradient descent and backpropagation to adjust the network's weights across all its layers and nodes to minimize the error. In many applications, this process is repeated among many labeled inputs in the training dataset to produce a network that can produce correct outputs when provided with inputs different from those present in the training dataset. This generalization is achieved in high-dimensional settings such as large images when sufficiently large and diverse training datasets are available.

[0268] The model training workflow 1105 and the model application workflow 1117 can be executed by processing logic performed by the processor of a computing device. For example, these workflows 1105 and 1117 can be executed by... Figure 25 This is achieved through one or more machine learning modules implemented in the intraoral scanning application, the tooth modeling application, and / or the tooth modeling logic 2550 executed on the processing unit 2502 of the computing device 2500 shown. Additionally, the following... Figures 12 to 22 Example operations and / or methods associated with training a machine learning model or applying a trained machine learning model to an input image are described.

[0269] For model training workflow 1105, a training dataset containing hundreds, thousands, tens of thousands, hundreds of thousands, or more images should be used to form the training dataset. In an embodiment, up to millions of patient dental cases that have undergone restorative procedures can be used to form the training dataset, wherein each case includes information about the different activities performed for that case and the time points at which those activities were performed. Each case may include, for example, data showing an initial 3D model of one or more tooth sites generated from an intraoral scan, data showing any modifications made to the 3D model by a laboratory technician and / or edge lines drawn on the 3D model by a laboratory technician, data showing whether the dentist accepted the modified 3D model, data showing whether the modified 3D model produced a successful dental prosthesis, and so on. This data can be processed to generate a training dataset for training one or more machine learning models. For example, the machine learning model can be trained to automate one or more processes performed manually by a laboratory technician, such as the process of marking edge lines on a 3D model of a tooth and / or the process of adjusting the surface of a 3D model of a tooth. Such a trained machine learning model can reduce the standard turnaround time of approximately 24 hours for processing 3D models generated at a dental clinic to a few minutes to a few hours.

[0270] In one embodiment, a first machine learning model 1155 is trained to mark edge lines in a 2D image of a prepared tooth. A collection of numerous (e.g., thousands to millions) 3D models of prepared teeth with marked edge lines 1112 can be collected. For each 3D model with marked edge lines, at box 1130C, a set of images (e.g., height maps) can be generated. Each image can be generated by projecting the 3D model (or a portion of the 3D model) onto a 2D surface or plane. In some embodiments, different images of the 3D model can be generated by projecting the 3D model onto different 2D surfaces. For example, a first image of the 3D model can be generated by projecting the 3D model onto a 2D surface at a top-down viewpoint, a second image can be generated by projecting the 3D model onto a 2D surface at a first lateral viewpoint (e.g., buccal view), a third image can be generated by projecting the 3D model onto a 2D surface at a second lateral viewpoint (e.g., lingual view), and so on. Each image may include a height map that includes depth values ​​associated with each pixel of the image. For each image, a probability map or mask can be generated based on the marked edge lines in the 3D model and the 2D surface on which the 3D model is projected. The size of the probability map or mask can be equal to the pixel size of the generated image. Each point or pixel in the probability map or mask can include a probability value indicating the probability that the point represents an edge line. For example, a point that does not represent an edge line can have a value of 0 (0%), while a point that represents an edge line can have a value of 1 (100%).

[0271] At box 1138, a first machine learning model is trained using image pairs generated from a 3D model with labeled edge lines. The first machine learning model (e.g., a first deep learning model) can be trained to determine edge lines in an image of a prepared tooth. Specifically, the first machine learning model can be trained to generate a probability map, where each point in the probability map corresponds to a pixel in the input image and indicates the probability that the pixel represents an edge line of the prepared tooth.

[0272] Figure 12 A flowchart of a method 1200 for training a machine learning model to determine edge lines in an image of prepared teeth, according to an embodiment, is shown. At block 1205 of method 1200, processing logic receives a virtual 3D model of the dental arch, which has marked edge lines. An example 3D model 1210 is shown as having marked edge lines 1215.

[0273] At box 1220, for each virtual 3D model, the processing logic generates one or more images, which include height maps from the virtual 3D model. As described above, each image can be generated by projecting the 3D model onto a 2D surface. In one embodiment, approximately 10-150 grayscale height maps are generated for each case or patient. Each image may include an associated mask or probability map indicating which pixels in the image represent edge lines. Example image 1225 and associated mask or probability map 1230 are shown. In one embodiment, each virtual 3D model includes a specific tooth number and / or a label indicating a specific feature.

[0274] At box 1235, the processing logic feeds the training dataset, including the heightmap, into an untrained machine learning model. At box 1240, the processing logic trains the untrained machine learning model based on the training dataset to generate a trained machine learning model that identifies the edge lines in the heightmap of the prepared teeth. Training can be performed by feeding the images into the machine learning model one at a time. For each input image, the machine learning model generates a probability map that indicates the probability that each pixel in the image represents a heightmap. The processing logic can then compare the generated probability map with a known probability map or mask, and can perform backpropagation to update the weights of the nodes in the machine learning model. This process can be repeated using a large portion of the training dataset, where each iteration slightly refines the accuracy of the machine learning model. Once the machine learning model is trained, it can be tested using a retained portion of the training dataset.

[0275] Additionally, in one embodiment, the machine learning model is trained to identify teeth, gums, and / or excess material. In one embodiment, the machine learning model is also trained to determine one or more specific tooth numbers and / or identify specific indications (or multiple indications) in the input image. Thus, a single machine learning model can be trained to identify and / or correct edge lines, and can also identify teeth in general, specific tooth numbers, gums, and / or specific indications (e.g., cavities, cracks, etc.). In an alternative embodiment, a separate machine learning model is trained for each specific tooth number and each specific indication. Thus, tooth numbers and / or indications (e.g., specific dental prostheses to be used) can be indicated (e.g., can be input by the user), and a suitable machine learning model can be selected based on the specific tooth number and / or specific indication.

[0276] Additionally or alternatively, in one embodiment, a machine learning model (or a different machine learning model) is trained to determine model orientation, insertion path for the restoration or bridge, and / or the positioning of the 3D model within a CAM template. The machine learning model may be trained to process images of teeth (e.g., height maps) and output data (e.g., vectors, matrices, etc.) containing additional information such as model orientation, insertion path, and / or the positioning of the 3D model within the CAM template. For example, the machine learning model may output a vector identifying the insertion path and / or may output a matrix representing the model orientation.

[0277] In embodiments, the machine learning model can be trained to output identifiers of edge lines and individual information indicating one or more of the above (e.g., insertion path, model orientation, tooth identifier, gingival identifier, extraneous material identifier, etc.). In one embodiment, the machine learning model (or different machine learning models) is trained to perform one or more of the following: identify teeth represented in the height map, identify gingival tissue represented in the height map, identify extraneous material in the height map (e.g., material that is not gingival or tooth tissue), and / or identify edge lines in the height map. In some cases, edge lines identified by such a machine learning model trained to identify teeth, gingival tissue, extraneous material, and edge lines can have improved accuracy because the machine learning model can learn to ignore which tooth / gingival boundaries and which artifacts.

[0278] For embodiments where the machine learning model is trained to output insertion paths, the training data may include a height map that includes the target insertion path. For embodiments where the machine learning model is trained to output model orientations, the training data may include a height map that includes labeled model orientations. For embodiments where the machine learning model is trained to output tooth identifiers, the training data may include a height map that includes labeled teeth. For embodiments where the machine learning model is trained to output gingival identifiers, the training data may include a height map that includes labeled gingival tissue. For embodiments where the machine learning model is trained to output identifiers of excess material, the training data may include a height map that includes labeled excess material. For embodiments where the machine learning model is trained to output multiple pieces of information (e.g., edge identification, insertion path, tooth number identifier, gingival identifier, excess material identifier, and / or model orientation), the training data may include a height map having targets / labels identifying the type of information the model is to output.

[0279] In one embodiment, the machine learning model is trained to determine a confidence score for each pixel, which indicates the confidence that the pixel represents an edge line. In some embodiments, the confidence score can be used to determine the quality value of a segment of the edge line.

[0280] return Figure 11In one embodiment, a second machine learning model 1175 is trained to modify an image (e.g., height maps) in a manner that can correct one or more features of the teeth shown and / or generate edge lines. At box 1110A, a collection of numerous (e.g., thousands to millions) original 3D models of teeth can be collected. At box 1110B, a corresponding collection of numerous modified 3D models of teeth can also be collected. Furthermore, 3D models approved by doctors and / or dental laboratories can be collected. Each modified 3D model may correspond to a specific original 3D model and may have been generated by manually adjusting the surface of the 3D model by a laboratory technician (e.g., adding or clarifying edge lines in the original 3D model). For each original 3D model, one or more height maps are generated. Additionally, for each corresponding modified 3D model, one or more height maps are generated. Each height map generated for the original 3D model is generated by projecting the 3D model onto a 2D surface, and each height map generated for the corresponding modified 3D model is generated by projecting the modified 3D model onto the same 2D surface onto which the original 3D model is projected. Therefore, paired heightmaps can be generated, where the heightmap from the original 3D model is the input to the second machine learning model, and the corresponding heightmap from the modified 3D model is the target of the second machine learning model associated with the heightmap of the original 3D model. At box 1135, the original heightmap and the corresponding modified heightmap can be correlated to generate a training dataset.

[0281] At box 1140, a second machine learning model is trained using a training dataset consisting of the original heightmap and a modified heightmap (or an image including the original and modified heightmaps). The second machine learning model (e.g., a second deep learning model) can be trained to adjust the surface of the tooth (e.g., adjust the surface of the prepared tooth and / or add edge lines to a portion of the prepared tooth). The first machine learning model 1155 can identify existing edge lines already present in the heightmap, while the second machine learning model 1175 can change the shape of the surfaces represented in the heightmap to add edge lines where they were not previously represented.

[0282] Figure 13A flowchart of a method 1300 for training a machine learning model to correct an image of a tooth, according to an embodiment, is shown. At block 1305 of method 1300, the processing logic receives an original virtual 3D model of the dental arch, which has interfering surfaces (e.g., surfaces requiring correction). Example 3D model 1315 is shown as an interfering surface 1310 with a segment having an occluding edge line. At block 1320, the processing logic receives an associated corrected virtual 3D model where the interfering surface has been removed. Example corrected 3D model 1325 is shown as a fabricated segment with an edge line 1322 where the interfering surface 1310 has been located. The corrected 3D model may include additional information that can instruct a dental laboratory to take actions to correct the 3D model. Examples of such actions include cleaning the model of surrounding soft tissue, removing artifacts (e.g., those caused by blood, saliva, obstructions such as cotton rolls or shrink sutures), etc. Furthermore, model orientation, insertion path of the restoration or bridge, positioning of the 3D model within a computer-aided manufacturing (CAM) template, and / or other information may be included in the corrected 3D model.

[0283] At box 1330, for each virtual 3D model, the processing logic generates one or more images, which include height maps from the virtual 3D model. As described above, each image can be generated by projecting the 3D model onto a 2D surface. Image (e.g., height map) pairs are generated for each original virtual 3D model, where the first image is generated from the original 3D model and the second image is generated from the corresponding corrected 3D model. In one embodiment, approximately 10-150 grayscale height maps are generated for each virtual 3D model. In another embodiment, approximately 100 grayscale height maps are generated for each virtual 3D model.

[0284] At box 1335, the processing logic inputs the training dataset, including the heightmap, into the untrained machine learning model. At box 1340, the processing logic trains the untrained machine learning model based on the training dataset to generate a trained machine learning model that generates a corrected or modified heightmap with altered tooth surfaces and / or includes added edge lines. Training can be performed by inputting images generated from the original 3D model into the machine learning model one at a time. For each input image, the machine learning model generates a modified heightmap. The processing logic can then compare the generated modified heightmap with a known corrected heightmap generated from a corrected 3D model corresponding to the original 3D model, and can perform backpropagation to update the weights of nodes in the machine learning model. This process can be repeated using a large portion of the training dataset, where each iteration slightly refines the accuracy of the machine learning model. Once the machine learning model is trained, a retained portion of the training dataset can be used to test the model.

[0285] In one embodiment, the machine learning model (or a different machine learning model) is also trained to determine quality scores for surfaces and / or portions of surfaces. For example, the training dataset may include labeled images and / or 3D models with high-quality surfaces and labeled images and / or 3D models with low-quality surfaces. This training dataset can be used to train the machine learning model to determine the surface quality of different surfaces and / or portions of surfaces represented in the images. Once such a machine learning model is trained, images generated by projecting the 3D model onto the 2D surface can be applied to the machine learning model in a manner discussed elsewhere herein, and the machine learning model can output a mask that identifies surface quality values ​​for different surface portions (e.g., for each pixel of the input image). These surface quality values ​​can then be projected onto the 3D model from which the images are generated (e.g., as a texture). The surfaces of the 3D model can then be labeled or highlighted based on their surface quality values. For example, surfaces with low surface quality values ​​can be labeled or highlighted.

[0286] Additionally, in one embodiment, the machine learning model is trained to identify teeth, gums, and / or excess material. In one embodiment, the machine learning model is also trained to determine one or more specific tooth numbers and / or identify specific indications (or multiple indications) in the input image. Thus, a single machine learning model can be trained to identify and / or correct edge lines, and can also identify teeth in general, specific tooth numbers, gums, and / or specific indications (e.g., cavities, cracks, etc.). In an alternative embodiment, a separate machine learning model is trained for each specific tooth number and each specific indication. Thus, tooth numbers and / or indications (e.g., specific dental prostheses to be used) can be indicated (e.g., can be input by the user), and a suitable machine learning model can be selected based on the specific tooth number and / or specific indication.

[0287] Additionally or alternatively, in one embodiment, a machine learning model (or a different machine learning model) is trained to determine model orientation, the insertion path for the restoration or bridge, and / or the positioning of the 3D model within a CAM template. The machine learning model may be trained to process images of teeth (e.g., height maps) and output data containing additional information such as model orientation, insertion path, and / or the positioning of the 3D model within the CAM template. For example, the machine learning model may output a vector identifying the insertion path and / or may output a matrix representing the model orientation.

[0288] In an embodiment, the machine learning model may be trained to output a modified height map (or other modified surface) having cleaned edge lines, optionally with edge line identifiers, and individual information indicating one or more of the above (e.g., insertion path, model orientation, tooth identifier, gingival identifier, excess material identifier, etc.). In one embodiment, the machine learning model (or a different machine learning model) is trained to perform one or more of the following: outputting a modified height map having cleaned surfaces and / or cleaned edge lines, identifying teeth represented in the height map, identifying gingival tissue represented in the height map, identifying excess material (e.g., material that is not gingival or tooth tissue) in the height map, and / or identifying edge lines in the height map.

[0289] For embodiments where the machine learning model is trained to output insertion paths, the training data may include a height map that includes the target insertion path. For embodiments where the machine learning model is trained to output model orientations, the training data may include a height map that includes labeled model orientations. For embodiments where the machine learning model is trained to output tooth identifiers, the training data may include a height map that includes labeled teeth. For embodiments where the machine learning model is trained to output gingival identifiers, the training data may include a height map that includes labeled gingival tissue. For embodiments where the machine learning model is trained to output identifiers of excess material, the training data may include a height map that includes labeled excess material. For embodiments where the machine learning model is trained to output multiple pieces of information (e.g., edge line identifiers, modified height maps, insertion paths, tooth number identifiers, gingival identifiers, excess material identifiers, and / or model orientations), the training data may include a height map having targets / labels that identify the type of information the model is to output.

[0290] Return to reference Figure 11The first and / or second trained models 1155, 1175 can be trained using embeddings including grayscale images, which include height maps (also simply referred to as grayscale images). The embeddings can be input to a machine learning model that has been projected into a more convenient representation space. In some embodiments, the embedding includes a grayscale image and a timestamp. The scan may change over time, and the timestamps provide additional data that can be used to further identify edge lines and / or adjust the surface of the teeth. In some embodiments, different portions of the 3D model can be generated from blended images, each of which may have an associated timestamp. Images generated from the 3D model may include one or more timestamps from the blended images used to generate a portion of the 3D model represented in the projected image. Timestamps may be associated with specific pixels. For example, a first set of pixels may include a first timestamp from a first blended image, and a second set of pixels may include a second timestamp from a second blended image.

[0291] An embodiment has been described with reference to the generated image, which was generated by projecting a 3D model onto a 2D surface. Alternatively or additionally, the new image may be a composite image generated from intraoral scan data. In this case, the composite image may include a timestamp.

[0292] In some embodiments, the embedding includes a grayscale image and an associated color image. The color image can be generated from the 3D model by projecting the 3D model onto the same 2D surface onto which the 3D model is projected to generate the heightmap. Color data can improve the accuracy of identifying edge lines and / or improve the accuracy of modifying the image / heightmap.

[0293] In some embodiments, a generic first and second trained model, agnostic to the dentist and dental laboratory, can be generated. In other embodiments, the first and second trained models can be trained using embeddings including height maps and identifiers of the dentist and / or dental laboratory. When new images are fed into the trained machine learning model, information about the dentist and / or laboratory can be input to improve the model's accuracy. Alternatively, separate models can be generated for specific dentists, specific dental laboratories, and / or specific combinations of dentists and dental laboratories. Such models can be updated continuously or periodically as new data is received for a specific dentist and / or dental laboratory. Customized machine learning models can provide improved accuracy. In some embodiments, a generic model is initially used (e.g., for a new dentist, a new laboratory, or a new dentist / laboratory combination). As new data is received for a new dentist, a new laboratory, or a new dentist / laboratory combination, the new data can be used to refine the model for a specific dentist and / or dental laboratory, thereby providing a customized experience.

[0294] In some embodiments, a trained machine learning model can be continuously trained (e.g., via reinforcement learning). For example, the trained machine learning model can be used to process images generated from a 3D model, and the output of the machine learning model can be used to update the 3D model. The updated 3D model generated based on the output of the machine learning model may be accepted or rejected by a doctor or dental laboratory. If the updated 3D model is rejected, the doctor or dental laboratory can manually correct the 3D model. In this case, the manually corrected 3D model can be projected onto the same plane used for the image and the modified image. The projected image of the manually corrected 3D model can then be used as a target. The original image and / or the modified image can be used to further train the machine learning model, with the projected image of the manually corrected 3D model as the target, to further refine the machine learning model.

[0295] Figure 14 A flowchart of a method 1400 for training a machine learning model using image data according to an embodiment is shown. In the embodiment, method 1400 may be performed to train a first machine learning model 1155 and / or a second machine learning model 1175, and method 1400 may be performed in conjunction with method 1200 and / or method 1300.

[0296] At block 1402 of method 1400, an untrained machine learning model is initialized. The initialized machine learning model can be a deep learning model such as an artificial neural network. One type of artificial neural network that can be initialized and then trained is MobileNet. In one embodiment, MobileNet is initialized with an initial module. Initialization of the artificial neural network may include selecting initial parameters for the neural network. The solution of a non-convex optimization algorithm depends at least in part on the initial parameters, therefore the initialization parameters should be appropriately selected. In one embodiment, the parameters are initialized using a Gaussian or uniform distribution with arbitrary set variance.

[0297] At box 1405, the untrained machine learning model receives a first data point from the training dataset. For example, the first data point could be an image / heightmap 1225 and a mask 1230 showing edge lines. Method 1400 is shown together with the example heightmap 1225 and mask 1230 used to train the machine learning model to identify edge lines. However, method 1400 can also be performed to train the machine learning model to modify the heightmaps / images of teeth to correct those images and / or add edge lines to those images.

[0298] At box 1410, the size of the mask and / or image can be adjusted. For example, machine learning models can be used with images within a specific range of pixel sizes, and if an image falls outside this range, its size can be adjusted. Training images may be of varying sizes. However, many deep learning algorithms only accept images with a fixed size. Therefore, the size of the images (and their accompanying masks) in the training dataset can be adjusted to have a fixed size. For example, methods such as nearest neighbor interpolation or box sampling can be used to resize the images. At box 1415, the image data can then be augmented. Training large-scale neural networks typically uses tens of thousands of images, which are not readily available in many real-world applications. Data augmentation can be used to artificially increase the effective sample size. Common techniques include randomly rotating, shifting, cropping, flipping, etc., existing images to increase the sample size.

[0299] At box 1420, the processing logic optimizes the parameters of the machine learning model from the data points. The machine learning model applies a classification or label to the image based on the current parameter values. The artificial neural network includes an input layer, which consists of values ​​from the data points (e.g., the intensity and / or height values ​​of pixels in image 1225). The next layer is called a hidden layer, and each node in the hidden layer receives one or more input values. Each node contains parameters (e.g., weights) applied to the input values. Thus, each node essentially feeds the input values ​​into a multivariate function (e.g., a nonlinear mathematical transformation) to produce an output value. The next layer can be another hidden layer or an output layer. In either case, the nodes in the next layer receive output values ​​from the nodes in the previous layer, and each node applies weights to these values ​​and then generates its own output value. This can be performed at each layer. The final layer is the output layer, where there is one node for each category. For the artificial neural network being trained, there can be a first category (no edge lines) and a second category (edge ​​lines). Furthermore, the category is determined for each pixel in the image. For each pixel in the image, the final layer applies the probability that the pixel belongs to the first category (no edge lines) and the probability that the pixel belongs to the second category (edge ​​lines).

[0300] The processing logic compares the classification, label, or other output of the machine learning model (e.g., a modified image) with the provided classification, label, or other target (in this case, mask 1230) to determine one or more classification errors. An error term or increment can be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more parameters (weights of one or more inputs to a node) for one or more of its nodes. Parameters can be updated via backpropagation, such that nodes at the highest layer are updated first, then nodes at the next lower layer, and so on. An artificial neural network contains multiple layers of "neurons," where each layer receives values ​​as input from neurons in the previous layer. The parameters of each neuron include weights associated with the values ​​received from each neuron in the previous layer. Therefore, adjusting the parameters can include adjusting the weights assigned to each input of one or more neurons at one or more layers in the artificial neural network.

[0301] Once the model parameters have been optimized, model validation can be performed at box 1425 to determine if the model has been improved and to determine the current accuracy of the deep learning model. At box 1430, the processing logic determines whether a stopping criterion has been met. The stopping criterion can be a target accuracy level, a target number of processed images from the training dataset, a target amount of parameter change on one or more previous data points, a combination of these, and / or other criteria. In one embodiment, the stopping criterion is met when at least a minimum number of data points have been processed and at least a threshold accuracy has been achieved. The threshold accuracy can be, for example, 70%, 80%, or 90% accuracy. In one embodiment, the stopping criterion is met if the accuracy of the machine learning model has stopped improving. If the stopping criterion is not met, the method can return to box 1420 to further optimize the model based on another data point from the training dataset. Alternatively, in an embodiment, the method can return to box 1405. If the stopping criterion has been met, the method continues to box 1435 and the machine learning model is trained. As described herein, the machine learning model can be an artificial neural network (or other deep learning model), such as MobileNet. However, other types of machine learning models can also be used.

[0302] A trainable first machine learning model can output a probability map for an input image (e.g., an input image including a heightmap or an input heightmap) that has the same resolution as the input image (e.g., the same number of horizontal and vertical pixels). The probability map can be a binary mask that includes a first value for a pixel if it represents an edge line, and a second value for a pixel if it does not represent an edge line. Alternatively, the probability map can include numerical values ​​ranging from 0 to 1, where each pixel is assigned a value representing a probability from 0% to 100% that the pixel represents an edge line. Thus, the trained machine learning model makes pixel-level decisions for each pixel in the input image regarding whether that pixel represents an edge line and / or the probability that that pixel represents an edge line.

[0303] A trainable second machine learning model can output a modified output image with the same resolution as the input image, given an input image (e.g., an input image including a heightmap or an input heightmap). The modified output image may resemble the input image but may have an adjusted surface, where occlusions such as blood, saliva, soft tissue (e.g., gums), and receding lines have been removed, and the underlying tooth surfaces (e.g., including edge lines) have been depicted. Thus, the trained machine learning model makes pixel-level decisions for each pixel in the input image about whether and / or how the pixel should be adjusted to correct the input image.

[0304] Return again Figure 11 Once the first machine learning model is trained, it is stored in model storage 1145. Similarly, once the second machine learning model is trained, it is stored in model storage 1145. Model storage 1145 may include storing one or more machine learning models in persistent storage (e.g., a storage server), which may include solid-state storage devices, hard disk drives, tape drives, etc.

[0305] The model application workflow 1117 begins by receiving and / or generating new 3D models 1148 of one or more dental sites (e.g., prepared teeth and / or adjacent teeth). In one embodiment, the new 3D model is generated from intraoral scan data generated by a physician. For example, a physician can perform an intraoral scan of a patient using scanner 150 and can generate a 3D model from the intraoral scan. In one embodiment, a single intraoral image generated during the intraoral scan is processed using a third machine learning model during image capture. The third machine learning model can identify and remove soft tissue from the intraoral image. These modified intraoral images can then be used to generate a 3D model. The processing logic can more easily detect edge lines from the 3D model generated using such modified intraoral images.

[0306] At box 1149, a new image (e.g., a new heightmap) 1150 is generated by projecting a 3D model (or a portion thereof) onto a 2D surface. The new image 1150 is then fed into a first trained model 1155, which can be trained as described above. The first trained machine learning model 1155 determines edge lines in the new image 1150 and outputs a probability map, where each point in the probability map corresponds to a pixel in the new image and indicates the probability that the pixel represents an edge line. At box 1165, the probability map 1160 is projected onto the 3D model to update the 3D model. In one embodiment, probability information from the probability map is projected onto the 3D model as a texture. The updated 3D model can then include the probability that one or more points, vertices, or voxels (e.g., vertices on a 3D mesh representing the surface of the 3D model) represent edge lines. At box 1168, the processing logic can then compute edge lines based on the probability values ​​associated with points on the surface of the 3D model. Edge lines can be calculated using a cost function that finds the contour comprising the set of connected points that together have the minimum cost. (Reference) Figure 16 The calculation of the edge line is described in more detail. The edge line can then be drawn on the 3D model. In some embodiments, different cost values ​​are calculated for different segments of the edge line. The different cost values ​​can be compared with a maximum cost, and if any cost value exceeds the maximum cost, the associated edge line segment can be highlighted in the 3D model. The highlighted segments of the edge line represent segments that are unclear, inaccurate, and / or otherwise unacceptable.

[0307] Additionally or alternatively, the new image 1150 can be fed into a second trained model 1175, which can be trained as described above. In one embodiment, the new image 1150 is fed into the second trained model 1175 if one or more regions of the new 3D model are identified as unclear, inaccurate, or otherwise unacceptable. In one embodiment, the new image 1150 is fed into the second trained model if one or more segments of an edge line have been identified as unacceptable. Alternatively, the updated 3D model 1165 can be used to generate different new images, which can be fed into the second trained model 1175.

[0308] A second trained machine learning model 1175 generates a modified image (e.g., a modified height map) 1180. At box 1185, the modified image 1180 is used to update the 3D model. In one embodiment, data from the modified image 1180 is used to rewrite portions of the 3D model, thereby changing the shape of the 3D model's surface. In addition to the modified image, a probability map can also be generated for the modified image, indicating the probability that pixels in the modified image represent edge lines. The probability map can be used to determine the probability that points on the surface of the 3D model represent edge lines, and ultimately, edge lines are drawn on the 3D model. All segments of the edge lines in the updated 3D model should be sharp and have an acceptable level of accuracy. Therefore, edge lines can be computed and then drawn on the 3D model.

[0309] In one embodiment, a new image is generated from the updated 3D model, and the new image is processed by a first trained model to update the labeling of the edge lines on the 3D model (e.g., the operation of boxes 1155-1169 can be repeated using the new image generated from the updated 3D model).

[0310] Figure 15 A flowchart of a method 1500 for identifying edge lines in a 3D model of a tooth region according to an embodiment is shown. At block 1505 of method 1500, processing logic receives intraoral scan data of a tooth region including a prepared tooth. At block 1510, the processing logic generates a 3D model of the tooth region from the intraoral scan data.

[0311] At block 1515, processing logic receives or generates an image of the prepared tooth, wherein the image includes a height map of the prepared tooth. For example, a grayscale height map may be received or generated. In one embodiment, at block 1518, processing logic projects a 3D model onto a 2D surface to generate an image of the prepared tooth. In one embodiment, at block 1519, processing logic selects an intraoral image from intraoral scan data. In one embodiment, the intraoral image is a composite image that combines multiple distinct intraoral images to create a contrast.

[0312] At box 1520, the processing logic processes the image using a trained machine learning model that has been trained to identify the edge lines of the prepared teeth. The trained machine learning model can output a probability map that includes the probability that each pixel in the image represents an edge line. In one embodiment, the trained machine learning model corresponds to... Figure 11 The first trained model 1155. In one embodiment, multiple different machine learning models have been trained, each trained for a specific tooth number and / or a specific indication. An appropriate machine learning model can be selected based on the specific tooth number and / or specific indication, and an image (e.g., a height map) is input into the selected machine learning model.

[0313] At box 1525, the processing logic updates the 3D model of the tooth site by marking edge lines on the representation of the prepared tooth based on a probabilistic map. In one embodiment, method 1600 is executed to mark edge lines on the 3D model.

[0314] The operations in boxes 1515-1525 can be performed on many (e.g., up to approximately one hundred or more) images generated from a single 3D model of the tooth site. In an embodiment, data from multiple images in the aggregation can provide an accurate representation of the edge lines.

[0315] Figure 16 Another flowchart of a method 1600 for identifying edge lines in a 3D model of tooth locations according to an embodiment is shown. At block 1630 of method 1600, processing logic projects probabilistic information from a probability map (e.g., output by a machine learning model) onto the 3D model. The probability map may be associated with a height map generated from the 3D model. The height map can be used to determine a corresponding point on the 3D model for each pixel in the height map. The probability of the associated pixel can then be assigned to the determined corresponding point on the 3D model as texture.

[0316] At box 1635, the processing logic computes edge lines by applying a cost function to points on the 3D model. In one embodiment, the processing logic generates a matrix that identifies the probability that a point represents an edge line for each point (e.g., an edge, vertex, voxel, etc. on the surface of the 3D model). For example, an entry in the matrix with an nt chance of representing an edge line has a specified 0% probability.

[0317] The processing logic uses a cost function to create the nearest contour passing through points with a high probability of representing an edge line. In one embodiment, the total cost of the contour drawn for an edge line is the sum of all edges (e.g., vertices) included in the edge line, adjusted by a weight associated with each vertex. The weight of each vertex can be a function of the probability assigned to that vertex. The cost of vertices included in the edge line can be approximately 1 / (A+P), where A is a small constant and P is the probability that the vertex represents the edge line. The smaller the probability of a vertex, the greater the cost of including that vertex in the edge line. The cost of a segment of the edge line can also be calculated based on the sum of the costs of vertices included in those segments. When the probability is close to 100%, the cost is approximately 1, adjusted by length.

[0318] In one embodiment, the pathfinding operation or algorithm is applied to the 3D model using values ​​from the matrix as the cost basis. Any pathfinding algorithm can be used. Some examples of possible pathfinding algorithms include dynamic programming, Dijkstra's algorithm, and A*. Search algorithms, incremental heuristic search algorithms, etc. Pathfinding algorithms can apply a cost function to determine the path of an edge line.

[0319] Pathfinding algorithms can search for paths with either maximum or minimum cost, using the probabilities representing edge lines in a matrix as the cost basis. The cost function described above uses a probability-based inverse function to search for the minimum cost. Alternatively, a direct probability-based cost function can be used, where the maximum cost is searched. If the pathfinding algorithm is run to maximize cost, paths between vertices that produce the maximum aggregation of probability values ​​are determined. The probability scores of the vertices can be input into the pathfinding algorithm to find the path with maximum cost for each probability score. Pathfinding algorithms can be used to define the contours representing edge lines.

[0320] At box 1640, the processing logic marks the calculated edge lines on the representation of the teeth in the 3D model. At box 1645, the processing logic calculates the individual costs for the different segments of the edge lines, as described above. For example, the processing logic may determine multiple segments of the edge line, each segment comprising a set of connected or adjacent vertices. For each segment, the processing logic can use a cost function to calculate the segment's cost. Cost values ​​can be calculated for overlapping and / or non-overlapping segments. Alternatively, such individual costs may have already been calculated at box 1635.

[0321] At box 1650, the processing logic determines whether any segment has a cost value / score that fails to meet the cost criteria. For example, the processing logic might determine whether any segment has a cost exceeding a cost threshold (if the cost function is optimized for minimum cost). Alternatively, the processing logic might determine whether any segment has a cost value / score below a cost threshold (if the cost function is optimized for maximum cost). If all segments meet the cost criteria, the method continues to box 1665. If any segment fails to meet the cost criteria, the method continues to box 1655.

[0322] At box 1665, optionally, the processing logic highlights segments whose edge lines meet cost criteria but are close to not meeting them. For example, the processing logic could highlight segments with the highest cost.

[0323] At box 1655, the processing logic determines that one or more segments of the edge line that fail to meet cost standards have unacceptable levels of uncertainty or sharpness. At box 1660, the processing logic highlights those segments of the edge line that have unacceptable levels of uncertainty or sharpness.

[0324] Figure 17 A flowchart of a method 1700 for updating a 3D model of a tooth region according to an embodiment is shown. For example, in some embodiments, method 1700 may be performed after method 1500 and / or method 1600.

[0325] At block 1705 of method 1700, the processing logic automatically locks one or more regions of the 3D model of one or more tooth locations, which include segments of the calculated edge lines with an acceptable level of uncertainty (e.g., regions depicting segments of the edge lines that meet edge line cost criteria). At block 1710, optionally, the processing logic automatically erases regions of the 3D model that include segments of the edge lines with an unacceptable level of uncertainty (e.g., those with costs that fail to meet cost criteria).

[0326] At box 1715, the processing logic can highlight areas of the 3D model that require new scan data (e.g., erased areas). Additionally, the processing logic can instruct the physician to generate one or more intraoral scans of a portion of the prepared tooth, associated with the erased area of ​​the 3D model.

[0327] At box 1720, the processing logic receives new scan data, which includes at least one intraoral image depicting segments with calculated edge lines exhibiting unacceptable levels of uncertainty. At box 1730, the processing logic updates the 3D model using the new intraoral image to output an updated 3D model. A first region of the 3D model, previously including segments with calculated edge lines exhibiting unacceptable levels of uncertainty, is replaced with information from the new intraoral image (or multiple images). A locked region of the 3D model, which includes segments with calculated edge lines exhibiting acceptable levels of uncertainty, remains unchanged during the update.

[0328] Figure 18 Another flowchart of a method 1800 for updating a 3D model of a tooth region according to an embodiment is shown. For example, in some embodiments, method 1800 may be performed after method 800 and / or method 1600.

[0329] At block 1805 of method 1800, the processing logic automatically locks one or more regions of the 3D model of one or more tooth locations, the one or more regions including segments of calculated edge lines with an acceptable level of uncertainty (e.g., regions depicting segments where the edge lines meet edge line cost criteria). Optionally, at block 1805, the processing logic automatically erases regions of the 3D model that include segments of the edge lines with an unacceptable level of uncertainty (e.g., those with costs that fail to meet cost criteria).

[0330] At box 1815, the processing logic determines that the intraoral scan data used to generate the 3D model includes mixed intraoral images, wherein each mixed intraoral image is an image based on a combination of multiple other clear intraoral images. At box 1820, the processing logic accesses individual intraoral images used to generate at least some of the mixed images from the intraoral scan data. At box 1825, the processing logic identifies a subset of the individual intraoral images that depicts segments with computed edge lines having an unacceptable level of uncertainty.

[0331] At box 1828, the processing logic selects a new intraoral image from the subset, wherein the new intraoral image includes an improved depiction of the edge line segments. For example, some of the sharp intraoral images in a particular blending image used to generate the 3D model may already include collapsed gingiva that obscures the edge lines. However, one or more other sharp intraoral images used to generate the blending image may show gingiva in a non-collapsed position. Intraoral images in which the gingiva is not collapsed can be selected. In one embodiment, by using Figure 11 The first trained model 1155 or another trained machine learning model that identifies the edge lines processes the image to evaluate it.

[0332] In one embodiment, intraoral images are evaluated by processing them using a trained machine learning model, which has been trained to identify the quality of depicted edge lines in a given intraoral image. The machine learning model may have been trained using a training dataset that includes intraoral images labeled with first-level unacceptable edge lines and intraoral images labeled with second-level acceptable edge lines. In one embodiment, when processing intraoral images of prepared teeth, the trained machine learning model outputs an image that includes an indication of acceptable or unacceptable edge lines. In another embodiment, when processing intraoral images of prepared teeth, the trained machine learning model outputs a quality score for the edge lines.

[0333] Each individual intraoral image can have an associated timestamp indicating when it was generated relative to other intraoral images that can be analyzed. In one embodiment, the embedding input into the machine learning model includes the intraoral image and the associated timestamp. In one embodiment, a recurrent neural network (RNN) is used for the machine learning model. Intraoral images can be input into the machine learning model in ascending order based on timestamps, and the machine learning model can evaluate the quality of the edge lines in part based on the changes between intraoral images. For example, the machine learning model can be able to identify when the quality of the edge lines deteriorates over time (e.g., due to gingival collapse after the removal of shrinkage material, due to bleeding from the patient over time, etc.). Thus, the machine learning model can detect deterioration of the surface (e.g., the edge lines). Furthermore, the machine learning model can identify when a physician wiped away blood or performed other actions that made subsequent images better than previous images.

[0334] At box 1830, the processing logic updates the 3D model with a new intraoral image selected from a subset of intraoral images used to generate a blended intraoral image that depicts a region with unacceptable edge segments.

[0335] At box 1835, the processing logic determines whether to accept the 3D model with updated edge lines. The updated 3D model can be presented to the doctor, who can review the 3D model and accept or reject it (and / or the edge lines of the 3D model or specific segments of the edge lines). If the 3D model is accepted, the method proceeds to box 1840 and uses the automatically generated updated 3D model. If the 3D model is not accepted, the method continues to box 1845.

[0336] At box 1845, the processing logic selects one or more additional individual intraoral images from the subset. At box 1850, the processing logic generates a different updated 3D model from each of the selected additional individual intraoral images. The user can then scroll through different options for the updated 3D model, where each option is based on using a different individual intraoral image. At box 1855, the processing logic receives the user's selection of one of the updated 3D models. At box 1860, the processing logic uses the selected updated 3D model (e.g., to send it to a dental laboratory).

[0337] Figure 19 Another flowchart of a method 1900 for identifying edge lines in a 3D model of a tooth region according to an embodiment is shown. At block 1905 of method 1900, processing logic receives intraoral scan data of a tooth region including a prepared tooth. At block 1910, processing logic generates a 3D model of the tooth region from the intraoral scan data.

[0338] At block 1915, processing logic receives or generates an image of the prepared tooth, wherein the image includes a height map of the prepared tooth. For example, a grayscale height map may be received or generated. In one embodiment, at block 1918, processing logic projects a 3D model onto a 2D surface to generate an image of the prepared tooth. In one embodiment, at block 1920, processing logic selects an intraoral image from intraoral scan data. In one embodiment, the intraoral image is a composite image that contrasts with multiple distinct, clear intraoral images combined together.

[0339] In box 1925, the processing logic processes the image using a trained machine learning model that has been trained to identify the edge lines of the prepared teeth. The trained machine learning model can output a probability map that includes the probability that each pixel of the image represents an edge line. In one embodiment, the trained machine learning model corresponds to... Figure 11 The first trained model 1155.

[0340] At box 1930, the processing logic projects the probability information from the probability map onto the 3D model. At box 1935, the processing logic calculates the edge lines by applying a cost function to the points in the 3D model. At box 1940, the processing logic determines whether the combined cost of the edge lines meets a cost criterion (e.g., exceeding a cost threshold if the objective is minimum cost, or falling below a cost threshold if the objective is maximum cost). If the combined cost meets the cost criterion, the method proceeds to box 1955 and draws the edge lines on the 3D model. If the combined cost fails to meet the cost criterion, the method continues to box 1945.

[0341] At box 1945, the processing logic determines that the edge line has an acceptable level of uncertainty. At box 1950, the edge line is not shown in the 3D model.

[0342] Figure 20 A flowchart 2000 illustrates a method for correcting the representation of a tooth in a 3D model of a tooth region according to an embodiment. At block 2005 of method 2000, processing logic receives intraoral scan data of a tooth region, including a tooth. The tooth may or may not be a prepared tooth. At block 2010, the processing logic generates a 3D model of the tooth region from the intraoral scan data. The 3D model includes a representation of the tooth and also includes a representation of an interfering surface that obscures a portion of the tooth. The interfering surface may be or include blood, saliva, soft tissue (e.g., gingiva), shrinkage material, etc.

[0343] At block 2015, processing logic receives or generates an image of the prepared tooth, wherein the image includes a height map of the prepared tooth. For example, a grayscale height map may be received or generated. In one embodiment, at block 2018, processing logic projects a 3D model onto a 2D surface to generate an image of the prepared tooth. In one embodiment, at block 2019, processing logic selects an intraoral image from intraoral scan data. In one embodiment, the intraoral image is a composite image that contrasts with multiple distinct, clear intraoral images combined together.

[0344] At box 2020, the processing logic processes the image to generate a modified image that includes a modified height map. The modified image shows the portions of the teeth that are occluded by interfering surfaces in the image. Therefore, the representation of the interfering surfaces can be removed from the modified image.

[0345] In one embodiment, at box 2022, a modified height map is generated using a trained machine learning model, which has been trained to generate a modified height map of the teeth from an input height map. An image can be input into the trained machine learning model, which can output the modified height map. In some embodiments, the trained machine learning model can also output a probability map that includes the probability that each pixel of the image represents an edge line. In one embodiment, the trained machine learning model corresponds to... Figure 11 The second trained model 1175. In one embodiment, multiple different machine learning models have been trained, each trained for a specific tooth number and / or a specific indication. An appropriate machine learning model can be selected based on the specific tooth number and / or specific indication, and an image (e.g., a height map) is input into the selected machine learning model.

[0346] At box 2025, the processing logic updates the 3D model of the tooth area using the modified heightmap. The 3D model can be updated by replacing a portion of the original surface of the 3D model with the updated surface based on the modified heightmap.

[0347] The operations in boxes 815-825 can be performed on many (e.g., up to approximately one hundred or more) images generated from a single 3D model of the tooth site. In an embodiment, data from multiple images in the aggregation can provide an accurate representation of the edge lines.

[0348] In some embodiments, the updated 3D model and / or modified image (e.g., modified heightmap) may be further processed to determine whether the modifications to the 3D model and / or image are acceptable. In one embodiment, a blended image is used to generate the 3D model. In this embodiment, the processing logic may access a sharpened image used to generate a blended image associated with segments of edge lines. The modified image can then be compared with the sharpened image, and the sharpened image that is closest to the modified image can be selected. The selected image can then be used to update the 3D model instead of the modified image. Alternatively, the selected image can be used to further update the 3D model after the modified image has been used to update it.

[0349] In some embodiments, a machine learning model that generates a probabilistic map of edge lines is used to process each sharpened image. In one embodiment, the machine learning model takes as input an embedding that includes a height map of teeth and a timestamp. Alternatively, the machine learning model may take as input an embedding that includes a grayscale height map or an embedding that includes a grayscale height map plus an embedding of a 2D color image without height information. A cost function can then be applied to each image to compute edge lines in the image. For each sharpened image, the edge lines computed from the image can be compared with edge lines identified in the modified image. A sharpened image with edge lines that are closest to the edge lines in the modified image can be selected. The selected image can then be used instead of the modified image to update the 3D model. Alternatively, the selected image can be used to further update the 3D model after the modified image has been used to update the 3D model.

[0350] In some embodiments, multiple different updated versions of the 3D model are generated, each version being generated using different selected images. These different updated versions of the 3D model can be presented to a physician, who can scroll through them and select the version that best reflects the actual margins of the patient's prepared teeth.

[0351] Figure 21A flowchart of a method 2100 for representing the edge lines of a prepared tooth in a 3D model of a corrected dental site according to an embodiment is shown. At block 2105 of method 2100, processing logic generates a 3D model of the dental site from intraoral scan data. At block 2110, processing logic detects edge lines in the 3D model of the prepared tooth from one or more images of the prepared tooth. In one embodiment, 10 to 150 grayscale height maps are generated by projecting the 3D model onto multiple different 2D surfaces. At block 2115, processing logic determines a quality score for each of the multiple segments of the edge line.

[0352] At box 2120, the processing logic determines whether any segment of the edge line has a quality score below a quality threshold. For example, an edge line segment may have a quality score below the quality threshold if it has a cost that fails to meet a cost criterion. The quality score can be calculated from the cost calculated for the edge line segment, for example. Alternatively, the cost can be used as the quality score. If any quality score is below the quality threshold, the method proceeds to box 2130. If all quality scores meet or exceed the quality threshold, the method proceeds to box 2125 and the edge line is identified as acceptable.

[0353] At box 2130, the processing logic can lock one or more portions of the 3D model, which include segments of edge lines with quality scores that meet a quality threshold. At box 2135, the processing logic can erase a portion of the 3D model associated with segments of edge lines that have quality scores below a quality threshold.

[0354] At box 2140, a new image is received. This new image may be an intraoral image generated by an intraoral scanner in response to a notification that a segment of the edge line has a quality score below a quality threshold. Alternatively, the new image may be a previously generated image, selected to provide an improved depiction of the edge line segment. For example, blended images may have been used to generate the 3D model, where each blended image is generated from multiple sharp images. The new image may be one such sharp image selected from the available sharp images, as described above regarding the selection of sharp images for generating the blended image. Alternatively, the new image may be generated automatically by projecting a portion of the 3D model onto a surface to generate a height map, which is then fed into a trained machine learning model (e.g., Figure 11 The second trained model (1175) was trained to modify intraoral images.

[0355] At box 2145, the processing logic updates the 3D model by replacing the portions of the 3D model associated with segments of the edge lines that have a quality score below the quality threshold with image data from the new image. Locked portions of the 3D model may remain unchanged during the update.

[0356] Figure 22 A flowchart of a method 2200 for generating 3D models of multiple tooth sites according to an embodiment is shown. At block 2205 of method 2200, processing logic receives intraoral scan data, which includes multiple intraoral images of at least a first tooth site (e.g., a first tooth) and a second tooth site (e.g., an adjacent second tooth). Each intraoral image may include a timestamp and a height map. In one embodiment, each intraoral image is based on a blended image that combines multiple sharp intraoral images.

[0357] At box 2210, the processing logic determines a first subset of intraoral images for generating a first 3D model depicting a portion of the first tooth region. The first subset may be determined at least in part based on: a) timestamps of the intraoral images in the first subset, and b) geometric data of the intraoral images in the first subset. At box 2215, the processing logic determines a second subset of intraoral images for generating a second portion of the 3D model depicting the second tooth region. The second subset may be determined at least in part based on: a) timestamps of the intraoral images in the second subset, and b) geometric data of the intraoral images in the second subset.

[0358] The first and second subsets can be generated by feeding multiple intraoral images into a trained machine learning model, which has been trained to classify the intraoral images for generating 3D models of tooth locations. In one embodiment, an embedding is generated for each intraoral image, which includes the intraoral image (or data from intraoral images) and a timestamp associated with that intraoral image. The embedding can be fed into the machine learning model. For each intraoral image, the machine learning model can output a first score associated with a first tooth location and a second score associated with a second tooth location. Alternatively, a separate trained machine learning model can be used for each tooth number. Thus, images can be fed into a first machine learning model that outputs the first scores of those images, and images can be fed into a second machine learning model that outputs the second scores of those images. The first subset can then be determined by selecting those images with a first score exceeding a quality threshold. The second subset can be determined by selecting those images with a second score exceeding a quality threshold. In one embodiment, the machine learning model is an RNN. In this embodiment, the intraoral images can be fed into the machine learning model in ascending chronological order.

[0359] At box 2225, the processing logic determines whether any area of ​​the tooth region is unclear (e.g., based on one or more of the techniques described above for identifying and / or grading edge lines). If no area of ​​the tooth region is unclear, the method proceeds to box 2230 and the 3D model is considered acceptable. If any area of ​​the tooth region is identified as unclear, the method continues to box 2235.

[0360] At box 2235, the processing logic identifies unclear areas in the dental region. These unclear areas can be identified as described in the techniques described above. At box 2240, the processing logic determines that the multiple intraoral images are composite images and accesses the individual images used to generate at least some of the composite images.

[0361] At box 2245, the processing logic identifies a subset of multiple individual intraoral images depicting an unclear region. At box 2250, the processing logic selects a specific image from the subset of multiple individual intraoral images, wherein the specific image includes an improved depiction of the region. The selection can be performed as described above with reference to the previous method. At box 2255, the processing logic updates the 3D model using the specific image.

[0362] In one embodiment, the processing logic generates multiple different versions of the updated 3D model, each of which is based on a different individual intraoral image from a subset of multiple individual intraoral images. The different versions can then be presented to a doctor. For example, the doctor can scroll or swipe through the different options until a suitable option is displayed. The processing logic can then receive the user's selection of a specific version of the updated 3D model and can use that version of the updated 3D model.

[0363] Figure 23 The marking of edge lines in a 3D model of a prepared tooth according to an embodiment is shown. As shown, a 3D model 2305 of the tooth region is projected 2310 onto a 2D surface to form a height map 2315, wherein the height map is a grayscale image including the depth or height value of each pixel. The height map 2315 is then processed 2320 using a machine learning model that has been trained to identify edge lines in the height map. The ML model outputs a probability map 2325, wherein each pixel in the probability map corresponds to a pixel in the height map 2315 and represents the probability that a pixel in the height map 2315 represents an edge line. In one embodiment, the probability map 2325 is a mask. The probability map 2325 is then projected back to the 3D model 2305 using height data from the height map 2315. The probability information can be represented as a texture on the 3D model. The probability information can then be used to draw edge lines 2340 in the 3D model 2305 to obtain an updated 3D model 2335.

[0364] Figure 24A A first example of automatic correction of a 3D model of a tooth according to an embodiment is shown. As shown, a 3D model 2405 of the tooth region is projected 2410 onto a 2D surface to form a height map 2415. In the 3D model 2405 of the tooth region, region 2408 has soft tissue covering a portion of the edge lines. The height map 2415 is processed 2420 using a machine learning model that has been trained to modify an image of the prepared tooth in a manner that clarifies and / or improves edge lines in the image. The ML model outputs a modified height map 2425, which may have the same number of pixels as height map 2415. The modified height map 2435 includes cleaned surfaces and / or edge lines. The modified height map 2435 is then projected back onto the 3D model 2405, and data from the modified image 2435 is used to rewrite a portion of the 3D model. As shown, region 2408 in the updated 3D model 2435 has cleaned edge lines. In some embodiments, in addition to the modified heightmap, the machine learning model also outputs a probability map, wherein each pixel in the probability map indicates the probability that the pixel represents an edge line. In one embodiment, the probability map is a binary mask, wherein each pixel has a value of 1 (100% probability) or a value of 0 (0% probability). In one embodiment, the modified heightmap includes labeled edge lines. For example, the pixels representing edge lines in the modified heightmap may include specific flags or values. Thus, the updated 3D model 2435 may include drawn edge lines determined from the probability map or other edge line indications output by the machine learning model.

[0365] Figure 24B A second example of automatic correction of a 3D model of a tooth according to an embodiment is shown. As shown, a 3D model 2450 of the tooth region is a starting 3D model. This 3D model can be projected onto a 2D surface to form a height map. A machine learning model can be used to process the height map, which has been trained to a) modify the image of the prepared tooth in a manner that clarifies and / or improves edge lines in the image (e.g., cleans the edge lines), and b) identify the edge lines. The ML model can output a modified height map, which may have the same number of pixels as the input height map. The modified height map is then projected back to the 3D model 2450, and data from the modified image is used to rewrite a portion of the 3D model to produce an updated 3D model 2455. In one embodiment, the updated 3D model 2455 includes labeled and cleaned edge lines 2458.

[0366] Figure 25A schematic representation of a machine in an example form of computing device 2500 is shown, in which a set of instructions can execute any or more methods discussed herein to cause the machine to perform. In alternative embodiments, the machine may be connected (e.g., networked) to other machines on a local area network (LAN), intranet, extranet, or Internet. For example, computing device 2500 may correspond to... Figure 1 The computing device 105 and / or computing device 106. The machine can operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), cellular phone, Globe device, server, network router, switch, or bridge, or any machine capable of executing a set of instructions (sequentially or otherwise) specifying the actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered as any collection of machines (e.g., computers) that individually or jointly execute a set (or more) of instructions to perform any or more of the methods discussed herein.

[0367] Example computing device 2500 includes processing device 2502, main memory 2504 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), static memory 2506 (e.g., flash memory, static random access memory (SRAM), etc.) and auxiliary memory (e.g., data storage device 2528), which communicate with each other via bus 2508.

[0368] Processing device 2502 represents one or more general-purpose processors such as microprocessors, central processing units, etc. More specifically, processing device 2502 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 2502 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processing device 2502 is configured to execute processing logic (instructions 2526) for performing the operations and steps discussed herein.

[0369] The computing device 2500 may also include a network interface device 2522 for communicating with the network 2564. The computing device 2500 may also include a video display unit 2510 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 2512 (e.g., a keyboard), a cursor control device 2514 (e.g., a mouse), and a signal generation device 2520 (e.g., a speaker).

[0370] Data storage device 2528 may include machine-readable storage medium (or more specifically, non-transitory computer-readable storage medium) 2524, on which one or more sets of instructions 2526 embodying any one or more methods or functions described herein are stored, such as instructions for tooth modeling logic 2550. Non-transitory storage medium refers to storage medium other than a carrier wave. During the execution of instructions 2526 by computer device 2500, instructions 2526 may also reside wholly or at least partially in main memory 2504 and / or processing device 2502, which also constitute computer-readable storage media.

[0371] The computer-readable storage medium 2524 can also be used to store tooth modeling logic 2550, which may include one or more machine learning modules and can perform the operations described above. The computer-readable storage medium 2524 can also store a software library containing methods for the tooth modeling logic 2550. Although the computer-readable storage medium 2524 is shown as a single medium in the example embodiment, the term "computer-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more sets of instructions. The term "computer-readable storage medium" should also be considered to include any medium other than a carrier wave capable of storing or encoding a set of instructions for machine execution and causing the machine to perform any one or more methods of this disclosure. Therefore, the term "computer-readable storage medium" should be considered to include, but is not limited to, solid-state memory, as well as optical and magnetic media.

[0372] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent upon reading and understanding the above description. Although embodiments of this disclosure have been described with reference to specific example embodiments, it should be recognized that this disclosure is not limited to the described embodiments but can be practiced with modifications and variations within the spirit and scope of the appended claims. Therefore, the specification and drawings should be considered illustrative and not restrictive. Consequently, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A computer-readable medium including instructions, which, when executed by a processing means, cause the processing means to perform operations including: A three-dimensional model of the tooth region is generated from scan data of the tooth region, the three-dimensional model including a representation of the tooth, wherein, The three-dimensional model includes interfering surfaces that obscure the portion of the teeth; Receive or generate an image of the tooth, wherein the image depicts the interfering surface; The image is automatically processed to generate a modified image, wherein the portion of the tooth that is occluded by the interfering surface in the image is shown in the modified image, and wherein the automatic processing of the image includes inputting data from the image into a trained machine learning model, the trained machine learning model being trained to modify images of teeth, wherein the trained machine learning model outputs data of the modified image; The three-dimensional model of the tooth is updated by replacing the portion of the three-dimensional model including the interfering surface with the modified image, wherein the portion of the tooth that was occluded in the three-dimensional model is shown in the updated three-dimensional model. Display at least one of the three-dimensional model or the updated three-dimensional model; and Receive input regarding whether to accept the updated 3D model.

2. A system comprising: Memory; as well as A processing device, operatively connected to the memory, wherein the processing device: A three-dimensional model of the tooth region is generated from scan data of the tooth region, the three-dimensional model including a representation of the tooth, wherein a portion of the three-dimensional model includes interfering surfaces that occlude portions of the tooth. Receive or generate an image of the tooth, wherein the image depicts the interfering surface and includes height information; The image is processed using a trained machine learning model to generate a modified image, the trained machine learning model being trained to modify images of teeth, wherein the trained machine learning model outputs data for the modified image, wherein the portion of the tooth occluded by the interfering surface in the original image is shown in the modified image, and wherein the modified image includes modified height information; and The three-dimensional model of the tooth is updated by replacing the portion of the three-dimensional model including the interfering surface with the modified image, wherein the portion of the tooth that was occluded in the three-dimensional model is shown in the updated three-dimensional model.

3. A method comprising: Detecting edge lines in a three-dimensional model of the prepared tooth from one or more images, wherein each of the one or more images includes a height map; Determine the quality fraction of each of the multiple segments of the edge line; Determine whether any one of the plurality of segments of the edge line has a quality score below a quality threshold; and In response to determining that a segment of the edge line has a quality score below the quality threshold, the three-dimensional model of the prepared tooth is updated by replacing the portion of the three-dimensional model associated with the segment of the edge line with image data from a new image.

4. A method comprising: A three-dimensional model of the tooth region is generated from scan data of the tooth region, the three-dimensional model including a representation of the tooth, wherein a portion of the three-dimensional model includes interfering surfaces that occlude portions of the tooth. Receive or generate an image of the tooth, the image including a height map, wherein the image depicts the interfering surface; The image is processed to generate a modified image, wherein the modified image includes a modified height map, and wherein the portion of the tooth that was occluded by the interfering surface in the original image is shown in the modified image; and The 3D model of the tooth region is updated by replacing the portion of the 3D model including the interference surface with the modified image including the modified height map, wherein the portion of the tooth that is occluded in the 3D model is shown in the updated 3D model.

5. A method comprising: Receive scan data, the scan data including multiple images of at least a first tooth region and a second tooth region, wherein each of the multiple images includes a timestamp and a height map; A first subset of the plurality of images is determined for use in the first tooth site, wherein the first subset is determined at least in part based on: a) timestamps of the images in the first subset, and b) geometric data of the images in the first subset; A second subset of the plurality of images is determined for use in the second tooth region, wherein the second subset is determined at least in part based on: a) timestamps of the images in the second subset, and b) geometric data of the images in the second subset; and Generate a three-dimensional model of at least a portion of a dental arch, the three-dimensional model including a representation of a first tooth region generated using a first subset and a representation of a second tooth region generated using a second subset.

6. A method comprising: While the intraoral scanning application is in the first scanning mode, the first multiple intraoral scans of the dental arch are received; The first plurality of intraoral scans are processed using one or more algorithms configured to determine the three-dimensional surface of a static tooth site while the intraoral scanning application is in the first scanning mode. The processing device determines that a partial shrinkage scan of the first prepared tooth will be performed or has already been performed, wherein the partial shrinkage scan includes an intraoral scan of the prepared tooth that has not yet been filled with retraction cord; Activate partial intraoral scanning mode; Receive a second or more intraoral scans; and The second plurality of intraoral scans are processed using one or more algorithms configured to determine the three-dimensional surface of a non-static tooth region, including collapsed gingiva, while the intraoral scan application is in the partially contracted intraoral scan mode.

7. A method comprising: Determine whether a partial shrinkage scan of the first prepared tooth will be performed or has been performed, wherein the partial shrinkage scan includes an intraoral scan of the prepared tooth that has not yet been filled with retraction cord; Receives multiple intraoral scans generated by an intraoral scanner; Based on the partially contracted intraoral scanning pattern, the processing device uses a stitching algorithm to process the multiple intraoral scans together, the processing including: Determine conflict surfaces of pairs of intraoral scans from the plurality of intraoral scans, wherein, with respect to the conflict surface, the first intraoral scan of the pair of intraoral scans has a first distance from the probe of the intraoral scanner, and the second intraoral scan of the pair of intraoral scans has a second distance from the probe. It is determined that the second distance is greater than the first distance; Determine whether the difference between the first distance from the probe and the second distance from the probe is greater than a difference threshold; and In response to determining that the difference is greater than the difference threshold, the representation of the conflict surface from the first intra-port scan is discarded, wherein the representation of the conflict surface from the second intra-port scan is used for the conflict surface.

8. A method comprising: After the gingival retraction tool has temporarily retracted a first portion of the gingiva around the prepared tooth to partially expose the finish line, a first intraoral scan of the prepared tooth is received, wherein the first portion of the finish line is exposed in the first intraoral scan. After receiving the first intraoral scan, a second intraoral scan of the prepared tooth is received, wherein the first portion of the completion line is obscured by the first portion of the gingiva in the second intraoral scan; The processing device compares the first intraoral scan with the second intraoral scan; Between the first intraoral scan and the second intraoral scan, identify the conflicting surfaces at the region corresponding to the first portion of the finish line of the prepared tooth; Discard or mark data from the region of the prepared tooth obtained from the second intraoral scan; and The first intraoral scan and the second intraoral scan are stitched together to generate a virtual model of the prepared tooth, wherein data from the region of the prepared tooth in the first intraoral scan is used to generate the virtual model of the prepared tooth, and wherein data from the region of the prepared tooth in the second intraoral scan is not used to generate the virtual model of the prepared tooth.

9. A method comprising: Receive intraoral scan data of the prepared teeth; The first surface of the prepared tooth is generated using the intraoral scan data and one or more algorithms, wherein the first surface depicts the prepared tooth without gingival surface information. The second surface of the prepared tooth is generated using the intraoral scan data and one or more algorithms, wherein the second surface depicts the prepared tooth with the gingival surface information. Select at least one of the first surface or the second surface; and Display at least one of the selected first surface or second surface.

10. A method comprising: After the gingival retraction tool has temporarily retracted a first portion of the gingiva around the prepared tooth to partially expose the finish line, a first intraoral scan of the prepared tooth is received, wherein a first portion of the finish line is exposed in the first intraoral scan, and wherein a second portion of the finish line is obscured by the gingiva in the first intraoral scan. After the gingival retraction tool has temporarily constricted a second portion of the gingiva surrounding the prepared tooth to partially expose the finishing line, a second intraoral scan of the prepared tooth is received, wherein the second portion of the finishing line is exposed in the second intraoral scan, and wherein the first portion of the finishing line is obscured by the gingiva in the second intraoral scan; and A processing device generates a virtual model of the prepared tooth using a first intraoral scan and a second intraoral scan, wherein the first intraoral scan is used to generate a first region of the virtual model, the first region representing a first portion of the finish line, and wherein the second intraoral scan is used to generate a second region of the virtual model, the second region representing a second portion of the finish line.

11. A method comprising: Receive first intraoral scan data of the prepared tooth, which was generated after removing the retraction line around the prepared tooth to expose the completion line; The first surface of the prepared tooth is generated using the first intraoral scan data and one or more algorithms; For the portion of the first surface depicting a part of the prepared tooth, it is determined that the finish line is obscured by gingival tissue; Using at least one of a) the first intraoral scan data or the second intraoral scan data and b) a second or more algorithms, a second surface of the portion of the prepared tooth that is obscured by the finish line is generated; as well as Replace the portion of the first surface with the second surface.

12. A non-volatile computer-readable medium including instructions that, when executed by a processing means, cause the processing means to perform operations including: Detecting edge lines in a 3D model of the prepared tooth from one or more images, wherein, The edge line is detected at least in part by applying a cost function to multiple points on the 3D model, wherein the cost function selects points that together form a profile with the lowest combined cost, and wherein, for each point, the cost of the point is related to the inverse of the probability that the point draws the edge line; Determine the quality fraction of each of the multiple segments of the edge line; Determining that a segment of the edge line from among the plurality of segments of the edge line has a quality score below a quality threshold, wherein the segments of the edge line are generated based on a first image set; and In response to determining that the segment of the edge line has a quality score below the quality threshold, the three-dimensional model of the prepared tooth is updated by replacing the portion of the three-dimensional model associated with the segment of the edge line with image data from at least one of a new image or a second image set, which is different from the first image set.

13. A system comprising: Memory; as well as A processing device, operatively connected to the memory, is configured to: Edge lines in a 3D model of a prepared tooth are detected from one or more images of the prepared tooth, wherein the edge lines are detected at least in part by applying a cost function to a plurality of points on the 3D model, wherein the cost function selects points that together form a profile with the lowest combined cost, wherein for each point, the cost of the point is related to the inverse of the probability that the point depicts the edge line; Determine the quality fraction of each of the multiple segments of the edge line; Determining that a segment of the edge line from among the plurality of segments of the edge line has a quality score below a quality threshold, wherein the segments of the edge line are generated based on a first image set; and In response to determining that the segment of the edge line has a quality score below the quality threshold, the three-dimensional model of the prepared tooth is updated by replacing the portion of the three-dimensional model associated with the segment of the edge line with image data from at least one of a new image or a second image set, which is different from the first image set.

14. A non-volatile computer-readable medium including instructions that, when executed by a processing means, cause the processing means to perform operations including: Detecting edge lines in a 3D model of the prepared tooth from one or more images, wherein, The three-dimensional model is generated from scan data comprising multiple blended images, wherein each of the multiple blended images is based on a combination of multiple individual intraoral images generated by an intraoral scanner; Determine the quality fraction of each of the multiple segments of the edge line; Determining that a segment of the edge line from among the plurality of segments of the edge line has a quality score below a quality threshold, wherein the segments of the edge line are generated based on a first image set; and In response to determining that the segment of the edge line has a quality score lower than the quality threshold, the following operations are performed: Access the plurality of individual intraoral images used to generate at least some of the plurality of mixed images; A subset of the plurality of individual intraoral images that identify the segment depicting the edge line; A specific image is determined from the subset of the plurality of individual intraoral images, the specific image including a representation of the segment of the edge line, the representation being most similar to the fabricated version of the segment of the edge line; and The three-dimensional model of the prepared tooth is updated by replacing the portion of the three-dimensional model associated with the segment of the edge line with image data from the specific image.

15. A system comprising: An intraoral scanner is configured to generate scan data including multiple images of at least a first tooth region and a second tooth region, wherein each of the multiple images includes a timestamp and a height map; and The computing device is configured as follows: Receive the scan data; A first subset of the plurality of images is determined for use in the first tooth site, wherein the first subset is determined at least in part based on: a) timestamps of the images in the first subset, and b) geometric data of the images in the first subset; A second subset of the plurality of images is determined for use in the second tooth region, wherein the second subset is determined at least in part based on: a) timestamps of the images in the second subset, and b) geometric data of the images in the second subset; and Generate a three-dimensional model of at least a portion of a dental arch, the three-dimensional model including a representation of a first tooth region generated using a first subset and a representation of a second tooth region generated using a second subset.

16. A system comprising: Memory; as well as A processing device, operatively connected to the memory, wherein the processing device: Receive one or more intraoral scans generated in the first moment from a portion of the dental site; The first three-dimensional (3D) surface of the portion of the tooth region is determined based on the one or more intraoral scans; Receive the second three-dimensional (3D) surface of the tooth region determined from second intraoral scan data generated at a second time earlier than the first time. Based on the comparison between the first 3D surface and the second 3D surface, conflicting surfaces between regions of the first 3D surface and corresponding regions of the second 3D surface are determined. as well as The second 3D surface is updated by replacing the data of the region of the second 3D surface with the data of the region of the first 3D surface.

17. A system comprising: Intraoral scanner; Memory; as well as A processing device, operatively connected to the memory, wherein the processing device: The first intraoral scan of the dental area is received, and the first intraoral scan has been generated by the intraoral scanner in the first instant. Receive a second intraoral scan of at least a portion of the tooth region, the second intraoral scan having been generated by the intraoral scanner at a second time; Based on the comparison between the second intraoral scan and the first intraoral scan, conflict surfaces between the portion of the tooth region from the first intraoral scan and the portion of the tooth region from the second intraoral scan are determined. as well as A three-dimensional (3D) surface is generated at least in part based on the first intraoral scan and the second intraoral scan, wherein data from the portion of the second intraoral scan of the tooth region is used to generate the 3D surface, and wherein data from the portion of the first intraoral scan of the tooth region is not used to generate the 3D surface.

18. A method comprising: The processing device receives intraoral scan data of the prepared tooth, wherein the intraoral scan data is acquired during the process of exposing the subgingival portion of the prepared tooth using a gingival retraction tool; The processing device determines a conflict surface associated with the prepared tooth based on the intraoral scan data, wherein the conflict surface corresponds to a region in which a first intraoral scan of the intraoral scan data depicts the subgingival portion of the prepared tooth, and a second intraoral scan of the intraoral scan data depicts the gingiva covering the subgingival portion of the prepared tooth. The processing device determines the depth value of the point on the collision surface; and The processing device processes data of the conflict surface based on the depth value to determine the surface of the subgingival portion of the prepared tooth.

19. A system comprising: Intraoral scanner; Gingival retraction tools are configured to expose the margin line of the prepared teeth; as well as The computing device is configured as follows: Receive intraoral scan data of the prepared tooth, the intraoral scan data being acquired by the intraoral scanner during the process of exposing the subgingival portion of the prepared tooth using the gingival retraction tool; Based on the intraoral scan data, a conflict surface associated with the prepared tooth is determined, wherein the conflict surface corresponds to a region in which a first intraoral scan of the intraoral scan data depicts the subgingival portion of the prepared tooth, and a second intraoral scan of the intraoral scan data depicts the gingiva covering the subgingival portion of the prepared tooth. Determine the depth value of the point on the conflict surface; and Data on the conflict surface is processed based on the depth value to determine the surface of the subgingival portion of the prepared tooth.

20. A method comprising: The processing device receives intraoral scan data of the prepared teeth, including the subgingival margin, during the intraoral scanning operation. The processing device uses a model to process the intraoral scan data to identify the probability of three-dimensional (3D) points on the edge line; The edge line is determined based on the probability of the 3D point on the edge line; Generate and display an indication of the edge line; as well as The indication based on the edge line provides guidance regarding the intraoral scanning operation.

21. A system comprising: An intraoral scanner is configured to generate intraoral scan data of the prepared tooth, including the subgingival margin line; as well as The computing device is configured as follows: The intraoral scan data is processed using a model to identify the probability of three-dimensional (3D) points on the edge line; The edge line is determined based on the probability of the 3D point on the edge line; Generate and display an indication of the edge line; as well as The indication based on the edge line provides guidance regarding intraoral scanning procedures.

Images