Progressive surface quality feedback during intraoral scanning

Abstract

Various embodiments relate to techniques for assisting a user of an intraoral scanner during an intraoral scan. In one embodiment, a method includes receiving an intraoral scan of a dental site, generating a 3D surface of the dental site based on the intraoral scan, determining a surface quality score for a region of the 3D surface, determining that the surface quality score for the region fails to satisfy one or more criteria, and outputting a notification for at least one of stopping generation of a new intraoral scan of the region or continuing to a next region.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the dental field, and more specifically to providing a visual graphical user interface during intraoral scanning. Background Technology

[0002] In oral prosthodontic procedures designed to implant dental prostheses in the mouth, in many cases, the tooth position to which the prosthesis will be implanted must be accurately measured and carefully studied in order to correctly design and determine the dimensions of the prosthesis, such as a crown, denture, or bridge, so that it fits properly in place. For example, a good fit allows for the proper transmission of mechanical stress between the prosthesis and the jaw and prevents gingival infection through the interface between the prosthesis and the tooth position.

[0003] Some surgeries also require the fabrication of removable prostheses to replace one or more missing teeth, such as partial or full dentures. In these cases, it is necessary to accurately reproduce the surface contours of the area where the teeth are missing so that the resulting prosthesis can fit the edentulous area under uniform pressure applied to the soft tissue.

[0004] In some practices, dentists prepare the tooth position and construct a positive physical model of it using known methods. Alternatively, the tooth position can be scanned to provide 3D data. In either case, either a virtual or physical model of the tooth position is sent to a dental laboratory, which manufactures the restoration based on the model. However, if the model is flawed or ambiguous in some aspects, or if the preparation is not optimally configured for the receiving restoration, the restoration design may not be ideal. For example, if the insertion path implied by the preparation for a veneer crown would cause the restoration to collide with adjacent teeth, the crown geometry must be altered to avoid collisions, which may result in an undesirable crown design. Furthermore, if the preparation area containing the termination line is not defined, the termination line may not be correctly determined, and therefore the lower edge of the crown may not be designed correctly. In fact, in some cases, the model is discarded, and the dentist rescans the tooth position or redoes the preparation so that a suitable restoration can be produced.

[0005] In orthodontic surgery, providing one or two models of the jaw can be important. If the orthodontic procedure is designed virtually, a virtual model of the mouth is also beneficial. This virtual model can be obtained by directly scanning the mouth, or by producing a physical model of the dentition and then scanning that model using a suitable scanner.

[0006] Therefore, obtaining a three-dimensional (3D) model of the tooth positions in the oral cavity is the initial procedure performed in both prosthodontic and orthodontic surgeries. When the 3D model is a virtual model, the more complete and accurate the scan of the tooth positions, the higher the quality of the virtual model, and thus the greater the ability to design the optimal restoration or (multiple) orthodontic appliances.

[0007] Some intraoral scanning systems provide bi-state user feedback on the adequacy of the 3D surface / 3D model. The bi-state feedback indicates whether the 3D surface is sufficient or insufficient. However, the user performing the scan does not know whether they are making progress toward adequacy; they can only guess whether they are making progress. Summary of the Invention

[0008] In a first implementation, a method includes: receiving multiple intraoral scans of a tooth position from an intraoral scanner during an intraoral scanning session; generating a three-dimensional (3D) surface of the tooth position based on the multiple intraoral scans; determining a first surface quality fraction for a first region of the 3D surface; outputting a first view of the 3D surface to a display, wherein the first region of the 3D surface is displayed using a first visualization associated with the first surface quality fraction; receiving one or more additional intraoral scans of the tooth position during the intraoral scanning session; updating the 3D surface based on the one or more additional intraoral scans; determining a new surface quality fraction for the first region of the updated 3D surface; and outputting a first view of the updated 3D surface to the display, wherein the first region of the updated 3D surface is shown using a second visualization associated with the new surface quality fraction.

[0009] The second implementation can further extend the first implementation. In the second implementation, the method further includes: inputting a first region of the 3D surface or at least one of a plurality of intraoral scans associated with the first region into a trained machine learning model, wherein the trained machine learning model outputs a first surface quality score.

[0010] The third implementation can further extend the first or second implementation. In the third implementation, multiple intraoral scans are generated by projecting structured light, including multiple features, onto the tooth position and capturing multiple features on the tooth position, and wherein the first surface quality fraction is determined at least in part based on the number of multiple features associated with a first region of the 3D surface.

[0011] The fourth implementation can further extend the third implementation. In the fourth implementation, multiple features include multiple light spots.

[0012] The fifth implementation can further extend the first implementation to the fourth implementation. In the fifth implementation, the 3D surface is generated based on multiple points from multiple intraoral scans, and the method further includes: determining a quality score for each of the multiple points; determining a subset of the multiple points associated with a first region; and determining a first surface quality score for the first region based on the quality scores of the subset of multiple points and the number of subsets of multiple points.

[0013] The sixth implementation can further extend the fifth implementation. In the sixth implementation, the quality score of one of the multiple points is calculated based at least in part on the following: a) the distance between the point and at least one of the first or second cameras of the intraoral scanner that captures the point when generating one of the multiple intraoral scans and b) the distance between the first and second cameras.

[0014] The seventh implementation can further extend the fifth or sixth implementation. In the seventh implementation, the quality score of one of the multiple points is calculated based at least in part on: a) the distance between the point and at least one of the cameras of the intraoral scanner that captures the point when generating one of the multiple intraoral scans; and b) the distance between the camera and the structured light projector of the intraoral scanner that projects structured light onto the point.

[0015] The eighth implementation can further extend the fifth to the seventh implementation. In the eighth implementation, the intraoral scanner includes multiple cameras, and the quality score of one of the multiple points is calculated at least in part based on the number of cameras that captured that point when generating one of the multiple intraoral scans.

[0016] The ninth implementation can further extend the fifth implementation to the eighth implementation. In the ninth implementation, one of the multiple points includes a point projected by a structured light projector of an intraoral scanner, and the quality fraction of that point is at least partially based on the spot size of that point.

[0017] The tenth implementation can further extend the fifth to the ninth implementations. In the tenth implementation, the quality score of one of the multiple points is calculated at least in part based on the angle between the normal of the 3D surface at that point and the imaging axis of the intraoral scanner.

[0018] The eleventh implementation can further extend the fifth to tenth implementations. In the eleventh implementation, the method further includes: determining the material type of the tooth at one of a plurality of points, wherein the mass fraction at that point is calculated at least in part based on the material type of the tooth at that point.

[0019] The twelfth implementation can further extend the fifth to eleventh implementations. In the twelfth implementation, the quality score of one of the multiple points is calculated based on at least one of the following: a) the distance between the point and a first or second camera of the intraoral scanner that captures the point when generating one of the multiple intraoral scans, and b) the distance between the first and second cameras; a) the distance between the point and the camera of the intraoral scanner that captures the point when generating the intraoral scan, and b) the distance between the camera and the structured light projector of the intraoral scanner that projects structured light onto the point; the number of cameras of the intraoral scanner that capture the point when generating the intraoral scan; the spot size associated with the point; the angle between the normal of the 3D surface at the point and the imaging axis of the camera that captures the point when generating the intraoral scan; or the material type of the tooth at the point.

[0020] The thirteenth implementation can further extend the first implementation to the twelfth implementation. In the thirteenth implementation, the first visualization includes a first color, and the second visualization includes a second color.

[0021] The fourteenth implementation can further extend the first implementation to the thirteenth implementation. In the fourteenth implementation, the first visualization includes a first level of transparency, and the second visualization includes a second level of transparency.

[0022] The fifteenth implementation can further extend the fourteenth implementation. In the fifteenth embodiment, the method further includes: outputting a background to a display, wherein the background appears behind a 3D surface such that the background is visible through a first transparency level and a second transparency level.

[0023] The sixteenth implementation can further extend the fourteenth or fifteenth implementation. In the sixteenth implementation, the new surface quality fraction is greater than the first surface quality fraction, and the second transparency level corresponding to the new surface quality fraction is lower than the first transparency level corresponding to the first surface quality fraction.

[0024] The seventeenth implementation can further extend the first implementation to the sixteenth implementation. In the seventeenth implementation, the first visualization includes a first flicker rate, and the second visualization includes a second flicker rate.

[0025] The eighteenth implementation can be further implemented as the seventeenth. In the eighteenth implementation, the first flicker rate and the second flicker rate include flickering of at least one of the color or transparency levels.

[0026] The nineteenth implementation can further extend the first implementation to the eighteenth implementation. In the nineteenth implementation, the first visualization includes an arrow pointing to the first region, and the second visualization includes an updated arrow pointing to the second region.

[0027] The twentieth implementation can further extend the nineteenth implementation. In the twentieth implementation, the new surface quality fraction is greater than the first surface quality fraction, and the updated arrow includes at least one of a shorter length, a smaller thickness, or a lower scintillation rate than the arrow.

[0028] The twenty-first implementation can further extend the first implementation to the twentyth implementation. In the twenty-first embodiment, the method further includes: outputting a second view of the 3D surface to a display adjacent to the first view, wherein the second view of the 3D surface lacks visualization corresponding to the surface quality fraction.

[0029] The twenty-second implementation can further extend the twenty-first implementation. In the twenty-second embodiment, the second view of the 3D surface includes a color view, a monochrome view, or a grayscale view displaying the captured color of the tooth position.

[0030] The twenty-third implementation can further extend the first implementation to the twenty-second implementation. In the twenty-third embodiment, the method further includes: determining a third surface quality fraction of a second region of the 3D surface; wherein the second region of the 3D surface is shown using a third visualization associated with the third surface quality fraction.

[0031] The twenty-fourth implementation can further extend the twenty-third implementation. In the twenty-fourth embodiment, the method further includes: determining a first rating gauge for a first region; determining a first visualization corresponding to a first surface quality score and a second visualization corresponding to a new surface quality score based on the first rating gauge; determining a second rating gauge for a second region; and determining a third visualization corresponding to a third surface quality score based on the second rating gauge.

[0032] The twenty-fifth implementation can further extend the twenty-fourth implementation. In the twenty-fifth implementation, the method further includes: classifying a first region as belonging to a first dental object category, wherein a first rating gauge is associated with the first dental object category; and classifying a second region as belonging to a second dental object category, wherein a second rating gauge is associated with the second dental object category.

[0033] The twenty-sixth implementation can further extend the twenty-fifth implementation. In the twenty-sixth implementation, the first dental object category includes prepared teeth, transgingival contours, or margins, while the second dental object category includes standard teeth.

[0034] The twenty-seventh implementation can further extend the fourth implementation to the twenty-sixth implementation. In the twenty-sixth implementation, the first rating gauge associates the first surface quality score with the first visualization, and the second rating gauge associates the second surface quality score with the first visualization.

[0035] The twenty-eighth implementation can further extend the first implementation to the twenty-seventh implementation. In the twenty-seventh implementation, the method further includes: determining whether to perform restorative or orthodontic treatment on the tooth position; and selecting a rating rub that associates the surface quality score with visualization based on whether restorative or orthodontic treatment is chosen.

[0036] The twenty-ninth implementation can further extend the first implementation to the twenty-eighth implementation. In the twenty-ninth implementation, the method further includes: receiving a plurality of two-dimensional (2D) images of a tooth position; and determining the number of a plurality of 2D images depicting a first region; wherein a first surface quality fraction of the first region is based at least in part on the number of a plurality of 2D images depicting the first region.

[0037] The thirtieth implementation can further extend the twenty-ninth implementation. In the thirtieth implementation, the method further includes: for each of a plurality of 2D images, determining the angle between the normal of the 3D surface at the first region and the imaging axis of the intraoral scanner; and determining the number of a plurality of 2D images in which the angle between the normal of the 3D surface at the first region and the imaging axis is within an angle threshold; wherein the first surface quality fraction of the first region is at least partially based on the number of a plurality of 2D images depicting the first region and in which the angle between the normal of the 3D surface at the first region and the imaging axis is within an angle threshold.

[0038] The thirty-first implementation can further extend the twenty-ninth or thirtieth implementation. In the thirty-first implementation, the multiple 2D images include at least one of multiple color images or multiple near-infrared (NIR) images.

[0039] The thirty-second implementation can further extend the first implementation to the thirty-first implementation. In the thirty-second implementation, the method further includes: determining a first roughness and a first resolution associated with a first region of the 3D surface; wherein the first surface quality fraction is determined at least in part based on the first roughness and the first resolution.

[0040] The thirty-third implementation can further extend the thirty-second implementation. In the thirty-third implementation, the first resolution is determined at least in part based on the number of points captured in the first region; and the first roughness is determined based on the distance between the points of one or more intraoral scans from the first region used to generate the 3D surface and the nearest point on the 3D surface.

[0041] The thirty-fourth implementation can further extend the thirty-third implementation. In the thirty-fourth implementation, the method further includes: determining the standard deviation of the distance between points from one or more intraoral scans and the nearest point on the 3D surface, wherein the first roughness is determined at least in part based on the standard deviation.

[0042] The thirty-fifth implementation can be a further extension of the thirty-fourth implementation. In the thirty-fifth implementation, an increase in the standard deviation corresponds to an increase in the first roughness.

[0043] The thirty-sixth implementation can further extend the first implementation to the thirty-fifth implementation. In the thirty-sixth implementation, the method further includes: determining that the new surface quality fraction of the first region fails to improve within a threshold time period, and that the new surface scan quality fraction is below the surface quality threshold; determining one or more scan suggestions that, if implemented, would result in an improvement in the new surface quality fraction of the first region; and outputting the one or more scan suggestions to a display.

[0044] The thirty-seventh implementation can further extend the thirty-sixth implementation. In the thirty-seventh implementation, the method further includes: determining a first region associated with the distal molars of the scanned patient, wherein one or more scanning suggestions include suggestions for the patient to move the jaw to the right or left.

[0045] The thirty-eighth implementation can further extend the thirty-sixth or thirty-seventh implementation. In the thirty-eighth implementation, the method further includes: determining that a first region is at least partially obscured by soft tissue, wherein one or more scanning suggestions include suggestions from a physician to roll an intraoral scanner around a prescribed axis and / or along a prescribed direction to move the soft tissue.

[0046] The thirty-ninth implementation can further extend the thirty-sixth to the thirty-eighth implementations. In the thirty-ninth implementation, the method further includes: determining a first region associated with the anterior teeth of the scanned patient, and the first region being at least partially obscured by the patient's lips, wherein one or more scanning suggestions include instructions for at least one of: a) pulling the patient's lips away from the anterior teeth or b) sliding an intraoral scanner between the anterior teeth and the patient's lips.

[0047] The fortieth implementation can be further extended to the thirty-sixth to thirty-ninth implementations. In the fortieth embodiment, one or more scanning suggestions include paths followed by the intraoral scanner to capture other intraoral scans.

[0048] The forty-first implementation can be further extended from the thirty-sixth to the fortieth implementation. In the forty-first implementation, one or more scanning suggestions include placing an intraoral scanner to capture at least one of the target locations or target orientations of other intraoral scans.

[0049] The forty-second implementation can be further extended to the forty-first implementation. In the forty-second implementation, the method further includes: receiving a first two-dimensional (2D) image of a tooth position corresponding to the current field of view of the intraoral scanner at a first time; outputting the first 2D image to a display; and outputting a first overlay to the display, the first overlay being on the first 2D image, the first overlay including a first shape located at approximately the center of the 2D image and a second shape located at a target position.

[0050] The forty-third implementation can be further extended to the forty-second implementation. In the forty-third implementation, the method further includes: receiving a second two-dimensional (2D) image of a tooth position corresponding to the current field of view of the intraoral scanner at a second time after the intraoral scanner has been repositioned to move toward the target position; outputting the second 2D image to a display; and outputting a second overlay on the second 2D image to the display, the second overlay including a first shape located at approximately the center of the 2D image and a second shape located at the target position, wherein the first shape overlaps with the second shape in the second overlay.

[0051] The forty-fourth implementation can be further extended to the forty-second or forty-third implementation. In the forty-third implementation, the first shape includes a crosshair, a circle, a ring, or a square.

[0052] The forty-fifth implementation can be further extended from the forty-first to the forty-fourth implementations. In the forty-fifth implementation, the method further includes: outputting a first overlay on a 3D surface to a display, the first overlay including a shape located at the target position and indicating the target orientation.

[0053] The forty-sixth implementation can further extend the thirty-sixth to the forty-fifth implementations. In the forty-sixth implementation, the method further includes: determining the current position of the intraoral scanner in the patient's mouth, wherein one or more scanning suggestions are determined at least in part based on the current position of the intraoral scanner.

[0054] The forty-seventh implementation can further extend the thirty-sixth to the forty-sixth implementation. In the forty-seventh implementation, one or more scanning suggestions include animations showing how to move the intraoral scanner to achieve the desired result.

[0055] The forty-eighth implementation can further extend the thirty-sixth to forty-seventh implementations. In the forty-eighth implementation, the method further includes: determining that the first region is at least partially obscured by the patient's tongue, wherein one or more scanning suggestions include a suggestion that the patient moves the tongue up, down, to the right, or to the left.

[0056] The forty-ninth implementation can further extend the first implementation to the forty-eighth implementation. In the forty-ninth implementation, the method further includes: determining that the new surface quality score of the first region fails to improve within a threshold time period and that the new surface scan quality score is lower than the surface quality threshold; and increasing the scaling setting of the first region.

[0057] The fiftieth implementation can further extend the forty-ninth implementation. In the fiftieth implementation, the method further includes: determining that the intraoral scanner is focused on the first region before increasing the scaling setting of the first region.

[0058] The fifty-first implementation can be further extended from the first implementation to the fiftieth implementation. In the fifty-first implementation, the method further includes: determining that the intraoral scanner remains focused on the first region within a threshold time period; and increasing the scaling setting of the first region.

[0059] The fifty-second implementation can further extend the fifty-first implementation. In the fifty-second implementation, the 3D surface includes all dental arches that have been scanned so far, wherein the first view of the 3D surface includes the entire 3D surface before the scaling setting of the first region is increased, and wherein the first view of the 3D surface includes only a portion of the 3D surface after the scaling setting of the first region is increased.

[0060] The fifty-third implementation can be further extended to the fifty-first or fifty-second implementation. In the fifty-third implementation, the first view of the 3D surface includes at least one of an interlocking view, a bird's-eye view, a far-to-near view, or a near-to-far view.

[0061] The fifty-fourth implementation can further extend the first to the fifty-third implementations. In the fifty-fourth implementation, the intraoral scanner includes a plurality of cameras, each camera having at least one of different positions or orientations within the intraoral scanner, and the method further includes: receiving a plurality of two-dimensional (2D) images, each 2D image generated by a different camera among the plurality of cameras; determining one of the plurality of 2D images associated with improved intraoral scan quality; and outputting the determined 2D image to a display, wherein in response to the output of the determined 2D image, a physician using the intraoral scanner will reposition the intraoral scanner in a manner that results in the improved intraoral scan quality.

[0062] The fifty-fifth implementation can further extend the first implementation to the fifty-fourth implementation. In the fifty-fifth implementation, the method further includes: determining that the new surface quality fraction of the first region fails to improve within a threshold time period; and determining that additional intraoral scanning of the first region does not improve the surface quality fraction of the region.

[0063] The fifty-sixth implementation can further extend the fifty-fifth implementation. In the fifty-sixth implementation, the method further includes: determining that the additional surface quality fraction of one or more additional regions adjacent to the first region is equal to or higher than a surface quality threshold, wherein the determination is made in response to determining that the additional surface quality fraction of one or more additional regions adjacent to the first region is greater than or equal to the surface quality threshold.

[0064] The fifty-seventh implementation can further extend the fifty-sixth implementation. In the fifty-seventh implementation, one or more additional regions include multiple additional regions surrounding the first region.

[0065] The fifty-eighth implementation can further extend the fifty-fifth to fifty-seventh implementations. In the fifty-eighth implementation, the method further includes: generating a notification to stop generating a new intraoral scan of the first region or to continue at least one of the next regions.

[0066] The fifty-ninth implementation can be further extended to the fifty-fifth to fifty-eighth implementations. In the fifty-ninth implementation, the first region includes at least one of the following region types: an orifice having at least one of a bottom or one or more sidewalls that cannot be imaged by an intraoral scanner; a surface that cannot be imaged by an intraoral scanner due to an achievable angle of the intraoral scanner relative to the orifice being too steep; a surface covered by at least one of blood or saliva; or a surface covered by collapsed gingiva.

[0067] The sixtieth implementation can be further extended to the fifty-ninth implementation. In the sixtieth implementation, the method also includes: determining the region type of the first region; and generating a notification that identifies the region type.

[0068] The sixty-first implementation can be further extended from the fifty-fifth to the sixtieth implementation. In the sixty-first implementation, at least one of the following is performed using a trained machine learning model: a) determining that the new surface quality score of the first region fails to improve within a threshold time amount or b) determining that an additional intraoral scan of the first region does not improve the surface quality score of the region.

[0069] The sixty-second implementation can further extend the fifty-fifth to the sixty-first implementations. In the sixty-second implementation, the method further includes: marking the first region as a void on the updated 3D surface.

[0070] In the sixty-third implementation, a method includes: receiving multiple intraoral scans of a tooth position from an intraoral scanner during an intraoral scanning session; generating a three-dimensional (3D) surface of the tooth position based on the multiple intraoral scans; identifying areas of the tooth position that are difficult for the user of the intraoral scanner to scan or areas of the tooth position that are difficult to scan; and performing one or more actions to assist the user in scanning the areas of the tooth position.

[0071] The sixty-fourth implementation can further extend the sixty-third implementation. In the sixty-fourth implementation, determining the region that is difficult for the user to scan or the region that is difficult to scan the tooth position includes: determining the surface quality fraction of the region of the 3D surface corresponding to the region of the tooth position; and determining that the surface quality fraction of the region of the 3D surface is lower than the surface quality threshold.

[0072] The sixty-fifth implementation can be further extended to the sixty-third to sixty-fourth implementations. In the sixty-fifth implementation, determining the area that is difficult for the user to scan includes: determining that the intraoral scanner maintains focus on the area of ​​the tooth position within a threshold time, while the surface quality of the area of ​​the 3D model corresponding to the tooth position does not reach the threshold improvement amount within the threshold time.

[0073] The sixty-sixth implementation can be further extended from the sixty-third to the sixty-fifth implementations. In the sixty-sixth implementation, performing one or more actions to assist the user in scanning the tooth position includes: increasing the scaling setting of the area of ​​the 3D surface corresponding to the tooth position; and outputting a view of the 3D surface increased by the scaling setting to a display.

[0074] The sixty-seventh implementation can further extend the sixty-sixth implementation. In the sixty-seventh implementation, the 3D surface includes all dental arches that have been scanned so far, wherein before increasing the scaling setting of the region, the view of the 3D surface includes the entire 3D surface without increasing the scaling setting, and wherein after increasing the scaling setting of the region of the 3D surface, the view of the 3D surface includes only a portion of the 3D surface.

[0075] The sixty-eighth implementation can be further extended to the sixty-sixth to the sixty-seventh implementations. In the sixty-eighth implementation, the view of the 3D surface includes at least one of an interlocking view, a bird's-eye view, a far-to-near view, or a near-to-far view.

[0076] The sixty-ninth implementation can be further extended from the sixty-third to the sixty-eighth implementations. In the sixty-ninth implementation, performing one or more actions to assist the user in scanning a tooth position includes: determining one or more scanning suggestions that, if implemented, would result in an improvement in the surface quality fraction of the region; and outputting one or more scanning suggestions to a display.

[0077] The seventieth implementation can be further extended to the sixty-ninth implementation. In the seventieth implementation, the method further includes: determining a region associated with the distal molars of the patient being scanned, wherein one or more scanning suggestions include suggestions for the patient to move the jaw to the right or left.

[0078] The seventy-first implementation can be further extended from the sixty-ninth to the seventieth implementation. In the seventy-first implementation, the method further includes: identifying an area at least partially obscured by soft tissue, wherein one or more scanning suggestions include suggestions from a physician to roll an intraoral scanner around a prescribed axis and / or along a prescribed direction to move the soft tissue.

[0079] The seventy-second implementation can further extend the sixty-ninth to seventy-first implementations. In the seventy-second implementation, the method further includes: determining a region associated with the anterior teeth of the scanned patient, and the region being at least partially obscured by the patient's lips, wherein one or more scanning suggestions include instructions for at least one of: a) pulling the patient's lips away from the anterior teeth or b) sliding an intraoral scanner between the anterior teeth and the patient's lips.

[0080] The seventy-third implementation can further extend the sixty-ninth to seventy-second implementations. In the seventy-third embodiment, one or more scanning suggestions include paths followed by the intraoral scanner to capture other intraoral scans.

[0081] The seventy-fourth implementation can further extend the sixty-ninth to seventy-third implementations. In the seventy-fourth implementation, one or more scanning suggestions include placing an intraoral scanner to capture at least one of the target locations or target orientations of other intraoral scans.

[0082] The seventy-fifth implementation can further extend the seventy-fourth implementation. In the seventy-fifth implementation, the method further includes: receiving a first two-dimensional (2D) image of a tooth position corresponding to the current field of view of the intraoral scanner at a first time; outputting the first 2D image to a display; and outputting a first overlay on the first 2D image to the display, the first overlay including a) a first shape located at a first position of the 2D image associated with the center of the current field of view of the intraoral scanner and b) a second shape located at a target position.

[0083] The seventy-sixth implementation can further extend the seventy-fifth implementation. In the seventy-sixth implementation, the method further includes: receiving a second two-dimensional (2D) image of a tooth position corresponding to the current field of view of the intraoral scanner at a second time after the intraoral scanner has been repositioned to move toward the target position; outputting the second 2D image to a display; and outputting a second overlay on the second 2D image to the display, the second overlay including a first shape located at a first position in the 2D image and a second shape located at the target position, wherein the first shape overlaps with the second shape in the second overlay.

[0084] The seventy-seventh implementation can be further extended to the seventy-fifth or seventy-sixth implementation. In the seventy-seventh implementation, the first shape includes a crosshair, a circle, a ring, or a square.

[0085] The seventy-eighth implementation can be further extended from the seventy-fourth to the seventy-seventh implementation. In the seventy-eighth implementation, the method further includes: outputting a first overlay on a 3D surface to a display, the first overlay including a shape located at the target position and indicating the target orientation.

[0086] The seventy-ninth implementation can further extend the sixty-ninth to the seventy-eighth implementations. In the seventy-ninth implementation, the method further includes: determining the current position of the intraoral scanner in the patient's mouth, wherein one or more suggestions are determined at least in part based on the current position of the intraoral scanner.

[0087] The 80th implementation can be further extended to the 69th to 79th implementations. In the 80th implementation, one or more scanning suggestions include an animation showing how to move the intraoral scanner to achieve the target result.

[0088] The 81st implementation can be further extended from the 69th to the 80th implementation. In the 81st implementation, the method further includes: determining an area that is at least partially obscured by the patient's tongue, wherein one or more scanning suggestions include a suggestion that the patient move the tongue up, down, to the right, or to the left.

[0089] The 82nd implementation can further extend the 63rd to the 81st implementations. In the 82nd implementation, the method further includes: determining, before performing one or more actions, that the movement of the intraoral scanner has stopped in the region for at least a certain threshold amount of time.

[0090] The 83rd implementation can be further extended from the 63rd to the 82nd implementation. In the 83rd implementation, the method further includes: determining that the intraoral scanner has begun to move away from the region before performing one or more actions.

[0091] The 84th implementation can further extend the 63rd to 83rd implementations. In the 84th implementation, the method further includes: in response to receiving a scan assistance request from a user of an intraoral scanner, performing one or more actions.

[0092] The 85th implementation can be further extended to the 63rd to 84th implementations. In the 85th implementation, the method further includes: receiving an intraoral image set from an intraoral scanner, each intraoral image in the set having been generated by a different camera of the intraoral scanner; determining at least one of the current position or current orientation of the intraoral scanner relative to a 3D surface; determining at least one of the target position or target orientation of the intraoral scanner relative to the 3D surface; determining that, if selected, it may cause a user of the intraoral scanner to reposition an image in the intraoral image set of the intraoral scanner in a manner that causes at least one of the current position or current orientation of the intraoral scanner relative to the 3D surface to move toward at least one of the target position or current orientation of the intraoral scanner relative to the 3D surface; selecting the determined image; and outputting the selected image to a display.

[0093] In the 86th implementation, a method includes: receiving at least one of a plurality of intraoral scans or a plurality of two-dimensional (2D) images of a tooth position from an intraoral scanner; determining a velocity of the intraoral scanner relative to the tooth position based at least in part on the plurality of intraoral scans or the plurality of 2D images; generating a three-dimensional (3D) surface of the tooth position based on the plurality of intraoral scans; determining a scaling setting for displaying at least a portion of the 3D surface of the tooth position based on the determined velocity; and outputting a view of at least a portion of the 3D surface, wherein the at least a portion of the 3D surface has the determined scaling setting.

[0094] The 87th implementation can further extend the 86th implementation. In the 87th implementation, the method further includes: determining the current field of view of the intraoral scanner based on at least one of the most recent intraoral scan of multiple intraoral scans or the most recent 2D image of multiple 2D images; wherein a portion of the 3D surface corresponds to a portion of the tooth position within the current field of view of the intraoral scanner.

[0095] The 88th implementation can further extend the 86th to 87th implementations. In the 88th implementation, a method includes: receiving at least one of a second plurality of intraoral scans or a second plurality of 2D images of a tooth position from an intraoral scanner; determining a new speed of the intraoral scanner relative to the tooth position based at least in part on the second plurality of intraoral scans or the second plurality of 2D images; updating a 3D surface of the tooth position based on the second plurality of intraoral scans; determining a new scaling setting for displaying at least a portion of the 3D surface of the tooth position based on the new speed; and outputting a view of at least a portion of the updated 3D surface, wherein the at least portion of the updated 3D surface has the new scaling setting.

[0096] The 89th implementation can be further extended from the 86th to the 88th implementation. In the 89th implementation, the scaling setting is inversely proportional to the speed.

[0097] The 90th implementation can be further extended to the 86th to 89th implementations. In the 90th implementation, the method further includes: determining whether the speed is below a speed threshold; and in response to determining that the speed is below the speed threshold, outputting a second view of at least a portion of the 3D surface, the second view being located next to a view of at least a portion of the 3D surface, wherein the portion of the 3D surface has a first orientation in the first view and a second orientation in the second view.

[0098] The 91st implementation can be further extended from the 86th to the 90th implementation. In the 91st implementation, the method further includes: determining the resolution of the portion used for the 3D surface based on the velocity.

[0099] The 92nd implementation can be further extended from the 86th to the 91st implementation. In the 92nd implementation, determining the velocity of the intraoral scanner relative to the tooth position includes: determining the average velocity of the intraoral scanner relative to the tooth position.

[0100] The 93rd implementation can further extend the 92nd implementation. In the 93rd implementation, the average velocity is a weighted average, which weights the velocities calculated from at least one scanned from a more recent 2D image or less recent 2D image that are greater than the velocities measured from at least one scanned from a less recent 2D image or less recent 2D image.

[0101] The 94th implementation can be a further extension of the 93rd implementation. In the 94th implementation, the average velocity is calculated using an infinite impulse response filter.

[0102] The 95th implementation can further extend the 86th to 94th implementations. In the 95th implementation, determining the scaling settings based on the determined speed is performed using a lookup table that maps the speed to the scaling settings.

[0103] The 96th implementation can further extend the 86th to the 95th implementation. In the 96th implementation, determining the scaling settings based on the determined speed is performed using a function that maps the speed to the scaling settings.

[0104] The 97th implementation can be further extended from the 86th to the 96th implementation. In the 97th implementation, determining the speed of the intraoral scanner includes determining the speed of a virtual point located at a set distance from the intraoral scanner.

[0105] In a ninety-eighth implementation, a method includes: receiving multiple intraoral scans of tooth positions from an intraoral scanner during an intraoral scanning session; determining a quality score for each of a plurality of points in one or more of the multiple intraoral scans; generating a three-dimensional (3D) surface based on the plurality of points from one or more of the multiple intraoral scans; determining a first surface quality score for a first region of the 3D surface based at least in part on a) the number of points associated with a first region from the plurality of points and b) the quality score of the points associated with the first region; and outputting a view of the 3D surface to a display, wherein the first region of the 3D surface is shown using a first visualization associated with the first surface quality score.

[0106] The 99th implementation can further extend the 98th implementation. In the 99th implementation, the quality score of each point is determined before determining the depth value of each point.

[0107] The hundredth implementation can be further extended to the ninety-eighth or ninety-ninth implementation. In the hundredth implementation, the quality score of each point is determined before determining the depth value of each point.

[0108] The 101st implementation can be further extended from the 98th to the 100th implementation. In the 101st implementation, the method further includes: inputting multiple points from one or more intraoral scans from a plurality of intraoral scans into a trained machine learning model, wherein the trained machine learning model outputs a quality score for each of the multiple points.

[0109] The 102nd implementation can be further extended from the 98th implementation to the 101st implementation. In the 102nd implementation, the method further includes: determining a first roughness and a first resolution associated with a first region of the 3D surface; wherein the first surface quality fraction is determined at least in part based on the first roughness and the first resolution.

[0110] The 103rd implementation can further extend the 102nd implementation. In the 103rd implementation, the first resolution is determined at least in part based on the number of points associated with the first region; and the first roughness is determined based on the distance between the points associated with the first region and the nearest point on the 3D surface.

[0111] The 104th implementation can further extend the 103rd implementation. In the 104th implementation, the method further includes: determining the standard deviation of the distance between the point associated with the first region and the nearest point on the 3D surface, wherein the first roughness is determined at least in part based on the standard deviation.

[0112] The 105th implementation can be further extended to the 104th implementation. In the 105th implementation, the increase in standard deviation corresponds to the increase in the first roughness.

[0113] The 106th implementation can be further extended from the first implementation to the 105th implementation. In the 106th implementation, an intraoral scanning system includes an intraoral scanner; and a computing device for performing the method of any of the first to 105th implementations.

[0114] The 107th implementation can be further extended from the first implementation to the 105th implementation. In the 107th implementation, a computer-readable medium includes instructions that, when executed by a processing device, cause the processing device to perform the method of any of the first to 105th implementations.

[0115] The 108th implementation can be further extended from the first implementation to the 105th implementation. In the 100th implementation, the computing device includes a memory and one or more processors, wherein the one or more processors are used to execute the method of any of the first to 105th implementations.

[0116] In a first implementation, an intraoral scanning system includes an intraoral scanner and a computing device, wherein the computing device is configured to: receive multiple intraoral scans of a tooth position from the intraoral scanner during an intraoral scanning session; generate a three-dimensional (3D) surface of the tooth position based on the multiple intraoral scans; determine a first surface quality fraction of a first region of the 3D surface; receive one or more additional intraoral scans of the tooth position during the intraoral scanning session; update the 3D surface based on the one or more additional intraoral scans; update the surface quality fraction of the first region based on the updated 3D surface; determine that the surface quality fraction of the first region fails to meet one or more criteria; and output a notification to stop generating a new intraoral scan of the first region or to continue to at least one of the next regions.

[0117] The 110th implementation can be further extended to the 109th implementation. In the 110th implementation, in order to determine that the surface quality score of the first region fails to meet one or more criteria, the computing device is used to: determine that the surface quality score of the first region fails to improve within a threshold time amount; and determine that additional intraoral scanning of the first region does not improve the surface quality score of the region.

[0118] The 111th implementation can be further extended to the 109th or 110th implementation. In the 111th implementation, in order to determine that the surface quality score of the first region fails to meet one or more criteria, the computing device is used to: determine that the surface quality score of the first region is lower than the surface quality threshold and that the additional surface quality score of one or more additional regions close to the first region is equal to or higher than the surface quality threshold.

[0119] In a 112th implementation, an intraoral scanning system includes an intraoral scanner and a computing device, wherein the computing device is configured to: receive multiple intraoral scans of a tooth position from the intraoral scanner during an intraoral scanning session; generate a three-dimensional (3D) surface of the tooth position based on the multiple intraoral scans; determine a surface quality fraction of a first region of the 3D surface; receive one or more additional intraoral scans of the tooth position during the intraoral scanning session; update the 3D surface based on the one or more additional intraoral scans; update the surface quality fraction of the first region based on the updated 3D surface; determine that the surface quality fraction of the first region fails to meet one or more criteria; and output a notification associated with scanning the first region.

[0120] The 113th implementation can further extend the 112th implementation. In the 113th implementation, in order to determine that the surface quality score of the first region fails to meet one or more criteria, the computing device is used to: determine that the surface quality score of the first region fails to improve within a threshold time period and / or determine that the surface quality score of the first region is below a surface quality threshold.

[0121] The 114th implementation can be further extended to the 112th or 113th implementation. In the 114th implementation, in order to determine that the surface quality fraction of the first region fails to meet one or more criteria, the computing device is used to: determine that the additional surface quality fraction of one or more additional regions adjacent to the first region is equal to or higher than the surface quality threshold.

[0122] The 115th implementation can be further extended from the 112th to the 114th implementations. In the 115th implementation, in order to determine that the surface quality fraction of the first region fails to meet one or more criteria, the computing device is used to: determine that additional intraoral scanning of the first region will not improve the surface quality fraction of the first region.

[0123] The 116th implementation can be further extended from the 112th to the 115th implementation. In the 116th implementation, the computing device is further configured to: output a first view of the 3D surface to a display, wherein a first region of the 3D surface is shown using a first visualization associated with a surface quality fraction; and output an updated second view of the 3D surface to the display, wherein the first region of the updated 3D surface is shown using a second visualization associated with an updated surface quality fraction.

[0124] The 117th implementation can be further extended from the 112th to the 116th implementation. In the 117th implementation, the computing device is also used to: input at least one of a first region of the 3D surface or a plurality of intraoral scans associated with the first region into a trained machine learning model, wherein the trained machine learning model outputs a surface quality score.

[0125] The 118th implementation can be further extended from the 112th to the 117th implementation. In the 118th implementation, multiple intraoral scans are generated by projecting structured light, including multiple features, onto the tooth position and capturing multiple features on the tooth position, and wherein the surface quality fraction is determined at least in part based on the number of multiple features associated with a first region of the 3D surface.

[0126] The 119th implementation can be further extended from the 112th to the 118th implementations. In the 119th implementation, the 3D surface is generated based on multiple points from multiple intraorbital scans, and the computing device is also used to: determine the quality fraction of each of the multiple points; determine a subset of the multiple points associated with a first region; and determine the surface quality fraction of the first region based on the quality fraction of the subset of multiple points and the number of subsets of multiple points.

[0127] The 120th implementation can be further extended to the 119th implementation. In the 120th implementation, the quality score of one of the multiple points is calculated based on at least one of the following: a) the distance between the point and a first or second camera of the intraoral scanner that captures the point when generating one of the multiple intraoral scans; b) the distance between the first and second cameras; c) the distance between the point and the camera of the intraoral scanner that captures the point when generating the intraoral scan; d) the distance between the camera and the structured light projector of the intraoral scanner that projects structured light onto the point; the number of cameras of the intraoral scanner that capture the point when generating the intraoral scan; the spot size associated with the point; the angle between the normal of the 3D surface at the point and the imaging axis of the camera that captures the point when generating the intraoral scan; or the material type of the tooth at the point.

[0128] The 121st implementation can be further extended from the 112th to the 120th implementation. In the 121st implementation, the computing device is also configured to: receive multiple two-dimensional (2D) images of a tooth position; and determine the number of multiple 2D images depicting a first region; wherein the surface quality fraction of the first region is based at least in part on the number of multiple 2D images depicting the first region.

[0129] The 122nd implementation can be further extended from the 112th to the 121st implementation. In the 122nd implementation, the computing device is also used to: determine a first roughness and a first resolution associated with a first region of the 3D surface; wherein the surface quality fraction of the first region is determined at least in part based on the first roughness and the first resolution.

[0130] The 123rd implementation can be further extended from the 112th to the 122nd implementation. In the 123rd implementation, the computing device is further configured to: determine one or more scanning recommendations that, if implemented, would improve the surface quality fraction of the first region; wherein the notification includes one or more scanning recommendations.

[0131] The 124th implementation can be further extended from the 112th to the 123rd implementation. In the 124th implementation, the computing device is also configured to: increase the scaling setting of the first region in response to determining that the surface quality fraction of the first region fails to meet one or more criteria.

[0132] The 125th implementation can be further extended from the 112th to the 124th implementations. In the 125th implementation, the notification includes at least one of the following: a notification to stop generating a new intraoral scan of the first region; a notification to continue to the next region; or a notification that the surface quality fraction of the first region has not improved over a period of time.

[0133] The 126th implementation can be further extended from the 112th to the 125th implementations. In the 126th implementation, the intraoral scanner includes a plurality of cameras, each camera having at least one of different positions or orientations within the intraoral scanner, and wherein the computing device is further configured to: receive a plurality of two-dimensional (2D) images, each 2D image generated by a different camera among the plurality of cameras; determine one of the plurality of 2D images associated with improved intraoral scan quality; and output the determined 2D image to a display, wherein in response to the output of the determined 2D image, a physician using the intraoral scanner will reposition the intraoral scanner in a manner that results in the improved intraoral scan quality. Attached Figure Description

[0134] Embodiments of this disclosure are illustrated by way of example and not by way of limitation in the various figures of the accompanying drawings.

[0135] Figure 1 An embodiment of a system 100 for performing intraoral scanning and / or generating virtual three-dimensional models of tooth positions is shown.

[0136] Figure 2A This is a schematic illustration of a handheld intraoral scanner according to some applications of the present disclosure, wherein multiple cameras of the handheld intraoral scanner are disposed in a probe at the distal end of the intraoral scanner.

[0137] Figures 2B to 2C Schematic illustrations of the positioning configuration of a camera and structured light projector for an intraoral scanner, including some applications according to this disclosure.

[0138] Figure 2D It is a diagram depicting several different configurations of the positions of the structured light projector and camera in the probe of an intraoral scanner according to some applications of this disclosure.

[0139] Figure 3A This is a flowchart of a method for displaying the 3D surface of a tooth position in a manner that displays surface quality during intraoral scanning, according to an embodiment of the present disclosure.

[0140] Figure 3B This is a flowchart of a method for performing actions based on the surface quality fraction of a 3D surface according to an embodiment of the present disclosure.

[0141] Figure 4 This is a flowchart of a method for displaying the 3D surface of a tooth position in a manner that displays surface quality during intraoral scanning, according to an embodiment of the present disclosure.

[0142] Figure 5 This is a flowchart of a method for displaying the 3D surface of a tooth position during intraoral scanning in accordance with embodiments of the present disclosure, in a manner that displays different surface quality fractions in different regions.

[0143] Figure 6 This is a flowchart of a method for classifying regions of a 3D surface of a tooth position and determining a surface quality score associated with a region, according to embodiments of the present disclosure, using one or more trained machine learning models.

[0144] Figure 7 This is a flowchart of a method for determining a surface quality fraction associated with a 3D surface of a tooth position using a trained machine learning model, according to embodiments of the present disclosure.

[0145] Figure 8 This is a flowchart of a method for determining a surface quality fraction associated with a 3D surface of a tooth position according to embodiments of the present disclosure.

[0146] Figure 9 The illustration shows an angle-based scoring of a tooth position region according to an embodiment of the present disclosure.

[0147] Figure 10 This is a flowchart of a method for determining the surface quality fraction of a 2D image according to embodiments of the present disclosure.

[0148] Figure 11 This is a flowchart of a method for determining the surface quality fraction of a region under consideration based on roughness and / or resolution, according to embodiments of the present disclosure.

[0149] Figure 12 The illustration shows a scoring of a tooth position region based on roughness and / or point density according to an embodiment of the present disclosure.

[0150] Figures 13A to 13C The illustration shows a view of a graphical user interface for an intraoral scanning application including a 3D surface with surface quality feedback, according to an embodiment of the present disclosure.

[0151] Figure 14A This is a flowchart of a method for providing user assistance during intraoral scanning according to an embodiment of the present disclosure.

[0152] Figure 14B This is a flowchart of a method for providing user assistance during intraoral scanning according to an embodiment of the present disclosure.

[0153] Figure 14C This is a flowchart of a method for providing user assistance during intraoral scanning in an instance where the surface quality of a region is not improved, according to embodiments of the present disclosure.

[0154] Figure 15A This is a flowchart of a method for adjusting scaling settings to assist intraoral scanning according to embodiments of the present disclosure.

[0155] Figure 15B This is a flowchart of a method for determining and outputting recommendations to assist intraoral scanning during intraoral scanning, according to embodiments of the present disclosure.

[0156] Figure 16A This is a flowchart of a method for generating and outputting scan-guided overlays according to embodiments of the present disclosure.

[0157] Figure 16B This is a flowchart of a method for selecting an image (e.g., a viewfinder image) to influence how tooth positions are scanned, according to embodiments of this disclosure.

[0158] Figures 17A to 17B The illustration shows a collection of intraoral images generated by a camera array of an intraoral scanner according to an embodiment of the present disclosure.

[0159] Figure 18 The illustration shows a reference frame for a plurality of cameras of an intraoral scanner according to an embodiment of the present disclosure.

[0160] Figure 19 The flowchart illustrates a method for generating and outputting a suggested path to be followed during intraoral scanning, according to embodiments of the present disclosure.

[0161] Figure 20 This is a flowchart of a method for guiding a user to place an intraoral scanner during scanning, according to an embodiment of the present disclosure.

[0162] Figure 21 The illustration shows a view of the 3D surface of a scanned tooth according to an embodiment of the present disclosure, the position and orientation of the probe head of an intraoral scanner relative to the 3D surface, and the next suggested position and orientation of the probe head relative to the 3D surface.

[0163] Figures 22A to 22C The illustration shows a 3D surface of a scanned tooth and a suggested path for scanning the tooth according to an embodiment of the present disclosure.

[0164] Figure 23 The illustration shows a 3D surface of a scanned tooth and a proposed view of a rotating intraoral scanner according to an embodiment of the present disclosure.

[0165] Figure 24The illustration shows a 3D surface of a scanned tooth position and a suggested view of patient movement of the jaw according to an embodiment of the present disclosure.

[0166] Figure 25 This is a flowchart illustrating an embodiment of a method for determining the scan quality of a region of a 3D surface during intraoral scanning and outputting recommendations for improving scan quality.

[0167] Figure 26 This is a flowchart of a method for adjusting the scaling settings of a 3D surface based on scanner speed, according to an embodiment of the present disclosure.

[0168] Figure 27 This is a flowchart of a method for adjusting the scaling settings of a 3D surface based on scanner speed, according to an embodiment of the present disclosure.

[0169] Figure 28A The illustration shows a view of a 3D surface of a tooth position having a first scaling setting according to an embodiment of the present disclosure.

[0170] Figure 28B The illustration shows a view of a 3D surface of a tooth position having a second scaling setting according to an embodiment of the present disclosure.

[0171] Figure 29 A block diagram of an example computing device according to an embodiment of the present disclosure is illustrated. Detailed Implementation

[0172] This document describes methods and systems for providing useful, progressive, real-time scanning feedback, such as visualization of intraoral objects (e.g., tooth positions) reflecting scan quality during intraoral scanning. Methods and systems for assisting users in intraoral scanning are also described, such as by providing suggestions for improving scans, by automatically adjusting scaling settings during intraoral scanning, by automatically selecting images for display during intraoral scanning, by automatically adjusting the resolution for a portion of a 3D surface during intraoral scanning, etc. Examples of real-time visualizations that may be provided in embodiments include representations of the 3D surface of tooth positions, representations of the intraoral scanner relative to (multiple) 3D surfaces, representations of suggested positions / orientations of the intraoral scanner relative to the 3D surfaces, representations of 3D surface regions where the visualization represents quality fractions associated with these regions, representations of recommended paths followed by the intraoral scanner for subsequent scans, representations of target positions and / or orientations of the intraoral scanner relative to its current position and / or orientation, representations of voids in the scanned surface, etc.

[0173] In various embodiments, the intraoral scanning application can continuously update the 3D surface and adjust the view of the 3D surface and / or the intraoral scanner during intraoral scanning. As the scan progresses, more and more information about the regions of the 3D surface is received. When additional information about the regions of the 3D surface is received, this additional information can be used to update the 3D surface, and the quality score associated with that region can be updated. The regions of the 3D surface can be shown using a visualization determined based on the current quality score associated with that region. In various embodiments, a variety of different techniques can be used to determine the quality scores of the regions of the 3D surface, points on the 3D surface, and / or the intraoral scan and / or the image. Thus, the processing logic displays the user's accumulated surface quality during scanning so that the user can see progressive improvement in problem areas and can decide how much effort to put into scanning these areas. In some instances, improvement may not be possible in problem areas. This may cause the processing logic to determine that the problem area cannot be successfully scanned and output a notification to the user to stop attempting to scan the problem area.

[0174] Intraoral scanners may present multiple surface capture challenges, such as dental objects with reflective surface materials that are difficult to capture; tooth positions where the surface of the tooth is at a high angle to the imaging axis (making the surface difficult to capture accurately); portions of the tooth position far from the intraoral scanner and therefore having high noise and / or error; portions of the tooth position too close to the intraoral scanner and having error; tooth positions captured when the scanner is moving too fast, causing data blurring and / or localized capture of the area; and accumulation of blood and / or saliva on the tooth position. Some or all of these challenges can result in high noise levels in the generated intraoral scan. Additionally, some capture areas of the tooth position may have a small number of capture points (e.g., only a few points per square millimeter), resulting in low accuracy of the generated 3D surface. The embodiments address each of these challenges and enable the user to receive progressive real-time or near-real-time surface quality feedback that may take into account some or all of these challenges that the scanner may be encountering or is currently encountering. In the embodiments, even with high noise levels, the construction algorithm is able to estimate the 3D surface with high accuracy if a large number of points are captured for the area.

[0175] In various embodiments, the surface quality score of regions of the 3D surface of the tooth position is updated continuously or periodically during intraoral scanning. This allows the user to determine whether they are making progress during the scan. By providing progressive feedback based on the quality of the generated 3D surface (or regions thereof) during the scan, the user can determine which areas of the 3D surface are improving and which areas are not improving. In various embodiments, different regions may be associated with different quality thresholds. For example, the quality threshold associated with preparing the tooth may be higher compared to other teeth. The user can see small improvements in the quality of regions of the 3D surface and can focus their attention and scan on important areas that require high quality during the scan (e.g., preparing the tooth) until the quality of these surfaces reaches the threshold quality.

[0176] In various embodiments, the speed of the scanner and / or virtual point (e.g., the focal point within the scanner's field of view) is determined, and the scaling settings used to display the generated 3D surface are controlled based on this speed. In various embodiments, the scaling settings of the 3D surface are automatically adjusted as the speed changes. Smoothing operations can be used to smooth the adjustment of the scaling settings to avoid sudden changes in the scaling settings.

[0177] In various embodiments, the processing logic evaluates information received during intraoral scanning and, based on that information, determines whether the user of the intraoral scanner is experiencing a problem when scanning a region of a tooth position. For example, if the intraoral scanner remains focused on the region for a threshold amount of time, if the quality score of the region does not improve, or if the intraoral scanner speed and / or focus is below a threshold, the processing logic may determine that the user is experiencing a problem when scanning that region. In response to determining that the user is experiencing a problem when scanning the region, the processing logic may determine one or more actions that may improve scan quality and / or provide one or more recommendations for improving scan quality. Many different actions may be performed and / or many different recommendations may be provided, examples of which are provided below. Alternatively, the processing logic may determine that no improvement in scan quality can be obtained for that region and may output a recommendation to stop scanning that region and / or continue scanning the next region.

[0178] Various embodiments are described herein. It should be understood that these various embodiments can be implemented as independent solutions and / or can be combined. Thus, reference to an embodiment or an embodiment can refer to the same embodiment and / or different embodiments. Some embodiments are discussed herein with reference to intraoral scans and intraoral images. However, it should be understood that the embodiments described with reference to intraoral scans are also applicable to laboratory scans or model / impression scans. Laboratory scans or model / impression scans may include one or more images of tooth positions or models or impressions of tooth positions, which may or may not include height maps and may or may not include intraoral two-dimensional (2D) images (e.g., 2D color images).

[0179] Figure 1 An embodiment of a system 101 for performing intraoral scanning and / or generating three-dimensional (3D) surfaces and / or virtual 3D models of tooth positions is illustrated. System 101 includes a dental clinic 108 and optionally one or more dental laboratories 110. The dental clinic 108 and dental laboratory 110 each include computing devices 105, 106, wherein computing devices 105, 106 can be interconnected with each other via a network 180. Network 180 can be a local area network (LAN), a public wide area network (WAN) (e.g., the Internet), a private WAN (e.g., an intranet), or a combination thereof.

[0180] Computing device 105 may be coupled to one or more intraoral scanners 150 (also referred to as scanners) and / or data repository 125 via wired or wireless connections. In one embodiment, multiple scanners 150 in a dental clinic 108 are wirelessly connected to computing device 105. In one embodiment, scanners 150 are wirelessly connected to computing device 105 via a direct wireless connection. In one embodiment, scanners 150 are wirelessly connected to computing device 105 via a wireless network. In one embodiment, the wireless network is a Wi-Fi network. In one embodiment, the wireless network is a Bluetooth network, a Zigbee network, or some other wireless network. In one embodiment, the wireless network is a wireless mesh network, examples of which include Wi-Fi mesh networks, Zigbee mesh networks, etc. In one example, computing device 105 may be physically connected to one or more wireless access points and / or wireless routers (e.g., Wi-Fi access points / routers). Intraoral scanners 150 may include wireless modules (such as Wi-Fi modules) and can join wireless networks via wireless access points / routers through the wireless modules.

[0181] Computing device 106 may also be connected to a data repository (not shown). The data repository may be a local data repository and / or a remote data repository. Computing device 105 and computing device 106 may each include one or more processing devices, memory, secondary storage devices, one or more input devices (e.g., such as a keyboard, mouse, tablet, touch screen, microphone, camera, etc.), one or more output devices (e.g., display, printer, touch screen, speaker, etc.) and / or other hardware components.

[0182] In various embodiments, scanner 150 includes an inertial measurement unit (IMU). The IMU may include an accelerometer, gyroscope, magnetometer, pressure sensor, and / or other sensors. For example, scanner 150 may include one or more microelectromechanical systems (MEMS) IMUs. The IMU can generate inertial measurement data (also referred to as motion data), which includes acceleration data, rotation data, etc.

[0183] Computing device 105 and / or data repository 125 may be located in dental clinic 108 (as shown), dental laboratory 110, or one or more other locations such as a server farm providing cloud computing services. Computing device 105 and / or data repository 125 may be connected to components located in the same or different locations as computing device 105 (e.g., components in a second location away from dental clinic 108, such as a server farm providing cloud computing services). For example, computing device 105 may be connected to a remote server, where some operations of intraoral scanning application 115 are performed on computing device 105 and some operations of intraoral scanning application 115 are performed on the remote server.

[0184] Some additional computing devices may be physically connected to computing device 105 via a wired connection. Some additional computing devices may be wirelessly connected to computing device 105 via a wireless connection, which may be a direct wireless connection or a wireless connection via a wireless network. In various embodiments, one or more additional computing devices may be mobile computing devices, such as laptops, notebook computers, tablets, mobile phones, portable game consoles, etc. In various embodiments, one or more additional computing devices may be traditionally stationary computing devices, such as desktop computers, set-top boxes, game consoles, etc. Additional computing devices may act as thin clients of computing device 105. In one embodiment, the additional computing device uses the Remote Desktop Protocol (RDP) to access computing device 105. In one embodiment, the additional computing device uses Virtual Network Control (VNC) to access computing device 105. Some additional computing devices may be passive clients that do not have control over computing device 105 and receive visualization of the user interface of intra-port scanning application 115. In one embodiment, one or more additional computing devices may operate in master mode, and computing device 105 may operate in slave mode.

[0185] The intraoral scanner 150 may include a probe (e.g., a handheld probe) for optically capturing three-dimensional structures. The intraoral scanner 150 can be used to perform intraoral scans of a patient's oral cavity. An intraoral scanning application 115 running on computing device 105 can communicate with the scanner 150 to perform the intraoral scan. The result of the intraoral scan may be intraoral scan data 135A, 135B to 135N, which may include one or more sets of intraoral scans and / or sets of intraoral 2D images. Each intraoral scan may include 3D or point clouds, which may include depth information (e.g., a height map) of a portion of a tooth position. In various embodiments, the intraoral scan includes x, y, and z information.

[0186] In various embodiments, the intraoral scan data 135A to 135N may further include color 2D images of tooth positions and / or images at specific wavelengths (e.g., near-infrared (NIRI) images, infrared images, ultraviolet images, etc.). In various embodiments, the intraoral scanner 150 alternates between generating 3D intraoral scans and one or more types of 2D intraoral images (e.g., color images, NIRI images, etc.) during scanning. For example, one or more 2D color images can be generated between a fourth and a fifth intraoral scan by outputting white light and using multiple cameras to capture the reflection of the white light.

[0187] The intraoral scanner 150 may include multiple different cameras (e.g., each camera may include one or more image sensors) that simultaneously generate 2D images (e.g., 2D color images) of different regions of the patient's dental arch. These 2D color images may be stitched together to form a single 2D image representation of a larger field of view that combines the fields of view of the multiple cameras. The intraoral 2D images may include 2D color images, 2D infrared or near-infrared (NIRI) images, and / or 2D images generated under other specific lighting conditions (e.g., 2D ultraviolet images). The user of the intraoral scanner can use the 2D images to determine where the scanning plane of the intraoral scanner is pointing and / or to determine other information about the position of the scanned teeth.

[0188] The scanner 150 can transmit intraoral scan data 135A, 135B to 135N to the computing device 105. The computing device 105 can store the intraoral scan data 135A to 135N in the data storage repository 125.

[0189] According to one example, a user (e.g., a physician) can subject a patient to an intraoral scan. In doing so, the user can apply scanner 150 to one or more intraoral locations of the patient. The scan can be divided into one or more segments (also referred to as roles). As an example, such segments may include the patient's lower dental arch, the patient's upper dental arch, one or more preparatory teeth of the patient (e.g., teeth of the patient to which a dental appliance, such as a crown or other dental restoration, will be applied), one or more teeth in contact with the preparatory teeth (e.g., teeth that are not affected by the dental appliance itself, but which are located next to or engage with one or more such teeth when the mouth is closed), and / or the patient's occlusion (e.g., a scan performed with the patient's mouth closed, where the scan is directed towards the interface area between the patient's upper and lower teeth). Through this scanner application, scanner 150 can provide intraoral scan data 135A to 135N to computing device 105. Intraoral scan data 135A to 135N can be provided in the form of intraoral scan datasets, each of which may include 2D intraoral images (e.g., color 2D images) and / or 3D intraoral scans of regions of specific teeth and / or tooth positions. In one embodiment, separate intraoral scan datasets are created for the maxillary arch, mandibular arch, patient occlusion, and each prepared tooth. Alternatively, a large single intraoral scan dataset (e.g., for the mandibular arch and / or maxillary arch) can be generated. Intraoral scans can be provided from scanner 150 to computing device 105 in the form of one or more points (e.g., one or more pixels and / or groups of pixels). For example, scanner 150 may provide intraoral scans as one or more point clouds. Intraoral scans may each include height information (e.g., a height map indicating the depth of each pixel).

[0190] The method by which a patient's oral cavity is scanned can depend on the procedure to be performed. For example, if a maxillary or mandibular denture is to be created, the entire edentulous arch of the mandible or maxilla may be scanned. Conversely, if a dental bridge is to be created, only a portion of the entire arch may be scanned, including the edentulous area, adjacent prepared teeth (e.g., mating teeth), and the opposing arch and dentition. Alternatively, if a dental bridge is to be created, a full scan of the upper and / or lower dental arch may be performed.

[0191] By way of non-limiting example, dental procedures can be broadly categorized into prosthodontics (restoration) and orthodontics, which are further subdivided into specific forms of these procedures. Additionally, dental procedures can include the identification and treatment of gingival diseases, sleep apnea, and intraoral conditions. The term prosthodontics specifically refers to any procedure involving the oral cavity and involving the design, fabrication, or installation of a dental prosthesis or its real or virtual model at a tooth position within the oral cavity (tooth position), or involving the design and preparation of tooth positions to receive such a prosthesis. For example, prostheses can include any restoration such as crowns, veneers, inlays, onlays, implants, and bridges, and any other artificial partial or complete dentures. The term orthodontics specifically refers to any procedure involving the oral cavity and involving the design, fabrication, or installation of orthodontic elements or their real or virtual models at tooth positions within the oral cavity, or involving the design and preparation of tooth positions to receive such orthodontic elements. These elements can be appliances, including but not limited to brackets and archwires, retainers, clear aligners, or functional appliances.

[0192] In various embodiments, intraoral scans can be performed on a patient's oral cavity during a visit to dental clinic 108. Intraoral scans can be performed, for example, as part of a semi-annual or annual dental health check. Intraoral scans can also be performed before, during, and / or after one or more dental treatments, such as orthodontic treatment and / or prosthodontic treatment. Intraoral scans can be complete or partial scans of the upper and / or lower dental arches and can be performed to acquire information for performing dental diagnoses, generating treatment plans, determining the progress of treatment plans, and / or for other purposes. Dental information generated from intraoral scans (intraoral scan data 135A to 135N) can include 3D scan data, 2D color images, NIRI and / or infrared images, and / or ultraviolet images of all or part of the maxilla and / or mandible. Intraoral scan data 135A to 135N can also include one or more intraoral scans illustrating the relationship between the upper and lower dental arches. These intraoral scans can be used to determine the patient's occlusion and / or to determine the patient's occlusal contact information. Patient occlusion can include the relationship between the teeth in the upper dental arch and the teeth in the lower dental arch.

[0193] For many dental prosthetic procedures (e.g., creating crowns, bridges, veneers, etc.), a patient's existing teeth are ground down into stumps. The ground-down tooth is referred to in this text as a prepared tooth, or simply a preparation. A prepared tooth has a marginal line (also called an end line), which is the boundary between the natural (unground) portion and the prepared (ground) portion of the prepared tooth. Prepared teeth are typically created so that crowns or other prostheses can be mounted or placed on them. In many cases, the marginal line of a prepared tooth is the sub-gingival margin (below the gingival line).

[0194] An intraoral scanner operates by moving scanner 150 inside a patient's mouth to capture all viewpoints of one or more teeth. In some embodiments, during a scan, scanner 150 calculates distances to a solid surface. In some embodiments, these distances may be recorded as an image or point cloud referred to as a "height map". Each scan (e.g., optionally, a height map or point cloud) is algorithmically overlaid or "stitched" with a set of previous scans to generate a long 3D surface. Thus, each scan is associated with a rotation or projection in space to determine how it fits into the 3D surface.

[0195] During an intraoral scan, the intraoral scan application 115 can register and stitch together two or more intraoral scans generated remotely from the intraoral scan session to generate a grown 3D surface. In one embodiment, performing registration includes: capturing 3D data of individual points of the surface in multiple scans; and registering the scans by calculating transformations between the scans. During the intraoral scan, one or more 3D surfaces can be generated based on the registered and stitched intraoral scans. One or more 3D surfaces can be output to a display, allowing a physician or technician to view their scan progress to date. As each new intraoral scan is captured and registered to a previous intraoral scan and / or 3D surface, one or more 3D surfaces can be updated, and the updated 3D surface(s) can be output to the display. The view of the 3D surfaces(s) can be updated periodically or continuously according to one or more viewing modes of the intraoral scan application. In one viewing mode, the 3D surface can be continuously updated such that the orientation of the displayed 3D surface is aligned with the field of view of the intraoral scanner (e.g., such that a portion of the 3D surface based on the most recently generated intraoral scan is approximately centered on the display or a window of the display), and the user sees what the intraoral scanner is seeing. In another viewing mode, the position and orientation of the 3D surface are static, and optionally, an image from the intraoral scanner is shown moving relative to the static 3D surface.

[0196] Intraoral scanning application 115 can generate one or more 3D surfaces from an intraoral scan and can display these 3D surfaces to a user (e.g., a physician) via a graphical user interface (GUI) during the intraoral scan. In various embodiments, separate 3D surfaces are generated for the maxilla and mandible. This process can be performed in real-time or near real-time to provide an updated view of the captured 3D surfaces during the intraoral scanning process. When scans are received, these scans can be registered and stitched onto the 3D surfaces. As discussed in detail below, quality scores for individual regions of the 3D surfaces can be determined based on one or more criteria. The quality scores can be updated continuously or periodically as information is added from other intraoral scans. As the quality scores gradually change, the visualization of the regions may change accordingly, allowing the user to receive real-time or near real-time feedback on surface quality during the scan. Additionally or alternatively, the scanning process can be monitored upon receiving a scan to determine if the user is experiencing problems when scanning any region of a tooth position (e.g., the upper or lower dental arch). If it is determined that the user is experiencing problems while scanning the area of ​​the tooth, one or more remedial measures and / or one or more suggestions can be performed. Additionally or alternatively, while a scan is being performed, the scaling settings for displaying the (multiple) 3D surfaces can be dynamically determined based on one or more criteria, such as scanner speed and / or scanner focus. In various embodiments, the user can select to enable or disable automatic scaling and / or automatic suggestions via a GUI. For example, the user can enter a scan assistance request, which may result in the activation of automatic scaling and / or scan suggestions. These and other operations can be performed during scanning to improve the quality of the (multiple) 3D surfaces, speed up the scan, assist users in difficult areas, etc.

[0197] When a scanning session, or part of a scanning session associated with a specific scanning role (e.g., maxillary role, mandibular role, occlusal role, etc.), is completed (e.g., all scans or tooth positions have been captured), the intraoral scanning application 115 can generate (e.g., maxillary and mandibular) virtual 3D models of one or more scanned tooth positions. The final 3D model can be a collection of 3D points and their connections to each other (i.e., a mesh). To generate the virtual 3D model, the intraoral scanning application 115 can register and stitch together intraoral scans associated with a specific scanning role generated from the intraoral scanning session. Registration performed at this stage can be more accurate than registration performed during the capture of the intraoral scan and can take more time to complete. In one embodiment, performing scan registration includes: capturing 3D data of individual points of a surface in multiple scans; and registering the scans by calculating transformations between the scans. The 3D data can be projected into the 3D space of the 3D model to form part of the 3D model. By applying appropriate transformations to the points of each registered scan and projecting each scan into 3D space, intraoral scans can be integrated into a common reference frame.

[0198] In one embodiment, registration is performed for adjacent or overlapping intraoral scans (e.g., each consecutive frame of an intraoral video). A registration algorithm is performed to register two adjacent or overlapping intraoral scans and / or to register an intraoral scan with a 3D model. This essentially involves determining the transformation that aligns one scan with the other and / or with the 3D model. Registration may involve identifying multiple points (e.g., point clouds) in each scan of the scan pair (or scan and 3D model), performing surface fitting on these points, and using a local search around the points to match the points of the two scans (or scan and 3D model). For example, an intraoral scan application 115 may match points from one scan with the nearest interpolated point on the surface of the other scan, iteratively minimizing the distance between the matched points. Other registration techniques may also be used.

[0199] The intraoral scan application 115 can repeatedly register for all intraoral scans in the intraoral scan sequence to obtain a transformation for each intraoral scan, registering each intraoral scan with (multiple) previous intraoral scans and / or with a common reference frame (e.g., with a 3D model). The intraoral scan application 115 can integrate intraoral scans into a single virtual 3D model by applying transformations, determined in an appropriate manner, to each intraoral scan within the intraoral scan sequence. Each transformation may include rotations about one to three axes and translations in one to three planes.

[0200] The intraoral scanning application 115 can generate one or more 3D models from an intraoral scan and display these 3D models to a user (e.g., a physician) via a graphical user interface (GUI). The physician can then visually examine the 3D models. The physician can virtually manipulate the 3D models via the user interface using appropriate user controls (hardware and / or virtual) about up to six degrees of freedom (i.e., translation and / or rotation about one or more of three mutually orthogonal axes) to allow viewing the 3D models from any desired direction.

[0201] Now, for reference Figure 2A ,Should Figure 2A This is a schematic illustration of an intraoral scanner 20 including an elongated handheld rod, according to some applications of this disclosure. In various embodiments, the intraoral scanner 20 may correspond to... Figure 1 The intraoral scanner 150. The intraoral scanner 20 includes a plurality of structured light projectors 22 and a plurality of cameras 24, which are coupled to a rigid structure 26 disposed within a probe 28 located at the distal end 30 of the intraoral scanner 20. In some applications, the probe 28 is inserted into the oral cavity of a subject or patient during an intraoral scanning procedure.

[0202] For some applications, the structured light projectors 22 are positioned within the probe 28 such that each structured light projector 22 faces the object 32 placed outside the intraoral scanner 20 within its illumination field, rather than positioning the structured light projector at the proximal end of the handheld stick and illuminating the object by light reflected from a mirror and then reflected onto the object. Alternatively, the structured light projectors can be positioned at the proximal end of the handheld stick. Similarly, for some applications, the cameras 24 are located within the probe 28 such that each camera 24 faces the object 32 placed outside the intraoral scanner 20 within its field of view, rather than positioning the camera in the proximal end of the intraoral scanner and viewing the object by light exiting from a mirror and then entering the camera. This positioning of the projectors and cameras within the probe 28 allows the scanner to have a large overall field of view while maintaining a low-profile probe. Alternatively, the cameras can be positioned in the proximal end of the handheld stick.

[0203] In some applications, each of the cameras 24 has a large field of view β (beta) of at least 45 degrees (e.g., at least 70 degrees, e.g., at least 80 degrees, e.g., 85 degrees). In some applications, the field of view may be less than 120 degrees, e.g., less than 100 degrees, e.g., less than 90 degrees. In one embodiment, the field of view β (beta) of each camera is between 80 and 90 degrees, which can be particularly useful because it provides a good balance between pixel size, field of view and camera overlap, optical quality and cost. The camera 24 may include an image sensor 58 and an objective lens 60 comprising one or more lenses. To achieve near-focus imaging, the camera 24 may focus on an object focal plane 50 located between 1 mm and 30 mm away from the lens, e.g., between 4 mm and 24 mm, e.g., between 5 mm and 11 mm, e.g., between 9 mm and 10 mm, where the lens is furthest from the sensor. In some applications, camera 24 can capture images at a frame rate of at least 30 frames per second (e.g., at least 75 frames per second, or at least 100 frames per second). In some applications, the frame rate may be less than 200 frames per second.

[0204] Accuracy can be improved by combining the respective fields of view of all cameras, particularly in edentulous regions where there may be fewer high-resolution 3D features with smooth and clear gingival surfaces, due to reduced image stitching errors. A larger field of view allows for the inclusion of large, smooth features, such as the overall curves of the teeth, in each image frame, thus improving the stitching accuracy of the corresponding surfaces obtained from multiple such image frames.

[0205] Similarly, each of the structured light projectors 22 can have a large illumination field α (alpha) of at least 45 degrees, for example, at least 70 degrees. In some applications, the illumination field α (alpha) may be less than 120 degrees, for example, less than 100 degrees.

[0206] For some applications, to improve image capture, each camera 24 has multiple discrete preset focus positions, in which the camera focuses on the focal plane 50 of the corresponding object. Each camera 24 may include an autofocus actuator that selects a focus position from the discrete preset focus positions to improve a given image capture. Additionally or alternatively, each camera 24 includes an optical aperture phase mask that extends the depth of focus of the camera, such that the image formed by each camera remains focused at all object distances located between 1 mm and 30 mm from the lens, for example, between 4 mm and 24 mm, for example, between 5 mm and 11 mm, for example, between 9 mm and 10 mm, where the lens is furthest from the sensor.

[0207] In some applications, the structured light projector 22 and camera 24 are coupled to the rigid structure 26 in a closely spaced and / or alternating manner, such that (a) a large portion of the field of view of each camera overlaps with the field of view of the adjacent camera; and (b) a large portion of the field of view of each camera overlaps with the illumination field of the adjacent projector. Optionally, at least 20%, for example, at least 50%, for example, at least 75% of the projected light pattern is within the field of view of at least one of the cameras at the object focal plane 50, which is located at least 4 mm away from the lens, which is furthest from the sensor. Due to the different possible configurations of the projectors and cameras, some projection patterns may never be seen in the field of view of any of the cameras, and some projection patterns may be obscured by the object 32 and cannot be observed as the scanner moves around during scanning.

[0208] The rigid structure 26 can be a non-flexible structure to which the structured light projector 22 and camera 24 are coupled to provide structural stability to the optics within the probe 28. Coupling all projectors and cameras to a common rigid structure helps maintain the geometric integrity of the optics of each structured light projector 22 and each camera 24 under varying environmental conditions, such as mechanical stresses that may be induced by the subject's mouth. Additionally, the rigid structure 26 helps maintain the stable structural integrity and positioning of the structured light projectors 22 and cameras 24 relative to each other.

[0209] Now, for reference Figures 2B to 2C ,Should Figures 2B to 2C Schematic illustrations of positioning configurations for camera 24 and structured light projector 22, respectively, according to some applications of this disclosure. For some applications, to improve the overall field of view and illumination field of the intraoral scanner 20, camera 24 and structured light projector 22 are positioned such that they are not both facing the same direction. For some applications, such as... Figure 2B In the illustrated application, multiple cameras 24 are coupled to a rigid structure 26 such that the angle θ (θ) between two corresponding optical axes 46 of at least two cameras 24 is 90 degrees or less, for example, 35 degrees or less. Similarly, for some applications, such as... Figure 2C In the application shown, multiple structured light projectors 22 are coupled to a rigid structure 26 such that the angle φ (phi) between two corresponding optical axes 48 of at least two structured light projectors 22 is 90 degrees or less, for example, 35 degrees or less.

[0210] Now, for reference Figure 2D ,Should Figure 2D This is a diagram depicting several different configurations of the structured light projector 22 and camera 24 within the probe 28 for some applications according to this disclosure. The structured light projector 22... Figure 2DThe center is represented by a circle, and the camera 24 is in Figure 2D The rectangle represents the field of view β (beta) of each image sensor 58 and each camera 24, which is typically 1:2 in aspect ratio. Therefore, the rectangle is used to represent the camera. Figure 2D Column (a) shows a bird's-eye view of various configurations of the structured light projector 22 and camera 24. The x-axis, as marked in the first row of column (a), corresponds to the central longitudinal axis of the probe 28. Column (b) shows side views of the camera 24 in various configurations, as seen from a line of sight coaxial with the central longitudinal axis of the probe 28 and substantially parallel to the observation axis of the intraoral scanner. Figure 2B Similar to what is shown, Figure 2D Column (b) shows a camera 24 positioned such that the optical axes 46 are at an angle of 90 degrees or less (e.g., 35 degrees or less) about each other. Column (c) shows a side view of the camera 24 in various configurations as seen from a line of sight perpendicular to the central longitudinal axis of the probe 28.

[0211] Typically, the furthest end (facing) Figure 2D The closest camera 24 (facing the positive x-direction in Figure 3D) and the nearest camera 24 (facing the negative x-direction in Figure 3D) are positioned such that their optical axes 46 are slightly (e.g., at an angle of 90 degrees or less, such as 35 degrees or less) rotated inward about the next nearest camera 24. More centrally positioned cameras 24 (i.e., neither the farthest nor the closest camera 24) are positioned directly outward from the probe, with their optical axes 46 substantially perpendicular to the central longitudinal axis of the probe 28. It should be noted that in row (xi), the projector 22 is located at the farthest point of the probe 28, and therefore the optical axis 48 of the projector 22 points inward, allowing more cameras 24 to see a larger number of light spots 33 projected from that particular projector 22.

[0212] In various embodiments, the number of structured light projectors 22 in probe 28 can be two (e.g., as shown in the figure). Figure 2D The number of cameras 24 in probe 28 can typically range from four (as shown in rows (iv) and (v)) to seven (as shown in row (ix)). It should be noted that... Figure 2D The various configurations shown are by way of example and not limitation, and the scope of this disclosure includes additional configurations not shown. For example, the scope of this disclosure includes fewer or more than five projectors 22 located in probe 28 and fewer or more than seven cameras located within probe 28.

[0213] In one example application, an apparatus for intraoral scanning (e.g., an intraoral scanner 150) includes an elongated handheld rod comprising a probe located at its distal end, at least two light projectors disposed within the probe, and at least four cameras disposed within the probe. Each light projector may include at least one light source configured to generate light when activated; and a pattern-generating optics configured to generate a light pattern when light is transmitted through it. Each of the at least four cameras may include a camera sensor (also referred to as an image sensor) and one or more lenses, wherein each of the at least four cameras is configured to capture multiple images depicting at least a portion of a light pattern projected onto an intraoral surface. The at least two light projectors and most of the at least four cameras may be arranged in at least two rows, each row being generally parallel to the longitudinal axis of the probe, the at least two rows including at least a first row and a second row.

[0214] In another application, the farthest camera along the longitudinal axis of at least four cameras and the nearest camera along the longitudinal axis are positioned such that their optical axes are at an angle of 90 degrees or less to the line of sight perpendicular to the longitudinal axis about each other. Cameras in the first row and cameras in the second row can be positioned such that the optical axes of the cameras in the first row are at an angle of 90 degrees or less to the line of sight coaxial with the longitudinal axis of the probe about the optical axes of the cameras in the second row. Apart from the farthest and nearest cameras, the remaining at least four cameras have optical axes substantially parallel to the longitudinal axis of the probe. Each of the at least two rows may include an alternating sequence of light projectors and cameras.

[0215] In another application, at least four cameras include at least five cameras, at least two light projectors include at least five light projectors, the nearest-end component in the first row is a light projector, and the nearest-end component in the second row is a camera.

[0216] In another application, the farthest camera along the longitudinal axis and the nearest camera along the longitudinal axis are positioned such that their optical axes are at an angle of 35 degrees or less with respect to the line of sight perpendicular to the longitudinal axis. The cameras in the first row and the cameras in the second row can be positioned such that the optical axes of the cameras in the first row are at an angle of 35 degrees or less with respect to the optical axes of the cameras in the second row with respect to the line of sight coaxial with the longitudinal axis of the probe.

[0217] In another application, at least four cameras can have a combined field of view of 25 mm to 45 mm along the longitudinal axis and a field of view of 20 mm to 40 mm along the z-axis corresponding to the distance from the probe.

[0218] Back Figure 2AFor some applications, there is at least one uniform light projector 118 coupled to the rigid structure 26 (which may be an unstructured light projector that projects light across a wavelength range). The uniform light projector 118 can transmit white light onto the object 32 being scanned. At least one camera (e.g., one of the cameras in camera 24) uses the illumination from the uniform light projector 118 to capture a two-dimensional color image of the object 32.

[0219] Processor 96 can run a surface reconstruction algorithm that can generate a 3D surface of object 32 using a detection pattern (e.g., a dot matrix pattern) projected onto object 32. In some embodiments, processor 96 can combine at least one 3D scan captured using illumination from structured light projector 22 with multiple intraoral 2D images captured using illumination from uniform light projector 118 to generate a digital 3D image of the intraoral 3D surface. The combination of structured light and uniform illumination enhances the overall capture of the intraoral scanner and may help reduce the number of options that processor 96 needs to consider when running a corresponding algorithm used to detect depth values ​​of object 32. In one embodiment, the intraoral scanner and corresponding algorithm described in U.S. Application No. 16 / 446,181, filed June 19, 2019, are used. The entire contents of U.S. Application No. 16 / 446,181, filed June 19, 2019, are incorporated herein by reference. In various embodiments, processor 96 may be Figure 1 The processor of the computing device 105. Alternatively, the processor 96 may be a processor integrated into the intraoral scanner 20.

[0220] For some applications, all data points acquired at a specific time are used as rigid point clouds, and multiple such point clouds are captured at a frame rate of more than 10 captures per second. These multiple point clouds are then stitched together using a registration algorithm (e.g., Iterative Nearest Point (ICP)) to create a dense point cloud. A surface reconstruction algorithm can then be used to generate a representation of the surface of object 32.

[0221] For some applications, at least one temperature sensor 52 is coupled to the rigid structure 26 and measures the temperature of the rigid structure 26. A temperature control circuitry system 54 disposed within the intraoral scanner 20 (a) receives data indicating the temperature of the rigid structure 26 from the temperature sensor 52, and (b) activates a temperature control unit 56 in response to the received data. The temperature control unit 56 (e.g., a PID controller) maintains the probe 28 at a desired temperature (e.g., between 35 and 43 degrees Celsius, between 37 and 41 degrees Celsius, etc.). Maintaining the probe 28 at a temperature above 35 degrees Celsius (e.g., above 37 degrees Celsius) when it enters the intraoral cavity reduces fogging of the glass surface of the intraoral scanner 20 through which the structured light projector 22 projects and through which the camera 24 observes, the intraoral cavity typically at or above 37 degrees Celsius. Maintaining the probe 28 below 43 degrees Celsius (e.g., below 41 degrees Celsius) prevents discomfort or pain.

[0222] In some embodiments, heat can be drawn from the probe 28 via a heat-conducting element 94 (e.g., a heat pipe) disposed within the intraoral scanner 20, such that the distal end 95 of the heat-conducting element 94 contacts the rigid structure 26, and the proximal end 99 contacts the proximal end 100 of the intraoral scanner 20. Heat is thus transferred from the rigid structure 26 to the proximal end 100 of the intraoral scanner 20. Alternatively or additionally, a fan disposed in the handle region 174 of the intraoral scanner 20 can be used to draw heat away from the probe 28.

[0223] Figures 2A to 2D An intraoral scanner of one type that can be used in embodiments of the present disclosure is illustrated. However, it should be understood that the embodiments are not limited to the type of intraoral scanner illustrated. In one embodiment, intraoral scanner 150 corresponds to the intraoral scanner described in U.S. Application No. 16 / 910,042, filed June 23, 2020, entitled “Intraoral 3D Scanner Employing Multiple Miniature Cameras and Multiple Miniature Pattern Projectors,” which is incorporated herein by reference. In one embodiment, intraoral scanner 150 corresponds to the intraoral scanner described in U.S. Application No. 16 / 446,181, filed June 19, 2019, entitled “Intraoral 3D Scanner Employing Multiple Miniature Cameras and Multiple Miniature Pattern Projectors,” which is incorporated herein by reference.

[0224] In some embodiments, an intraoral scanner that performs confocal focusing to determine depth information can be used. Such an intraoral scanner may include a light source and / or illumination module that emits light (e.g., a focused beam or array of focused beams). The light passes through a polarizer and then through a one-way mirror or beamsplitter (e.g., a polarizing beamsplitter) that allows the light to pass. The light may pass through a pattern before or after the beamsplitter to make the light patterned. Along the optical path of the light after the one-way mirror or beamsplitter are optics that may include one or more lens groups. Any lens group may consist of only a single lens or multiple lenses. One lens group may include at least one movable lens.

[0225] Light can pass through an endoscopic probe member, which may include a rigid light-transmitting medium. This medium may be a hollow object defining a light transmission path within it, or an object made of a light-transmitting material (e.g., a glass body or tube). In one embodiment, the endoscopic probe member includes a prism, such as a folding prism. At the end of the prism, the endoscopic probe member may include a mirror that ensures total internal reflection. Therefore, the mirror can guide an array of light beams toward a tooth segment or other object. The endoscopic probe member thus emits light, which optionally passes through one or more windows and then incident on the surface of an intraoral object.

[0226] The light can comprise an array of beams propagating along the Z-axis, arranged in the XY plane of a Cartesian coordinate system, corresponding to the imaging or viewing axis of an intraoral scanner. Because the surface on which the incident beam is projected is not flat, the illuminated spot can be at different (X... i Y i The light spots at one location are shifted relative to each other along the Z-axis. Therefore, while a light spot at one location may be at the focal point of a confocal focusing optics, light spots at other locations may be out of focus. Consequently, the intensity of the returning beam from the focused spot will be at its peak, while the intensity at other spots will be below the peak. Therefore, for each illuminated spot, multiple intensity measurements are performed at different locations along the Z-axis. For such a (X) i Y i Each (X) position i Y i The position can be used to obtain the derivative of intensity with respect to distance (Z), where the maximum derivative Z0 is obtained at Z. i It is the focusing distance.

[0227] Light is reflected from an object within the opening and passes through a window (if present), then a mirror, through an optical system, and is reflected by a beam splitter onto a detector. The detector is an image sensor with a matrix of sensing elements, each representing a pixel of a scan or image. In one embodiment, the detector is a charge-coupled device (CCD) sensor. In another embodiment, the detector is a complementary metal-oxide-semiconductor (CMOS) image sensor. Other types of image sensors may also be used for the detector. In one embodiment, the detector detects the light intensity at each pixel, which can be used to calculate height or depth.

[0228] Alternatively, in some embodiments, an intraoral scanner using stereo imaging is used to determine depth information.

[0229] Figure 3 to Figure 8 , Figures 10 to 11 , Figures 14A to 16B , Figures 19 to 20 and Figures 25 to 27 This is a flowchart illustrating various methods related to providing user feedback during intraoral scanning to assist the scanning process. These methods can be executed by processing logic, which may include hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions running on a processing device to execute hardware emulation), firmware, or a combination thereof. In one embodiment, at least some operations of the method are performed by the scanning system's computing device and / or server computing device (e.g., by...). Figure 1 Computing device 105 or Figure 29 The computing device 2900 is used for execution.

[0230] Figure 3A This is a flowchart of a method 300 for displaying the 3D surface of a tooth position in a manner that displays surface quality during an intraoral scan, according to embodiments of the present disclosure. At block 302 of method 300, processing logic receives one or more intraoral scans of the tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR) 2D images, 2D images generated under ultraviolet light, etc. Each intraoral scan may include three-dimensional information about a captured portion of the tooth position. For example, each intraoral scan may include a point cloud. In various embodiments, each intraoral scan includes three-dimensional information (e.g., x, y, z coordinates) of a plurality of points on the dental surface. Each of the plurality of points may correspond to a spot or feature of structured light projected onto the tooth position by a structured light projector of the intraoral scanner and captured in an image generated by one or more cameras of the intraoral scanner.

[0231] At block 304, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position. This may include registering and stitching multiple intraoral scans together and / or registering and stitching one or more intraoral scans to an already generated 3D surface to update the 3D surface. The registration and stitching process may be performed as described in more detail above. As other intraoral scans are received, these intraoral scans may be registered and stitched to the 3D surface to add more information about regions / parts of the 3D surface and / or improve the quality of one or more regions / parts of an existing 3D surface. In some embodiments, the generated surface is an approximate surface whose quality may be lower than that of a surface calculated later.

[0232] At box 306, the processing logic determines the surface quality fraction of one or more regions of the 3D surface. In some embodiments, the 3D surface is divided into multiple regions, which may have the same size or different sizes. For example, in one embodiment, the 3D surface is divided into regions of approximately mm in size (e.g., a 1 mm square region, a 1 mm diameter region, a 1 mm cube region, etc.). Alternatively, larger or smaller regions may be used. A surface quality fraction may be determined for each region. This may include determining a first surface quality fraction for a first region of the 3D surface. In various embodiments, the processing logic determines the surface quality fraction of a single point on the surface and / or a group of points on the surface. Thus, the size of the region can range from a single point location to an entire tooth, a group of multiple teeth, or more. In some embodiments, regions are determined by grouping adjacent and / or nearby points with the same or similar surface quality fractions together.

[0233] Various techniques can be used to determine the surface quality score of a 3D surface region. In some instances, intraoral scans are scored, and the scores from the intraoral scans are used to determine the score of a relevant region generated based on those scans. In some instances, individual points on the intraoral scans are scored, and the scores of these points are applied to relevant points on the 3D surface. In some instances, 2D images and / or points on 2D images are scored, and these scores are applied to relevant points on the 3D surface. In some instances, the regions are scored at least in part based on multiple color and / or NIRI images depicting these regions. Multiple color and / or NIRI images depicting a region can be identified, and if the number is below a threshold, the user can be notified to generate more images and / or a low surface quality score can be indicated. In some instances, individual points on the 3D surface are scored, and the region is determined based on grouping points scored in a similar manner. In some embodiments, data associated with regions of the 3D surface of a tooth position is fed into a trained machine learning model that outputs one or more surface quality scores for the region. In some instances, multiple measurement data points in a region are counted, and the surface quality fraction of the region is determined based at least on the number of points (e.g., the number of points per square millimeter).

[0234] Different scoring criteria can be applied to determine the surface quality score of a region on a 3D surface. Some information that can be used to calculate the surface quality score includes the point density of the region, the quality score of the points constituting the region, the roughness of the 3D surface at the region, the distance between the region and one or more cameras generating an intraoral scan including the points in the region, the distance between multiple cameras capturing one or more points in the region, the distance between the structured light projector and the cameras capturing one or more points in the region, the number of cameras capturing one or more points in the region, the spot size of one or more points in the region, the angle of the 3D surface at the region relative to the intraoral scanner, the material type of the tooth at the region, etc. Several different techniques for determining the surface quality score are discussed with reference to the following figures, any of which can be applied at box 306.

[0235] At box 308, the processing logic determines one or more visualizations for each region assigned to the 3D surface based on its surface quality score. This may include determining a first visualization for a first region of the 3D surface based on the surface quality score determined for that first region. In some embodiments, the processing logic inputs the surface quality score into a function, and the function outputs a visualization for the surface quality score. In some embodiments, the processing logic performs a lookup in a lookup table that associates different surface quality scores with different visualizations. For example, the lookup table may include multiple entries, each of which can associate a specific visualization with a range of surface quality score values. In some embodiments, different gauges are used to determine the visualization for the selected surface quality score, where the gauge may be selected based on the characteristics of the regions of the 3D surface and / or based on the classification of the regions of the 3D surface. For example, a region classified as a preparing tooth may be associated with a first gauge, while a region classified as a standard tooth may be associated with a second gauge. In another example, a region classified as an edge line may be associated with a first gauge, a region classified as a preparing tooth may be associated with a second gauge, and a region classified as a standard tooth may be associated with a third gauge.

[0236] In one example, the surface quality score is associated with a color value. Therefore, a color can be selected for a region based on its surface quality score. For example, a low surface quality score might be represented using a red family, a slightly higher surface quality score using an orange family, a medium surface quality score using a yellow family, a high surface quality score using a green family, and so on. In another example, the surface quality score is associated with a transparency value. For example, a low surface quality score might be associated with a high transparency value, while a high surface quality score might be associated with a low transparency value. Thus, as the surface quality score of a region improves, the transparency level used to represent that region might gradually decrease. In yet another example, the surface quality score can be associated with flash or flicker frequency. For example, a low surface quality score might be associated with a high flash frequency, while a high surface quality score might be associated with a low flash frequency, such as being associated with no flash. Any of these visualization techniques and / or other visualization techniques can be used to represent the surface quality of regions of a 3D surface.

[0237] At box 310, the processing logic outputs one or more views of the 3D surface to the display. Each region of the 3D surface can be shown using a corresponding visualization determined for that region based on its surface quality score. Thus, a first region can be shown in one or more views using a visualization associated with its surface quality score.

[0238] At box 312, the processing logic determines whether the scan is complete. If the scan is not complete (e.g., more intraoral scans are still being generated), the method returns to box 302 and receives and processes additional intraoral scans. This may result in an update of the 3D surface at box 304 and may result in a change in the surface quality score of one or more regions of the 3D surface at box 306. In various embodiments, as other intraoral scans containing information about these regions are received and processed, the surface quality score of the regions may gradually change over time, and the 3D surface is updated based on these intraoral scans. As the surface quality scores gradually change, the visualization associated with these surface quality scores also gradually changes. This allows the user of the intraoral scanner to receive real-time feedback on the surface quality and to know whether they are making progress during the scan or whether they are stuck in a region without improving the surface quality of that region.

[0239] Figure 3B This is a flowchart of a method 350 for performing actions based on the surface quality fraction of a 3D surface according to an embodiment of this disclosure. At block 352 of method 350, processing logic receives one or more intraoral scans of a tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR) 2D images, 2D images generated under ultraviolet light, etc. Each of the plurality of points may correspond to a spot or feature of structured light projected onto the tooth position by a structured light projector of the intraoral scanner and captured in an image generated by one or more cameras of the intraoral scanner.

[0240] At box 354, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position. This may include registering and stitching multiple intraoral scans together and / or registering and stitching one or more intraoral scans to an already generated 3D surface to update the 3D surface.

[0241] At box 356, the processing logic determines the surface quality fraction of one or more regions of the 3D surface, as discussed in box 306 of reference method 300.

[0242] The processing logic can determine one or more visualizations for each region of the 3D surface based on its surface quality score, and can output one or more views of the 3D surface to a display. Each region of the 3D surface can be shown using a corresponding visualization determined for that region based on its surface quality score. Thus, a first region can be shown in one or more views using a visualization associated with its surface quality score. In some embodiments, no visualization based on surface quality score is output to the display.

[0243] At block 358, the processing logic may determine whether one or more first criteria for a first region are met, which may be a region currently being scanned. The first criteria may be associated with criteria such as the amount of scans already performed on the first region, the amount of scans generated for the first region, and the amount of time spent scanning the first region. In various embodiments, the first criteria may additionally include a criterion that the first region is a region currently being scanned. If the first region meets one or more first criteria, the method may proceed to block 359. Otherwise, the method may proceed to block 362.

[0244] At box 359, the processing logic may determine whether the surface quality score of the first region meets one or more second criteria. The one or more second criteria may include one or more surface quality score criteria, such as a criterion that the surface quality score of the region meets a surface quality threshold, a criterion that the surface quality score improves by at least a threshold amount over time, etc. In one example, the processing logic may determine that the surface quality score of the first region fails to improve within a threshold amount of time and / or determine that the surface quality score of the first region is below a surface quality threshold and / or determine that additional intraoral scanning of the first region will not improve the surface quality score of the region. In one embodiment, to determine that the surface quality score of the first region fails to meet one or more criteria, the processing logic will determine that the additional surface quality score of one or more additional regions adjacent to the first region is equal to or higher than the surface quality threshold, while the surface quality score of the first region is lower than the surface quality threshold.

[0245] If the surface quality fraction of the first region fails to meet one or more surface quality fraction criteria, the method continues to box 360. If one or more surface quality fraction criteria of the first region are not met, the method continues to box 362.

[0246] At box 360, the processing logic outputs a notification associated with a scan of the first region. In one embodiment, the processing logic determines one or more scan recommendations that, if implemented, would result in an improvement in the surface quality score of the first region. The notification may include one or more scan recommendations. In one embodiment, the notification includes a notification to stop generating a new intraoral scan of the first region, a notification to continue to the next region, and / or a notification that the surface quality score of the first region has not improved over a period of time. In one embodiment, in response to determining that the surface quality score of the first region fails to meet one or more criteria, the processing logic automatically increases the scaling setting of the first region. In one embodiment, the intraoral scanner includes a plurality of cameras, each camera being at least one of having a different position or orientation within the intraoral scanner. The 2D processing logic may receive a plurality of two-dimensional (2D) images, each 2D image generated by a different camera among the plurality of cameras; determine one of the plurality of 2D images associated with improved intraoral scan quality; and output the determined 2D image to a display, wherein, in response to the output of the determined 2D image, a physician using the intraoral scanner will reposition the intraoral scanner in a manner that results in the improved intraoral scan quality.

[0247] At box 362, the processing logic determines whether the scan is complete. If the scan is not complete (e.g., more intraoral scans are still being generated), the method returns to box 352 and receives and processes additional intraoral scans. This may result in an update of the 3D surface at box 354 and may result in a change in the surface quality fraction of one or more regions of the 3D surface at box 356. In various embodiments, as other intraoral scans containing information about these regions are received and processed, the surface quality fraction of the regions may gradually change over time, and the 3D surface may be updated based on these intraoral scans.

[0248] Figure 4 This is a flowchart of a method 400 for displaying the 3D surface of a tooth position in a manner that displays surface quality during an intraoral scan, according to embodiments of the present disclosure. At block 402 of method 400, processing logic receives one or more intraoral scans of the tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR) 2D images, 2D images generated under ultraviolet light, etc. Each intraoral scan may include three-dimensional information about a captured portion of the tooth position. For example, each intraoral scan may include a point cloud. In various embodiments, each intraoral scan includes three-dimensional information (e.g., x, y, z coordinates) of a plurality of points on the dental surface. Each of the plurality of points may correspond to a spot or feature of structured light projected onto the tooth position by a structured light projector of the intraoral scanner and captured in an image generated by one or more cameras of the intraoral scanner.

[0249] At box 403, the processing logic determines a quality score for each point in the received intraoral scan. Various techniques can be used to determine the quality score for each point in the intraoral scan. In some embodiments, the intraoral scan is fed into a trained machine learning model that outputs one or more quality scores for each point in the intraoral scan. Different scoring criteria can be applied to determine the quality score for each point in the intraoral scan. Some information that can be used to calculate the quality score of a point includes the distance between the point and one or more cameras that generated the intraoral scan, the distance between multiple cameras that captured the point, the spacing between the structured light projector that projected the point and the camera that captured the point, the number of cameras that captured the point, the spot size associated with the point, the angle of the 3D surface at the point relative to the intraoral scanner, the material type of the tooth at the point, etc. Several different techniques for determining the quality score are discussed with reference to the following figures, any of which can be applied at box 403.

[0250] At box 404, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position. This may include registering and stitching multiple intraoral scans together and / or registering and stitching one or more intraoral scans to an already generated 3D surface to update the 3D surface. The registration and stitching processes can be performed as described in more detail above. Through stitching, information from one or more points from the intraoral scans can be added to the 3D surface. As other intraoral scans are received, these scans can be registered and stitched to the 3D surface to add more point information to the 3D surface.

[0251] At box 406, the processing logic determines the surface quality score of one or more regions of the 3D surface. The surface quality score of a region can be determined at least in part based on a) the number of points associated with that region and b) the quality score of the points associated with that region. Other criteria may also be used for scoring, such as the surface roughness at that region. In various embodiments, the size of the region can range from a single point location to an entire tooth or a group of multiple teeth. In some embodiments, regions are determined by grouping adjacent and / or nearby points with the same or similar surface quality scores together. Several different techniques for determining surface quality scores are discussed with reference to the following figures, any of which can be applied at box 406.

[0252] In some embodiments, the processing logic determines the surface quality fraction of a region without actually generating a 3D surface including those regions. The processing logic may generate such a surface quality fraction based on the quality fractions of points from multiple intraoral scans and / or 2D images that depict the same general region or location of the tooth position. Estimating the surface quality fraction without actually generating a 3D surface improves the speed of surface quality fraction estimation.

[0253] In some embodiments, intraoral scanning initially lacks depth (z) information, and a correspondence algorithm is used to determine the depth information. Calculating depth information using a correspondence algorithm can be a processor-intensive task. Therefore, in some embodiments, a quality score is determined for each point before determining depth information for these points (e.g., based on 2D information). This may provide a coarse quality value for the points. These coarse quality values ​​can then be used to very quickly determine the surface quality score of the region. After the depth information of the points has been determined, an updated quality score for the points with higher accuracy can then be calculated, and this updated quality score can be used to refine the surface quality score of regions on the 3D surface.

[0254] At box 408, the processing logic determines one or more visualizations for each region assigned to the 3D surface based on its surface quality score. This may include determining a first visualization for a first region of the 3D surface based on the surface quality score determined for that first region. In some embodiments, the processing logic inputs the surface quality score into a function, and the function outputs a visualization for the surface quality score. In some embodiments, the processing logic performs a lookup in a lookup table that associates different surface quality scores with different visualizations. For example, the lookup table may include multiple entries, each of which can associate a specific visualization with a range of surface quality score values. In some embodiments, different gauges are used to determine the visualization for the selected surface quality score, where the gauge may be selected based on the characteristics of the regions of the 3D surface and / or based on the classification of the regions of the 3D surface. For example, a region classified as a preparing tooth may be associated with a first gauge, while a region classified as a standard tooth may be associated with a second gauge. In another example, a region classified as an edge line may be associated with a first gauge, a region classified as a preparing tooth may be associated with a second gauge, and a region classified as a standard tooth may be associated with a third gauge.

[0255] At box 410, the processing logic outputs one or more views of the 3D surface to the display. Each region of the 3D surface can be shown using a corresponding visualization determined based on the surface quality score of that region. Thus, a first region can be shown in one or more views using a visualization associated with the surface quality score of the first region.

[0256] At box 412, the processing logic determines whether the scan is complete. If the scan is not complete (e.g., more intraoral scans are still being generated), the method returns to box 402 and receives and processes additional intraoral scans. This may result in an update of the 3D surface at box 404 and may result in a change in the surface quality score of one or more regions of the 3D surface at box 406. In various embodiments, as other intraoral scans containing information about these regions are received and processed, the surface quality score of the regions may gradually change over time, and the 3D surface is updated based on these intraoral scans. As the surface quality scores gradually change, the visualization associated with these surface quality scores also gradually changes. This allows the user of the intraoral scanner to receive real-time feedback on the surface quality and to know whether they are making progress during the scan or whether they are stuck in a certain region without improving the surface quality of that region.

[0257] In various embodiments, different gauges may be applied to translate surface quality fractions into different visualizations. Each gauge may be associated with a specific type of treatment to be performed on the tooth surface category and / or information being scanned. Figure 5 This is a flowchart of a method 500 for displaying the 3D surface of a tooth position during an intraoral scan, according to embodiments of the present disclosure, using different gauges to show different surface quality fractions for different regions. At block 501 of method 500, processing logic can determine whether the intraoral scan is for orthodontic treatment or for restorative treatment. In various embodiments, the different gauges used to translate the surface quality fraction into visualization can be applied to different types of treatment. For example, different surface quality thresholds can be used to assess the adequacy of the 3D surface generated for orthodontic treatment compared to the surface quality of the 3D surface generated for restorative treatment.

[0258] At box 502, the processing logic receives one or more intraoral scans of the tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR) 2D images, 2D images generated under ultraviolet light, etc. Each intraoral scan may include three-dimensional information about a captured portion of the tooth position. For example, each intraoral scan may include a point cloud. In various embodiments, each intraoral scan includes three-dimensional information (e.g., x, y, z coordinates) of multiple points on the dental surface. Each of the multiple points may correspond to a spot or feature of structured light projected onto the tooth position by a structured light projector of the intraoral scanner and captured in an image generated by one or more cameras of the intraoral scanner.

[0259] At box 504, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position. This may include registering and stitching multiple intraoral scans together and / or registering and stitching one or more intraoral scans to an already generated 3D surface to update the 3D surface. The registration and stitching process can be performed as described in more detail above. As other intraoral scans are received, these intraoral scans can be registered and stitched to the 3D surface to add more information about regions / parts of the 3D surface and / or improve the quality of one or more regions / parts of the existing 3D surface.

[0260] At box 506, the processing logic determines the surface quality fraction of one or more regions of the 3D surface. This may include determining a first surface quality fraction for a first region of the 3D surface. In various embodiments, the processing logic determines the surface quality fraction of a single point on the surface and / or a group of points on the surface. Thus, the size of the region can range from a single point location to an entire tooth or a group of multiple teeth. In some embodiments, regions are determined by grouping adjacent and / or nearby points with the same or similar surface quality fractions together, as discussed in more detail with reference to other figures.

[0261] At box 510, processing logic may determine the classification of one or more regions of a 3D surface, which may include determining a first classification of a first region of the 3D surface. In one embodiment, a user manually marks a region on the 3D surface as a tooth preparation area. Alternatively, a user may instruct the user to begin scanning a tooth preparation area, and the scanned region may be marked as a tooth preparation area upon receiving such user input. In one embodiment, the 3D surface is classified and / or segmented using one or more trained machine learning models. In some embodiments, processing logic uses one or more trained machine learning models (e.g., neural networks) to perform classification of tooth positions, where at least one category is for a restoration object. Other categories of dental objects that can be identified include teeth, gums, tooth preparation areas (which may be considered a restoration object), margins, palate, tongue, lips, gingival-dental lines (referred to as transgingival contours), etc. In various embodiments, the machine learning model receives data associated with a region of the 3D surface, which may include the 3D surface, one or more intraoral scans depicting the region, one or more 2D images depicting the region, at least one height map of the region, etc. The machine learning model may process the data and output a dental object classification for the region. The trained machine learning model can perform image-level classification / scan-level classification, pixel-level classification, or classification of groups of pixels. In various embodiments, classification is performed using a trained machine learning model, as discussed in, for example, U.S. Application No. 17 / 230,825, filed April 14, 2021, the entire contents of which are incorporated herein by reference.

[0262] At box 512, the processing logic determines a first gauge for determining the visualization associated with a first surface quality score of the first region. The first gauge may be determined based on at least one of the type of treatment to be performed (e.g., restorative versus orthodontic treatment) or the dental object category determined for the first region. For example, accuracy may be particularly important for certain areas of the dental arch when performing an intraoral scan. Specifically, for restorative treatment, one or more teeth may be prepared with a margin line. Generally, the level of accuracy required for preparing the tooth is higher than the level of accuracy required for producing a dental restoration (e.g., crown, bridge, cap, etc.) that fits correctly onto the prepared tooth. Additionally, the level of accuracy required for preparing the margin line of the tooth may be higher than the level of accuracy required for preparing the rest of the tooth. Thus, different gauges may be applied depending on the type of treatment and / or the dental object category. Each evaluation gauge may include one or more surface quality thresholds (e.g., an upper surface quality threshold, where any region with a surface quality that at least meets the upper surface quality threshold is considered sufficient), one or more functions for determining the visualization based on the surface quality score, one or more lookup tables for determining the visualization based on the surface quality score, etc. Therefore, depending on the gauge applied to that surface quality fraction, the same surface quality fraction can be translated into different visualizations. In the example, a surface quality fraction may be sufficient for a standard tooth and may be shown in green when applied to a prepared tooth; while the same surface quality fraction may be insufficient for a prepared tooth and may be shown in yellow, orange, or red when applied to a prepared tooth.

[0263] At box 514, the processing logic determines a first visualization of a first region based on a first surface quality score and a first gauging. This may include determining one or more functions, lookup tables, thresholds, etc., from the first gauging to apply to the first surface quality score. The first gauging may indicate one or more first techniques for visualizing the surface quality score. For example, the first gauging may indicate that the surface quality score should be shown based on the first of color, texture, shading, transparency, flash, pointers, arrows, markers, etc. In some embodiments, the first dental object category is teeth, and the second dental object is gingiva. How a region of a 3D surface is visualized based on its surface quality score may differ for different types of tissue (e.g., dental tissue versus gingival tissue). Because gingival tissue is generally less important than dental tissue in orthodontic treatment, a low surface quality score for gingiva may still be shown using visualizations indicating a high or sufficient surface quality score. Similarly, the palatal region may be less important than dental tissue, and a low surface quality score may be shown as sufficient. In one embodiment, different gaugings are associated with different quality thresholds. For example, a surface quality threshold of 40 points per cubic millimeter can be used for teeth, 20 points per cubic millimeter for gingiva, and 10 points per cubic millimeter for the palate. For instance, a tooth region with 30 points per cubic millimeter might be shown in yellow or orange (indicating medium quality), while a gingival or palatal region with 30 points per cubic millimeter (indicating high quality) might be shown in green. For some treatments, such as orthodontics, the gingival-dental line (the line where teeth intersect the gingiva) can be important for proper fit of the aligner. Therefore, in some instances, a dental object category for the gingival-dental line can be assigned to a region, and the gauge associated with the gingival line might have a higher threshold for surface quality values. For example, the gingival-dental line might have a surface quality threshold of 50 points per cubic millimeter.

[0264] At box 516, the processing logic determines a second surface quality score for a second region of the 3D surface. In one embodiment, the operation of box 516 is performed at box 506. At box 518, the processing logic may determine a second classification for the second region, wherein the second classification of the second region differs from the first classification of the first region. In one embodiment, the operation of box 518 is performed at box 510 using one or more trained machine learning models.

[0265] At box 520, the processing logic determines a second visual gauge for determining the second surface quality fraction associated with the second region. The second gauge may be determined based on at least one of the type of treatment to be performed (e.g., restorative treatment versus orthodontic treatment) or a dental object category determined for the second region. In various embodiments, the second gauge to be used for the second region differs from the first gauge to be used for the first region.

[0266] At box 522, the processing logic determines a second visualization of the second region based on a second surface quality score and a second gauge. This may include determining one or more functions, lookup tables, thresholds, etc., from the second gauge to apply to the second surface quality score. The second gauge may indicate one or more second techniques for visualizing the surface quality score. For example, the second gauge may indicate that the surface quality score should be shown based on a second of color, texture, shadow, transparency, flashing, pointers, arrows, markers, etc. For example, the surface quality score in the first region may be depicted using color and / or transparency levels, and the surface quality score in the second region may be depicted using flashing.

[0267] At box 524, the processing logic outputs one or more views of the 3D surface to a display. Each region of the 3D surface can be displayed using a corresponding visualization determined for that region based on its surface quality score and / or a gauge applied to that region. Thus, a first region can be shown using a first visualization associated with a first surface quality score as applied to one or more criteria of a first gauge, and a second region can be shown using a second visualization associated with a second surface quality score as applied to a second first gauge. In one example, the first surface quality score of the first region may be the same as the second surface quality score of the second region. However, because different gauges are applied to the first and second regions, the first visualization for the first region may differ from the second visualization for the second region, even though the first and second regions have the same surface quality score.

[0268] Figure 6 This is a flowchart of a method 600 for classifying regions of a 3D surface of a tooth and determining a surface quality score associated with each region using one or more trained machine learning models, according to embodiments of the present disclosure. At block 602 of method 600, processing logic inputs data associated with one or more regions of the 3D surface into one or more trained machine learning models.

[0269] In one embodiment, at block 604, the processing logic receives output from one or more machine learning models that classify one or more regions of a 3D surface (and / or intraoral scan and / or tooth position) into one or more dental categories. In one embodiment, one or more trained machine learning models are trained to perform dental category classification and / or segmentation on images, intraoral scans, and / or 3D surfaces. In various embodiments, the processing logic may input intraoral scans, 2D images, 3D surfaces, projections of 3D surfaces onto one or more planes, points from the 3D surfaces, and / or other data into the trained machine learning models. One implementation uses a deep neural network to learn how to map input images, intraoral scans, and / or 3D surfaces to human-labeled dental categories, where dental categories include regular teeth and one or more restoration objects. The result of this training is a trained machine learning model that can predict labels directly from the input scan data and / or 3D surface data. Input data can be individual intraoral scans (e.g., height maps), 3D surface data (e.g., 3D surfaces from multiple scans or projections of such 3D surfaces onto a plane), and / or other images (e.g., color images and / or NIRI images). Such data can be available in real time concurrently with the scan. Additionally, the intraoral scan data associated with individual scans can be large enough (e.g., the scanner can have a sufficiently large FOV) to include at least one tooth and its surrounding environment. Given input based on a single intraoral scan, a trained neural network is able to predict whether the scan (e.g., a height map) contains any of the dental categories described above. The nature of this prediction can be probabilistic: each category has a probability of being presented on the intraoral scan. This approach allows the system to identify regions on the 3D surfaces and / or 3D models generated from the intraoral scan that are relevant to the object being restored and therefore should be treated differently from natural teeth.

[0270] In one embodiment, at block 606, the processing logic receives output from one or more trained machine learning models, the output including one or more regions of a 3D surface and / or one or more intraoral scans of surface quality scores.

[0271] In various embodiments, one or more machine learning models are trained to perform one or both of the operations of boxes 604 and 606. Each task can be performed by a separate machine learning model. Alternatively, a single machine learning model can perform each task or a subset of tasks. In one example, one or more machine learning models can be trained, wherein the trained ML model is a single shared neural network with multiple shared layers and multiple higher-level, distinct output layers, wherein each output layer outputs a different prediction, classification, label, etc.

[0272] One type of machine learning model that can be used to perform some or all of the tasks mentioned above is an artificial neural network, such as a deep neural network. Artificial neural networks typically include feature representation components with classifier or regression layers that map features to a desired output space. For example, a convolutional neural network (CNN) hosts multiple convolutional filter layers. Pooling and nonlinear processing are performed at lower layers, often topped by a multilayer perceptron, which maps the top-level features extracted by the convolutional layers to a decision (e.g., a classification output). Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each subsequent layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure of layers, where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and comprehensive representation. For example, in image recognition applications, the raw input can be a pixel matrix; the first representation layer can abstract the pixels and encode edges; the second layer can construct and encode the arrangement of edges; the third layer can encode higher-level shapes (e.g., teeth, lips, gums, etc.); and the fourth layer can identify the scanned character. It's worth noting that the deep learning process can learn on its own which features will be optimally placed at which level. The "depth" in "deep learning" refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a large Credit Assignment Path (CAP) depth. CAP is the transformation chain from input to output. CAP describes the underlying causal relationship between the input and output. For feedforward neural networks, the CAP depth can be the network depth and can be the number of hidden layers plus one. For recurrent neural networks where signals may propagate through layers more than once, the CAP depth may be unlimited.

[0273] Training neural networks can be achieved through supervised learning, which involves feeding the network a training dataset consisting of labeled inputs, observing its output, constraining the error (by measuring the difference between the output and the label values), and using techniques such as deep gradient descent and backpropagation to tune the network's weights across all its layers and nodes to minimize the error. In many applications, repeating this process across many labeled inputs in the training dataset produces a network that can produce correct outputs when presented with inputs different from those present in the training dataset. This generalization can be achieved in high-dimensional settings such as large images when sufficiently large and diverse training datasets are available.

[0274] Figure 7This is a flowchart of a method 700 for determining a surface quality score associated with a 3D surface of a tooth position using a trained machine learning model, according to embodiments of the present disclosure. At block 702, processing logic inputs data from one or more intraoral scans (e.g., points from one or more intraoral scans and / or the intraoral scans themselves) into one or more trained machine learning models. At block 704, the processing logic receives output from the trained machine learning model (e.g., a trained neural network), which includes a quality score for each point from the input one or more intraoral scan points.

[0275] Figure 8 This is a flowchart of a method 800 for determining a surface quality score associated with a 3D surface of a tooth position according to embodiments of the present disclosure. At block 802, the method 800 includes: determining a quality score for each point of a received intraoral scan. The points of the intraoral scan may correspond to points (e.g., spots or features) of structured light projected onto the tooth position and captured by one or more cameras of an intraoral scanner. The points of the intraoral scan may correspond to x, y, z coordinates measured by a camera or other optical sensor of the intraoral scanner, where z may represent the distance from the intraoral scanner. The quality score of a point may represent the error and / or confidence level associated with that point. Many different criteria can be used to score points, some of which are listed herein.

[0276] In some embodiments, the intraoral scanner includes a plurality of cameras, some or all of which can capture a point during the generation of the intraoral scan. In one embodiment, at block 804, to determine the quality score of a point in the intraoral scan and / or 3D surface, processing logic determines the distance between the point and the intraoral scanner (e.g., at least one of a first or second camera of the intraoral scanner that captures the point during the generation of the intraoral scan). The processing logic may additionally determine the distance between the first camera that captured the point and the second camera that also captured the point and / or one or more additional cameras that may have captured the point. In various embodiments, the maximum distance between any two cameras that captured the point is determined. The processing logic may determine the triangulation angle of the point based on the distance from the point and the distance between the cameras that captured the point. The lower the distance from the point and the greater the distance between the cameras, the larger the triangulation angle, and therefore the higher the accuracy of the determined distance from the point. Thus, the quality score can be directly proportional to the distance between the cameras or inversely proportional to the distance from the point. In one embodiment, the function e=z is used. 2 / bf determines the error of the determined distance from the point, where e is the error of the determined distance from the point, z is the determined distance from the point, f is the focal length of the camera, and b is the reference (the distance between the two cameras). In various embodiments, an error associated with the triangulation angle of the point is determined, and this error is used to calculate the mass fraction of the point, where the mass fraction is inversely proportional to the error.

[0277] In some embodiments, the intraoral scanner includes one or more structured light projectors that project a structured light pattern onto a dental surface; and one or more cameras that capture images of the structured light on the dental surface. In one embodiment, at block 806, to determine the quality fraction of a point in the intraoral scan and / or 3D surface, processing logic determines the distance between the point and the intraoral scanner (e.g., the camera of the intraoral scanner that captures the point when generating the intraoral scan). The processing logic may additionally determine the distance between the camera that captures the point and the structured light projector that projects the light from the point onto the tooth position. The processing logic may determine the triangulation angle of the point based on the distance from the point and the distance between the camera and the structured light projector. The lower the distance from the point and the larger the distance between the camera and the structured light projector, the larger the triangulation angle, and therefore the higher the accuracy of the determined distance from the point. Thus, the quality fraction can be directly proportional to the distance between the camera and the structured light projector or inversely proportional to the distance from the point. In one embodiment, the function e=z is used. 2 / bf determines the error of the determined distance from the point, where e is the error of the determined distance from the point, z is the determined distance from the point, f is the focal length of the camera, and b is the reference (the distance between the camera and the structured light projector). In various embodiments, an error associated with the triangulation angle of the point is determined, and this error is used to calculate the mass fraction of the point, where the mass fraction is inversely proportional to the error.

[0278] In some embodiments, the intraoral scanner includes a plurality of cameras, wherein some or all of the cameras can capture a point during the generation of the intraoral scan. The more cameras that capture the point and agree or nearly agree on its coordinates, the higher the confidence level of the point and the lower the error associated with the point. In one embodiment, at block 808, in order to determine the quality score of a point in the intraoral scan and / or 3D surface, processing logic determines the number of cameras that capture the point. The quality score of the point may be proportional to the number of cameras that capture the point, wherein the quality score increases as the number of cameras capturing the point increases.

[0279] In some embodiments, a structured light projector projects structured light / patterned light, including a spot and / or other shapes / features, onto a tooth position. In various embodiments, the spot size is a function of the distance from the intraoral scanner. The size of the spot or other feature typically increases with the distance from the intraoral scanner. The processing logic may include calibration data indicating approximate spot / feature sizes that should be detected at various distances. For example, a spot projected onto a surface at a first relatively close distance may have a first general spot size, a spot projected onto a surface at a second distance greater than the first distance may have a larger second general spot size, and a spot projected onto a surface at a third distance greater than the second distance may have an even larger third general spot size. At block 810, the processing logic may predict the spot size at the point projected onto the tooth position based on the determined distance to that point. The processing logic may measure the spot size at that point and then compare the measured spot size projected onto that point with the predicted spot size. A quality score may be determined based on the difference between the estimated spot size and the measured spot size. The quality score may be indirectly proportional to the difference. Therefore, as the difference increases, the quality score at that point may decrease.

[0280] In some embodiments, a structured light projector projects structured light / patterned light, including a spot and / or other shapes / features (e.g., a checkerboard pattern including checkerboard features), onto the tooth position. In various embodiments, the intensity of the spot or other projected feature is a function of the distance from the intraoral scanner. The intensity of the spot or other feature generally decreases with distance from the intraoral scanner. The processing logic may include calibration data indicating approximate spot / feature intensities that should be detected at various distances. For example, a spot projected onto a surface at a first relatively close distance may have a first general intensity, a spot projected onto a surface at a second distance greater than the first distance may have a lower second intensity, and a spot projected onto a surface at a third distance greater than the second distance may have an even lower third intensity. At block 811, the processing logic may predict the intensity of the spot / feature at the point projected onto the tooth position based on the determined distance to that point. The processing logic may measure the intensity at that point and then compare the measured intensity of the projected spot / feature at that point with the predicted intensity. A quality fraction may be determined based on the difference between the estimated intensity and the measured intensity. The quality score may be indirectly proportional to the difference. Therefore, as the difference increases, the quality score at that point may decrease.

[0281] In some embodiments, the accuracy of the distance coordinates of a point varies based on the angle between the surface at that point and the camera capturing the image of that point and / or the light source projecting structured light onto that point. At block 812, the processing logic can determine the angle between the normal to the 3D surface at that point and the imaging axis of the camera capturing that point during intraorbital scanning. This angle can then be used to determine the quality fraction of that point. As the angle increases, the quality fraction of the point decreases.

[0282] Figure 9 The illustration shows an angle-based scoring of a tooth position region according to an embodiment of the present disclosure. Figure 9 An intraoral scanner 910 is shown (e.g., it may correspond to...). Figure 1 The intraoral scanner 910 and the ray 912 of one or more cameras that roughly correspond to the imaging axis of the intraoral scanner 910. Figure 9 The scanned tooth position 915 is also shown. In a first region associated with a first point set 920, a first normal 921 of the surface of tooth position 915 is approximately parallel to ray 912. Therefore, the first point set will have high accuracy, and the quality score associated with the first point set will be relatively high. In a second region associated with a second point set 922, a second normal 923 of the surface of tooth position 915 is approximately at 45 degrees to ray 912. Therefore, the second point set will have moderate accuracy, and the quality score associated with the second point set 922 will be lower than the quality score of the first point set 920. In a third region associated with a third point set 924, a third normal 925 of the surface of tooth position 915 is approximately at 90 degrees to ray 912. Therefore, the third point set will have low accuracy, and the quality score associated with the third point set 924 will be lower than the quality score of the second point set 920.

[0283] Back Figure 8 In some embodiments, the material type of the tooth position may affect the accuracy of the capture point. For example, it may be more difficult to generate accurate scans of highly reflective objects, such as those made of titanium, which may include certain types of restoration objects (e.g., dental implants, abutments, etc.). Therefore, in some embodiments, at block 814, the processing logic determines the material type used for the tooth position and determines the quality score of the point based at least in part on the material of the tooth position associated with that point. In some embodiments, the user inputs the material type. In some embodiments, intraoral scans and optional associated 2D images (such as color images and / or NIR images) are input into a trained machine learning model that classifies tooth positions / objects according to material. The trained machine learning model may be, for example, a convolutional neural network as described above.

[0284] Other parameters or metrics can also be determined for a point and can be used to calculate the quality score for that point. For example, an intraoral scanner may have lower accuracy when capturing points that are beyond a certain distance from the scanner. Therefore, points that are too close to or too far from the scanner (i.e., beyond the optimal scanner distance) may produce lower accuracy and thus can be scored with a lower quality score compared to points within the optimal scanner distance. Additionally, if the scanner moves too quickly during the scan, the resulting scan may be blurry, resulting in points of lower quality. Furthermore, if the tooth position is partially obscured by saliva and / or blood, this may cause the measurement point to not actually be part of the tooth surface. Therefore, in various embodiments, saliva and / or blood detection can be performed on the intraoral scan (e.g., by using a trained machine learning model, such as a neural network, to classify regions as teeth, blood, and / or saliva). Points classified as blood and / or saliva may have a lower surface quality score compared to points classified as teeth. In one embodiment, as described in U.S. Application No. 16 / 809,451, filed March 4, 2020, the detection of blood and / or saliva, the entire contents of which are incorporated herein by reference.

[0285] In some embodiments, multiple parameters or metrics are defined for a point, such as those defined at boxes 804-814 (e.g., parameters / metrics such as triangulation angles relative to the point, the number of cameras capturing the point, the difference between the estimated spot size and the measurement tube sheet size, the difference between the estimated intensity and the measured intensity, the surface angle of the point relative to the imaging axis of the intraoral scanner, material type, etc.). Each of these parameters can provide clues about the accuracy of the point's measurement location (e.g., x, y, and / or z position). In various embodiments, some or all of these parameters (and optionally others) are used to calculate the point's quality score. In various embodiments, some or all of these parameters are combined into a function to statistically estimate the point's quality. Some of the parameters mentioned may have a greater impact on the accuracy of the point's measurement location. Therefore, some parameters may be weighted more heavily than others when determining the point's quality score. In some embodiments, some or all of the techniques discussed herein are used to determine a quality score (e.g., at boxes 804 to 814), and a weighted or unweighted average of the quality scores is calculated to arrive at a final quality score for that point.

[0286] In some embodiments, once some or all of the parameters described above are determined for one or more points of an intraoral scan and / or region of a 3D surface, regression analysis can be performed on the one or more points and their associated parameters to determine the final quality score of the one or more points. In some embodiments, once some or all of the parameters described above are determined for one or more points of an intraoral scan and / or region of a 3D surface, the data for the one or more points can be fed into a trained machine learning model, such as a neural network, which can output the final quality score of the one or more points. In one embodiment, the regression analysis and / or machine learning model operates on a single point at a time to output the quality score for that point. In one embodiment, the regression analysis and / or machine learning model operates on multiple points in parallel to determine the quality scores for multiple points, where parameters of one point (e.g., x, y, z position, distance, triangulation angle, surface angle, number of cameras capturing that point, etc.) affect not only the quality score of that point but also the quality scores of other nearby points.

[0287] At box 820, the processing logic can determine a region of the 3D surface (e.g., a 1x1mm square region or a 1x1x1mm cube region) and identify a subset of points associated with that region. At box 824, the processing logic determines the surface quality score of the region based on the quality score of the points included in the region and the number of points contained in the region. The surface quality of the region may be a function of the number of points in the region and the error level of each of those points. For example, the surface quality score of a 1mmx1mmx1mm voxel region can be estimated by calculating the following formula: It is the standard error of the weighted mean, where E is the surface quality fraction representing the error of the region, i is a given point, and e is the quality fraction representing the error of the given point.

[0288] As the quality fraction of points in the region increases, the surface quality fraction of the region also increases. Additionally, as the number and / or density of points in the region (which is related to the resolution of the region) increases, the surface quality fraction of the region also increases. In some embodiments, the surface quality fraction of a region may also depend on the surface roughness of the region. The 3D surface at the region can be determined by determining the surface as an average of multiple points. The 3D surface may or may not correspond to the actual location of the points. The surface roughness of the region can be calculated by determining the distance between a point in the region and the nearest corresponding location on the 3D surface. The greater the distance, the higher the roughness and the lower the surface quality fraction. In various embodiments, the roughness can be calculated as the average of the distances between a point in the region and the nearest corresponding location on the 3D surface in the region.

[0289] Figure 10 This is a flowchart of a method 1000 for determining the surface quality fraction of a 2D image according to embodiments of the present disclosure. At block 1001 of method 1000, processing logic receives a 2D image of a tooth position. In various embodiments, the 2D image may include a color image and / or a NIR image. An intraoral scanner may interweave the generation of an intraoral scan with the generation of a 2D image. Thus, a 2D image can also be generated along with the generation of an intraoral scan.

[0290] At box 1002, the processing logic determines the number of 2D images associated with a region of the 3D surface to be determined. In various embodiments, the greater the number of 2D images, the more data is available for that region, and the higher the final surface quality score for that region.

[0291] At block 1004, the processing logic determines the image score for each 2D image. In one embodiment, at block 1006, the processing logic determines the angle between the imaging axis of the intraoral scanner and the normal to the 3D surface in that region. At block 1008, the processing logic may determine the image score of the image based at least in part on the angle. In some embodiments, a larger angle results in a lower image score, similar to... Figure 9 The relationship between mass fraction and angle is shown.

[0292] At block 1014, the processing logic may determine the surface quality score of the region based at least in part on the number of 2D images associated with the region of the 3D surface and the quality score of the 2D images associated with that region. In some embodiments, the surface quality score determined at block 1014 may be combined with the surface quality score determined at block 824 of method 800 and / or the surface quality score determined at block 606 of method 600, such as with a weighted average or an unweighted average, to produce a final surface quality score for the region.

[0293] Figure 11 This is a flowchart of a method 1100 for determining a surface quality score of a region to be determined based on roughness and / or resolution, according to embodiments of the present disclosure. In various embodiments, roughness and resolution can be used to characterize the quality of a 3D surface, where roughness can be a measure of the noise level of the 3D surface, and resolution relates to the ability to reveal the detail of the 3D surface. For a given number of points (e.g., each point has x, y, z coordinates) and noise level, processing logic can reduce roughness by performing a smoothing operation on the 3D surface. However, in various embodiments, performing a smoothing operation reduces resolution. In some embodiments, for a given scan point or region, the value calculated by multiplying roughness by resolution is approximately constant. Thus, increasing resolution may reduce roughness, and increasing roughness may reduce resolution, but the result of multiplying these two values ​​for a given region should remain approximately constant. In some embodiments, processing logic may automatically select a balance between roughness and resolution for each region.

[0294] At block 1101 of method 1100, the processing logic determines the roughness associated with the region to be determined. In one embodiment, the roughness of the region is determined as stated in blocks 1108 through 1110. In one embodiment, at block 1108, the processing logic determines the distance between the nearest point on the 3D surface and one or more intraoral scan points associated with the region of the 3D surface. At block 1110, the processing logic determines the standard deviation of the distance between the nearest point on the 3D surface and one or more intraoral scan points.

[0295] At box 1114, the processing logic determines the resolution associated with the region to be determined. In one embodiment, the resolution is determined by identifying multiple points associated with the region from an intraoral scan. The more points in the region, the higher the resolution of the region.

[0296] At block 1118, the processing logic determines the surface quality score of the region to be determined, at least in part, based on roughness and resolution. The surface quality score may be proportional to the resolution, such that the surface quality score increases with increasing resolution. Additionally, the surface quality score may be inversely proportional to the roughness, such that the surface quality score decreases with increasing roughness. In some embodiments, the surface quality score determined at block 1118 may be combined with the surface quality score determined at block 1014 of method 1000, the surface quality score determined at block 824 of method 800, and / or the surface quality score determined at block 606 of method 600, such as with a weighted average or an unweighted average, to produce a final surface quality score for the region.

[0297] Figure 12The illustration shows a scoring of a tooth region based on roughness and / or point density according to an embodiment of the present disclosure. Figure 12 In the diagram, line 1205 represents the original dental surface, line 1212 represents the generated 3D surface based on an intraoral scan of the original dental surface, and point 1215 is the point captured during the intraoral scan. The first region 1220 of the 3D surface has too few data points and is considered to be pores or voids in the surface. Therefore, line 1212 does not extend into the first region 1220. The second region 1225 of the 3D surface has high smoothness and high resolution due to the large number of points 1215 closely distributed along line 1212, and thus has a high surface quality fraction. The third region 1230 of the 3D surface has a low to moderate number of points 1215 and high roughness because the points 1215 in the third region 1220 are generally far from line 1212, and therefore has a low surface quality fraction.

[0298] Figures 13A to 13C The illustration shows a view of a graphical user interface for an intraoral scanning application including a 3D surface with surface quality feedback, according to an embodiment of the present disclosure. Figure 13A A first view of a graphical user interface 1300 for an intraoral scanning application according to an embodiment of the present disclosure is illustrated. The graphical user interface includes a first 3D surface 1310A of the current field of view (FOV) of the intraoral scanner and a first 2D image 1323A. Additionally, an outline 1350 of the intraoral scanner, including an outline 1351 of the intraoral scanner's field of view, can be shown relative to the 3D surface 1310A. Figure 13B A second view of a graphical user interface 1330 for an intraoral scanning application is illustrated, the second view including a second 3D surface 1310B of the current field of view of the intraoral scanner and a second 2D image 1323B. Additionally, an outline 1350 of the intraoral scanner, including an outline 1351 of the intraoral scanner's field of view, can be shown relative to the 3D surface 1310A. Figure 13CA third view of a graphical user interface 1360 for an intraoral scanning application is illustrated, including a third 3D surface 1310C of the current field of view (FOV) of the intraoral scanner and a third 2D image 1323C. Additionally, the contour 1350 of the intraoral scanner, including the contour 1351 of the intraoral scanner's field of view, can be shown relative to the 3D surface 1310A. The 3D surfaces 1310A, 1310B, and 1310C are generated by registering and stitching together multiple intraoral scans captured during an intraoral scanning session. As each new intraoral scan is generated, the scan is registered to the 3D surface and then stitched together. Thus, with each intraoral scan, the 3D surface becomes increasingly accurate until it is complete. A 3D model can then be generated based on the intraoral scan. 3D surface 1310A is the initial 3D surface of the tooth position shortly after the scan begins. 3D surface 1310B is an updated version of 3D surface 1310A after other intraoral scans have been captured. 3D surface 3100C is an updated version of 3D surface 1310B after other intraoral scans have been captured.

[0299] During intraoral scanning, users of intraoral scanners often find it difficult to determine whether they are making progress while scanning one or more areas of a tooth position. This is especially true for hard-to-capture areas such as posterior molars, teeth with deep depressions, interproximal spaces between teeth, and areas where high accuracy is desired (such as preparation materials). Therefore, in various embodiments, the processing logic continuously calculates surface quality scores for multiple areas of 3D surfaces 1310A to 1310C and determines visualizations for these areas to visually indicate the surface quality scores. In some embodiments, the visualizations used to depict the surface quality scores are based on color and / or shading. For example, green may indicate a high surface quality score, yellow may indicate a medium surface quality score, and red may indicate a low surface quality score.

[0300] like Figure 13AAs shown in the 3D surface 1310A, a first region is depicted using a first visualization 1352 indicating a high surface quality score, a second region is depicted using a second visualization 1354 indicating a medium surface quality score, and a third region is depicted using a third visualization 1356 indicating a low surface quality score. Other visualizations that can be used to indicate surface quality scores, besides color, include transparency levels, flickering or flashing frequencies, etc. If transparency levels are used to indicate surface quality scores, background images, colors, textures, patterns, etc., can be displayed behind the 3D surfaces 1310A to 1310C to make different transparency levels easier to see in some embodiments. In some embodiments, the accurate color of the scanned tooth position is determined, and the determined tooth position color is used to represent the 3D surface. In such embodiments, color is not used to represent surface quality, but other visualization options (such as transparency and / or flickering) can be used to represent surface quality. One advantage of using transparency to mark surface quality is that the natural worst-case scenario for surface quality is no data at all, which is related to 100% transparency. In various embodiments, a smooth transition can be provided between no data (100% transparent), a small amount of data (very transparent), more data (slightly transparent), and sufficient data (completely opaque). In some embodiments, flickering can be performed on transparency (e.g., flickering between transparency levels), color (e.g., flickering between colors, etc.). In various embodiments, the flickering rate may be inversely proportional to the surface quality level.

[0301] In some embodiments, markings, such as arrows pointing to various regions, can be used to indicate surface quality scores. The surface quality score of a region can be reflected in the characteristics of the arrows or other markings pointing to and / or indicating that region. For example, the thickness of the arrow (e.g., where increased thickness indicates a lower surface quality score), the length of the arrow (e.g., where increased length indicates a lower surface quality score), the flashing rate of the arrow (e.g., where increased flashing frequency indicates a lower surface quality score), etc., can be based on the surface quality score of the region the arrow points to. In some embodiments, even in instances where the region itself is not visible, such as if the region is occluded by other surfaces in the current view of the 3D surface, the markings and / or arrows of the region may be visible. In one embodiment, as set forth in U.S. Patent No. 9,510,757, published December 6, 2016, marking regions using markings indicating surface quality scores is provided, the entire contents of which are incorporated herein by reference.

[0302] Figure 13B3D surface 1310B shows an updated 3D surface after the additional intraoral scan has been registered and stitched onto 3D surface 1310A. Therefore, the area shown in 3D surface 1310A using second visualization 1354 can be shown in 3D surface 1320B using first visualization 1352. Additionally, the area shown in 3D surface 1310A using third visualization 1356 can be shown in 3D surface 1320B using second visualization 1354. Additionally, the 3D surface may have grown to other areas of the scanned tooth position.

[0303] Figure 13C The 3D surface 1310C shows an updated 3D surface after the additional intraoral scan has been registered and stitched onto 3D surface 1310B. Therefore, the area shown in 3D surface 1310B using the second visualization 1354 can be shown in 3D surface 1320C using the first visualization 1352. Additionally, the area shown in 3D surface 1310B using the third visualization 1356 can be shown in 3D surface 1320C using the second visualization 1354. Additionally, the 3D surface may have grown to other areas of the scanned tooth position.

[0304] The GUI for intraoral scanning applications can display 2D images 1323A to 1323C in an area of ​​the GUI's display. In various embodiments, the 2D images can be generated at a frame rate of approximately 20 frames per second (updated every 50 milliseconds) to approximately 15 frames per second (updated every 66 milliseconds).

[0305] In one embodiment, as shown, the scan segment indicator 1330 may include a maxillary arch segment indicator 1332, a mandibular arch segment indicator 1334, and an occlusal segment indicator 1336. The maxillary arch segment indicator 1332 may be active (e.g., highlighted) while the maxillary arch is being scanned. Similarly, the mandibular arch segment indicator 1334 may be active while the mandibular arch is being scanned, and the occlusal segment indicator 1336 may be active while the patient's occlusion is being scanned. A user can select specific segment indicators 1332, 1334, and 1336 to display the 3D surface associated with the selected segment. The user can also select specific segment indicators 1332, 1334, and 1336 to indicate that a scan of that specific segment should be performed. Alternatively, the processing logic may automatically determine the segment being scanned and may automatically select that segment to make it active.

[0306] The GUI of the intraoral scanning application may also include a taskbar with multiple operating modes or intraoral scanning stages. Selecting the patient selection mode 1340 allows the physician to input patient information and / or select patients already entered into the system. Selecting the scan mode 1342 enables an intraoral scan of the patient's oral cavity. After the scan is complete, selecting the post-processing mode 1344 prompts the intraoral scanning application to generate one or more 3D models based on the intraoral scan and / or 2D images generated during the intraoral scan, and optionally performs analysis on the 3D models(s). Examples of analyses that can be performed include detecting regions of interest, evaluating the quality of the 3D models(s), etc. Once the physician is satisfied with the 3D models, these 3D models can be used to generate orthodontic and / or dental restoration prescriptions. Selecting the prescription fulfillment mode 1346 allows the generated orthodontic and / or dental restoration prescriptions to be sent to a laboratory or other facility to generate dental restoration devices (e.g., crowns, bridges, prostheses, etc.) or orthodontic devices (e.g., clear aligners).

[0307] Figures 13A to 13C A single 3D surface is shown, with various visualizations used to depict the surface quality score. In some embodiments, multiple copies of the 3D surface (or regions of the 3D surface) are output to the display in parallel (e.g., displayed together in a GUI). For example, one instance of the 3D surface may be monochrome or depict color information of the scanned tooth position, while another instance of the 3D surface may include a visualization showing the surface quality score. In some embodiments, a second instance of the 3D surface is shown when one or more regions have a surface quality score below a surface quality threshold. In some embodiments, the second instance of the 3D surface with the visualization indicating the surface quality score is depicted using a smaller size and / or scaling setting than the first instance of the 3D surface. In one embodiment, a user can select or click on an instance of the 3D surface with the visualization indicating the surface quality score to zoom in on that instance of the 3D surface (e.g., to become the primary or primary instance of the 3D surface). This may also result in a change in the arrangement of the first and second instances of the 3D surface in the display. In some embodiments, more than two instances of the 3D surface are shown. For example, a first instance of a 3D surface with tinting that reflects the color of the dental arch, a second instance of a 3D surface lacking color or with the exact same color, and a third instance of a 3D surface demonstrating surface quality can be output to a display in parallel. At any given time, one of the multiple instances of the 3D surface can be a primary or secondary instance that is larger than the other instances of the 3D surface. At any time, a doctor or other user can select one of the other instances of the 3D surface to make the selected instance the primary or secondary instance (and thus zoom in and / or move it to different positions on the display).

[0308] Figure 14AThis is a flowchart of a method 1400 for providing user assistance during an intraoral scan according to an embodiment of this disclosure. At block 1402 of method 1400, processing logic receives one or more intraoral scans and / or 2D images of a tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR or NIRI) 2D images, 2D images generated under ultraviolet light, etc. Each intraoral scan may include three-dimensional information about a captured portion of the tooth position. For example, each intraoral scan may include a point cloud. In various embodiments, each intraoral scan includes three-dimensional information (e.g., x, y, z coordinates) of a plurality of points on the dental surface. Each of the plurality of points may correspond to a spot or feature of structured light projected onto the tooth position by a structured light projector of the intraoral scanner and captured in an image generated by one or more cameras of the intraoral scanner.

[0309] At block 1404, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position. This may include registering and stitching multiple intraoral scans together and / or registering and stitching one or more intraoral scans to an already generated 3D surface to update the 3D surface. The registration and stitching process may be performed as described in more detail above. As other intraoral scans are received, these intraoral scans may be registered and stitched to the 3D surface to add more information about regions / parts of the 3D surface and / or improve the quality of one or more regions / parts of an existing 3D surface. In some embodiments, the generated surface is an approximate surface whose quality may be lower than that of a surface calculated later.

[0310] At box 1406, processing logic determines a surface quality fraction for one or more regions of the 3D surface. In some embodiments, the 3D surface is divided into multiple regions, which may have the same size or different sizes. For example, in one embodiment, the 3D surface is divided into regions of approximately mm in size (e.g., a 1 mm square region, a 1 mm diameter region, etc.). Alternatively, larger or smaller regions may be used. A surface quality fraction may be determined for each region.

[0311] At box 1408, the processing logic determines one or more visualizations for each region of the 3D surface based on the surface quality score assigned to that region.

[0312] At box 1410, the processing logic outputs one or more views of the 3D surface to the display. Each region of the 3D surface can be shown using a corresponding visualization determined for that region based on its surface quality score. Thus, a first region can be shown in one or more views using a visualization associated with its surface quality score.

[0313] At box 1412, processing logic determines whether the surface quality score of one or more regions is below a surface quality threshold. This may include determining one or more surface quality thresholds to be applied to one or more regions based on the region's classification. For example, region-associated data (e.g., from intraoral scans, 3D surfaces, 2D images, etc.) may be fed into a trained machine learning model that outputs a dental object classification for the region. Different dental object classifications may be associated with different thresholds. In one embodiment, a gauge is determined based on the dental object classification of the region, wherein the gauge includes the surface quality threshold to be applied. If the surface quality score of the region is below the determined surface quality threshold, the method continues to box 1414. If the surface quality score of the region(s) is equal to or greater than the determined surface quality threshold, the method may continue to box 1415. In one embodiment, the region to be determined corresponds to a region of a tooth currently in the field of view of an intraoral scanner. In one embodiment, the region to be determined corresponds to a region of a tooth that has recently been in the field of view of an intraoral scanner (e.g., a time prior to its presence in the field of view that was less than a threshold time amount).

[0314] At box 1415, the processing logic determines whether the scan is complete. If the scan is not complete (e.g., more intraoral scans are still being generated), the method returns to box 1402 and receives and processes additional intraoral scans. This may result in an update of the 3D surface at box 1404 and may result in a change to the surface quality fraction of one or more regions of the 3D surface at box 1406. If the scan is complete, the method terminates.

[0315] At block 1414, the processing logic may determine whether a threshold time has elapsed without improvement in the surface quality score, or whether the surface quality score has not improved by at least the threshold amount. Alternatively or additionally, the processing logic may determine whether the speed of the intraoral scanner is below a speed threshold. Alternatively or additionally, the processing logic may determine whether the intraoral scanner has begun to move away from the target area without the surface quality score reaching the threshold surface quality. In various embodiments, if one or more of these conditions are met, the method continues to block 1416. If none of these conditions are met, the method may return to block 1402.

[0316] At box 1416, the processing logic performs one or more actions to assist the user of the intraoral scanner in scanning the area to be scanned and / or to alert the user that scanning the area to be scanned may not be successful. In some embodiments, at box 1416, the processing logic determines that an additional intraoral scan of the first area will not improve the surface quality score of that area. Different types of actions may be performed to assist the user. In some embodiments, the processing logic automatically increases the zoom setting to magnify the area to be scanned, making it easier for the user to view the area of ​​the 3D surface.

[0317] In some embodiments, the processing logic generates a notification to the user to stop generating a new intraoral scan for that region and / or to continue to the next region. In some embodiments, the processing logic marks the region as a gap.

[0318] In some embodiments, the processing logic generates an overlay and outputs an overlay on the 3D surface and / or 2D image of the intraoral scanner's current FOV to show the position / mode of the intraoral scanner relative to its current positioning and / or orientation. In some embodiments, the processing logic generates an overlay showing the path followed by the intraoral scanner in capturing the intraoral scan, which would improve the surface quality of the region. In some embodiments, the processing logic determines the classification of the region (e.g., using a trained machine learning model, such as a neural network) and determines one or more recommendations, at least in part, based on the region classification, for physicians and / or patients to perform to improve the quality of the intraoral scan of the captured region.

[0319] In some embodiments, the processing logic identifies one or more problems with a region of the 3D surface (e.g., obscured by lips, obscured by gums, obscured by blood and / or saliva, obscured by collapsed gums in the region, poor angle, etc.) and outputs one or more suggestions for the patient and / or physician to perform in an intraoral scan to improve the quality of the captured region. Problems can be identified by feeding data associated with the 3D surface (e.g., intraoral scans, 2D images, the 3D surface itself, projections of the 3D surface, etc.) into a trained machine learning model that can output indications of one or more detected problems.

[0320] In some embodiments, the intraoral scanner includes multiple cameras, each with a different position and / or orientation at the head of the intraoral scanner (and therefore, a different field of view). Due to the different positions / orientations of the various cameras, some cameras may be better able to capture information about an area or a portion of an area. Each camera can generate a 2D image, any of which can be used as a viewfinder image to show the current field of view of the intraoral scanner. Processing logic can select the camera that best captures high-quality data for the area and can display the image generated by the selected camera. This may allow the user to adjust the position / orientation of the intraoral scanner so that they can better see the area as captured by the selected camera. In various embodiments, this can improve the quality of the intraoral scan of the area. In various embodiments, any one or more of these and / or other actions can be performed to improve the quality of the 3D surface.

[0321] In some embodiments, the processing logic automatically adjusts one or more algorithms used to process intraoral scan data associated with the region. In one embodiment, the processing logic determines one or more algorithms for processing intraoral scan data (e.g., intraoral scans, 2D images, regions of 3D surfaces, etc.). Examples of algorithms include registration algorithms, stitching algorithms, moving tissue detection and removal algorithms, object detection algorithms, soft tissue detection algorithms, etc.

[0322] Figure 14B This is a flowchart of a method 1450 for providing user assistance during an intraoral scan according to embodiments of the present disclosure. In various embodiments, method 1450 operates in conjunction with method 1400 to provide user assistance during a scan. At block 1452 of method 1450, processing logic receives one or more intraoral scans and / or 2D images of a tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR or NIRI) 2D images, 2D images generated under ultraviolet light, etc.

[0323] At box 1454, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position.

[0324] At box 1456, the processing logic determines whether the user is experiencing a problem or has already experienced a problem while scanning the area of ​​the tooth position (e.g., if the intraoral scanner is moving away from the area or is currently scanning a different area). Several different techniques can be used to determine whether the user is experiencing a problem or has already experienced a problem while scanning the area of ​​the tooth position. In one embodiment, the processing logic can determine whether the user is experiencing a problem or has already experienced a problem while scanning that area if the intraoral scanner remains focused on or maintains focus on the area of ​​the tooth position for at least a threshold amount of time. In one embodiment, a surface quality score for the area can be calculated; and if the surface quality score falls below a threshold after a threshold amount of time, the processing logic can determine whether the user is experiencing a problem or has already experienced a problem while scanning that area. In one embodiment, the processing logic can determine whether the user is experiencing a problem while scanning that area if the intraoral scanner is moving at a speed below a threshold speed and is focused on the area.

[0325] At box 1460, the processing logic determines whether the intraoral scanner is focused on the region or was previously focused on the region (e.g., moving away from the region). This can be determined by comparing the current field of view with the region to determine whether there is a threshold amount of overlap between the current field of view and the region (e.g., by comparing the most recent intraoral scan with the region of the 3D surface).

[0326] At box 1462, the processing logic performs one or more actions to assist the user of the intraoral scanner in scanning the area. Any of the actions previously described that provide user assistance may be performed. In some embodiments, user assistance is not initiated until the user begins to move the intraoral scanner away from the area.

[0327] Figure 14C This is a flowchart of method 1470 for providing user assistance during intraoral scanning in an instance where the surface quality of a region is not improved, according to embodiments of the present disclosure. At block 1472 of method 1470, processing logic receives one or more intraoral scans and / or 2D images of a tooth position. The processing logic may also receive one or more two-dimensional (2D) images of the tooth position, which may include color 2D images, near-infrared (NIR or NIRI) 2D images, 2D images generated under ultraviolet light, etc. Each intraoral scan may include three-dimensional information about a captured portion of the tooth position. For example, each intraoral scan may include a point cloud. In various embodiments, each intraoral scan includes three-dimensional information (e.g., x, y, z coordinates) of a plurality of points on the dental surface. Each of the plurality of points may correspond to a spot or feature of structured light projected onto the tooth position by a structured light projector of the intraoral scanner and captured in an image generated by one or more cameras of the intraoral scanner.

[0328] At box 1474, the processing logic uses one or more received intraoral scans to generate a 3D surface representing the scanned tooth position. This may include registering and stitching together multiple intraoral scans and / or registering and stitching one or more intraoral scans to an already generated 3D surface to update the 3D surface. The registration and stitching process may be performed as described in more detail above. As other intraoral scans are received, these intraoral scans may be registered and stitched to the 3D surface to add more information about regions / parts of the 3D surface and / or improve the quality of one or more regions / parts of an existing 3D surface. In some embodiments, the generated surface is an approximate surface whose quality may be lower than that of a surface calculated later.

[0329] At box 1476, the processing logic determines a surface quality fraction for one or more regions of the 3D surface. In some embodiments, the 3D surface is divided into multiple regions, which may have the same size or different sizes. For example, in one embodiment, the 3D surface is divided into regions of approximately mm in size (e.g., a 1 mm square region, a 1 mm diameter region, etc.). Alternatively, larger or smaller regions may be used. A surface quality fraction may be determined for each region.

[0330] At box 1478, the processing logic determines that the surface quality score of the scanned region is below a surface quality threshold. This may include determining one or more surface quality thresholds to be applied to the region based on the region's classification. For example, data associated with the region (e.g., from intraoral scans, 3D surfaces, 2D images, etc.) may be fed into a trained machine learning model that outputs a dental object classification for the region. Different dental object classifications may be associated with different thresholds. In one embodiment, a gauge is determined based on the dental object classification of the region, wherein the gauge includes the surface quality thresholds to be applied. In one embodiment, the region to be determined corresponds to a region of a tooth position currently in the field of view of an intraoral scanner. In one embodiment, the region to be determined corresponds to a region of a tooth position recently in the field of view of an intraoral scanner (e.g., a period of time in the field of view prior to its presence that was less than a threshold amount of time).

[0331] At box 1478, the processing logic may additionally or alternatively determine that a threshold amount of time has elapsed without any improvement in the surface quality score or at least no improvement in the surface quality score by a threshold amount.

[0332] At box 1480, the processing logic determines the surface quality score of one or more additional regions adjacent to the region to be determined. In one embodiment, this includes determining the surface quality score of one or more regions that individually or together at least partially surround the region to be determined.

[0333] At box 1482, the processing logic determines that the surface quality score of one or more adjacent regions meets or exceeds the surface quality score threshold.

[0334] At box 1486, based on the determinations made at boxes 1478 and / or 1482, the processing logic can determine that additional scanning of the region to be determined will not improve the surface quality score of that region.

[0335] In some embodiments, at block 1486, processing logic determines the region type of the area. The region type may be associated with the area failing to meet a surface quality threshold. The region type may indicate the reason why the area failed to meet the surface quality threshold. Different region types include, for example, an opening having at least one of a bottom or one or more sidewalls that cannot be imaged by an intraoral scanner; a surface whose achievable angle relative to the opening is too steep for the intraoral scanner to image; a surface covered by at least one of blood or saliva; or a surface covered by collapsed gum tissue. For some region types, the problem preventing the area from reaching the threshold may be solvable, such as by wiping away blood or saliva, applying dental floss to retract the gum tissue, and exposing the underlying preparation. For some region types, the problem preventing the area from reaching the threshold may be unsolvable. For example, the region type may be an unsolvable deep opening, a steep angle, etc. In some embodiments, the region type is determined by applying machine learning or artificial intelligence. For example, one or more intraoral scans, 2D images, a portion of a 3D surface, combinations thereof, etc., may be input into a trained machine learning model that can output the region type of the area.

[0336] At box 1488, the processing logic may label the region to be defined based on the region type. In some embodiments, the processing logic additionally or alternatively labels the region as a void. In the absence of data, a void may be an empty space and may or may not represent an actual physical hole in the 3D surface.

[0337] At box 1490, the processing logic may output a notification to stop scanning the current region and / or continue scanning the next region. Additionally or alternatively, the processing logic may output a notification regarding the region type determined for that region.

[0338] Figure 15AThis is a flowchart of method 1500 for adjusting scaling settings to assist intraoral scanning according to embodiments of the present disclosure. For example, method 1500 may be performed at block 1416 of method 1400 and / or block 1462 of method 1450. At block 1502 of method 1500, processing logic increases the scaling settings of the area currently being scanned. This may include increasing the scaling settings of the entire 3D surface generated so far. At block 1504, processing logic updates the view of the 3D surface so that a region of the 3D surface is emphasized and highlighted (e.g., in the center of the display). Due to the increase in scaling settings and the focused area, one or more other regions of the 3D surface may not be shown. This contrasts with standard techniques for showing the 3D surface of a scanned tooth position, which typically involve displaying the entire 3D surface.

[0339] Figure 15B This is a flowchart of a method 1520 for determining and outputting recommendations to assist intraoral scanning during intraoral scanning, according to embodiments of the present disclosure. For example, method 1520 may be performed at block 1416 of method 1400 and / or block 1462 of method 1450. In various embodiments, method 1520 may be performed in place of method 1500 or as a supplement to method 1500.

[0340] At block 1522 of method 1520, the processing logic determines one or more recommendations that, if implemented, would likely improve the surface quality score of an area currently being scanned or previously scanned (e.g., a region to be determined). In various embodiments, the current position of the intraoral scanner within the patient's mouth is determined, and one or more recommendations are determined at least in part based on this current position. In various embodiments, the processing logic feeds data of the 3D surface region into a trained machine learning model (e.g., a neural network) that can output one or more classifications and / or characteristics of the region. Alternatively, conventional image processing techniques and / or one or more heuristics can be applied to the image data and / or surface data to determine one or more characteristics of the region. In various embodiments, the processing logic may determine, for example, whether the region is associated with a distal molar, whether the region is at least partially occluded by soft tissue, whether the region is associated with anterior teeth, whether the region is at least partially occluded by the patient's lips, whether the region is at least partially occluded by the tongue, etc.

[0341] In one embodiment, at box 1524, processing logic determines that the region to be determined (e.g., the region currently being scanned) is associated with the distal molars of the patient being scanned. The processing logic may generate one or more scanning suggestions for the patient to move their jaw to the left or right. This might include guiding the patient to move their jaw to the left and then to the right, or guiding the patient to move their jaw to the right and then to the left. For example, the processing logic may generate animation of the patient moving their jaw in a specific manner (e.g., to the left or right). By having the patient move their jaw in a specified direction, this could cause the patient's jaw to pull the tip of the intraoral scanner into the region being scanned and could potentially help scan the back of the patient's distal molars (e.g., the left or right third molars). Similarly, the processing logic may determine that the region to be determined is associated with the distal molars of the patient being scanned and may generate suggestions for the physician to move the intraoral scanner, which could achieve a similar result to having the patient move their jaw from side to side.

[0342] In one embodiment, at block 1526, the processing logic determines that a region to be identified (e.g., the region currently being scanned) is obscured by soft tissue. The processing logic may generate one or more scanning suggestions from the physician to move the soft tissue by rolling the intraoral scanner around a specified axis and / or along a specified direction. This may cause the obscured area of ​​the region to be revealed.

[0343] In one embodiment, at box 1528, the processing logic determines that the region to be determined (e.g., the region currently being scanned) is associated with the patient's anterior teeth and is obscured by the patient's lips. The processing logic may generate one or more scanning suggestions, including instructing the physician to pull the patient's lips away from the anterior teeth and / or instructing the physician to slide the head of the intraoral scanner between the anterior teeth and the patient's lips.

[0344] In one embodiment, at block 1530, processing logic determines one or more scanner positions / orientations, upon which an intraoral scan should be generated to provide missing details of regions on a 3D surface. The processing logic can then determine the path followed by the intraoral scanner capturing the scan from each of the determined positions / orientations. The processing logic can take into account the current position / orientation of the intraoral scanner relative to the tooth position, as well as the distances between the various determined positions / orientations and the distances from the current position / orientation of the intraoral scanner. The determined path can be the most efficient and / or simplest path followed from the current position / orientation of the intraoral scanner through each of the determined positions / orientations. In various embodiments, a trained machine learning model (e.g., a neural network) can be used to determine the position / orientation and / or path. For example, the current position / orientation of the intraoral scanner and information about one or more regions of the tooth position can be input into a trained machine learning model that can output the determined position / orientation or the determined path. Once the path is determined, a superposition showing the path and / or the one or more target positions / orientations of the intraoral scanner included in the path can be generated. In some embodiments, the generated overlay includes one or more target locations / orientations of the intraoral scanner, but does not include the path that the intraoral scanner is to follow. In some embodiments, the processing logic generates an animation of the path followed by the intraoral scanner.

[0345] In one embodiment, at box 1532, processing logic determines that a region to be determined (e.g., the region currently being scanned) is obscured by the patient's tongue. The processing logic may generate one or more suggestions for the patient's tongue to move upwards, downwards, to the left, or to the right. This may cause the obscured area to be revealed.

[0346] At box 1534, the processing logic outputs one or more determined suggestions to the display. In one embodiment, the suggestions are output to the display as text and / or graphics. In one embodiment, the suggestions are output to the display as animation. In one embodiment, the suggestions are output to the display as an overlay, which is output above or on top of a 2D image (e.g., a viewfinder image) and / or a 3D surface.

[0347] Figure 16AThis is a flowchart of a method 1600 for generating and outputting a scan-guided overlay according to embodiments of this disclosure. At block 1602 of method 1600, processing logic determines at least one of a target location or target orientation to position an intraoral scanner for performing one or more other intraoral scans. The location / orientation of the intraoral scanner can be determined by: identifying regions with low surface quality fractions (e.g., surface quality fractions below a quality threshold), determining the location of missing data points in those regions, and determining the location / orientation of the intraoral scanner from which the missing data points can be captured.

[0348] At block 1604, the processing logic receives a first 2D image of the tooth position corresponding to the current field of view of the intraoral scanner. At block 1606, the processing logic outputs the first 2D image to a display. At block 1608, the processing logic generates a first overlay, which includes a first shape located at approximately the center of the 2D image or at another location on the 2D image associated with the current field of view of the intraoral scanner (e.g., showing the center of the field of view of the intraoral scanner and / or a particular camera of the intraoral scanner). The shape can be a crosshair, circle, ring, donut shape, square, rectangle, or any other shape. The generated overlay additionally includes a second shape at the target location. The second shape can be a crosshair, circle, ring, donut shape, square, rectangle, bar shape, area block shape, pyramid shape, or any other shape. In various embodiments, the second shape can be shaped according to a target orientation. For example, the second shape can indicate the target orientation and / or the angle between the current orientation and the target orientation. In one example, the first shape can be a ring or a hollow circle (e.g., a donut shape), and the second shape can be a solid circle, a cylinder, a rod, etc. The rod can have an orientation associated with the scanner's target orientation and a position associated with the scanner's target location. If the angle of the intraoral scanner coincides with the angle of the target orientation, the cylinder / rod shape may be shown as a circle. If the angle of the intraoral scanner is at a 90-degree angle to the target orientation, the cylinder / rod shape may be shown as a roughly linear or rectangular shape. Once the overlay is generated, it can be output to a display on the first 2D image. Therefore, the scanner's current position / orientation and the target position / orientation can be shown in the 2D image.

[0349] In response to the guidance provided by the overlay, the user can move the intraoral scanner in an attempt to align the first shape with the second shape. At box 1610, after the intraoral scanner has been repositioned to move toward the target location, the processing logic receives a second 2D image of the tooth position corresponding to the updated current field of view of the intraoral scanner. At box 1612, the processing logic outputs the second 2D image to the display, thereby replacing the first 2D image.

[0350] At box 1614, the processing logic generates a second overlay, which includes a first shape located at the center of the approximately 2D image or at another location on the 2D image associated with the current field of view of the intraoral scanner. The generated second overlay additionally includes a second (or third) shape at the target location. In various embodiments, the second / third shape can be shaped based on the difference between the current orientation and the target orientation of the intraoral scanner. As the user moves the scanner closer to the target location, the second shape will be closer to the first shape in the second overlay. Once the second overlay is generated, it can be output to a display on the second 2D image. Thus, the scanner's new current position / orientation and target position / orientation can be shown in the 2D image.

[0351] As the user moves the intraoral scanner, the operations of boxes 1610 to 1614 can be repeated, and for each updated 2D image, a new overlay can be generated and output on the updated 2D image. In this way, the user can be provided with guidance on how to move / position the intraoral scanner and how close they are to the target location / orientation. In one embodiment, the scanner is at the target location and orientation (and optionally, shares a common center and / or is concentric with the second shape) when the first shape overlaps with the second shape. For example, the first shape can be a donut shape, and the second shape can be a circle; the scanner can reach the target location / orientation when the circle is completely surrounded by the donut shape. In some embodiments, when the scanner reaches the target location / orientation, a visualization such as flashing can be output. Additional or alternative user feedback can be provided to indicate that the scanner has reached the target location and orientation, such as using haptic feedback, by outputting a beep or other audio signal, by outputting a visual indicator to the display, etc.

[0352] In some embodiments, in addition to generating an overlay for output on a 2D image, or as an alternative, the processing logic generates an overlay for output on a 3D surface. In this embodiment, at blocks 1606 and 1612, the 3D surface may be updated, and a view of the updated 3D surface may be output to a display. The overlay determined at blocks 1608 and 1614 may be similar to the overlays discussed above, but may be projected onto a 3D surface in addition to or as an alternative to overlaying onto a 2D image. Since the 3D surface contains 3D data that is the opposite of the 2D data, the overlay projected onto the 3D surface may contain additional 3D information, such as depth information, not included in the overlay projected onto the 2D surface. Additionally, the 3D surface includes information about areas outside the current field of view of the intraoral scanner, so target locations / orientations that are outside the scanner's field of view and therefore not visible in the overlay on the 2D surface can be shown on the 3D surface.

[0353] In some embodiments, augmented reality and / or mixed reality systems are used, wherein the physician may wear an augmented reality display. In such embodiments, the superposition of a first shape and a second shape can be projected onto the viewing surface of the augmented reality display (e.g., glasses) directly above the scanned tooth position. This may allow the physician to continue to focus on the patient while also seeing the current position / orientation and target position / orientation of the intraoral scanner relative to the tooth position without taking their gaze away from the patient. In one embodiment, the augmented reality system is used as described in U.S. Patent No. 10,467,815, issued November 5, 2019, the entire contents of which are incorporated herein by reference. In one embodiment, the augmented reality system is used as described in U.S. Patent No. 10,888,399, issued January 12, 2021, the entire contents of which are incorporated herein by reference.

[0354] Figure 16B This is a flowchart of a method 1650 for selecting an image (e.g., a viewfinder image) to influence how a tooth position is scanned, according to embodiments of this disclosure. As previously mentioned, in some embodiments, the intraoral scanner may include multiple cameras (e.g., a camera array), wherein each camera may have a different position and / or orientation on the intraoral scanner, and wherein each camera may provide a different viewpoint of the surface being scanned. Each camera may periodically generate intraoral images (also simply referred to herein as images). An image set may be generated, wherein the image set may include images generated by each camera. Processing logic may perform one or more operations on the received image set to select which images to output to a display and / or select which camera. The selected image may be a scanned image that will allow the user to position the intraoral scanner in a beneficial manner to capture the scanned area. The selected image may be, for example, a viewfinder image showing the current field of view of the cameras of the intraoral scanner.

[0355] At block 1652 of method 1650, the processing logic receives a set of intraoral 2D images from an intraoral scanner. In various embodiments, the intraoral 2D images may be color 2D images. Alternatively or additionally, the 2D images may be monochrome images, NIR images, or other types of images. Each image in the image set may have been generated simultaneously or substantially simultaneously by one or more different cameras. For example, the image set may correspond to... Figure 17A Images 1711 to 1717 or Figure 17B Images from 1721 to 1726.

[0356] At box 1654, the processing logic can determine the current position and / or orientation of the intraoral scanner relative to the generated 3D surface. At box 1656, the processing logic can determine the target position and / or orientation of the intraoral scanner relative to the 3D surface. At box 1658, the processing logic can then determine an image in the image set, and if that image is selected, it may cause the user of the intraoral scanner to reposition the intraoral scanner in a manner that moves the current position and / or orientation relative to the 3D surface toward the target position and / or orientation of the intraoral scanner relative to the 3D surface.

[0357] In one embodiment, processing logic determines whether any intraoral image in the intraoral image set satisfies one or more image selection criteria. In one embodiment, the image selection criteria include a highest score criterion. A score (also referred to as a value) may be calculated for each image in the image set based on one or more characteristics of the image, and the image with the highest score satisfies the image selection criteria. In some embodiments, the score may be determined based on the number of pixels or the amount of region in an image with a particular classification. The score of a single image may be adjusted based on the scores of one or more surrounding or other images (such as using a weighted matrix in some embodiments). In some embodiments, determining whether any intraoral image in the intraoral images satisfies one or more criteria includes: feeding the intraoral image set into a trained machine learning model that outputs a recommendation for selecting a camera associated with one of the input images. Other image selection criteria and / or techniques may also be used. In one embodiment, the score is determined based on the location where a user might aim at the scanner in response to selecting an image. Each image may be scored based on the distance between the location where a user might aim at the scanner in response to displaying an image and the location where the user should aim at the scanner to reach a target location and / or orientation.

[0358] At box 1660, the processing logic selects a camera associated with an intraoral image that meets one or more criteria. In one embodiment, the image with the highest score is selected. In another embodiment, an image recommended by a machine learning model is selected.

[0359] At block 1662, the processing logic outputs the intraoral image associated with the selected camera (e.g., the intraoral image with the highest score) to a display. This can provide the user with information about the current field of view of the selected camera, and consequently, information about the intraoral scanner (or at least a portion thereof). In one embodiment, image selection is determined according to the description of U.S. Patent Application No. 63 / 434,031, filed December 20, 2022, the entire contents of which are incorporated herein by reference. In one embodiment, image selection is determined according to the statements in U.S. Patent Application No. 63 / 434,031, but using a selection rule different from that stated in that patent application.

[0360] Figure 17A The illustrations show 2D images (e.g., intraoral images) 1701, 1702, 1703, 1704, 1705, and 1706 of a first tooth position generated by a camera array of an intraoral scanner according to embodiments of the present disclosure. In one embodiment, the 2D images are respectively generated by a camera array of an intraoral scanner having Figure 18 Camera generation for the reference frame shown.

[0361] Figure 17B The illustration shows 2D images 1711, 1712, 1713, 1714, 1715, and 1716 of a second tooth position generated by a camera array of an intraoral scanner according to an embodiment of the present disclosure. In one embodiment, the 2D images are respectively generated by a camera array of an intraoral scanner having Figure 18 Camera generation for the reference frame shown.

[0362] Figure 18 The illustration shows multiple cameras of an intraoral scanner according to an embodiment of the present disclosure relative to a scanned intraoral object 1816. In the illustrated example, the scanner includes six cameras, each with a different reference frame 1802, 1804, 1806, 1808, 1810, 1812. In some embodiments, a center or average reference frame 1814 may be calculated based on multiple reference frames.

[0363] Figure 19 This is a flowchart of a method 1900, according to embodiments of the present disclosure, for generating and outputting an overlay of a suggested path to be followed during an intraoral scan. At block 1902 of method 1900, processing logic generates a 3D surface based on an intraoral scan received during the intraoral scan. At block 1904, the processing logic determines at least one of a target location or target orientation to position the intraoral scanner for performing one or more other intraoral scans. The position / orientation of the intraoral scanner can be determined by identifying regions of the 3D surface with low surface quality fractions (e.g., surface quality fractions below a quality threshold), determining the location of missing data points in those regions, and determining the position / orientation of the intraoral scanner from which the missing data points can be captured.

[0364] At box 1906, the processing logic determines the current position and / orientation of the intraoral scanner relative to the 3D surface based on one or more most recent intraoral scans. At box 1908, the processing logic outputs a view of the 3D surface. The processing logic also generates and outputs an overlay that includes a first shape (e.g., such as...) showing the current position and orientation of the intraoral scanner relative to the 3D surface. Figure 13A The scanner outline (1350) and a second shape showing the target position and orientation of the intraoral scanner relative to the 3D surface.

[0365] In various embodiments, the second shape can be shaped according to the target orientation. For example, the second shape can indicate the target orientation and / or the angle between the current orientation and the target orientation. In one example, the first shape can be a ring or a hollow circle (e.g., a donut shape), and the second shape can be a solid circle, a cylinder, a rod, etc. The rod can have an orientation associated with the target orientation of the scanner and a position associated with the target position of the scanner. If the angle of the intraoral scanner coincides with the angle of the target orientation, the cylindrical / rod shape may be shown as a circle. If the angle of the intraoral scanner is at a 90-degree angle to the target orientation, the cylindrical / rod shape may be shown as a generally linear or rectangular shape. Once the overlay is generated, it can be output to a display on top of the first 2D image. Thus, the current position / orientation of the scanner and the target position / orientation can be shown in the 2D image.

[0366] Responding to the guidance provided by the overlay, the user can move the intraoral scanner in an attempt to align the first shape with the second shape. At box 1910, after the intraoral scanner has been repositioned to move toward the target location, the processing logic receives one or more additional intraoral scans of the tooth position. At box 1912, the processing logic updates the 3D surface based on the additional intraoral scans(s).

[0367] At block 1914, the processing logic determines the updated position and / or orientation of the intraoral scanner relative to the 3D surface. At block 1918, the processing logic generates an updated overlay including a first shape (or third shape) showing the current position and orientation of the intraoral scanner relative to the 3D surface and a second shape (or fourth shape) showing the target position and orientation of the intraoral scanner relative to the 3D surface. The processing outputs an updated view of the 3D surface, wherein the updated overlay is overlaid on top of the view of the 3D surface. In various embodiments, the second / fourth shape can be shaped based on the difference between the current orientation and the target orientation of the intraoral scanner. As the user moves the scanner closer to the target position, the second shape will be closer to the first shape in the second overlay.

[0368] As the user moves the intraoral scanner, the operations in boxes 1910 to 1918 can be repeated, and for each updated 3D image and / or new intraoral scan, a new overlay can be generated and output on the updated 3D image. In this way, the user can be provided with guidance on how to move / position the intraoral scanner and how close they are to the target location / orientation. In one embodiment, the scanner is at the target location and orientation (and optionally, shares a common center and / or is concentric with the second shape) when the first shape overlaps with the second shape. For example, the first shape can be a donut shape, and the second shape can be a circle; the scanner can reach the target location / orientation when the circle is completely surrounded by the donut shape. In some embodiments, a visualization such as flashing can be output when the scanner reaches the target location / orientation. Additional or alternative user feedback can be provided to indicate that the scanner has reached the target location and orientation, such as using haptic feedback, by outputting a beeping sound or other audio signal, by outputting a visual indicator to the display, etc.

[0369] In one embodiment, at box 1919, a path is generated illustrating the suggested movement of the intraoral scanner relative to the dental arch, causing the intraoral scanner to move from its current position / orientation to a target position / orientation. At box 1920, an overlay illustrating the generated path can be output above a 3D surface. In one embodiment, at box 1922, an animation illustrating the intraoral scanner moving along the path is generated. Then, at box 1924, the animation can be displayed.

[0370] Paths, animations, and / or target shapes can precisely instruct users of intraoral scanners on how to move / reposition the scanner to capture areas of teeth that are difficult for the user to reach. For example, path, animation, or target shape placement can instruct the user on how and when to rotate the scanner. In some embodiments, the processing logic initially provides coarse instructions on how to position / move the scanner, and as the user gets closer to the target location / orientation, it outputs more detailed instructions (e.g., a magnified view of the area on a 3D surface and an overlaid target shape).

[0371] In some embodiments, there may be multiple areas requiring additional scanning (e.g., areas with low surface quality fractions). In various embodiments, a target location / orientation for the scanner can be determined for each of these areas. The processing logic can then determine a path that takes into account the multiple target locations / orientations. By following the path, the user can generate the desired scan for each of these areas.

[0372] Figure 20This is a flowchart of a method 2000 for guiding a user to place an intraoral scanner during scanning, according to an embodiment of the present disclosure. At block 2002, processing logic receives an intraoral scan of a tooth position (e.g., maxillary or mandibular). At block 2004, the processing logic uses the intraoral scan to generate a 3D surface of the tooth position. At block 2006, the processing logic determines a first view of the 3D surface, which has a first position and orientation in the first view. At block 2008, the processing logic outputs the first view of the 3D surface to a display.

[0373] At box 2009, the processing logic determines the position and / or orientation of the intraoral scanner probe relative to the 3D surface of the tooth position based on an intraoral scan (e.g., based on the most recent near intraoral scan that has been successfully registered and stitched to the 3D surface). At box 2010, the processing logic may output a representation of the determined position and / or orientation of the probe head relative to the 3D surface to a display.

[0374] At box 2011, the processing logic determines the next suggested position and / or orientation of the probe tip relative to the 3D surface of the tooth position. In one embodiment, the next suggested position and orientation of the probe may be determined based on the scanning difficulty of specific subsequent areas of the tooth position that have not yet been scanned. In some embodiments, the processing logic may access one or more previously generated 3D models of the tooth position (e.g., 3D models generated during previous patient visits). The processing logic may evaluate these one or more 3D models to determine tooth crowding and / or particularly difficult-to-scan tooth geometry. Thus, the processing logic may determine suggested scan speed, suggested position and / or orientation of the scanner capturing difficult-to-scan areas, suggested position and / or orientation sequence of the scanner capturing difficult-to-scan areas, etc. In one embodiment, the processing logic determines a suggested trajectory for the intraoral scanner, which may include a recommended position and orientation sequence of the intraoral scanner.

[0375] At block 2012, the processing logic outputs additional representations of the probe head at suggested positions and / or suggested orientations relative to the 3D surface of the tooth position. A first visualization may be used to show the representation of the probe head, showing its current position and orientation; and a second visualization, different from the first visualization, may be used to show additional representations of the probe head, showing its next suggested position and orientation. The first visualization may include a first color, a first transparency level, a first line type, a first scaling level (also referred to as a magnification level), etc.; the second visualization may include a second color, a second transparency level, a second line type, a second scaling level (also referred to as a magnification level), etc. In one embodiment, the processing logic shows additional representations of the probe moving according to a determined recommended trajectory of the intraoral scanner. In various embodiments, recommendations for a trajectory or path followed by the intraoral scanner when scanning a region (or sequence of regions) of a tooth position may be determined and displayed based on the statements in U.S. Application No. 17 / 894,096, filed August 23, 2022, the entire contents of which are incorporated herein by reference.

[0376] Figure 21 The illustration shows a view of the 3D surface of a scanned tooth according to an embodiment of the present disclosure, the position and orientation of the probe head of an intraoral scanner relative to the 3D surface, and the next suggested position and orientation of the probe head relative to the 3D surface. As shown, a first visualization can be used for the current position and orientation of the probe head 2110, and a second visualization can be used for the recommended position and orientation of the probe head 2120. In various embodiments, animations are shown of the probe head 2120 moving through a sequence of positions and orientations according to a recommended trajectory 2130.

[0377] Figures 22A to 22C The illustration shows a view of a 3D surface 2205 of a tooth position being scanned and a proposed path 2215 for scanning the tooth position, according to an embodiment of the present disclosure. As shown, multiple regions 2220, 2225, 2230 have low surface quality and / or voids. Therefore, additional scans should be performed on these regions 2220 to 2230. Figure 22A The illustration shows a view of the 3D surface 2205 and the initial position and orientation of the probe head of the intraoral scanner 2210 relative to the 3D surface 2205. The proposed path first moves to a first target position / orientation of the intraoral scanner 2210 to capture region 2220, then to a second target position / orientation of the intraoral scanner 2210 to capture region 2225, and finally to a third target position / orientation of the intraoral scanner 2210 to capture region 2230. In various embodiments, the path begins at the region 2220 closest to the intraoral scanner 2210 and moves across the most efficient path to capture all regions with low surface quality fractions and / or voids. Figure 22B The illustration shows a view of the 3D surface 2205 after the scanner 2210 has moved past the suggested target position / orientation for the scan area 2220 and moved to the suggested target positioning / orientation for the scan area 2225. Figure 22C The illustration shows a view of the 3D surface 2205 after the scanner 2210 has been moved to the suggested target position / orientation for the scan area 2230. In various embodiments, the movement of the scanner 2210 through... Figures 22A to 22C Animation of the position and orientation sequence shown.

[0378] Figure 23 The illustration shows a 3D surface 2310 of a scanned tooth and a proposed view of a rotating intraoral scanner according to an embodiment of the present disclosure. Figure 23 The illustration shows a view of a graphical user interface 2300 for an intraoral scanning application according to an embodiment of the present disclosure. The graphical user interface 2300 includes a 3D surface 2310 of the current field of view (FOV) of the intraoral scanner and a 2D image 2323. Additionally, an outline 2350 of the intraoral scanner, including an outline 2351 of the intraoral scanner's field of view, can be shown relative to the 3D surface 2310. The 3D surface 2310 is generated by registering and stitching together multiple intraoral scans captured during an intraoral scanning session. When each new intraoral scan is generated, the scan is registered to the 3D surface and then stitched together. Thus, with each intraoral scan, the 3D surface becomes increasingly accurate until it is complete. A 3D model can then be generated based on the intraoral scan.

[0379] Some areas of the dental arch may be particularly difficult to scan. For example, areas with deep pits or fissures and the most distal central molars may be particularly difficult to scan. In some embodiments, processing logic may analyze 3D surface 2310 or data from 3D surface (e.g., surface quality fraction data) and determine that the user is experiencing difficulty when scanning an area. Other clues, such as the user lingering in a certain area, or the surface quality of that area not improving over time, may also indicate that the user is experiencing difficulty when scanning that area. The processing logic may determine one or more suggestions that, if implemented, would increase the surface quality fraction of the scanned area. One such suggestion might be to rotate the intraoral scanner about a specific axis and / or in a specific direction. As illustrated, a suggestion to rotate the intraoral scanner may be shown as a superimposition on 2D image 2323 and / or a graphic depiction of intraoral scanner 2350. This suggestion may be presented as rotation icons 2354A and / or rotation icons 2354B. In various embodiments, rotation icons 2354A through 2354B may indicate the direction of rotation, the axis of rotation, and / or the amount of rotation.

[0380] In one embodiment, as shown, the scan segmentation indicator 2330 may include an upper dental arch segmentation indicator 2332, a lower dental arch segmentation indicator 2334, and an occlusal segmentation indicator 2336. The GUI of the intraoral scanning application may also include a taskbar with multiple operating modes or intraoral scanning stages. Selecting the patient selection mode 2340 allows the physician to enter patient information and / or select patients already entered into the system. Selecting the scan mode 2342 enables an intraoral scan of the patient's oral cavity. After the scan is complete, selecting the post-processing mode 2344 prompts the intraoral scanning application to generate one or more 3D models based on the intraoral scan and / or 2D images generated during the intraoral scan, and optionally performs analysis on the 3D models(s). Selecting the prescription fulfillment mode 2346 allows the generated orthodontic and / oral prosthodontic prescription to be sent to a laboratory or other facility to generate prosthodontic devices (e.g., crowns, bridges, prostheses, etc.) or orthodontic devices (e.g., clear aligners).

[0381] Figure 24 The illustration shows a 3D surface 2410 of a scanned tooth position and a suggested view of patient movement of the jaw according to an embodiment of the present disclosure. Figure 24 The illustration shows a view of a graphical user interface 2400 for an intraoral scanning application according to an embodiment of the present disclosure. The graphical user interface 2300 includes a 3D surface 2410 of the current field of view (FOV) of the intraoral scanner and a 2D image 2423. Additionally, an outline 2450 of the intraoral scanner, including an outline 2451 of the intraoral scanner's field of view, can be shown relative to the 3D surface 2410.

[0382] Some areas of the dental arch may be particularly difficult to scan. For example, areas with deep pits or fissures and the most distal central molars may be particularly difficult to scan. In some embodiments, the processing logic may analyze the 3D surface 2410 or data from the 3D surface (e.g., surface quality fraction data) and determine that the user is experiencing difficulty while scanning the area. Other clues, such as the user lingering in a certain area, or the surface quality of that area not improving over time, may also indicate that the user is experiencing difficulty while scanning that area. The processing logic may determine one or more suggestions that, if implemented, would increase the surface quality fraction of the scanned area. One such suggestion might be for the patient to move their jaw to the left and / or right (e.g., left then right or right then left). As illustrated, suggestions for the patient to move their jaw may be displayed in a pop-up window 2450 and may include text and / or graphics.

[0383] In one embodiment, as shown, the scan segmentation indicator 2430 may include an upper dental arch segmentation indicator 2432, a lower dental arch segmentation indicator 2434, and an occlusal segmentation indicator 2436.

[0384] Figure 25 This is a flowchart illustrating an embodiment of a method 2500 for determining the scan quality of regions of a 3D surface during intraoral scanning and outputting recommendations for improving scan quality. At block 2502 of method 2500, processing logic receives multiple intraoral scans during an intraoral scanning session in a patient's oral cavity. At block 2504, the processing logic uses the multiple intraoral scans to generate a 3D surface of a tooth position within the patient's oral cavity. At block 2506, the processing logic determines one or more regions of the 3D surface and / or scan quality metrics (e.g., surface quality scores) of the received one or more intraoral scans. At block 2508, the processing logic determines whether one or more regions of the 3D surface and / or scan quality metrics (e.g., surface quality scores) of the one or more intraoral scans are below a scan quality threshold (or fail to meet one or more scan quality criteria). At block 2509, the processing logic may additionally identify one or more regions of interest (ROIs) of the tooth position that have not yet been scanned or have only been partially scanned. The processing logic may additionally or alternatively identify one or more hard-to-scan regions that have not yet been scanned or have only been partially scanned. In one embodiment, regions of interest (AOIs) and / or hard-to-scan areas are determined based on analysis of a previously generated 3D model of the tooth positions (e.g., the patient's dental arch). For example, a physician may have scanned the patient's dental arch during a previous patient visit or earlier in the current patient visit. The processing logic can analyze this 3D model to identify areas of crowded teeth, areas with small gaps (e.g., the buccal side of the posterior molars), etc.

[0385] At box 2510, the processing logic determines the current position (and optionally one or more past positions) of the intraoral scanner probe relative to the 3D surface, at least in part, based on the most recent near intraoral scan successfully stitched to the 3D surface. At box 2512, the processing logic determines one or more suggested scanning parameters for one or more next intraoral scans for the intraoral scanning session. Scanning parameters may include the relative position and / or orientation of the intraoral scanner probe relative to a portion of the tooth to be scanned next. Scanning parameters may also include the speed at which the intraoral scanner moves, the distance between the scanner and the tooth, the angle of the scanner relative to the tooth, etc. The processing logic may additionally determine one or more unscanned areas of the patient's oral cavity. Additionally, the processing logic may determine scan quality metrics for scanned areas and may identify those areas where one or more scan quality metrics fall outside the target range of scan quality metrics. Additionally, the processing logic may identify one or more AOIs on the 3D surface.

[0386] At box 2514, the processing logic outputs one or more suggested scanning parameters on the display (e.g., in the GUI of an intraoral scanning application) for one or more next intraoral scans. The one or more suggested scanning parameters may include, for example, the next suggested position and / or orientation of the intraoral scanner probe relative to a 3D surface, the next distance between the probe tip and the 3D surface, the movement speed between the current position and the next position of the probe tip, etc. The (multiple) next suggested positions and / or (multiple) next suggested orientations and / or other suggested scanning parameters of the probe tip relative to the patient's oral cavity (e.g., dental arch) may be positions and / or orientations suitable for scanning one or more unscanned areas, rescanning the AOI, etc. For example, the next suggested position and / or orientation of the probe tip relative to the patient's oral cavity may be positions and / or orientations and / or other suggested scanning parameters suitable for rescanning scanned areas where the scan quality metric fails to meet scan quality criteria (e.g., outside the target scan quality metric range). When used, suggested scanning parameters (e.g., position and / or orientation of the scanning head, scanning speed, scanning distance, scanning angle, etc.) may result in higher scanning quality metrics for one or more areas with unacceptable scanning quality metrics. Additionally or alternatively, the next suggested position and / or next suggested orientation and / or other suggested scanning parameters of the probe head relative to the patient's oral cavity (e.g., dental arch) may be a suitable position and / or orientation for rescanning the AOI. In one embodiment, at block 2516, the processing logic outputs a representation of the probe head moving from its current position to the next position of the probe head relative to the 3D surface of the tooth, based on one or more suggested scanning parameters.

[0387] In one example, the processing logic determines the current angle of the probe head relative to the 3D surface and can determine whether the angle of the probe head is within a target angle range (e.g., 40 to 60 degrees) relative to the probe head. In response to determining that the angle of the probe head is outside the target angle range, the processing logic can determine one or more angle adjustments to the probe head, wherein one or more suggested scanning parameters may include one or more angle adjustments.

[0388] In one example, processing logic determines the ratio of the distal to the proximal surface represented in the 3D surface of the tooth position. Based on this ratio, the processing logic can determine whether the distal or proximal surface is dominant. In response to determining that the distal surface is dominant, the processing logic can determine one or more first angle adjustments to the probe tip that increase the amount of mesial surface captured. In response to determining that the mesial surface is dominant, the processing logic can determine one or more second angle adjustments to the probe tip that increase the amount of distal surface captured. The processing logic can then determine one or more suggested scanning parameters, which include one or more first angle adjustments or one or more second angle adjustments.

[0389] In one embodiment, the processing logic determines a scan rate associated with one or more intraoral scans and / or one or more regions of a 3D surface. The processing logic may determine that the scan rate is outside a scan rate range and may suggest one or more scan parameters for one or more next intraoral scans that will cause the scan rate to fall within a target scan rate range.

[0390] In one embodiment, the processing logic determines the trajectory of the intraoral scanner during intraoral scanning. The processing logic projects the trajectory into the future and optionally compares the area to be scanned with a previously generated 3D model of the tooth position. The processing logic may determine whether the subsequent area to be scanned is a difficult or easy area to scan. If a difficult area is subsequently identified during the intraoral scanning session, the processing logic may output an alert to the user to slow down the scanning speed (e.g., slow down the probe head speed) for scanning the difficult area. The processing logic may additionally determine one or more recommended scanning parameters (in addition to scanning speed) for scanning the difficult area and may output recommendations for using one or more of these parameters.

[0391] Figure 26 This is a flowchart of a method 2600 for adjusting the scaling settings of a 3D surface based on scanner speed, according to embodiments of the present disclosure. When a user moves an intraoral scanner during an intraoral scan, the amount of information useful to a particular area (e.g., the area currently being scanned) may be related to the speed at which the user moves the intraoral scanner. For example, if the user slows down the movement of the intraoral scanner (e.g., pauses the intraoral scanner to capture a tooth that is preparing to be captured or is difficult to capture), the user may have more information about the area (e.g., a tooth) that the scanner is currently focusing on. Therefore, in various embodiments, the scaling settings of the resulting 3D surface are automatically controlled at least in part based on the speed of the intraoral scanner (e.g., controlled without user input).

[0392] At block 2602 of method 2600, one or more intraoral scans and / or associated 2D images of the tooth position are received. At block 2604, processing logic generates a 3D surface of the tooth position based on the intraoral scans. Optionally, the 2D images can also be used to generate the 3D surface (e.g., to provide color information for the 3D surface).

[0393] At block 2606, for each intraoral scan and / or 2D image, the processing logic determines the position of the intraoral scanner that generated the intraoral scan or 2D image relative to the 3D surface. Since the intraoral scan comprises numerous points, the distance information of these points indicates the distance between these points and the intraoral scanner during the intraoral scan. Therefore, the distance between the intraoral scanner and the tooth position (and thus to the 3D surface registered and stitched by the intraoral scan) is known and / or can be easily calculated. The intraoral scanner may alternate between generating intraoral scans and 2D images; therefore, in various embodiments, the distance between the intraoral scanner and the tooth position (and / or 3D surface) associated with the 2D image can be interpolated based on the distances associated with intraoral scans generated before and after the 2D image.

[0394] At box 2607, for each intraoral scan and / or 2D image, the processing logic can determine the focal point of the intraoral scanner and / or the position of a virtual point at a set distance from the intraoral scanner that generated the intraoral scan / 2D image. In one embodiment, the focal point and / or virtual point is a point at an x, y position approximately 10 mm from the intraoral scanner, located approximately at the center of the intraoral scanner's field of view.

[0395] At block 2608, the processing logic determines the speed of the intraoral scanner relative to the tooth position based at least in part on the intraoral scans and / or 2D images. In one embodiment, the processing logic determines the difference between the positions of the intraoral scanners associated with a plurality of intraoral scans / 2D images, and determines the speed based on the determined difference between the positions and the time difference between generating the intraoral scans / 2D images. In one embodiment, the processing logic determines the difference between the positions of the focal points and / or virtual points of the intraoral scanners associated with a plurality of intraoral scans / 2D images, and determines the speed based on the determined difference between the positions and the time difference between generating the intraoral scans / 2D images.

[0396] At block 2610, the processing logic determines the scaling setting based on a determined speed (e.g., the speed of the intraoral scanner and / or the focus of the intraoral scanner). In one embodiment, the determined speed is used as a key value for performing a lookup in a lookup table. The lookup table can associate speed with scaling settings. Thus, the processing logic can determine the entry in the lookup table associated with the determined speed, and can determine the scaling setting in that entry. In one embodiment, the scaling setting is determined by inputting the speed into a function that associates speed with scaling settings. In various embodiments, the scaling setting is inversely proportional to the speed. Thus, as the intraoral scanner accelerates, the processing logic shrinks the 3D surface. Similarly, when the intraoral scanner slows down and / or pauses, the processing logic magnifies the 3D surface.

[0397] In one embodiment, at block 2612, the processing logic determines the resolution for a portion of the 3D surface and / or for intraoral scanning based on a determined speed (e.g., the speed of the intraoral scanner and / or the focus of the intraoral scanner). In one embodiment, the determined speed is used as a key value for performing a lookup in a lookup table. The lookup table can associate speed with a resolution setting. Thus, the processing logic can determine the entry in the lookup table associated with the determined speed, and can determine the resolution setting in that entry. In one embodiment, the resolution setting is determined by inputting the speed into a function that associates speed with a resolution setting.

[0398] In one embodiment, at block 2613, the processing logic determines one or more algorithms for processing intraoral scan data (e.g., intraoral scans, 2D images, regions of 3D surfaces, etc.). Examples of algorithms include registration algorithms, stitching algorithms, motion tissue detection and removal algorithms, object detection algorithms, soft tissue detection algorithms, etc.

[0399] At box 2614, the processing logic determines the current field of view of the intraoral scanner. The determined field of view can be a combined field of view of multiple cameras of the intraoral scanner, as determined based on the most recently received set of 2D images. Alternatively, the field of view can be the field of view associated with the most recently received intraoral scan.

[0400] At box 2616, the processing logic determines a portion of the 3D surface associated with the current field of view. This could be a portion or region of the 3D surface that the intraoral scanner is currently focusing on (e.g., the area currently being imaged).

[0401] At box 2618, the processing logic uses the determined scaling settings and optionally the determined resolution to output a view of at least a determined portion / region of the 3D surface. For example, this view could be an occlusal view, a bird's-eye view, a distal-to-mesoscopic view, a mesoscopic-to-distal view, etc. The processing logic may also use one or more selected algorithms to process regions of the 3D surface and / or intraoral scan data used to generate those regions. At the current scaling settings, some parts of the 3D surface may not be displayed. For example, if the processing logic zooms in on a particular region, that region may be shown along with surrounding regions adjacent to it. However, other regions farther away from the currently scanned region may not be visible. In some embodiments, the processing logic determines whether the velocity is below a velocity threshold. If the velocity is below the velocity threshold, the processing logic may display a second view of the 3D surface and a first view of the 3D surface. The two different views of the 3D surface may have different scaling settings and / or different translation / rotation settings. Therefore, the first view of the 3D surface can be a magnified view of the 3D surface viewed from a first angle (e.g., corresponding to the current angle of the intraoral scanner relative to the tooth position), while the second view of the 3D surface can be a magnified view of the 3D surface viewed from a different angle (e.g., an occlusal view, a lingual view, etc.). In some embodiments, additional information is output to the display in response to the speed dropping below a threshold speed. Such information may include icons and / or windows indicating surface quality fractions, indications of areas with missing data, etc.

[0402] At box 2620, the processing logic determines whether the scan is complete. If the scan is not complete, the method returns to box 2602 and repeats operations 2602 through 2618. This process may repeat as long as the scan is in progress. Thus, as the intraoral scanner speed changes, the processing logic can automatically zoom in and out while the scan is being performed. If the user slows down the scanner's movement, the processing logic can zoom in on the current area to show it in more detail; and if the user speeds up the scanner's movement, the processing logic may zoom out to provide a more detailed view of the 3D surface.

[0403] In some embodiments, users can enable or disable the auto-zoom function of the intraoral scanning application via a GUI. In one embodiment, if the processing logic detects that the user has manually zoomed in and / or zoomed out once or more during an intraoral scan, the processing logic can provide a suggestion to activate the auto-zoom function. In one embodiment, the processing logic outputs a suggestion to activate the auto-zoom function in response to detecting that the user is experiencing a problem in the area of ​​the scanned tooth position.

[0404] Figure 27This is a flowchart of a method 2700 for adjusting the scaling settings of a 3D surface based on scanner speed, according to an embodiment of this disclosure. At block 2705 of method 2700, processing logic determines the image received from the intraoral scanner during intraoral scanning and / or the scanner position during intraoral scanning. The scanner position may be a 3D position and may be represented as... , where W is the scanner position and n is the image or intraoral scan. In various embodiments, the scanner position can be estimated continuously or periodically, allowing for the cascading of individual frames, for example, using processes such as chain stitching. At block 2710, the processing logic determines the position of a virtual point associated with the scanner position, where the virtual point is located at a target distance from the intraoral scanner. For example, the position of the virtual point can be associated with the focus of the intraoral scanner. The virtual point provides a reliable estimate of how the field of view of the intraoral scanner changes over time. The virtual point can be a point located at approximately the center of the FOV of the intraoral scanner at a typical depth. In one embodiment, the virtual point is a point within the field of view of the intraoral scanner at a distance of approximately 10 mm from the intraoral scanner. Thus, the virtual point can have a position in the scanner coordinate system with x,y,z={0,0,10}. The virtual point position can be represented as , where P is the position of the virtual point in the world coordinate system (e.g., the coordinate system of a 3D surface), and F is a function applied to the scanner position to determine the position of the virtual point in the global coordinate system.

[0405] In various embodiments, the virtual position of the point may change frequently as the scan progresses (e.g., with each new intraoral scan and / or 2D image generated). This can cause abrupt changes in the virtual position. Therefore, in various embodiments, a smoothing operation is performed to smooth the position of the virtual point over time. In one embodiment, smoothing logic 2715 is applied to one and / or more virtual positions associated with previously received intraoral scans and / or 2D images. The smoothing operation can reduce or eliminate abrupt changes in the virtual point over time.

[0406] In one embodiment, smoothing logic 2715 is an infinite impulse response (IIR) filter. An IIR filter is a recursive filter where the filter's output is computed using the current input, the previous input, and the previous output. In one embodiment, at block 2720, the current virtual position is added to the previous virtual position delayed at block 2725, and multiplied by a multiplier a0 at block 2730. The multiplier a0 can be a scalar value with any value from 0 to 1, which controls the averaging function. The output of smoothing logic 2715 is the average position of virtual point 2735. .

[0407] At block 2740, processing logic determines the velocity of the virtual point's average position over two or more recent intraoral scans and / or 2D images. In one embodiment, the previous average position is stored in delay logic 2745, and summation logic 2750 adds the latest average position of the virtual point to the negative of the previous average position of the virtual point. Thus, the difference between the current position and the previous position of the virtual point is determined. This difference can be divided by the time difference between generating the latest intraoral scan and / or 2D image and generating the previous intraoral scan and / or 2D image to determine the velocity of the virtual point associated with the current intraoral scan or 2D image. , where v is velocity.

[0408] In various embodiments, the velocity of the points may change frequently as the scan progresses (e.g., with each new intraoral scan and / or 2D image generated). This can cause abrupt changes in the velocity of the virtual locations. Therefore, in various embodiments, a smoothing operation is performed to smooth the velocity of the virtual points over time. In one embodiment, smoothing logic 2760 is applied to the velocity of one and / or more virtual locations associated with previously received intraoral scans and / or 2D images. The smoothing operation can reduce or eliminate velocity abrupt changes.

[0409] In one embodiment, smoothing logic 2760 is an infinite impulse response (IIR) filter. In one embodiment, at block 2765, the current velocity is added to the previous velocity delayed at block 2770, and multiplied by a multiplier a1 at block 2775. The multiplier a1 can be a scalar value with any value from 0 to 1, which controls the averaging function. The output of smoothing logic 2760 is the average velocity at virtual point 2780. .

[0410] At box 2785, the processing logic determines a scaling factor for the image or scan n based on the average velocity associated with the image or scan n. In one embodiment, the scaling factor is determined by inputting the average velocity into a function, which outputs the scaling factor. This function can be represented as follows: In one embodiment, the function is a monotonically decreasing function. In another embodiment, the function that maps velocity to a scaling factor is as follows: Where M is a constant representing the maximum scaling and α is a constant representing the sensitivity.

[0411] In one embodiment, the scaling factor is determined by performing a lookup in a lookup table using the average speed as the key.

[0412] Figure 28A The illustration shows a view of a 3D surface 2810A of a tooth position having a first scaling setting according to an embodiment of the present disclosure. Figure 28A The illustration shows a view of a graphical user interface 2800 for an intraoral scanning application according to an embodiment of the present disclosure. The graphical user interface 2800 includes a 3D surface 2810A of the current field of view (FOV) of the intraoral scanner and a 2D image 2823A. Additionally, an outline 2850 of the intraoral scanner, including an outline 2351 of the intraoral scanner's field of view, can be shown relative to the 3D surface 2810A. The 3D surface 2810A is generated by registering and stitching together multiple intraoral scans captured during an intraoral scanning session. When each new intraoral scan is generated, the scan is registered to the 3D surface and then stitched together. Thus, with each intraoral scan, the 3D surface becomes increasingly accurate until the 3D surface is complete. A 3D model can then be generated based on the intraoral scan. In one embodiment, as shown, the scan segmentation indicator 2830 may include an upper dental arch segmentation indicator 2832, a lower dental arch segmentation indicator 2834, and an occlusal segmentation indicator 2836.

[0413] Some areas of the dental arch can be particularly difficult to scan. For example, areas with deep pits or fissures and the most distal central molars can be especially difficult to scan. When scanning these areas, the user can naturally slow down the movement of the intraoral scanner. In some embodiments, the processing logic can calculate the speed of the intraoral scanner (or virtual points within the field of view of the intraoral scanner) and determine the scaling factor setting based on the speed. The processing logic can then automatically adjust the scaling factor setting based on the speed. Figure 28A As shown, the scanner speed may be very high, resulting in a relatively low scaling setting (e.g., scaling set to 1x).

[0414] Figure 28B The illustration shows a view of a 3D surface 2810B with a second scaling setting for a tooth position according to an embodiment of the present disclosure. The 3D surface 2810B may be an updated version of the 3D surface 2810A, which has been updated after one or more additional intraoral scans have been stitched to the 3D surface. Figure 28B The illustration shows a view of a graphical user interface 2820 for an intraoral scanning application according to an embodiment of the present disclosure. The graphical user interface 2820 includes a 3D surface 2810B of the current field of view (FOV) of the intraoral scanner and a 2D image 2823B. Additionally, an outline 2850 of the intraoral scanner, including an outline 2351 of the intraoral scanner's field of view, can be shown relative to the 3D surface 2810B. The 3D surface 2810B is generated by registering and stitching one or more intraoral scans onto the 3D surface 2810A. Processing logic may have determined that... Figure 28AThe associated speed slows down the scanner, and the scaling factor settings may have been automatically adjusted based on the updated speed (e.g., automatically zooming in on the current area being scanned). Figure 28B As shown, the scanner speed may be low, resulting in relatively high zoom settings (e.g., zoom settings of 2x, 3x, 4x, 5x, 6x, 7x, 8x, 9x, 10x, etc.).

[0415] Figure 29 The illustration shows a graphical representation of a machine in an example form of computing device 2900, in which a set of instructions can be executed to cause the machine to perform one or more of the methods discussed herein. In alternative embodiments, the machine may be connected (e.g., networked) to other machines on a local area network (LAN), intranet, extranet, or the Internet. Computing device 2900 may correspond to, for example... Figure 1 The computing devices 105 and / or 106. The machine can operate as a server or client in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), cellular phone, network device, server, network router, switch, or bridge, or any machine capable of executing a (continuous or discontinuous) set of instructions specifying actions to be taken by that machine. Further, although only a single machine is illustrated, the term "machine" should also be considered as including any collection of machines (e.g., computers) that individually or in combination execute one or more sets of instructions to perform any one or more of the methods discussed herein.

[0416] Example computing device 2900 includes processing device 2902, main memory 2904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), static memory 2906 (e.g., flash memory, static random access memory (SRAM), etc.), and secondary memory (e.g., data storage device 2928), which communicate with each other via bus 2908.

[0417] Processing device 2902 represents one or more general-purpose processors, such as microprocessors, central processing units, etc. More specifically, processing device 2902 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. Processing device 2902 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processing device 2902 is configured to execute processing logic (instructions 2926) for performing the operations and steps discussed herein.

[0418] The computing device 2900 may also include a network interface device 2922 for communicating with the network 2964. The computing device 2900 may also include a video display unit 2910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 2912 (e.g., a keyboard), a cursor control device 2914 (e.g., a mouse), and a signal generation device 2920 (e.g., a speaker).

[0419] Data storage device 2928 may include a machine-readable storage medium (or more specifically, a non-transitory computer-readable storage medium) 2924 thereon storing a set 2926 of one or more instructions (such as instructions for intraoral scanning application 115) embodying any one or more methods or functions described herein, the machine-readable storage medium 2924 possibly corresponding to Figure 1 Intraoral scanning application 115. Non-transitory storage medium refers to storage medium other than carrier wave. Instruction 2926 may also reside wholly or at least partially in main memory 2904 and / or processing device 2902 during execution by computing device 2900, which also constitute computer-readable storage medium.

[0420] The computer-readable storage medium 2924 may also store a software library containing methods for intraoral scanning application 115. Although the computer-readable storage medium 2924 is shown as a single medium in the example embodiment, the term "computer-readable storage medium" should be understood to include a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) storing one or more instruction sets. The term "computer-readable storage medium" should also be understood to include any medium other than a carrier wave capable of storing or encoding instruction sets for execution by a machine and causing the machine to perform any one or more methods of the methods disclosed herein. The term "computer-readable storage medium" should therefore be understood to include, but is not limited to, solid-state memory and optical and magnetic media.

[0421] It should be understood that the above description is intended to be exemplary and not restrictive. Many other embodiments will become apparent upon reading and understanding the above description. Although embodiments of this disclosure have been described with reference to specific exemplary embodiments, it should be recognized that this disclosure is not limited to the above embodiments, but can be practiced with modifications and variations within the spirit and scope of the appended claims. Therefore, the specification and drawings are to be regarded for illustrative purposes and not for limiting purposes. Consequently, the scope of this disclosure should be determined by reference to the appended claims and their full scope and equivalents.

Claims

1. An intraoral scanning system, comprising: Intraoral scanner; as well as Computing device, wherein the computing device is used for: Multiple intraoral scans of tooth positions are received from the intraoral scanner during an intraoral scanning session; The three-dimensional (3D) surface of the tooth position is generated based on the multiple intraoral scans; Determine the surface mass fraction of a first region of the 3D surface; One or more additional intraoral scans of the tooth position are received during the intraoral scanning session; The 3D surface is updated based on the one or more additional intraoral scans; The surface quality fraction of the first region is updated based on the updated 3D surface; It was determined that the surface quality fraction of the first region failed to meet one or more criteria; as well as Output a notification associated with the scan of the first region.

2. The intraoral scanning system of claim 1, wherein, in order to determine that the surface quality fraction of the first region fails to meet one or more criteria, the computing device is configured to: The surface quality fraction of the first region is determined to have failed to improve within a threshold time period and / or the surface quality fraction of the first region is determined to be below a surface quality threshold.

3. The intraoral scanning system according to any one of claims 1 to 2, wherein, in order to determine that the surface quality fraction of the first region fails to meet one or more of the criteria, the computing device is configured to: Determine that the additional surface quality fraction of one or more additional regions adjacent to the first region is equal to or higher than the surface quality threshold.

4. The intraoral scanning system according to any one of claims 1 to 3, wherein, in order to determine that the surface quality fraction of the first region fails to meet one or more of the criteria, the computing device is configured to: It was determined that additional intraoral scanning of the first region would not improve the surface quality fraction of the first region.

5. The intraoral scanning system according to any one of claims 1 to 4, wherein the computing device is further configured to: A first view of the 3D surface is output to a display, wherein the first region of the 3D surface is shown using a first visualization associated with the surface quality fraction; and A second view of the updated 3D surface is output to the display, wherein the first region of the updated 3D surface is shown using a second visualization associated with the updated surface quality fraction.

6. The intraoral scanning system according to any one of claims 1 to 5, wherein the computing device is further configured to: The first region of the 3D surface or at least one of the plurality of intraoral scans associated with the first region is input into a trained machine learning model, wherein the trained machine learning model outputs the surface quality score.

7. The intraoral scanning system according to any one of claims 1 to 6, wherein the plurality of intraoral scans are generated by projecting structured light comprising a plurality of features onto the tooth position and capturing the plurality of features on the tooth position, and wherein the surface quality fraction is determined at least in part based on the number of the plurality of features associated with the first region of the 3D surface.

8. The intraoral scanning system according to any one of claims 1 to 7, wherein the 3D surface is generated based on a plurality of points from the plurality of intraoral scans, and wherein the computing device is further configured to: Determine the quality score for each of the plurality of points; Determine a subset of the plurality of points associated with the first region; and The surface quality score of the first region is determined based on the quality score of the subset of the plurality of points and the number of the subset of the plurality of points.

9. The intraoral scanning system of claim 8, wherein the quality score of one of the plurality of points is calculated based on at least one of the following: a) the distance between the point and at least one of the first or second cameras of the intraoral scanner that captured the point when generating one of the plurality of intraoral scans; and b) the distance between the first camera and the second camera; c) the distance between the point and the camera of the intraoral scanner that captures the point when generating the intraoral scan and d) the distance between the camera and the structured light projector of the intraoral scanner that projects structured light onto the point; The number of cameras of the intraoral scanner that capture the points when generating the intraoral scan; The spot size associated with the point; The angle between the normal of the 3D surface at the point and the imaging axis of the camera that captures the point during the intraoral scan; or The material type of the tooth at the specified point.

10. The intraoral scanning system according to any one of claims 1 to 9, wherein the computing device is further configured to: Receive multiple two-dimensional (2D) images of the tooth position; and Determine the number of the plurality of 2D images depicting the first region; The surface quality fraction of the first region is based at least in part on the number of the plurality of 2D images depicting the first region.

11. The intraoral scanning system according to any one of claims 1 to 10, wherein the computing device is further configured to: Determine a first roughness and a first resolution associated with the first region of the 3D surface; The surface quality fraction of the first region is determined at least in part based on the first roughness and the first resolution.

12. The intraoral scanning system according to any one of claims 1 to 11, wherein the computing device is further configured to: One or more scanning recommendations, if implemented, would result in an improvement in the surface quality fraction of the first region; The notification mentioned therein includes one or more scan recommendations.

13. The intraoral scanning system according to any one of claims 1 to 12, wherein the computing device is further configured to: In response to determining that the surface quality fraction of the first region fails to meet one or more criteria, the scaling setting of the first region is increased.

14. The intraoral scanning system according to any one of claims 1 to 11, wherein the notification comprises at least one of the following: Notification to stop generating new intraoral scans of the first region; Continue with notifications for the next area; or Notification that the surface quality score of the first region has not improved over a period of time.

15. The intraoral scanning system according to any one of claims 1 to 14, wherein the intraoral scanner comprises a plurality of cameras, each camera having at least one of a different position or different orientation in the intraoral scanner, and wherein the computing device is further configured to: Receive multiple two-dimensional (2D) images, each 2D image generated by a different camera among the multiple cameras; Determine one of the plurality of 2D images associated with the improved intraoral scan quality; and The determined 2D image is output to the display, wherein, in response to the output of the determined 2D image, the physician using the intraoral scanner will reposition the intraoral scanner in a manner that results in the improved intraoral scan quality.

Images