Foreign body identification, image enhancement, and / or filtering for intraoral scanning

Abstract

To provide a method for generating a virtual 3D model of a dental site.SOLUTION: Scan data comprising a plurality of images of a dental site is received during an intraoral scan. An analysis of an image is performed. A representation of foreign matter potentially having known properties is identified in the image based on the analysis. A virtual 3D model of the dental site is generated based on the plurality of images. The image and / or the virtual 3D model of the dental site is modified by adding additional data about the foreign matter to an intraoral image and / or the virtual 3D model or removing the data about the foreign matter from the intraoral image and / or the virtual 3D model. When the image is modified, it may be modified prior to generation of the virtual 3D model, and the virtual 3D model may be generated by using the modified image.SELECTED DRAWING: Figure 1B

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of intraoral scanning, and more particularly, to systems and methods for performing real-time filtering of intraoral images generated during intraoral scanning.

Background Art

[0002] In prosthodontic procedures planned to insert dental prostheses into the oral cavity, accurate measurements and careful consideration are often required for the dental sites where the prostheses are to be implanted, so that the prostheses (e.g., crowns, dentures, or bridges, etc.) can be appropriately designed and sized to fit precisely in place. By fitting precisely, mechanical stress can be properly transmitted between the prosthesis and the jaw, and for example, it is possible to prevent infection of the gingiva from the boundary between the prosthesis and the dental site due to mechanical stress.

[0003] Depending on the procedure, it is also necessary to create removable prostheses (e.g., partial dentures or complete dentures) to replace one or more missing teeth. In that case, it is necessary to accurately reproduce the surface contour of the area where the teeth are missing so that the created prosthesis applies uniform pressure to the soft tissue and fits precisely over the edentulous area.

[0004] Depending on the treatment, the dental area may be prepared by a practicing dentist, and an explicit physical model of the dental area may be created using known methods. Alternatively, the dental area may be scanned to obtain 3D data of the dental area (i.e., data in the form of intraoral images such as height maps). In either case, a virtual or real model of the dental area is sent to a dental laboratory, which then fabricates the prosthesis based on that model. However, if a specific range of the model is incomplete or uncertain, or if the shape of the preparation is not optimal for receiving the prosthesis, the design of the prosthesis may be suboptimal. For example, if the insertion path determined by the preparation for a tight coping fit results in the prosthesis colliding with an adjacent tooth, the geometry of the coping must be modified to avoid collision, which may result in a suboptimal coping design. Furthermore, if the range of the preparation, including the finish line, is unclear, it may be impossible to properly determine the finish line, and therefore, the lower edge of the coping cannot be properly designed. In fact, depending on the situation, the model may be deemed unacceptable, and the dentist may need to rescan the dental area or redo the preparation to ensure that a suitable prosthesis can be fabricated.

[0005] In orthodontic treatment, it is sometimes important to have models of one or both jaws. When virtually designing such orthodontic treatment, a virtual model of the oral cavity is also useful. Such virtual models can be obtained by directly scanning the oral cavity, or by creating a physical model of the dentition and scanning that model with an appropriate scanner.

[0006] Therefore, in both prosthetic and orthodontic treatments, obtaining a three-dimensional (3D) model of the dental area within the oral cavity is the first step. If the 3D model is a virtual model, the more complete and accurate the scan of the dental area, the higher the quality of the virtual model, and therefore the more likely it is that the design of the prosthesis or orthodontic appliance will be optimal.

[0007] During an intraoral scan session, foreign objects are often present in the patient's oral cavity. The presence of such objects can reduce the accuracy of the generated virtual model. For example, image processing algorithms may consider a foreign object to be part of the dental arch and adjust the shape of one or more regions of the dental arch based on the shape of that object. Furthermore, foreign objects may move between the generation of different intraoral images, which can interfere with image registration. [Overview of the project] [Problems that the invention aims to solve]

[0008] This invention was made to solve the problems of the above-mentioned conventional technology. [Means for solving the problem]

[0009] A first aspect of the present disclosure includes a method comprising: receiving scan data including an intraoral image during an intraoral scan of a dental site; identifying representations of foreign objects in the intraoral image based on analysis of the scan data; modifying the intraoral image by removing the representations of foreign objects from the intraoral image; receiving additional scan data including a plurality of additional intraoral images of the dental site during the intraoral scan; and generating a virtual three-dimensional (3D) model of the dental site using the modified intraoral image and the plurality of additional intraoral images.

[0010] A second aspect of the present disclosure may extend the first aspect of the present disclosure. In the second aspect of the present disclosure, the method further includes the steps of: identifying an additional representation of a foreign object in one or more additional intraoral images of a plurality of additional intraoral images of a dental site; and modifying one or more additional intraoral images by removing the additional representation of a foreign object from one or more additional intraoral images, wherein the foreign object is a stationary object located at the same position of the dental site in the intraoral image and one or more additional intraoral images.

[0011] A third aspect of the present disclosure may extend the first or second aspect of the present disclosure. In the third aspect of the present disclosure, the method further includes the steps of: analyzing scan data to identify at least one of the reflectance of the foreign object to a wavelength of light, the refractive index of the foreign object to a wavelength of light, the responsiveness of the foreign object to a wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object; and identifying a representation of the foreign object based on at least one of the reflectance of the foreign object to a wavelength of light, the refractive index of the foreign object to a wavelength of light, the responsiveness of the foreign object to a wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object.

[0012] A fourth aspect of the present disclosure may extend the first to third aspects of the present disclosure. In the fourth aspect of the present disclosure, the method further includes the steps of performing image processing on an intraoral image to identify a plurality of shapes in the intraoral image, wherein the plurality of shapes include the shape of a foreign object; comparing the plurality of shapes in the intraoral image with a plurality of known shapes of a reference object; and determining, as a result of the comparison, whether the shape of the foreign object matches a known shape of the reference object.

[0013] A fifth aspect of the present disclosure may extend the fourth aspect of the present disclosure. In the fifth aspect of the present disclosure, the method further includes the steps of entering a training mode before intraoral scanning of a dental site; receiving a plurality of images of a foreign object during the training mode; generating a virtual model of the foreign object based on the plurality of images; and adding the virtual model of the foreign object, or at least one of the plurality of images of the foreign object, to a reference object library which includes a plurality of known shapes of reference objects.

[0014] A sixth aspect of the present disclosure may extend the first to fifth aspects of the present disclosure. In the sixth aspect of the present disclosure, the method further includes the steps of: performing image processing on an intraoral image to identify at least one of a plurality of textures or a plurality of surface patterns in the intraoral image; identifying at least one of the plurality of textures that does not naturally exist in the dental area, or at least one of the plurality of surface patterns that does not naturally exist in the dental area; and determining the contour of a region in the intraoral image having at least one of the textures or surface patterns that does not naturally exist in the dental area, wherein the region within the contour in the intraoral image is removed from the intraoral image.

[0015] A seventh aspect of the present disclosure may extend the sixth aspect of the present disclosure. In the seventh aspect of the present disclosure, the step of identifying at least one of a texture or surface pattern that is not naturally present in a dental site includes at least one of the following: a) comparing a plurality of textures identified in an intraoral image with a plurality of known textures of a reference object and determining, as a result of the comparison, whether at least one texture matches a known texture of the reference object; b) comparing a plurality of surface patterns identified in an intraoral image with a plurality of known surface patterns of a reference object and determining, as a result of the comparison, whether at least one surface pattern matches a known surface pattern of the reference object; c) comparing a plurality of textures identified in an intraoral image with a plurality of known textures that are naturally present in a dental site and determining, as a result of the comparison, whether at least one texture does not match any known texture that is naturally present in the oral cavity; or d) comparing a plurality of surface patterns identified in an intraoral image with a plurality of known surface patterns that are naturally present in a dental site and determining, as a result of the comparison, whether at least one surface pattern does not match any known surface pattern that is naturally present in the oral cavity.

[0016] An eighth aspect of the present disclosure may extend the first to seventh aspects of the present disclosure. In the eighth aspect of the present disclosure, the method further includes the steps of: performing image processing on an intraoral image to identify a plurality of colors in the intraoral image; identifying at least one of the plurality of colors that is not naturally present in a dental area; and determining the contour of a region in the intraoral image having at least one color that is not naturally present in a dental area, wherein the region within the contour of the intraoral image is removed from the intraoral image.

[0017] A ninth aspect of the present disclosure may extend the eighth aspect of the present disclosure. In the ninth aspect of the present disclosure, the step of identifying at least one color that does not naturally exist in a dental area includes, for each pixel or voxel of an intraoral image, the step of examining at least one of the saturation value or intensity value for each color channel of a color encoding system, the step of determining a tuple containing at least one of the saturation value or intensity value for each color channel of the color encoding system, and the step of determining whether the tuple falls outside a specified color range, and the step of determining the contour of a region of an intraoral image having at least one color that does not naturally exist includes the step of identifying a plurality of consecutive pixels or voxels having a tuple that falls outside a specified color range.

[0018] A tenth aspect of the present disclosure may extend the first to ninth aspects of the present disclosure. In the tenth aspect of the present disclosure, the method further includes the step of receiving an indication that a foreign body is located in a dental site, the indication including identification information of the foreign body; the step of querying a reference object library using the identification information of the foreign body; the step of receiving a response to the query, the response including one or more known properties of the foreign body, the one or more known properties including at least one of the known reflectance of the foreign body to a wavelength of light, the known refractive index of the foreign body to a wavelength of light, the known responsiveness of the foreign body to a wavelength of light, the known texture of the foreign body, the known surface pattern of the foreign body, the known color of the foreign body, or the known shape of the foreign body; and the step of identifying a representation of the foreign body in an intraoral image using one or more known properties of the foreign body.

[0019] An eleventh aspect of the present disclosure may extend the first to tenth aspects of the present disclosure. In the eleventh aspect of the present disclosure, the method further includes the steps of generating a view of a dental site based on a modified intraoral image and one or more additional intraoral images during an intraoral scan; generating the contour of a foreign object; adding the contour of the foreign object to the view; and providing a user interface that enables the user to select the contour of the foreign object, the user's selection of the contour of the foreign object causing the representation of the foreign object to be returned to the intraoral image, and further causing the foreign object to be added to a reference object library of known objects that should not be removed by the filter.

[0020] A twelfth aspect of the present disclosure may extend the first to eleventh aspects of the present disclosure. In the twelfth aspect of the present disclosure, the method further includes the steps of: determining a confidence value associated with the representation of a foreign object before modifying the intraoral image; determining whether the confidence value falls below a confidence threshold; presenting the user an option to a) remove the representation of the foreign object from the intraoral image or b) leave the representation of the foreign object in the intraoral image; and receiving the user's choice to remove the representation of the foreign object from the intraoral image.

[0021] A thirteenth aspect of the present disclosure may extend the twelfth aspect of the present disclosure. In the thirteenth aspect of the present disclosure, the method further includes the step of identifying one or more properties of a foreign object, wherein the one or more properties include at least one of the reflectance of the foreign object with respect to a wavelength of light, the refractive index of the foreign object with respect to a wavelength of light, the responsiveness of the foreign object with respect to a wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object; and the step of adding an entry for the foreign object to a reference object library of known foreign objects to be removed by filtering.

[0022] A 14th aspect of the Disclosure may extend a 12th or 13th aspect of the Disclosure. In a 14th aspect of the Disclosure, the step of identifying a foreign body representation includes processing an intraoral image using a machine learning model trained to identify foreign bodies in a dental site, and receiving the output of the machine learning model, the output including a binary mask having the same number of entries as pixels or voxels in the intraoral image, wherein entries associated with pixels or voxels that are part of the foreign body have a first value, and entries associated with pixels or voxels that are not part of the foreign body have a second value, and the intraoral image is modified using the binary mask.

[0023] A 15th aspect of the present disclosure includes a method comprising: receiving intraoral scan data including a plurality of images of a dental site; generating a virtual three-dimensional (3D) model of the dental site based on the plurality of images; performing an analysis on the images of the dental site; identifying a representation of a reference object in the images, wherein the reference object has one or more known characteristics; and modifying at least one of the images of the dental site or the virtual 3D model by adding additional data relating to the reference object based on one or more known characteristics of the reference object.

[0024] The 16th aspect of the present disclosure may extend the 15th aspect of the present disclosure. In the 16th aspect of the present disclosure, the image is one of a plurality of images of a dental site used for generating a virtual 3D model, the image is corrected by adding additional data regarding a reference object to the image, and when the image is corrected, it is used for generating a virtual 3D model of the dental site.

[0025] The 17th aspect of the present disclosure may extend the 16th aspect of the present disclosure. In the 17th aspect of the present disclosure, the method further includes the step of performing image processing on the image to identify likely objects represented in the image, the likely objects including one or more physical characteristics, the step of identifying as described above, the step of comparing one or more physical characteristics of the likely objects with one or more known characteristics of the reference object, the step of determining whether one or more physical characteristics of the likely objects match one or more known characteristics of the reference object as a result of the comparison, and the step of determining whether the likely object is the reference object.

[0026] The 18th aspect of the present disclosure may extend the 17th aspect of the present disclosure. In the 18th aspect of the present disclosure, the one or more physical characteristics include at least one of reflectivity, refractive index, reactivity with respect to the wavelength of light, and at least one of texture, surface pattern, color, or shape.

[0027] The 19th aspect of the present disclosure may extend any of the 15th to 18th aspects of the present disclosure. In the 19th aspect of the present disclosure, the step of performing analysis of the image includes the step of inputting the image into a machine learning model trained to identify one or more types of reference objects, and the machine learning model outputs what indicates the representation of the reference object in the image.

[0028] The 20th aspect of the present disclosure may extend the 15th or 19th aspect of the present disclosure. In the 20th aspect of the present disclosure, the method further includes the step of generating the image on which the analysis is performed by projecting the virtual 3D model onto a plane.

[0029] The 21st aspect of the present disclosure may extend the 20th aspect of the present disclosure. In the 21st aspect of the present disclosure, the method further includes, during an intraoral scan, generating a view of a dental site based on a modified image, the view including a contour of a reference object based on additional data regarding the reference object; receiving a plurality of additional images of the dental site during the intraoral scan; replacing one or more portions of the shape of the reference object based on the plurality of additional images; and updating the view of the dental site, the updated view showing the replaced portion of the shape of the reference object.

[0030] The 22nd aspect of the present disclosure may extend any of the 15th to 21st aspects of the present disclosure. In the 22nd aspect of the present disclosure, the step of identifying a representation of a reference object includes processing an image using a machine learning model trained to identify one or more foreign objects in a dental site, and receiving an output of the machine learning model, the output including a binary mask having the same number of entries as the pixels or voxels in the image, an entry associated with a pixel or voxel that is part of the reference object having a first value, and an entry associated with a pixel or voxel that is not part of the reference object having a second value.

[0031] The 23rd aspect of the present disclosure may extend any of the 15th to 22nd aspects of the present disclosure. In the 23rd aspect of the present disclosure, the method further includes entering a training mode before receiving intraoral scan data; receiving a plurality of images of a reference object during the training mode; generating a virtual model of the reference object based on the plurality of images of the reference object; and adding the virtual model of the reference object or at least one of the plurality of images of the reference object to a reference object library including a plurality of entries of the reference object.

[0032] A 24th aspect of the present disclosure may extend any of the 15th to 23rd aspects of the present disclosure. In the 24th aspect of the present disclosure, the method further includes the step of receiving an indication that a reference object is located in a dental area, the indication including identification information of the reference object; the step of querying a reference object library including entries for multiple reference objects using the identification information of the reference object; the step of receiving a response to the query, the response including data associated with the reference object; and the step of identifying the reference object in an image using the data.

[0033] A 25th aspect of this disclosure may extend any of the 15th to 24th aspects of this disclosure. In the 25th aspect of this disclosure, one or more known properties include at least one of the following: a known reflectance of the reference object with respect to a wavelength of light, a known refractive index of the reference object with respect to a wavelength of light, a known responsiveness of the reference object with respect to a wavelength of light, a known texture of the reference object, a known surface pattern of the reference object, a known color of the reference object, or a known shape of the reference object.

[0034] A 26th aspect of the Disclosure may extend any of the 15th to 25th aspects of the Disclosure. In the 26th aspect of the Disclosure, the Method further includes the steps of determining a confidence value associated with a reference object; determining whether the confidence value falls below a confidence threshold; presenting the option of a) adding additional data relating to the reference object to at least one of the image or virtual 3D model, or b) leaving the representation of the reference object in at least one of the image or virtual 3D model unchanged; and receiving the user selection to add additional data relating to the reference object to at least one of the image or virtual 3D model.

[0035] A 27th aspect of this disclosure may extend any of the 15th to 26th aspects of this disclosure. In the 27th aspect of this disclosure, the reference object includes at least one of a shape or material that causes a decrease in the accuracy of the intraoral scan of the reference object.

[0036] Further aspects of this disclosure include computer-readable media containing instructions that, when executed by a processing device, cause the processing device to perform a method of one or more of the first to 27 aspects of the above disclosure.

[0037] Further aspects of the present disclosure include a system comprising a handheld scanner for performing intraoral scanning and a non-temporary computer-readable medium containing instructions that, when executed by the processor, cause the processor to perform an operation of one or more of the methods of aspects 1 to 27 of the present disclosure.

[0038] Further aspects of this disclosure include a system comprising a handheld scanner for performing intraoral scanning and a computing device for performing a method of one or more aspects of the 1st to 27th aspects of this disclosure.

[0039] The figures in the attached drawings are provided as examples, not as limitations. [Brief explanation of the drawing]

[0040] [Figure 1A] This document describes one embodiment of a system for performing intraoral scans and generating virtual 3D models of dental areas. [Figure 1B] A flowchart of a method for identifying the representation of foreign objects in intraoral images according to embodiments of this disclosure is shown. [Figure 2] This diagram shows a flowchart of a method for removing the representation of foreign objects from an intraoral image using a filter, according to an embodiment of the present disclosure. [Figure 3] A flowchart of a method for identifying foreign objects in intraoral images according to an embodiment of the present disclosure is shown. [Figure 4] This diagram shows a flowchart of a method for adding an entry for a foreign object to a foreign object reference library according to an embodiment of the present disclosure. [Figure 5] A flowchart illustrating a method for identifying foreign objects in intraoral images using a foreign object reference library according to an embodiment of this disclosure is shown. [Figure 6] A flowchart of a method for identifying foreign objects in an intraoral image based on color, according to an embodiment of the present disclosure, is shown. [Figure 7] A flowchart of a method for providing a user interface for managing foreign objects in intraoral images according to an embodiment of the present disclosure is shown. [Figure 8] A flowchart of another method for providing a user interface for managing foreign objects in intraoral images according to embodiments of the present disclosure is shown. [Figure 9] A flowchart illustrating a method for filtering out foreign object representations from intraoral images based on the use of a trained machine learning model according to embodiments of this disclosure is shown. [Figure 10] This shows a portion of an exemplary dental arch with a foreign object placed on it. [Figure 11] A first intraoral image including a first representation of a foreign body according to one embodiment of the present disclosure is shown. [Figure 12] Figure 11 shows a modified version of the first intraoral image according to one embodiment of the present disclosure, in which the foreign matter has been removed by a filter. [Figure 13A] This shows a view of an exemplary portion of a dental arch and a foreign object placed thereon, according to one embodiment of the present disclosure. [Figure 13B] Figure 13A shows a view of an unscanned area of ​​foreign matter according to one embodiment of the present disclosure. [Figure 14] A block diagram of an exemplary computing device according to an embodiment of the present disclosure is shown. [Figure 15] This document illustrates a method of orthodontic treatment using multiple instruments according to an embodiment. [Figure 16] A method for designing orthodontic appliances according to an embodiment is shown. [Figure 17] This document describes a method for digitally planning orthodontic treatment according to an embodiment. [Modes for carrying out the invention]

[0041] This specification describes methods and apparatus for improving the quality of intraoral scans performed on dental sites containing one or more foreign objects. Interference situations often occur during intraoral scans, each involving the presence of one or more foreign objects in the intraoral image generated during the scan. In some embodiments, the intraoral image may be a height map, or may include a height map. For example, the intraoral image may be a 2D height map, which includes the depth value of each pixel in the height map. An example of an intraoral scanner that generates such a height map during an intraoral scan is the iTero® intraoral digital scanner from Align Technology, Inc.

[0042] Foreign objects in intraoral scans (e.g., intraoral images and / or height maps) may include objects that are naturally present in the oral cavity but are not part of the dental arch being scanned (e.g., saliva, blood, tongue, foam, etc.). Other foreign objects may not be naturally present in the oral cavity (e.g., cotton rolls, the operator's fingers of the intraoral scanner, suction devices, air blow devices, dental mirrors, cords, hands, gloves, etc.). The presence of foreign objects in intraoral scan data (i.e., intraoral images) can slow down the processing speed of those intraoral images and reduce the accuracy of the virtual 3D model generated from them. Therefore, embodiments described herein provide a method for filtering out (i.e., removing) the depiction of foreign objects in real time or near real time (e.g., at the time the image is generated) before the foreign objects are used for image registration and / or virtual model generation. Removing foreign objects from intraoral images before further processing of those images can improve the speed and / or accuracy of that further processing. For example, it is possible to increase the speed of image registration, increase the accuracy of image registration, reduce the number of instances of image registration failure, increase the accuracy of the generated 3D virtual model, and increase the generation speed of the 3D virtual model. Furthermore, by considering and omitting processing-intensive operations that mitigate problems caused by foreign objects, it is possible to further increase the speed of image processing.

[0043] Notably, detecting and removing foreign objects from intraoral images can be done using a single image and is applicable to static (i.e., non-moving) objects. This contrasts with many conventional methods (known as space carving) that identify and remove unwanted moving objects from images. Such conventional space carving methods, which identify and remove moving objects, cannot operate on a single image and are ineffective for objects that do not move across multiple frames of a video or multiple images of a region. In contrast, the method described herein can identify and remove stationary objects and can operate on a single image.

[0044] In another embodiment, a known reference object may be identified in the image of the dental area (for example, an intraoral image generated by an intraoral scanner, or an image generated by projecting a virtual 3D model of the dental area onto a 3D surface or plane) and / or in the virtual 3D model of the dental area. The image of the dental area and / or the virtual 3D model may then be modified by adding additional information about the reference object to the image of the dental area and / or the virtual 3D model based on the known characteristics of the reference object.

[0045] In one embodiment, the computer implementation method includes the steps of: receiving scan data including an intraoral image during an intraoral scan of a dental site; and identifying representations of foreign objects in the intraoral image based on analysis of the scan data. The computer implementation method further includes the step of modifying the intraoral image by removing representations of foreign objects from the intraoral image. The computer implementation method further includes the step of receiving additional scan data including a plurality of additional intraoral images of the dental site during the intraoral scan. The computer implementation method further includes the step of generating a virtual three-dimensional (3D) model of the dental site using the modified intraoral image and the plurality of additional intraoral images.

[0046] In one embodiment, a non-temporary computer-readable medium includes instructions that cause the processing device to perform a series of operations when executed by the processing device. These operations include receiving an intraoral image during an intraoral scan of a dental site, and performing image processing on the intraoral image to identify a plurality of possible objects in the intraoral image, each of which includes one or more physical characteristics. These operations further include comparing, for each of the plurality of possible objects, one or more physical characteristics of the possible object with a plurality of known characteristics of a reference object. These operations further include determining, as a result of the comparison, whether one or more physical characteristics of one of the plurality of possible objects match one or more known characteristics of the reference object. These operations further include determining whether the possible object is a foreign object in its intraoral location, and then modifying the intraoral image by adding additional data about the foreign object to the intraoral image based on one or more known characteristics of the reference object.

[0047] In one embodiment, a method for generating a virtual 3D model of a dental area includes the step of receiving intraoral scan data containing multiple images of the dental area. The method further includes the step of generating a virtual three-dimensional (3D) model of the dental area based on the multiple images. The method further includes the step of performing an analysis on the images of the dental area. This image may be one of the images contained in the received intraoral scan data. Alternatively, this image may be generated by projecting the virtual 3D model of the dental area onto a 2D surface or plane. This image may be a height map of the dental area, or may include a height map. The method further includes the step of identifying a representation of a reference object in the image, wherein the reference object has one or more known characteristics. The method further includes the step of modifying at least one of the images of the dental area or the virtual 3D model by adding additional data about the reference object based on one or more known characteristics of the reference object to at least one of the images of the dental area or the virtual 3D model. This makes it possible to improve the images and virtual 3D models by augmenting them with accurate information about known reference objects. Therefore, by using the methods described herein, it is possible to improve the accuracy of reference objects that are difficult to scan (for example, reference objects made of titanium) in images and virtual 3D models.

[0048] In one embodiment, the system includes a handheld scanner for performing intraoral procedures and a non-temporary computer-readable medium containing instructions that cause the processor to perform the aforementioned series of operations when executed by the processor.

[0049] Figure 1A shows one embodiment 100 of a system for performing intraoral scanning and / or generating a virtual three-dimensional (3D) model of a dental site. In one embodiment, system 100 performs one or more operations of the methods described later with reference to Figures 1B-9. In system 100, a computing device 105 may be coupled with a scanner 150 and / or a data store 110.

[0050] The computing device 105 may include a processing unit, memory, secondary storage, one or more input devices (e.g., keyboard, mouse, tablet, etc.), one or more output devices (e.g., display, printer, etc.), and / or other hardware components. The computing device 105 may be connected to the data store 110 directly or via a network. The network may be a local area network (LAN), a public wide area network (WAN) (e.g., the Internet), a private WAN (e.g., an intranet), or a combination thereof. In some embodiments, the computing device 105 and / or the data store 110 may be integrated into the scanner 150 to enhance performance and portability.

[0051] The datastore 110 may be an internal datastore or an external datastore connected to the computing device 105 directly or via a network. Examples of network datastores include storage area networks (SANs), network-attached storage (NAS), and storage services provided by cloud computing service providers. The datastore 110 may include a file system, a database, or other data storage configuration.

[0052] In some embodiments, a scanner 150 (referred to as an intraoral scanner) that acquires three-dimensional (3D) data of the dental area in the patient's oral cavity may also be operationally connected to the computing device 105. The scanner 150 may include a probe (e.g., a handheld probe) that optically captures the three-dimensional structure (e.g., by confocal focusing of an array of light beams). An example of such a scanner 150 is the iTero® intraoral digital scanner from Align Technology, Inc. Other examples of intraoral scanners include the True Definition Scanner from 3M®, and the Apollo DI and CEREC AC intraoral scanners from Sirona®.

[0053] The scanner 150 may be used to perform an intraoral scan of a patient's oral cavity. The result of the intraoral scan may be a series of discretely generated intraoral images (for example, by pressing the scanner's "image generation" button for each image). Alternatively, the result of the intraoral scan may be one or more videos of the patient's oral cavity. The operator may start video recording with the scanner 150 at a first position in the oral cavity, move the scanner 150 to a second position in the oral cavity while continuing to record video, and stop the video recording there. In some embodiments, recording may start automatically when the scanner identifies that it has been inserted into the patient's oral cavity. The scanner 150 may send discrete intraoral images (e.g., height maps) or intraoral videos (collectively referred to as image data 135) to the computing device 105. In some embodiments, the computing device may be integrated into the scanner 150. The computing device 105 may store the image data 135 in a data store 110. Alternatively, the scanner 150 may be connected to another system that stores image data in a data store 110. In such an embodiment, the scanner 150 does not need to be connected to the computing device 105.

[0054] The computing device 105 may include a foreign object identification module 115, an image registration module 120, and / or a model generation module 125. Modules 115, 120, and 125 may be software modules that are components of a single software application (e.g., various libraries of a single application), each may be a separate software application, each may be firmware, each may be a hardware module, and / or a combination thereof.

[0055] The foreign body identification module 115 can identify foreign bodies in the intraoral images of the received image data 135 (e.g., discrete intraoral images, or intraoral images which are frames of an intraoral video) by analyzing the image data 135 using one or more of the methods described herein. In some embodiments, the data store 110 may include a reference object library containing entries for one or more reference objects 130. The reference objects 130 may include well-known instruments that are standard in dental and / or orthodontic clinics. Such reference objects 130 may be default reference objects pre-stored in the data store 110. Some reference objects 130 may be custom reference objects which may be generated by a dental practitioner in training mode. Each entry for a reference object may include one or more physical properties of that reference object (e.g., shape (e.g., a 3D virtual model of the reference object), color, texture, surface pattern, etc.). Some foreign object identification methods used in the foreign object identification module 115 may include the steps of identifying the physical properties of an object in a received intraoral image and comparing those physical properties of a reference object 130 in a reference object library. If a match is found, the object represented in the intraoral image may be identified as a specific foreign object associated with the match.

[0056] In one embodiment, the user may put the foreign object identification module 115 into training mode. During training mode, the user may perform a 3D scan of an object using the scanner 150. Next, a virtual 3D model of the foreign object may be generated using the image of the 3D object. Then, an entry for the scanned foreign object may be added to the reference object 130. This entry may be for the same object and / or for another very similar object. This entry may include the generated virtual 3D model, the name of the object, and / or one or more physical properties of the object.

[0057] When the foreign body identification module 115 identifies a foreign body in the intraoral image, it may remove the foreign body from the intraoral image using a filter, leave the foreign body as is, or add more detailed information about the foreign body to the intraoral image (or enhance the representation of the foreign body in the intraoral image in another way).

[0058] The image registration module 120 registers the intraoral images processed by the foreign body identification module 115 using one or more image registration methods. For example, the image registration module 120 may perform image registration using a modified intraoral image from which the representation of foreign bodies has been removed. After the completion of image registration, or during the performance of image registration, the model generation module 125 generates a 3D virtual model of the imaged dental area.

[0059] The user interface 140 may function during an intraoral scan session. When the scanner 150 generates intraoral images and these images are processed by the foreign body identification module 115, a view screen displaying a two-dimensional (2D) or three-dimensional representation of the scanned dental area may be output to a display (e.g., of the computing device 105). Foreign bodies filtered out by the foreign body identification module 115 may not be displayed in the 2D or 3D representation. Images can be stitched together during the intraoral scan session to continuously update the representation of the dental area during the session. Image registration and model generation processes may be performed in a similar manner to those performed by the image registration module 120 and the model generation module 125, but more quickly and with lower accuracy. The representation of the dental area can show the user of the scanner 150 which areas are sufficiently imaged and which areas would benefit from further imaging.

[0060] In some cases, the foreign body identification module 115 may not be able to accurately determine whether an object in the dental area is a foreign body or not. In such cases, the foreign body identification module 115 may or may not filter out the foreign body. If the foreign body is filtered out, the user interface may continue to display the foreign body on the view screen as a black and white outline, a semi-transparent representation, a mesh representation, or other unusual representation. When the user then selects the representation of the foreign body displayed on the view screen, they may be presented with the option to proceed with filtering out the foreign body or to return the foreign body to the intraoral image. If the user chooses to filter out the foreign body, the outline representation or other unusual representation of the foreign body may be removed from the view screen. Furthermore, data about the foreign body (e.g., physical properties) may be added to the entry of the reference object 130. Then, in subsequent scans of the same patient and / or another patient, it is possible to correctly identify further instances of the foreign body using the new reference object 130 and its associated data. Therefore, it is possible to continuously improve the accuracy of the foreign object identification module. If the user chooses not to remove the foreign object with a filter, the foreign object may be returned to the intraoral image (or the original intraoral image may be saved and used instead of the modified intraoral image). If the user does not select a foreign object, the initial decision to remove the foreign object with a filter may be upheld. In such a case, a new reference object may or may not be added to the data store 110 with respect to that foreign object.

[0061] If a foreign object is not initially removed by a filter, it will appear on the view screen alongside other features and objects in the dental area. However, the foreign object may be highlighted by contouring, highlighting, or other means. When the user then selects the representation of the foreign object on the view screen, they may be presented with the option to remove the foreign object with a filter or leave it in the intraoral image. If the user chooses to remove the foreign object with a filter, it may be removed from the intraoral image of the intraoral scan depicting the foreign object. Furthermore, data about the foreign object (e.g., physical properties) may be added to the entry for reference object 130. Subsequent scans of the same patient and / or another patient can then use the new reference object 130 and its associated data to correctly identify further instances of the foreign object. Thus, the accuracy of the foreign object identification module can be continuously improved. If the user chooses to leave the foreign object (which presumably indicates that the object is not actually a foreign object but a non-foreign object), the intraoral image including the representation of the object does not need to be modified, and the outline or other emphasis of the object may be removed from the view screen. If the user does not make a selection regarding the foreign object, the initial decision to leave the foreign object may be accepted.

[0062] In some cases, foreign objects may be intentionally introduced into a patient's oral cavity for imaging and / or dental treatment or orthodontic treatment. In such cases, the foreign object identification module 115 does not need to filter out the representation of the foreign object (which may be a reference object). Instead, the foreign object identification module 115 may identify the foreign object and use its known physical properties (e.g., shape, color, etc.) to augment the image with further information about the foreign object. As an addition or alternative, the known properties of the foreign object may be used to augment a virtual 3D model generated from the image or otherwise associated with the image. For example, a user may want to scan a foreign object to obtain an accurate and complete representation of it, but the foreign object may have geometry that is difficult to capture completely and / or may be made of a material that is difficult to image (e.g., titanium, etc.). Once the foreign object identification module 115 identifies the foreign object as matching a known reference object 130, it can use the known physical properties of the reference object to augment the representation of the foreign object in the intraoral image and / or in the virtual 3D model. For example, it is possible to identify the position and orientation of foreign objects in an image, and use this position and orientation to add data about the foreign object (e.g., shape data or surface data) to the image and / or virtual 3D model. This process can be useful, for example, when the foreign object is a scanned body, brace, etc.

[0063] Once a foreign object to be augmented using the reference object 130 is identified, the user interface 140 may add further information about that foreign object to the view screen. For example, if the user scans the edge of a scan body and it is identified as matching the reference object 130, the unscanned portion of the scan body may be registered in the correct position and orientation relative to other features on the dental area of ​​the view screen and displayed in that position and orientation. This further unscanned portion of the scan body may be displayed, for example, as a mesh or contour. This makes it possible to show the user additional images that need to be acquired to collect enough images to fully capture the foreign object. When a new image is generated, the mesh or contour representation of the foreign object may be replaced with the actual scan data of that foreign object. If there are areas that the user ultimately cannot image correctly, the information from the reference object 130 may be used to fill in the missing information. Such missing information to be filled in may be filled on the intraoral image and / or on a virtual 3D model generated from or otherwise associated with the intraoral image.

[0064] Figures 1B-19 illustrate methods related to intraoral scanning and the identification and processing of foreign objects in intraoral images generated during intraoral scanning. These methods may be implemented by processing logic including hardware (e.g., circuits, dedicated logic, programmable logic, microcode, etc.), software (e.g., instructions executed on a processing unit), or a combination thereof. Various embodiments may be implemented by processing logic executed on the computing device 105 shown in Figure 1A. These methods may be implemented in real-time or near real-time at the time the image is generated by the scanner and / or at the time the image is received from the scanner.

[0065] Figure 1B shows a flowchart of a method 152 for identifying the representation of foreign objects in an intraoral image according to an embodiment of the present disclosure. In block 154, multiple areas of the patient's dentition and / or oral cavity and / or one or more tools may be pre-treated for intraoral scanning. Pre-treatment of the dentition and / or oral cavity may include spraying such areas with a dye of a color not naturally present in the patient's oral cavity. For example, the patient's tongue may be spray-coated, or areas of the patient's dental arch that are not to be scanned may be spray-coated. Similarly, one or more tools to be inserted into the patient's oral cavity may be spray-coated. Some tools may already be of a specific color not naturally present in the patient's oral cavity. For example, the dentist may wear gloves of the same color as the spray-coating dye.

[0066] In block 156, a dental practitioner scans the dental arch (or part of the dental arch) using the scanner 150 shown in Figure 1A. In block 157, processing logic processes one or more intraoral images acquired by the intraoral scan to identify all foreign objects. The intraoral images may be intraoral images included in the scan data generated by the scanner 150. Alternatively, a 3D model may be generated from the intraoral images generated by the scanner 150. This 3D model may then be projected onto a 2D plane to generate a new image that potentially contains data from multiple intraoral images generated by the scanner 150. This new image may be processed to identify all foreign objects within the new image. Figures 2-9 show various foreign object identification methods, which may be used separately or together. In block 158, the user may be presented with options to provide user input regarding the identified foreign objects. For example, the user may have options such as marking an identified foreign object as not being a foreign object, marking an identified non-foreign object as being a foreign object, or acknowledging an identified foreign object as being a foreign object. If the user makes any user input in block 158, such user input is recorded in block 166. Such user input may then be used to train the foreign object identification module 115. This will be described in more detail later.

[0067] Two different operations may be performed on the identified foreign object. In the first operation, a virtual object (e.g., a virtual 3D model or other representation of the foreign object) may be added to one or more intraoral images and / or associated virtual 3D models in an appropriate position and orientation (block 160). Alternatively, the detected foreign object may be filtered out of the intraoral image (i.e., removed) (block 162). If two or more foreign objects are detected, a decision may be made individually for each foreign object whether to filter out the detected foreign objects or to snap a virtual object onto the detected foreign objects.

[0068] In one example of snapping virtual objects onto intraoral images and / or virtual 3D models, brackets may be identified on the patient's teeth. Brackets can be difficult to scan, and images and / or 3D models of brackets may be of low quality. However, by using known data of the identified brackets, it is possible to complete the representation of the brackets in the images and / or virtual 3D models, displaying the complete brackets (for example, as if they were precisely shaped).

[0069] Next, in block 164, a virtual model is generated based on the modified intraoral image (which may include additional data about the foreign object, or may not include any representation of the foreign object). In some embodiments, instead of snapping the virtual object to the image (in block 160), a virtual model constructed from one or more intraoral images based on a foreign object identified in one or more intraoral images may be snapped to the virtual object. For example, the intraoral images used to generate the virtual 3D model may be used to determine the position and orientation of a known object, and this information may be used to add information about the known object to the virtual 3D model. In another example, an image may be generated by projecting the virtual 3D model onto a 2D plane, and by processing that image, the identification of known objects in the image, as well as the determination of the position and orientation of the known objects, may be performed. Then, the determined position and orientation of the known objects may be used to add information about the known objects to the virtual 3D model. Alternatively, the virtual object may be snapped to both the intraoral image and the virtual 3D model.

[0070] As described above, in block 180, the processing logic may train the foreign body identification module to better identify foreign bodies in intraoral images. The foreign body identification module may be trained using data from a number of different sources. In some embodiments, the foreign body identification module is pre-trained and fed in with default settings 170. Such default settings may include global color data (or other wavelength reflectance, wavelength reactivity, wavelength refractive index, etc.) for colors that are commonly present in the oral cavity and / or not present in the oral cavity. Such default settings may further include textures, patterns, shapes, etc. that are commonly present in the oral cavity and / or those that are not present in the oral cavity. In one example, data for several common dental tools may be pre-filled in a default reference object library. Each entry in the default reference object library may include information such as a virtual 3D model, shape information, color information, surface pattern information, texture information, etc. for a particular dental tool.

[0071] In some embodiments, block 172 may receive patient data indicating one or more foreign objects (e.g., a specific bracket attached to the patient's tooth) present in the oral cavity of a particular patient. This allows for narrowing the search range for foreign objects and adjusting the search criteria and / or search boundaries considering the fact that there is a 100% probability that the indicated foreign object is present in the oral cavity. Foreign objects may be indicated based on the user scanning a barcode, manually entering a code or product identifier, selecting a foreign object from a drop-down menu, etc. Based on the provided information, it is possible to identify the foreign object in a reference object library and determine the physical characteristics to be searched for in the intraoral image.

[0072] In some embodiments, block 174 may receive an indication that a particular scanning procedure has been performed. For example, some types of scanning procedures may involve scanning to obtain specific information. For example, a retainer scanning procedure may be performed to scan a patient's oral cavity to depict retainers on the patient's dentition. A retainer scanning procedure may include retrieving or looking up bracket information (e.g., from a reference object library). Similarly, a restorative scanning procedure may include retrieving or looking up restorative object information (e.g., from a reference object library). The foreign body detection module may then adjust its parameters to find brackets (in an instance of a retainer scanning procedure) or restorative objects (in an instance of a restorative scanning procedure).

[0073] In some embodiments, the system may enter training mode in block 178. While the foreign object identification module is in training mode, the user may use scanner 150 to generate a 3D scan of the foreign object. This may include generating a series of 3D images and / or 3D videos of the object. These images and / or videos may then be used to generate a virtual 3D model of the object. Furthermore, one or more image processing algorithms may be applied to these images and / or videos to identify one or more physical properties of the object. These images, videos, virtual 3D models, and / or physical properties may then be added to a new entry for the foreign object in a reference object library.

[0074] Once training is complete, the foreign object identification module will be able to more accurately identify specific foreign objects in block 157.

[0075] Figure 2 shows a flowchart of method 200 for filtering out foreign object representations from intraoral images according to an embodiment of the present disclosure. A dental practitioner may perform an intraoral scan of a dental area (e.g., the maxillary and / or mandibular dental arch, or a portion of the maxillary and / or mandibular dental arch). In block 205 of method 200, during or after the intraoral scan, processing logic may receive scan data containing one or more intraoral images of the dental area. Each intraoral image may be a three-dimensional (3D) image generated by an intraoral scanner (e.g., scanner 150 in Figure 1).

[0076] In block 210, the processing logic analyzes the scan data and identifies representations of foreign objects in the intraoral images based on the analysis. In some embodiments, analyzing the scan data includes processing one or more intraoral images in the scan data using one or more image processing algorithms. Examples of image processing algorithms that can be used to process intraoral images include edge detection algorithms, 3D object detection algorithms, and feature extraction algorithms (e.g., Speeded Up Robust Features (SURF) algorithm, Scale-Invariant Feature Transform (SIFT) algorithm, Haar-like feature algorithm, Histogram of Oriented Gradients (HOG) algorithm, etc.). After the intraoral images have been processed, their shape, edges, features, etc., may be determined. These shapes, edges, features, etc., may then be further analyzed to distinguish between non-foreign objects and foreign objects. In one example, foreign objects may often have a uniform surface pattern and / or texture in at least a portion of the object. Such a uniform surface pattern and / or texture may be used to define the contour of a foreign object, which extends around the area having the uniform surface pattern and / or texture.

[0077] In one embodiment, in block 215, the processing logic analyzes the scan data to identify one or more physical properties represented in the intraoral image. This may include analyzing the intraoral image before or after any of the image processing algorithms described above are applied to the intraoral image. Multiple different types of physical properties may be detected, including reflectance, refractive index, responsiveness to wavelengths of light, as well as texture, surface pattern, color, and / or shape. Other physical and / or optical properties may also be detected.

[0078] In some cases, foreign body identification may be based on only one physical characteristic (for example, color). For example, dental instruments, dentist's gloves, cotton rolls, etc., may all be dyed in colors that do not naturally exist in the oral cavity. For example, blue is not a color that naturally exists in teeth or gums. Therefore, shades of blue may be used. Furthermore, a dental practitioner may spray objects inserted into the oral cavity with dyes of colors that do not naturally exist in the oral cavity. The dental practitioner may also spray the dye on the patient's tongue so that the tongue can be identified as a foreign body based on its color. If there are areas with unnatural colors in an intraoral image, they can be easily identified as belonging to a foreign body with minimal processing. Simple color filters may be applied to intraoral images, for example, color filters that remove one or more colors (for example, a range of colors) that do not naturally exist in the oral cavity.

[0079] In some cases, it is possible to identify an object as a foreign object by using a combination of multiple physical properties of that object. For example, the shape of an object and its surface pattern or texture may be used together to identify a foreign object. The more identifiable physical properties an object has, the easier and more accurate it becomes to determine whether it is a foreign object or not.

[0080] Regarding data on responsiveness to light wavelengths, specific wavelengths of light may be shone on the dental area while the intraoral image is being generated. An example of usable wavelengths is near-infrared (NIR) light. Other examples include IR light, ultraviolet (UV) light, near-ultraviolet (NUV) light, mid-infrared (MIR) light, far-infrared (FIR) light, extreme ultraviolet (EUV) light, and ultra-high frequency (EHF) electromagnetic radiation. Non-foreign objects in the dental area may respond to light wavelengths in one way, while foreign objects in the dental area may respond to light wavelengths in another way. For example, teeth and tissues in the oral cavity may have a specific reflectivity to NIR light, while metallic objects may have a very different reflectivity to NIR light. For instance, foreign objects typically appear much darker to NIR light than teeth and tissues. Furthermore, the reflectivity of blood and saliva to NIR light may differ from that of teeth and tissues, making it possible to detect blood and / or saliva in intraoral images using NIR light. Such blood and / or saliva may be identified as foreign matter depending on the embodiment.

[0081] In one embodiment, the processing logic identifies bubbles in an intraoral image. Bubbles may be a sign of saliva and are not generally present as part of the teeth or gingival tissue. Therefore, the presence of bubbles can be considered another physical characteristic that can be detected and used to identify saliva and / or blood.

[0082] In block 220, the processing logic identifies a representation of a foreign object as belonging to a foreign object based on one or more physical properties (for example, reflectance, refractive index, responsiveness to wavelengths of light, and texture, surface pattern, color, and / or shape). This may include comparing the identified physical properties with the physical properties of one or more reference objects in a reference object library. If a match is found between one or more physical properties of an object in an intraoral image and one or more physical properties of a reference object, the representation of that object may be identified as a foreign object.

[0083] In block 225, the processing logic modifies the intraoral image by removing the representation of foreign objects from the intraoral image. In block 230, the processing logic receives additional scan data, including additional intraoral images of dental areas, during the intraoral scan. These intraoral images may be received first, together with the first intraoral image, before the operation in block 210 is performed. Alternatively, one or more additional images may be received after the first image has been received, while (or after) the operations in blocks 210 and / or 225 are being performed on the first intraoral image.

[0084] In block 235, the processing logic analyzes the additional scan data in the same manner as described in relation to block 210. Next, in block 240, the processing logic determines whether any of the additional intraoral images contain representations of foreign objects. If so, those additional images are also modified in block 245 by removing the foreign objects from the additional intraoral images. If they do not contain foreign objects, the process proceeds to block 255.

[0085] In block 255, the processing logic registers (i.e., "stitches") the intraoral images together. This may include, for example, registering a modified first intraoral image to a modified second intraoral image, or registering a modified first intraoral image to an unmodified second intraoral image. In one embodiment, performing image registration includes capturing 3D data of various points on the surface in multiple images and registering those images together by calculating transformations between them. These images may then be unified and placed into a common reference frame by applying appropriate transformations to the points in each registered image. The processing logic then generates a virtual 3D model of the dental site based on the registration using the modified intraoral images and additional intraoral images (which may or may not be modified). The virtual 3D model may be a digital model that shows the surface features of the dental site.

[0086] Figure 3 shows a flowchart of a method 300 for identifying foreign objects in an intraoral image according to an embodiment of the present disclosure. In block 305 of method 300, the processing logic performs image processing on the intraoral image to identify the shape, texture, surface pattern, color, and reflectance, refractive index, and / or responsiveness to wavelengths of light in the intraoral image. This image processing may be performed using an edge detection algorithm, a feature extraction algorithm, and / or other image processing algorithms (e.g., the image processing algorithms described above).

[0087] In block 310, the processing logic identifies shapes, textures, surface patterns, colors, and reflectivity, refractive index, and / or responsiveness to wavelengths of light in the intraoral image that do not naturally exist in dental areas. This may include, in block 315, comparing the identified shapes, textures, surface patterns, colors, and reflectivity, refractive index, and / or responsiveness to wavelengths of light in the intraoral image with shapes, textures, surface patterns, colors, and reflectivity, refractive index, and / or responsiveness to wavelengths of light that are known to naturally exist in the oral cavity. Then, in block 320, the processing logic may, based on this comparison, identify shapes, textures, surface patterns, colors, and reflectivity, refractive index, and / or responsiveness to wavelengths of light from the intraoral image that do not naturally exist in the oral cavity.

[0088] In block 325, the processing logic determines the contour of a region in the intraoral image that has at least one of the following characteristics: shape, texture, surface pattern, color, and reflectance, refractive index, and / or responsiveness to wavelengths of light, which are determined not to be naturally present in the oral cavity. In block 330, the processing logic may compare the contour, shape, texture, surface pattern, color, and reflectance, refractive index, and / or responsiveness to wavelengths of light in the intraoral image with the known shape, texture, surface pattern, color, and reflectance, refractive index, and / or responsiveness to wavelengths of light of a reference object. In block 335, the processing logic may determine, in view of the comparison performed in block 330, whether the shape, texture, surface pattern, color, and reflectance, refractive index, and / or responsiveness to wavelengths of light in the intraoral image match the shape, texture, surface pattern, color, and reflectance, refractive index, and / or responsiveness to wavelengths of light of any known reference object. If a match is found, the method proceeds to block 340, where the representation of the foreign object is identified. Subsequently, the representation of the foreign object may be removed from the intraoral image. However, in some embodiments, shapes, textures, surface patterns, colors, and reflectance, refractive index, and / or responsiveness to wavelengths of light that do not naturally exist in the oral cavity may be removed from the intraoral image in block 310 or block 325 without comparison to a library of reference objects. If no match is found in block 335, the method proceeds to block 345, and no foreign object is identified.

[0089] Figure 4 shows a flowchart of Method 400 for adding an entry for a foreign object to a foreign object reference library according to an embodiment of the present disclosure. In block 405 of Method 400, the processing logic enters training mode (for example, based on user input). In block 410, the processing logic receives one or more images of foreign objects to be added to a known reference object library. Image reception may occur during a 3D scan of the foreign object. In block 415, the processing logic generates a virtual model of the foreign object based on the images. The processing logic may also perform image processing on the images and / or virtual model to identify one or more physical properties of the foreign object. In block 420, the processing logic may add the virtual model of the foreign object, the images of the foreign object, and / or the physical properties of the foreign object to the foreign object reference library. Thereafter, the representation of the object is identified as a representation of the foreign object in an intraoral image.

[0090] Figure 5 shows a flowchart of a method 500 for identifying foreign objects in an intraoral image using a foreign object reference library, according to an embodiment of the present disclosure. In block 505 of method 500, the processing logic receives an indication that a foreign object is present in the dental area. This indication may include identification information of the foreign object (e.g., by name, unique identifier, characteristics, etc.). For example, an orthodontist may indicate that a patient is wearing a specific brace bracket. In block 510, the processing logic queries the reference object library using the identification information of a known object. In block 515, the processing logic receives a response to that query. The response may include one or more physical characteristics of the foreign object, a virtual model of the foreign object, metadata associated with the entry for the foreign object, and / or other information. In block 520, the processing logic identifies the foreign object in the intraoral image using one or more known characteristics of the foreign object. For example, the processing logic may search the intraoral image for objects having the physical characteristics identified in the response.

[0091] Figure 6 shows a flowchart of a method 600 for identifying foreign objects in an intraoral image based on color, according to an embodiment of the present disclosure. In block 605 of method 600, the processing logic selects a pixel (or voxel) in the intraoral image. In block 610, the processing logic examines at least one of the saturation value or intensity value associated with the pixel (or voxel) for each color channel of the color encoding system. In block 615, the processing logic determines a tuple containing at least one of the saturation value or intensity value for each color channel of the color encoding system. For example, if an RGB color encoding system is used, the tuple may be a 3-tuple (r,g,b).

[0092] In block 620, the processing logic compares the generated tuple to a specified color range. The specified color range may cover some or all of the colors naturally present in the oral cavity, or it may cover some or all of the colors not naturally present in the oral cavity. In block 625, the processing logic determines whether the tuple is outside the color range (if the color range represents colors naturally present in the oral cavity). If the tuple is outside the color range (and the color range represents colors naturally present in the oral cavity), the process proceeds to block 630, where the pixel (or voxel) is marked as potentially associated with a foreign object. If the tuple is within the color range, the process proceeds to block 635.

[0093] Alternatively, in block 625, the processing logic may determine whether the tuple is within the color range (if the color range indicates a color not naturally present in the oral cavity). If the tuple is within the color range (and the color range indicates a color not naturally present in the oral cavity), the method proceeds to block 630, where the pixel (or voxel) is marked as potentially associated with a foreign object. If the tuple is outside the color range, the method proceeds to block 635.

[0094] In block 635, the processing logic determines whether all pixels of the intraoral image have been processed. If there are further pixels to process, the method returns to block 605, where another pixel is selected for analysis. If all pixels of the intraoral image have been processed, the method proceeds to block 640. In block 640, the processing logic identifies tuples that fall outside a specified color range (if the color range represents colors naturally present in the oral cavity), or consecutive pixels or voxels that have tuples within a specified color range (if the color range represents colors not naturally present in the oral cavity). In block 645, the processing logic determines the contour around the identified consecutive pixels or voxels. In block 650, the processing logic identifies pixels or voxels within the contour as representations of a foreign object. Alternatively, the operations in blocks 640-645 may be omitted. Instead, all pixels marked as potentially associated with a foreign object may be filtered out of the intraoral image.

[0095] Figure 7 shows a flowchart of Method 700, which provides a user interface for managing foreign objects in intraoral images according to an embodiment of the present disclosure. In block 705 of Method 700, the processing logic receives scan data, including an intraoral image, during an intraoral scan of a dental site. In block 710, the processing logic identifies representations of foreign objects in the intraoral image based on analysis of the scan data. In block 720, the processing logic modifies the intraoral image by removing the representations of foreign objects from the intraoral image. In block 730, during the intraoral scan, the processing logic generates a view of the dental site based on the modified image.

[0096] In block 735, the processing logic generates a contour, mesh representation, translucent representation, or other representation of the foreign object. In block 745, the processing logic adds the contour (or other representation) of the foreign object to the view. In block 750, the processing logic provides a user interface that allows the user to select the foreign object. In block 755, the processing logic receives the user selection of the foreign object. The user selection indicates that the object should not be removed from the intraoral image (for example, the object may be identified as a non-foreign object). In block 760, the processing logic returns the foreign object to the intraoral image.

[0097] Figure 8 shows a flowchart of another method 800 that provides a user interface for managing foreign objects in intraoral images according to an embodiment of the present disclosure. In block 805 of method 800, processing logic receives scan data, including intraoral images, during an intraoral scan of a dental site. In block 810, processing logic identifies representations of foreign objects in the intraoral images based on analysis of the scan data. In block 815, processing logic determines a confidence value associated with that representation. In block 820, processing logic determines whether the confidence value falls below a threshold. If the confidence value falls below a threshold, this indicates that the processing logic cannot correctly determine whether the identified object should be classified as a foreign object or a non-foreign object.

[0098] In block 830, during intraoral scanning, the processing logic generates a view of the dental area based on the modified image. In block 835, the processing logic generates possible foreign body contours, mesh representations, translucent representations, or other representations. In block 845, the processing logic adds the foreign body contour (or other representation) to the view. In block 850, the processing logic provides a user interface that allows the user to a) remove the foreign body representation from the intraoral image, or b) leave the foreign body representation in the intraoral image.

[0099] In block 855, the processing logic receives a user selection to remove the representation from the intraoral image. In block 860, the processing logic identifies one or more physical properties of the foreign object. In block 865, the processing logic removes the foreign object from the intraoral image. In block 870, the processing logic adds an entry for the foreign object to the reference object library. The entry may include the identified physical properties of the foreign object. The processing logic can then identify the foreign object with improved reliability and accuracy in subsequent intraoral images.

[0100] In some embodiments, machine learning models such as convolutional neural networks (CNNs) are used to identify foreign objects in intraoral images. Intraoral images may be images generated by an intraoral scanner and / or images generated by projecting a virtual 3D model of a dental site onto a 2D plane or surface. Such images may be height maps or may include height maps. The machine learning model may, as an alternative or additional method, directly identify foreign objects in the virtual 3D model (for example, by inputting data representing the virtual 3D model into the machine learning model). In one exemplary embodiment, a method for training a machine learning model includes the step of receiving a training dataset comprising a plurality of intraoral images, each of the plurality of intraoral images comprising a depiction of a dental site and a label indicating whether the intraoral image contains a foreign object. Intraoral images containing foreign objects may further include a given outline of the foreign object. Intraoral images may be real images and / or computer-generated images. The given outline may be generated by a manual process in which a user outlines the foreign object in the image, or by an automatic or semi-automatic process. In some embodiments, intraoral images containing foreign bodies may be further labeled with the class, type, or category of the foreign body. Each type of foreign body may be, for example, appropriately labeled.

[0101] Training a machine learning model involves inputting a training dataset into an untrained machine learning model (e.g., a CNN or other neural network) and training the untrained model based on that training dataset to produce a trained machine learning model that identifies foreign objects in intraoral images. The machine learning model may also be trained to generate the contours of foreign objects. In one embodiment, the machine learning model is trained to generate a binary mask, where the first value of the binary mask represents pixels associated with foreign objects and the second value of the binary mask represents pixels associated with non-foreign objects. Furthermore, the trained machine learning model may output an indication of the type of foreign object identified in the intraoral image for the input image.

[0102] The processing logic typically processes each image in the training dataset one by one. The processing logic processes the intraoral images to determine an output (classification or label), and compares the determined classification or label (or multiple classifications or labels) with a set of classifications or labels associated with the intraoral images to reveal one or more classification errors. An error term or delta may be determined at each node of the artificial neural network. Based on this error, the artificial neural network adjusts one or more of its parameters (weights for one or more inputs to the node) for one or more of its nodes. The parameters may be updated in a backpropagation manner, so that the nodes in the top layer are updated first, followed by the nodes in the next layer. The artificial neural network contains multiple layers of "neurons," each layer receiving values ​​as input from the neurons in the preceding layer. The parameters of each neuron include weights associated with the values ​​received from each of the neurons in the preceding layer. Therefore, adjusting the parameters may involve adjusting the weights assigned to each of the inputs to one or more neurons in one or more layers of the artificial neural network.

[0103] In some embodiments, the initial machine learning model is first trained using a small dataset containing manually labeled and contoured foreign objects. This machine learning model can then be used to automatically define contours around foreign objects in additional images of the training dataset, which are then used to train a more accurate machine learning model (or to further train the initial machine learning model). The contours generated for the images in the training dataset can be reviewed by the user and corrected if incorrect. The final model can then be trained using the expanded dataset to define contours around foreign objects in intraoral images and to identify foreign objects in intraoral images. This iterative process of generating machine learning models that define contours around foreign objects makes it possible to prepare a large training dataset that minimizes the time the user spends on image annotation.

[0104] Figure 9 shows a flowchart of Method 900, according to embodiments of the present disclosure, for filtering out foreign object representations from intraoral images based on the use of a trained machine learning model (e.g., a deep learning model, an artificial neural network, a convolutional neural network, etc.). In block 902 of Method 900, the processing logic may perform feature extraction on the intraoral image to identify features in the image. This may include performing edge detection, object detection, and / or other feature extraction techniques described herein.

[0105] In block 905, the processing logic processes intraoral images (and / or extracted features of intraoral images) using a trained machine learning model. In one embodiment, the trained machine learning model is a deep neural network. In one embodiment, the trained machine learning model is a CNN. A convolutional neural network (CNN) hosts, for example, multiple layers of convolutional filters. Pooling is performed in the lower layers (and nonlinearity is often dealt with), and a multilayer perceptron is usually added on top of the lower layers to map the top layer features extracted by the convolutional layers to a decision (e.g., classification output). Deep learning is a class of machine learning algorithms that uses a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer takes the output from the preceding layer as input. Deep neural networks can be trained in a supervised manner (e.g., classification) and / or unsupervised manner (e.g., pattern analysis). A deep neural network contains a hierarchy of layers, where different layers learn different levels of representation corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in image recognition applications, the raw input may be a matrix of pixels, the first representation layer may abstract the pixels and encode the edges, the second layer may construct and encode the array of edges, the third layer may encode higher-level shapes (e.g., teeth, lips, gums, etc.), and the fourth layer may recognize that the image contains a face or define the contours around foreign objects in the image. In particular, the deep learning process can learn which features should be optimally placed on themselves at which level. The "deep" in "deep learning" means that the data is transformed across several layers. More precisely, a deep learning system has a substantial credit assignment path (CAP) depth. A CAP is a chain of transformations from input to output. A CAP describes the latent causal relationships between input and output. In the case of a feedforward neural network, the CAP depth may be the network depth, or it may be the number of hidden layers plus 1.In the case of a recurrent neural network where a signal can propagate by passing through a single layer more than once, the CAP depth is potentially unlimited. In one embodiment, the machine learning model is a region convolutional neural network (R-CNN). R-CNN is a type of CNN that is capable of searching for and detecting objects in an image.

[0106] Machine learning models classify or label images based on their current parameter values. An artificial neural network includes an input layer consisting of data point values ​​(e.g., RGB values ​​of pixels in an intraoral image). The next layer is called a hidden layer, and each layer in the hidden layer receives one or more of the input values. Each node contains parameters (e.g., weights) applied to the input values. Thus, each node essentially inputs the input values ​​into a multivariable function (e.g., a nonlinear mathematical transformation) to produce an output value. The next layer may be another hidden layer or an output layer. In either case, the nodes in the next layer receive output values ​​from the nodes in the previous layer, and each node applies weights to those values ​​to produce its own output value. This may be done in each layer. The final layer is the output layer, where there is one node per class. In the case of an artificial neural network, there may be a first class (containing foreign objects) and a second class (not containing foreign objects). Furthermore, the class may be determined for each pixel in the image. The final layer applies, for each pixel in the image, the probability that the pixel belongs to a first class (foreign object) and the probability that the pixel belongs to a second class (non-foreign object). In embodiments where the machine learning model is trained to identify multiple different types of foreign objects, different classes may be determined for each pixel in the image.

[0107] In block 910, the processing logic receives the output of the machine learning model. This output may include a binary mask having the same number of entries as pixels or voxels in the intraoral image, where entries associated with pixels or voxels that are part of a foreign object have a first value, and entries associated with pixels or voxels that are not part of a foreign object have a second value. In block 915, the processing logic modifies the intraoral image using the mask. Pixels or voxels with the first value may be removed from the intraoral image, while pixels or voxels with the second value may remain in the intraoral image.

[0108] Figure 10 shows a portion of an exemplary dental arch 1005 with a foreign object 1030 placed on it. As shown in the figure, the portion of the dental arch 1005 includes molars 1010 and 1020. The foreign object 1030 is a dental mirror, which is placed in the interdental region between molars 1010 and 1020. Two intraoral images 1058 and 1060 of the dental arch 1005 are generated.

[0109] Figure 11 shows a first intraoral image 1058 including a first representation of a foreign object 1030 according to one embodiment of the present disclosure. The first intraoral image 1058 may be processed using one or more of the methods described herein to identify the foreign object 1030 in the intraoral image 1058.

[0110] Figure 12 shows a modified version 1068 of the first intraoral image in Figure 11 according to one embodiment of the present disclosure, in which the foreign body has been removed by a filter. Next, a virtual 3D model of the dental arch 1005 may be generated using the modified image 1068. By removing the foreign body 1030 from the intraoral image 1058 before generating the virtual 3D model, it is possible to improve the generation speed and accuracy of the virtual model.

[0111] Furthermore, the mirror surface of the dental mirror may include reflections of teeth and / or gingival tissue. These reflections may appear to have the physical properties of a non-foreign object in the intraoral image 158. However, the mirror surface is surrounded by a non-reflective edge. In the embodiment, the processing logic can ignore information surrounded by edges having a specific shape and / or other criteria. This prevents reflections of natural features appearing in reflective foreign objects from confusing the processing logic and causing the foreign object to be misidentified as a non-foreign object.

[0112] Figure 13A shows a view of a portion of an exemplary dental arch 1305 and a foreign object 1310 placed thereon, according to one embodiment of the present disclosure. The foreign object 1310 may be, for example, a bracket, attachment, scan body, stock implant abutment, or other permanently mounted object. In some embodiments, the foreign object may be made of a material that is difficult to image (such as titanium).

[0113] Figure 13B shows a view of Figure 13A, indicating an unscanned region of the foreign object 1310 according to one embodiment of the present disclosure. The foreign object 1310 may be identified using one of the methods described above. The processing logic may then embed the overall representation of the foreign object into a view of the dental arch 1305. As shown in the figure, the unscanned portion of the foreign object 1310 is indicated by a mesh contour. As the user generates additional intraoral images of the foreign object, image data from the additional intraoral images may be embedded in the mesh contour.

[0114] Embodiments have been described relating to the processing of images of dental areas (e.g., intraoral images) to identify foreign objects (e.g., reference objects) present therein. Note that the same or equivalent methods described herein can be used to identify foreign objects in a virtual 3D model generated by stitching together multiple intraoral images. Furthermore, the images processed to identify foreign objects may be generated by an intraoral scanner, or by projecting a virtual 3D model of the dental area onto a 2D plane or surface. Identified foreign objects may be removed from the image and / or virtual 3D model by filtering, or they may be enhanced in the image and / or virtual 3D model as described above.

[0115] Figure 14 shows a schematic diagram of an exemplary machine, computing device 1400, on which a set of instructions may be executed to cause the machine to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine may be connected to other machines (e.g., network connection) on a local area network (LAN), intranet, extranet, or internet. The machine may operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), mobile phone, web-based electronic device, server, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential instructions or other types of instructions) that specify the actions to be performed by the machine. Furthermore, although the figure shows only one machine, the term “machine” should also be interpreted to encompass any collection of machines (e.g., computers) that individually or collectively execute a set (or set) of instructions that implement any one or more of the methodologies discussed herein.

[0116] The exemplary computing device 1400 includes a processing unit 1402, a main memory 1404 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) (e.g., synchronous DRAM (SDRAM), etc.)), a static memory 1406 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 1428), which communicate with each other via a bus 1408.

[0117] The processing unit 1402 represents one or more general-purpose processors (e.g., microprocessors, central processing units, etc.). More specifically, the processing unit 1402 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing another instruction set, or a processor implementing a combination of instruction sets. The processing unit 1402 may also be one or more dedicated processing units (e.g., application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc.). The processing unit 1402 is configured to execute processing logic (instructions 1426) that perform the operations and steps discussed herein.

[0118] The computing device 1400 may further include a network interface device 1422 for communicating with the network 1464. The computing device 1400 may also include a video display device 1410 (e.g., a liquid crystal display (LCD) or cathode ray tube (CRT)), an alphanumeric input device 1412 (e.g., a keyboard), a cursor control device 1414 (e.g., a mouse), and a signal generating device 1420 (e.g., a speaker).

[0119] The data storage device 1428 may include a machine-readable storage medium (or more specifically, a non-temporary computer-readable storage medium) 1424 which stores one or more instruction sets 1426 that perform any one or more of the methods or functions described herein, provided that non-temporary storage medium means storage medium other than carrier. The instructions 1426 may also be in whole or in part in the main memory 1404 and / or the processing unit 1402 while being executed by the computer device 1400, and the main memory 1404 and the processing unit 1402 also constitute computer-readable storage medium.

[0120] The computer-readable storage medium 1424 may also be used to store a foreign object identification module 1450, which may correspond to a similarly named foreign object identification module 115 in Figure 1. The computer-readable storage medium 1424 may also store a software library containing methods for calling the foreign object identification module 1450. Although the computer-readable storage medium 1424 is shown as a single medium in one exemplary embodiment, the term “computer-readable storage medium” should be interpreted to encompass a single or multiple mediums (e.g., a centralized or distributed database, and / or associated caches and servers) that store one or more instruction sets. The term “computer-readable storage medium” should also be interpreted to encompass any medium capable of storing or encoding instruction sets executed by a machine, thereby enabling the machine to implement any one or more of the methodologies of this disclosure. Accordingly, the term “computer-readable storage medium” should be interpreted to encompass (but not limited to) solid-state memory, as well as optical and magnetic media.

[0121] Figure 15 shows a method 1550 of orthodontic treatment using multiple appliances according to an embodiment. Method 1550 may be performed using a customized orthodontic appliance or set of appliances generated based on a virtual 3D model of the patient's dental arch. The virtual 3D model may be generated using intraoral images that have been processed and optionally modified according to the embodiments described herein. In block 1560, a first orthodontic appliance is applied to the patient's teeth to rearrange them from a first dental arrangement to a second dental arrangement. In block 1570, a second orthodontic appliance is applied to the patient's teeth to rearrange them from a second dental arrangement to a third dental arrangement. Method 1550 may be repeated as many times as necessary using any appropriate number and combination of orthodontic appliances to incrementally rearrange the patient's teeth from an initial arrangement to a target arrangement. Orthodontic appliances may be produced in their entirety at the same stage (e.g., at the start of a stage of treatment), i.e., as a set or batch, or manufactured one at a time, and the patient may wear each appliance until the pressure of each appliance on the teeth is no longer felt, or until the maximum amount of tooth movement specified for that given stage is achieved. Multiple different orthodontic appliances (e.g., sets) can be designed and even manufactured before the patient wears any of those appliances. After wearing an orthodontic appliance for an appropriate period of time, the patient may replace the current appliance with the next appliance in that set, and this can be done until there are no remaining appliances. Orthodontic appliances are generally not glued to the teeth, and the patient can place or replace appliances at any time during treatment (e.g., removable orthodontic appliances). The last or several orthodontic appliances in a series may have a geometry selected to overcorrect the alignment of the teeth. For example, one or more orthodontic appliances may have a geometry that moves individual teeth beyond the alignment selected as the "final" alignment (if fully achieved).Such overcorrection is considered desirable to offset potential relapse after the repositioning method is completed (for example, to allow individual teeth to move back towards their original orthodontic positions). Overcorrection can also help increase the speed of orthodontic treatment (for example, an orthodontic appliance with geometry positioned beyond the desired intermediate or final position can move individual teeth to the desired position more quickly). In such cases, the use of the orthodontic appliance may be terminated before the teeth reach the position set by the appliance. Furthermore, overcorrection may be applied intentionally to compensate for any inaccuracies or limitations of the orthodontic appliance.

[0122] Figure 16 shows a method 1600 for designing an orthodontic appliance. Some or all blocks of method 1600 may be performed by any suitable data processing system or data processing device (e.g., one or more processors configured with appropriate instructions). In block 1610, a movement path is determined for moving one or more teeth from an initial alignment to a target alignment. The initial alignment may be determined from a type or scan of the patient's teeth or oral tissue, using, for example, wax bite, direct contact scan, radiography, radiography, ultrasound imaging, and other techniques to obtain information about the position and structure of the teeth, jaw, gums, and other orthodontic-related tissues. From the acquired data, a digital dataset representing the initial (e.g., pre-treatment) alignment of the patient's teeth and other tissues may be derived. Optionally, the initial digital dataset is processed so that each component of the tissue is segmented from one another. For example, a data structure in which individual tooth crowns are digitally represented may be generated. Advantageously, a digital model of the entire tooth, including the measured or extrapolated hidden surface and root structure, as well as the surrounding bone and soft tissue, may be generated.

[0123] The target tooth alignment (e.g., the desired and intended final outcome of orthodontic treatment) may be received from the clinician in the form of a prescription, calculated from basic orthodontic principles, and / or extrapolated by calculation from the clinician's prescription. The specifications of the desired final positions of the teeth, and the digital representation of the teeth themselves, allow for the specification of the final position and surface geometry of each tooth to form a complete model of the tooth alignment at the final treatment goal.

[0124] Given both the initial and target positions of each tooth, it is possible to define the movement path for each tooth. In some embodiments, the movement path is configured to move the teeth from their initial positions to the desired target positions in the fastest possible manner, minimizing round trips. The tooth path may be optionally segmented, and these segments may be calculated such that the movement of each tooth within the segment remains within the limits of linear and rotational translation. In this way, the endpoints of each path segment can constitute a clinically viable rearrangement, and the aggregate of the segment endpoints can constitute a clinically viable sequence of tooth positions, thereby ensuring that moving from one point to the next in the sequence does not cause collisions between teeth.

[0125] In block 1620, a force system is determined that generates the movement of one or more teeth along the movement path. The force system may include one or more forces and / or one or more torques. Various force systems can cause various types of tooth movement, such as tilting, translation, rotation, eruption, intrusion, and root movement. Biomechanical principles, modeling techniques, force calculation / measurement techniques, etc., including knowledge and approaches commonly used in orthodontics, may be used to determine the appropriate force system to be applied to the teeth to achieve tooth movement. When determining the force system to be applied, sources may be considered, including literature, force systems determined by experiment or virtual modeling, computer-based modeling, clinical experience, and minimization of unnecessary forces.

[0126] The determination of the force system may include constraints on the permissible force (e.g., permissible direction and magnitude), as well as the desired movement that should be brought about by the applied force. For example, when fabricating a palatal dilator, the desired movement strategy may differ depending on the patient. For instance, the magnitude of the force required for palatal separation may vary depending on the patient's age, because in very young patients, the sutures may not be fully formed. Therefore, in younger patients or other patients whose palatal sutures are not fully closed, it may be possible to perform palatal dilation with a smaller force. Also, slowing palatal movement can help bone growth to close the widened sutures. In other patients, faster dilation may be desired, which can be achieved by applying a larger force. These requirements may be incorporated as needed when selecting the structure and materials of the orthodontic appliance, for example, by selecting a palatal dilator capable of applying a large force to separate the palatal sutures and / or cause rapid palatal dilation. Subsequent stages of orthodontic treatment may be designed to apply forces of varying magnitudes, for example, by applying a larger force initially to separate the sutures, and then by applying smaller forces to maintain the separation of the sutures or to gradually expand the palate and / or dental arch.

[0127] Determining the force system may also involve modeling the patient's facial structure (e.g., the skeletal structure of the jaw and palate). For example, scan data of the palate and dental arch (X-ray data or 3D optical scan data, etc.) can be used to determine the parameters of the patient's oral skeletal and muscular systems so that sufficient force is determined to achieve the desired expansion of the palate and / or dental arch. In some embodiments, the thickness and / or density of the middle palatine suture may be measured (input) by the therapist. In another embodiment, the therapist may select the appropriate treatment based on the patient's physiological characteristics. For example, palatal characteristics may be evaluated based on factors such as the patient's age. For example, younger patients typically require less force to expand the suture than older patients because the suture is not yet fully formed.

[0128] In block 1630, an orthodontic appliance configured to generate a force system is determined. The determination of the orthodontic appliance, its geometry, material composition, and / or properties may be carried out using a simulation environment for treatment or force application. The simulation environment may be, for example, a computer modeling system, a biomechanical system, or an apparatus.

[0129] Block 1640 generates instructions for manufacturing orthodontic appliances or molds used in the manufacture of orthodontic appliances. These instructions may be configured to control a manufacturing system or apparatus to manufacture orthodontic appliances and / or molds. In some embodiments, these instructions are configured to manufacture orthodontic appliances by direct manufacturing (e.g., stereolithography, selective laser sintering, fused deposition modeling, 3D printing, continuous direct manufacturing, multi-material direct manufacturing, etc.) according to various methods described herein. In alternative embodiments, these instructions may be configured to perform indirect manufacturing of orthodontic appliances, for example, by directly 3D printing a mold and then thermoforming the mold with a plastic sheet.

[0130] Method 1600 may further include 1) a block for generating three-dimensional data of the palate and maxillary dental arch by performing an intraoral scan of the patient's maxillary dental arch and palate, and 2) a block for determining the three-dimensional shape profile of an orthodontic appliance to give the gap-to-tooth engagement structure described herein.

[0131] Figure 17 shows a method 1700 for digitally planning the design or fabrication of orthodontic treatment and / or orthodontic appliances according to an embodiment. Method 1700 may be applied to any treatment procedure described herein and may be carried out by any suitable data processing system.

[0132] In block 1710, a digital representation of the patient's teeth is received. The digital representation may include surface topography data of the patient's oral cavity (including teeth, gingival tissue, etc.). Surface topography data may be generated by directly scanning the oral cavity, a physical model of the oral cavity (explicit or implicit), or an impression of the oral cavity using an appropriate scanning device (e.g., a handheld scanner, a desktop scanner, etc.).

[0133] In block 1720, one or more treatment stages are generated based on the digital representation of the teeth. These treatment stages may be incremental repositioning stages of orthodontic treatment procedures designed to move one or more of the patient's teeth from an initial alignment to a target alignment. For example, the generation of treatment stages may be performed by determining the initial alignment shown by the digital representation, determining the target alignment, and determining the movement paths of one or more teeth in the initial alignment necessary to achieve the target alignment. Optimization of the movement paths may be based on minimizing the total distance of movement, avoiding collisions between teeth, avoiding more difficult tooth movements, or any other appropriate criteria.

[0134] In block 1730, at least one orthodontic appliance is fabricated based on the generated treatment stages. For example, a set of orthodontic appliances may be fabricated, the shape of which is determined according to the tooth alignment specified by one of the treatment stages, so that the teeth can be incrementally rearranged from the initial alignment to the target alignment by the patient wearing the orthodontic appliances sequentially. The set of orthodontic appliances may include one or more of the orthodontic appliances described herein. The fabrication of the orthodontic appliance may include creating a digital model of the orthodontic appliance to be used as input to a computer-controlled fabrication system. The orthodontic appliance may be formed using direct fabrication methods, indirect fabrication methods, or a combination thereof, as necessary.

[0135] In some cases, staging, such as various arrangements or treatment stages, may not be essential to the design and / or fabrication of orthodontic appliances. As shown by the dashed lines in Figure 17, the design and / or fabrication of orthodontic appliances, and possibly certain orthodontic treatments, may include using a representation of the patient's teeth (e.g., receiving a digital representation of the patient's teeth (1710)) and subsequently designing and / or fabricating orthodontic appliances based on the representation of the patient's teeth in the arrangement represented by the received representation.

[0136] Naturally, the above description is illustrative, not restrictive. Many other embodiments will become apparent upon reading and understanding the above description. While embodiments of this disclosure have been described with reference to specific examples of embodiments, it should be understood that this disclosure is not limited to the embodiments described and may be implemented with modifications and changes that do not deviate from the spirit and scope of the appended claims. Accordingly, this specification and drawings should be taken as illustrative, not restrictive. Therefore, the scope of this disclosure should be determined by reference to the appended claims, together with the entire scope of equivalents to which such claims are entitled. [Note 1] The steps include receiving intraoral scan data containing multiple images of the dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area based on the aforementioned multiple images, The steps include performing an analysis on the image of the dental area, A step of identifying a representation of a reference object in the aforementioned image, wherein the reference object has one or more known characteristics, The steps of modifying at least one of the image or the virtual 3D model by adding additional data relating to the reference object to at least one of the image or the virtual 3D model of the dental area, based on one or more known characteristics of the reference object, A method that includes this. [Note 2] The method according to Appendix 1, wherein the image is one of a plurality of images of the dental area used to generate the virtual 3D model, the image is modified by adding the additional data relating to the reference object to the image, and the modified image is used to generate the virtual 3D model of the dental area. [Note 3] A step of performing image processing on the image to identify a likely object represented in the image, wherein the likely object includes one or more physical characteristics, The steps include comparing one or more physical properties of the likely object with one or more known properties of the reference object, As a result of the comparison, the step of determining whether one or more physical properties of the likely object match one or more known properties of the reference object, The steps include determining whether the likely object is the reference object, The method described in Appendix 2, further including the above. [Note 4] The method according to Appendix 3, wherein the one or more physical properties include reflectance, refractive index, responsiveness to the wavelength of light, and at least one of texture, surface pattern, color, or shape. [Note 5] The method according to Appendix 1, wherein the step of performing the analysis of the image includes inputting the image to a machine learning model trained to identify one or more types of reference objects, the machine learning model outputs a representation of the reference objects in the image. [Note 6] The step of generating the image on which the analysis is performed by projecting the virtual 3D model onto a plane. The method described in Appendix 1, further including the above. [Note 7] During an intraoral scan, a step of generating a view of the dental area based on the modified image, wherein the view includes the contour of the reference object based on the additional data relating to the reference object; The steps include receiving multiple additional images of the dental area during the intraoral scan, The steps include replacing one or more parts of the shape of the reference object based on the aforementioned additional images, A step of updating the view of the dental area, wherein the updated view shows the replaced portion of the shape of the reference object, The method described in Appendix 6, further including the method described in Appendix 6. [Note 8] The step of identifying the representation of the reference object is: The steps include processing the image using a machine learning model trained to identify one or more foreign objects in the dental area, A step of receiving the output of the machine learning model, wherein the output includes a binary mask having the same number of entries as pixels or voxels in the image, wherein entries associated with pixels or voxels that are part of the reference object have a first value, and entries associated with pixels or voxels that are not part of the reference object have a second value, The method described in Appendix 1, including the method described in Appendix 1. [Note 9] Before receiving the aforementioned intraoral scan data, Steps to enter training mode, The steps include receiving multiple images of the reference object during the training mode, The steps include generating a virtual model of the reference object based on the plurality of images of the reference object, The steps of adding at least one of the virtual model of the reference object or the plurality of images of the reference object to a reference object library that includes entries for a plurality of reference objects, The method described in Appendix 1, further comprising carrying out the following. [Note 10] A step of receiving an indication that the reference object is located in the dental area, wherein the indication includes identification information of the reference object, The steps include: using the identification information of the reference object, querying a reference object library that includes entries for multiple reference objects; A step of receiving a response to the query, wherein the response includes data associated with the reference object, Using the aforementioned data, the steps include identifying the reference object in the image, The method described in Appendix 1, further including the above. [Note 11] The method according to Appendix 1, wherein the one or more known properties include at least one of the known reflectance of the reference object with respect to the wavelength of light, the known refractive index of the reference object with respect to the wavelength of light, the known responsiveness of the reference object with respect to the wavelength of light, the known texture of the reference object, the known surface pattern of the reference object, the known color of the reference object, or the known shape of the reference object. [Note 12] The steps include determining the confidence value associated with the aforementioned reference object, The steps include determining whether the aforementioned confidence value falls below a confidence threshold, A step of presenting the option of a) adding the additional data relating to the reference object to at least one of the image or the virtual 3D model, or b) leaving the representation of the reference object in at least one of the image or the virtual 3D model unchanged, A step of receiving a user selection to add the additional data relating to the reference object to at least one of the image or the virtual 3D model, The method described in Appendix 1, further including the above. [Note 13] The method according to Appendix 1, wherein the reference object includes at least one of a shape or material that causes a decrease in the accuracy of the intraoral scan of the reference object. [Note 14] When executed by the processing unit, The steps include receiving intraoral scan data, which includes intraoral images of the dental area, The steps include: performing an analysis of the intraoral image to a) identify a foreign object represented in the intraoral image, and b) determining whether the foreign object is an instance of a reference object having one or more known characteristics; The step of modifying the intraoral image by adding additional data relating to the foreign object based on one or more known characteristics of the reference object, The steps include receiving additional intraoral scan data, which includes multiple additional intraoral images of the aforementioned dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area based on the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processing unit to perform an operation including the above. [Note 15] The step of performing the analysis of the intraoral image is: A step of performing image processing on the intraoral image to identify a plurality of possible objects in the intraoral image, wherein each of the plurality of possible objects includes one or more physical characteristics, For each of the aforementioned multiple possible objects, the step is to compare one or more physical properties of the aforementioned possible object with multiple known properties of the reference object. As a result of the above comparison, the step of determining whether one or more physical properties of one of the multiple likely objects match one or more known properties of the reference object, The steps include determining whether the aforementioned likely object is the foreign object located in the dental area, Non-temporary computer-readable media as described in Appendix 14, including the above. [Note 16] The non-temporary computer-readable medium described in Appendix 15, wherein one or more of the aforementioned physical properties include reflectance, refractive index, responsiveness to wavelengths of light, and at least one of texture, surface pattern, color, or shape. [Note 17] The aforementioned intraoral scan data and the aforementioned additional intraoral scan data are received during the intraoral scan, the additional data includes the shape of the foreign object, at least a portion of the shape is not shown in the intraoral image, and the operation further, During the intraoral scan, a step of generating a view of the dental area based on the modified intraoral image, wherein the view includes the contour of the foreign object based on the additional data relating to the foreign object. A step of replacing one or more parts of the shape of the foreign object based on the plurality of additional intraoral images, A step of updating the view of the dental area, wherein the updated view shows the replaced portion of the shape of the foreign object, Non-temporary computer-readable media as described in Appendix 14, including the above. [Note 18] The non-temporary computer-readable medium described in Appendix 17, wherein at least a portion of the foreign object represented in the virtual 3D model is based on the additional data relating to the foreign object and not on the received intraoral image. [Note 19] The above operation is performed before receiving the intraoral scan data. Steps to enter training mode, The steps include receiving multiple images of the foreign object during the training mode, The steps include generating a virtual model of the foreign object based on the aforementioned plurality of images, The steps include adding at least one of the virtual model of the foreign object or the plurality of images of the foreign object to a reference object library which includes entries for a plurality of reference objects, Non-temporary computer-readable media as described in Appendix 14, further including the implementation of the above. [Note 20] The aforementioned operation is, A step of receiving an indication that the foreign object is located in the dental area, wherein the indication includes identification information of the foreign object, The steps include: using the identification information of the foreign object, querying a reference object library containing entries for multiple reference objects; The step of receiving a response to the query, wherein the response includes one or more known properties of the foreign object, the one or more known properties include at least one of the known reflectance of the foreign object with respect to the wavelength of light, the known refractive index of the foreign object with respect to the wavelength of light, the known responsiveness of the foreign object with respect to the wavelength of light, the known texture of the foreign object, the known surface pattern of the foreign object, the known color of the foreign object, or the known shape of the foreign object, The steps include identifying the foreign object in the intraoral image using one or more known characteristics of the foreign object, Non-temporary computer-readable media as described in Appendix 14, further including the above. [Note 21] The above operation is performed before correcting the intraoral image. The steps include determining the confidence value associated with the foreign substance, The steps include determining whether the aforementioned confidence value falls below a confidence threshold, The step of presenting the options of a) adding the additional data relating to the foreign object to the intraoral image, or b) leaving the representation of the foreign object in the intraoral image unchanged, The step of receiving a user selection to add the additional data relating to the foreign object to the intraoral image, Non-temporary computer-readable media as described in Appendix 14, further including the implementation of the above. [Note 22] The aforementioned operation is, Step 1: Add the entry for the foreign object to the library of reference objects for known foreign objects. Non-temporary computer-readable media as described in Appendix 21, further including the above. [Note 23] A handheld scanner for performing intraoral scans, When executed by the processor, The steps include receiving intraoral images associated with an intraoral scan of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of responsiveness of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The steps include modifying the oral cavity image by adding a more complete representation of the foreign object to the oral cavity image, The steps include receiving multiple additional intraoral images of the dental area during the intraoral scan, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processor to perform an operation including the above, A system that includes this. [Note 24] The steps include receiving scan data, including intraoral images, during an intraoral scan of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on the analysis of the scan data, The steps include modifying the oral cavity image by removing the representation of the foreign object from the oral cavity image, The steps include receiving additional scan data during the intraoral scan, which includes a plurality of additional intraoral images of the dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A method that includes this. [Note 25] The steps include identifying additional representations of the foreign body in one or more additional intraoral images of the dental region, The steps include modifying one or more additional intraoral images by removing the additional representation of the foreign object from one or more additional intraoral images, It further includes, The foreign object is a stationary object located at the same position in the dental region in the intraoral image and the one or more additional intraoral images. The method described in Appendix 24. [Note 26] The steps include analyzing the scan data to identify at least one of the following: the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object. A step of identifying the representation of the foreign object based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The method described in Appendix 24, further including the method described in Appendix 24. [Note 27] A step of performing image processing on the intraoral image to identify a plurality of shapes in the intraoral image, wherein the plurality of shapes include the shape of the foreign object, The steps include comparing the plurality of shapes in the intraoral image with a plurality of known shapes of a reference object, As a result of the above comparison, the step of determining whether the shape of the foreign object matches the known shape of the reference object, The method described in Appendix 24, further including the method described in Appendix 24. [Note 28] Prior to the intraoral scan of the aforementioned dental area, Steps to enter training mode, The steps include receiving multiple images of the foreign object during the training mode, The steps include generating a virtual model of the foreign object based on the aforementioned plurality of images, The steps include adding at least one of the virtual model of the foreign object or the plurality of images of the foreign object to a reference object library that includes the plurality of known shapes of the reference object, The method described in Appendix 27, further including the implementation of the following. [Note 29] The steps include performing image processing on the intraoral image to identify at least one of a plurality of textures or a plurality of surface patterns in the intraoral image, A step of identifying at least one of the above-mentioned multiple textures that does not naturally exist in the dental area, or a surface pattern that does not naturally exist in the dental area, A step of determining the contour of a region in the intraoral image that has at least one of the texture or surface pattern that does not naturally exist in the dental area, wherein the region within the contour in the intraoral image is removed from the intraoral image. The method described in Appendix 24, further including the method described in Appendix 24. [Note 30] The step of identifying at least one of the texture or surface pattern that does not naturally exist in the dental area is, a) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures of a reference object, and determining whether the texture matches a known texture of the reference object as a result of the comparison. b) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns of a reference object, and determining whether the surface patterns match the known surface patterns of the reference object as a result of the comparison. c) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures naturally present in the dental area, and determining whether the texture does not match any known texture naturally present in the oral cavity as a result of the comparison, or d) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns naturally present in the dental area, and determining, as a result of the comparison, whether the surface pattern does not match any known surface pattern naturally present in the oral cavity. The method described in Appendix 29, comprising at least one of the steps. [Note 31] The steps include performing image processing on the intraoral image to identify multiple colors in the intraoral image, A step of identifying at least one of the aforementioned multiple colors that does not naturally exist in the dental area, A step of determining the contour of a region having at least one color that does not naturally exist in the dental area, wherein the region within the contour of the intraoral image is removed from the intraoral image. The method described in Appendix 24, further including the method described in Appendix 24. [Note 32] The step of identifying the at least one color that does not naturally exist in the dental area is performed for each pixel or voxel of the intraoral image, For each color channel of the color encoding system, the step is to examine at least one of the saturation value or intensity value, A step of determining a tuple that includes at least one of the saturation value or intensity value of each color channel of the color encoding system, The steps include determining whether the tuple falls outside the specified color range, This includes implementing, The step of determining the contour of the region having at least one color that does not naturally exist in the intraoral image includes the step of identifying a plurality of consecutive pixels or voxels having a tuple that falls outside the designated color range. The method described in Appendix 31. [Note 33] A step of receiving an indication that the foreign object is located in the dental area, wherein the indication includes identification information of the foreign object, The steps include: executing a query against a reference object library using the identification information of the foreign object; The step of receiving a response to the query, wherein the response includes one or more known properties of the foreign object, the one or more known properties include at least one of the known reflectance of the foreign object with respect to the wavelength of light, the known refractive index of the foreign object with respect to the wavelength of light, the known responsiveness of the foreign object with respect to the wavelength of light, the known texture of the foreign object, the known surface pattern of the foreign object, the known color of the foreign object, or the known shape of the foreign object, The steps include identifying the representation of the foreign object in the intraoral image using one or more known characteristics of the foreign object, The method described in Appendix 24, further including the method described in Appendix 24. [Note 34] During the intraoral scan, the step of generating a view of the dental area based on the modified intraoral image and the plurality of additional intraoral images, The step of generating the outline of the foreign object, The steps include adding the contour of the foreign object to the view, A step of providing a user interface that enables a user to select the contour of the foreign object, wherein the user's selection of the contour of the foreign object causes the representation of the foreign object to be returned to the intraoral image, and further causes the foreign object to be added to a reference object library of known objects that should not be removed by the filter, The method described in Appendix 24, further including the method described in Appendix 24. [Note 35] Before correcting the aforementioned intraoral image, A step of determining the confidence value associated with the representation of the foreign substance, The steps include determining whether the aforementioned confidence value falls below a confidence threshold, The step of presenting the options of a) removing the representation of the foreign object from the intraoral image, or b) leaving the representation of the foreign object in the intraoral image, The steps include receiving a user selection to remove the representation of the foreign object from the intraoral image, The method described in Appendix 24, further including the implementation of the following. [Note 36] A step of identifying one or more properties of the foreign matter, wherein the one or more properties include at least one of the reflectance of the foreign matter with respect to the wavelength of light, the refractive index of the foreign matter with respect to the wavelength of light, the responsiveness of the foreign matter with respect to the wavelength of light, the texture of the foreign matter, the surface pattern of the foreign matter, the color of the foreign matter, or the shape of the foreign matter, The steps include adding an entry for the foreign object to a reference object library of known foreign objects to be removed by the filter, The method described in Appendix 35, further including the method described in Appendix 35. [Note 37] The step of identifying the representation of the foreign substance is: The steps include processing the intraoral images using a machine learning model trained to identify foreign objects in the dental area, A step of receiving the output of the machine learning model, wherein the output includes a binary mask having the same number of entries as pixels or voxels in the intraoral image, wherein entries associated with pixels or voxels that are part of the foreign body have a first value, and entries associated with pixels or voxels that are not part of the foreign body have a second value, Includes, The intraoral image is modified using the binary mask. The method described in Appendix 35. [Note 38] When executed by the processing unit, The steps include receiving scan data, including intraoral images, during an intraoral scan of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on the analysis of the scan data, The steps include modifying the oral cavity image by removing the representation of the foreign object from the oral cavity image, The steps include receiving additional scan data during the intraoral scan, which includes a plurality of additional intraoral images of the dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processing unit to perform an operation including the above. [Note 39] The aforementioned operation is, The steps include identifying additional representations of the foreign body in one or more additional intraoral images of the dental region, The steps include modifying one or more additional intraoral images by removing the additional representation of the foreign object from one or more additional intraoral images, It further includes, The foreign object is a stationary object located at the same position in the dental region in the intraoral image and the one or more additional intraoral images. The method described in Appendix 38. [Note 40] The aforementioned operation is, The steps include analyzing the scan data to identify at least one of the following: the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object. A step of identifying the representation of the foreign object based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, Non-temporary computer-readable media as described in Appendix 38, further including the above. [Note 41] The aforementioned operation is, A step of performing image processing on the intraoral image to identify a plurality of shapes in the intraoral image, wherein the plurality of shapes include the shape of the foreign object, The steps include comparing the plurality of shapes in the intraoral image with a plurality of known shapes of a reference object, As a result of the above comparison, the step of determining whether the shape of the foreign object matches the known shape of the reference object, Non-temporary computer-readable media as described in Appendix 38, further including the above. [Note 42] The aforementioned operation is, Steps to enter training mode, The steps include receiving multiple images of the foreign object during the training mode, The steps include generating a virtual model of the foreign object based on the aforementioned plurality of images, The steps include adding at least one of the virtual model of the foreign object or the plurality of images of the foreign object to a reference object library that includes the plurality of known shapes of the reference object, Non-temporary computer-readable media as described in Appendix 41, further including the above. [Note 43] The aforementioned operation is, The steps include performing image processing on the intraoral image to identify at least one of a plurality of textures or a plurality of surface patterns in the intraoral image, A step of identifying at least one of the above-mentioned multiple textures that does not naturally exist in the dental area, or a surface pattern that does not naturally exist in the dental area, A step of determining the contour of a region in the intraoral image that has at least one of the texture or surface pattern that does not naturally exist in the dental area, wherein the region within the contour in the intraoral image is removed from the intraoral image. Non-temporary computer-readable media as described in Appendix 38, further including the above. [Note 44] The step of identifying at least one of the texture or surface pattern that does not naturally exist in the dental area is, a) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures of a reference object, and determining whether the texture matches a known texture of the reference object as a result of the comparison. b) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns of a reference object, and determining whether the surface patterns match the known surface patterns of the reference object as a result of the comparison. c) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures naturally present in the dental area, and determining whether the texture does not match any known texture naturally present in the oral cavity as a result of the comparison, or d) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns naturally present in the dental area, and determining, as a result of the comparison, whether the surface pattern does not match any known surface pattern naturally present in the oral cavity. A non-temporary computer-readable medium as described in Appendix 43, comprising at least one of the steps of the above. [Note 45] The aforementioned operation is, The steps include performing image processing on the intraoral image to identify multiple colors in the intraoral image, A step of identifying at least one of the aforementioned multiple colors that does not naturally exist in the dental area, A step of determining the contour of a region having at least one color that does not naturally exist in the dental area, wherein the region within the contour of the intraoral image is removed from the intraoral image. Non-temporary computer-readable media as described in Appendix 38, further including the above. [Note 46] A handheld scanner for performing intraoral scans, When executed by the processor, The steps include receiving intraoral images during intraoral scanning of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of responsiveness of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The steps include modifying the oral cavity image by removing the representation of the foreign object from the oral cavity image, The steps include receiving multiple additional intraoral images of the dental area during the intraoral scan, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processor to perform an operation including the above, A system that includes this.

Claims

1. A step of receiving a plurality of first two-dimensional (2D) intraoral images of a dental area from an intraoral scanner during intraoral scanning of the dental area, wherein the plurality of first 2D intraoral images include at least one color 2D intraoral image, The steps include performing color analysis on at least one of the aforementioned color 2D intraoral images, A step of identifying the representation of a foreign object in the at least one color 2D intraoral image based at least partially on the color analysis, wherein a pixel of the at least one color 2D intraoral image having a color value within a color value range is identified as part of the foreign object; The steps include modifying one or more 2D intraoral images by removing the representation of the foreign object from one or more of the first plurality of 2D intraoral images, The steps include receiving a second plurality of 2D intraoral images of the dental area from the intraoral scanner during the intraoral scan, The steps of converting the modified one or more 2D intraoral images from which the representation of the foreign object has been removed, and one or more additional 2D intraoral images from the second plurality of 2D intraoral images, into three-dimensional (3D) data, A step of generating a 3D surface of the dental area based at least partially on the 3D data, A method that includes this.

2. The step of performing image processing on the at least one color 2D intraoral image to determine a plurality of features of the at least one color 2D intraoral image is as follows: The method according to claim 1, further comprising:

3. The method according to claim 2, wherein the image processing includes at least one of an edge detection algorithm, an object detection algorithm, or a feature detection algorithm.

4. Prior to the intraoral scan of the aforementioned dental area, Steps to enter training mode, The steps include receiving one or more images of the foreign object during the training mode, A step of determining the color value range using one or more images of the foreign object, The method according to claim 1, further comprising carrying out the following.

5. The second plurality of 2D intraoral images include at least one additional color 2D intraoral image, and the method is The steps include performing the color analysis of at least one additional color 2D intraoral image, The steps include identifying additional representations of the foreign body in the at least one additional color 2D intraoral image based on the color analysis of the at least one additional color 2D intraoral image, The method according to claim 1, further comprising the step of modifying the one or more additional 2D intraoral images by removing the additional representation of the foreign body from the one or more additional 2D intraoral images before converting the one or more additional 2D intraoral images into 3D data.

6. The method according to claim 5, wherein the foreign object is a stationary object located at the same position in the dental area in the one or more 2D intraoral images and the one or more additional 2D intraoral images.

7. The method according to claim 1, comprising the step of performing at least one additional analysis of the at least one color 2D intraoral image, wherein the at least one additional analysis includes at least one of the following: analysis of reflectance associated with wavelength of light, analysis of refractive index associated with wavelength of light, analysis of reactivity associated with wavelength of light, analysis of texture, or analysis of shape.

8. The method according to claim 7, wherein the representation of the foreign body in the at least one color 2D intraoral image is determined based on the color analysis and the at least one additional analysis.

9. The step of receiving a third plurality of two-dimensional (2D) intraoral images of the dental area from the intraoral scanner during the intraoral scan, wherein the third plurality of 2D intraoral images include at least one further color 2D intraoral image. The steps include performing color analysis of at least one further color 2D intraoral image, A step of identifying a representation of a second foreign body in the at least one further color 2D intraoral image, based at least partially on the color analysis of the at least one further color 2D intraoral image, wherein a pixel of the at least one further color 2D intraoral image having a color value within a second color value range is identified as part of the second foreign body. The steps include modifying one or more of the third plurality of 2D intraoral images by removing the representation of the second foreign body from one or more of the third plurality of 2D intraoral images, A step of updating the 3D surface of the dental area based at least partially on one or more of the modified 2D intraoral images from the third plurality of 2D intraoral images, The method according to claim 1, further comprising:

10. A step of determining the contour of the representation of the foreign object, wherein the region of one or more 2D intraoral images within the contour is removed from the one or more 2D intraoral images, The method according to claim 1, further comprising:

11. The step of outputting the 3D plane to a display during the intraoral scan, wherein the foreign object is not included in the 3D plane, The method according to claim 1, further comprising:

12. Before removing the foreign matter, A step of determining the confidence value associated with the representation of the foreign substance, The steps include determining whether the confidence value is below the confidence threshold, The method according to claim 1, further comprising:

13. Before removing the foreign matter, The step of presenting the option of a) removing the representation of the foreign object from one or more 2D intraoral images, or b) leaving the representation of the foreign object in one or more 2D intraoral images, A step of receiving a user selection for removing the representation of the foreign object from one or more 2D intraoral images, The method according to claim 12, further comprising:

14. A system comprising an intraoral scanner and a computing device operably coupled to the intraoral scanner, wherein the computing device is configured to perform the method according to any one of claims 1 to 13.

15. A computer-readable medium that, when executed by a processing device, includes an instruction causing the processing device to perform the method described in any one of claims 1 to 13.

16. The process involves receiving a first set of multiple intraoral images of the dental area from a handheld scanner during intraoral scanning of the dental area, The steps include processing at least one of the first plurality of intraoral images using a machine learning model trained to identify foreign objects in the dental area, A step of receiving the output of the trained machine learning model, wherein the output includes showing a plurality of pixels from the at least one intraoral image associated with a foreign object in the dental site, The steps include modifying one or more of the first multiple intraoral images by removing data associated with the plurality of pixels from one or more intraoral images based on the output of the trained machine learning model, The steps include receiving a second set of intraoral images of the dental area from the handheld scanner during the intraoral scan, 1) A step of generating a three-dimensional (3D) plane of the dental area based at least partially on one or more modified intraoral images from the first plurality of intraoral images, and 2) the second plurality of intraoral images. A method that includes this.

17. The method according to claim 16, wherein the output includes a binary mask having the same number of entries as pixels or voxels of the at least one intraoral image, wherein entries associated with pixels or voxels that are part of the foreign body have a first value, and entries associated with pixels or voxels that are not part of the foreign body have a second value, and the one or more intraoral images are modified using the binary mask.

18. The method according to claim 16, wherein the at least one intraoral image includes a color intraoral image, and the trained machine learning model is trained to identify foreign objects in dental sites at least partially based on color information.

19. A system comprising a handheld scanner and a computing device operably coupled to the handheld scanner, wherein the computing device is configured to perform the method according to any one of claims 14 to 18.

20. A handheld scanner for performing intraoral scans, When executed by the processor, The process involves receiving one or more two-dimensional (2D) intraoral images during an intraoral scan of the dental area, The steps include identifying the representation of a foreign object in one or more 2D intraoral images based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of responsiveness of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The step of modifying one or more 2D intraoral images by removing the representation of the foreign object from one or more 2D intraoral images, The steps include receiving a plurality of additional 2D intraoral images of the dental area during the intraoral scan, The steps of converting the modified one or more 2D intraoral images from which the representation of the foreign object has been removed, and the plurality of additional 2D intraoral images, into three-dimensional (3D) data, The steps include generating a virtual 3D model of the dental area using the aforementioned 3D data, A non-temporary computer-readable medium containing instructions that cause the processor to perform an operation including the above, A system that includes this.

Images