Foreign body identification, image enhancement, and / or filtering for intraoral scanning

Abstract

This invention relates to a system and method for performing real-time filtering of intraoral images generated during intraoral scanning. [Solution] During an intraoral scan, scan data containing multiple images of the dental area is received. Image analysis is performed. Based on this analysis, representations of foreign objects potentially possessing known characteristics are identified in the images. A virtual 3D model of the dental area is generated based on the above multiple images. Modifying the images and / or virtual 3D model of the dental area is done by adding additional data about the foreign object to the intraoral images and / or virtual 3D model, or by removing data about the foreign object from the intraoral images and / or virtual 3D model. If the images are modified, the images may be modified before the virtual 3D model is generated, and the virtual 3D model may be generated using the modified images.

Claims

1. The steps include receiving intraoral scan data containing multiple images of the dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area based on the aforementioned multiple images, The steps include performing an analysis on the image of the dental area, A step of identifying a representation of a reference object in the aforementioned image, wherein the reference object has one or more known characteristics, The steps of modifying at least one of the image or the virtual 3D model by adding additional data relating to the reference object to at least one of the image or the virtual 3D model of the dental area, based on one or more known characteristics of the reference object, A method that includes this.

2. The method according to claim 1, wherein the image is one of a plurality of images of the dental area used to generate the virtual 3D model, the image is modified by adding the additional data relating to the reference object to the image, and the modified image is used to generate the virtual 3D model of the dental area.

3. A step of performing image processing on the image to identify a likely object represented in the image, wherein the likely object includes one or more physical characteristics. The steps include comparing one or more physical properties of the likely object with one or more known properties of the reference object, As a result of the comparison, the step of determining whether one or more physical properties of the likely object match one or more known properties of the reference object, The steps include determining whether the likely object is the reference object, The method according to claim 2, further comprising:

4. The method according to claim 3, wherein the one or more physical properties include at least one of reflectance, refractive index, responsiveness to the wavelength of light, and texture, surface pattern, color, or shape.

5. The method according to claim 1, wherein the step of performing the analysis of the image includes inputting the image to a machine learning model trained to identify one or more types of reference objects, the machine learning model outputs a representation of the reference objects in the image.

6. The step of generating the image on which the analysis is performed by projecting the virtual 3D model onto a plane. The method according to claim 1, further comprising:

7. During an intraoral scan, a step of generating a view of the dental area based on the modified image, wherein the view includes the contour of the reference object based on the additional data relating to the reference object; The steps include receiving multiple additional images of the dental area during the intraoral scan, The steps include replacing one or more parts of the shape of the reference object based on the plurality of additional images, A step of updating the view of the dental area, wherein the updated view shows the replaced portion of the shape of the reference object, The method according to claim 6, further comprising:

8. The step of identifying the representation of the reference object is: The steps include processing the image using a machine learning model trained to identify one or more foreign objects in the dental area, A step of receiving the output of the machine learning model, wherein the output includes a binary mask having the same number of entries as pixels or voxels in the image, wherein entries associated with pixels or voxels that are part of the reference object have a first value, and entries associated with pixels or voxels that are not part of the reference object have a second value, The method according to claim 1, including the method described in claim 1.

9. Before receiving the aforementioned intraoral scan data, Steps to enter training mode, The steps include receiving multiple images of the reference object during the training mode, The steps include generating a virtual model of the reference object based on the plurality of images of the reference object, The steps of adding at least one of the virtual model of the reference object or the plurality of images of the reference object to a reference object library that includes entries for a plurality of reference objects, The method according to claim 1, further comprising carrying out the following.

10. A step of receiving an indication that the reference object is located in the dental area, wherein the indication includes identification information of the reference object, The steps include: using the identification information of the reference object, querying a reference object library that includes entries for multiple reference objects; A step of receiving a response to the query, wherein the response includes data associated with the reference object, Using the aforementioned data, the steps include identifying the reference object in the image, The method according to claim 1, further comprising:

11. The method according to claim 1, wherein the one or more known properties include at least one of the known reflectance of the reference object with respect to the wavelength of light, the known refractive index of the reference object with respect to the wavelength of light, the known responsiveness of the reference object with respect to the wavelength of light, the known texture of the reference object, the known surface pattern of the reference object, the known color of the reference object, or the known shape of the reference object.

12. The steps include determining the confidence value associated with the aforementioned reference object, The steps include determining whether the aforementioned confidence value falls below a confidence threshold, A step of presenting the option of a) adding the additional data relating to the reference object to at least one of the image or the virtual 3D model, or b) leaving the representation of the reference object in at least one of the image or the virtual 3D model unchanged, The steps include receiving a user selection to add the additional data relating to the reference object to at least one of the image or the virtual 3D model, The method according to claim 1, further comprising:

13. The method according to claim 1, wherein the reference object includes at least one of a shape or material that causes a decrease in the accuracy of the intraoral scan of the reference object.

14. When executed by the processing unit, The steps include receiving intraoral scan data, which includes intraoral images of the dental area, The steps include: a) analyzing the intraoral image to identify the foreign object represented in the intraoral image, and b) determining whether the foreign object is an instance of a reference object having one or more known characteristics; The step of modifying the intraoral image by adding additional data relating to the foreign object based on one or more known characteristics of the reference object, The steps include receiving additional intraoral scan data, which includes multiple additional intraoral images of the aforementioned dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area based on the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processing unit to perform an operation including the above.

15. The step of performing the analysis of the intraoral image is: A step of performing image processing on the intraoral image to identify a plurality of possible objects in the intraoral image, wherein each of the plurality of possible objects includes one or more physical characteristics, For each of the aforementioned multiple possible objects, the step is to compare one or more physical properties of the aforementioned possible object with multiple known properties of the reference object. As a result of the above comparison, the step of determining whether one or more physical properties of one of the multiple likely objects match one or more known properties of the reference object, The steps include determining whether the aforementioned likely object is the foreign object located in the dental area, A non-temporary computer-readable medium according to claim 14, including the following:

16. The non-transient computer-readable medium according to claim 15, wherein the one or more physical properties include reflectance, refractive index, responsiveness to the wavelength of light, and at least one of texture, surface pattern, color, or shape.

17. The aforementioned intraoral scan data and the aforementioned additional intraoral scan data are received during the intraoral scan, the additional data includes the shape of the foreign object, at least a portion of the shape is not shown in the intraoral image, and the operation further, During the intraoral scan, a step of generating a view of the dental area based on the modified intraoral image, wherein the view includes the contour of the foreign object based on the additional data relating to the foreign object. The steps include replacing one or more parts of the shape of the foreign object based on the plurality of additional intraoral images, A step of updating the view of the dental area, wherein the updated view shows the replaced portion of the shape of the foreign object, A non-temporary computer-readable medium according to claim 14, including the following:

18. The non-temporary computer-readable medium according to claim 17, wherein at least a portion of the foreign object represented in the virtual 3D model is based on the additional data relating to the foreign object and not on the received intraoral image.

19. The above operation is performed before receiving the intraoral scan data. Steps to enter training mode, The steps include receiving multiple images of the foreign object during the training mode, The steps include generating a virtual model of the foreign object based on the aforementioned plurality of images, The steps include adding at least one of the virtual model of the foreign object or the plurality of images of the foreign object to a reference object library which includes entries for a plurality of reference objects, The non-temporary computer-readable medium according to claim 14, further comprising performing the following:

20. The aforementioned operation is, A step of receiving an indication that the foreign object is located in the dental area, wherein the indication includes identification information of the foreign object, The steps include: using the identification information of the foreign object, querying a reference object library containing entries for multiple reference objects; The step of receiving a response to the query, wherein the response includes one or more known properties of the foreign object, the one or more known properties include at least one of the known reflectance of the foreign object with respect to the wavelength of light, the known refractive index of the foreign object with respect to the wavelength of light, the known responsiveness of the foreign object with respect to the wavelength of light, the known texture of the foreign object, the known surface pattern of the foreign object, the known color of the foreign object, or the known shape of the foreign object, The steps include identifying the foreign object in the intraoral image using one or more known characteristics of the foreign object, A non-temporary computer-readable medium according to claim 14, further comprising:

21. The above operation is performed before correcting the intraoral image. The steps include determining the confidence value associated with the foreign substance, The steps include determining whether the aforementioned confidence value falls below a confidence threshold, The step of presenting the following options: a) add the additional data relating to the foreign object to the intraoral image, or b) leave the representation of the foreign object in the intraoral image unchanged. The step of receiving a user selection to add the additional data relating to the foreign object to the intraoral image, The non-temporary computer-readable medium according to claim 14, further comprising performing the following:

22. The aforementioned operation is, Step 1: Add the entry for the foreign object to the library of reference objects for known foreign objects. A non-temporary computer-readable medium according to claim 21, further comprising:

23. A handheld scanner for performing intraoral scans, When executed by the processor, The steps include receiving intraoral images associated with an intraoral scan of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of responsiveness of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The steps include modifying the intraoral image by adding a more complete representation of the foreign object to the intraoral image, The steps include receiving multiple additional intraoral images of the dental area during the intraoral scan, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processor to perform an operation including the above, A system that includes this.

24. The steps include receiving scan data, including intraoral images, during an intraoral scan of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on the analysis of the scan data, The steps include modifying the oral cavity image by removing the representation of the foreign object from the oral cavity image, The steps include receiving additional scan data during the intraoral scan, which includes a plurality of additional intraoral images of the dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A method that includes this.

25. The steps include identifying additional representations of the foreign body in one or more additional intraoral images of the dental region, The steps include modifying one or more additional intraoral images by removing the additional representation of the foreign object from one or more additional intraoral images, It further includes, The foreign object is a stationary object located at the same position in the dental region in the intraoral image and the one or more additional intraoral images. The method according to claim 24.

26. The steps include analyzing the scan data to identify at least one of the following: the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object. A step of identifying the representation of the foreign object based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The method according to claim 24, further comprising:

27. A step of performing image processing on the intraoral image to identify a plurality of shapes in the intraoral image, wherein the plurality of shapes include the shape of the foreign object, The steps include comparing the plurality of shapes in the intraoral image with a plurality of known shapes of a reference object, As a result of the above comparison, the step of determining whether the shape of the foreign object matches the known shape of the reference object, The method according to claim 24, further comprising:

28. Prior to the intraoral scan of the aforementioned dental area, Steps to enter training mode, The steps include receiving multiple images of the foreign object during the training mode, The steps include generating a virtual model of the foreign object based on the aforementioned plurality of images, The steps include adding at least one of the virtual model of the foreign object or the plurality of images of the foreign object to a reference object library that includes the plurality of known shapes of the reference object, The method according to claim 27, further comprising carrying out the following.

29. The steps include performing image processing on the intraoral image to identify at least one of a plurality of textures or a plurality of surface patterns in the intraoral image, A step of identifying at least one of the plurality of textures that does not naturally exist in the dental area, or a surface pattern that does not naturally exist in the dental area, A step of determining the contour of a region in the intraoral image that has at least one of the texture or surface pattern which is not naturally present in the dental area, wherein the region within the contour in the intraoral image is removed from the intraoral image. The method according to claim 24, further comprising:

30. The step of identifying at least one of the texture or surface pattern that does not naturally exist in the dental area is, a) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures of a reference object, and determining whether the texture matches a known texture of the reference object as a result of the comparison. b) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns of a reference object, and determining whether the surface patterns match the known surface patterns of the reference object as a result of the comparison. c) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures naturally present in the dental area, and determining, as a result of the comparison, whether the texture does not match any known texture naturally present in the oral cavity, or d) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns naturally present in the dental area, and determining, as a result of the comparison, whether the surface pattern does not match any known surface pattern naturally present in the oral cavity. The method according to claim 29, comprising at least one of the steps.

31. The steps include performing image processing on the intraoral image to identify multiple colors in the intraoral image, A step of identifying at least one of the aforementioned multiple colors that does not naturally exist in the dental area, A step of determining the contour of a region having at least one color that does not naturally exist in the dental area, wherein the region within the contour of the intraoral image is removed from the intraoral image. The method according to claim 24, further comprising:

32. The step of identifying the at least one color that does not naturally exist in the dental area is performed for each pixel or voxel of the intraoral image, For each color channel of the color encoding system, the step is to examine at least one of the saturation value or intensity value, A step of determining a tuple that includes at least one of the saturation value or intensity value of each color channel of the color encoding system, The steps include determining whether the tuple falls outside the specified color range, This includes implementing, The step of determining the contour of the region having at least one color that does not naturally exist in the intraoral image includes the step of identifying a plurality of consecutive pixels or voxels having a tuple that falls outside the designated color range. The method according to claim 31.

33. A step of receiving an indication that the foreign object is located in the dental area, wherein the indication includes identification information of the foreign object, The steps include: executing a query against a reference object library using the identification information of the foreign object; The step of receiving a response to the query, wherein the response includes one or more known properties of the foreign object, the one or more known properties include at least one of the known reflectance of the foreign object with respect to the wavelength of light, the known refractive index of the foreign object with respect to the wavelength of light, the known responsiveness of the foreign object with respect to the wavelength of light, the known texture of the foreign object, the known surface pattern of the foreign object, the known color of the foreign object, or the known shape of the foreign object, The steps include identifying the representation of the foreign object in the intraoral image using one or more known characteristics of the foreign object, The method according to claim 24, further comprising:

34. During the intraoral scan, a step of generating a view of the dental area based on the modified intraoral image and the plurality of additional intraoral images, The step of generating the outline of the foreign object, The steps include adding the contour of the foreign object to the view, A step of providing a user interface that enables a user to select the contour of the foreign object, wherein the user's selection of the contour of the foreign object causes the representation of the foreign object to be returned to the intraoral image, and further causes the foreign object to be added to a reference object library of known objects that should not be removed by the filter, The method according to claim 24, further comprising:

35. Before correcting the aforementioned intraoral image, A step of determining the confidence value associated with the representation of the foreign substance, The steps include determining whether the aforementioned confidence value falls below a confidence threshold, The step of presenting the following options: a) remove the representation of the foreign object from the intraoral image, or b) leave the representation of the foreign object in the intraoral image. The steps include receiving a user selection to remove the representation of the foreign object from the intraoral image, The method according to claim 24, further comprising carrying out the following.

36. A step of identifying one or more properties of the foreign matter, wherein the one or more properties include at least one of the reflectance of the foreign matter with respect to the wavelength of light, the refractive index of the foreign matter with respect to the wavelength of light, the responsiveness of the foreign matter with respect to the wavelength of light, the texture of the foreign matter, the surface pattern of the foreign matter, the color of the foreign matter, or the shape of the foreign matter. The steps include adding an entry for the foreign object to a reference object library of known foreign objects to be removed by the filter, The method according to claim 35, further comprising:

37. The step of identifying the representation of the foreign substance is: The steps include processing the intraoral images using a machine learning model trained to identify foreign objects in the dental area, A step of receiving the output of the machine learning model, wherein the output includes a binary mask having the same number of entries as pixels or voxels in the intraoral image, wherein entries associated with pixels or voxels that are part of the foreign body have a first value, and entries associated with pixels or voxels that are not part of the foreign body have a second value, Includes, The intraoral image is modified using the binary mask. The method according to claim 35.

38. When executed by the processing unit, The steps include receiving scan data, including intraoral images, during an intraoral scan of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on the analysis of the scan data, The steps include modifying the oral cavity image by removing the representation of the foreign object from the oral cavity image, The steps include receiving additional scan data during the intraoral scan, which includes a plurality of additional intraoral images of the dental area, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processing unit to perform an operation including the above.

39. The aforementioned operation is, The steps include identifying additional representations of the foreign body in one or more additional intraoral images of the dental region, The steps include modifying one or more additional intraoral images by removing the additional representation of the foreign object from one or more additional intraoral images, It further includes, The foreign object is a stationary object located at the same position in the dental region in the intraoral image and the one or more additional intraoral images. The method according to claim 38.

40. The aforementioned operation is, The steps include analyzing the scan data to identify at least one of the following: the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object. A step of identifying the representation of the foreign object based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the reactivity of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, A non-temporary computer-readable medium according to claim 38, further comprising:

41. The aforementioned operation is, A step of performing image processing on the intraoral image to identify a plurality of shapes in the intraoral image, wherein the plurality of shapes include the shape of the foreign object, The steps include comparing the plurality of shapes in the intraoral image with a plurality of known shapes of a reference object, As a result of the above comparison, the step of determining whether the shape of the foreign object matches the known shape of the reference object, A non-temporary computer-readable medium according to claim 38, further comprising:

42. The aforementioned operation is, Steps to enter training mode, The steps include receiving multiple images of the foreign object during the training mode, The steps include generating a virtual model of the foreign object based on the aforementioned plurality of images, The steps include adding at least one of the virtual model of the foreign object or the plurality of images of the foreign object to a reference object library that includes the plurality of known shapes of the reference object, A non-temporary computer-readable medium according to claim 41, further comprising:

43. The aforementioned operation is, The steps include performing image processing on the intraoral image to identify at least one of a plurality of textures or a plurality of surface patterns in the intraoral image, A step of identifying at least one of the plurality of textures that does not naturally exist in the dental area, or a surface pattern that does not naturally exist in the dental area, A step of determining the contour of a region in the intraoral image that has at least one of the texture or surface pattern which is not naturally present in the dental area, wherein the region within the contour in the intraoral image is removed from the intraoral image. A non-temporary computer-readable medium according to claim 38, further comprising:

44. The step of identifying at least one of the texture or surface pattern that does not naturally exist in the dental area is, a) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures of a reference object, and determining whether the texture matches a known texture of the reference object as a result of the comparison. b) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns of a reference object, and determining whether the surface patterns match the known surface patterns of the reference object as a result of the comparison. c) A step of comparing the plurality of textures identified in the intraoral image with a plurality of known textures naturally present in the dental area, and determining, as a result of the comparison, whether the texture does not match any known texture naturally present in the oral cavity, or d) A step of comparing the plurality of surface patterns identified in the intraoral image with a plurality of known surface patterns naturally present in the dental area, and determining, as a result of the comparison, whether the surface pattern does not match any known surface pattern naturally present in the oral cavity. A non-temporary computer-readable medium according to claim 43, comprising at least one of the steps of:

45. The aforementioned operation is, The steps include performing image processing on the intraoral image to identify multiple colors in the intraoral image, A step of identifying at least one of the aforementioned multiple colors that does not naturally exist in the dental area, A step of determining the contour of a region having at least one color that does not naturally exist in the dental area, wherein the region within the contour of the intraoral image is removed from the intraoral image. A non-temporary computer-readable medium according to claim 38, further comprising:

46. A handheld scanner for performing intraoral scans, When executed by the processor, The steps include receiving intraoral images during intraoral scanning of the dental area, A step of identifying the representation of a foreign object in the intraoral image based on at least one of the reflectance of the foreign object with respect to the wavelength of light, the refractive index of the foreign object with respect to the wavelength of light, the degree of responsiveness of the foreign object with respect to the wavelength of light, the texture of the foreign object, the surface pattern of the foreign object, the color of the foreign object, or the shape of the foreign object, The steps include modifying the oral cavity image by removing the representation of the foreign object from the oral cavity image, The steps include receiving multiple additional intraoral images of the dental area during the intraoral scan, The steps include generating a virtual three-dimensional (3D) model of the dental area using the modified intraoral image and the plurality of additional intraoral images, A non-temporary computer-readable medium containing instructions that cause the processor to perform an operation including the above, A system that includes this.

Images