An intraoral scanning system for providing a feedback signal that includes a quality level of a three-dimensional model

Abstract

The present disclosure relates to an intraoral scanner system that is configured to display a feedback signal on a three-dimensional (3D) model of a dental object, wherein the system includes at least one projector unit configured to emit a probe light with a plurality of configurations in the form of an illumination pattern; at least one image sensor unit accommodating an array of sensor elements configured to provide two-dimensional (2D) images with three-dimensional (3D) data based on captured reflections of the probe light from a dental object, one or more processors configured to: determine a 3D model of the dental object by stitching the 2D images, update the 3D model by stitching 2D images provided by the at least one image sensor unit onto the 3D model, determine a quality level of the data in the 3D model at a position in the 3D model, and determine a feedback signal that includes the quality level of data in the 3D model and a corresponding position of the data in the 3D model, determine a feedback trigger signal when the one or more processors is not updating the 3D model. The system further includes a displaying unit configured to display in real-time the 3D model, and the displaying unit is further configured to display when the feedback trigger signal is determined, the quality level of the data at the position on the 3D model determined by the feedback signal.

Description

[0001] AN INTRAORAL SCANNING SYSTEM FOR PROVIDING A FEEDBACK SIGNAL

[0002] THAT INCLUDES A QUALITY LEVEL OF A THREE-DIMENSIONAL MODEL

[0003] FIELD

[0004] The disclosure relates to an intraoral scanning system that is configured to determine a feedback signal on a three-dimensional model of a dental object. More specifically, the disclosure relates to an improved way of determining and visualizing a quality level of parts of a three-dimensional model.

[0005] BACKGROUND

[0006] Three-dimensional (3D) representation is relevant in many fields such as medicine or civil engineering. In dentistry, a 3D digital representation of a dental object represents a clear advantage to help dentists give the best treatment to a patient. Indeed, digital dentistry gives the opportunity of expanding dentist expertise with new treatment options and provides possibilities for more cost-effective ways of taking care of their patients.

[0007] Intraoral scanner systems are devices or systems for capturing a digital impression of a patient's dentition within dentistry. In a typical scanning scenario using a handheld intraoral scanning device, a digital 3D representation is reconstructed in real-time and simultaneously rendered to a display to provide visual guidance and user feedback to the user via the display. This implies that the user needs to focus on a display to see whether he / she is progressing in their scanning as the 3D representation is constructed in real-time on the display providing visual feedback on the progress and data coverages.

[0008] During intraoral scanning the user may regularly change his / her view between the patients intraoral cavity and the display, but for some users his / hers view is mostly focused towards the patients intraoral cavity, and for these users, the intraoral scanning system generates a large of amount feedback data to be displayed which the user is not paying any attention to.

[0009] SUMMARY It is an aspect of the present disclosure to overcome the above-mentioned disadvantage of generating a larger amount of feedback data to be displayed which user is not paying any attention to.

[0010] According to the aspect, an intraoral scanning system is disclosed. The intraoral scanning system is configured to display a feedback signal on a three-dimensional (3D) model of a dental object. The system may include at least one projector unit configured to emit a probe light with a plurality of configurations in the form of an illumination pattern, and at least one image sensor unit accommodating an array of sensor elements configured to provide two-dimensional (2D) images with three-dimensional (3D) data based on captured reflections of the probe light from a dental object. The system further includes one or more processors that is configured to: determine a 3D model of the dental arch by stitching the 2D images and to update the 3D model by stitching 2D images provided by the at least one image sensor unit onto the 3D model. The one or more processors is further configured to determine a quality level of the data in the 3D model at a position in the 3D model, and determine a feedback signal that includes a quality level of data in the 3D model and a corresponding position of the data in the 3D model.

[0011] The data in the 3D model may include color data of the 3D model, structural data of the 3D model and / or the diagnostic data of the 3D model. The one or more processors is further configured to determine a feedback trigger signal when the one or more processors is not updating the 3D model. 15. The diagnostic data may include a difference between infrared data and color data captured by the image sensor unit. Color data may be red, green, blue, fluorescent red and fluorescent green. The emitted infrared light reflects partly off a surface of the dental object and mostly within internal surfaces of the dental object that are below the surface of dental object. The emitted visible light is mostly reflected off the surface of the dental object, and by determine a difference between the infrared data and the color data would result in diagnostic data that includes infrared data with reduced amount of surface reflections. Which means caries that is visible seen in infrared data would have an improved visibility, i.e. contrast in the diagnostic data. The system includes a displaying unit configured to display in real-time the 3D model, and the displaying unit is further configured to display when the feedback trigger signal is determined, the quality level of the data at the position in the 3D model determined by the feedback signal.

[0012] The feedback trigger signal is determined when the one or more processor is not updating the 3D model. This may indicate that a handheld intraoral scanning device which includes at least the projector unit and the at least one image sensor unit is not generating two- dimensional images that can be used for updating the 3D model. That could either be caused by a wrong movement of the handheld intraoral scanning device inside the oral cavity of the patient or that the complete handheld intraoral scanning device is pulled out of the oral cavity of the patient. When the user pulls out the tip of the handheld intraoral scanning device from the oral cavity may indicate that the user is looking at the displaying unit for inspecting the 3D model and the quality of the 3D model. The system is then configured to update the displayed 3D model together with the quality level of the data at the position in the 3D model determined by the feedback signal. Thereby, the quality level is not being updated in real time as this would cause unnecessary consumption of the processing power of the one or more processors.

[0013] The no updating of the feedback trigger signal includes no stitching of 2D images onto the 3D model provided by the at least one image sensor unit onto the 3D model.

[0014] A 3D model should be understood as a point cloud, a point cloud with additional attributes like point normals, point colors, a triangulated point cloud (a triangle mesh), a polygonal mesh, a volumetric representation / voxel model, parametric representations, e.g. a spline representation. The 3D model may additionally include a virtual representation of an object in three dimensions in which internal regions (structures, etc.) are arranged within the volume in three physical dimensions in proportion and relative to the other internal and surface features of the object which is being modelled. For example, a volumetric representation of a tooth may include the outer surface as well as internal structures within the tooth (beneath the tooth surface) proportionately arranged relative to the tooth, so that a section through the volumetric model would substantially correspond to a section through the tooth, showing position and size of internal structures.

[0015] A feedback signal should be understood herein as a signal provided to an output device for providing visual feedback to a user. Alternatively, a feedback signal may simply refer to the actual feedback provided to the user when the scanning session is paused. The feedback may relate to a variety of information related to scanning the dental object. Examples include whether scan data is acquired, the rate of which new scan data is acquired, whether scan patches are correctly registered, if registration of scan patches are unsuccessful, the progress of the scanning, whether the dental object is completely scanned, deviation from recommended scan paths etc.

[0016] The intraoral scanning system comprises an output device for providing feedback as described herein. As an example, the output device may be configured to receive a feedback signal, which is instant displayed without being modulated based on e.g. a rate of which scan data is acquired.

[0017] The system may include a handheld intraoral scanning device that includes the at least one projector unit, the at least one image sensor unit, a gyroscope configured to determine an orientation of the handheld intraoral scanning device and / or an accelerometer configured to determine a motion of handheld intraoral scanning device. The one or more processors may be configured to determine the feedback trigger signal by comparing the orientation and / or the motion of the handheld intraoral scanning device to a scanning orientation range and / or a scanning motion pattern of the handheld intraoral scanning device. A user may stop moving the handheld intraoral scanning device which would be indicated to the one or more processors via an output from the accelerometer. In this example, the accelerometer would provide an output that includes an acceleration that it is at a level which correspond to a handheld intraoral scanning device that does not move. The user may move around the handheld intraoral scanning device wherein the movement does not correspond to a scanning scenario. The gyroscope that provides orientation of the handheld intraoral scanning device to the one or more processor may be used for determining whether the movement of the handheld intraoral scanning device corresponds to a scanning scenario. For example, the orientation of the handheld intraoral scanning device which does not correspond to a scanning scenario may be used by the one or more processors to determine the trigger signal or to maintain the displaying of the 3D model with the quality level. In another scenario, when the user moves the handheld intraoral scanning device when orienting the device that corresponds to a scanning scenario will remove the displaying of the quality level and instead only displaying / updating the 3D model in real time.

[0018] The intraoral scanning system may be denoted as a system.

[0019] The intraoral scanning system may include a memory unit that includes the scanning orientation range of the handheld intraoral scanning device, and the feedback trigger signal may be determined when the orientation of the handheld intraoral scanning device is outside the scanning orientation range. The orientation of the handheld intraoral scanning device is outside the scanning orientation range corresponds to a non-scanning scenario, and thereby, the quality level is displayed in the 3D model. For example, if moving the handheld intraoral scanning device when orienting a tip of the scanning device directly upwards would correspond to a very unlikely position of the patient being scanned. In this specific example, the orientation of handheld intraoral scanning device corresponds to the patient is lying on his / her’ s stomach and looking down towards the floor.

[0020] The feedback trigger signal may be determined when the motion of the handheld intraoral scanning device is not similar to reference motion patterns, or, when no motion is measured, and the reference motion patterns are stored in a memory unit of the system, and the reference motion patterns correspond to motion patterns in a scanning situation of a dental object. Motion patterns that correspond to a scanning situation may be with an acceleration that is below an acceleration threshold that has been predetermined during the manufacture of the handheld intraoral scanning device. For example, if the user moves the handheld intraoral scanning device that would not allow the system for stitching the acquired two-dimensional images to the 3D model corresponds to a non-scanning situation. That kind of movement may occur when the user moves around the handheld intraoral scanning device within a room or between rooms. The handheld intraoral scanning device may comprise a user interface, wherein the feedback trigger signal is determined when the user interface receives an input from a user of the intraoral scanning system. For example, irrespective of the orientation and the movement of the handheld intraoral scanning device, a user requests the quality level to be displayed in the 3D model by pressing a button on the handheld intraoral scanning device or on an external device of the system. That button may be a virtual button that is displayed on the displaying unit. In another example, the user may want to display the quality level while scanning the patient, the user may then press the button on the handheld intraoral scanning device, and the quality level is displayed in the 3D model. When the user releases the button then the quality level is no more seen in the 3D model. During the scanning of the patient and while the user is pressing the button, the displayed quality level is being updated with a second frame rate that is lower than a first frame rate for updating the 3D model by stitching 2D images onto the 3D model.

[0021] The first frame rate may be between 20 and 40 2D images per second, wherein all pixels of the 2D images are in focus. A 2D images in focus may be a combination of in focus pixels of a set of 2D images. The second frame rate may be between 3 and 5 times lower than the first frame rate, which means the second frame rate may be between 4 and 8 2D images per second. The second frame rate would be at a level which is not perceived as being real time.

[0022] The feedback signal may include the quality level of three-dimensional geometrical data, color data, and / or diagnostic data of the dental object. The quality level is displayed in the 3D model as a colored overlayer that is added to the 3D model. The brightness level and / or the transparency level of the colored overlayer corresponds to the quality level of the data in the 3D model. F.x. a high brightness or a high transparency may correspond to a low quality. The quality level may be displayed with one or more colors, or, with one or more levels of transparencies in the 3D model. The quality level may be displayed as a pattern or a rough surface of the 3D model. The projector unit may include a plurality of light sources, wherein the plurality of light sources includes a first light source configured to emit visible light, a second light source configured to emit infrared light, and / or a third light source configured to emit UV light, and wherein the visible light includes the probe light. The real time mode is determined by a first frame rate, and wherein the at least one image sensor unit is configured to capture fluorescent signals from the dental object based on the emitted UV light at a second frame rate, and / or, the at least one image sensor unit is configured to capture reflected infrared light at a third frame rate, the first frame rate is larger than the second frame rate and the third frame rate, and wherein the feedback trigger signal is determined at the second frame rate or the third frame rate. The second frame rate and the third frame rate may be a factor of 3 to 5 times lower than the first frame rate. The second frame rate and the third frame rate do not correspond to real time. The user is able to be informed about the quality level in the 3D model in a non-real time manner for the purpose of reducing the processing power for generating a visualization of the feedback signal. This would not be the case if the feedback signal was generated according to the first frame rate which provide the real time updating of the 3D model.

[0023] The real time mode may be determined by the first frame rate, and wherein the one or more processors may be configured to determine the feedback trigger signal at a frame rate being at least 1 / 2 of the first frame rate.

[0024] The quality level may be determined based on the captured two-dimensional images, wherein one or more properties, i.e. data, of the captured two-dimensional images are analysed by the one or more processors.

[0025] The quality level may be determined by, i.e. one or more properties, such asthe amount of data at a position in the 3D model, amount of data from different angles relative to a position in the 3D model, a quality of the data at a position in the 3D model, quality of data from different angles relative to a position in the 3D model, signal -to-noise ratio, sharpness, contrast, color, accuracy, and / or dynamic range. The signal-to-noise ratio may be in the captured two-dimensional images. For example, poor light conditions, a poor image sensor performance and / or a poor light source performance, high level of moisture inside the patient’s mouth could cause a degradation of the signal-to-noise ratio, and thereby, the quality level is lowered.

[0026] The dynamic range of the focusing in the two-dimensional images could also reduce the quality level.

[0027] The quality level may be lower than a quality threshold level, the 3D model is full transparent or has a specific color at the corresponding position of the determined quality level determined by the feedback signal. When the quality level is larger than the quality threshold level and keeps improving, the 3D model becomes less transparent or the color is changing at the corresponding position of the determined quality level.

[0028] The quality threshold level may vary between different parts in the 3D model. For example, at a first part of the 3D model no treatment or preparation is to be performed, and at a second part of the 3D model treatment or preparation is to be performed. In most situations there would be a need to have more accurate data in the 3D model where at least a treatment or preparation is to be carried out. For example, a treatment could be for a caries, crack or a tissue disease, such as inflammation, periodontitis, and gingivitis. A preparation could be for a restoration. The quality level and the position in the feedback signal is determined based on a type of treatment or a type of preparation, and wherein the type of treatment or preparation may be one or more of following a caries treatment, a crack treatment, a tissue disease, and a restoration preparation.

[0029] Dental objects may include “restorations”, which may be generally understood to include components that restore the structure or function of existing dentition, such as crowns, bridges, veneers, inlays, onlays, amalgams, composites, and various substructures such as copings and the like, as well as temporary restorations for use while a permanent restoration is being fabricated. In vicinity of these restorations, the accuracy and the quality level is extra important, and thus, the threshold is changed. For some users it would be an advantage to couple an audio feedback to the feedback signal, which means that a frequency of the audio feedback corresponds to a quality level at the position in the feedback signal. For example, in a scanning scenario, the user moves the handheld intraoral scanning device from one area to another area on the dental object of the patient the frequency of the audio would change as the quality level is different between the two areas. Furthermore, the frequence of the audio feedback signal determines the rate of the feedback trigger signal being generated. The system may include a loudspeaker for playing the audio feedback. The one or more processors may be configured to provide an audio feedback via a loudspeaker of the system, and wherein a timing of playing the audio feedback and the determining of the feedback trigger signal is synchronised.

[0030] The synchronizing of the audio feedback and the visualization of the feedback signal would enhance the users ability to keep track on the quality of the 3D model while scanning.

[0031] The one or more processors may be configured to change the geometry of the illumination pattern, and wherein the one or more processors may be configured to determine a feedback trigger signal when the illumination pattern is a specific pattern, such as a line pattern, a checkerboard pattern or any patterns that is different from the illumination pattern being used in the probe light for providing the 2D images with 3D data what are used for determining a 3D model.

[0032] BRIEF DESCRIPTION OF THE FIGURES

[0033] Aspects of the disclosure may be best understood from the following detailed description taken in conjunction with the accompanying figures. The figures are schematic and simplified for clarity, and they just show details to improve the understanding of the claims, while other details are left out. Throughout, the same reference numerals are used for identical or corresponding parts. The individual features of each aspect may each be combined with any or all features of the other aspects. These and other aspects, features and / or technical effect will be apparent from and elucidated with reference to the illustrations described hereinafter in which:

[0034] FIG.1 illustrates an example of an intraoral scanning system;

[0035] FIG.2 illustrates an example of an intraoral scanning system;

[0036] FIGS.3 A to 3E illustrates an example of scenarios that includes an intraoral scanning system;

[0037] FIGS. 4A and 4B illustrates an example of an intraoral scanning system and a scan sequence schedule;

[0038] FIGS. 5A to 5C illustrate different examples of displaying a feedback signal on a 3D model; and

[0039] FIGS. 6A and 6B illustrate yet another of displaying a feedback signal on a 3D model.

[0040] DETAILED DESCRIPTION

[0041] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. Several aspects of the devices, systems, mediums, programs and methods are described by various blocks, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). Depending upon particular application, design constraints or other reasons, these elements may be implemented using electronic hardware, computer program, or any combination thereof.

[0042] The electronic hardware may include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. Computer program shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0043] A scanning for providing intra-oral scan data may be performed by a dental scanning system that may include an intraoral scanning device such as the TRIOS series scanners from 3 Shape A / S. The dental scanning system may include a wireless capability as provided by a wireless network unit. The scanning device may employ a scanning principle such as triangulation-based scanning, confocal scanning, focus scanning, ultrasound scanning, x-ray scanning, stereo vision, structure from motion, optical coherent tomography OCT, or any other scanning principle. In an embodiment, the scanning device is capable of obtaining surface information by operated by projecting a pattern and translating a focus plane along an optical axis of the scanning device and capturing a plurality of 2D images at different focus plane positions such that each series of captured 2D images corresponding to each focus plane forms a stack of 2D images. The acquired 2D images are also referred to herein as raw 2D images, wherein raw in this context means that the images have not been subject to image processing. The focus plane position is preferably shifted along the optical axis of the scanning system, such that 2D images captured at a number of focus plane positions along the optical axis form said stack of 2D images (also referred to herein as a sub-scan) for a given view of the object, i.e. for a given arrangement of the scanning system relative to the object. After moving the scanning device relative to the object or imaging the object at a different view, a new stack of 2D images for that view may be captured. The focus plane position may be varied by means of at least one focus element, e.g., a moving focus lens. The scanning device is generally moved and angled relative to the dentition during a scanning session, such that at least some sets of sub-scans overlap at least partially, in order to enable reconstruction of the digital dental 3D model by stitching overlapping 3D subscans together in real-time and display the progress of the virtual 3D model on a display as a feedback to the user. The result of stitching is the digital 3D representation of a surface larger than that which can be captured by a single sub-scan, i.e. which is larger than the field of view of the 3D scanning device. Stitching, also known as registration and fusion, works by identifying overlapping regions of 3D surface in various sub-scans and transforming sub-scans to a common coordinate system such that the overlapping regions match, finally yielding the digital 3D model. An Iterative Closest Point (ICP) algorithm may be used for this purpose. Another example of a scanning device is a triangulation scanner, where a time varying pattern is projected onto the dental object and a sequence of images of the different pattern configurations are acquired by one or more cameras located at an angle relative to the projector unit.

[0044] Color texture of the dental object may be acquired by illuminating the object using different monochromatic colors such as individual red, green and blue colors or my illuminating the object using multi chromatic light such as white light. A 2D image may be acquired during a flash of white light.

[0045] Generally the process of obtaining surface information in real time of a dental object to be scanned requires the scanning device to illuminate the surface and acquire high number of 2D images. Typically a high speed camera is used with a framerate of 300-2000 2D frames pr second dependent on the technology and 2D image resolution. The high amount of image data needed to be handled by the scanning device to eighter directly forward the raw image data stream to an external processing device or performing some image processing before transmitting the data to an external device or display. This process requires that multiple electronic components inside the scanner is operating with a high workload thus requiring a high demand of current.

[0046] The scanning device comprises one or more light projectors configured to generate an illumination pattern to be projected on a three-dimensional dental object during a scanning session. The light projector(s) preferably comprises a light source, a mask having a spatial pattern, and one or more lenses such as collimation lenses or projection lenses. The light source may be configured to generate light of a single wavelength or a combination of wavelengths (mono- or polychromatic). The combination of wavelengths may be produced by using a light source configured to produce light (such as white light) comprising different wavelengths. Alternatively, the light projector(s) may comprise multiple light sources such as LEDs individually producing light of different wavelengths (such as red, green, and blue) that may be combined to form light comprising the different wavelengths. Thus, the light produced by the light source may be defined by a wavelength defining a specific color, or a range of different wavelengths defining a combination of colors such as white light. In an embodiment, the scanning device comprises a light source configured for exciting fluorescent material of the teeth to obtain fluorescence data from the dental object. Such a light source may be configured to produce a narrow range of wavelengths. In another embodiment, the light from the light source is infrared (IR) light, which is capable of penetrating dental tissue. The light projector(s) may be DLP projectors using a micro mirror array for generating a time varying pattern, or a diffractive optical element (DOF), or back-lit mask projectors, wherein the light source is placed behind a mask having a spatial pattern, whereby the light projected on the surface of the dental object is patterned. The back-lit mask projector may comprise a collimation lens for collimating the light from the light source, said collimation lens being placed between the light source and the mask. The mask may have a checkerboard pattern, such that the generated illumination pattern is a checkerboard pattern. Alternatively, the mask may feature other patterns such as lines or dots, etc.

[0047] The scanning device preferably further comprises optical components for directing the light from the light source to the surface of the dental object. The specific arrangement of the optical components depends on whether the scanning device is a focus scanning apparatus, a scanning device using triangulation, or any other type of scanning device. A focus scanning apparatus is further described in EP 2 442 720 Bl by the same applicant, which is incorporated herein in its entirety.

[0048] The light reflected from the dental object in response to the illumination of the dental object is directed, using optical components of the scanning device, towards the image sensor(s). The image sensor(s) are configured to generate a plurality of images based on the incoming light received from the illuminated dental object. The image sensor unit may be a high-speed image sensor such as an image sensor configured for acquiring images with exposures of less than 1 / 1000 second or frame rates in excess of 250 frames pr. second (fps). As an example, the image sensor may be a rolling shutter (CCD) or global shutter sensor (CMOS). The image sensor(s) may be a monochrome sensor including a color filter array such as a Bayer filter and / or additional filters that may be configured to substantially remove one or more color components from the reflected light and retain only the other non-removed components prior to conversion of the reflected light into an electrical signal. For example, such additional filters may be used to remove a certain part of a white light spectrum, such as a blue component, and retain only red and green components from a signal generated in response to exciting fluorescent material of the teeth.

[0049] The network unit may be configured to connect the dental scanning system to a network comprising a plurality of network elements including at least one network element configured to receive the processed data. The network unit may include a wireless network unit or a wired network unit. The wireless network unit is configured to wirelessly connect the dental scanning system to the network comprising the plurality of network elements including the at least one network element configured to receive the processed data. The wired network unit is configured to establish a wired connection between the dental scanning system and the network comprising the plurality of network elements including the at least one network element configured to receive the processed data.

[0050] The dental scanning system preferably further comprises a processor configured to generate scan data (such as extra-oral scan data and / or intra-oral scan data) by processing the two-dimensional (2D) images acquired by the scanning device. The processor may be part of the scanning device. As an example, the processor may comprise a Field- programmable gate array (FPGA) and / or an Advanced RISC Machines (ARM) processor located on the scanning device. The scan data comprises information relating to the three- dimensional dental object. The scan data may comprise any of: 2D images, 3D point clouds, depth data, texture data, intensity data, color data, and / or combinations thereof. As an example, the scan data may comprise one or more point clouds, wherein each point cloud comprises a set of 3D points describing the three-dimensional dental object. As another example, the scan data may comprise images, each image comprising image data e.g. described by image coordinates and a timestamp (x, y, t), wherein depth information can be inferred from the timestamp. The image sensor(s) of the scanning device may acquire a plurality of raw 2D images of the dental object in response to illuminating said object using the one or more light projectors. The plurality of raw 2D images may also be referred to herein as a stack of 2D images. The 2D images may subsequently be provided as input to the processor, which processes the 2D images to generate scan data. The processing of the 2D images may comprise the step of determining which part of each of the 2D images are in focus in order to deduce / generate depth information from the images. The internal depth information may be used to generate 3D point clouds comprising a set of 3D points in space, e.g., described by cartesian coordinates (x, y, z). The 3D point clouds may be generated by the processor or by another processing unit. Each 2D / 3D point may furthermore comprise a timestamp that indicates when the 2D / 3D point was recorded, i.e., from which image in the stack of 2D images the point originates. The timestamp is correlated with the z-coordinate of the 3D points, i.e., the z-coordinate may be inferred from the timestamp. Accordingly, the output of the processor is the scan data, and the scan data may comprise image data and / or depth data, e.g. described by image coordinates and a timestamp (x, y, t) or alternatively described as (x, y, z). The scanning device may be configured to transmit other types of data in addition to the scan data. Examples of data include 3D information, texture information such as infra-red (IR) images, fluorescence images, reflectance color images, x-ray images, and / or combinations thereof.

[0051] FIG. 1 illustrates an example of an intraoral scanning system 1. In this example, the system 1 includes a handheld intraoral scanning device 100 that includes at least one projector unit 7, at least one image sensor unit 5, and one or more processors 11. In this example, the system 1 includes one or more external computers 102 and a server 103 which communicates with an external computer 102. The one or more processors 11 may be distributed between the external computers 102, the server 103 and the handheld intraoral scanning device 100. The at least one projector unit 7 is configured to emit a probe light with a plurality of configurations in the form of an illumination pattern, and the at least one image sensor unit 5 accommodating an array of sensor elements configured to provide two-dimensional (2D) images with three-dimensional (3D) data based on captured reflections of the probe light from a dental object. The one or more processors is configured to determine a 3D model of the dental object by stitching the 2D images, update the 3D model by stitching 2D images provided by the at least one image sensor unit onto the 3D model, determine a quality level of the data in the 3D model at a position in the 3D model, and determine a feedback signal that includes a quality level of data in the 3D model and a corresponding position of the data in the 3D model, determine a feedback trigger signal when the one or more processors is not updating the 3D model. The system 1 includes a displaying unit 105, which is configured to display in real-time the 3D model, and the displaying unit 105 is further configured to display when the feedback trigger signal is determined , the quality level of the data at the position in the 3D model determined by the feedback signal.

[0052] FIG. 2 illustrates the one or more processors 11. The one or more processors 11 receives 2 the acquired two-dimensional images from the image sensor unit 5 and determines 4 the 3D model of the dental object being scanned. Further 2D images that are acquired by the image sensor unit 5 are stitched 4 onto the 3D model. A quality level of the data, e.g. the quality of the image data in the acquired 2D images, is determined 6, and a feedback signal 6 is determined which includes the quality level of data in the 3D model and a corresponding position of the data in the 3D model. In this example, a feedback trigger signal 10 is determined since the 3D model is not being updated 3 by the one or more processors 11. The no updating of the feedback trigger signal includes no stitching of 2D images onto the 3D model. The one or more processors 11 is configured to prepare 8 the 3D model to be displayed on the displaying unit 105. The one or more processors 11 is configured to prepare 8 the feedback signal to be displayed on the displaying unit 105. The feedback signal includes the quality level and a corresponding position, and the one or more processors 11 is configured to align the position of the quality level relative to the 3D model and apply the quality level onto the 3D model as an overlayer to the 3D model. The overlayer may be a colored overlayer where the color may change or diminish according to the quality level. The overlayer may be a layer with a transparency level, wherein the transparency level is adjusted according to the quality level. The overlayer is applied to the 3D model at the position determined by the feedback signal.

[0053] FIGS. 3A to 3E illustrates different scenarios with the handheld intraoral scanning device 100. In these examples, the handheld intraoral scanning device includes a gyroscope configured to determine an orientation of the handheld intraoral scanning device and / or an accelerometer configured to determine a motion of handheld intraoral scanning device. The one or more processors 11 is configured to determine the feedback trigger signal by comparing the orientation and / or the motion of the handheld intraoral scanning device to a scanning orientation range and / or a scanning motion pattern of the handheld intraoral scanning device 100. FIG. 3A illustrates a scanning scenario where a tip of the handheld intraoral scanning device 100 is inside the mouth 31 of a patient, and the scanner is moving 32 according to a typical movement that reflects a scanning scenario. In this example, no feedback trigger signal is determined. Furthermore, FIG.

[0054] 3 A does also illustrate that the user is mainly focused on the mouth 31 of the patient, and therefore, there are no needs for updating the 3D model with the feedback signal on the displaying unit 105. FIG. 3B illustrates a scenario where the scanning has been paused by pulling the handheld intraoral scanning device 100 out the mouth 31. In this example, the one or more processors 11 would not update 3 the 3D model as no relevant 2D images are acquired by the image sensor unit 5. FIG. 3B could also reflect a scenario where the scanning is either static or moving in a direction that does not reflects a scanning scenario, and in these two scenarios, the feedback trigger signal is generated. The feedback trigger signal is determined when the motion of the handheld intraoral scanning device 100 is not similar to reference motion patterns, or, when no motion is measured, and the reference motion patterns are stored in a memory unit of the system 1, and the reference motion patterns correspond to motion patterns in a scanning situation of a dental object 32. FIG. 3C illustrates a scenario where the tip of the scanner 100 is placed inside the mouth 32 of the patient still acquiring 2D images but there are no movement of the scanner 100. The one or more processors 11 stop stitching the acquired 2D images to the 3D model as the quality level of the part of the 3D model which is overlapped by the acquired 2D images is obtained. In this example, since the 3D model is not being updated the feedback trigger signal is generated. FIG. 3D illustrates an example where the orientation 33 of the handheld intraoral scanning device 100 does not reflect a scanning scenario, and in this example, the feedback trigger signal is generated. The system 1 includes a memory unit that includes the scanning orientation range of the handheld intraoral scanning device 100, and the feedback trigger signal is determined when the orientation of the handheld intraoral scanning device 100 is outside the scanning orientation range. In this specific example, the orientation is outside the scanning orientation range which is reflected by the tip of the handheld intraoral scanning device 100 is pointing upwards and in such a way which would reflect a very unnormal position of the patient, if the patient is being scanned. FIG. 3E illustrates an example where the feedback trigger signal 10 is generated when the user is pressing a button 35 on the scanner 100. When the user either presses the button 35 one more time or released the button 35 from the first button press, then the feedback signal is no more displayed.

[0055] The orientation may be defined by a XYZ coordinate system.

[0056] FIGS. 4A and 4B illustrate an example where the feedback trigger signal 10 is being generated according to a scan sequence schedule 40 which includes time slots 45 for emitting light. In FIG. 4A the scan sequence schedule 40 includes time slots for emitting the probe light (45A,45B,45C) and time slots for emitting ultraviolet light (45D, 45D’), and the feedback trigger signal 10 is being generated every time a fluorescence signal is being captured based on the emitted ultraviolet light (45D, 45D’). In FIG. 4B, the feedback trigger signal 10 is being generated every time a fluorescence signal is being captured based on the emitted ultraviolet light (45D,45D’) and / or every time a reflected infrared light is being captured based on the emitted infrared light (45E,45E’). In FIG. 4A, the real time mode is determined by a first frame rate of emitting the probe light (45 A,45B,45C), and wherein the at least one image sensor unit 5 is configured to capture fluorescent signals from the dental object based on the emitted UV light (45D,45D’) at a second frame rate, and the first frame rate is larger than the second frame rate, and wherein the feedback trigger signal 10 is determined at the second frame rate. In FIG 4B, the real time mode is determined by a first frame rate of emitting the probe light (45 A,45B,45C), and wherein the at least one image sensor unit 5 is configured to capture fluorescent signals from the dental object based on the emitted UV light (45D,45D’) at a second frame rate, and / or, the at least one image sensor unit 5 is configured to capture reflected infrared light at a third frame rate based on the emitted infrared light (45E,45E’), the first frame rate is larger than both the second frame rate and the third frame rate, and wherein the feedback trigger signal 10 is determined at the second frame rate or the third frame rate. FIGS. 5A to 5C illustrate an example of displaying the feedback signal 52 on the 3D model 50. In FIG. 5A the quality level of the feedback signal 52 is displayed on the 3D model 50 as a colored overlayer, and in FIG. 5B, the quality level is displayed as a transparent area in the 3D model at the position determined by the feedback signal, and in FIG. 5C, the quality level (52A,52B) is depicted as a colored gradient. For example, at the position 52B the color of the quality level is different from the other position 52A, and the color is determined based on the quality level.

[0057] FIGS. 6A and 6B illustrate an example where a quality threshold level may change between certain parts in the 3D model 50. In FIG. 6A it is seen that the 3D model 50 includes a marked area 62 which includes an area of the 3D model where the user has planned to apply a treatment or a preparation of the corresponding dental object of the patient. In this example, if the quality level 52 is lower than a quality threshold level, the 3D model is full transparent or has a specific color at the corresponding position of the determined quality level determined by the feedback signal. When the quality level 52 is larger than the quality threshold level and keeps improving, the 3D model becomes less / more transparent or the color is changing at the corresponding position of the determined quality level. The quality threshold level changes between different parts in the 3D model. For example, at a first part 63 of the 3D model 50 no treatment or preparation is to be performed, and at a second part 62 of the 3D model 50 treatment or preparation is to be performed. In most situations there would be a need to have more accurate data in the 3D model 50 where at least a treatment or preparation is to be carried out. For example, a treatment could be for a caries, crack or a tissue disease, such as inflammation, periodontitis, and gingivitis. A preparation could be for a restoration. The quality level and the position in the feedback signal is determined based on a type of treatment or a type of preparation, and wherein the type of treatment or preparation may be one or more of following a caries treatment, a crack treatment, a tissue disease, and a restoration preparation.

[0058] Many modifications and other embodiments of the inventions set forth herein will come to mind of one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or functions, it should be appreciated that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0059] ITEM

[0060] 1. An intraoral scanning system configured to display a feedback signal on a three- dimensional (3D) model of a dental object, wherein the system includes:

[0061] • at least one projector unit configured to emit a probe light with a plurality of configurations in the form of an illumination pattern;

[0062] • at least one image sensor unit accommodating an array of sensor elements configured to provide two-dimensional (2D) images with three-dimensional (3D) data based on captured reflections of the probe light from a dental object,

[0063] • one or more processors configured to: o determine a 3D model of the dental object by stitching the 2D images, o update the 3D model by stitching 2D images provided by the at least one image sensor unit onto the 3D model, o determine a quality level of the data in the 3D model at a position in the 3D model, and o determine a feedback signal that includes a quality level of data in the 3D model and a corresponding position of the data in the 3D model, o determine a feedback trigger signal when the one or more processors is not updating the 3D model,

[0064] • a displaying unit configured to display in real-time the 3D model, and the displaying unit is further configured to display when the feedback trigger signal is determined, the quality level of the data at the position in the 3D model determined by the feedback signal.

[0065] 2. The intraoral scanning system according to item 1, wherein the no updating of the feedback trigger signal includes no stitching of 2D images provided by the at least one image sensor unit onto the 3D model,

[0066] 3. The intraoral scanning system according to any of the previous items, comprising a handheld intraoral scanning device that includes:

[0067] • the at least one projector unit,

[0068] • the at least one image sensor unit,

[0069] • a gyroscope configured to determine an orientation of the handheld intraoral scanning device and / or an accelerometer configured to determine a motion of the handheld intraoral scanning device, and wherein the one or more processors is configured to determine the feedback trigger signal by comparing the orientation and / or the motion of the handheld intraoral scanning device to a scanning orientation range and / or a scanning motion pattern of the handheld intraoral scanning device.

[0070] 4. The intraoral scanning system according to item 3, comprising a memory unit that includes the scanning orientation range of the handheld intraoral scanning device, and the feedback trigger signal is determined when the orientation of the handheld intraoral scanning device is outside the scanning orientation range.

[0071] 5. The intraoral scanning system according to any of items 3 and 4, wherein the feedback trigger signal is determined when the motion of the handheld intraoral scanning device is not similar to reference motion patterns, or, when no motion is measured, and the reference motion patterns are stored in a memory unit of the system, and the reference motion patterns correspond to motion patterns in a scanning situation of a dental obj ect. 5. The intraoral scanning system according to any of the previous items, comprising a user interface, wherein the feedback trigger signal is determined when the user interface receives an input from a user of the intraoral scanning system.

[0072] 6. The intraoral scanning system according to any of the previous items, wherein the feedback signal includes a quality level of three-dimensional geometrical data, color data, and / or diagnostic data of the dental object.

[0073] 7. The intraoral scanning system according to any of the previous items, wherein the projector unit includes a plurality of light sources, wherein the plurality of light sources includes a first light source configured to emit visible light, a second light source configured to emit infrared light, and / or a third light source configured to emit UV light, and wherein the visible light includes the probe light.

[0074] 8. The intraoral scanning system according to item 7, wherein the real time mode is determined by a first frame rate, and wherein the at least one image sensor unit is configured to capture fluorescent signals from the dental object based on the emitted UV light at a second frame rate, and / or, the at least one image sensor unit is configured to capture reflected infrared light at a third frame rate, the first frame rate is larger than the second frame rate and the third frame rate, and wherein the feedback trigger signal is determined at the second frame rate or the third frame rate

[0075] 9. The intraoral scanning system according to item 8, wherein the second frame rate and the third frame rate do not correspond to real time.

[0076] 10. The intraoral scanning system according to any of items 1 to 7, wherein the real time mode is determined by a first frame rate, and wherein the one or more processors is configured to determine the feedback trigger signal at a frame rate being at least 1 / 2 of the first frame rate.

[0077] 11. The intraoral scanning system according to any of the previous items, wherein the quality level is determined by one or more of following: • amount of data at a position in the 3D model,

[0078] • amount of data from different angles relative to a position in the 3D model,

[0079] • quality of data at a position in the 3D model, and

[0080] • quality of data from different angles relative to a position in the 3D model.

[0081] 12. The intraoral scanning system according to any of the previous items, wherein the quality level is displayed with one or more colors, or, with one or more levels of transparencies in the 3D model.

[0082] 13. The intraoral scanning system according to any of the previous items, wherein the quality level is below a quality threshold level, the 3D model is full transparent or has a specific color at the corresponding position of the quality level determined by the feedback signal.

[0083] 14. The intraoral scanning system according to any of the previous items, wherein the quality level and the position in the feedback signal is determined based on a type of treatment or a type of preparation, and wherein the type of treatment or preparation is one of following:

[0084] • a caries treatment,

[0085] • a crack treatment,

[0086] • a periodontal treatment, and

[0087] • a restoration preparation.

[0088] 15. The intraoral scanning system according to item 6, wherein the diagnostic data includes a difference between infrared data and color data captured by the image sensor unit.

[0089] 16. The intraoral scanning system according to any of the previous items, wherein the one or more processors is configured to provide an audio feedback via a loudspeaker of the system, and wherein a timing of playing the audio feedback and the determining of the feedback trigger signal is synchronised.

Claims

CLAIMS1. An intraoral scanning system configured to display a feedback signal on a three- dimensional (3D) model of a dental object, wherein the system includes:• at least one projector unit configured to emit a probe light with a plurality of configurations in the form of an illumination pattern;• at least one image sensor unit accommodating an array of sensor elements configured to provide two-dimensional (2D) images with three-dimensional (3D) data based on captured reflections of the probe light from a dental object,• one or more processors configured to: o determine a 3D model of the dental object by stitching the 2D images, o update the 3D model by stitching 2D images provided by the at least one image sensor unit onto the 3D model, o determine a quality level of the data in the 3D model at a position in the 3D model, wherein the quality level may be determined based on one or more properties of the two-dimensional images, o determine a feedback signal that includes the quality level of data in the 3D model and a corresponding position of the data in the 3D model, o determine a feedback trigger signal when the one or more processors is not updating the 3D model,• a displaying unit configured to display in real-time the 3D model, and the displaying unit is further configured to display when the feedback trigger signal is determined, the quality level of the data at the position on the 3D model determined by the feedback signal.

2. The intraoral scanning system according to claim 1, wherein the no updating of the feedback trigger signal includes no stitching of 2D images onto the 3D model.

3. The intraoral scanning system according to any of the previous claims, comprising a handheld intraoral scanning device that includes:• the at least one projector unit,• the at least one image sensor unit,• a gyroscope configured to determine an orientation of the handheld intraoral scanning device and / or an accelerometer configured to determine a motion of handheld intraoral scanning device, and wherein the one or more processors is configured to determine the feedback trigger signal by comparing the orientation and / or the motion of the handheld intraoral scanning device to a scanning orientation range and / or a scanning motion pattern of the handheld intraoral scanning device.

4. The intraoral scanning system according to claim 3, comprising a memory unit that includes the scanning orientation range of the handheld intraoral scanning device, and the feedback trigger signal is determined when the orientation of the handheld intraoral scanning device is outside the scanning orientation range.

5. The intraoral scanning system according to any of claims 3 and 4, wherein the feedback trigger signal is determined when the motion of the handheld intraoral scanning device is not similar to reference motion patterns, or, when no motion is measured, and the reference motion patterns are stored in a memory unit of the system, and the reference motion patterns correspond to motion patterns in a scanning situation of a dental obj ect.

6. The intraoral scanning system according to any of the previous claims, wherein the feedback signal includes a quality level of three-dimensional geometrical data, color data, and / or diagnostic data of the dental object.

7. The intraoral scanning system according to any of the previous claims, wherein the projector unit includes a plurality of light sources, wherein the plurality of light sources includes a first light source configured to emit visible light, and a second light source configured to emit infrared light, and wherein the visible light includes the probe light.

8. The intraoral scanning system according to claim 7, wherein the real time mode is determined by a first frame rate, and wherein the at least one image sensor unit is configured to capture fluorescent signals from the dental object based on the emitted UVlight at a second frame rate, and / or, the at least one image sensor unit is configured to capture reflected infrared light at a third frame rate based on the emitted infrared light, and wherein the first frame rate is larger than both the second frame rate and the third frame rate, and wherein the feedback trigger signal is determined at the second frame rate or the third frame rate.

9. The intraoral scanning system according to claim 8, wherein the second frame rate and the third frame rate do not correspond to real time.

10. The intraoral scanning system according to any of claims 1 to 7, wherein the real time mode is determined by a first frame rate, and wherein the one or more processors is configured to determine the feedback trigger signal at a frame rate being at least 1 / 2 of the first frame rate.

11. The intraoral scanning system according to any of the previous claims, wherein the quality level is determined by one or more of following properties of the two- dimensional images:• amount of data at a position in the 3D model,• amount of data from different angles relative to a position in the 3D model,• quality of data at a position in the 3D model, and• quality of data from different angles relative to a position in the 3D model.

12. The intraoral scanning system according to any of the previous claims, wherein the quality level is displayed with one or more colors, or, with one or more levels of transparencies in the 3D model.

13. The intraoral scanning system according to any of the previous claims, wherein the quality level is below a quality threshold level, the 3D model is full transparent or has a specific color at the corresponding position of the quality level determined by the feedback signal.

14. The intraoral scanning system according to any of the previous claims, wherein the quality level and the position in the feedback signal is determined based on a type of treatment or a type of preparation, and wherein the type of treatment or preparation is one of following: • a caries treatment,• a crack treatment,• a tissue disease, and• a restoration preparation.

15. The intraoral scanning system according to claim 6, wherein the diagnostic data includes a difference between infrared data and color data captured by the image sensor unit.

Images