An intraoral scanning system with aligned focused images to a 3D surface model

Abstract

The present disclosure relates to an intraoral scanning system that is configured to determine three-dimensional surface model and a plurality of focused images. The intraoral scanning system comprising a projector unit configured to emit light onto a dental object, wherein the emitted light includes an infrared wavelength and structured light that includes a first visible wavelength; a first image sensor unit including; a first image sensor configured to acquire first reflected light from the dental object, wherein the first reflected light includes the infrared wavelength, and wherein the first image sensor includes an array of pixels, a micro-lens array arranged in front of the array of pixels and configured to convey the first reflected light to the array of pixels, wherein one or more micro lenses of the micro-lens array directs the first reflected light to one or more pixels of the array of pixels; a second image sensor unit including; a second image sensor configured to acquire second reflected light from the dental object, wherein the second reflected light includes the structured light, a processor unit configured to determine; three-dimensional (3D) data of the dental object based on the second reflected light, a three-dimensional (3D) surface model of the dental object by merging the 3D data, and a plurality of focused images at different focus depth into the dental object by selection of a set of pixels from the array of pixels associated with each microlens of the micro-lens array, and wherein the processor unit is configured to align a position of each of the plurality of focused images to a position in the 3D surface model based on the 3D data.

Description

[0001] AN INTRAORAL SCANNING SYSTEM WITH ALIGNED FOCUSED IMAGES

[0002] TO A 3D SURFACE MODEL

[0003] FIELD

[0004] The disclosure relates to an intraoral scanning system. More specifically, the intraoral scanning system provides an improved surface 3D model with aligned focused images of internal structures of a dental object.

[0005] BACKGROUND

[0006] Many dental and orthodontic procedures can benefit from accurate three-dimensional (3D) descriptions of a patient's dentition and intraoral cavity. In particular, it would be helpful to provide a three-dimensional description of both the surface and internal structures of the teeth, including the enamel and dentin, as well as caries and the general internal composition of the tooth volume. Although pure surface representations of the 3D surfaces of teeth have proven extremely useful in the design and fabrication of dental restorations (e.g., crowns or bridges) the ability to image internal structures including the development of caries and dental cracks in the enamel and underlying dentin would be tremendously useful, particularly in conjunction with a surface topographical mapping.

[0007] State of the art, ionizing radiation (e.g., X-rays) has been used to image the teeth for diagnostic purposes. For example, X-ray bitewing radiographs are often used to provide non-quantitative images of the teeth's internal structures. However, in addition to the risk of ionizing radiation, such images are typically limited in their ability to show early tooth mineralization changes (e.g. initial caries) resulting in underestimation of the demineralization depth; they are unable to assess the presence or not of micro-cavitation; they result in frequent overlap of the approximal tooth surfaces which requires repetition of radiograph acquisition and thus may involve a lengthy and expensive procedure.

[0008] Some intraoral features such as soft tissues and dental plaque are usually not visualized via X-ray because of their low density. Other techniques, such as cone beam computed tomography (CBCT) may provide tomographic images and be used to collect more information about the tissues and internal structure, but still require ionizing radiation. Furthermore, it is known that near-infrared (NIR) light can be used for assessing internal structure of a tooth and tooth surface in the form of transillumination of teeth or light reflection and backscattering from teeth. The NIR range offers a non-ionizing and safe approach to assess dental caries, restorations, cracks, enamel and dentin defects.

[0009] To further improve the accuracy of measuring focus depth of multiple 2D images of a dental object a light field camera or an image sensor unit with an array of pixels with a micro-lens array arranged in-front of has been shown very efficient.

[0010] However, the disadvantage of such a light field camera is the lack of accuracy in determining a 3D surface model of a dental object, and therefore, using a light field camera or an image sensor unit with an array of pixels with a micro-lens array arranged infront of is not suitable to provide an accurate 3D surface model with aligned focused images with different focus depths.

[0011] Furthermore, a single camera has a limited field of view and it is known to expand the field of view of an intraoral scanner by applying multiple cameras into an intraoral scanner arranged in either in a linear or circular arrangement. Unfortunately, the pricing of the intraoral scanner will increase significantly.

[0012] SUMMARY

[0013] It is an aspect of the present disclosure to overcome the above-mentioned disadvantages. By combining a first image sensor unit with a micro-lens array configured to acquire reflected light from within a dental object and a second image sensor unit configured to acquire structured light from a surface of the dental object into an intraoral scanning system will improve the accuracy of the acquired reflected light by the first image sensor unit. Furthermore, the combination of the first image sensor unit and the second image sensor unit provides a user of the intraoral scanning system the ability to adjust the focus depth into the dental object.

[0014] According to the aspect, an intraoral scanning system is disclosed. The intraoral scanning system may comprise a projector unit configured to emit light onto a dental object, wherein the emitted light includes an infrared wavelength and structured light that includes a first visible wavelength. The infrared wavelength may be between 800 nm and 1150 nm, and the visible wavelength may be between 350 nm and 750 nm. The structured light may be a static pattern or a time-varying pattern which may be provided by a structural pattern unit arranged such that the emitted light from the projector unit may be received by the structural pattern unit. The structural pattern unit may be configured to change the pattern dynamically. The structural pattern may be part of the intraoral scanning system. Furthermore, the intraoral scanning system may include a first image sensor unit that includes a first image sensor configured to acquire first reflected light from the dental object, wherein the first reflected light includes the infrared wavelength. The first image sensor may include an array of pixels. The first image sensor unit may include a micro-lens array arranged in front of the array of pixels of the first image sensor, and wherein the micro-lens array may be configured to convey the first reflected light to the array of pixels. The one or more micro-lenses of the micro-lens array may direct the first reflected light to one or more pixels of the array of pixels. The micro-lens array may generate a multitude of similar focused images onto the array of pixels, and from the slight angle differences between each of the focused images, the direction of the light can be calculated. Since each of the focused images is in part redundant, this step decreases the resolution of the system. In this example, the first image sensor unit may be a light field camera or a plenoptic camera. The first image sensor unit may be configured to capture a radiant intensity of reflected light in a plane. The first image sensor unit may capture the entire reflected light field in a plane.

[0015] The emitted light that includes infrared wavelength(s) may include a structured pattern, such as a static pattern or a time-varying pattern. The processing unit may be configured to register datapoints of the plurality of focused images by the help of the structured pattern in the first reflected light that includes infrared wavelengths.

[0016] A static pattern is a pattern that does not vary in time, e.g. a static checkerboard pattern or a static line pattern. A time varying pattern is a pattern that varies in time, i.e. the embedded spatial structure varies in time. May also be termed “time varying illumination pattern”. In the following also termed “fringes”.

[0017] The light field consists of radiant intensity and direction of the light. Thus, knowing the radiant intensity and direction of the light in one plane, it is possible to calculate the intensity and direction of this light in other planes. This enables the first image sensor unit to refocus the image onto other planes. The refocused image may include intraoral structures within a dental object determined by at least the first reflected light.

[0018] The second image sensor unit may be a photo image sensor or a high-speed photo image sensor configured with a frame rate of 25 frames per second or more. Each frame includes a subscan of multiple two-dimensional images of a specific wavelength range determined by the second reflected light. The multiple two-dimensional images of the subscan may be used for generating a point cloud in a three-dimensional Euclidean coordinate system. The point cloud may then be used for generating a 3D subscan which is then stitched together with other 3D subscans acquired previously by the intraoral scanning system.

[0019] The first image sensor unit may have a lower resolution than the second image sensor unit, however, this may be compensated by combining with 3D data taken earlier or later with any of the image sensors, i.e. the first image sensor unit or the second image sensor unit, thus achieving high resolution in the final 3D model.

[0020] The second image sensor unit of the intraoral scanning system may include a second image sensor configured to acquire second reflected light from the dental object, wherein the second reflected light includes the structured light. The second image sensor unit may include a plurality of pixels which receives the structured light from variable planes which causes the structures within the structured light to vary in size. The second image sensor unit outputs the two-dimensional image with depth information, wherein the depth information may be determined by the size variations of the structures in the structured light. The second image sensor may be configured to acquire second reflected light from the dental object via a moveable focus lens or via a triangulation arrangement of the second image sensor and the projector unit.

[0021] The first image sensor unit and the second image sensor unit may be configured to acquire the first and second reflected light, respectively, in parallel or sequential.

[0022] The intraoral scanning system may include a processor unit configured to determine three- dimensional (3D) data of the dental object based on the second reflected light, a three- dimensional (3D) surface model of the dental object by merging the 3D data, and a plurality of focused images at different focus depth into the dental object by selection of a set of pixels from the array of pixels associated with each micro-lens of the micro-lens array. Furthermore, the processor unit may be configured to align a position of the plurality of focused images to a position in the 3D model based on the 3D data.

[0023] The advantage of combining the first image sensor unit and the second image sensor unit is that the user or the processor unit is able to locate more precisely the plurality of focused images on a dentition of a patient, and wherein the dentition may be represented by the 3D surface model. In comparison to a system that includes a regular IR camera in combination with the second image sensor unit would not be able to achieve a depth resolution of a 3D surface model that is comparable to the combination of the first and the second image sensor unit. Matter of fact, a regular IR camera is able to acquire infrared images at a single focused image plane.

[0024] The first and the second image sensor unit in combination with the projector unit may be arranged within a handheld intraoral scanner, and the processing unit may be arranged within or external to the intraoral handheld scanner. In an example where the processing unit may be arranged external to the intraoral handheld scanner, the processing unit may be part of a computer, a server, a cloud server or any external computing device.

[0025] The intraoral scanning system may include a handheld intraoral scanner that includes a first optical path that guides the first and the second reflected light to the first image sensor unit and the second image sensor unit, respectively. In this example, the reflected light that are acquired by the two image sensor units are guided in parallel by optical components within a housing of the handheld intraoral scanner. At least partly between a mirror and the image sensor units the first and the second optical path are parallel. In another example, the handheld intraoral scanner may include a first optical path and a second optical for the first image sensor unit and the second image sensor unit, respectively, wherein the first and second optical path may be configured to guide the first and second reflected light to the first and second image sensor unit, respectively. The first optical path may be partly or fully orthogonal to the second optical path. Between the mirror and the second image sensor unit the first and the second optical path are orthogonal. The mirror receives the reflected light via a scanner window placed in the tip of the handheld intraoral scanner, and the mirror is configured to reflect the reflected light towards the first image sensor unit and / or the second image sensor unit. The scanner window maybe arranged within a housing of the handheld intraoral scanner, and the scanner window may be an opening in the housing or made of glass.

[0026] The second image sensor unit may include a Bayer filter arranged in-front of the second image sensor unit. The Bayer filter receives and forwards the reflected light towards an array of pixels of the second image sensor unit. The reflected light may be white, and the Bayer filter may be configured to filtrate the reflected light into red, green and blue light which are then forwarded to separate pixels of the array of pixels.

[0027] The optical components may include one or more mirrors, one or more beam splitters and / or one or more lenses.

[0028] The second image sensor unit may include a plurality of cameras that each includes an array of pixels. The plurality of cameras may include between four and eight cameras divided into eight one or two groups, wherein each of the two groups includes four cameras. The cameras of the one or two groups may be arranged around a light source, and wherein the light source is part of the projector unit. In the example with two groups of cameras, the projector unit includes two light sources configured to emit structured light A separate infrared light source may be arranged separately from the one or two groups of cameras, and the infrared light source may be part of the projector unit. The handheld intraoral scanner may include a housing which accommodates the first and the second image sensor unit and a mirror configured for directing the reflected light from the dental object to at least the second image sensor unit. The housing may include a tip end configured for being inserted into a mouth of a patient during a scanning sequence and a distal end that is opposite to the tip end. Furthermore, the housing may include a midpoint that has an equal distance to the tip end and the distal end. At the distal end the handheld intraoral scanner may include a power unit configured to powering the projector unit, the image sensor units, the processor unit, a memory unit and / or other components within the handheld intraoral scanner. At the midpoint the housing may include a user interface, such as a button, a touchpad or a different button mean configured for controlling the handheld intraoral scanner and / or a graphical user interface. In one example, the mirror is arranged at the tip end and the first and second image sensor unit are arranged in vicinity of the midpoint or the distal end. In this example the optical path from the dental object to the image sensor units have been increased to an extent which resolves in a handheld intraoral scanner that is less sensitive to where the user places the handheld intraoral scanner relative to the dental object while performing the scanning. Furthermore, it also reduces the size of the tip end that only the mirror is placed at the tip end, i.e. in the tip of the housing. In this example, the first and the second optical path may be parallel between the mirror and the image sensor units. Alternatively, the first and the second optical path are the same between the mirror and a beam splitter that is configured to split the first optical path and the second optical path, such that the first image sensor unit receives the first reflected light via the first optical path and the second image sensor unit receives the second reflected light via the second optical path. In another example, the first image sensor unit may be arranged in vicinity to the tip end and the second image sensor unit may be arranged in vicinity to the midpoint. In this example, a scanner window is arranged within the housing and at the tip end. The scanner window is arranged on a first side of the mirror, and the first image sensor unit is arranged on a second side of the mirror wherein a field of view of the first image sensor unit points directly towards the scanner window. Thereby, in this example the first reflected light that includes infrared wavelengths are not redirected by the mirror, instead, the first reflected light travels through the scanner window and directly to the first image sensor unit from the dental object. The second reflected light is redirected by the mirror towards the second image sensor unit. In this example, the first optical path is between the scanner window and the first image sensor unit, and the second optical path is between the mirror and the second image sensor unit.

[0029] In another example, the handheld intraoral scanner may include a housing which accommodates the first and the second image sensor unit. In this example, the field-of- view of both image sensor units are pointing directly towards the scanner window. In this example, the distance between the dental object and the image sensor units is shortened significantly which results in a more overall compact design of the handheld intraoral scanner.

[0030] The processing unit may be configured to determine the three-dimensional (3D) surface model of the dental object by merging the 3D data, and the merging of 3D data may include stitching of multiple 3D subscans generated by multiple two-dimensional images that includes the second reflected light. The second reflected light includes visible wavelengths.

[0031] The processor unit may be configured to align a position of the plurality of focused images to a position in the 3D model based on the 3D data. The alignment may involve alignment of the plurality of focused images onto the 3D model which is then displayed and visualized by the user as the plurality of focused images is part of the 3D model. In another example, the alignment may involve displaying the 3D model in a first window of a graphical user interface displayed on a display unit of the intraoral scanning system. Furthermore, the one or more of the plurality of focused images is displayed on a second window of the graphical user interface. A user is able to select manually one or more of the plurality of focused images on the second window via the 3D model. The manually selection may be provided by moving a marker on the 3D model, and the position of the marker on the 3D model corresponds to the aligned positioned of the selected one or more focused images. To improve the intraoral scanning system ability to emit light with both infrared wavelengths and visible wavelengths at the same time and acquire the reflected light with both the first image sensor unit and the second image sensor unit simultaneously or about the same time, an infrared blocker filter may be arranged in-front of the second image sensor unit for preventing reflected light including infrared wavelengths from being acquired by the second image sensor unit, and an infrared pass filter in-front of the first image sensor unit for preventing reflected light outside the infrared wavelength range from being acquired by the first image sensor unit. The infrared pass filter may be configured to pass wavelengths above 800 nm or between 800 nm and 1150 nm to the first image sensor unit.

[0032] The processor unit may be configured to align the plurality of focused images onto the 3D surface model. The processor unit may be configured to display on a graphical user interface the aligned plurality of focused images onto the 3D surface model and to allow a user of the intraoral scanning system to change a focus depth of one or more of the aligned plurality of focused images. The one or more focused images may be selected by the user by moving a marker on the 3D surface model to different positions on the 3D model, and the change of focus depth may be determined by a focus mean on the graphical user interface. The focus mean may be a sliding bar which the user may move for adjusting the focus. The marker may be a window, wherein the part of the 3D surface model which is within the window is affected by the adjustment of the focus. The one or more of plurality of focused images that are within the window are selected for being adjusted in focus depth via the focus mean.

[0033] The processor unit may be configured to merge the plurality of focused images into the 3D surface model. The processor unit may be configured to determine a volumetric point cloud within the 3D surface model based on a neural radiance field model applied to each of the plurality of focused images, and wherein the processor unit may be configured to modify the three-dimensional (3D) surface model by applying the volumetric point clouds to the 3D surface model based on the aligned position of the plurality of focused images to the position in the 3D model. When scanning a dental object the projector unit emits light and the image sensor units acquire reflected light according to a scanning sequence, and during the scanning sequence the plurality of focused images and the 3D data are provided with time stamps. The time stamp of each of the plurality of focused images and the 3D data is used for aligning the position of the plurality of focused images to a position in the 3D surface model.

[0034] The plurality of focused images may include a first time stamp and the 3D data may include a second time stamp, and wherein the alignment of the position of the plurality of focused images to the position in the 3D model may be performed by comparing the first time stamp and the second time stamp. The first time stamp may be the same or about the same as the second time stamp which indicates that the focused image with the first time stamp has been acquired at the same position as where the 3D data has been generated on the dental obj ect.

[0035] The first time stamp and the second time stamp may correspond to a scanning sequence, wherein the first time stamp corresponds to a time which the first image sensor unit acquires the first reflected light, and the second time stamp corresponds to a time which the second image sensor unit acquires the second reflected light or when the 3D data was generated based on the second reflected light.

[0036] The emitted light that includes the first visible wavelength may be white light. In another example, the intraoral scanning system may use the second image sensor unit for multiple purposes, such as for acquiring surface data or fluorescence data of the dental object, and wherein the purpose decides the wavelength of the emitted light of the projector unit.

[0037] The emitted light includes a second visible wavelength, and wherein the second visible wavelength may be equal to or different from the first visible wavelength. The first image sensor unit may be configured to acquire third reflected light from the dental object, wherein the third reflected light may include the second visible wavelength. In the example, where the second visible wavelength is equal to the first visible wavelength, the first image sensor unit may be configured to extend the field of view of the intraoral scanning system. In this example, both the first image sensor unit and the second image sensor unit are arranged within an intraoral scanner which results in an extended field of view of the second image sensor unit. The field of view of the first image sensor unit and the second image sensor unit may overlap.

[0038] The first image sensor unit may have a first field-of-view, and the second image sensor unit may have a second field-of-view, and wherein a total field of view of the intraoral scanning system includes a combination of the first and the second field-of-view, wherein the total field-of-view is larger than first field-of-view and the second field-of-view, respectively.

[0039] The stitching of the 3D subscans is based on an overlap between two or more 3D subscans, and the size of the overlap between the two or more 3D subscans may be depended on how fast the user is moving the intraoral scanner or by the field-of-view of the camera. By extending the field-of-view would resolve in an increased overlap between the 3D subscans, and therefore, the user is able to move the intraoral scanner faster and still maintain a proper overlap between the 3D subscans for performing the stitching.

[0040] The extended field-of-view may be achieved by combining the respective fields-of-view of all the image sensor units which will improve accuracy due to reduced amount of image stitching errors, especially in edentulous regions, where the gum surface is smooth and there may be fewer clear high resolution 3-D features. Having an extended field-of-view enables large smooth features, such as the overall curve of the tooth, to appear in each image frame, which improves the accuracy of stitching respective surfaces obtained from multiple such image frames.

[0041] The field-of-view of the first and the second image sensor unit may be partial overlapped. In another example, the field-of-view of the first and the second image sensor unit may not overlap, and in this example, the stitching of the 3D subscans is based on knowing the location of the field of view of both image sensor units. In this example, the processor unit may be configured to monitor a first position of a first field-of-view of the first image sensor unit on a dental object, and use the first position to determine a second position of the second field-of-view of the second image sensor unit. The processor unit may be configured to stitch the 3D subscans provided by the two image sensor units based on the first position and the second position. The second position may be determined based on a geometrical relation between the first and the second field-of-view. The geometrical relation between the first and the second field-of-view is calibrated and stored into a memory unit of the intraoral scanning system.

[0042] The first image sensor unit may include a first field-of-view, and the second image sensor unit may include a second field-of-view, and wherein a total field of view of the intraoral scanning system includes an overlap of the first and the second field-of-view, such that the total field of view of the intraoral scanning system may be equal to the first field-of-view or the second field-of-view. In this example, no extension of the field-of-view of the intraoral scanning system is obtained. However, the purpose of the none-extended field-of- view is to obtain the plurality of focused images and 3D data which are overlapping. The overlapping of the plurality of focused images and the 3D data are used for aligning the position of each of the plurality of focused images to a position in the 3D surface model based on the 3D data.

[0043] The penetration depth of the first reflected light, i.e. the infrared reflected light, into the dental object is determined mainly by the wavelength of the first reflected light, and thus, the amount of micro-lenses in the micro-lens array determines the resolution of the focus depth into the dental object. Each of the plurality of focused images corresponds to a focus depth inside the dental object. The processor unit may be configured to determine a focus depth for each of the plurality of focused images by selecting two or more pixels of the set of pixels which acquires at least a same point on the dental object, and wherein the selection of the two or more pixels corresponds to the focus depth. The selection of the two or more pixels may include a certain combination of the two or more pixels, and wherein the processor unit may be configured to retrieve the focus depth that corresponds to the combination of the one or more pixels from a memory unit of the intraoral scanning system. The processor unit may be configured to determine a focus depth for each of the plurality of focused images by selecting two or more pixels of the set of pixels which acquires at least a same point on the dental object, determining a position of the same point on the object by performing ray tracing of the first reflected light from each of the selected two or more pixels to the same point on the dental object, and performing triangulation between the two or more pixels and the position of the same point on the object. The ray tracing may be performed at least through one or more dental elements of within the dental object, and wherein the processor unit may be configured to determine or retrieve from a memory unit of the intraoral scanning system a refractive index of each of the one or more dental elements, and the ray tracing may be performed by including the refractive index of each of the one or more dental elements. The refractive index may correspond to an angle of incident of a ray of the first reflected light into the one or more dental elements. A dental element may be a dentine, an enamel, a pulp, a caries, a crack, a filling etc. The type of the dental element may be determined by a machine learning model. The machine learning model includes determining one or more dental elements for each of the plurality of focused images by determining an intensity level for each of the pixels of the array of pixels of the first image sensor unit, and correlating the intensity level to a trained intensity level which corresponds to a specific type of a dental element. The trained intensity level may be trained by manually identifying the type of a dental element on each of a plurality of focused images or on an infrared (IR) image(s) generated based on infrared reflected light. The identification on each of the plurality of focused images or the IR image(s) may be converted to a trained intensity level. In another example, the identification on each of the plurality of focused images or the IR image may be converted to a trained intensity level and a trained dental element position. In this example, the machine learning model includes determining one or more dental elements for each of the plurality of focused images by determining an intensity level for each of the pixels of the array of pixels of the first image sensor unit, determining a position of the intensity level relative to the dental object in the plurality of focused images, and determining the type of a dental element by correlating the intensity level to a trained intensity level and the position to the trained dental element position. The training of the trained intensity levels and / or the trained dental element position may be based on plurality of focuses images or IR image(S) from different patients.

[0044] The type of dental element may be determined by the processor unit based on the neural radiance field model. The neural radiance field model may include determine a set of input parameters for a casting object corresponding to each of the plurality of pixels of the first image sensor unit, wherein the set of input parameters comprises spatial location information and viewing angle information. The viewing angle information represents the viewing angle of the field-of-view of the first image sensor unit, and the spatial location information is a relative position between the first image to the dental object while acquiring the reflected light. The casting object may comprise a plurality of point coordinates; and a continuous volumetric machine learning model configured to receive and process the set of input parameters to determine an intensity value and a density value of a three-dimensional inner geometry of the 3D surface model, wherein the continuous volumetric machine learning model is configured to be trained using plurality of NIR images or focused images.

[0045] The continuous volumetric machine learning model may be trained by receiving the set of input parameters that corresponds to the casting object for each of the plurality of IR images or focused images, wherein the casting object includes the plurality of point coordinates associated with the 3D surface model, generating, based on the continuous volumetric machine learning model using the set of input parameters, the intensity value and the density value for each of the plurality of point coordinates, determining a synthetic pixel value for the casting object based on the corresponding determined intensity value and the density value for each of the plurality of point coordinates; and minimizing a loss function between the synthetic pixel value and a corresponding true pixel value of the plurality of pixels of the plurality of IR images or focused images by changing the intensity value and the density value for each of the plurality of point coordinates.

[0046] Each of the plurality of focused images corresponds to a focus depth which the processor unit may be configured to convert into a penetration depth in the dental object. The penetration depth may be in relation to a reference focus depth which is determined by the 3D data of the dental object. The processor unit may be configured to determine a surface focus depth based on the three-dimensional (3D) data of the dental object, and wherein the surface focus depth corresponds to a surface of the dental object. The processor unit may be configured to determine a penetration depth into the dental object of each of the plurality of focused images, and wherein the penetration depth is a difference between the reference focus depth and the focus depth of each of the plurality of focused images.

[0047] The penetration depth may be indicated on a graphical user interface of a display unit on or next to each of the plurality of focused images or the 3D model which includes the plurality of focused images.

[0048] The intraoral scanning system may include a display unit that may be configured to display the 3D model and one or more focused images of the plurality focused images of the dental object. The display unit may further display a focus adjustment mean, and wherein the processor unit may be configured to select the one or more focused images based on a focus depth that may be determined by the focus adjustment mean. The focus adjustment mean may be a range slider. The user may adjust the position of a slider on the focus adjustment mean, and while moving the slider the focus depth is changing and so does the selection of one or more focused images of the plurality of focused images.

[0049] The focus adjustment mean may not be visible on the graphical user interface of the display unit, and the focus depth is adjusted via the focus adjustment mean by swiping across the 3D model with the plurality of focused images or on a separate window displaying the one or more of plurality of focused images. In another window the 3D model is displayed.

[0050] BRIEF DESCRIPTION OF THE FIGURES

[0051] Aspects of the disclosure may be best understood from the following detailed description taken in conjunction with the accompanying figures. The figures are schematic and simplified for clarity, and they just show details to improve the understanding of the claims, while other details are left out. Throughout, the same reference numerals are used for identical or corresponding parts. The individual features of each aspect may each be combined with any or all features of the other aspects. These and other aspects, features and / or technical effect will be apparent from and elucidated with reference to the illustrations described hereinafter in which:

[0052] FIGS. 1 A to IE illustrate different examples of an intraoral scanning system;

[0053] FIG. 2 illustrates another example of an intraoral scanning system

[0054] FIGS. 3 A, 3B, and 3C illustrate different examples of a field-of-view of image sensor units;

[0055] FIGS. 4 A and 4B illustrate a plurality of focused images and a surface focused depth, respectively;

[0056] FIGS. 5A, 5B and 5C illustrate different examples of a first image sensor unit;

[0057] FIGS. 6A, 6B, and 6C illustrate different examples of a sequence of projected light from a projecting unit;

[0058] FIGS. 7A and 7B illustrate different examples of a neural radiance field model; and FIGS. 8A and 8B illustrate different examples on displaying of a 3D surface model and one or more plurality of focused images.

[0059] DETAILED DESCRIPTION

[0060] The detailed description set forth below in connection with the appended drawings is intended as a description of various configurations. The detailed description includes specific details for the purpose of providing a thorough understanding of various concepts. However, it will be apparent to those skilled in the art that these concepts may be practiced without these specific details. Several aspects of the devices, systems, mediums, programs and methods are described by various blocks, functional units, modules, components, circuits, steps, processes, algorithms, etc. (collectively referred to as “elements”). Depending upon particular application, design constraints or other reasons, these elements may be implemented using electronic hardware, computer program, or any combination thereof.

[0061] The electronic hardware may include microprocessors, microcontrollers, digital signal processors (DSPs), field programmable gate arrays (FPGAs), programmable logic devices (PLDs), gated logic, discrete hardware circuits, and other suitable hardware configured to perform the various functionality described throughout this disclosure. Computer program shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, executables, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

[0062] A scanning for providing intra-oral scan data may be performed by a dental scanning system that may include an intraoral scanning device such as the TRIOS series scanners from 3 Shape A / S. The dental scanning system may include a wireless capability as provided by a wireless network unit. The scanning device may employ a scanning principle such as triangulation-based scanning, confocal scanning, focus scanning, ultrasound scanning, x-ray scanning, stereo vision, structure from motion, optical coherent tomography OCT, or any other scanning principle. In an embodiment, the scanning device is capable of obtaining surface information by operated by projecting a pattern and translating a focus plane along an optical axis of the scanning device and capturing a plurality of 2D images at different focus plane positions such that each series of captured 2D images corresponding to each focus plane forms a stack of 2D images. The acquired 2D images are also referred to herein as raw 2D images, wherein raw in this context means that the images have not been subject to image processing. The focus plane position is preferably shifted along the optical axis of the scanning system, such that 2D images captured at a number of focus plane positions along the optical axis form said stack of 2D images (also referred to herein as a sub-scan) for a given view of the object, i.e. for a given arrangement of the scanning system relative to the object. After moving the scanning device relative to the object or imaging the object at a different view, a new stack of 2D images for that view may be captured. The focus plane position may be varied by means of at least one focus element, e.g., a moving focus lens. The scanning device is generally moved and angled relative to the dentition during a scanning session, such that at least some sets of sub-scans overlap at least partially, in order to enable reconstruction of the digital dental 3D model by stitching overlapping 3D subscans together in real-time and display the progress of the virtual 3D model on a display as a feedback to the user. The result of stitching is the digital 3D representation of a surface larger than that which can be captured by a single sub-scan, i.e. which is larger than the field of view of the 3D scanning device. Stitching, also known as registration and fusion, works by identifying overlapping regions of 3D surface in various sub-scans and transforming sub-scans to a common coordinate system such that the overlapping regions match, finally yielding the digital 3D model. An Iterative Closest Point (ICP) algorithm may be used for this purpose. Another example of a scanning device is a triangulation scanner, where a time varying pattern is projected onto the dental arch and a sequence of images of the different pattern configurations are acquired by one or more cameras located at an angle relative to the projector unit.

[0063] Color texture of the dental arch may be acquired by illuminating the object using different monochromatic colors such as individual red, green and blue colors or my illuminating the object using multi chromatic light such as white light. A 2D image may be acquired during a flash of white light.

[0064] Generally the process of obtaining surface information in real time of a dental arch to be scanned requires the scanning device to illuminate the surface and acquire high number of 2D images. Typically, a high speed camera is used with a framerate of 100-3000 2D frames pr second dependent on the technology and 2D image resolution. The high amount of image data needed to be handled by the scanning device to either directly forward the raw image data stream to an external processing device or performing some image processing before transmitting the data to an external device or display. This process requires that multiple electronic components inside the scanner is operating with a high workload thus requiring a high demand of current.

[0065] The scanning device comprises one or more light projectors configured to generate an illumination pattern to be projected on a three-dimensional dental arch during a scanning session. The light projector(s) preferably comprises a light source, a mask having a spatial pattern, and one or more lenses such as collimation lenses or projection lenses. The light source may be configured to generate light of a single wavelength or a combination of wavelengths (mono- or polychromatic). The combination of wavelengths may be produced by using a light source configured to produce light (such as white light) comprising different wavelengths. Alternatively, the light projector(s) may comprise multiple light sources such as LEDs individually producing light of different wavelengths (such as red, green, and blue) that may be combined to form light comprising the different wavelengths. Thus, the light produced by the light source may be defined by a wavelength defining a specific color, or a range of different wavelengths defining a combination of colors such as white light. In an embodiment, the scanning device comprises a light source configured for exciting fluorescent material of the teeth to obtain fluorescence data from the dental arch. Such a light source may be configured to produce a narrow range of wavelengths. In another embodiment, the light from the light source is infrared (IR) light, which is capable of penetrating dental tissue. The light projector(s) may be DLP projectors using a micro mirror array for generating a time varying pattern, or a diffractive optical element (DOF), or back-lit mask projectors, wherein the light source is placed behind a mask having a spatial pattern, whereby the light projected on the surface of the dental arch is patterned. The back-lit mask projector may comprise a collimation lens for collimating the light from the light source, said collimation lens being placed between the light source and the mask. The mask may have a checkerboard pattern, such that the generated illumination pattern is a checkerboard pattern. Alternatively, the mask may feature other patterns such as lines or dots, etc.

[0066] The scanning device preferably further comprises optical components for directing the light from the light source to the surface of the dental arch. The specific arrangement of the optical components depends on whether the scanning device is a focus scanning apparatus, a scanning device using triangulation, or any other type of scanning device. A focus scanning apparatus is further described in EP 2 442 720 Bl by the same applicant, which is incorporated herein in its entirety.

[0067] The light reflected from the dental arch in response to the illumination of the dental arch is directed, using optical components of the scanning device, towards the image sensor(s). The image sensor(s) are configured to generate a plurality of images based on the incoming light received from the illuminated dental arch. The image sensor unit may be a high-speed image sensor such as an image sensor configured for acquiring images with exposures of less than 1 / 1000 second or frame rates in excess of 250 frames pr. second (fps). As an example, the image sensor may be a rolling shutter (CCD) or global shutter sensor (CMOS). The image sensor(s) may be a monochrome sensor or include a color filter array such as a Bayer filter and / or additional filters that may be configured to substantially remove one or more color components from the reflected light and retain only the other non-removed components prior to conversion of the reflected light into an electrical signal. For example, such additional filters may be used to remove a certain part of a white light spectrum, such as a blue component, and retain only red and green components from a signal generated in response to exciting fluorescent material of the teeth.

[0068] The network unit may be configured to connect the dental scanning system to a network comprising a plurality of network elements including at least one network element configured to receive the processed data. The network unit may include a wireless network unit or a wired network unit. The wireless network unit is configured to wirelessly connect the dental scanning system to the network comprising the plurality of network elements including the at least one network element configured to receive the processed data. The wired network unit is configured to establish a wired connection between the dental scanning system and the network comprising the plurality of network elements including the at least one network element configured to receive the processed data.

[0069] The dental scanning system preferably further comprises a processor configured to generate scan data (such as extra-oral scan data and / or intra-oral scan data) by processing the two-dimensional (2D) images acquired by the scanning device. The processor may be part of the scanning device. As an example, the processor may comprise a Field- programmable gate array (FPGA) and / or an Advanced RISC Machines (ARM) processor located on the scanning device. The scan data comprises information relating to the three- dimensional dental arch. The scan data may comprise any of: 2D images, 3D point clouds, depth data, texture data, intensity data, color data, and / or combinations thereof. As an example, the scan data may comprise one or more point clouds, wherein each point cloud comprises a set of 3D points describing the three-dimensional dental arch. As another example, the scan data may comprise images, each image comprising image data e.g. described by image coordinates and a timestamp (x, y, t), wherein depth information can be inferred from the timestamp. The image sensor(s) of the scanning device may acquire a plurality of raw 2D images of the dental arch in response to illuminating said object using the one or more light projectors. The plurality of raw 2D images may also be referred to herein as a stack of 2D images. The 2D images may subsequently be provided as input to the processor, which processes the 2D images to generate scan data. The processing of the 2D images may comprise the step of determining which part of each of the 2D images are in focus in order to deduce / generate depth information from the images. The internal depth information may be used to generate 3D point clouds comprising a set of 3D points in space, e.g., described by cartesian coordinates (x, y, z). The 3D point clouds may be generated by the processor or by another processing unit. Each 2D / 3D point may furthermore comprise a timestamp that indicates when the 2D / 3D point was recorded, i.e., from which image in the stack of 2D images the point originates. The timestamp is correlated with the z-coordinate of the 3D points, i.e., the z-coordinate may be inferred from the timestamp. Accordingly, the output of the processor is the scan data, and the scan data may comprise image data and / or depth data, e.g. described by image coordinates and a timestamp (x, y, t) or alternatively described as (x, y, z). The scanning device may be configured to transmit other types of data in addition to the scan data. Examples of data include 3D information, texture information such as infra-red (IR) images, fluorescence images, reflectance color images, x-ray images, and / or combinations thereof.

[0070] FIGS.1 A to IE illustrate different examples of an intraoral scanning system 1. In the examples, the system 1 includes a projector unit 2 configured to emit light and structured light onto a dental object 11. The emitted light includes an infrared wavelength, and the structured light includes a first visible wavelength. The system 1 includes a first image sensor unit 4 including a first image sensor 5 configured to acquire first reflected light from the dental object 11, wherein the first reflected light includes the infrared wavelength. The first image sensor 5 includes an array of pixels 5, and a micro-lens array 7 arranged in front of the array of pixels 5 and configured to convey the first reflected light to the array of pixels 5. One or more micro lenses of the micro-lens array 7 directs the first reflected light to one or more pixels of the array of pixels 5. The system 1 further includes a second image sensor unit 3 including a second image sensor 3B configured to acquire second reflected light from the dental object 11, wherein the second reflected light includes the structured light. The system 1 further includes a processor unit 6 configured to determine three-dimensional (3D) data of the dental object 11 based on the second reflected light, a three-dimensional (3D) surface model of the dental object 11 by merging the 3D data, and a plurality of focused images at different focus depth into the dental object 11 by selection of a set of pixels from the array of pixels associated with each micro-lens of the micro-lens array 7. The processor unit 6 is configured to align a position of the plurality of focused images to a position in the 3D surface model based on the 3D data. In the examples illustrated in FIGs 1 A to ID, the projector unit 2, image sensor units (3, 4), the processor 6 and a mirror 9 are arranged within a handheld intraoral scanner 10. The image sensor units (3,4) and the projector unit 2 are electrically connected to the processor unit 6. In FIG. IE, the handheld intraoral scanner 10 includes the projector unit 2, image sensor units (3, 4), and the processor unit 6. In another example, the intraoral scanning system 1 includes a plurality of processors distributed into an external computer or a server and the handheld intraoral scanner 10. In FIG. 1 A, the mirror reflects the emitted light from the projector unit 2 onto the dental object 11, and the reflected light from the dental object 11 is redirected by the mirror 9 to the image sensors (3, 4). FIG. IB, illustrates the second image sensor unit 3 includes a second image unit 12 that includes an array of pixels 12, and a Bayer filter 13 is arranged in-front of the array of pixels 12. The Bayer filter receives and forwards the reflected light towards the array of pixels 12. The reflected light may be white, and the Bayer filter may be configured to filtrate the reflected light into red, green and blue light which are then forwarded to separate pixels of the array of pixels 12. FIG. 1C illustrates an example, where the first image sensor unit 4 is arranged in vicinity to the mirror 9 wherein the field- of-view of the first image sensor unit 4 is directed towards a scanning window 15. In this example, the first image sensor unit 4 includes an additional lens to direct the reflected light towards the micro-lens array 7. Further to this example, the projector unit 2 includes a first light source 2A that is configured to emit structured light with visible wavelengths and a second light source 2B configured to emit light including an infrared wavelength, and the emitted infrared light is directly transmitted towards the scanning window 15, and wherein the emitted structured light is directed towards the scanning window 15 via the mirror 9. The intraoral scanning system 1 in FIG. ID is similar to the system 1 in FIG. 1C, however, the second image sensor unit 3 includes plurality of cameras (3 A, 3B). In this example, the plurality of cameras (3 A, 3B) includes two cameras. In FIG. 1 A, the handheld intraoral scanner 10 includes a housing 10 which has a tip end 20, a distal end 21 which is opposite to the tip end, and a midpoint 22 which is equally positioned relative to the tip end 20 and the distal end 21. In the examples illustrated in FIGS. 1A to ID the first image sensor unit 4 is arranged in vicinity to the tip end 20 or the midpoint 21. Furthermore, the second image sensor unit 3 is arranged in vicinity to the midpoint 21.

[0071] In FIG. IE, the first and second image sensor unit (3, 4) and the projector unit 2 are arranged in vicinity to the tip end 20 of the housing 10. In this example, the field-of-view of both image sensor units (3, 4) are directed towards the scanning windows 15, and the projector unit 2 is configured to emit directly towards the scanning window 15.

[0072] FIG. 2 illustrates yet another example of an intraoral scanning system 1 that comprises a projector unit 2 configured to emit light and structured light onto a dental object 11, wherein the emitted light includes an infrared wavelength and the structured light includes a first visible wavelength. The system 1 further includes a first image sensor unit 5 including a first image sensor 4 configured to acquire reflected light from the dental object, and wherein the first image sensor 4 includes an array of pixels 5, and a micro-lens array 7 arranged in front of the array of pixels and configured to convey the reflected light to the array of pixels, wherein one or more micro lenses of the micro-lens array 7 directs the reflected light to one or more pixels of the array of pixels 5. Furthermore, the system 1 includes a processor unit 6 configured to determine three-dimensional (3D) data based on the reflected light that includes the structured light, a three-dimensional (3D) surface model of the dental object 11 by merging the 3D data, and a plurality of focused images at different focus depth into the dental object 3 by selection of a set of pixels from the array of pixels 5 associated with each micro-lens of the micro-lens array 7. The processor unit 6 is configured to align a position of the plurality of focused images to a position in the 3D model based on the 3D data.

[0073] In another example, in relation to FIGs. 1 A to IE, the second image senor unit 3 is configured to capture the first reflected light that includes infrared wavelength(s). In this example, the second image sensor unit 3 includes a Bayer filter 13 that includes one or more red, green and blue filter channels that are configured to filter the white light of the second reflected light into the different colors and forward these colors to respective pixels of the pixel array of the second image sensor 12. The Bayer filter 13 includes one or more combined filter channels where at least a color and infrared wavelengths are filtered and forwarded to the array of pixels, such as green and infrared. In this example, the projector unit 2 is configured to switch between visible wavelengths and infrared wavelengths such that visible light and infrared light are being emitted sequentially, i.e. at different time slots, and not at the same time slot, i.e. in parallel. The switching between visible wavelengths and infrared wavelengths may include turning on and off the respective light sources of the projector unit 2 that emit the respective wavelengths, or turning up and down the power of the light sources such that the capturing of both visible and infrared wavelengths with the combined filter channel has minimal negative affection to the 3D data or the infrared data. In another example, the Bayer filter 13 includes the one or more red, green, infrared and blue filter channels that are configured to filter the white light and infrared such that each of the pixels of the array of pixels of the second image sensor unit 3 receives either the red, green, infrared or blue light. In the example, where the second image sensor unit 3 is configured to captured infrared wavelengths and provide a reference infrared image to the processor unit 6. The processor unit 6 is configured to improve the resolution of the plurality of focused images by comparing each of the plurality of focused images with the reference infrared image for selecting a focused image which has the same or about the same focus depth. The selected focused image, the plurality of focused images and the reference infrared image are fed into a machine learning algorithm which are configured to improve the resolution of the plurality of focused images by performing an interpolation of the pixels of each of the plurality of focused images based on a comparison of the selected focused image and the reference infrared image.

[0074] FIGs. 3A, 3B and 3C illustrate different examples of a field-of-view (30A, 30B) of the image sensor units (3,4). The first image unit sensor unit 4 has a first field-of-view 30A and the second image sensor unit 3 has a second field-of-view 3 OB. In the example illustrated in FIG. 3A, the two field-of-views (30A, 30B) are arranged such that the first field-of-view 30A extends the second field-of-view 30B. In this example, the processor unit 6 is configured to align the 3D data provided by the images captured by the two image sensor units (3,4) based on a time stamp of each of the captured images. Alternatively, the stitching of the 3D subscans is based on knowing the location of the field of view of both image sensor units (3,4). In this example, the processor unit 6 may be configured to monitor a first position of a first field-of-view of the first image sensor unit 3 on a dental object 11, and use the first position to determine a second position of the second field-of- view of the second image sensor unit 4. The processor unit 6 may be configured to stitch the 3D subscans provided by the two image sensor units (3,4) based on the first position and the second position. The second position may be determined based on a geometrical relation between the first and the second field-of-view (30A,30B). The geometrical relation between the first and the second field-of-view (30A,30B ) is calibrated and stored into a memory unit of the intraoral scanning system 1.

[0075] The stitching of the 3D subscans is based on an overlap between two or more 3D subscans, and the size of the overlap between the two or more 3D subscans may be depended on how fast the user is moving the intraoral scanner 10 or by the field-of-view of the camera (3,4). By extending the field-of-view would resolve in an increased overlap between the 3D subscans, and therefore, the user is able to move the intraoral scanner faster and still maintain a proper overlap between the 3D subscans for performing the stitching.

[0076] The extended field-of-view may be achieved by combining the respective fields-of-view of all the image sensor units (3,4) which will improve accuracy due to reduced amount of image stitching errors, especially in edentulous regions, where the gum surface is smooth and there may be fewer clear high resolution 3-D features. Having an extended field-of- view enables large smooth features, such as the overall curve of the tooth, to appear in each image frame, which improves the accuracy of stitching respective surfaces obtained from multiple such image frames. In. Fig. 3B a partially overlap between the two field-of- view is illustrated, and the disadvantage with solution is a reduction of the extended field- of-view seen in relation to the example with no overlap, but the stitching of the 3D subscans provided by the two image sensor units (3,4) is a lot simpler. In Fig. 3C, the two field-of-views (30 A, 3 OB) are fully overlapping, which means no extension of the field of view view of either image sensor units (3,4) is obtained. In this example, the purpose of the none-extended field-of-view 30 is to obtain the plurality of focused images and 3D data which are overlapping. The overlapping of the plurality of focused images and the 3D data are used for aligning the position of each of the plurality of focused images to a position in the 3D surface model based on the 3D data.

[0077] FIGS. 4 A and 4B illustrate the plurality of focused images (41 A - 41E) and the surface focused depth 40. The first reflected light 43 is a sum of multiple internal reflections from inside the dental object 11 which is sorted by selecting a focused image of the plurality of focused images 41. Each of the plurality of focused images correspond to a focus depth. The structured reflected light 42 is used for determining a reference focus depth 40 which corresponds to the surface of the dental object 11. In FIG. 4B, a penetration depth is determined by the processor unit 6 based on the reference focus depth 40 and the focus depth of each of the plurality of focused images (41A-41E). The penetrations depth (44A- 44E) is determined for each of the plurality of focused images (41A-41E) as a difference between the reference focus depth 40 and each of the focus depth of the plurality of focused images (41A-41E). The penetration depth (44A-44E) may be converted to a distance by the processing unit 6 based on a calibration factor stored in a memory unit of the system 1. The calibration factor may be determined during the manufacture of the system 1 or the handheld intraoral scanner 10.

[0078] FIGS. 5A - 5C illustrate different examples of the first image sensor unit 4. In the example, the first reflected light is guided by an optical lens module 50 towards micro-lens array 7 which are then redirecting the first reflected light towards one or more pixels of the array of pixel 5. In this example, the plurality of focused images includes two focused images from two different focus planes (54,55). In FIG. 5B, the focused images include one focused image 55 from a focus depth that is within the enamel 51 and another focused image 54 from a focus depth within the dentine 52. In FIG. 5C, a caries 56 is seen in the enamel 51 of the dental object 11, and in this example, the focused image 55 corresponds to the focus plane 55 which depicts a caries 56 inside the enamel 51 in focus. FIGS. 6A-6C illustrate different examples of a sequence (60,60A,60B) of projected light from the projecting unit 2. FIG. 6A illustrates an example, where visible wavelengths including white 61 and blue wavelengths, are emitted in parallel to the emitted infrared wavelengths 63. The projector unit 2 is configured to emit pulses of the different wavelengths, however, in FIG 6B, the projector unit 2 is configured to constantly emit infrared wavelengths 63 while the white 61 and blue wavelengths are interchangeably switched on and off. In FIG. 6C, the projector unit 2 is configured to switch between three different wavelengths (61,62,63), wherein the emitted white 61 and blue 62 wavelengths are interchangeably switched on and off. The emitted infrared wavelengths 63 is turned up and down in power.

[0079] FIGS. 7 A and 7B illustrate different examples of a neural radiance field model used by the processing unit 2 to determine a volumetric point cloud within 3D surface model based on one or more of the plurality of focused images 718. The neural radiance field model may include determine a set of input parameters for a casting object 724 corresponding to each of the plurality of pixels of the first image sensor unit 4, wherein the set of input parameters comprises spatial location information and viewing angle information. The viewing angle information represents the viewing angle of the field-of-view of the first image sensor unit 4, and the spatial location information is a relative position between the first image to the dental object while acquiring the reflected light. The casting object (724- 728) may comprise a plurality of point coordinates (726A-726E), (732A - 732E); and a continuous volumetric machine learning model configured to receive and process the set of input parameters to determine an intensity value and a density value of a three- dimensional inner geometry of the 3D surface model, wherein the continuous volumetric machine learning model is configured to be trained using plurality of NIR images or focused images. In FIG. 7A, the casting object 724 is a ray and in FIG. 7B, the casting object 728 is a cone. The processors 6 may be configured to determine an average of content within a visible volume 734 for the pixel 722. For example, the content within the visible volume 734 of the pixel 722 of the array of pixels 5 may indicate a color intensity of the pixel 722. Furthermore, the average of the color intensity within the visible volume 734 may enable to identify presence of, for example, part or point of teeth, caries, lesion, cracks, dentin, enamel junction, filling, or any other object present on or inside the point of teeth captured by the pixel 722. For example, the processors 6 may be configured to render or project the cone 728 or the ray 724 as the casting object corresponding to the pixel 722 based on the corresponding average of content. Further, the cone 728 may be rendered such that the cone 728 models a whole volume of space that may be visible through the pixel 722 based on the average of content or color intensity within the visible volume 734.

[0080] The cone 728 may include a plurality of point coordinates, depicted as point coordinates 732A, 732B, 732C, 732D and 732E (collectively referred to as point coordinates 732, hereinafter). Further, the cone 728 may be sliced into conical frustums corresponding to the point coordinates 732. For example, a conical frustum 736 may correspond to the point coordinate 732D. To this end, each of the point coordinates 732 along the cone 728 may be transformed with a positional encoding of a volume of the corresponding conical frustums. The point coordinates 732 may be sampled along the cone 728. Spatial location information and viewing angle information corresponding to each transformed point coordinate is used to determine the set of input parameters for optimizing the continuous volumetric scene function of the continuous volumetric ML model 108, wherein the continuous volumetric ML model is the mip-NeRF based neural network.

[0081] The ray -based casting object 724 may be casted or projected through a pixel 722 of one or more of the plurality of focused images 718. The ray 724 may include a plurality of point coordinates, depicted as point coordinates 726 A, 726B, 726C, 726D and 726E. For example, each of the point coordinates 726 along the ray 724 may have to be transformed with a positional encoding, for example, using a gamma function. The point coordinates 726 may be sampled along the ray 724. Spatial location information and viewing angle information corresponding to each transformed point coordinate is used to determine the set of input parameters for optimizing the continuous volumetric scene function of the continuous volumetric ML model.

[0082] FIGS. 8A and 8B illustrate different examples on displaying of the 3D surface model and one or more of the plurality of focused images on a graphical user interface 80 of the system 1. In. FIG. 8 A, the plurality of focused images 41 has been aligned onto the 3D surface model 81. The focus depth is adjusted by a focus adjustment mean 82 which in this example is a sliding bar. The focus adjustment mean 82 can be a button which is rotatable. In FIG. 8B, a focused image (41 A-41D) of the plurality of focused image 41 is displayed in a window separate from the window showing the 3D surface model 81. The focused image (41 A-41D) is selected by moving the position of a marker 83 relative to the 3D surface model 81, wherein the processing unit 6 is configured to select the focused image (41A-41D) by comparing position of the marker 83 relative to the 3D surface model 81 and a position of each of the plurality focused images. The position may be defined by geometrical coordinates provided by a motion sensor within the handheld intraoral scanner device 10 or by a timestamp.

[0083] Many modifications and other embodiments of the inventions set forth herein will come to mind of one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is to be understood that the inventions are not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, although the foregoing descriptions and the associated drawings describe example embodiments in the context of certain example combinations of elements and / or functions, it should be appreciated that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions than those explicitly described above are also contemplated as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0084] ITEM

[0085] 1. An intraoral scanning system comprising;

[0086] • a projector unit configured to emit light onto a dental object, wherein the emitted light includes an infrared wavelength and structured light that includes a first visible wavelength;

[0087] • a first image sensor unit including; o a first image sensor configured to acquire first reflected light from the dental object, wherein the first reflected light includes the infrared wavelength, and wherein the first image sensor includes an array of pixels, o a micro-lens array arranged in front of the array of pixels and configured to convey the first reflected light to the array of pixels, wherein one or more micro lenses of the micro-lens array directs the first reflected light to one or more pixels of the array of pixels;

[0088] • a second image sensor unit including; o a second image sensor configured to acquire second reflected light from the dental object, wherein the second reflected light includes the structured light,

[0089] • a processor unit configured to determine; o three-dimensional (3D) data of the dental object based on the second reflected light, o a three-dimensional (3D) surface model of the dental object by merging the 3D data, and o a plurality of focused images at different focus depth into the dental object by selection of a set of pixels from the array of pixels associated with each micro-lens of the micro-lens array, and wherein the processor unit is configured to align a position of the plurality of focused images to a position in the 3D surface model based on the 3D data.

[0090] 2. The intraoral scanning system according to item 1, comprising an infrared blocker filter that is configured to block reflected light including infrared wavelengths from being acquired by the second image sensor unit, and an infrared pass filter configured to pass through wavelengths above 800 nm or between 800 nm and 1100 nm.

[0091] 3. The intraoral scanning system according to any of the previous items, wherein the processor unit is configured to align the plurality of focused images onto the 3D surface model. 4. The intraoral scanning system according to any of the previous items, wherein the emitted light includes a second visible wavelength, and wherein the second visible wavelength is equal to or different from the first visible wavelength, and the first image sensor unit is configured to acquire third reflected light from the dental object, wherein the third reflected light includes the second visible wavelength.

[0092] 5. The intraoral scanning system according to any of the previous items, wherein the first image sensor unit includes a first field-of-view, and the second image sensor unit includes a second field-of-view, and wherein a total field of view of the intraoral scanning system includes a combination of the first and the second field-of-view, wherein the total field-of-view is larger than first field-of-view and the second field-of- view, respectively.

[0093] 6. The intraoral scanning system according to any of the items 1 to 4, wherein the first image sensor unit includes a first field-of-view, and the second image sensor unit includes a second field-of-view, and wherein a total field of view of the intraoral scanning system includes an overlap of the first and the second field-of-view, such that the total field of view of the intraoral scanning system is equal to the first field-of-view or the second field-of-view.

[0094] 7. The intraoral scanning system according to any of the previous items, wherein the processor unit is configured to determine a focus depth for each of the plurality of focused images by selecting two or more pixels of the set of pixels which acquires at least a same point on the dental object, and wherein the selection of the two or more pixels corresponds to the focus depth.

[0095] 8. The intraoral scanning system according to item 7, wherein the selection of the two or more pixels includes a certain combination of the two or more pixels, and wherein the processor unit is configured to retrieve the focus depth that corresponds to the combination of the one or more pixels from a memory unit of the intraoral scanning system. 9. The intraoral scanning system according to any of the previous items, wherein the processor unit is configured to determine a focus depth for each of the plurality of focused images by:

[0096] • selecting two or more pixels of the set of pixels which acquires at least a same point on the dental obj ect,

[0097] • determining a position of the same point on the object by performing ray tracing of the first reflected light from each of the selected two or more pixels to the same point on the dental object, and

[0098] • performing triangulation between the two or more pixels and the position of the same point on the object.

[0099] 10. The intraoral scanning system according to item 9, wherein the ray tracing is performed at least through one or more dental elements of within the dental object, and wherein the processor unit is configured to determine or retrieve from a memory unit of the intraoral scanning system a refractive index of each of the one or more dental elements, and the ray tracing is performed by including the refractive index of each of the one or more dental elements.

[0100] 11. The intraoral scanning system according to item 10, wherein the refractive index corresponds to an angle of incident of a ray of the first reflected light into the one or more dental elements.

[0101] 12. The intraoral scanning system according to any of the previous items, wherein the processor unit is configured to determine a reference focus depth based on the three- dimensional (3D) data of the dental object, and wherein the reference focus depth corresponds to a surface of the dental object.

[0102] 13. The intraoral scanning system according to item 12, wherein the processor unit is configured to determine a penetration depth into the dental object of each of the plurality of focused images, and wherein the penetration depth is a difference between the reference focus depth and the focus depth of each of the plurality of focused images. 14. The intraoral scanning system according to any of the previous items, wherein the processor unit is configured to determine a volumetric point cloud within the dental object based on a neural radiance field model applied to each of the plurality of focused images.

[0103] 15. The intraoral scanning system according to item 14, wherein the processor unit is configured to modify the three-dimensional (3D) surface model by applying the volumetric point clouds to the 3D surface model based on the aligned position of the plurality of focused images to the position in the 3D model.

[0104] 16. The intraoral scanning system according to any of the previous items, wherein the plurality of focused images includes a first time stamp and the 3D data includes a second time stamp, and wherein the alignment of the position of the plurality of focused images to the position in the 3D model is performed by comparing the first time stamp and the second time stamp.

[0105] 17. The intraoral scanning system according to item 16, wherein the first time stamp and the second time stamp correspond to a scanning sequence, wherein the first time stamp corresponds to a time which the first image sensor unit acquires the first reflected light, and the second time stamp corresponds to a time which the second image sensor unit acquires the second reflected light.

[0106] 18. The intraoral scanning system according to any of the previous items, wherein the first image sensor unit and the second image sensor unit are configured to acquire the first and second reflected light, respectively, in parallel or sequential.

[0107] 19. The intraoral scanning system according to any of the previous items, wherein the first visible wavelength is between 400 nm and 700 nm, a second visible wavelength is between 350 nm to 500 nm, and the infrared wavelength is between 800 nm and 1150 nm. 20. The intraoral scanning system according to any of the previous items, comprising a handheld intraoral scanner that includes a first optical path that guides the first and the second reflected light to the first image sensor unit and the second image sensor unit, respectively, or, the handheld intraoral scanner includes a first optical path and a second optical for the first image sensor unit and the second image sensor unit, respectively, wherein the first and second optical path are configured to guide the first and second reflected light to the first and second image sensor unit, respectively.

[0108] 21. The intraoral scanning system according to any of the previous items, comprising a display unit configured to:

[0109] • display the 3D model and one or more focused images of the plurality focused images of the dental object,

[0110] • display a focus adjustment mean, and wherein the processor unit is configured to select the one or more focused images based on a focus depth that is determined by the focus adjustment mean.

[0111] 22. The intraoral scanning system according to any of items 21, wherein the focus adjustment mean is a range slider.

[0112] 23. The intraoral scanning system according to any of the previous items, wherein the processor unit is configured to display the plurality of focused images aligned onto the 3D model.

Claims

CLAIMS1. An intraoral scanning system comprising;• a projector unit configured to emit light onto a dental object, wherein the emitted light includes an infrared wavelength and structured light that includes a first visible wavelength;• a first image sensor unit including; o a first image sensor configured to acquire first reflected light from the dental object, wherein the first reflected light includes the infrared wavelength, and wherein the first image sensor includes an array of pixels, o a micro-lens array arranged in front of the array of pixels and configured to convey the first reflected light to the array of pixels, wherein one or more micro lenses of the micro-lens array directs the first reflected light to one or more pixels of the array of pixels;• a second image sensor unit including; o a second image sensor configured to acquire second reflected light from the dental object via a moveable focus lens or via a triangulation arrangement of the second image sensor and the projector unit, wherein the second reflected light includes the structured light,• a processor unit configured to determine; o three-dimensional (3D) data of the dental object based on the second reflected light, o a three-dimensional (3D) surface model of the dental object by merging the 3D data, and o a plurality of focused images at different focus depth into the dental object by selection of a set of pixels from the array of pixels associated with each micro-lens of the micro-lens array, and wherein the processor unit is configured to align a position of the plurality of focused images to a position in the 3D surface model based on the 3D data.

2. The intraoral scanning system according to claim 1, comprising an infrared blocker filter that is configured to block reflected light including infrared wavelengths from being acquired by the second image sensor unit, and an infrared pass filter configured to pass through wavelengths above 800 nm or between 800 nm and 1100 nm.

3. The intraoral scanning system according to any of the previous claims, wherein the processor unit is configured to align the plurality of focused images onto the 3D surface model.

4. The intraoral scanning system according to any of the previous claims, wherein the emitted light includes a second visible wavelength, and wherein the second visible wavelength is equal to or different from the first visible wavelength, and the first image sensor unit is configured to acquire third reflected light from the dental object, wherein the third reflected light includes the second visible wavelength.

5. The intraoral scanning system according to any of the previous claims, wherein the first image sensor unit includes a first field-of-view, and the second image sensor unit includes a second field-of-view, and wherein a total field of view of the intraoral scanning system includes a combination of the first and the second field-of-view, wherein the total field-of-view is larger than first field-of-view and the second field-of- view, respectively.

6. The intraoral scanning system according to any of the claims 1 to 4, wherein the first image sensor unit includes a first field-of-view, and the second image sensor unit includes a second field-of-view, and wherein a total field of view of the intraoral scanning system includes an overlap of the first and the second field-of-view, such that the total field of view of the intraoral scanning system is equal to the first field-of-view or the second field-of-view.

7. The intraoral scanning system according to any of the previous claims, wherein the processor unit is configured to determine a focus depth for each of the plurality of focused images by selecting two or more pixels of the set of pixels which acquires atleast a same point on the dental object, and wherein the selection of the two or more pixels corresponds to the focus depth.

8. The intraoral scanning system according to claim 7, wherein the selection of the two or more pixels includes a certain combination of the two or more pixels, and wherein the processor unit is configured to retrieve the focus depth that corresponds to the combination of the one or more pixels from a memory unit of the intraoral scanning system.

9. The intraoral scanning system according to any of the previous claims, wherein the processor unit is configured to determine a focus depth for each of the plurality of focused images by:• selecting two or more pixels of the set of pixels which acquires at least a same point on the dental obj ect,• determining a position of the same point on the object by performing ray tracing of the first reflected light from each of the selected two or more pixels to the same point on the dental object, and• performing triangulation between the two or more pixels and the position of the same point on the object.

10. The intraoral scanning system according to claim 9, wherein the ray tracing is performed at least through one or more dental elements of within the dental object, and wherein the processor unit is configured to determine or retrieve from a memory unit of the intraoral scanning system a refractive index of each of the one or more dental elements, and the ray tracing is performed by including the refractive index of each of the one or more dental elements.

11. The intraoral scanning system according to claim 10, wherein the refractive index corresponds to an angle of incident of a ray of the first reflected light into the one or more dental elements.

12. The intraoral scanning system according to any of the previous claims, wherein the processor unit is configured to determine a reference focus depth based on the three- dimensional (3D) data of the dental object, and wherein the reference focus depth corresponds to a surface of the dental object.

13. The intraoral scanning system according to claim 12, wherein the processor unit is configured to determine a penetration depth into the dental object of each of the plurality of focused images, and wherein the penetration depth is a difference between the reference focus depth and the focus depth of each of the plurality of focused images.

14. The intraoral scanning system according to any of the previous claims, wherein the processor unit is configured to determine a volumetric point cloud within the dental object based on a neural radiance field model applied to each of the plurality of focused images.

15. The intraoral scanning system according to claim 14, wherein the processor unit is configured to modify the three-dimensional (3D) surface model by applying the volumetric point clouds to the 3D surface model based on the aligned position of the plurality of focused images to the position in the 3D model.

16. The intraoral scanning system according to any of the previous claims, wherein the plurality of focused images includes a first time stamp and the 3D data includes a second time stamp, and wherein the alignment of the position of the plurality of focused images to the position in the 3D model is performed by comparing the first time stamp and the second time stamp.

17. The intraoral scanning system according to claim 16, wherein the first time stamp and the second time stamp correspond to a scanning sequence, wherein the first time stamp corresponds to a time which the first image sensor unit acquires the first reflected light, and the second time stamp corresponds to a time which the second image sensor unit acquires the second reflected light.

18. The intraoral scanning system according to any of the previous claims, wherein the first image sensor unit and the second image sensor unit are configured to acquire the first and second reflected light, respectively, in parallel or sequential.

19. The intraoral scanning system according to any of the previous claims, wherein the first visible wavelength is between 400 nm and 700 nm, a second visible wavelength is between 350 nm to 500 nm, and the infrared wavelength is between 800 nm and 1150 nm.

20. The intraoral scanning system according to any of the previous claims, comprising a handheld intraoral scanner that includes a first optical path that guides the first and the second reflected light to the first image sensor unit and the second image sensor unit, respectively, or, the handheld intraoral scanner includes a first optical path and a second optical for the first image sensor unit and the second image sensor unit, respectively, wherein the first and second optical path are configured to guide the first and second reflected light to the first and second image sensor unit, respectively.

21. The intraoral scanning system according to any of the previous claims, comprising a display unit configured to:• display the 3D model and one or more focused images of the plurality focused images of the dental object,• display a focus adjustment mean, and wherein the processor unit is configured to select the one or more focused images based on a focus depth that is determined by the focus adjustment mean.

22. The intraoral scanning system according to any of claims 21, wherein the focus adjustment mean is a range slider.

23. The intraoral scanning system according to any of the previous claims, wherein the processor unit is configured to display the plurality of focused images aligned onto the 3D model.

Images