Computerized Dental Visualization

Abstract

A method for visualizing a dental condition, comprising: projecting a color-coded pattern type with an intraoral scanner, providing a plurality of projection images of the dental condition, detecting one or more anatomical features in each of the projection images, removing the color-coded pattern information from at least one of the projection images, and rendering color information to the one or more anatomical features in the projection images via machine learning to generate a plurality of corresponding at least partially artificially colored projection images, where the pattern information is removed through rendering. The present teachings also relate to a system, an intraoral scanner, a software product, and a storage medium.

Description

[Technical field]

[0001] The present invention relates to a computer-implemented method and system for dental scanning, and more particularly for intraoral scanning, and visualization thereof. [Background technology]

[0002] Dental scanners, such as intraoral dental scanners, or simply referred to as intraoral scanners, are used to capture digital dental impressions of a patient. The impression is usually in the form of a three-dimensional (3D) model and may relate to features of interest of the oral anatomy, such as the teeth and / or soft and / or hard tissues in the patient's mouth. The impression may be stored digitally and / or transmitted to a dental practitioner. Digital impressions may also be used for other purposes, such as the manufacture of dental restorations and the planning or performance of dental procedures.

[0003] It is known to use intraoral scanners based on the principle of structured light. Such scanners comprise means for projecting a light pattern onto the intraoral geometry. When projected onto the intraoral geometry, the pattern appears contoured according to the shape of the geometry. The scanner also comprises a camera for capturing the light reflected from the geometry, which contains information related to the contours formed in the projected pattern. The signal from the camera is then sent to a computing unit for extracting shape information of the intraoral geometry. A digital impression or a 3D model of the intraoral geometry can then be reconstructed from the shape information, which may be in the form of a point cloud.

[0004] The light patterns used in structured light imaging may be of different types, for example stripes or dots. Furthermore, the patterns may include different color components, for example stripes or dots of different colors. The main goal of structured light imaging is to obtain a digital impression that is as accurate as possible. Different types of patterns and colors may offer different advantages in achieving this.

[0005] While an intraoral scan is being performed, the user or practitioner, or even the patient, may want to follow a live image of the area of ​​the tooth condition or geometry being scanned. A live picture or video of the area in front of the scanning tip of the scanner is therefore useful for the user and / or the patient. Therefore, many structured light scanners provide a visual picture mode. In the visual picture mode, the structured light pattern is switched off and the normal light source can be switched on to capture a visual picture of the scan area. In some cases, even a separate camera for visual imaging may be provided, but the same camera may be usable in both the structured light imaging mode and the visual imaging mode. In either case, this requires interleaving the structured light imaging mode with the visual imaging mode, which may at least slow down the scanning operation. Thus, the generation of a digital impression may take longer than necessary. Furthermore, additional hardware such as a switching unit and a light source may be required to provide the visual imaging mode in addition to the structured light imaging mode, and usually only the latter is used to obtain a digital impression.

[0006] The intraoral scanner Omnicam by Dentsply Sirona uses structured light for 3D reconstruction. To be able to display a live image while scanning with the Omnicam, the scanner includes a second white light source and optical elements such as a beam splitter, allowing to switch between two imaging modes: structured light and live view image. Thus, compared to a scanner constructed according to the invention, the Omnicam requires an additional light source and additional optical elements.

[0007] In the following description, reference is made to the following documents [A] to [E]:

[0008] [A]: Estimating a surface texture of a tooth, US10682209B2, Align Technology, Inc. [B]: Dental imaging and illumination device, US10458848B2, 3M. [C]: Light field intraoral 3D scanner with structured light illumination, US20190388194 A1, Align Technology, Inc. [D]: Intraoral scanner with integrated viewfinder, US20200348005A1, Align Technology, Inc. [E]: 3D geometric modeling and motion capture using both single and dual imaging, US8208719B2, Mantis Vision Ltd.

[0009] The patent document in reference [A] includes a description of an intraoral scanner with multiple light sources because "the use of dental devices (e.g., optical devices, intraoral devices, etc.) may result in capturing the entire shape of the tooth and may lack the information required by the dental technician to generate a natural-looking dental restoration...". In a way, reference [A] addresses the opposite problem of the present invention. In a similar vein, reference [B] describes a dental scanner with several light paths to enable accurate measurement of color. Another disclosure in reference [C] describes an approach very similar to Omnicam, based on structured light, in which an additional "uniform light projector sends white light onto the object being scanned" to enable imaging of the color and texture of the surface. Similarly, reference [D] claims a "second illumination unit that generates a second light that generates a viewfinder image". In one non-dental patent, see Reference [E], it is acknowledged that for surface reconstruction based on coded light patterns, some additional data input source is required to render the surface texture ("The method in which the 3D mesh is further processed to add texture data of the imaged object").

[0010] Recently, self-scanning or self-use dental scanners have also been introduced. Such scanners can enable patients to scan their own oral cavity or can generally enable non-practitioners to perform the scanning operation. This can save the user costs compared to having a dental practitioner or specialist perform the scan, and / or dental health monitoring can be improved by performing more regular and / or faster scans at home. When addressing the direct to consumer market, lower cost solutions are generally more attractive.

[0011] Thus, there is a recognized need for improved intraoral scanning methods and devices. Summary of the Invention

[0012] Images with projected light patterns are not suitable for human operators. Dental professionals prefer live camera images while performing intraoral scans (Fig. 2 right). These live camera images are not actually required to reconstruct the geometry of the patient's dental situation. If color textures on the reconstructed 3D surfaces are of interest, these can be generated from the live view. Nevertheless, to generate live camera images usable by humans, it is necessary so far for structured light based intraoral scanners to rely on some kind of switching mechanism between the structured light used for the actual reconstruction and the white light used to generate the live view images.

[0013] This switching requires additional electronics and / or optics, such as a beam splitter, to be built in, thus making such structured light based intraoral scanners more expensive to manufacture. Furthermore, it may lead to larger hardware designs to accommodate the additional components, or slower scanning due to the alternating switching between structured light and white light modes.

[0014] The present invention relates to various types of structured light patterns, including, but not limited to, color-coded stripe patterns such as those used by Dentsply Sirona's Omnicam, monochromatic stripe patterns ("dark-light-dark"), dot patterns with or without color coding, dot patterns with shape or size coding, irregular dot patterns, or line patterns with non-uniform spacing.

[0015] Regardless of the type of structured light pattern used for reconstruction, the present invention addresses the problem of generating a human-usable live view image in software without additional light sources, optics, or switching mechanisms built into the scanner hardware.

[0016] Depending on the type of structured light used, the invention proposes appropriate operating modes for generating live view images: Inpainting fills artificial color information into image areas affected by the pattern; color correction corrects the color of image areas affected by the pattern; Inpainting is particularly suitable for all structured light patterns that leave parts of the scanned area unilluminated; color correction is particularly suitable for reconstructing the true (white light) appearance of surfaces illuminated by colored light (whether or not they also have unilluminated areas), for example for omni-comb type structured light patterns (Fig. 2 left).

[0017] At least some of the limitations associated with the foregoing may be overcome by the subject matter of the attached independent claims. At least some of the further advantageous alternatives will be outlined in the dependent claims.

[0018] Applicant has realized through inventive efforts that data-driven techniques can be utilized to create artificially colored visual images from structured light image data captured via an intraoral scanner. Thus, the need for a visual imaging mode can be eliminated. This can save costs and make intraoral scanning faster while still maintaining a visually followable scanning process for the user and / or patient. Thus, the scanning process can remain intuitive despite the elimination of a separate visual imaging mode.

[0019] More specifically, in a first aspect, there is provided a computer-implemented method for visualizing a dental condition, the method comprising: - projecting a pattern onto the patient's dental condition by means of an intraoral scanner; providing a plurality of projected images of the dental condition via an intraoral scanner; - detecting one or more anatomical features in each of the projection images; removing color-coded pattern information from at least one of the projected images; and - rendering texture and / or color information to the one or more anatomical features in the projection images via machine learning to generate a plurality of projection images that are at least partially artificially colored.

[0020] At least a part of the artificially colored projected image may be provided, for example, at an output interface, which may be connected to a human machine interface.

[0021] The method comprises: and displaying at least a portion of the artificially colored projected image via a human-machine interface ("HMI") unit.

[0022] Thus, color information is automatically provided in the image data received from the intraoral scanner. This may have the advantage that at least the visual imaging mode is prevented from consuming additional time that would otherwise be required to complete a digital impression of the dental condition. This may speed up the scanning operation. Furthermore, the cost of the intraoral scanner may be reduced while maintaining the same or similar advantages of the visual imaging mode for the user and / or patient. Thus, depending on the application, it may be possible to acquire more structured light images and / or to relax the specifications of the scanner device and / or to allow faster operation of the scanner during scanning. Yet another advantage of the proposed method is that the acquisition frequency of the image sensor of the intraoral scanner may be lower compared to conventional devices, thus lowering the requirements for hardware and again relaxing the specifications of the scanner device.

[0023] Preferably, the display operation of the artificially colored projection image is performed in real-time or near real-time with respect to the capture of the projection image in order to provide a better user experience. The term "real-time" is well known to those skilled in the art. A real-time processing or computational operation is one that provides a response time within a specified time. The specified time in this context does not require a precise definition, as it depends on the application at hand. However, typically, in modern computer-implemented processes, the time period for providing a response to a given input or stimulus is at most a few seconds or less. Preferably, the time period is in the range of seconds or less, more preferably milliseconds or microseconds, or even shorter if technically possible. In general, in the computational domain, real-time refers to operations without perceptible delay. Those skilled in the art should recognize that a more precise definition or specific values ​​are not required for the scope or generality of the present teachings. The rendering operations may thus be performed in a manner that allows for smooth visualization in synchronization with the speed at which a realistic intraoral scan is performed. The projection image may be provided directly or indirectly via an intraoral scanner. For example, in some cases, some or all of the projection image may be processed after being captured by the intraoral scanner. Thus, some or all of the projection images may be captured directly via the intraoral scanner or may be processed after capture by the intraoral scanner. The projection images may be provided to a computer memory or one or more memory storage units. In some cases, the computer memory and / or memory storage may be part of a cloud storage or cloud computing service.

[0024] A "projected image" may refer to a structured light image captured via an intraoral scanner. Typically, multiple structured light images are provided as image data generated by the intraoral scanner during a scanning operation.

[0025] In some cases, the projection images may be in the form of or included in a data stream, which may be generated, at least in part, via an intraoral scanner.

[0026] An "intra-oral scanner" or "intra-oral dental scanner" refers to a device used to capture a digital impression of an oral anatomy or dental condition. The terms may be used interchangeably and refer to a dental scanner that can be used to generate a dental impression for a patient. From this, an intra-oral scanner can be used to generate a digital model of the patient's mouth. The intra-oral scanner of the present disclosure is more specifically a device that relies on at least structured light imaging to provide a plurality of projected images. Thus, the intra-oral scanner comprises at least one structured light source and an image sensor for recording reflections of the structured light reflected from the patient's oral anatomy. The output of the image sensor or camera generated in response to the reflections is thus used to provide the projected images. The output may be in the form of a data stream that is generated continuously or intermittently as the intra-oral scanner transmits structured light and records its reflections from the patient's oral anatomy.

[0027] It is clear that the dental condition or oral anatomy refers to the anatomical parts within the oral cavity. The oral anatomy may therefore include any one or more of the teeth and / or soft and / or hard tissues. In some cases, the oral anatomy may also include other artificial parts, such as replacements or restorations, such as any one or more of dental fillings, crowns, bridges, braces, veneers, implants, etc. The intraoral scanner records a digital impression of the dental condition by detecting the features, especially the geometric features, of the oral anatomy. These intraoral features are then recorded in the digital impression.

[0028] "Tooth condition" more specifically refers to the condition of the oral anatomical structures at the time the multiple projection images are captured or an intraoral scan is performed to provide the projection images.

[0029] "Digital impression" refers to a 3D digital model obtained via an intraoral scan of a patient's mouth. The digital model may include a representation of the patient's teeth and / or hard and / or soft tissue in and around the patient's gums. The digital impression may be usable to provide the patient with any one or more of the following: dental treatments or procedures, surgical templates, custom devices, prosthetic restorations, and orthodontic aligners.

[0030] "Intra-oral features" refers to information about intra-oral structures such as the patient's dentition and / or gums and / or other intra-oral anatomical features of the oral anatomy. Alternatively or additionally, at least a part of the intra-oral structure on which the intra-oral features are recorded may be one or more dental replacements, e.g. one or more artificial structures such as crowns, bridges, veneers, etc. Alternatively or additionally, the intra-oral structure may be a scan body (or scanbody) or other natural and / or artificial structures, as the case may be. The intra-oral features, and more importantly the surface data, are recorded to build a digital impression of the patient's oral anatomy. Usually, structures that are rigidly attached to the patient's jaw are more important to make a digital impression.

[0031] "Anatomical features" refer to information related to a particular oral anatomical structure of a patient. For example, an anatomical feature may be information related to a particular tooth or group of teeth. Thus, an anatomical feature may even refer to information related to any one or more intraoral structures, such as dentition, gums, nerve channels, tooth extraction sites, jaw bones, and condyles. Alternatively or additionally, an anatomical feature may be one or more dental replacements, e.g., one or more artificial structures, such as crowns, braces, veneers, bridges, etc. Alternatively or additionally, an anatomical feature may be a scan body (or scanbody) or other natural and / or artificial structures, in some cases, attached to the patient's jaw. Alternatively or additionally, an anatomical feature may be information related to a medical condition or condition. As a non-limiting example, a medical condition or condition may be any one or more of a tooth or bone fracture, a cavity, a radiolucency, an impaction, or any other characterizable condition of the oral anatomical structure or any part thereof. Some non-limiting examples of anatomical features are any one or more of the shape, contour or border of any one or group of teeth, their arrangement, order in any direction, the presence or absence of expected features of a given tooth type, etc. Alternatively or additionally, the anatomical feature may be a shape, region, or spot detected within an area or interest, such as a contour or border. Alternatively or additionally, the anatomical feature may be or be detected through the interrelationship between two different anatomical features, such as the border between a tooth and gums.

[0032] Preferably, the detection of anatomical features is performed using a data-driven approach. Thus, the detection of anatomical features may be performed using anatomical data-driven logic, which may either refer to a single logic providing for the detection of multiple anatomical features, or collectively refer to a group of separate data-driven logic, at least one of which is used to detect a single anatomical feature, and / or at least one of which is used to detect a group of anatomical features.

[0033] "Structured light" in the context of "structured light imaging" refers to projected light that includes a predetermined pattern. The structured light or pattern is projected onto a target surface to determine the topography of the surface. The surface topography induces distortions in the pattern projected thereon. The distortions are recorded via an imaging sensor or camera and further analyzed to extract surface information and ultimately build a digital model of the surface. It should be understood that in the context of intraoral scanning, the target surface is typically the patient's oral anatomy. As the intraoral scanner is moved along areas or locations adjacent to the surface, the digital impression of the oral anatomy is extended by stitching together surface data from different locations. Thus, the term "structured light imaging" refers to the process of projecting a structured light pattern and extracting surface data from the reflection of said pattern.

[0034] "Visual imaging" refers to a process of producing a visual or photographic image of an object. More specifically, visual imaging refers to imaging that produces an image that shows the natural or near-natural color and / or texture of the object being imaged. Thus, visual imaging can be performed using white light, or a light that allows the resulting visual image to show the natural or near-natural color and / or texture to humans. Thus, when the visual image is shown via an HMI, the user can be provided with a natural reproduction of the object, at least in terms of color and / or texture. It should be understood that for a grayscale visual image, the texture is a more central feature, whereas for a color image, at least the color should be natural, and preferably both the color and the texture.

[0035] "Human Machine Interface" or "HMI" refers to a device or system that includes at least one video unit or display screen for displaying artificially colored projected images to a user.

[0036] Thus, the HMI system may comprise a visual display or screen for displaying visual images or video. Optionally, the HMI system may also comprise an audio device. For example, the HMI system may comprise a speaker for outputting sound. The video unit may comprise a video screen, which may be active or passive, such as an LCD screen, an OLED screen, or a projection-based video system. Alternatively, or additionally, the video unit may comprise an augmented reality ("AR") device.

[0037] The HMI may further include one or more input devices for receiving user input.

[0038] "Artificially colored projected image" refers to an image in which color information is at least artificially provided via data-driven logic.

[0039] The artificially colored projection image may be an artificially colored structured light image and / or may be an artificially colored projection view of a 3D model of the tooth condition, viewable on the HMI.

[0040] "Texture data-driven logic" refers to data-driven logic used to render texture and / or color information for one or more anatomical features in a projection image. The output of the texture data-driven logic is a plurality of projection images that are at least partially artificially colored.

[0041] The texture data driven logic may apply texture and / or color information to any one or more of the detected anatomical features. The detection operation may or may not be performed by the texture data driven logic.

[0042] As will be further described, in some cases, detection of anatomical features may be performed via anatomical data-driven logic. The texture data-driven logic may, in such cases, be arranged in a serial manner with respect to the anatomical data-driven logic, whereby the output of the anatomical data-driven logic is provided directly or indirectly as an input to the texture data-driven logic.

[0043] Thus, according to one embodiment, detection of one or more anatomical features in the projection images is performed using anatomical data driven logic.

[0044] "Anatomical data-driven logic" may refer to data-driven logic that is trained or pre-trained in a supervised manner using anatomical training data that includes projection images with annotation data that specify respective anatomical features. Alternatively, or additionally, the anatomical data-driven logic may be trained in an unsupervised manner.

[0045] The anatomical data driven logic is provided with the projection image as an input. The anatomical data driven logic can provide anatomical data as an output. Thus, the anatomical data can include information related to one or more anatomical features.

[0046] According to one aspect, the anatomical data driven logic performs one or more segmentation operations to detect one or more preselected anatomical features. Alternatively or additionally, the anatomical data driven logic performs detection and / or localization operations to detect one or more preselected anatomical features.

[0047] Thus, in general, anatomical data driven logic extracts features of interest, or preselected anatomical features, from the projection images.

[0048] In some cases, the anatomical data-driven logic and the texture data-driven logic may be the same logic, e.g., combined logic that is data-driven logic for performing detection of one or more anatomical features and for rendering color information as appropriate for each of the individual images. Thus, in such cases, the projection image may be provided via an input of the combined logic, and the artificially colored projection image may be provided via an output of the combined logic.

[0049] A "segmentation operation" in this context refers to a computerized processing operation that culminates in the demarcation of at least one anatomical feature with a computer-generated feature boundary.

[0050] The anatomical data driven logic can perform a segmentation operation to detect one or more preselected anatomical features. Thus, the anatomical data driven logic can segment at least one of the projection images for detection. Thus, the anatomical data driven logic can extract attributes or one or more objects of interest from at least one of the projection images. The anatomical data driven logic can classify every pixel in each projection image into a class according to its context, such that each pixel is assigned to a specific object. A non-limiting example of a segmentation operation when applied using the present teachings can be a contour around a preselected anatomical feature, such as a specific tooth or group of teeth. Another non-limiting example is a contour around a specific medical condition. The segmented contour corresponds to the shape of each anatomical feature visible in each projection image.

[0051] There may be multiple segmented objects in any given projection image.

[0052] In addition to data-driven approaches, the segmentation operation may be complemented by any suitable segmentation algorithm, for example point and line detection, edge linking and / or analytical models using thresholding methods, histograms, adaptation, etc.

[0053] A "localization" operation in this context refers to a computerized processing operation that terminates at a more general region or location where a particular anatomical feature is located in a projection image. Localization can also result in a boundary where the anatomical feature is located, but unlike a segmentation operation, no pixel-by-pixel evaluation is performed. Thus, the localization boundary, if generated, may be broader than a segmented boundary around a given anatomical feature. Additionally or alternatively, the localization operation may terminate with annotating or tagging the particular anatomical feature, for example, at any location on the anatomical feature.

[0054] Similarly, in addition to data-driven approaches, localization operations may be supplemented by analytical models using any suitable localization algorithms, e.g., edge and shape detection, and the like.

[0055] "Combined logic" refers to logic that includes at least one of two different data-driven logics disclosed herein. The combined logic may be pre-trained in a supervised manner using an end-to-end training process. An end-to-end training process in this context refers to at least partially annotated data that includes inputs and corresponding outputs. In some cases, the combined logic may be fully or partially trained in an unsupervised manner.

[0056] "Data-driven logic" refers to logic, such as artificial intelligence, deep learning, or machine learning, that may be implemented with one or more neural networks that derive their functionality at least in part from training data. In contrast to strict or analytical logic based on programmed instructions to perform specific logical tasks, data-driven logic may allow for forming such instructions at least in part automatically using training data. The use of data-driven logic may allow for describing logical and / or mathematical relationships without solving equations. This may reduce computational power and / or improve speed. Additionally, data-driven logic may be capable of detecting patterns or instructions (e.g., in inputs or input data provided to the data-driven logic) that may not otherwise be known or may be difficult to implement as an analytical logic form.

[0057] The data driven logic may be in the form of, for example, software and / or hardware executable via one or more computing units.

[0058] In the present context, it should be understood that data-driven logic refers to trained mathematical logic that is parameterized according to the respective training data set. For example, an anatomical data-driven logic is parameterized via its respective training data to detect one or more anatomical features. Untrained logic lacks this information. Thus, untrained logic or models cannot perform the desired detection. Thus, feature engineering and training with the respective training data set allows the parameterization of the untrained logic. The result of such a training phase is a respective data-driven model, which provides, as a result of the training process, and preferably only as a result of the training process, the interrelationships and logical capabilities that are relevant to the purpose for which the respective data-driven logic is used.

[0059] The data-driven logic preferably includes or is a regression logic. In some cases, the data-driven logic may be combined with or further include an analytical logic. The analytical logic may describe the relationship between its inputs and outputs as a function, one or more mathematical expressions, or logical criteria. Thus, any of the data-driven logics described herein may be a hybrid logic. Hybrid logic refers to logic that includes a first principles part, i.e., a so-called white box described via a set of equations, e.g., mathematical expressions, and a data-driven part as described above. The data-driven part is also called a black-box logic or model. Thus, the data-driven logic may in some cases be a combination of white-box logic or analytical logic and black-box logic. In some cases, the data-driven logic may be or have a grey-box part. Grey-box logic in this context refers to logic or a model that combines a partial analytical structure with a data-driven part using training data to complete the model.

[0060] According to one embodiment, the projection images may be used to build a visualized 3D model of the tooth condition. During a dental scan, a 3D visualization of the tooth condition is often rendered on the HMI. This visualization may help the dental practitioner and the patient know the completion status of the intraoral scan. As the scan progresses, the 3D visualization is incrementally built or completed using data from the projection images. Such a 3D visualization or view may be useful in guiding the user to indicate which parts of the oral anatomy have been scanned and which areas remain or are incomplete. At the start of the scan, the 3D view may be empty or may show the contours of the dental arch. As the intraoral scan progresses, the 3D view is enhanced with dental features rendered on it. To obtain a better viewing angle, the dental practitioner may adjust the projection or viewing angle of the 3D view. Thus, it may be possible to view the 3D visualization from different angles and / or lighting conditions.

[0061] It may be advantageous to render photorealistic features in a 3D view or model, preferably with photorealistic lighting, which may help the dental practitioner to get a more realistic feedback between the scan and the resulting visualization. The applicant has realized that the present teachings can provide not only an artificially colored projection image, but also a visualized 3D model that may be used as said 3D visualization for the dental practitioner and / or patient. The present teachings may enable to provide a more realistic 3D view.

[0062] Thus, a method for visualizing a dental condition can also be provided, the method comprising: - projecting the pattern onto the dental condition by means of an intraoral scanner; providing a plurality of projected images of the dental condition via an intraoral scanner; - constructing a visualized 3D model of the dental condition using at least some of the projection images; - Rendering color information to one or more anatomical features in the visualized 3D model via machine learning to generate an at least partially artificially colored 3D model, where pattern information is removed through rendering.

[0063] Those skilled in the art will appreciate that aspects of the visualized 3D model are also novel and patentable in their own right. The rendering of color information removes artifacts in the texture of anatomical features in the visualized model that are introduced by patterns in the projected image.

[0064] The 3D data-driven logic may be separate data-driven logic, or the texture data-driven logic and the 3D data-driven logic may be in a combined form. It is also contemplated that in some cases the anatomical data-driven logic, the texture data-driven logic, and the 3D data-driven logic may be the same combined logic.

[0065] The at least partially artificially colored 3D model is preferably displayed on the same HMI. - displaying, via a human-machine interface unit, the at least partially artificially colored 3D model.

[0066] Those skilled in the art will appreciate that steps may be repeated as the visualized 3D model is constructed. For example, the method may include: - incrementally building a visualized 3D model further in response to additional projection images captured via the intraoral scanner; - incrementally rendering color information to a correspondingly incrementally constructed visualized 3D model to further construct an at least partially artificially colored 3D model.

[0067] Thus, as the scanning position of the intraoral scanner changes, the visualized 3D model is further enhanced and color information is automatically rendered accordingly.

[0068] According to one embodiment, the anatomical data driven logic is trained using anatomical training data that includes a plurality of historical projection images tagged with at least one anatomical feature.

[0069] According to one aspect, the texture data driven logic is trained using texture training data including a plurality of historical projection images tagged with anatomical data and their corresponding respective visual images including texture and / or color information, ideally the visual images are photographic images acquired during an intraoral scan as each tagged historical projection image.

[0070] According to one embodiment, the 3D data-driven logic is trained using 3D training data that includes a plurality of historical 3D visualizations and their corresponding colored and / or textured morphologies. The historical 3D visualizations include historical visualization 3D models and may optionally also include historical projection images.

[0071] It should be understood that when any two or more of the data-driven logics are combined, the combined logic is trained using the corresponding data. For example, when the combined logic includes anatomical structure detection and rendering, the combined training data may include a plurality of historical projection images and their associated visual images. The visual images include texture and / or color information. Thus, end-to-end training may be performed, thus enabling the combined logic to provide an artificially colored projection image by analyzing the projection image.

[0072] Similarly, where the combined logic performs anatomical structure detection, rendering of both projection images and visualized 3D models, the combined training data may include multiple historical projection images, their associated visual images, and their corresponding colored 3D models.

[0073] According to one embodiment, the at least partially artificially colored 3D model is displayed from a viewpoint selectable via user input. Preferably, the rendering of texture and / or color information to the visualized 3D model is performed in a perspective-dependent manner. By doing so, a more realistic representation can be provided to the user instead of an artificial looking image. The 3D data-driven logic or combined logic can thus adapt the texture and / or color rendering in a perspective-dependent manner.

[0074] Applicant has also recognized that the structured light image already contains some photographic information. This information can be exploited to provide or improve the quality of the artificially colored projected image and / or the artificially colored 3D model. Because the structured light image contains known patterns in terms of color and / or profile, one or more image correction operations can be performed on the structured light image to obtain the artificially colored projected image and / or the artificially colored 3D model.

[0075] Therefore, according to another aspect, the rendering of texture and / or color information in the projected image and / or in the visualized 3D model includes at least one image correction operation.

[0076] Image correction operations may include, for example, extracting color and / or texture information from certain portions of the projected image. For example, one or more portions of the projected image where the structured light is more neutral may be used to detect or estimate the actual color of one or more portions of the dental condition for which the scan is being performed. This may be used to determine how the projected image should be modified to obtain an artificial white light visual image or an approximation thereof. This artificial white light visual image may be used to provide a more realistic artificially colored projected image and / or an artificially colored 3D model.

[0077] Additionally or alternatively, other operations such as filtering, e.g., color filtering, inversion, averaging, edge detection, or the like, may be performed for detection, followed by image correction operations. Thus, one or more of the projection images may be analyzed, and depending on the analysis, one or more image correction operations may be performed, e.g., on some or all of the projection images, to obtain an artificially colored projection image and / or an artificially colored 3D model. As a non-limiting example, the image correction operation may be removing color-coded pattern information from at least one of the projection images. Additionally or alternatively, the image correction operation may include replacing, substituting, or modifying information, e.g., replacing a color with another and / or replacing a texture with another. As another non-limiting example, the image correction operation may involve blending the color of one or more projected images and / or the visualized 3D model with a preselected color to obtain a corrected color of a portion or the entirety of the respective projected image and / or the visualized 3D model, to obtain a more color-corrected result, i.e., a plurality of at least partially artificially colored projected images and / or an at least partially artificially colored 3D model. Similarly, texture blending may be contemplated as an additional or alternative aspect. In some embodiments, the image correction operation may be part of the rendering operation.

[0078] It should be understood that the present teachings may be particularly advantageous for intraoral scanners that do not include the means or functionality for capturing white light visual images. Thus, significant cost savings can be obtained by eliminating the associated hardware and / or software. Even for intraoral scanners that have the ability to capture white light visual images, a fast mode can be provided to speed up the scanning process by at least partially preventing the white light capture mode. More intelligent solutions can be deployed to capture white light images only when a white light image is needed and provide artificially colored projected images for the remaining parts. A non-limiting example of triggering a white light image may be when the respective data-driven logic lacks the ability to provide a photorealistic image, for example due to an abnormal part of the tooth condition that may not have been parameterized via training data that was available at the time. Another example is to calibrate certain parameters such as color tone, texture, etc., for which a quick white light image can be used as a reference for the artificially colored projected image. Similar reasoning can also be applied to artificially colored 3D models. As will be appreciated, the dental scanning process can be performed faster while providing a more realistic image to the user.

[0079] Viewed from yet another perspective, a system may also be provided that includes means for performing any of the method steps disclosed herein, or that is configured to perform any of the method steps disclosed herein.

[0080] For example, viewed from another perspective, an intraoral scanner or intraoral scanning system may be provided, comprising: - a structured light source for projecting structured light onto the tooth condition; a sensor unit for receiving reflections of structured light from intraoral surfaces and generating a plurality of projected images of the dental condition; an interface for connecting to one or more computing units, any one of the computing units comprising: - Detect one or more anatomical features; removing color-coded pattern information from at least one of the projection images to determine texture and color information of the tooth condition being scanned; - rendering texture and / or color information to one or more anatomical features in the projection images via machine learning to generate a corresponding plurality of at least partially artificially colored projection images; configured to provide, via the output interface, at least a portion of the artificially colored projected image.

[0081] The output interface may be used to provide some or all of the artificially colored projected image to an HMI. In some cases, the interface and the output interface may be the same unit. The interface and / or the output interface may be hardware units and / or software modules. Either or both of the interfaces may be in the form of or comprise an application programming interface ("API").

[0082] Any of the systems disclosed herein may include one or more computing units. Optionally, any of the systems and / or any of the computing units may be configured to operably connect to a network, for example, via one or more network interfaces and / or connection interfaces. The sensor unit may be a camera, etc. In some cases, at least one of the computing units may be part of a cloud computing service.

[0083] In some cases, the intraoral scanner or intraoral scanning system may be equipped with an HMI for displaying at least a portion of the artificially colored projected image.

[0084] Alternatively or additionally, any of the computing units may include - using at least some of the projection images to construct a visualized 3D model of the dental condition; - may be configured to render, via machine learning, color information onto one or more anatomical features within the visualized 3D model to generate an at least partially artificially colored 3D model, where pattern information is removed through rendering.

[0085] Thus, the rendering of texture and / or color information to generate the at least partially artificially colored 3D model may be performed instead of, or in addition to, the rendering of texture and / or color information to generate a corresponding plurality of at least partially artificially colored projected images.

[0086] Viewed from another perspective, there may also be provided a non-transitory computer readable storage medium storing a computer software product, or program, comprising instructions that, when executed by a suitable computing unit, or a system comprising computing units, cause any one or more of the computing units to perform any of the steps of the methods disclosed herein.

[0087] For example, from another perspective, when executed by a suitable computing unit, or a system having computing units, the method may be configured to cause any of the computing units to: - detecting one or more anatomical features in each of a plurality of projection images of a dental condition, where the projection images are provided via an intraoral scanner; - rendering, via machine learning, color information into one or more anatomical features in the projection images to generate a corresponding plurality of at least partially artificially colored projection images, where pattern information is removed through the rendering; A non-transitory computer readable storage medium storing a computer software product, or program, comprising instructions may be provided.

[0088] The software product, or a non-transitory computer-readable storage medium storing the program, may be configured to: The device may also include instructions to display, via the human-machine interface unit, at least a portion of the projected image that is artificially colored.

[0089] The computer-readable data medium or carrier includes any suitable data storage device having stored thereon one or more sets of instructions (or software) for performing any one or more of the methods disclosed herein. The instructions may even reside partially or completely within a main memory and / or within a processor during execution thereof by a computing unit, and the main memory, which may constitute a computer-readable storage medium. The instructions may even be transmitted or received over a network via a network device.

[0090] The computer program for implementing one or more of the embodiments described herein may be stored / distributed on a suitable medium, such as an optical storage medium or a solid-state medium, supplied together with or as part of another hardware, but also distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems, etc. However, the computer program may also be presented over a network, such as the World Wide Web, and can be downloaded from such a network into the working memory of a data processor.

[0091] Furthermore, a data carrier or data storage medium for making a computer program product available for downloading may also be provided, which computer program product is configured to perform a method according to any one of the methods disclosed herein.

[0092] Viewed from another perspective, there may also be provided a computing unit comprising computer program code for performing the methods disclosed herein. There may also be provided a computing unit operably coupled to a memory storage device comprising computer program code for performing any of the methods disclosed herein.

[0093] A "computing unit", "computing device", "processing unit" or "processing device" may comprise or be a processing means or computer processor, such as a microprocessor, microcontroller, etc., having one or more computer processing cores.

[0094] A "computer processor" refers to any logic circuitry configured to perform basic operations of a computer or system, and / or generally, a device configured to perform calculations or logic operations. In particular, the processing means or computer processor may be configured to process basic instructions that run the computer or system. By way of example, the processing means or computer processor may comprise at least one arithmetic logic unit ("ALU"), at least one floating point unit ("FPU"), such as a mathematical coprocessor or numeric coprocessor, a number of registers, in particular registers configured to provide operands to the ALU and to store results of operations, and memory, such as L1 and L2 cache memories. In particular, the processing means or computer processor may be a multi-core processor. In particular, the processing means or computer processor may be or comprise a central processing unit ("CPU"). The processing means or computer processor may be a complex instruction set computing ("CISC") microprocessor, a reduced instruction set computing microprocessor ("RISC"), a very long instruction word ("VLIW") microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processing means may also be one or more dedicated processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a digital signal processor (DSP), a network processor, etc. The methods, systems, and devices disclosed herein may be implemented as software in a DSP, a microcontroller, or any other side processor such as a hardware unit in an ASIC, a CPLD, or an FPGA. It should be understood that the term processing means or processor may also refer to one or more processing devices, such as a distributed system of processing devices located across multiple computer systems (such as cloud computing), and is not limited to a single device unless otherwise specified.

[0095] A "memory storage device" may refer to a device for storing information in the form of data in a suitable storage medium. Preferably, the memory storage device is a digital storage device suitable for storing information in a machine-readable digital form, e.g., digital data that is readable via a computer processor. The memory storage device may thus be realized as a digital memory storage device that is readable by a computer processor. More preferably, the memory storage device on the digital memory storage device may also be operated by a computer processor. For example, any part of the data recorded on the digital memory storage device may be written and / or erased and / or overwritten, partially or entirely, with new data by the computer processor.

[0096] A "network" as discussed herein may be any suitable type of data transmission medium, wired, wireless, or a combination thereof. A particular type of network does not limit the scope or generality of the present teachings. A network may thus refer to any suitable interconnection between at least one communication endpoint and another communication endpoint. A network may comprise one or more distribution points, routers, or other types of communication hardware. Network interconnections may be formed by physical hard wiring, optical and / or wireless radio frequency ("RF") methods. A network may specifically be or comprise a physical network made entirely or partially by hard wiring, such as an optical fiber network, or a network made entirely or partially by conductive cables, or a combination thereof. A network may comprise, at least in part, the Internet.

[0097] "Network interface" refers to a device or grouping of one or more hardware and / or software components that enables an operative connection to a network.

[0098] A "connection interface" or a "communication interface" refers to a software and / or hardware interface for establishing communication, such as the transfer or exchange of signals or data. The communication can be either wired or wireless. The connection interface is preferably based on or supports one or more communication protocols. The communication protocol connection can be a wireless protocol, e.g., a short-range communication protocol such as Bluetooth or Wi-Fi, or a long-range communication protocol such as a cellular or mobile network, e.g., a second generation cellular network ("2G"), 3G, 4G, Long Term Evolution ("LTE"), or 5G. Alternatively or additionally, the connection interface can even be based on a proprietary short-range or long-range protocol. The connection interface can support any one or more standard and / or proprietary protocols.

[0099] It will be apparent to one skilled in the art that two or more components are "operably" coupled or connected. Without limitation, this means that there may be at least one communication connection between the coupled or connected components, e.g., they may be a network interface or any suitable interface. The communication connection may be either fixed or removable. Moreover, the communication connection may be either unidirectional or bidirectional. Furthermore, the communication connection may be wired and / or wireless. In some cases, the communication connection may also be used to provide control signals.

[0100] Certain aspects of the present teachings will be discussed with reference to the accompanying drawings, which illustrate said aspects by way of example. The drawings may not be drawn to scale, as the generality of the present teachings does not depend thereon. Method and system aspects may be described together for ease of understanding. Certain specific features shown in the drawings may be logical features shown together with physical features for understanding, without affecting the generality or scope of the present teachings. [Brief description of the drawings]

[0101] [Figure 1] FIG. 1 illustrates various structured light patterns that can be used in embodiments of the present invention. [Diagram 2] FIG. 2 shows a live view image (right side) according to an embodiment of the present invention, and the live view image (left side) is a comparative prior art example (Omnicam). [Diagram 3] FIG. 3 illustrates a scanning system according to an embodiment of the present invention. [Figure 4] FIG. 4 shows a flow chart according to an embodiment of the present invention. [Diagram 5] FIG. 5 shows a scanning system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0102] According to example aspects described herein, a method, system, and computer-readable storage medium for visualizing a dental condition may be provided.

[0103] The present invention solves this problem by "inpainting" and / or "color correction," which means that algorithms (based on machine learning, also called data-driven logic) are used to complete and / or correct images with structured light patterns in order to produce live camera images without structured light patterns to be displayed to a human operator.

[0104] For this purpose, live view intraoral images during the scan are -improving the user experience of the scanning process for operators; - Reducing scan time by providing direction to the operator; -Patient communication and engagement, is primarily important to

[0105] Therefore, the result achieved does not need to be a completely accurate representation of the patient's teeth, implants, existing dental restorations, and soft tissues, as it does not serve any medical purpose. Errors or inaccuracies in this result will in no way interfere with the dental procedure being performed (Figure 3).

[0106] This is made possible using machine learning. Traditional hand-coded algorithms for repairing and / or color correcting such images are expected to fail, since human programmers cannot anticipate all situations. However, by presenting a large amount of data to a machine learning model during the training phase, the machine learning model can integrate semantic information about anatomy, lighting conditions, and other things into the model. Thus, machine learning is expected to yield live view images that are useful to human operators in ways that would not be possible using more traditional algorithms.

[0107] Restoration and / or color correction can be performed algorithmically either directly on the structured light image or directly on the reconstructed tooth surface geometry after projection.

[0108] Overall, the system consists of an intraoral camera capable of generating only structured light images (e.g., coded or uncoded dots, lines, or other), a 3D reconstruction algorithm running on any suitable computing device that generates a 3D surface model of the dental situation based on these structured light patterns, and a restoration / texture completion algorithm for generating an image similar to a white light illumination image of the same area as the corresponding structured light image. The algorithm can be based on machine learning. Additionally, a device for displaying the algorithmically generated images to a human operator during the intraoral scanning procedure.

[0109] Possible implementations of the present invention rely on an extensive body of existing literature on machine learning-based literature on inpainting (e.g., 0,0,0). Inpainting refers to the algorithmic process of filling in missing areas of an image. For color correction, a deep neural network model is appropriate that takes color-encoded data as input and outputs a color-corrected image.

[0110] Known machine learning methods such as those described in [1] to [3] fill designated image areas with missing color information with artificial color information. The method disclosed herein is specifically directed to application to dental conditions using dental intraoral scanner projection patterns. Pixels of the projection image affected by the pattern are automatically located and adapted with respect to their color information. The method is able to cope with specific tooth patterns, materials, and lighting conditions.

[0111] The present invention comprises the following process:

[0112] First, to collect a large number of image pairs to train the machine learning model, where the first image of each pair depicts the structured light image used for dental reconstruction, and the second image of each pair depicts the true sought white light illumination image of the same intraoral area as the first image. This is called the "training data set." To generate this training data set, several possibilities exist, including (but are not limited to) using ray tracing (or simulated ray tracing) algorithms to create several renderings of image pairs (structured / colored light, white light, . . .) of available 3D dental models. These 3D models can be created using other existing high quality cameras, such as the Omnicam by Dentsply Sirona or Primescan.

[0113] Second, to create a deep learning model that is used to repair missing textures between areas illuminated by structured light projections ("texture completion"), or to correct color information when color-based projections are used. Additionally, the model can also perform color correction. The parameters of that model are adjusted to minimize a measure ("loss function") between the predicted results and the (true or simulated) white-light illuminated images in the training dataset. This is called the "training process."

[0114] Third, build a camera that can only generate structured light images. Use the structured light images for 3D reconstruction. In parallel or sequentially, feed those same images into a restoration model obtained from a training process. The model outputs an approximation of the actual appearance of the intra-oral situation. Display that output image to the user during the scan.

[0115] Another reasonable implementation of the present invention is as follows: - Use of machine learning models to remove only color information from color-coding patterns such as Omnicam (Figure 2, left). In this embodiment, no inpainting is required; - Use of image stacks to feed more information to the inpainting model: for example, take 3, 5 or 7 structured light images. Feed the stacks (3D data) to an inpainting algorithm (machine learning model) to generate, for example, one white light (live view) image representing the position of the central image in the stack of structured light images. In this way, the inpainting model has more information on which to base its predictions. The frequency of the live preview images is lower than the acquisition frequency of the scanner; - Use of image sequences to feed more information to the inpainting model. To perform this type of prediction, a time-series neural network model, e.g. LSTM, can be used.

[0116] The following references are given:

[0117] [1] Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros, “Context Encoders: Feature Learning by Inpainting,” Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2016.

[0118] [2]Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li, "High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis," Proc. International Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[0119] [3] Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa, “Globally and Locally Consistent Image Completion,” ACM Trans. on Graphics, Vol. 36, No. 4, Article 107, Publication date: July 2017.

[0120] The advantage of the present invention is that it allows to build an intraoral dental scanner that is smaller and cheaper to build. The disadvantage is the fact that the calculated texture only approximates the patient's true dental condition. This may be acceptable in some situations, since this information is not important for the dental procedure.

[0121] In particular, machine learning represents a suitable tool for implementing the present invention. Intraoral scan image data can be very heterogeneous and difficult to process due to many issues such as limited light, scattering when there is too much light, blockage through soft tissues, blockage through various liquids such as saliva and blood, etc. Machine learning methods can fill in the missing information and infer the dental anatomical structures based on the training data.

[0122] The present invention attenuates the effect of scattering in the presence of too much light by applying a pattern that introduces little light into the scan of the dental condition. This includes dots, lines, or other geometric patterns with a high percentage of light-blocked areas, such that the percentage of pixels in the projected image that show darkened or reduced light intensity color information is 80% or more. Figure 1 shows two examples of line and dot patterns. In a preferred embodiment of the present invention, the color information captured in these darkened pixels may be reduced to image noise or close to image noise. In another embodiment, the darkened pixels contain distorted color information of low intensity. The darkened pixels may contain scattered light information from surrounding anatomical structures or from blurring of the projected pattern.

[0123] Furthermore, the present invention includes removing color-coded patterns (e.g., parallel lines of equal or varying width) such that more than 50% of the pixels in the projection image show distorted color information of the anatomical features. The left side of Figure 2 shows an example of a tooth condition with such a color-coded pattern that covers up to 100% of the pixels in the projection image. The layered color-coded pattern distorts the natural color of the pixels.

[0124] The machine learning is trained with tooth data covering dental anatomical structures and dental materials, including light reflection, light scattering, and typical color characteristics. This allows the machine learning to estimate natural-looking colors for distorted or darkened image pixels. The input data for the machine learning may include a sequence of projection images to include information of other projection images. The input data for the machine learning may include a current estimate of the 3D model. These additional data extend the power of the machine learning. The machine learning is trained to render artificial color information for image pixels with distorted or darkened color information. This significantly removes or reduces patterns in the projection images. The step of rendering artificial color information corresponds to an image restoration or image correction step.

[0125] When the layered color-coded pattern covers up to 100% of the pixels in the projection image, no pixels with natural color information remain, and the color information of pixels with distorted color information cannot be derived directly from other pixels. Machine learning can estimate natural-looking colors for these pixels because its training data includes the tooth condition. If the training data includes the specific pattern applied, the resulting color information can be improved.

[0126] If a pattern reduces the light intensity of all pixels in a larger image area to image noise, there may not be sufficient reliable color information in neighboring pixels to directly derive natural-looking color information for the pixels in this image area. Machine learning can estimate natural-looking colors because its training data includes the condition of the teeth. The resulting color information can be improved if the machine learning input data includes a sequence of projection images and / or a current estimate of the 3D model. Machine learning can also be trained to derive color information from scattered or reflected light.

[0127] One possible machine learning method applicable to the present invention may be a neural network. Considering the case where the machine learning method takes a single projection image as input and produces an artificially colored image, suitable network architectures include, but are not limited to, convolutional neural networks that implement an encoder-decoder architecture. Such networks later apply layers of convolutional filters with learnable parameters to the input image, usually reducing the resolution but increasing the per-pixel feature descriptiveness. These layers are followed by a deconvolution layer, resulting in an image with the same resolution as the input image, but with the combined filters of the previous layers applied to make it more similar to the desired output. This is a common architecture for related image processing networks, but several extensions have been presented for improvement. These can include, but are not limited to, residual connections, regularization techniques like dropout layers, variants of activation functions, and pooling layers. The last layer of the network in the training configuration represents a loss function designed to be minimized when the output image has the desired properties. In supervised learning, the training data is represented by pairs of inputs and desired outputs, and the loss function is a similarity calculated for the network output given the training inputs and the desired outputs. Training is performed by backpropagation and gradient descent, utilizing the chain rule in deriving the loss function for the trainable parameters.

[0128] Considering machine learning methods that rely on supervised learning, the problem of training set generation arises. For methods that generate pattern-free images from projected images, this means providing suitable input-output pairs. Several ways of doing this are possible. For example, rendering methods are known to those skilled in the art that use any method and device that provides a realistically colored 3D model of an intraoral situation, possibly using an intraoral scanner, and specifying a suitable illumination source for the rendering method / framework to provide a photorealistic rendering of a view of said model. Photorealistic in this context means (almost) indistinguishable by a human observer from an image generated by a physical camera device. Such methods may include, but are not limited to, ray tracing, possibly with subsurface scattering or radiosity. This allows generating images of the situation from any viewpoint, and the light source may be set to emulate the projection of a structured light pattern onto the model or to emulate the white light generated by a light-emitting diode. Pairs of images generated by a rendering method set with two different light sources can directly serve as training pairs.

[0129] Although the above techniques are possible, it is usually difficult to rely solely on artificial training data to train a machine learning method to operate on data provided by a physical camera. Another way to generate training data is to use a training camera operating in a conventional mode alternating between structured and white light exposure. The same structured light pattern can be used as for the camera without white light exposure. In this way, a set of projection images with structured light exposure and a set of white light images are generated. However, usually none of these images are captured with a camera that is in exactly the same position and orientation relative to the intraoral situation. Since it is desirable for a machine learning method to provide an image that would have been seen from the same position and orientation as the projection images it operates on, training output images must be provided accordingly by applying a pre-processing step to the white light exposure images captured by the training camera. Several ways to mitigate this problem are possible. A simple approach is to use a number of white light exposure images that are close to the projection images and to use 2D registration and filter techniques to calculate a fusion image from the white light images that resembles the image that would have been observed when capturing from the position and orientation of the projection images. These 2D alignment techniques may include, but are not limited to, phase correlation, keypoint-based methods and intensity-based methods, as well as appropriate pre-processing steps such as edge detection. Another possibility is to use view synthesis methods to generate an artificial white light exposure image at the position and orientation of the patterned projected image. View synthesis methods provide a means to provide a rendering of a scene as seen from a specified position and orientation, giving a number of 2d images of the scene that do not necessarily include images captured at said positions and orientations. Neural network architectures have been proposed that can be trained on a number of images and generate renderings given orientation and position at inference. For example, the Neural Radiance Field ("NeRF") architecture is suitable for the task.

[0130] FIG. 5 shows a block diagram of an intraoral scanning system. The system includes an intraoral scanner 104 shown in FIG. 5 that is used to scan the dental condition 106 of a patient 108. The intraoral scanner 104 is operably connected to a computing module 110 having at least one computing unit. In some cases, the computing module 110 may be at least partially located within the intraoral scanner 104. In some cases, the computing module 110 may be at least partially part of a cloud service 112. Thus, the computing module 110 may be a local device at the location of the intraoral scanner 104 and / or at least partially a remotely located device or system, such as one or more cloud services 112 and / or a remote server. The computing module 110 is also operably connected to a memory storage 114 that may be at least partially part of the cloud service 112. In this case, the memory storage 114 is shown as part of the cloud service 112. The intraoral scanner 104 may be connected to an external computing module 110 via a connection interface (not explicitly shown in FIG. 5). Any local device, such as the intraoral scanner 104 and / or the local computing module 110, may connect to the cloud service 112 via one or more networks 116. A network interface (not explicitly shown in FIG. 1 ) may be used to connect the local device to a remote device, such as the cloud service 112. In some cases, the network interface and the connection interface may be the same unit.

[0131] The computing module 110 is operatively connected to a human machine interface 118 ("HMI"), shown in FIG. 5 as a computer screen. The connection between the computing module 110 and the HMI 118 may be via an output interface (not explicitly shown in FIG. 5), which may be, for example, a computer port or bus, e.g., a display port, HDMI, USB, or the like. However, the output interface may even be an API for providing data to one or more computing units on the HMI 118 side. In some cases, the output interface may be part of a network interface and / or a connection interface. Thus, it is possible that the three interfaces are the same device.

[0132] The intraoral scanner 104 includes a structured light source (not explicitly shown in FIG. 5 ) for projecting structured light onto the tooth condition 106. The intraoral scanner 104 also includes a sensor unit (not explicitly shown in FIG. 5 ) for receiving reflections of the structured light from the surfaces of the tooth condition 106. In response to receiving the reflections, multiple projection images of the tooth condition 106 are generated. Thus, during the scan, the intraoral scanner 104 is operated to capture multiple projection images of the tooth condition 106. The projection images may be in the form of a data stream. These projection images are provided to a computing module 110. The computing module 110 is configured to detect one or more anatomical features in each of the projection images. Advantageously, the detection of the one or more anatomical features is performed using anatomical data driven logic.

[0133] The computing module 110 uses texture data driven logic for rendering texture and / or color information 120 to one or more anatomical features in the projection images to generate a corresponding at least partially artificially colored projection image 122. The texture and / or color information 120 may be added, for example, by editing the projection images and / or by adding one or more layers to the respective projection images. Thus, the at least partially artificially colored projection image 122 is also preferably in the form of a data stream that is displayed as a video on the HMI 118. Thus, photorealistic images or videos can be provided to the user or patient 108 without the need to capture white light images.

[0134] Alternatively or in addition to providing the at least partially artificially colored projected image 122, an at least partially artificially colored 3D model 124 may be provided. The HMI 118 may in this manner present a more photorealistic visual representation of the dental condition 106 as the intraoral scan progresses. The at least partially artificially colored 3D model 124 may also be provided with texture and / or color information that may correspond to the viewing angle or projected view of the model 124 on the screen 118.

[0135] 4 shows a flowchart 200 that may be implemented as one or more routines executed by one or more computing units. At least certain relevant system aspects will also become apparent from the following. The flowchart 200 may be implemented as one or more routines in an intraoral scanner and / or an intraoral scanning system.

[0136] At block 202, a plurality of projection images of the patient's dental condition are provided. The projection images may be provided directly or indirectly via an intraoral scanner. Thus, an intraoral scanner is used to provide projection images related to the patient's dental condition.

[0137] In block 204, one or more anatomical features are detected in each of the projection images. In particular, the dental features may vary from projection image to projection image, as the projection images change as the scan progresses to different parts of the dental condition. The detection is advantageously performed via anatomical data driven logic. In block 206, texture and / or color information is rendered to one or more anatomical features in the projection images via texture data driven logic. In this manner, a plurality of artificially colored projection images, at least partially artificially colored, are generated. The texture data driven logic and the anatomical data driven logic may be different logic, or they may be the same logic, e.g., combined logic. Optionally, in block 208, a visualized 3D model of the dental condition is constructed or rendered using at least a portion of the projection images. Also, optionally, in block 210, texture and / or color information is rendered to one or more anatomical features in the visualized 3D model via 3D data driven logic to generate an at least partially artificially colored 3D model.

[0138] It should be understood that the routine may be implemented using block 206 while excluding blocks 208 and 210, or they may be combined in any suitable processing order, or the routine may be implemented using blocks 208 and 210 while excluding block 206.

[0139] In either case, the plurality of at least partially artificially colored projected images and / or the at least partially artificially colored 3D model may be provided to the HMI, for example via an output interface.

[0140] The steps of the method may be performed in the order as enumerated and shown in the examples or aspects. However, it should be noted that under certain circumstances, a different order may be possible. Furthermore, it is also possible to perform one or more of the steps of the method once or repeatedly. These steps may be repeated at regular or irregular time periods. Furthermore, two or more of the steps of the method may be performed simultaneously or overlapping in time, particularly when some or more of the steps of the method are performed repeatedly. The method may comprise further steps not enumerated.

[0141] The word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single processing means, processor or controller or other similar unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage. Any different signs in the claims should not be interpreted as limiting the scope.

[0142] Furthermore, it should be noted that in this disclosure, the terms "at least one," "one or more," or similar expressions indicating that a feature or element may be present one or more times may typically be used only once when introducing each feature or element. Thus, in some cases, unless expressly stated otherwise, when referring to each feature or element, the expressions "at least one" or "one or more" may not be repeated, despite the fact that each feature or element may be present one or more times.

[0143] Furthermore, the terms "preferably", "more preferably", "particularly", "more particularly", "particularly", "more particularly" or similar terms are used in conjunction with optional features without limiting the possibilities of substitution. From this, any features introduced by these terms are optional features and are not intended to limit the scope of the claims in any way. The present teachings can be implemented by using alternative features, as the skilled artisan will recognize. Similarly, features introduced by "according to one embodiment" or similar expressions are intended to be optional features, without any limitations on the alternative forms of the present teachings, without any limitations on the scope of the present teachings, and without any limitations on the possibilities of combining the features so introduced with other optional or non-optional features of the present teachings.

[0144] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0145] Various examples have been disclosed above for methods, systems, devices, uses, software programs, and computing units with computer program codes for implementing the methods disclosed herein. For example, a method for visualizing a dental condition is disclosed, comprising providing a plurality of projection images of the dental condition, detecting one or more anatomical features in each of the projection images, and rendering texture and / or color information to the one or more anatomical features in the projection images to generate a corresponding plurality of at least partially artificially colored projection images via texture data-driven logic. The present teachings also relate to systems, intraoral scanners, software products, and storage media. However, those skilled in the art will understand that changes and modifications can be made to the examples without departing from the spirit and scope of the appended claims and their equivalents. It will be further recognized that aspects from the method and product embodiments discussed herein can be freely combined.

Claims

1. 1. A computer-implemented method for visualizing a dental condition, comprising: - projecting a pattern onto said dental condition by means of an intraoral scanner; - providing, via said intraoral scanner, a number of projected images of reflection patterns from said dental condition; - detecting one or more anatomical features in each of said projection images; - rendering color information via machine learning methods for the one or more anatomical features in the plurality of projection images to generate at least partially artificially colored projection images, wherein the patterns are removed through the rendering.

2. The computer-implemented method of claim 1 , wherein the machine learning method is trained on projection images as input and produces pattern-free images as output.

3. 3. The computer-implemented method of claim 2, wherein the machine learning method is implemented through a convolutional neural network following an encoder-decoder architecture.

4. 3. The computer-implemented method of claim 2, wherein training pairs for input and output of the machine learning method are generated by photorealistic rendering of a 3D model including geometry and surface color information.

5. 3. The computer-implemented method of claim 2, wherein the input and output pairs of the machine learning method are provided by a movable camera following an optical configuration that captures a projected image and a white light exposure image at different times.

6. 6. The computer-implemented method of claim 5, wherein different camera positions and orientations due to motion between capture of different images are compensated for by interpolating positions and orientations between projected images, which are obtained by 3D registration of geometry derived from the projected images.

7. The computer-implemented method of claim 5 , wherein different camera positions and orientations due to motion between capture of different images are compensated for by 2D registration of the white light exposure image to the projected image.

8. 8. The computer-implemented method of claim 7, wherein the images are pre-processed to enable registration of a projected image with a white light exposure image.

9. 6. The computer-implemented method of claim 5, wherein different camera positions and orientations due to motion between capture of different images are compensated for by using a projection pattern and / or white light exposure to generate a synthetic view.

10. A synthetic view is generated by a view synthesis network, which is trained on either the entire set of projection images of the intraoral scene or on white exposure images of the intraoral scene; when generating synthetic views of a projection image, said synthetic views are generated from the position and orientation of the white-balanced image, forming a training pair with said white-balanced image, respectively; or The computer-implemented method of claim 9, wherein when generating a synthetic view of a white light exposure image, the synthetic view is generated from a position and orientation of the projection image, each of which forms a training pair with the projection image.

11. The computer-implemented method of claim 10 , wherein the network employs principles of neural radiance fields ("NeRF").

12. The computer-implemented method of claim 1 , wherein pattern information distorts more than 50% of the per-pixel color information of the one or more anatomical features in the originally provided projection image.

13. 13. The computer-implemented method of claim 1, wherein the pattern information darkens, i.e. reduces the intensity of, more than 80% of the per-pixel color information of the one or more anatomical features in the originally provided projection image.

14. - constructing a visualized 3D model of the dental condition using at least some of said plurality of projection images; - rendering color information for one or more anatomical features in the visualized 3D model via machine learning methods to generate an at least partially artificially colored 3D model, wherein pattern information is removed through the rendering.

15. A computer-implemented method according to any one of claims 1 to 14, further comprising displaying at least a part of the partially artificially coloured projected image and / or the at least partially artificially coloured 3D model via a human-machine interface unit.

16. - incrementally building the visualized 3D model further in response to further projection images captured via the intraoral scanner; - incrementally rendering color information onto a correspondingly incrementally constructed visualized 3D model to further construct an at least partially artificially colored 3D model, wherein pattern information is removed throughout said rendering.

17. 17. A computer-implemented method according to claim 1, wherein an at least partially artificially coloured 3D model is displayed, and the rendering of colour information in the visualized 3D model is performed depending on the perspective.

18. The computer-implemented method of claim 1 , wherein the rendering of the color information in the projected image and / or in the visualized 3D model involves at least one image correction operation.

19. 19. The computer-implemented method of claim 1, wherein the machine learning method uses a sequence of projected images to perform the rendering of color information.

20. A non-transitory computer readable storage medium storing a computer software product, or a program, comprising instructions which, when executed by one or more computing units, cause any of the computing units to perform the steps of any one of claims 1 to 19 of the above method.

21. A system comprising means for carrying out the steps of any of the above method claims 1 to 19.

22. 1. An intraoral scanning system comprising: - a structured light source for projecting structured light onto the tooth condition; a sensor unit for receiving reflections of said structured light from intraoral surfaces and generating a plurality of projected images of the dental condition; one or more computing units, any of said computing units - detecting one or more anatomical features in said projection images; configured to render, via machine learning, color information for one or more anatomical features in said projection images to generate a plurality of at least partially artificially colored projection images, where pattern information is removed through said rendering; - a human machine interface unit for displaying at least a part of said artificially colored projected image.

Images