System and method for intraoral scanning

Abstract

The present disclosure relates to an intraoral scanning system (102) configured to generate an overlay of associated 2D internal geometric features on a 3D surface model of a dental object. The system is configured to receive visible light information (128) and IR information (130) from one or more sensors; generate a 3D surface model (118) of the dental object (146) based on the visible light information; generate a plurality of 2D internal feature images (302, 306, 310, 314) based on the visible light information and the IR information; process the plurality of 2D internal feature images to associate 2D internal geometric features of the dental object with at least one reference frame (712) of the dental object in the 3D surface model; and output an overlay (120) of the associated 2D internal geometric features of the dental object on the 3D surface model. The plurality of 2D internal feature images are indicative of 2D internal geometric features of the dental object.

Description

Technical Field

[0001] Exemplary embodiments of this disclosure generally relate to intraoral scan registration, and more specifically, to intraoral scan systems and methods for intraoral scans to generate three-dimensional internal geometry of teeth. Background Technology

[0002] An intraoral scanner is an electronic device that can be used, for example, to capture digital images of a subject's oral cavity. In one example, an intraoral scanner may include a light source that projects light onto the object being scanned, such as teeth, gums, and other intraoral structures within the subject's oral cavity. In some cases, computer-aided design processes can be used to create virtual 3D models of a subject's teeth using digital images captured by an intraoral scanner. The digital images of the teeth are imported into a computer-aided design (CAD) program, which creates a final virtual 3D model of the subject's dental arch.

[0003] Intraoral scanners are typically used for examinations or treatments within a patient's oral cavity. Using an intraoral scanner eliminates the need for conventional impression materials and plaster models, streamlining dental procedures and reducing discomfort for dentists. However, a virtual 3D model of the dentition will only provide surface information related to the patient's teeth or the surface of the dentition. In other words, a conventional intraoral scanner may not be suitable for examining the internal structures of teeth, such as the enamel and dentin structure within each tooth. In particular, an intraoral scanner may not be able to determine the overall internal composition of the teeth within the patient's oral cavity.

[0004] Therefore, 3D models of the surface of a patient's teeth, generated using digital images created by a conventional intraoral scanner, are unsuitable for detecting abnormal defects within the patient's teeth. For example, a 3D model will not provide any information related to caries, cracks, and / or bacterial development within the enamel and subdental matrix; bleeding or any other damage within the enamel and subdental matrix; and / or deep cracks or marginal lines, or other errors in prepared dental prostheses (e.g., crowns, bridges, implants, inlays, deposits, etc.). In particular, for effective patient treatment and to ensure large-scale restorations of natural teeth or prosthetic implants, combining information about the internal structure of the natural teeth or prosthetic implants with surface information is crucial.

[0005] In some cases, conventional methods can be used to construct 3D models of teeth by projecting 2D images of teeth onto a 3D model. For example, segmentation and stitching techniques can be used to segment white light images and place them onto a 3D model counterpart to create a 3D model of the object's teeth. In one example, a traditional method involves using an intraoral scanner to capture a large number of images from various perspectives to construct a 3D model of the teeth using white light images. However, conventional methods for generating 3D models of teeth have several drawbacks.

[0006] In one example, conventional methods cannot reliably eliminate the different compositions of the tooth's internal geometry in a 3D model. Furthermore, conventional methods require a large number of images captured from numerous viewpoints because reliable scattering coefficients can only be determined when multiple viewpoints are covered. This increases the computational load and causes delays in generating the 3D model. Consequently, operations performed on such a tooth 3D model may not be real-time. Additionally, 3D models generated using conventional methods store the captured tooth color on the surface of the 3D model. Therefore, projecting images of the tooth's subsurface structure onto the 3D model to reveal its internal composition when viewed from different angles can easily result in incorrect displacement.

[0007] Therefore, improved intraoral scanning systems and methods are needed to overcome the shortcomings of traditional methods used to generate 3D models of tooth structures. Summary of the Invention

[0008] An intraoral scanning system, method, and computer-programmable product are provided for generating an overlay of associated 2D internal geometric features on a 3D surface model of a tooth using a handheld intraoral scanner. The handheld intraoral scanner includes sensors for detecting infrared (IR) or near-infrared (NIR) light and visible light.

[0009] Some embodiments are based on the understanding that visualization of IR or NIR image information of a tooth structure can be used to create a 3D model of the tooth structure that provides internal geometric information. In this respect, the IR or NIR image information includes sub-surface information of the tooth structure. When projected onto a 3D surface model, the IR or NIR image information creates a 3D model with internal geometric information of the tooth structure.

[0010] Some implementations are based on the understanding that if a 3D surface model is moved (e.g., rotated about an axis) to be viewed from different directions, subsurface structures within the 3D model of a dental structure may shift. In other words, when the 3D model is moved to be viewed from different directions, the IR or NIR images projected onto the 3D surface model may appear in incorrect positions. As a result, such a conventional 3D model will not provide accurate subsurface structure information for the tooth structure. This can lead to incorrect examination, diagnosis, and treatment of the subject.

[0011] The purpose of this disclosure is to provide a technique for accurately superimposing 2D internal geometric features of a tooth structure (e.g., a tooth) onto a 3D surface model of the tooth.

[0012] In one aspect, an intraoral scanning system is provided, configured to generate a 3D model for one or more teeth. The intraoral scanning system includes a handheld intraoral scanner configured to operate in conjunction with one or more sensors to detect infrared (IR) light and visible light. The one or more sensors include image sensors. The intraoral scanning system includes one or more processors operatively connected to the handheld intraoral scanner. The one or more processors are configured to: receive visible light information and IR information from the one or more sensors; generate a three-dimensional (3D) surface model of the tooth from the one or more teeth based on the visible light information; and generate a plurality of two-dimensional (2D) internal feature images based on the visible light information and IR information. The plurality of 2D internal feature images indicate 2D internal geometric features of the tooth. The one or more processors are configured to process the plurality of 2D internal feature images to associate the 2D internal geometric features of the tooth with at least one reference frame of the tooth in the 3D surface model, and output a superposition of the associated 2D internal geometric features of the tooth on the 3D surface model.

[0013] According to some exemplary embodiments, at least one reference frame of the tooth is perpendicular to two or more planes of the tooth in the 3D surface model.

[0014] According to some exemplary embodiments, each of two or more planes of the tooth in the 3D surface model includes at least a first plane and a second plane. In one example, the first plane and the second plane are aligned with the buccal-lingual plane and the mesiodistal-mesiodistal plane, respectively.

[0015] According to some exemplary embodiments, two or more planes of the tooth body include at least one of the occlusal plane, buccal plane, lingual plane, mesial plane, distal plane, or labial plane.

[0016] According to some exemplary embodiments, in order to associate the 2D internal geometry features of a tooth with at least one reference frame of the tooth in a 3D surface model, one or more processors are further configured to: align the position of a template object model with the position of the tooth in the 3D surface model, and associate the position of each of a plurality of 2D internal feature images of the tooth with the position of the tooth in the 3D surface model. The template object model includes a 3D coordinate system indicating the orientation of the tooth having the 3D surface model. For example, the order in which visible light information of the 3D surface model of the tooth is received is within a predetermined range from the order in which IR information of the plurality of 2D internal feature images is received. The one or more processors are also configured to associate the viewing orientation of each of the plurality of 2D internal feature images of the tooth with the 3D coordinate system of the template object model of the tooth.

[0017] According to some exemplary embodiments, the one or more processors are further configured to connect multiple 2D internal feature images of the tooth to generate one or more 2D internal geometric panoramic images of the tooth based on the viewing direction; and to overlay one or more 2D internal geometric panoramic images on a 3D surface model.

[0018] According to some exemplary embodiments, the coordinate axes of the 3D coordinate system are aligned with the occlusal reference frame of at least one reference frame of the tooth, and wherein the occlusal reference frame is perpendicular to the occlusal plane of the tooth in the 3D surface model.

[0019] According to some exemplary embodiments, the template object model has a corresponding tooth type, wherein the tooth type of the template object model is at least one of central incisor, lateral incisor, canine, first premolar, second premolar, first molar, second molar, or third molar.

[0020] According to some exemplary embodiments, the one or more processors are configured to: estimate one or more object poses of the internal geometry of the tooth using a trained machine learning model based on a plurality of 2D internal feature images and a 3D surface model; determine one or more intermediate poses of the tooth based on smooth interpolation between two object poses indicated by a pair of adjacent images from the plurality of 2D internal feature images; and overlay the plurality of 2D internal feature images of the tooth onto the 3D surface model based on the one or more intermediate poses and one or more orientations defined by the object poses of the one or more internal geometries.

[0021] According to some exemplary embodiments, the one or more processors are configured to: generate an offset surface of a 3D surface model of the tooth, such that the offset surface is within a constant distance field from the tooth; superimpose a plurality of 2D internal feature images on the offset surface based on one or more orientations defined by at least one of one or more object poses or one or more intermediate poses of the tooth; and display the superposition of one or more intermediate poses of the tooth with the offset surface. Furthermore, the superposition can be performed based on the intersection points between points on the offset surface and rays extending from the tooth toward the offset surface in a first direction.

[0022] According to some exemplary embodiments, the one or more processors are configured to: update the pose matrix of each of one or more intermediate poses of the tooth to map a superposition with updated viewpoints; and based on the updated pose matrix, generate an updated superposition of associated 2D internal geometric features of the tooth on a 3D surface model for the updated viewpoints.

[0023] According to some exemplary embodiments, one or more processors are configured to: generate an object mask using visible light information captured from a scanning location; apply the object mask to a plurality of 2D internal feature images to segment at least a portion of the plurality of 2D internal feature images indicating non-object information; and remove the segmented portions of the plurality of 2D internal feature images indicating non-object information. The scanning location corresponds to the location of one or more sensors.

[0024] According to some exemplary embodiments, multiple 2D internal feature images include synthetic 2D internal geometric features of the tooth.

[0025] On the other hand, a method is provided for generating an overlay of associated 2D internal geometric features on a 3D surface model of one or more teeth. The method is implemented using an intraoral scanning system comprising a handheld intraoral scanner and one or more processors configured to operate with one or more sensors to detect infrared (IR) light and visible light, the one or more processors being operatively connected to the handheld intraoral scanner. The method includes: receiving visible light information and IR information from the one or more sensors; generating a three-dimensional (3D) surface model of the teeth from the one or more teeth based on the visible light information; generating a plurality of two-dimensional (2D) internal feature images based on the visible light information and IR information; processing the plurality of 2D internal feature images to associate the 2D internal geometric features of the teeth with at least one reference frame of the teeth in the 3D surface model; and outputting an overlay of associated 2D internal geometric features of the teeth on the 3D surface model. In one example, the plurality of 2D internal feature images indicate the 2D internal geometric features of the teeth.

[0026] In another aspect, a computer-programmable product is provided. The computer-programmable product includes a non-transitory computer-readable medium storing computer-executable instructions thereon, which, when executed by processing circuitry, cause the processing circuitry to perform operations. The operations include: receiving visible light information and IR information from one or more sensors; generating a three-dimensional (3D) surface model of a tooth from one or more teeth based on the visible light information; generating a plurality of two-dimensional (2D) internal feature images based on the visible light information and IR information; processing the plurality of 2D internal feature images to associate 2D internal geometric features of the tooth with at least one reference frame of the tooth in the 3D surface model; and outputting a superposition of the associated 2D internal geometric features of the tooth on the 3D surface model. In one example, the plurality of 2D internal feature images indicate the 2D internal geometric features of the tooth.

[0027] The foregoing summary is illustrative only and is not intended to be limiting in any way. Other aspects, embodiments, and features will become apparent from the accompanying drawings and the detailed description below, in addition to the illustrative aspects, embodiments, and features described above.

[0028] Effects of the present invention

[0029] According to the present invention, an intraoral scanning system, method, and computer-programmable product are provided. One object of this disclosure is to provide a precise three-dimensional representation of the tooth with its subsurface structures.

[0030] Conventional systems may include intraoral scanners, which capture two-dimensional (2D) intraoral scans of a patient's teeth. However, conventional intraoral scanners have limited processing capabilities and can only capture 2D intraoral scans and / or 3D models of the surface of the subject's teeth. However, a 3D surface model of the teeth may not be able to detect the internal structures of the teeth, such as the structure of the enamel and dentin within the subject's teeth. As a result, diseases or abnormalities within the internal structures of the teeth may not be identified unless the disease has grown to the tooth surface or the dentist uses other devices such as X-ray machines, which are ionizing radiation that can cause cytopathic effects on human cells upon exposure and should be avoided. For example, unless the bone or tooth surface is affected, a conventional intraoral scanner may not be able to detect early-onset abnormalities or diseases occurring inside the teeth, such as dentin erosion, enamel erosion, internal cracks or cavities, bacterial growth, etc. Due to the delayed diagnosis of diseases or abnormalities inside the teeth, irreparable damage may occur within the teeth, leading to the need for surgical removal. This will cause significant discomfort to the patient.

[0031] Furthermore, traditional intraoral scanners may fail to detect abnormalities occurring within dental prostheses. Identification of any abnormalities occurring within dental prostheses is crucial for ensuring their longevity. Therefore, combining information related to the internal geometry of the tooth or dental prosthesis with surface information is essential for early diagnosis, effective treatment, and ensuring large-scale restorations with natural teeth or prostheses.

[0032] In some conventional methods, the internal structure of a tooth can be visualized using intraoral scans. Conventional methods for generating the internal structure of a tooth may include visualizing IR or NIR image information on a 3D surface model of the tooth as 2D IR image information. Typically, the captured colors of an intraoral scan of the oral cavity containing the tooth are stored on the surface of a 3D surface model of the tooth. Specifically, the color values ​​of the intraoral scan are associated with portions of the surface mesh forming the 3D surface model of the tooth. The 2D IR image information, indicating internal structure or sub-surface information, is then projected onto the 3D surface model.

[0033] However, projecting 2D IR image information onto a 3D mesh alone is unreliable for capturing information about the internal structure or subsurface information of a tooth. For example, due to overlay and / or missing data, and different angles from which the IR image is captured, the 2D IR image information projected onto the 3D mesh, or the 3D surface model, may not provide accurate details of the internal structure. In particular, when 2D IR image information is projected onto a 3D surface model, if the 3D model is moved, the internal structure of the subsurface structure information may be displaced. For example, examiners such as dentists and / or medical practitioners may have to move the 3D model of the tooth to obtain knowledge of the surface and internal structure of the tooth from various directions and / or orientations. However, any movement of the 3D model can cause displacement of the projected 2D IR image information, resulting in inaccurate or incorrect projection of internal structural information. Therefore, such displacement of 2D IR image information due to changes in the viewing angle of the 3D model is undesirable and may hinder accurate diagnosis and / or treatment. In some cases, displacement of 2D IR image information may bring incorrect information about the tooth, potentially leading to incorrect diagnosis and / or treatment.

[0034] Furthermore, this processing of a large number of 2D IR images corresponding to the teeth would be processing-intensive. It would likely require significant computing power, which would increase the size and cost of the intraoral scanning system.

[0035] To this end, the intraoral scanning system disclosed herein includes a handheld intraoral scanner and one or more processors for generating a superposition of 2D IR information on a 3D surface model, ensuring accurate representation of the 3D model of the tooth even from different viewing angles. In one embodiment, the superposition and subsequent output can be performed in real time or as a post-data acquisition processing step. The intraoral scanning system can provide enhanced processing capabilities within a handheld intraoral scanner, eliminating the need for additional equipment or devices to generate the 3D model of the tooth.

[0036] Embodiments of this disclosure provide visualized 2D internal feature image information on a 3D surface model by stitching together multiple 2D internal feature images (particularly 2D IR hyperspectral images) covering at least a portion of the tooth structure. This disclosure provides techniques for setting and aligning stitched images between the 3D surface model and the viewpoint of the 3D surface model to project 2D internal feature image information onto the 3D surface model. Visualizing 2D internal feature image information on a 3D surface model is done from the angle of the tooth or tooth surface and the viewpoint of the observer (e.g., a dentist or examiner).

[0037] Embodiments of this disclosure allow for the direct visualization of certain diagnoses using 3D models that provide information on the subsurface structure of a tooth (e.g., an object's tooth). In one example, the projection of 2D internal feature image information onto a 3D surface model is accomplished by mapping and aligning multiple 2D internal feature images associated with the tooth onto the 3D surface model using one or more estimated object poses of the tooth. The projection is further completed by generating an offset mesh for the 3D surface model, such that the offset mesh represents a constant distance field from the surface of the object's tooth (i.e., the surface of the tooth).

[0038] In this way, the technique for superimposing 2D internal feature image information on a 3D surface model according to this disclosure can estimate the depth of different layers within the surface of a tooth. For example, for a tooth, the sub-surface information in the superimposed layers of the 3D surface model makes it possible to estimate the characteristics of the dentin layer, enamel layer, etc., of the tooth's internal structure. This allows an observer to observe and analyze the internal or internal structure of a subject's tooth from multiple angles to accurately determine the tooth's condition and characteristics.

[0039] Subsurface information can also be used to simulate how images are acquired from an intraoral scanner. This will be useful for optimizing image analysis for diagnosis and / or treatment. In particular, different features of the teeth in an image can be captured using an intraoral scanner by controlling the frequency, angle, etc., of the scanner's operation, allowing different features of the image to be used for different analyses. Embodiments of this disclosure also provide improved image stitching techniques, particularly for panoramic image stitching. Attached Figure Description

[0040] This disclosure is illustrated in the accompanying drawings by way of example and not limitation, wherein the same reference numerals denote the same elements, and wherein:

[0041] Figure 1A A network environment for implementing an intraoral scanning system for oral cavity scanning according to an exemplary embodiment is shown;

[0042] Figure 1B and Figure 1C An exemplary schematic diagram of an intraoral scanning system 102 according to various exemplary embodiments is shown;

[0043] Figure 1D The coordinate system of the tooth body according to an exemplary embodiment is shown;

[0044] Figure 2 A sequence diagram illustrating the generation of a 3D surface model according to an exemplary embodiment is shown;

[0045] Figure 3A , Figure 3B , Figure 3C and Figure 3D An exemplary schematic diagram is shown for generating multiple 2D internal feature images according to various exemplary embodiments;

[0046] Figure 4A A flowchart illustrating a method for generating an object mask for multiple 2D interior feature images according to an exemplary embodiment is shown.

[0047] Figure 4B A schematic diagram of an application object masking according to an exemplary embodiment is shown;

[0048] Figure 5A A flowchart illustrating a method for estimating the pose of one or more objects for a tooth, according to an exemplary embodiment, is shown.

[0049] Figure 5B A schematic diagram of aligning template object models is shown according to an exemplary embodiment;

[0050] Figure 6A A flowchart of a method for associating 2D internal geometric features with a 3D model of a tooth, according to an exemplary embodiment, is shown.

[0051] Figure 6B and Figure 6C A schematic diagram illustrating the overlay of an object pose onto a 3D model according to an exemplary embodiment is shown;

[0052] Figure 7A A flowchart of a method for overlaying a 2D internal feature image with a 3D model of a tooth, according to an exemplary embodiment, is shown.

[0053] Figure 7B and Figure 7C A schematic diagram of an exemplary offset surface according to an exemplary embodiment is shown;

[0054] Figure 7D , Figure 7E and Figure 7F A schematic diagram is shown for overlaying a 2D interior feature image onto an offset surface according to an exemplary embodiment;

[0055] Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G The method of merging different spectra used to generate a 2D internal feature image is illustrated according to an exemplary embodiment;

[0056] Figure 9 A block diagram of a handheld intraoral scanner according to an exemplary embodiment is shown; and

[0057] Figure 10 The preprocessing steps for visible light and IR images are shown in the example. Detailed Implementation

[0058] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of this disclosure. However, it will be apparent to those skilled in the art that this disclosure may be practiced without these specific details. In other instances, various systems and methods are shown only in block diagram form to avoid obscuring this disclosure.

[0059] The terms "an embodiment" or "embodiment" as used in this specification mean that at least one embodiment of this disclosure includes a particular feature, structure, or characteristic described in connection with that embodiment. The phrase "in one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment, nor is it necessarily a single or alternative embodiment that is mutually exclusive with other embodiments. Furthermore, the terms "a" and "an" do not imply a limitation of quantity, but rather indicate the presence of at least one of the referenced items. In addition, various features that may be demonstrated by some embodiments but not by others are described. Similarly, various requirements that may be required by some embodiments but not by others are described.

[0060] Some embodiments of the present disclosure will now be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, of the embodiments of the present disclosure. In fact, various embodiments of the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure will satisfy any applicable legal requirements. The same reference numerals denote the same elements. As used herein, the terms “data,” “content,” “information,” and similar terms are used interchangeably to refer to data that can be transmitted, received, and / or stored according to embodiments of the present disclosure. Furthermore, the terms “processor,” “controller,” and “processing circuitry,” and similar terms are used interchangeably to refer to a processor capable of processing information according to embodiments of the present disclosure. Additionally, the terms “electronic device,” “electronic equipment,” and “device” are used interchangeably to refer to an electronic device monitored by a system according to embodiments of the present disclosure. Therefore, none of these terms should be used to limit the spirit and scope of the embodiments of the present disclosure.

[0061] Various embodiments are described herein for illustrative purposes, and these embodiments are subject to numerous variations. It should be understood that various equivalents are contemplated as potentially implying or creating advantageous situations without departing from the spirit or scope of this disclosure, but are intended to cover the application or implementation. Furthermore, it should be understood that the phrases and terms used herein are for descriptive purposes and should not be considered limiting. Any headings used in this description are for convenience only and have no legal or limiting effect.

[0062] The terms “for example” and “such as” as used in this specification and claims, as well as the verbs “comprising,” “having,” “including,” and other verbal forms thereof, when used with a series of one or more components or other items, are each interpreted as open-ended, meaning that such components or items should not be considered as excluding other additional components or items. Other terms should be interpreted using their broadest reasonable meaning unless they are used in a context that requires a different interpretation.

[0063] An intraoral scanning system, method, and computer-programmable product are provided for overlaying 2D internal feature image information onto a 3D surface model of a target tooth to provide an overall view of the tooth from different perspectives.

[0064] For example, refer to the following Figure 1A , Figure 1B , Figure 1C and Figure 1D An exemplary network environment is provided for an intraoral scanning system for oral cavity scanning and generating overlays.

[0065] Figure 1A An exemplary network environment 100 is shown for an implementation of an intraoral scanning system 102 for intraoral scanning according to an exemplary embodiment. The intraoral scanning system 102 can be used to generate an overlay of 2D internal feature image information (specifically, 2D IR hyperspectral image information) of teeth within a target oral cavity onto a 3D surface model. For example, the intraoral scanning system 102 can be used by a user, such as a person with dental knowledge, like a dentist, dental technician, etc. Furthermore, one or more components can be rearranged, altered, added, and / or removed without departing from the scope of this disclosure.

[0066] The intraoral scanning system 102 includes a handheld intraoral scanner 104 and one or more processors 106. The network environment 100 may also include a communication channel 108, which can be configured to establish communication links between the various components of the intraoral scanning system 102, i.e., the handheld intraoral scanner 104 and one or more processors 106 (hereinafter referred to as processors 106)).

[0067] For example, processor 106 can be implemented as one or more of a variety of hardware processing devices, such as a coprocessor, microprocessor, controller, digital signal processor (DSP), processing element with or without an accompanying DSP, or various other processing circuits including integrated circuits such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, dedicated computer chips, etc. Thus, in some embodiments, processor 106 may include one or more processing cores configured to execute independently. Multi-core processors can implement multiple processing within a single physical package. Additionally or alternatively, processor 106 may include one or more processors configured in series via a bus to enable independent execution of instructions, pipelines, and / or multithreading. Additionally or alternatively, processor 106 may include one or more processors capable of handling large workloads and operations to provide support for big data analytics. In an exemplary embodiment, processor 106 may communicate with other components of the intraoral scanning system 102 (hereinafter referred to as system 102) via a bus or communication channel 108 to transfer information between components of system 102.

[0068] In one example, when processor 106 is implemented as an executor of software instructions, the instructions may specifically configure processor 106 to perform the algorithms and / or operations described herein when the instructions are executed. However, in some cases, processor 106 may be a processor-specific device (e.g., a mobile terminal or a fixed computing device) configured to further configure processor 106 to employ embodiments of this disclosure by executing instructions that perform the algorithms and / or operations described herein. Processor 106 may include a clock, an arithmetic logic unit (ALU), and logic gates configured to support the operation of processor 106. A network environment (e.g., network environment 100) may be accessed using communication channel 108.

[0069] Communication channel 108 can be any combination of wired, wireless, or wired and wireless communication networks, such as cellular, Wi-Fi, the Internet, local area networks, etc. According to one embodiment, communication channel 108 can be one or more wireless full-duplex communication channels. In one embodiment, communication channel 108 can include one or more networks, such as a data network, a wireless network, a telephone network, or any combination thereof. It is conceivable that the data network is any local area network (LAN), metropolitan area network (MAN), wide area network (WAN), public data network (e.g., the Internet), short-range wireless network, or any other suitable packet-switched network (e.g., a commercially owned proprietary packet-switched network, such as a proprietary cable or fiber optic network), or any combination thereof. Furthermore, the wireless network can be, for example, a cellular network, and can employ various technologies, including Enhanced Data Rate for Global Evolution (EDGE), General Packet Radio Service (GPRS), Global System for Mobile Communications (GSM), Internet Protocol Multimedia Subsystem (IMS), Universal Mobile Telecommunications System (UMTS), and any other suitable wireless medium, such as Global Microwave Access Interoperability (WiMAX), Long Term Evolution (LTE) networks (e.g., LTE-Advanced Pro), 5G New Radio Networks, ITU-IMT 2020 networks, Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Wi-Fi, Wireless LAN (WLAN), Bluetooth, Internet Protocol (IP) Data Broadcast, satellite, Mobile Ad Hoc Networks (MANET), and any combination thereof. The handheld intraoral scanner 104 can be configured to communicate with the processor 106 via communication channel 108.

[0070] The intraoral scanning system 102 can be configured to store, for example, data generated by the intraoral scanning system 102 in a memory. For example, the intraoral scanning system 102 can be configured to store data captured by a handheld intraoral scanner 104, namely multiple 2D images 110. In one example, the multiple 2D images 110 generated by the handheld intraoral scanner 104 include multiple two-dimensional (2D) infrared (IR) images 112 and multiple visible light images 114. Furthermore, the intraoral scanning system 102 can be configured to store data generated by the processor 106. Such data may include a three-dimensional (3D) model 116, namely a three-dimensional (3D) surface model 118 of the object's teeth, and an overlay 120 of 2D image information on the 3D surface model 118.

[0071] The intraoral scanning system 102 can be used for intraoral scanning registration and to generate overlays of teeth to display sub-surface information and surface information for the teeth. The intraoral scanning system 102 may include multiple components, such as a handheld intraoral scanner 104, a processor 106, and a memory (not shown), which can communicate with each other to register intraoral scans of teeth.

[0072] In one example, the handheld intraoral scanner 104 may include a processor 106. In another example, the handheld intraoral scanner 104 may be coupled to the processor 106, wherein the processor 106 can be remotely positioned and can perform operations associated with the intraoral scanning system 102. The intraoral scanning system 102 may have enhanced processing capabilities required to process multiple 2D IR images 112 and visible light images 114 in real time to generate multiple 2D internal feature images and further overlay the 2D internal feature image information (or 2D internal features) of the teeth onto a 3D surface model 118.

[0073] A handheld intraoral scanner 104 can be configured to capture multiple 2D images 110 during a scanning process of a subject's oral cavity. The multiple 2D images 110 may include multiple 2D IR images 112 indicating images of the subject's teeth captured using light emitted at wavelengths corresponding to the infrared or near-infrared (NIR) wavelength range. The multiple 2D IR images 112 may include images of the subject's teeth from various viewing angles or viewpoints. Furthermore, the multiple 2D images 110 may include visible light images 114 indicating images of the subject's teeth captured using light emitted at wavelengths corresponding to a range or selected sub-spectrum of visible white light wavelengths, such as blue light (400 to 495 nm), green light (490 to 570 nm), and fluorescence (300 to 800 nm). Visible light images 114 may also include images of the subject's teeth from various viewing angles. In some cases, the multiple 2D images 112 may also include images captured at other visible and / or non-visible light wavelengths.

[0074] In operation, processor 106 is configured to receive and / or acquire visible light and infrared information from one or more sensors (hereinafter referred to as sensors) of handheld intraoral scanner 104. The sensors of handheld intraoral scanner 104 may include, for example, light sensors and / or image sensors. For this purpose, based on the visible light and / or infrared light sensed by the sensors, the sensors can capture visible light and infrared light information corresponding to the tooth structure of the object and provide it to processor 106. In one example, processor 106 may receive or acquire visible light information as a plurality of 2D visible light images 114 (hereinafter referred to as visible light images 114) and infrared information as a plurality of 2D IR images 112 (hereinafter referred to as IR images 112).

[0075] In one example, the handheld intraoral scanner 104 may include a web server configured to communicate via a web network and establish a connection to communication channel 108. The handheld intraoral scanner 104 may be configured to cause the web server to provide visible light and infrared information acquired from sensors to processor 106. The handheld intraoral scanner 104 may also include a processing unit, a memory unit, a communication interface, sensors, and additional components. The processing unit, memory unit, communication interface, and additional components may be communicatively coupled to each other. For example, in... Figure 1B and Figure 9 The document further provides details of the components of the handheld intraoral scanner 104.

[0076] Furthermore, processor 106 is configured to determine 3D surface information of the object's tooth based on the captured visible light image 114 (specifically, white light information or a white light image). Specifically, processor 106 is configured to generate a 3D surface model 118 of the tooth based on the visible light information. The 3D surface information can be a digital representation of the dental arch (e.g., teeth, gingiva, or a set of teeth) of the object's tooth depicted in 3D space. The 3D surface information can include, for example, 3D point cloud data corresponding to the visible light image 114 or the white light image. The 3D point cloud data can correspond to 3D real-world coordinates. The 3D surface information can also include color textures reflected from the tooth surface (i.e., the surface of a tooth or a set of teeth).

[0077] Processor 106 may include the processing capabilities required to process visible light image 114 (or visible light information) and infrared image 112 (or infrared information). Processor 106 may be configured to establish a connection to communication channel 108. In one example, processor 106 may be located within handheld intraoral scanner 104. For example, processor 106 may receive visible light information or visible light image 114 and infrared information or infrared image 112 from the handheld intraoral scanner 104 or its sensors to generate 3D surface information of the tooth for a given object. Furthermore, a 3D surface model 118 of the tooth for the object is generated based on the 3D surface information derived from the visible light image (e.g., white light image) 114. Processor 106 may also render the 3D surface information or 3D surface model 118 of the tooth as an interactive 3D graphical representation. For example, in combination with... Figure 2 The details of the 3D surface model 118 for generating the tooth are further described.

[0078] The processor 106 is also configured to generate multiple two-dimensional (2D) internal feature images based on visible light and infrared information. These multiple 2D internal feature images indicate the 2D internal geometry of the tooth. In one example, the multiple 2D internal feature images can be a hybrid image generated based on IR and visible light information associated with one or more sub-spectrums. In one example, multiple 2D internal feature images can be generated based on a combination of white light image information, fluorescence (e.g., fluorescent red and / or fluorescent green) image information, and IR image information. However, such image spectra used to generate hybrid or hyperspectral images should not be interpreted as limiting. For example, combining... Figure 3A , Figure 3B , Figure 3C and Figure 3D The details of generating multiple 2D internal feature images are described.

[0079] In one example, multiple 2D internal feature images or 2D hyperspectral images can be generated to distinguish different enhanced internal structures of the tooth using different colors, making it easier for the user of system 102 to differentiate between them. In another example, different colors can be assigned to each of the multiple channels associated with the synthesized scan information (e.g., channels for visible light and IR information). To improve the differentiation of different enhanced internal structures, processor 106 can be configured to weight each of the multiple channels in different or similar ways using channel weighting coefficients. For example, combining... Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G The details of generating 2D internal feature images or 2D hyperspectral images are further described.

[0080] Once multiple 2D internal feature images indicative of the tooth's internal structural information or sub-surface information are generated, processor 106 is configured to process the multiple 2D internal feature images. Processor 106 is configured to associate the 2D internal geometry of the tooth with at least one reference frame of the tooth in a 3D surface model 118. It can be noted that the 3D surface model 118 may include 3D surface information that can be represented as a mesh. Furthermore, in order to overlay the multiple 2D internal feature images onto the 3D surface model 118, processor 106 may have to select locations within the 3D surface model 118 to position or overlay the multiple 2D internal feature images onto the 3D surface model 118. Selecting the location for overlaying the multiple 2D internal feature images ensures that the images are positioned at approximately a constant distance from the tooth surface, and that the placement and orientation of the multiple 2D internal feature images vary smoothly across the entire 3D surface model. For example, combining... Figure 5A , Figure 5B , Figure 6A , Figure 6B and Figure 6C The details of associating the 2D internal geometry of the tooth with the 3D surface model 118 are described.

[0081] In one example, a reference frame may include vertical lines or normals of a 3D mesh of a 3D surface model 118. Furthermore, reference frames (e.g., multiple reference frames) may indicate vectors transmitting information about the local surface orientation at each control point of the 3D mesh. For this purpose, control points can be used to map pixels of multiple 2D internal feature images to their corresponding locations to generate a superposition 120 of multiple 2D internal feature images on the 3D surface model 118. For this purpose, one or more reference frames for a tooth in the 3D surface model 118 can provide information related to the geometry and orientation of the surface of the 3D surface model 118. Subsequently, when the reference frames of the tooth are associated with 2D internal geometric features, they provide precise locations for superimposing multiple 2D internal feature images. For example, combining... Figure 1D Details of the reference frame are provided.

[0082] Based on relevant reference frames and 2D internal geometry features of the tooth structure, multiple 2D internal feature images can be superimposed and provided as output. In one example, the output may include a superposition 120 of 2D image information (i.e., 2D internal feature images) on a 3D surface model 118. The relevant reference frames and 2D internal geometry features can indicate how pixels from multiple 2D internal feature images should be shifted relative to each control point in the 3D mesh to accurately provide the viewer with surface and sub-surface information from different perspectives.

[0083] For example, a tooth overlay 120 can be presented as an interactive 3D graphical representation. This interactive 3D graphical representation can be displayed on the device's display unit. The interactive 3D graphical representation can be viewed by a viewer or user (e.g., a dentist in front of the monitor) to diagnose any diseases or abnormalities within the tooth. Users can modify the view or perspective of the interactive 3D graphical representation of overlay 120 based on their preferences. For example, the perspective of the interactive 3D graphical representation can be changed according to the user's preferences, or the interactive 3D graphical representation can be zoomed in or out.

[0084] In one example, the 3D graphic representation of overlay 120 can be presented by an intraoral scanner 104 or any other light projector, such that the overlay is presented directly above the surface of the object tooth. In another example, the 3D graphic representation of overlay 120 can be presented on a display device. The display device can be associated with any user-accessible device such as a display unit, monitor, mobile phone, smartphone, tablet, computer, artificial real estate (XR) device, etc. In some examples, the display device can be part of a user-accessible device. The display can be, for example, a touchscreen display. Additional, different, or fewer components can be provided. Furthermore, one or more components can be rearranged, altered, added, and / or removed without departing from the scope of this disclosure.

[0085] Figure 1B An exemplary schematic diagram 122 of an intraoral scanning system 102 according to an exemplary embodiment is shown. The intraoral scanning system 102 includes a handheld intraoral scanner 104 configured to scan a tooth 124. In one example, the handheld intraoral scanner 104 may emit light of various wavelengths in a pulsed manner. For example, light of different wavelengths (e.g., white light, blue light, infrared light, near-infrared light, fluorescence, etc.) may be used to capture images of the tooth 124. For this purpose, due to the high pulse repetition rate of the light emitted from the handheld intraoral scanner 104, visible light images 114 and / or infrared images 112 of the tooth can be captured simultaneously from the same location.

[0086] The handheld intraoral scanner 104 may include a projector unit configured to emit light of different wavelengths, such as near-infrared wavelengths, infrared wavelengths, white wavelengths, and / or color visible wavelengths, onto at least the tooth 124. In one example, the projector unit may be configured to emit light of different wavelengths onto at least the tooth 124 in a pulsating manner during different time periods, wherein the different wavelengths include near-infrared wavelengths, infrared wavelengths, and visible wavelengths.

[0087] In one example, the visible light emitted by the projector unit may include light of one or more colors (e.g., red, green, blue, and white) of a filtered visible light signal. The projector unit may include multiple light sources configured to emit one or more colors of light and infrared light. The multiple light sources may be arranged within a single module, which includes multiple light-emitting diodes (LEDs) configured to emit different wavelengths in the visible and invisible wavelength ranges. In another example, the light sources (i.e., one or more LEDs that can be configured to emit infrared light) may be arranged separately from the light sources configured to emit visible light.

[0088] The handheld intraoral scanner 104 may include an image sensor configured to capture visible light information and infrared information from the tooth 124, which is at least drawn by the emitted light from the projector unit. In another example, the image sensor is configured to capture visible light information and internal light information from the tooth 124, which is drawn by at least visible wavelengths and near-infrared and / or infrared wavelengths, respectively.

[0089] The image sensor unit may include multiple cameras, such as high-speed cameras. In one example, multiple cameras may be arranged around or adjacent to the projector unit.

[0090] An image sensor unit may include multiple pixels, wherein each of a plurality of monochrome channels and each of a plurality of combined color channels may be aligned with each of the plurality of pixels. In this example, each color channel may be superimposed with a pixel of the image sensor or a group of pixels of the image sensor.

[0091] Visible light and infrared information generated by the handheld intraoral scanner 104 can be transmitted to the processor 106 via a wired or wireless communication channel 108. In one example, the processor 106 can be implemented as a computing device 126A or a server 126B external to the handheld intraoral scanner 104.

[0092] System 102 includes one or more processors, an external computer 126A, and / or a server 126B disposed within a handheld intraoral scanner 104. The handheld intraoral scanner 104 may include a processor configured to process sensor data from sensors of the handheld intraoral scanner 104 into information, such as visible light and infrared information configured to be sent to the external computer 126A or server 126B. Furthermore, the external computer 126A or server 126B may include a processor 106 for processing the received visible light and infrared information.

[0093] For example, the subject may require dental treatment. In this case, system 102 can be used by a user (e.g., a dentist) to provide dental treatment to the subject. In one embodiment, the subject may be at a dental clinic. In this case, system 102 can be used in the treatment room of the dental clinic. In another embodiment, the subject may have already requested a home visit for dental treatment. In this case, system 102 can be used at the subject's home. To begin dental treatment, the user can utilize a handheld intraoral scanner 104 to capture visible light images 114 and infrared images 112 of the subject's teeth 124 using one or more sensors.

[0094] In one example, throughout the entire scanning of the object, the projector unit can be configured to continuously emit infrared light while emitting visible light. In this example, switching between on and off infrared light emission is avoided, thus preventing undesirable transients in infrared light emission. Furthermore, any timing issues between visible and infrared light are also avoided. In other embodiments, pulsed infrared and visible light can be emitted in a 2:1 time period; that is, for every 2 seconds of visible light emission or 2 pulses of visible light, IR light can be emitted in the next second or the next pulse, respectively.

[0095] Figure 1C Another exemplary schematic diagram of an intraoral scanning system 102 according to an embodiment is shown. According to this example, the processor 106 of the system 102 receives, for example, visible light information 128 or a visible light image 114 and infrared information 130 or an infrared image 112 from a handheld intraoral scanner 104. In one example, the processor 106 is configured to generate infrared information 130 based on subtracting a combined light signal from one or more color light signals of the acquired visible light information 128.

[0096] Furthermore, processor 106 is configured at 132 to generate multiple 2D internal feature images based on visible light information 128 and infrared information 130. The multiple 2D internal feature images indicate the 2D internal geometry of the tooth 124. For example, the internal geometry of the tooth 124 can be determined based on infrared information 130. Processor 106 is also configured at 134 to determine 3D data of the tooth based on visible light information 128 to further generate a 3D surface model 118 of the tooth 124. In one example, the 3D surface model 118 may include a mesh with multiple control points indicating surface color and / or texture information of the tooth 124. Subsequently, processor 106 is further configured at 136 to process the multiple 2D internal feature images to associate the 2D internal geometry of the tooth 124 with one or more normals of the mesh of the 3D surface model 118 of the tooth 124. Associating 2D internal geometry features with the normals of the 3D surface model 118 allows sub-surface information from 2D internal feature images to be mapped onto the 3D surface model 118. For example, the sub-surface information can provide color information, shadow information, and internal region information determined at least based on infrared information 130 or multiple 2D internal feature images. Furthermore, processor 106 is configured to output a superposition 120 of multiple 2D internal feature images to the 3D surface model 118 at 138. Superposition 120 includes the surface 3D geometry and internal geometry of the tooth 124 or tooth row. In one example, processor 106 can be configured to determine superposition 120 and the 3D surface model 118 in parallel based on infrared information 130 and visible light information 128.

[0097] exist Figure 1C In the example shown, system 102 includes display unit 140. For example, a 2D internal feature image can be rendered on display unit 140 as an overlay 142 onto a 3D surface model 118. Processor 106 is configured to display the overlay 142 in real time. The displayed overlay 142 includes 3D surface model and internal structure information in the form of pixels of a plurality of 2D internal feature images.

[0098] The processor 106 can determine the internal structure or subsurface information of the tooth 124 based on infrared information 130. The subsurface information may include information about dental features disposed within the tooth 124. Dental features may be one or more of anatomical features, disease features, and mechanical features. Anatomical features may indicate information relating to, for example, the enamel, dentin, or pulp of the tooth 124. Disease features may indicate information relating to, for example, dental plaque, dental fissures, or caries. Mechanical features may indicate information relating to, for example, fillings and / or composite restorations.

[0099] In one example, based on multiple 2D internal feature images, i.e., 2D hyperspectral infrared images, these images may need to be positioned at or above selected locations on the 3D surface model 118. For this purpose, such placement of the 2D internal feature images on the 3D surface model 118 will also need to display newly acquired information, for example, due to changes in how the 3D surface model 118 is viewed directly. This visualization of updated information about the internal structure of the tooth due to changes in perspective allows certain diagnoses to be visualized directly on the 3D surface model 118 or overlay 120, which may not be obtained solely using white light images or white light rendering within the 3D model. The projection or overlay of the 2D internal feature images is accomplished by mapping and aligning the pixels of the 2D internal feature images to estimate one or more object poses for the tooth 124. In one example, one or more object poses can be determined for each tooth of the object. Based on one or more object poses (or one or more tooth poses), an overlay of the 2D internal feature images on the 3D surface model can be generated.

[0100] Figure 1D A coordinate system 144 for a tooth body according to an exemplary embodiment is shown. Specifically, coordinate system 144 corresponds to the structure of the tooth body 146 or tooth, and different information provided by different directions and / or planes. Coordinate system 144 may be associated with a 3D surface model 118 for superimposing 2D internal feature information of the internal geometry of the tooth body 146 onto the 3D surface model.

[0101] Therefore, the coordinate system 144 of the tooth body 146 can have three axes (X, Y, and Z). In addition, the coordinate system 144 can have two or more planes, such as the XY plane 148, the YZ plane 150, and the XZ plane 152 (collectively referred to as plane 148, plane 150, and plane 152).

[0102] In one example, each of the planes 148, 150, and 152 of the tooth body or tooth 146 in the 3D surface model 118 includes at least a first plane and a second plane. For example, the XY plane 148 may be the first plane, and the YZ plane 150 may be the second plane. For this purpose, the XY plane 148 or the first plane may correspond to the buccal-lingual plane of the tooth body or tooth 146 (hereinafter referred to as tooth 146), and the YZ plane 150 or the second plane may correspond to the mesial-distal-mesial plane of tooth 146.

[0103] Therefore, different information is provided for different planes and axes of coordinate system 144 for tooth 146. In one example, the Y-axis may correspond to the tooth axis. For example, the tooth axis may indicate information about the depth of tooth 146.

[0104] In addition, the planes 148, 150 and 152 of the tooth body or tooth 146 include at least one of the occlusal plane, buccal plane, lingual plane, mesial plane, distal plane or labial plane.

[0105] Figure 2 A sequence of diagrams 200 illustrating the generation of a 3D surface model 118 according to an exemplary embodiment is shown. (In conjunction with...) Figure 1A , Figure 1B , Figure 1C and Figure 1D To explain the components Figure 2 Sequence diagram 200 may include a handheld intraoral scanner 104 and one or more processors 106. Sequence diagram 200 will depict operations performed by at least one of the handheld intraoral scanner 104 and processor 106.

[0106] In step 202, the projector unit of the handheld intraoral scanner 104 can illuminate the patient's oral cavity. The projector unit can illuminate the teeth in the oral cavity using visible or white light wavelength pulses and IR and / or NIR wavelength pulses. In one example, the teeth 124 can correspond to one or more structures, such as a single tooth, a group of teeth, and / or gums in the patient's oral cavity.

[0107] In step 204, one or more sensors of the handheld intraoral scanner 104 can detect visible light and IR or NIR light. This detected visible light and NIR light can be reflected or refracted from the surface or internal regions of the tooth 124. In other words, one or more sensors can detect visible light information 128 and infrared information 130 reflected or refracted from the tooth 124 or the tooth itself.

[0108] In step 206, one or more sensors of the handheld intraoral scanner 104 can capture a visible light image 114 and a 2D IR image 112. For example, the one or more sensors may include an image sensor that can be configured to capture a visible light image 114 based on detected visible light information 128 and an infrared image 112 based on detected infrared information 130. In this way, the one or more sensors can be configured to generate multiple 2D images 110 of the tooth 124.

[0109] In step 208, the processor 106 receives a plurality of 2D images 110 of the tooth 124. As described above, the plurality of 2D images 110 of the tooth 124 may include visible light images 114 and infrared images 112.

[0110] In step 210, visible light information 128 is used to determine 3D surface information. Specifically, visible light image 114 can be processed to determine 3D surface information. In one example, focus measurements in visible light image 114 can be determined. Furthermore, the projection features of tooth 124 can be tracked across the entire visible light image 114 or white light image. For example, a correspondence function can be executed or solved to triangulate depth information for tooth 124 and determine 3D surface information of tooth 124.

[0111] In step 212, a 3D surface model 118 is generated for the tooth 124 based on the 3D surface information. In one example, a 3D image patch corresponding to a portion of the surface of the tooth 124 or a set of teeth of an object can be generated by accessing calibration data from one or more sensors and converting the 3D surface information corresponding to that portion into real-world 3D coordinates and texture information. For example, different 3D image patches can be generated for different portions of the tooth surface using a visible light image 114 or a white light image. In some cases, overlays may exist in portions where different 3D image patches are superimposed. For example, by locating corresponding data points, a 3D image patch for a portion of the tooth surface can be registered or associated with one or more previously generated 3D image patches for other portions and / or superimposed portions of the tooth surface. Subsequently, the 3D image patch for that portion can be fused with one or more previously generated 3D image patches associated with other portions of the tooth surface to generate the 3D surface model 118.

[0112] In one example, the 3D surface model 118 may be a 3D point cloud comprising 3D points or 3D control points within voxels in a signed distance field, and the signed distance field may be converted into a 3D mesh for rendering the 3D surface model 118. In one example, the 3D surface model 118 is generated as an offset mesh of a raster or grid including the 3D control points. For example, an initial raster or grid of 3D control points for generating the 3D surface model 118 may be created on an input image (i.e., visible light image 114). The raster may be a collection of points (referred to as 3D control points) strategically placed based on the type of distortion or transformation to be performed on the image. These 3D control points may be used as reference values ​​to map image pixels in the corresponding grid to generate the 3D surface model 118. For example, each 3D control point may provide associated offset information defining how much the pixels at that control point should be moved or adjusted. The 3D surface model 118 may also include color and texture data of the tooth surface.

[0113] After generating a 3D surface model 118 of tooth 124 or a group of teeth, processor 106 can be configured to process visible light information as well as infrared information to generate a superposition 120 indicating sub-surface information for tooth 124 or a group of teeth. For example, combining 7A, Figure 7B , Figure 7C , Figure 7D , Figure 7E and Figure 7F The details of generating the overlay are described.

[0114] Figure 3A , Figure 3B , Figure 3C and Figure 3D An example diagram is shown for generating multiple 2D internal feature images based on IR information 130 and visible light information 128. Specifically, Figure 3A , Figure 3B , Figure 3C and Figure 3D The generation of 2D internal feature images is described based on different combinations (e.g., addition or subtraction) of visible and IR light spectra. As a result, mixed or hyperspectral images are generated. It should be noted that the different ways of generating 2D internal feature images or 2D hyperspectral images of tooth 124 are merely exemplary and should not be construed as limiting.

[0115] Embodiments of this disclosure can utilize mixed signals or information obtained from imaging the tooth 124 with different light sources (i.e., white, blue, and infrared or near-infrared) to obtain a 2D internal feature image or a hyperspectral image. It can be noted that the 2D internal feature image may include a 2D composition of the internal geometry of the tooth 124. The object of this disclosure is to combine information from the 2D internal feature image or the hyperspectral image (i.e., images taken with white / blue / infrared light) to generate a superposition 120 displaying various diagnoses (e.g., tooth cracks, caries, dental plaque, etc.).

[0116] In one example, visible light information 128 includes surface reflection, i.e., surface information provided, for example, by white light emitted by the handheld intraoral scanner 104. Furthermore, infrared information 130 includes sub-surface structure information provided by infrared and / or near-infrared light emitted by the handheld intraoral scanner 104.

[0117] refer to Figure 3AThe visible light information 128 may include, for example, white light information 128A provided by white light emitted by a handheld intraoral scanner 104. For this purpose, multiple 2D internal feature images 302 can be generated based on subtracting the white light information 128A from the infrared information 130. In one example, the visible light information 128 may be filtered, for example, using a monochromatic or multicolor channel to produce visible light of different wavelengths, such as white light 128A, red light, blue light, green light, etc. This filtered white light information 128A can be used to determine a 3D surface model 118 of the tooth 124. Furthermore, the 2D internal feature images 302 include enhanced internal structures 304, represented by restorations that are not visible in the infrared information 130 but can be easily identified in the synthesized or generated 2D internal feature images 302. The synthesized 2D internal feature images 302 can be mapped onto the 3D surface model 118 by the processor 106, such that the synthesized 2D internal feature images 302 provide 3D sub-surface information about the internal structure of the tooth 124.

[0118] refer to Figure 3B The visible light information 128 may, for example, include excitation fluorescence information 128B provided by green and / or blue light emitted by the handheld intraoral scanner 104. According to this example, multiple 2D internal feature images 306 can be generated based on subtracting the fluorescence information 128B from the infrared information 130. For example, the fluorescence information 128B can be used to apply fluorescence information to a 3D surface model 118 of the tooth 124. It can be noted that the infrared information 130 cannot independently display anomalies 308 in the internal regions of the tooth 124. However, the infrared information 130 with the subtracted fluorescence information 128B in the 2D internal feature image 306 enhances the visibility of the anomalies 308.

[0119] refer to Figure 3C The visible light information 128 may include, for example, green fluorescence information 128C excited by green light emitted by a handheld intraoral scanner 104. According to this example, multiple 2D internal feature images 310 can be generated based on a combination of the green fluorescence information 128C and the infrared information 130. For this purpose, the 2D internal feature images 310 include enhanced texture information regarding different layers within the internal region of the tooth 124. The layers within the internal region may correspond to, for example, enamel 312A and dentin 312B. As a result, due to the improved contrast in the 2D internal feature images 310, the dentin-enamel junction (DEJ) will be more easily or clearly seen.

[0120] It can be noted that enamel 312A and dentin 312B are seen more clearly in the synthesized 2D internal feature image 310 than in the infrared information 130. In some cases, multiple 2D internal feature images can also be generated using excited red fluorescence information (e.g., by combining infrared information 130, green fluorescence information 128C, and red fluorescence information). Such 2D internal feature images generated based on infrared information 130, green fluorescence information 128C, and red fluorescence information improve the visibility of the DEJ compared to regular or enhanced fluorescence information or independently compared to infrared information 130.

[0121] refer to Figure 3D The projector unit is configured to emit light pulses corresponding to visible light pulses or visible wavelength pulses. In one example, the wavelength of the visible light pulse may include blue pulse wavelengths and white pulse wavelengths. In other words, the projector unit is configured to emit visible light pulses having wavelengths corresponding to white light and blue light. The projector unit may also emit invisible light pulses including infrared wavelengths. In one example, the emitted blue wavelength pulse can be used to capture excited green fluorescence information 128C and excited red fluorescence information 128D.

[0122] For this purpose, the visible light signals (128A, 128C, and 128D) include surface information of the tooth 124 provided by emitted white wavelength pulses, fluorescent green wavelength pulses provided by emitted blue wavelength pulses, and fluorescent red wavelength pulses. The surface information is used to generate or update the 3D model 118. Furthermore, the fluorescence information (128C and 128D) can be used to generate multiple 2D internal feature images 314. In this example, the processor 106 can be configured to determine a first difference between infrared information 130 and green fluorescence information 128C, and a second difference between infrared information 130 and red fluorescence information 128D. Furthermore, the 2D internal feature images 314 are generated based on a combination of the first and second differences. The generated 2D internal feature images 314 provide enhanced internal structural information associated with the tooth 124. For example, compared to the independent infrared information 130, the 2D internal feature images 314 more clearly indicate sub-surface information such as anomalies, anatomical structures, etc.

[0123] In one example, the generated multiple 2D internal feature images may include non-object information. For example, the multiple 2D internal feature images may include portions corresponding to the tongue, walls, etc., of an object's oral cavity. However, the 3D surface model 118 and / or overlay 120 that generate the 2D internal features of the tooth 124 on the 3D surface model do not require such information. Furthermore, this information would increase the processing time and computational power required to generate the overlay 120. Therefore, it is necessary to eliminate such non-object information that may be irrelevant to the tooth and gum tissue in the multiple 2D internal feature images. For example, combining... Figure 4A and Figure 4B The details of removing non-object information are described.

[0124] Figure 4A A flowchart 400 illustrates a method for generating an object mask for an internal feature image according to an exemplary embodiment. As described above, a 2D internal feature image can be generated based on infrared information 130 and visible light information 128. For example, the infrared information 130 can be enhanced based on different spectra or color-based excitation fluorescence information and / or white light information. This enhancement of the infrared information 130 based on different spectra can result in a clearer internal structure of the tooth 124.

[0125] In one example, due to the high pulse repetition rate, certain infrared and visible light images can be collected from the same time frame and the same scan position. Therefore, an object mask generated from the visible light image can be implemented on top of or correspond to the infrared image.

[0126] Object masking can be used to filter or isolate specific portions of an image (e.g., multiple 2D internal feature images) while excluding or suppressing unwanted regions. For example, an object mask can be implemented as a binary image where each pixel is assigned a value of 1 (inclusive) or 0 (exclusive) based on a predefined criterion or pattern. Such a predefined criterion or pattern can be defined, for example, based on a visible light image. Object masking can be used to perform occlusion or occlusion-based filtering to apply selective processing to multiple 2D internal feature images.

[0127] At 402, processor 106 is configured to generate an object mask using visible light information captured from the scanning position. It can be noted that the scanning position corresponds to the position of one or more sensors of the handheld intraoral scanner 104. In one example, processor 106 is configured to generate multiple masks based on different movable scanning positions of the handheld intraoral scanner 104 (hereinafter referred to as scanner 104). Furthermore, for a specific scanning position, processor 106 is configured to identify visible light information or a visible light image captured by scanner 104. Based on the identified visible light image for the scanning position, one or more portions of the tooth 124 in the identified visible light image can be defined and identified.

[0128] In one example, an object mask can be automatically generated based on predefined criteria or image processing techniques. The object mask can be generated based on the identification of portions of tooth 124 in an image (e.g., a visible light image). Furthermore, pixel values ​​of "1" can be assigned to pixels corresponding to these portions of tooth 124 in the image. In this way, the object mask can define which parts of the image should be included or excluded in the filtering process. For example, techniques used to create the object mask can include, but are not limited to, thresholding, edge detection, and region segmentation.

[0129] At 404, the processor 106 is configured to apply an object mask over a plurality of 2D internal feature images. When generating the object mask for a scan location, a set of 2D internal feature images associated with the scan location can be identified. For example, this set of 2D internal feature images can be generated based on a combination of a visible light image and an IR image captured from the scan location. Furthermore, since the pulse repetition rate used to emit visible wavelength pulses and IR wavelength pulses is high, visible light images and IR images can be captured from the same scan location. Therefore, based on the generated object mask for the scan location, this set of 2D internal feature images can be segmented. At this point, the object mask can be applied to this set of 2D internal feature images to segment at least a portion of the 2D internal feature images. For example, combining... Figure 4B The details of applying object masking to a 2D internal feature image are described.

[0130] refer to Figure 4B According to an exemplary embodiment, a schematic diagram of applying an object mask 408 is shown. For example, the object mask 408 may be an image corresponding to a visible light image, such that the object mask 408 can assign pixel values ​​"1" to multiple portions of the tooth 124 in the visible light image. For example, as shown in 410A, the object mask 408 may be generated based on assigning pixel values ​​"1" to pixels corresponding to the tooth 124 in the visible light image. Furthermore, in some cases, as shown in 410B, the object mask 408 may be generated based on assigning pixel values ​​"1" to pixels corresponding to the outer edges of the tooth 124 in the visible light image.

[0131] Furthermore, object mask 408 can be applied to the segmentation portion of the set of 2D internal feature images that indicate non-object information. In this regard, the set of 2D internal feature images, generated based on a combination of a visible light image and an IR image taken from the same scanning position, can be segmented. Object mask 408 can, for example, be superimposed on the 2D internal feature images to perform element-wise multiplication. Each pixel value in the mask is multiplied by the corresponding pixel value in the image from the set of 2D internal feature images. If the mask pixel is 1, the corresponding pixel in the image remains unchanged; if the mask pixel is 0, the corresponding pixel in the input image is set to 0. In this way, object mask 408 is applied to the infrared information in the set of 2D internal feature images to generate filtered infrared information 414A, and to the visible light information (such as fluorescence information) in the set of 2D internal feature images to generate filtered visible light information 414B.

[0132] Return to Figure 4A At 406, processor 106 is configured to remove segmented portions of the 2D internal feature image that indicate non-object information. At this point, by applying object mask 408, the result of element-wise multiplication generates an image that retains only the pixels of the set of 2D internal feature images aligned with the "1"s in object mask 408. In this way, segmented portions indicating non-object information can be removed from the set of 2D internal feature images.

[0133] While this example describes applying an object mask to a 2D internal feature / hyperspectral image, in some cases, one or more object masks generated from a white light image can be applied to captured blue light and infrared images to remove background, redundant information, or non-tooth information from multiple 2D images 110. The object mask can segment portions corresponding to object information, such as teeth. This is possible because the white light image, blue light image, and IR image were captured from the same location at the same time frame.

[0134] To this end, one or more object masks can be generated based on different scanning positions of the scanner 104 to filter multiple 2D internal feature images. Furthermore, in some cases, the masked or filtered 2D internal feature images may undergo various filtering operations, such as blurring, sharpening, contrast adjustment, noise reduction, or any other image processing techniques, to enhance the visibility of the internal structure of the tooth 124. These operations are applied only to selected regions of interest defined by one or more object masks. Once segmented, multiple 2D internal feature images can be processed to correlate with the 3D surface model 118 for generating a superposition 120. For example, combining... Figure 5A and Figure 5B The details of processing 2D internal feature images are described.

[0135] Figure 5AA flowchart 500 of a method for estimating the poses of one or more objects, according to an exemplary embodiment, is shown. Processor 106 generates one or more object poses using a 2D internal feature image and a 3D model 118. For example, one or more object poses are generated using a machine learning model. Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 4A and Figure 4B The elements explained Figure 5A .

[0136] A trained machine learning model can be used to estimate one or more object poses of the internal geometry of the tooth 124. In addition, one or more object poses can be used to overlay multiple 2D internal feature images of the tooth (referred to as 2D internal feature images) onto a 3D surface model 118 (hereinafter referred to as the 3D model).

[0137] Therefore, for clarity, embodiments of this disclosure are explained with reference to the processing of 2D internal feature images associated with a single tooth for estimating the pose of one or more objects or teeth, and further, the 2D internal feature images of the teeth are superimposed onto a 3D model 118 at the tooth location. Such embodiments may be repeated for each tooth in this group of teeth of the object to generate an object superposition 120.

[0138] refer to Figure 5A At 502, processor 106 is configured to align the template object model with the position of the tooth in 3D model 118. In one example, each tooth in 3D model 118 may belong to a tooth type, such as central incisor, lateral incisor, canine, first premolar, second premolar, first molar, second molar, third molar, etc. Furthermore, template object models (also called template tooth models) can be predefined for different tooth types. In one example, the template object model can be identified and aligned based on the position of the tooth or tooth in 3D model 118 and the tooth type of the tooth or tooth. For example, the tooth type can be identified based on the position of the tooth in 3D model 118. Furthermore, the template tooth model of the identified tooth type can be aligned with the position of the tooth in 3D model 118. Subsequently, the template object model has the corresponding tooth type, such as one of central incisor, lateral incisor, canine, first premolar, second premolar, first molar, second molar, and third molar.

[0139] It can be noted that template object models can provide general feature information related to the tooth structure (i.e., the tooth) based on the tooth type. For example, template object models can indicate the shape, size, and anatomical structure of the tooth's surface and / or internal structure.

[0140] refer to Figure 5B A schematic diagram 510 for aligning a template object model 512 is shown according to an exemplary embodiment. In one example, the template object model 512 also includes a 3D coordinate system 514 indicating the orientation of the tooth having the 3D model 118. In other words, the template object model 512 may have a predefined 3D coordinate system 514 (which has coordinate axes). For example, different coordinate axes of the 3D coordinate system 514 (e.g., X, Y, and Z) may correspond to different tooth axes to provide structural information about tooth types in different directions in 3D. Furthermore, the values ​​of the coordinate axes of the three-dimensional coordinate system 514 of the template object model 512 may indicate the orientation of the tooth having the 3D model 118 or the template object model 512 of the tooth.

[0141] In one example, the template object model 512 can be set at the origin of the three-dimensional coordinate system 514. For example, the values ​​along the coordinate axes of the 3D coordinate system 514 can be manipulated to convert the properties of the template object model 512 into the properties of its corresponding tooth or dental structure.

[0142] According to one embodiment, the coordinate axes of the 3D coordinate system 514 are aligned with the occlusal reference frame 516 of the tooth in the 3D model 118. It can be noted that the occlusal reference frame 516 can be a normal or perpendicular line to the occlusal plane 518 of the tooth in the 3D model 118. Specifically, the occlusal reference frame 516 can pass through the surface of the tooth in the 3D model 118, i.e., the surface normal. The surface normal or the occlusal reference frame 516 can define the occlusal direction for aligning the 3D coordinate system 514 with the 3D model 118. This ensures that the template object model 512 is aligned within the surface of the tooth identified in the 3D model 118.

[0143] Furthermore, based on object mask 408, segmentation of the mesh, occlusal orientation, and 2D internal feature image of 3D model 118 is used to perform estimation of the pose of one or more objects for transforming template object model 512 into actual teeth.

[0144] return Figure 5AAt 504, processor 106 is configured to associate the position of each 2D internal feature image of the tooth with the position of the tooth in 3D model 118. The position of the tooth in the 3D model can be identified based on calibration data associated with the sensor of scanner 104. Furthermore, 2D internal feature images showing the tooth or associated with the tooth can be identified based on the mapping between scan positions and / or scan time frames of visible light information used to generate one or more control points of the tooth in 3D model 118.

[0145] In one example, based on the correlation between the position of each of the 2D internal feature images of the tooth and the position of the tooth in the 3D model 118, the 2D internal image of the tooth can be classified or moved to the position of the tooth in the 3D model 118. This can be done for each tooth in the group of teeth. For the entire group of teeth, by locating the 2D internal feature images based on their corresponding positions in the 3D surface model, certain gaps associated with the structure of the tooth's internal structure can be filled and / or redundant information can be eliminated. This can improve the accuracy and efficiency of further processing of the 2D internal feature images. In one example, based on correlation, the 2D internal feature image can be moved or located at the position of the tooth in the 3D model 118.

[0146] According to one embodiment, due to the high pulse repetition frequency of the projection unit emitting infrared and visible light (e.g., white and blue light), the scanner 104 can capture visible light and infrared light information of the tooth from the same location. As a result, the sequential reception or entry into the visible light information corresponding to the tooth and subsequent points corresponding to the 3D model 118 of the tooth can approach, for example, a predetermined range of sequential reception of infrared information used to generate a 2D internal feature image of the tooth. Therefore, the processor 106 is configured to determine or correlate the position of the 2D internal feature image with the position of the tooth in the 3D model 118 based on the correlation between the sequence of incident visible light information and the sequence of incident infrared information corresponding to the tooth. This sequence can be defined, for example, by calibration data of the scanner 104's sensors, the scanning position or location of the scanner 104 relative to the tooth, the scanning time frame, etc. In particular, if the IR information of the 2D internal feature images in a particular sequence is not mapped or correlated with the corresponding sequence of visible light information of the 3D surface model, the correlation will result in misalignment in the positioning of the 2D internal feature images relative to the 3D surface model.

[0147] Furthermore, correlation allows for the establishment of a relationship between a 2D internal feature image of the tooth and its corresponding 3D representation in the 3D model 119. This correlation enables the processor 106 to accurately determine the position of the tooth within the 3D scene or 3D model 118. It also allows for tracking the tooth's position in 3D real-world coordinates when rendering the 3D model. In some cases, associating the position of the 2D internal feature image with the position of the tooth in the 3D surface model can also be used for spatial registration, i.e., ensuring the 3D model is correctly aligned with the real-world environment.

[0148] Subsequently, at 506, the processor 106 is configured to associate the viewing direction of each 2D internal feature image of the tooth with the 3D coordinate system 514 of the template object model 512 of the tooth. In one example, the viewing directions of the IR information and visible light information for the same time frame of the tooth can be close to or the same as the wavelength pulse emitted at a high rate. Therefore, the viewing direction or camera orientation of the scanner 104 when capturing 2D internal feature images generated based on infrared and visible light information can be determined based on the calibration data of the scanner 104's sensors. Furthermore, for example, if the 2D internal feature images are captured from a forward-looking direction (e.g., 10 degrees from the Y-axis), the values ​​of pixels and / or points at 10 degrees from the Y-axis of the template object model can be adjusted, updated, or modified based on the 2D internal feature images.

[0149] In one example, the correlation between the viewing orientation of the 2D internal feature image and the 3D coordinate system 514 allows for the 3D reconstruction of the internal geometry of the tooth by transforming the template object model based on the 2D internal feature image. As the viewing orientation of the 2D internal feature image is mapped to the coordinates of the template object model, the mapping information from the 2D internal feature image to the template object model 514 becomes accurate and easy.

[0150] Furthermore, based on the correlation between the viewing orientation of different 2D internal feature images and the 3D coordinate system 514 of the template object model 512, as well as the position of the 2D internal feature images relative to the 3D model 118, one or more object poses of the tooth can be determined. Specifically, the values ​​of the 2D internal feature images can provide sub-surface information of the tooth from different directions. In this way, the internal geometry of the template model 512 or the tooth can be determined to transform the template object model 512 into one or more object poses (i.e., position and orientation) of the tooth. Specifically, the matrix values ​​of the template object model 512 can be updated based on the correlation. Furthermore, one or more poses (i.e., matrices) can be used to transform the template object model 512 into the dimensions of a scanned tooth or a scanned tooth. For example, combining... Figure 6A , Figure 6B and Figure 6CThe method for converting template object model 512 is further described.

[0151] Figure 6A A flowchart 600 of a method for associating 2D internal geometric features with a 3D model 118 of a tooth, according to an exemplary embodiment, is shown. Accurately determining the correlation between the 2D internal geometric features and the 3D model 118 is crucial. To this end, the overlay of the 2D internal feature image onto the 3D surface model is performed based on the correlation between the internal geometric features and the 3D surface model. Precise correlation also ensures accurate updating of the overlay of the 2D internal feature image even when the viewer changes the perspective of the 3D model (i.e., rotates, moves, scales the 3D model, etc.). Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 4A , Figure 4B , Figure 5A and Figure 5B To explain the elements Figure 6A .

[0152] At 602, processor 106 is configured to estimate one or more object poses of the 2D internal geometry of the tooth. In one example, a trained machine learning model can be trained to align template object model 512 with a 3D model. Furthermore, the position of the 2D internal feature image is correlated with the position of the tooth in the 3D model 118; and the viewing direction or camera orientation for the 2D internal feature image is correlated with the 3D coordinate system 514 of the aligned template object model 512 for the tooth. This correlation between the position of the 2D internal feature image and the position of the tooth in the 3D model 118 establishes a relationship between the 2D internal feature image and its corresponding 3D representation, thereby ensuring the tracking of the tooth's position within the 3D scene or model. Furthermore, by aligning the viewing direction with the 3D coordinate system 514 of the template object model 512, information from the 2D internal feature image can be mapped onto the template object model 514. Based on these correlations, one or more object poses (e.g., position and orientation) of the tooth's internal geometry can be estimated for one or more viewing directions. Determining one or more poses is crucial for manipulating the information of the 2D internal feature image of the tooth during overlay.

[0153] In one example, one or more object poses for the internal geometry of a tooth may include matrices. In other words, each object pose can be defined as a matrix that transforms the template object model 512 or template tooth into a scanned tooth for a corresponding viewing orientation or orientation. For this purpose, the object pose matrix may indicate the orientation or orientation corresponding to the transformation of the template object model 512 based on the viewing orientation of the 2D internal features. For example, combined with... Figure 5A and Figure 5B It describes the details of the estimation of the pose of one or more objects regarding the 2D internal geometry of the tooth.

[0154] In one example, the trained machine learning model can be implemented as a convolutional neural network (CNN) and / or other deep learning architectures. The trained machine learning model can take a template object model 512 and a 2D internal feature image as input, and predict the object pose for different orientations, which are typically represented as translation (X, Y, Z) and rotation (pitch, yaw, roll) values.

[0155] For example, to train a machine learning model for pose estimation, a training dataset consisting of pairs of template object models and their corresponding ground-based poses for different orientations can be fed into the machine learning model. This data is used to teach the model how to associate visual features in a 2D interior feature image with template object models having a specific pose or orientation. For example, during training, the machine learning model can extract relevant features such as keypoints, edges, corners, or other unique visual elements to learn how to map orientation or orientation information from the 2D interior feature image using the coordinate axes of the 3D template object model.

[0156] At 604, processor 106 is configured to determine one or more intermediate poses of the tooth. In one example, matrix interpolation techniques can be applied to an affine matrix to obtain one or more intermediate poses of the tooth. Notably, for the tooth, a 2D internal feature image can be located or set in an orientation or direction defined by the object pose used for the internal geometry of the tooth.

[0157] Furthermore, for neighboring regions indicated by one or more pairs of neighboring images—that is, images indicating information relating to neighboring regions located or occurring in the group of teeth or between two adjacent teeth—precise orientation for locating the 2D internal feature image may not be defined. For this purpose, the orientation (such as position and orientation) of the 2D internal feature image for locating the neighboring region can be obtained based on smooth interpolation between the poses of two objects defined by a corresponding pair of neighboring images from the located 2D internal feature image. For example, the pose of the neighboring region corresponding to each of the two objects includes an affine matrix.

[0158] In one example, an affine matrix can represent various types of geometric transformations that preserve parallel lines and distance ratios. In other words, to change the shape, size, or position of a template object model based on a tooth and to locate a 2D internal feature image, the affine matrix corresponding to the tooth's internal geometry can be changed. In this way, the template object model can be transformed based on the tooth, and the 2D internal feature image can be located on an orientation defined by the pose of the internal geometry. Affine matrices can be used to transform template object models while keeping lines straight and maintaining the same proportions of the shape. For example, an affine matrix can include a set of numbers organized in a grid, where the numbers in the matrix allow for scaling and / or translation of the image.

[0159] Affine matrices are subsets of linear transformations, and they contain information related to translation, rotation, scaling, shearing, and combinations thereof. Affine matrices are used to represent these transformations and apply them to points or vectors in the 3D model 118, particularly to localized 2D interior feature images. For example, affine matrices used for the pose or orientation of the 2D interior geometry indicated by the 2D interior feature image can be combined with linear transformations and translations applied to the 2D interior feature image. In one example, with rigid motion... Peaceful relocation The affine matrix (E) can be defined as:

[0160]

[0161] In one example, for two affine matrices corresponding to a pair of adjacent images ( and This allows for smooth interpolation as linear interpolation. Linear interpolation can be defined as:

[0162]

[0163] In some cases, for two affine matrices and Quaternion representation Spherical linear interpolation (SLERP) can also be used to perform smooth interpolation.

[0164] refer to Figure 6B and Figure 6CThis illustration shows a schematic diagram of superimposing the pose of one or more objects onto a 3D model 118 according to an exemplary embodiment. At this point, the three-dimensional coordinate system (depicted as three-dimensional coordinate systems 610A, 610B, 610C, 610D, collectively referred to as three-dimensional coordinate system 610) of a template tooth model corresponding to each tooth in the tooth group in the 3D model 118 can be located at the position of the tooth in the 3D model 118. In one example, for a tooth, a template tooth model transformed to indicate tooth features can also be located at the position of the tooth in the 3D model 118. Furthermore, the 3D coordinate system of the tooth is associated with the viewing or camera orientation of a 2D internal feature image of the tooth. Subsequently, the three-dimensional coordinate system can be located within the 3D model 118 to provide orientation information defined by the pose. This orientation information is used to superimpose the image onto the 3D model.

[0165] exist Figure 6B The alignment of the 3D coordinate system corresponding to the template object model of this set of teeth is shown. Figure 6C The image shows the alignment of the three-dimensional coordinate system with the 3D model 118 corresponding to the template object model of the set of teeth.

[0166] Subsequently, at 606, processor 106 is configured to overlay a 2D internal feature image of the tooth onto the 3D model 118. In one example, processor 106 is configured to position the 2D internal feature image of the tooth in one or more orientations defined by the object pose of one or more internal geometries. For the internal geometry of the tooth, particularly the object pose (or position and orientation information) 512 of the internal geometry of the template object model, the orientation or orientation can be defined based on the viewing direction from which the 2D internal feature image is captured. Based on the defined orientation, the 2D internal feature image of the tooth can be positioned. To this end, by positioning and orienting the 2D internal feature image of the tooth on the 3D model 118 based on the orientation of the identified internal geometry, the 2D internal feature image is correctly positioned and oriented relative to the virtual 3D model 118. Furthermore, this allows for the seamless integration of multiple 2D internal feature images into the 3D model 118.

[0167] Furthermore, the 2D internal feature image (or adjacent image) corresponding to the adjacent region associated with the tooth can be located based on the intermediate pose determined by interpolation of the poses of two objects corresponding to two adjacent teeth or two adjacent teeth.

[0168] In one example, if the 2D internal feature image is only located at the position corresponding to the tooth in the 3D model 118, the alignment of the image may not be correct when viewing the 3D model 118 from different angles. By locating the 2D internal feature image in an orientation identified by the pose and / or intermediate pose of the transformed template object model, the localization of the 2D internal feature image is more accurate. Furthermore, this correlation-based localization makes the superposition 120 of the 2D internal feature image on the 3D model less susceptible to changes in the viewing angle of the superposition 120.

[0169] In this way, for a specific viewpoint or various viewpoints of the 3D model 118, a 2D internal feature image can be positioned and aligned with the 3D model 118. Subsequently, 2D image information (e.g., the 2D internal composition of a tooth) can be projected onto the 3D model 118. In one example, the 2D internal feature image can be set and aligned onto the 3D model 118 from a viewpoint or perspective of the actual surface of the tooth (i.e., the surface of the primary tooth).

[0170] In some cases, processor 106 is configured to connect 2D internal feature images of the tooth's location. In one example, overlay 120 can be visualized by stitching or joining the located 2D internal feature images or hyperspectral images superimposed on the same or different parts of the tooth for a specific viewing orientation. Each 2D internal feature image can be superimposed on at least some parts of the tooth. The joined or stitched 2D internal feature images can form one or more 2D internal geometric panoramic images of the tooth (hereinafter referred to as 2D panoramic images) based on the corresponding viewing orientation. In one example, the 2D panoramic images can then be superimposed, placed, and aligned onto 3D model 118 based on the corresponding viewing orientation.

[0171] Figure 7A A flowchart 700 illustrates a method for overlaying a 2D internal feature image onto a 3D model 118 of a tooth according to an exemplary embodiment. To this end, overlay 120 of the 2D internal feature image onto a 3D model 120 is performed based on the correlation between the 2D internal geometric features and the 3D model 118. Combined with... Figure 1A , Figure 1B , Figure 1C , Figure 1D , Figure 2 , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 4A , Figure 4B , Figure 5A , Figure 5B , Figure 6A , Figure 6B and Figure 6C Let's use the elements in Figure 7 to explain.

[0172] At 702, processor 106 is configured to generate an offset surface for a 3D model 118 of the tooth. In one example, the offset surface is a mesh. The offset surface can be a grid or mesh used to map pixels of an image to their corresponding positions in the 3D model 118 to generate control points of overlay 120. Each control point has an associated offset that defines how many pixels should be moved or adjusted at that control point.

[0173] An offset mesh can correspond to a 3D model 118. An offset mesh can be generated on a visible light image. An offset mesh can include a grid of points or a set of keypoints strategically placed based on the type of deformation or transformation desired. In one example, an offset vector can be calculated for a control point in the mesh or offset surface. This offset vector can specify how much the pixel at that control point should be offset in the horizontal (X) and vertical (Y) directions. The offset value can be positive or negative and indicates the direction of displacement. The visible light image or visible light information can be distorted based on the offset value assigned to each control point. This can include moving each pixel in the visible light image according to its associated offset vector. According to this example, the offset mesh can be used for image stitching and / or image registration of visible light images. In one example, the offset surface can have a low resolution, and the offset surface can be configured to have a constant range of distances to the teeth or tooth structure.

[0174] refer to Figure 7B and Figure 7C The illustration shows a schematic diagram of an example offset surface 710, which may include 3D information associated with different teeth (i.e., different teeth of the object). In particular, the offset surface 710 may include a mesh or grid for associating pixel values ​​with the offset surface 710 grid.

[0175] like Figure 7B As shown, a symbolic distance field can be used to represent visible light information and / or visible light images to create the offset surface 710. For example, the symbolic distance field of the visible light information may include a data structure for storing information related to the distances of points in the offset surface 710 to their nearest surfaces or the boundaries of the teeth represented by the offset surface 710.

[0176] also, Figure 7C A low-resolution offset surface 710 is shown. Low-resolution offset surfaces can be generated using techniques such as traveling cubes. For example, the low resolution of offset surface 710 can smooth the surface of offset surface 710. The reference frame 712 or the normals generated from offset surface 710 can be consistent.

[0177] return Figure 7AAt 704, the processor 106 is configured to superimpose a 2D internal feature image of the tooth onto the offset surface based on one or more orientations. The one or more orientations may be defined by the object pose of the tooth's internal geometry. Furthermore, the orientation of one or more adjacent regions may be defined by the intermediate pose of the tooth. Additionally, the 2D internal feature image of the tooth may be superimposed on the offset surface 710 or the 3D model 118 based on the intersection points between points on the offset surface 710 and rays extending from the tooth toward the offset surface 710 in a first direction.

[0178] In one example, the interpolated intermediate pose and the object pose for the tooth body or tooth can be superimposed on the position corresponding to the tooth in the offset surface 710. Furthermore, the superimposed intermediate pose and object pose can indicate various orientations for the tooth. For example, the orientation, position, and pose of the offset surface 710 can be used to project or place a 2D internal feature image. Additionally, the tooth orientation can be used to select the optimal view of the tooth body. For this purpose, a 2D internal feature image superimposed on the 3D model 118 or the offset surface 710 can be presented from the selected optimal view or perspective.

[0179] In one example, the projection or placement of a 2D internal feature image onto an offset surface 710 or a 3D model 118 can be performed by generating a ray from the tooth in a first direction or an upward direction. The ray can extend from the tooth surface to a point on the offset mesh 710. Subsequently, at the intersection of the ray and the point on the offset surface 710, one or more 2D internal feature images can be placed, positioned, or superimposed.

[0180] Figure 7D , Figure 7E and Figure 7F A schematic diagram is shown for overlaying a 2D internal feature image onto an offset surface 710 according to an exemplary embodiment.

[0181] exist Figure 7D In this process, based on the corresponding position of the teeth in the offset surface 710, the offset surface 710 is mapped using a 3D coordinate system (depicted as 714A, 714B, and 714C, collectively referred to below as 3D coordinate system 714) of a template tooth model for each tooth. The 3D coordinate system can indicate various orientations or directions for overlaying images onto the offset surface 710.

[0182] exist Figure 7EIn this process, a first layer of images (depicted as 716A, 716B, and 716C, collectively referred to below as first layer image 716) from a 2D internal feature image is positioned or set on an offset surface 710. In one example, the first image layer 716 can be set based on the intersection of a point on the offset surface 710 and a ray extending from the tooth surface. For this purpose, the first image layer 716 may correspond to the innermost layer, for example, to the side opposite to the side where the ray extends toward the offset surface 710. Furthermore, the first image layer for a tooth can be positioned on the offset surface 710 corresponding to the tooth based on an orientation defined by a three-dimensional coordinate system associated with the tooth or a template tooth model.

[0183] exist Figure 7F In this process, multi-layer images (depicted as 718A, 718B, and 718C, hereinafter collectively referred to as multi-layer images 718) derived from 2D internal feature images are positioned or set on offset surface 710. In one example, multi-layer images 718 can be set based on the intersections of points on offset surface 710 and rays extending from the tooth surface. For this purpose, multi-layer images 718 can be projected or superimposed on offset surface 710 to create overlay 120. For example, 2D internal feature images can be projected onto offset mesh 710 or 3D model 118 based on the correlation between orientation defined by the pose and / or intermediate pose of one or more objects and the viewing direction and position of the 2D internal feature images.

[0184] return Figure 7A At 706, processor 106 is configured to display an overlay 120 of a 2D internal feature image of a tooth with offset surface 710. In one example, the overlay can be presented on a display device such as a computing device, smartphone, monitor, etc. Specifically, the 2D internal feature image is not part of the 3D offset surface 710 or surface model 118. Image projection of the 2D internal feature image can be performed by mapping and aligning the image to one or more object poses and / or one or more intermediate poses.

[0185] In some cases, the viewpoint of the displayed overlay 120 can be changed, for example, by a user or viewer to examine other parts of the object's teeth. In this case, processor 106 is configured to update the pose matrix of each of one or more object poses and one or more intermediate poses of the tooth. For example, to update the pose matrix, processor 106 can be configured to map the current overlay to view overlay 120 with the updated viewpoint. Furthermore, processor 106 can be configured to generate an updated overlay of the associated 2D internal geometry of the tooth on the 3D model or offset surface 710 for the updated viewpoint. For example, based on the updated viewpoint, the position of the 2D internal feature image is located within the offset mesh 710 or the 3D model 118.

[0186] Figure 8A , Figure 8B , Figure 8C , Figure 8D , Figure 8E , Figure 8F and Figure 8G This demonstrates a method for merging different spectra to generate 2D internal feature images and / or specific color images. These 2D internal feature images and / or specific color images can be used to enhance the quality of captured images.

[0187] according to Figure 8A A 2D internal feature image 802 can be generated using NIR wavelengths, fluorescence (Fluo) wavelengths, and white light wavelengths. Specifically, the excited green fluorescence information can be subtracted from the NIR wavelength information, i.e., NIR(G) - Fluo(G). Furthermore, the signal contrast can be readjusted after the subtraction. Additionally, white light wavelength information can be subtracted from the output obtained from the previous subtraction. In one example, object edges or tooth edges can be removed to reduce the signal from the tooth edges.

[0188] In some cases, NIR wavelengths can be captured with a blue background. In this case, superimposing the blue NIR information and white light information can produce an image with NIR highlighted by white edges.

[0189] according to Figure 8B A 2D internal feature image 804 can be generated using NIR wavelengths, fluorescence wavelengths, and white light wavelengths. Specifically, the 2D internal feature image 804 can be obtained based on NIR(G) - White(G) + 0.3(NIR(G) - Fluo(G)). In one example, the signal contrast can be readjusted before the second-order summation (i.e., 0.3(NIR(G) - Fluo(G))). Furthermore, object edges or tooth edges can be removed to reduce the signal from the tooth edges.

[0190] according to Figure 8C The 2D internal feature image 806 can be a synthetic image. In this case, image 806 can be generated using the red channel hue from fluorescence, and the green channel can be generated based on NIR(G)-Fluoro(G)-White(G). In addition, the blue channel provides NIR features.

[0191] according to Figure 8D A 2D internal feature image 808 can be synthesized by applying all pixels corresponding to visible light with (R>5 or G>5 or R>5) to a white light image.

[0192] according to Figure 8EA 2D internal feature image 810 can be synthesized by applying all pixels corresponding to visible light with (R>5 and G>5 and R>5) to a white light image.

[0193] according to Figure 8F and Figure 8G 2D internal feature images can be synthetic images. In this case, such as... Figure 8F As shown, image 812 can be generated using a white image with a purple / blue background. Furthermore, a red hue from fluorescence is superimposed on the background as overlay signal 1. Subsequently, as... Figure 8G As shown, image 814 is generated from the composite image using all pixels with (R > 5 and G > 5 and R > 5), and image 814 can be superimposed on superimposed signal 1 as superimposed signal 2.

[0194] Therefore, different color combinations can be formed using white and blue light. Furthermore, color channels or filters corresponding to different colors can be applied to the NIR background image to generate a 2D internal feature image.

[0195] Figure 9 A block diagram 900 of a handheld intraoral scanner 104 according to an exemplary embodiment is shown. The elements of Figures 1 through 8 are explained in conjunction with the diagram. Figure 9 The scanner 104 may include at least one processing unit (hereinafter also referred to as "processing unit 902"), a memory unit 904, a web server 906, a monitoring unit 908, a temporary storage unit 910, a scan feedback unit 912, an input / output (I / O) unit 914, and a communication interface 916.

[0196] Processing unit 902 can be implemented in a variety of different ways. For example, processing unit 902 can be implemented as one or more of various hardware processing devices such as a coprocessor, microprocessor, controller, digital signal processor (DSP), processing elements with or without a DSP, or various other processing circuits including integrated circuits such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontroller units (MCUs), hardware accelerators, and application-specific computer chips. In one embodiment, processing unit 902 can be implemented as a high-performance microprocessor with a series of system-on-chips (SoCs), including a relatively powerful and power-efficient graphics processing unit (GPU) and central processing unit (CPU) and a small form factor. Thus, in some embodiments, processing unit 902 may include one or more processing cores configured to execute independently. Multi-core processors can implement multiprocessing within a single physical package. Additionally or alternatively, processing unit 902 may include one or more processors configured in series via a bus to be able to execute instructions, pipeline, and / or multithread independently.

[0197] In some embodiments, processing unit 902 may be configured to use I / O unit 914 to detect near-infrared and visible light during a scanning process of a tooth of an object (e.g., a patient requiring dental treatment). The detected near-infrared and visible light can be used to generate multiple 2D images 110, such as multiple 2D infrared images 112 and visible light images 114. The multiple 2D images 110 may include images of the tooth structure 124 or the tooth of the object from various angles or viewpoints. For example, processing unit 902 may be configured to generate 3D surface information of the tooth based on the visible light image 114.

[0198] In an exemplary embodiment, the processing unit 902 may communicate with the memory unit 904 via a bus for transmitting information between components of the scanner 104.

[0199] Memory unit 904 may be non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, memory unit 904 may be an electronic storage device (e.g., a computer-readable storage medium) including gates configured to store data (e.g., bits) that can be retrieved by a machine (e.g., a computing device such as processing unit 902). Memory unit 904 may be configured to store information, data, content, applications, instructions, etc., enabling the device to perform various functions according to exemplary embodiments of this disclosure. For example, memory unit 904 may be configured to store detected infrared light and detected visible light after the tooth scanning process is completed. The detected IR light and visible light may be stored as IR information and visible light information, respectively, after the scanning process. In some cases, memory unit 904 may be configured to store compressed IR information and visible light information. In some embodiments, memory unit 904 may be configured to store calibration data required to measure the detected IR light and visible light to generate IR information, visible light information, white light images, IR images, and / or multiple 2D internal feature images. Figure 9As exemplarily shown, memory unit 904 can be configured to store instructions for execution by processing unit 902. Therefore, whether configured by hardware or software methods, or by a combination thereof, processing unit 902 can represent an entity (e.g., physically embodied in circuitry) capable of performing operations according to embodiments of this disclosure while being configured accordingly. Thus, for example, when processing unit 902 is implemented as a microprocessor, processing unit 902 can be hardware with a specific configuration for performing the operations described herein. Alternatively, as another example, when processing unit 902 is implemented as an executor of software instructions, the instructions can specifically configure processing unit 902 to perform the algorithms and / or operations described herein when the instructions are executed. Furthermore, processing unit 902 may include a clock, an arithmetic logic unit (ALU), and logic gates configured to support the operation of processing unit 902.

[0200] Web server 906 can be software, hardware, or a combination thereof, and can be configured to store data and provide data to a web browser associated with processor 106. For example, visible light information and IR information are provided to the web browser of processor 106 via web server 906. Since web server 906 can be accessed by any web browser, the need for processor 106 to install additional software to connect to web server 906 is eliminated. Web server 906 can communicate with one of the communication channels 108 via a web network. In one example, web server 906 and processor 106 can communicate with a public wireless full-duplex communication channel via a web network to send and receive visible light information and IR information. Web server 906 and web browser can communicate via, for example, Hypertext Transfer Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), or File Transfer Protocol (FTP). Once web server 906 and web browser are connected, web server 906 can provide web applications on the web browser.

[0201] The monitoring unit 908 can be software, hardware, or a combination thereof, and can be configured to monitor the bandwidth of one of the communication channels 108 (such as a wireless full-duplex communication channel) through which the scanner 104 and processor 106 are connected. Furthermore, the monitoring unit 908 can be configured to monitor the connection of one of the communication channels 108, wherein the scanner 104 and processor 106 can be connected via the communication channel 108.

[0202] In one embodiment, if the monitoring unit 908 determines that the bandwidth of the communication channel 108 is lower than the minimum bandwidth, the monitoring unit 908 may provide this information to the processing unit 902. The processing unit 902 may downsample the visible light information and IR information based on the received information. In another embodiment, if the monitoring unit 908 determines that the bandwidth of the communication channel 108 is lower than the minimum bandwidth for a longer period than the maximum period, the monitoring unit 908 may provide this information to the processing unit 902. The processing unit 902 may compress the visible light information and NIR information and store them in the storage unit 904. In some embodiments, if the monitoring unit 908 determines that the connection between the scanner 104 and the processor 106 is lost, the monitoring unit 908 may provide this information to the processing unit 902. In this case, the processing unit 902 may compress the visible light information and IR information and store them in the storage unit 904.

[0203] Temporary storage unit 910 can be software, hardware, or a combination thereof, and can be configured to store visible light information and IR information when the bandwidth of communication channel 108 (e.g., a wireless full-duplex communication channel) is determined to be below a minimum bandwidth. When the bandwidth is determined to be above or equal to the minimum bandwidth, temporary storage unit 910 can also send the stored visible light information and IR information to processor 106. Examples of temporary storage unit 910 may include, but are not limited to, random access memory (RAM) or cache memory.

[0204] The scan feedback unit 912 can be software, hardware, or a combination thereof, and can be configured to receive status input from the monitoring unit 908. Based on the received status input, the scan feedback unit 912 can provide scan feedback signals to the user (e.g., a dentist) of the handheld intraoral scanner 104. In one embodiment, the scan feedback signal is used to provide guidance to the user regarding areas of the teeth where the scan quality is low and insufficient visible light information and / or NIR information is received. For example, the scan feedback unit 912 can provide the scan feedback signal as, for example, acoustic feedback, tactile feedback, or visual feedback.

[0205] I / O unit 914 may include circuitry and / or software configured to provide output to a user of the handheld intraoral scanning device 104 and to receive, measure, or sense input information. I / O unit 914 may include a speaker 914A, a vibrator 914B, a projector unit 914C, and one or more sensors 914D. In one embodiment, speaker 914A may be configured to output an acoustic feedback signal to guide the user. Vibrator 914B may be, for example, a transducer configured to convert a scanning feedback signal, which may be an electrical signal, into a mechanical output (such as tactile feedback in the form of vibration) to guide the user.

[0206] It is understood that scanner 104 can be configured to detect IR and visible light that can be reflected from the teeth of a subject. In this regard, projector unit 914C can be configured to output one or more visible or white wavelength pulses, and one or more IR wavelength pulses. For example, the visible and IR wavelength pulses can be directed onto tooth 124 or the tooth to illuminate the teeth of a subject (e.g., a patient). Furthermore, the visible and IR wavelength pulses can be reflected or refracted from the surface and / or internal regions of the tooth. One or more sensors 914D can be configured to detect visible color wavelength pulses and IR wavelength pulses that can be reflected and / or refracted from the surface or internal regions of the tooth. In one example, one or more sensors 914D may include one or more image sensors, such as a camera. For example, the image sensors can be configured to generate a visible light image 114 and an infrared image 112 based on the tooth illumination using IR and visible light.

[0207] The communication interface 916 may include input and output interfaces for supporting communication with the handheld intraoral scanner 104. The communication interface 916 may be a device or circuit embodied in hardware or a combination of hardware and software, configured to receive and / or transmit data to / from the scanner 104. In this regard, the communication interface 916 may include, for example, an antenna (or multiple antennas) and supporting hardware and / or software for enabling communication with a wireless communication network. Additionally or alternatively, the communication interface 916 may include circuitry for interacting with the antenna to induce the transmission of signals via the antenna or to process the reception of signals received via the antenna. In some environments, the communication interface 916 may alternatively or additionally support wired communication. Thus, for example, the communication interface 916 may include a communication modem and / or other hardware and / or software for supporting communication via cable, Digital Subscriber Line (DSL), Universal Serial Bus (USB), or other mechanisms.

[0208] Figure 10 Preprocessing steps for a visible light image and an IR image 1006, according to one example, are shown. In one example, the visible light image may include a white light image 1002 and a blue light image 1004. As described above, the blue light image 1004 can be used to generate fluorescence information or images excited by green and / or red light. Furthermore, in one example, near-infrared wavelength pulses can be used to capture the infrared image 1006.

[0209] For example, the preprocessing steps for the white light image 1002, blue light image 1004, and infrared image 1006 may include contrast adjustment. In one example, the contrast adjustment of the white light image 1002 may include red (R) light contrast adjustment 1008A, green (G) light contrast adjustment 1008B, and blue (B) light contrast adjustment 1008C. Furthermore, the contrast adjustment of the blue light image 1004 may include red light contrast adjustment 1010A, green light contrast adjustment 1010B, and red-green (RG) light contrast adjustment 1010C. Additionally, the contrast adjustment of the IR light image 1006 may include red light contrast adjustment 1012A, green light contrast adjustment 1012B, and blue light contrast adjustment 1012C. Moreover, the preprocessed white light image 1002, blue light image (or fluorescent red and / or green light image) 1004, and IR image 1006 can be used to generate a 2D internal feature image.

[0210] Those skilled in the art will recognize the many modifications and other embodiments of this disclosure set forth herein, and will also appreciate the benefits of such disclosure with respect to the teachings presented in the foregoing description and associated drawings. Therefore, it should be understood that the disclosure is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Furthermore, although the foregoing description and associated drawings depict exemplary embodiments in the context of certain exemplary combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions different from those explicitly described above, as set forth in some of the appended claims, are also contemplated. While specific terminology is used herein, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. An intraoral scanning system (102) configured to generate an overlay (120) of associated 2D internal geometric features on a 3D surface model (118) of one or more teeth (146), the intraoral scanning system comprising: A handheld intraoral scanner (104) is configured to operate with one or more sensors to detect infrared (IR) light and visible light, wherein the one or more sensors include an image sensor; One or more processors (106) operatively connected to the handheld intraoral scanner, the one or more processors being configured to: Visible light information (128) and IR information (130) are received from the one or more sensors. A three-dimensional (3D) surface model (118) of the tooth (146) from the one or more teeth is generated based on the visible light information. Based on the visible light information and the IR information, a plurality of two-dimensional (2D) internal feature images (302, 306, 310, 314) are generated, wherein the plurality of 2D internal feature images indicate the 2D internal geometric features of the tooth. The plurality of 2D internal feature images are processed to associate the 2D internal geometric features of the tooth with at least one reference frame (712) of the tooth in the 3D surface model; as well as Output the superposition of the associated 2D internal geometric features of the tooth on the 3D surface model (120).

2. The intraoral scanning system (102) according to claim 1, wherein, The at least one reference frame (712) of the tooth (146) is perpendicular to two or more planes (148, 150, 152) of the tooth in the 3D surface model (118).

3. The intraoral scanning system (102) according to claim 2, wherein, Each of the two or more planes (148, 150, 152) of the tooth (146) in the 3D surface model (118) includes at least a first plane and a second plane, wherein the first plane and the second plane are aligned with the buccal-lingual plane and the mesiodistal-mesiodistal plane, respectively.

4. The intraoral scanning system (102) according to claim 2 or 3, wherein, The two or more planes (148, 150, 152) of the tooth (146) include at least one of the occlusal plane (518), buccal plane, lingual plane, mesial plane, distal plane, or labial plane.

5. The intraoral scanning system (102) according to any one of the preceding claims, wherein, In order to associate the 2D internal geometry of the tooth (146) with at least one reference frame (712) of the tooth in the 3D surface model (118), the one or more processors are configured to: Align (502) the position of the tooth in the template object model (512) with the position of the tooth in the 3D surface model, wherein the template object model includes a three-dimensional coordinate system (514) that indicates the orientation of the tooth in the 3D surface model; The position of each of the plurality of 2D internal feature images (302, 306, 310, 314) of the tooth is associated (504) with the position of the tooth in the 3D surface model, such that the order in which the visible light information (128) of the 3D surface model of the tooth is received is within a predetermined range of the order in which the IR information (130) of the plurality of 2D internal feature images is received; and The viewing orientation of each of the plurality of 2D internal feature images of the tooth is associated with the 3D coordinate system of the template object model of the tooth (506).

6. The intraoral scanning system (102) according to claim 5, wherein, The coordinate axes of the 3D coordinate system (514) are aligned with the occlusal reference frame (516) of the at least one reference frame (712) of the tooth (146), and wherein the occlusal reference frame is perpendicular to the occlusal plane (518) of the tooth in the 3D surface model (118).

7. The intraoral scanning system (102) according to any one of claims 5 or 6, wherein, The template object model (512) has a corresponding tooth type, wherein the tooth type of the template object model is at least one of the following types: central incisor, lateral incisor, canine, first premolar, second premolar, first molar, second molar, or third molar.

8. The intraoral scanning system (102) according to any one of claims 5 to 7, wherein, The one or more processors (106) are configured to: Based on the plurality of 2D internal feature images (302, 306, 310, 314) and the 3D surface model (118), a trained machine learning model is used to estimate (602) one or more object poses of the internal geometry of the tooth (146), wherein each of the one or more object poses indicates a matrix for converting the template object model (512) into the tooth. Based on smooth interpolation between two object poses indicated by a pair of adjacent images from a plurality of 2D internal feature images from the localization, determine (604) one or more intermediate poses for adjacent regions associated with the tooth, wherein each of the two object poses of the tooth includes an affine matrix; and Based on the one or more intermediate poses of the adjacent regions associated with the tooth and one or more orientations defined by the one or more object poses of the one or more internal geometries, the plurality of 2D internal feature images of the tooth are superimposed (606) on the 3D surface model.

9. The intraoral scanning system (102) according to claim 8, wherein, The one or more processors (106) are configured to: Multiple 2D internal feature images (302, 306, 310, 314) connecting the positioning of the tooth (146) are used to generate one or more 2D internal geometric panoramic images of the tooth based on the viewing orientation; as well as One or more 2D interior geometric panoramic images (118) are superimposed on the 3D surface model.

10. The intraoral scanning system (102) according to claim 8, wherein, The one or more processors (106) are configured to: Generate (702) an offset surface (710) of the 3D surface model (118) of the tooth (146) such that the offset surface is in a constant distance field from the tooth; Based on an orientation defined by at least one of the following, (704) the plurality of 2D internal feature images (302, 306, 310, 314) of the tooth are superimposed on the offset surface: the one or more object poses or the one or more intermediate poses of the tooth, and the intersection points between points on the offset surface and rays extending from the tooth toward the offset surface in a first direction; as well as Display (706) a superposition (120) of multiple 2D internal feature images of the tooth having the offset surface.

11. The intraoral scanning system (102) according to any one of claims 8 to 10, wherein, The one or more processors (106) are configured to: Update the pose matrix of each of the one or more intermediate poses of the tooth (146) to map the superposition (120) with the updated viewpoint. as well as Based on the updated pose matrix, an updated superposition of the associated 2D internal geometric features of the tooth is generated on the 3D surface model (118) for the updated viewpoint.

12. The intraoral scanning system (102) according to any one of the preceding claims, wherein, The one or more processors (106) are configured to: An object mask (402) is generated (402) using visible light information (128) captured from the scanning location, wherein the scanning location corresponds to the location of the one or more sensors; The object mask is applied (404) to the plurality of 2D internal feature images (302, 306, 310, 314) to segment at least a portion of the plurality of 2D internal feature images that indicate non-object information; and Remove (406) the segmentation portions in the plurality of 2D internal feature images that indicate the non-object information.

13. The intraoral scanning system (102) according to any one of the preceding claims, wherein, The plurality of 2D internal feature images (302, 306, 310, 314) include the synthesized 2D internal geometric features of the tooth (146).

14. A method for generating an overlay (120) of associated 2D internal geometric features on a 3D surface model (118) of one or more teeth, the method being implemented using an intraoral scanning system (102) comprising a handheld intraoral scanner (104) and one or more processors (106), the handheld intraoral scanner (104) being configured to operate with one or more sensors to detect infrared (IR) light and visible light, the one or more processors (106) being operatively connected to the handheld intraoral scanner, the method comprising: Visible light information (128) and IR information (130) are received from the one or more sensors. Based on the visible light information, a three-dimensional (3D) surface model (118) of the tooth (146) from the one or more teeth is generated (212). Based on the visible light information and the IR information, a plurality of two-dimensional (2D) internal feature images (302, 306, 310, 314) are generated, wherein the plurality of 2D internal feature images indicate the 2D internal geometric features of the tooth. The plurality of 2D internal feature images are processed to associate the 2D internal geometric features of the tooth with at least one reference frame (712) of the tooth in the 3D surface model; as well as Output the superposition of the associated 2D internal geometric features of the tooth on the 3D surface model (120).

15. A computer-programmable product comprising a non-transitory computer-readable medium having stored thereon computer-executable instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations, the operations including: Receive visible light information (128) and infrared (IR) information (130) from one or more sensors; Based on the visible light information, a three-dimensional (3D) surface model (118) of the tooth (146) is generated (212). Based on the visible light information and the IR information, a plurality of two-dimensional (2D) internal feature images (302, 306, 310, 314) are generated, wherein the plurality of 2D internal feature images indicate the 2D internal geometric features of the tooth. The plurality of 2D internal feature images are processed to associate the 2D internal geometric features of the tooth with at least one reference frame (712) of the tooth in the 3D surface model; as well as Output the superposition of the associated 2D internal geometric features of the tooth on the 3D surface model (120).

Images