Blood and saliva handling for intraoral scanning

Abstract

In a method of processing intraoral scan data, comprising: receiving intraoral scan data, the intraoral scan data comprising one or more intraoral images of a portion of a dental site generated using structured light projection during an intraoral scan of the dental site; generating a three-dimensional (3D) point cloud of the portion of the dental site using the intraoral scan data; detecting one or more regions in the 3D point cloud comprising bodily fluids; removing the one or more regions from the 3D point cloud; updating a 3D surface of the dental site using the 3D point cloud of the portion of the dental site; and outputting the 3D surface of the dental site to a display.

Description

Technical Field

[0001] Embodiments of this disclosure relate to the field of intraoral scanning, and more particularly to a system and method for performing detection, correction, and / or notification of blood and saliva in intraoral scans and / or three-dimensional (3D) models generated from intraoral scans. Background Technology

[0002] In designing restorative procedures for inserting dental prostheses into the oral cavity, in many cases, the dental site where the prosthesis will be implanted should be accurately measured and carefully studied so that the prosthesis (e.g., crown, denture, or bridge) can be correctly designed and its dimensions determined for proper placement. A good fit allows for the proper transfer of mechanical stress between the prosthesis and the jaw, and prevents gingival infection, for example, through the interface between the prosthesis and the dental site.

[0003] Some procedures also require the fabrication of removable restorations to replace one or more missing teeth, such as partial or complete dentures. In these cases, it is necessary to accurately reproduce the surface contours of the area where the tooth is missing so that the resulting restoration fits the area of ​​the missing tooth under uniform pressure on the soft tissue.

[0004] In some practices, the dental site is prepared by the dentist, who then constructs a positive physical model of the site using known methods. Alternatively, the dental site can be scanned to provide 3D data of the site (i.e., in the form of an intraoral image such as a height map). In either case, the virtual or physical model of the dental site is sent to the dental laboratory, which manufactures the restoration based on that model. However, if the model is defective or ambiguous in certain areas, or if the preparation is not optimally configured to accept the restoration, the restoration design may not be optimal. For example, if the insertion path implied by the preparation for a closely-fitting coping would cause the restoration to collide with adjacent teeth, the geometry of the coping must be altered to avoid collisions, which may result in an suboptimal coping design. Furthermore, if the area of ​​the preparation containing the finish line is poorly defined, the finish line may not be correctly determined, and therefore the lower edge of the coping may not be designed correctly. In fact, in some cases, the model is rejected, and the dentist then rescans the dental site or redoes the preparation so that a suitable restoration can be produced.

[0005] In orthodontic procedures, providing a model of one or both jaws can be important. In cases where the orthodontic procedure is a virtual design, a virtual model of the mouth is also beneficial. This virtual model can be obtained by directly scanning the mouth, or by generating a physical model of the dentition and then scanning that model with a suitable scanner.

[0006] Therefore, obtaining a three-dimensional (3D) model of the dental sites in the oral cavity is the initial procedure performed in restorative and orthodontic procedures. When the 3D model is a virtual model, the more complete and accurate the scan of the dental sites, the higher the quality of the virtual model, and thus the greater the ability to design optimal restorations or orthodontic appliances.

[0007] During an intraoral scanning session, blood and / or saliva often accumulate in the patient's mouth. This blood and / or saliva can alter the optical properties of the scanning surface, which can change the shape of the captured dental site (e.g., introduce deformation into the surface of the dental site) and reduce the accuracy of the generated virtual model. Summary of the Invention

[0008] In a first aspect of this disclosure, a method includes: receiving scan data including intraoral images during an intraoral scan of a dental site; identifying representations of foreign bodies in the intraoral images based on analysis of the scan data; modifying the intraoral images by removing representations of foreign bodies from the intraoral images; receiving additional scan data including multiple additional intraoral images of the dental site during the intraoral scan; and generating a virtual three-dimensional (3D) model of the dental site using the modified intraoral images and the multiple additional intraoral images.

[0009] In a second aspect of this disclosure, an intraoral scanning system includes: an intraoral scanner configured to generate intraoral scanning data; and a computing device configured to: receive the intraoral scanning data, the intraoral scanning data including one or more intraoral images of a portion of a dental part generated using structured light projection during an intraoral scan of a dental part; generate a three-dimensional (3D) point cloud of the portion of the dental part using the intraoral scanning data; detect one or more regions in the 3D point cloud that include bodily fluids; remove the one or more regions from the 3D point cloud; update a 3D surface of the dental part using the 3D point cloud of the portion of the dental part; and output the 3D surface of the dental part to a display.

[0010] In a third aspect of this disclosure, an intraoral scanning system includes: an intraoral scanner configured to generate intraoral scanning data; a first computing device configured to: receive the intraoral scanning data, the intraoral scanning data including one or more intraoral images of a portion of a dental part generated using structured light projection during an intraoral scan of a dental part; generate a three-dimensional (3D) point cloud of the portion of the dental part using the intraoral scanning data; and transmit the 3D point cloud to a second computing device; and the second computing device configured to: detect one or more regions in the 3D point cloud that include bodily fluids; remove one or more regions from the 3D point cloud; update a 3D surface of the dental part using the 3D point cloud of the portion of the dental part; and transmit the 3D surface of the dental part to the first computing device.

[0011] In a fourth aspect of this disclosure, an intraoral scanner includes: a plurality of structured light projectors for generating structured light patterns; a plurality of cameras for capturing a plurality of images of a portion of a dental part illuminated by the structured light patterns; and a processor configured to: generate a three-dimensional (3D) point cloud of the portion of the dental part using at least some of the plurality of images; detect one or more regions in the 3D point cloud that include bodily fluids; remove one or more regions from the 3D point cloud; and transmit the 3D point cloud to a computing device for further processing. Attached Figure Description

[0012] The present disclosure is illustrated by way of example rather than limitation in the figures of the accompanying drawings.

[0013] Figure 1 An embodiment of a system for performing intraoral scanning and generating virtual 3D models of dental sites is shown.

[0014] Figure 2A This is a schematic diagram of a handheld intraoral scanner according to some applications of this disclosure, wherein multiple cameras are disposed in a probe at the distal end of the intraoral scanner.

[0015] Figures 2B-2C The schematic diagram includes the positioning configuration of a camera and a structured light projector for an intraoral scanner, which are used in some applications according to this disclosure.

[0016] Figure 2D It is a diagram depicting several different configurations of structured light projectors and cameras in intraoral scanner probes for some applications according to this disclosure.

[0017] Figure 2E This is a flowchart outlining some applications of methods for generating digital three-dimensional images according to this disclosure.

[0018] Figure 2F This is an overview of some applications of this disclosure for performing Figure 2EA flowchart of a specific step in a method.

[0019] Figures 2G to 2J It describes some applications according to this disclosure. Figure 2F A simplified example of the steps is illustrated in the diagram.

[0020] Figure 3 A view of a graphical user interface for an intraoral scanning application according to an embodiment of the present disclosure is shown, the graphical user interface including a 3D surface of the intraoral scanner showing the current field of view of accumulated blood and a combined 2D image.

[0021] Figure 4 A flowchart illustrating a method for identifying and removing representations of body fluids from intraoral scan data according to an embodiment of the present disclosure is shown.

[0022] Figure 5 A flowchart illustrating a method for identifying and removing representations of body fluids from intraoral scan data according to an embodiment of the present disclosure is shown.

[0023] Figure 6 A flowchart of a method for processing body fluids in intraoral scan data according to an embodiment of the present disclosure is shown.

[0024] Figure 7 A flowchart illustrating a method for identifying body fluids in intraoral scan data according to an embodiment of the present disclosure is shown.

[0025] Figure 8A The present disclosure illustrates a model training workflow for training one or more machine learning models to identify the classification of bodily fluids and / or other dental objects in intraoral scan data, according to embodiments of the present disclosure.

[0026] Figure 8B The present disclosure illustrates a model application workflow for classifying bodily fluids and / or other dental objects in intraoral scan data using one or more machine learning models, according to embodiments of the present disclosure.

[0027] Figure 9 A block diagram of an example computing device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0028] This document describes methods and apparatus for identifying bodily fluids such as blood and / or saliva in intraoral scan data and improving the quality of intraoral scans of dental sites including one or more accumulated bodily fluids. The techniques described herein can be used to detect and prevent geometric distortions in intraoral scans and / or three-dimensional (3D) models generated from intraoral scans due to bodily fluids such as blood and saliva bubbles. During intraoral scanning, bodily fluids may gradually accumulate and pool within the patient's mouth. Opaque bodily fluids such as blood may obscure the shape of the underlying surface of a dental site covered by the opaque fluid, thereby reducing the accuracy of the intraoral scan data captured from the dental site. Similarly, transparent bodily fluids such as saliva may also obscure or distort the shape of the underlying surface of a dental site covered by the transparent fluid, also reducing the accuracy of the intraoral scan data captured from the dental site. Detection and correction of saliva can be particularly difficult due to its transparent or translucent properties.

[0029] Body fluids can gradually accumulate at dental sites, and intraoral scans and / or images may capture these fluids. These fluids can refract light (e.g., structured light) and cause crater-like deformations in reconstructed 3D models of dental sites. Including such fluids in intraoral scans and / or images also slows down the processing of these scans and / or images and reduces the accuracy of the virtual 3D model generated from the scans and / or images. Therefore, the embodiments described herein provide techniques for filtering (i.e., removing) body fluids in real-time or near real-time (e.g., as intraoral scans or images are generated). This detection and removal of body fluids from intraoral scan data can be performed before the fluid-occluded area is used for image registration and / or virtual model generation. Alternatively, this detection and removal of body fluids can be performed on the 3D model after its generation. By removing body fluids from these intraoral scans or images before subsequent processing, the speed and / or accuracy of further processing can be improved. For example, it can improve the speed of image or scan registration, improve the accuracy of image or scan registration, reduce the number of image or scan registration failures, improve the accuracy of the generated 3D virtual model, and / or improve the speed of generating the 3D virtual model. Furthermore, the cumbersome process of considering and mitigating problems caused by body fluids can be omitted, thereby further improving the speed of image or scan processing. The accuracy of 3D models of dental sites can also be improved by identifying and removing body fluids from the 3D model after its generation.

[0030] In some embodiments, in addition to removing body fluids from captured intraoral scan data (e.g., intraoral scans and / or images) and / or from 3D models generated from such intraoral scan data, or instead of removing body fluids from captured intraoral scan data (e.g., intraoral scans and / or images) and / or from 3D models generated from such intraoral scan data, an alarm or notification may be generated to notify the user of identified body fluids and / or suspected body fluids. For example, an alarm may be generated if the amount of body fluids and / or suspected body fluids exceeds a threshold. The alarm may include recommendations for the dentist to remove the body fluids (e.g., by wiping or using aspiration) and / or to take other corrective measures. In response to such an alarm, the dentist may pause the intraoral scan and remove the body fluids before continuing the intraoral scan.

[0031] The implementation can improve the accuracy of 3D models of a patient's dental arch and reduce the number of reworks. Rework may occur when a laboratory or other institution receiving a 3D model of a patient's dental arch determines that the 3D model is of such low quality as to be unusable for generating dental appliances (e.g., orthodontic appliances, palatal expanders, dental caps, crowns, bridges, etc.) and notifies the dentist to perform an intraoral scan of the patient's dental arch again to generate an improved 3D model of these arches.

[0032] Some embodiments focus on detecting and preventing geometric distortions caused by saliva bubbles and blood. In some embodiments, an active intraoral scanner illuminates a scanned surface with structured light and triangulates the location of geometric distortions. When structured light passes through fluids such as saliva and blood, reflection and refraction can alter the optical path of the structured light and lead to geometric distortions in the resulting 3D model. Intraoral scanners using structured light projection can be particularly susceptible to inaccuracies caused by bodily fluids because the structured light projector and camera capturing the image of reflected light on the intraoral surface are typically angled relative to each other. In contrast, intraoral scanners using confocal optics to determine depth can have a light projector and camera parallel to the imaging axis, thereby reducing sensitivity to geometric distortions caused by bodily fluids such as saliva bubbles, pooled blood, pooled saliva, etc.

[0033] The embodiments identify bodily fluids in intraoral scan data (e.g., point clouds generated from intraoral scan data) during scanning and enable the removal of the fluids during this scan. This allows dentists to see which areas are affected by the accumulation of bodily fluids such as blood and / or saliva, and to address these areas before continuing the scan with the fluids removed. One advantage of this method is that it reduces the confusion dentists may have regarding handling saliva and / or blood. Since the effects of deformation caused by saliva and / or blood often only become apparent in later stages (e.g., after the scan is complete and a 3D model has been generated), the method described herein can also save dentists time during scanning by drawing attention to areas with excessive blood and / or saliva.

[0034] In some embodiments, a series of operations are performed to identify bodily fluids in intraoral scan data and to remove representations of these bodily fluids from the intraoral scan data. In an example solution for identifying bodily fluids in intraoral scan data generated using structured light projection, a first operation detects suspected saliva pixels in 2D or 3D images (e.g., color 2D images) of a dental site generated by an intraoral scanner. The images may include a set of color images, each generated simultaneously (or substantially simultaneously) by different cameras of the intraoral scanner. A machine learning model, such as a neural network, may be trained from a large set of labeled images to solve a binary image segmentation problem, where each pixel of the input image can be assigned a probability score that estimates the probability that the pixel is a bodily fluid pixel. The probability scores may be captured in a segmentation mask or probability mask for each image. In embodiments, the number of suspected bodily fluid pixels may be compared to one or more thresholds, and an alert may be generated if the number of suspected bodily fluid pixels exceeds one or more thresholds.

[0035] In the second operation, structured light points or features from images of dental sites captured using structured light projection can be projected onto one or more generated segmentation masks. Each image can be generated by one of multiple cameras of an intraoral scanner and can correspond to one of the color images generated by the same camera. The fields of view of the multiple cameras can overlap. Therefore, each structured light point can be seen by multiple cameras. Each structured light point can be a 3D point generated by an algorithm that solves for correspondences between points or features captured in multiple images.

[0036] In embodiments, a combined score (e.g., an average score) can be calculated for each 3D point. The combined score can be a combined probability that the 3D point is a body fluid point. Using combined scores to identify body fluids can improve the stability of body fluid detection. In embodiments, a thresholding operation can be performed to determine whether each of the 3D points is a candidate for body fluid. In some embodiments, each 3D point in the generated point cloud is associated with one or more corresponding pixels in a captured image, where each pixel has a defined probability value indicating the probability that the pixel depicts body fluid. In some embodiments, a voting algorithm is used to determine whether a 3D point is a body fluid point based on the body fluid classification (e.g., probability value) of the pixels associated with the 3D point.

[0037] In the third operation, considering the fact that saliva bubbles and bodily fluids are usually continuous, spatial evidence of bodily fluids can be collected. Specifically, for each suspected bodily fluid point identified from the second operation, the density of additional suspected bodily fluid points around or near that suspected bodily fluid point can be calculated. In an embodiment, the area considered for determining the density can be a circular region centered on the suspected bodily fluid point. In an embodiment, bodily fluid points in the high-density region of bodily fluid points can be identified as bodily fluid points.

[0038] In the fourth operation, for each confirmed fluid point, a containing circle can be calculated. In an embodiment, all points in these containing circles are identified as fluid points. In some embodiments, fluid points can then be excluded from a surface reconstruction operation that introduces pores in these regions of the reconstructed 3D mesh of the dental arch.

[0039] This document describes embodiments of presentations that accurately identify, filter out, or remove bodily fluids from intraoral scan data and / or 3D surfaces generated from intraoral scan data. In particular, embodiments are described that perform exceptionally well in identifying, removing, or filtering bodily fluids from intraoral scan data generated using structured light projection. While embodiments are discussed with reference to areas depicting bodily fluids, the techniques discussed herein regarding bodily fluids can also be applied to other types of dental object categories. For example, the same or similar techniques can be used to identify and remove foreign objects (e.g., dental tools, hands, fingers, etc.), movable tissues (e.g., tongue, lips, cheeks, etc.), excess tissue, etc. In such cases, machine learning models can be trained to identify one or more other types of dental objects to be removed from intraoral scan data and / or 3D surfaces. Therefore, although the following discussion focuses on bodily fluids, it should be understood that the embodiments are also applicable to other types of dental object categories.

[0040] Now refer to the attached diagram, Figure 1An embodiment of a system 101 for performing intraoral scanning and / or generating three-dimensional (3D) surfaces and / or virtual 3D models of dental sites is illustrated, the system identifying and filtering the presentation of bodily fluids. System 101 includes a dental office 108 and optionally one or more dental labs 110. Dental office 108 and dental lab 110 each include computing devices 105, 106, which can be interconnected via a network 180. Network 180 can be a local area network (LAN), a public wide area network (WAN) (e.g., the Internet), a private WAN (e.g., an intranet), or a combination thereof.

[0041] The computing device 105 may be coupled to one or more intraoral scanners 150 (also referred to as scanners) and / or data storage 125 via wired or wireless connections. In one embodiment, multiple scanners 150 in a dental clinic 108 are wirelessly connected to the computing device 105. In one embodiment, the scanners 150 are wirelessly connected to the computing device 105 via a direct wireless connection. In one embodiment, the scanners 150 are wirelessly connected to the computing device 105 via a wireless network. In one embodiment, the wireless network is a Wi-Fi network. In one embodiment, the wireless network is a Bluetooth network, a Zigbee network, or some other wireless network. In one embodiment, the wireless network is a wireless mesh network, examples of which include Wi-Fi mesh networks, Zigbee mesh networks, etc. In one example, the computing device 105 may be physically connected to one or more wireless access points and / or wireless routers (e.g., Wi-Fi access points / routers). The intraoral scanner 150 may include a wireless module, such as a Wi-Fi module, and may join the wireless network via a wireless access point / router.

[0042] The computing device 106 may also be connected to a data storage device (not shown). The data storage device may be a local data storage device and / or a remote data storage device. The computing device 105 and the computing device 106 may each include one or more processing devices, memory, auxiliary memory, one or more input devices (e.g., such as a keyboard, mouse, tablet computer, touch screen, microphone, camera, etc.), one or more output devices (e.g., display, printer, touch screen, speaker, etc.) and / or other hardware components.

[0043] In this embodiment, the scanner 150 includes an inertial measurement unit (IMU). The IMU may include an accelerometer, gyroscope, magnetometer, pressure sensor, and / or other sensors. For example, the scanner 150 may include one or more microelectromechanical systems (MEMS) IMUs. The IMU can generate inertial measurement data (also known as motion data), including acceleration data, rotation data, etc.

[0044] The computing device 105 and / or data storage 125 may be located in a dental clinic 108 (as shown), a dental laboratory 110, or one or more other locations, such as a server farm providing cloud computing services. The computing device 105 and / or data storage 125 may be connected to components located in the same or different locations as the computing device 105 (e.g., components located in a second location away from the dental clinic 108, such as a server farm providing cloud computing services). For example, the computing device 105 may be connected to a remote server, where some operations of the intraoral scanning application 115 are performed on the computing device 105, and some operations of the intraoral scanning application 115 are performed on the remote server.

[0045] Additional computing devices may be physically connected to computing device 105 via wired connections. Additional computing devices may be wirelessly connected to computing device 105 via wireless connections, which may be direct wireless connections or wireless connections via a wireless network. In embodiments, one or more additional computing devices may be mobile computing devices, such as laptops, notebooks, tablets, mobile phones, portable game consoles, etc. In embodiments, one or more additional computing devices may be traditionally stationary computing devices, such as desktop computers, set-top boxes, game consoles, etc. Additional computing devices may act as thin clients of computing device 105. In one embodiment, the additional computing device uses Remote Desktop Protocol (RDP) to access computing device 105. In one embodiment, the additional computing device uses Virtual Network Control (VNC) to access computing device 105. Some additional computing devices may be passive clients that do not control computing device 105 and receive visualizations of the user interface of intraoral scanning application 115. In one embodiment, one or more additional computing devices may operate in master mode, and computing device 105 may operate in slave mode.

[0046] The intraoral scanner 150 may include a wand for optically capturing three-dimensional structures. The intraoral scanner 150 can be used to perform intraoral scans of a patient's oral cavity. An intraoral scanning application 115 running on a computing device 105 can communicate with the scanner 150 to perform the intraoral scan. The result of the intraoral scan may be intraoral scan data 135A, 135B to 135N, which may include one or more intraoral scans or a set of first images (e.g., images generated using structured light projection) and / or a set of second images (e.g., intraoral 2D images). The first and second images may be generated using different illumination conditions. For example, the first image may be an image generated using structured light projection, and the second image may be an image generated using non-structured light projection or illumination (e.g., uniform white light illumination). Each intraoral scan may include a 3D image or point cloud, which may include depth information (e.g., a height map) of portions of the dental site. In embodiments, the intraoral scan includes x, y, and z information. In some embodiments, each 3D intraoral scan includes or is based on a set of 2D images, each 2D image being generated by a different camera of the intraoral scanner using structured light projection. Features of structured light from projection (e.g., points, spots, checkerboard regions, etc.) can be captured from a 2D image set, and a correspondence problem can be solved for one or more of the features to generate a 3D mesh or point cloud.

[0047] In embodiments, intraoral scan data 135A-135N may include color 2D images of dental sites and / or images at specific wavelengths (e.g., near-infrared (NIRI) images, infrared images, ultraviolet images, etc.). In embodiments, the intraoral scanner 150 alternates between generating 3D intraoral scans (e.g., using structured light projection) and generating one or more types of 2D intraoral images (e.g., color images, NIRI images, etc.) during scanning. For example, one or more 2D color images can be generated between the generation of a fourth and fifth intraoral scan by outputting white light and using multiple cameras to capture the reflection of the white light.

[0048] The scanner 150 can transmit intraoral scan data 135A, 135B to 135N to the computing device 105. The computing device 105 can store the intraoral scan data 135A-135N in the data storage 125.

[0049] According to the example, a user (e.g., a practitioner) can perform an intraoral scan on a patient. In doing so, the user can apply scanner 150 to one or more intraoral locations of the patient. The scan can be divided into one or more segments (also called roles). As an example, a segment may include the patient's lower dental arch, the patient's upper dental arch, one or more prepared teeth (e.g., teeth to which dental appliances such as crowns or other dental restorations will be applied), one or more teeth in contact with the prepared teeth (e.g., teeth that are not themselves subject to dental appliances but are located near one or more such teeth or engage with one or more such teeth when the mouth is closed), and / or the patient's occlusion (e.g., scanning the area of ​​engagement of the patient's upper and lower teeth when the patient's mouth is closed). Through this scanner application, scanner 150 can provide intraoral scan data 135A-135N to computing device 105. Intraoral scan data 135A-135N can be provided in the form of intraoral scan datasets, each dataset may include 3D intraoral scans of a region of a specific tooth and / or dental site (e.g., a first 2D image set generated using structured light projection or a 3D point cloud generated from such a first 2D image set), a second intraoral image set (e.g., color images), and so on. In one embodiment, separate intraoral scan datasets are created for the maxillary arch, mandibular arch, patient occlusion, and / or each prepared tooth. Alternatively, a single large intraoral scan dataset (e.g., for the mandibular arch and / or maxillary arch) can be generated. Intraoral scans can be provided from scanner 150 to computing device 105 in the form of one or more points (e.g., one or more pixels and / or groups of pixels), 3D point clouds, 2D image sets, 3D meshes, or other data representations. For example, scanner 150 can provide intraoral scans as one or more 3D point clouds.

[0050] The method of scanning a patient's oral cavity can depend on the procedure being performed. For example, if a maxillary or mandibular denture is to be fabricated, a full scan of the edentulous arch of the mandible or maxilla may be performed. In contrast, if a dental bridge is to be fabricated, only a portion of the entire dental arch may be scanned, including the edentulous area, adjacent prepared teeth (e.g., adjacent teeth), and the opposing arch and dentition. Alternatively, if a dental bridge is to be fabricated, a full scan of the upper and / or lower dental arch may be performed.

[0051] As a non-limiting example, dental procedures can be broadly categorized into restorative and orthodontic procedures, and then further subdivided into specific forms of these procedures. Additionally, dental procedures may include the identification and treatment of gingival diseases, sleep apnea, and intraoral conditions. The term "restorative procedure" specifically refers to any process involving the oral cavity and relating to the design, manufacture, or installation of a dental restoration or its real or virtual model at a dental site (dental region) within the oral cavity, or the design and preparation of the dental site receiving such a restoration. Restorations can include any restoration such as crowns, veneers, inlays, onlays, implants, and bridges, as well as any other artificial partial or complete dentures. The term "orthodontic procedure" specifically refers to any process involving the oral cavity and relating to the design, manufacture, or installation of orthodontic elements or their real or virtual models at a dental site within the oral cavity, or the design and preparation of the dental site receiving such orthodontic elements. These elements can be appliances including, but not limited to, brackets and archwires, retainers, clear aligners, or functional appliances.

[0052] In this embodiment, an intraoral scan of the patient's oral cavity can be performed during a visit to dental clinic 108. For example, the intraoral scan may be performed as part of a semi-annual or annual dental health check. The intraoral scan may also be performed before, during, and / or after one or more dental treatments (e.g., orthodontic treatment and / or restorative treatment). The intraoral scan may be a scan of all or part of the upper and / or lower dental arches and may be performed to collect information for performing dental diagnoses, generating treatment plans, determining the progress of treatment plans, and / or for other purposes. Dental information generated from the intraoral scan (intraoral scan data 135A-135N) may include 3D scan data of all or part of the maxilla and / or mandible (e.g., 3D point clouds and / or image sets captured using structured light projection to generate 3D point clouds), 2D color images, NIRI and / or infrared images, and / or ultraviolet images. The intraoral scan data 135A-135N may also include one or more intraoral scans illustrating the relationship between the upper and lower dental arches. These intraoral scans can be used to determine a patient's occlusion and / or to determine the patient's occlusal contact information. Patient occlusion can include determining the relationship between the teeth in the upper dental arch and the teeth in the lower dental arch.

[0053] For many restorative procedures (e.g., crowns, bridges, veneers, etc.), the patient's existing teeth are ground down into stumps. The ground-down tooth is referred to herein as a preparation tooth, or simply a preparation. A preparation tooth has a marginal line (also called a termination line), which is the boundary between the natural (unground) portion and the prepared (ground) portion of the preparation tooth. Preparation teeth are typically fabricated so that crowns or other restorations can be mounted or placed on them. In many cases, the marginal line of the preparation tooth is subgingival (below the gingival line).

[0054] An intraoral scanner operates by moving scanner 150 within a patient's mouth to capture all viewpoints of one or more teeth. During scanning, in some embodiments, scanner 150 calculates distances to a solid surface. In some embodiments, these distances may be recorded as an image called a "height map" or as a point cloud. Each scan (e.g., an optional height map or point cloud) is algorithmically overlapped with or "stitched" with scans from a previous group to generate a grown 3D surface. Thus, each scan is associated with a spatial rotation or projection to determine how it fits into the 3D surface.

[0055] During an intraoral scan, the intraoral scan application 115 can register and stitch together two or more intraoral scans (e.g., 3D point clouds or meshes) generated so far from the intraoral scan session to generate a grown 3D surface. In one embodiment, performing registration includes capturing 3D data of individual points of the surface in multiple intraoral scans and registering the scans by calculating the transformations between the scans. During the intraoral scan, one or more 3D surfaces can be generated based on the registered and stitched intraoral scans. One or more 3D surfaces can be output to a display, allowing a physician or technician to view their scan progress to date.

[0056] In embodiments, body fluid detection is performed on each intraoral scan and / or 3D surface. For example, body fluid detection can be performed on each intraoral scan before registering it with other intraoral scans and / or 3D surfaces constructed from other intraoral scans. A body fluid detection algorithm or a series of algorithms or operations can be performed to identify body fluids in the intraoral scan. When stitching an intraoral scan (e.g., a 3D point cloud or mesh) to a 3D surface, points in the intraoral scan identified as depicting body fluids may not be included in the 3D surface. In some embodiments, this may include removing or filtering out body fluid points.

[0057] As each new intraoral scan is captured and registered with a previous intraoral scan and / or 3D surface, one or more 3D surfaces can be updated, and the updated 3D surfaces can be output to a display. Depending on one or more viewing modes of the intraoral scanning application, the view of the 3D surfaces can be updated periodically or continuously. In one viewing mode, the 3D surfaces can be continuously updated such that the orientation of the displayed 3D surfaces is aligned with the field of view of the intraoral scanner (e.g., such that a portion of the 3D surface based on the most recently generated intraoral scan is approximately centered on the display or a window of the display), and the user sees what the intraoral scanner is seeing. In another viewing mode, the position and orientation of the 3D surfaces are static, and the image from the intraoral scanner is optionally shown as moving relative to the stationary 3D surfaces. Other viewing modes may include a magnified viewing mode that shows a magnified view of one or more areas of the 3D surfaces (e.g., the intraoral area of ​​interest (AOI)). Other viewing modes are also possible.

[0058] In this embodiment, voids may be formed in the generated 3D surface because fluid points may not have been used. These voids may represent areas where fluid is present in the dental site. To fill these voids, the dentist can use blowing, suction, wiping, or other techniques to remove the fluid from the dental site. The dentist can then continue with intraoral scanning of these areas showing voids in the 3D surface. The newly captured intraoral scans can then be stitched onto the 3D surface, and data to fill the voids can be added.

[0059] In this embodiment, separate 3D surfaces are generated for the maxilla and mandible. This process can be performed in real-time or near real-time to provide an updated view of the captured 3D surfaces during intraoral scanning.

[0060] When a scanning session or part of a scanning session associated with a specific scanning role (e.g., maxillary role, mandibular role, occlusal role, etc.) is completed (e.g., all scans of the dental site or the dental site has been captured), the intraoral scanning application 115 can generate a virtual 3D model of one or more scanned dental sites (e.g., maxilla and mandible). The final 3D model can be a set of 3D points and their connections to each other (i.e., a mesh). To generate the virtual 3D model, the intraoral scanning application 115 can register and stitch together intraoral scans generated from the intraoral scanning session associated with the specific scanning role (e.g., maxillary arch, mandibular arch, occlusion, preparation, etc.). Registration performed at this stage may be more accurate than registration performed during the capture of the intraoral scan and may take longer to complete. In one embodiment, performing scan registration includes capturing 3D data of individual points of a surface in multiple scans and registering the scans by calculating transformations between scans. The 3D data can be projected into the 3D space of the 3D model to form part of the 3D model. By applying appropriate transformations to the points of each registered scan and projecting each scan into 3D space, intraoral scans can be integrated into a common reference frame.

[0061] In one embodiment, registration is performed on adjacent or overlapping intraoral scans (e.g., each consecutive frame of an intraoral video). A registration algorithm is performed to register two adjacent or overlapping intraoral scans and / or to register an intraoral scan with a 3D model. This essentially involves determining a transformation to align one scan with another scan and / or the 3D model. Registration may involve identifying multiple points in each scan (e.g., a point cloud) of the scan pair (or scan and 3D model), performing surface fitting on these points, and using a local search around the points to match the points of the two scans (or scan and 3D model). For example, an intraoral scan application 115 may match points of one scan with the nearest points interpolated on the surface of another scan and iteratively minimize the distance between the matched points. Other registration techniques may also be used.

[0062] The intraoral scanning application 115 can repeatedly register all intraoral scans of a sequence of intraoral scans to obtain a transformation for each intraoral scan, thereby registering each intraoral scan with previous(one or more) intraoral scans and / or a common reference frame (e.g., with a 3D model). The intraoral scanning application 115 can integrate intraoral scans into a single virtual 3D model by applying appropriate, determined transformations to each intraoral scan. Each transformation may include rotations about one to three axes and translations in one to three planes.

[0063] The intraoral scanning application 115 can generate one or more 3D models based on an intraoral scan and can display the 3D models to a user (e.g., a physician) via a graphical user interface (GUI). The physician can then visually examine the 3D models. The physician can virtually manipulate the 3D models (i.e., translate and / or rotate them relative to one or more of three mutually orthogonal axes) via the user interface using appropriate user controls (hardware and / or virtual) to enable viewing the 3D models from any desired orientation.

[0064] In some embodiments, the intraoral scanning application 115 may include a body fluid identifier 116, a scan registration module 118, a user interface 140, and / or a model generation module 120. Modules 116, 118, 120, and 140 may be software modules (e.g., different libraries of a single intraoral scanning application 115) that are components of a single software application, may be different software applications, may be firmware, may be hardware modules, and / or combinations thereof.

[0065] Body fluid identifier 116 can analyze the received intraoral scan data 135A-N using one or more of the techniques described below to identify body fluids in the intraoral images of the intraoral scan data 135A-N. Scan registration module 118 can register the intraoral scan images together (e.g., after body fluid points have been identified in the scan and optionally filtered out or removed), and model generation module 120 can generate a 3D model based on the registered intraoral scan.

[0066] In the example shown, the body fluid identifier 116, the scan registration module 118, and the model generation module 120 are all included in an intraoral scanning application executed on the computing device 115.

[0067] In some embodiments, the body fluid identifier 116, the scan registration module 118, and / or the model generation module 120 are performed on a remote server computing device. In some embodiments, intraoral scans can be transmitted from the scanner 150 or the computing device 105 to the remote server computing device, which can process the intraoral scans and transmit updated intraoral scans and / or 3D surfaces or 3D models, or portions thereof, to the computing device 105 (e.g., for output to a display on the computing device 105). Therefore, in some embodiments, at least some processing can be offloaded from the computing device 105 to the remote server computing device.

[0068] In the example, computing device 105 may receive intraoral scan data, which includes one or more intraoral images of a portion of the dental part generated using structured light projection during an intraoral scan of the dental part. Computing device 105 may then use the intraoral scan data to generate a three-dimensional (3D) point cloud of that portion of the dental part. Alternatively, computing device 105 may receive intraoral scan data including a 3D point cloud generated from one or more images. In either case, computing device 105 may transmit the 3D point cloud to a second computing device. The second computing device may detect one or more regions in the 3D point cloud that include bodily fluids, remove the one or more regions from the 3D point cloud, update the 3D surface of the dental part using the 3D point cloud of that portion of the dental part, and transmit the 3D surface of the dental part to the first computing device. The first computing device may then output the 3D surface to a display.

[0069] In some embodiments, the body fluid identifier 116 is integrated into the scanner 150. For example, the processor of the intraoral scanner 150 can process the captured images and / or intraoral scan data to identify body fluids, and can update the images and / or intraoral scan data before transmitting the images and / or intraoral scan data to the computing device 105 for further processing. In one embodiment, the intraoral scanner 150 is configured to: generate a structured light pattern using a plurality of structured light projectors; capture a plurality of images of a portion of a dental site illuminated by the structured light pattern using a plurality of cameras; generate a three-dimensional (3D) point cloud of that portion of the dental site using at least some of the plurality of images; detect one or more regions in the 3D point cloud that include body fluids; remove the one or more regions from the 3D point cloud; and transmit the 3D point cloud to the computing device for further processing.

[0070] User interface 140 may be a graphical user interface (GUI). User interface 140 may operate during an intraoral scanning session. When scanner 150 generates intraoral images, and these intraoral images are processed by body fluid identifier 116, a view showing a two-dimensional (2D) or 3D representation of the scanned dental site may be output to a display (e.g., the display of computing device 105). Body fluids that have been filtered out by body fluid identifier 116 may not be shown in the 2D or 3D representation.

[0071] Now for reference Figure 2A The figure is a schematic diagram of an intraoral scanner 20 including an elongated rod according to some applications of the present disclosure. In embodiments, the intraoral scanner 20 may correspond to Figure 1The intraoral scanner 150. The intraoral scanner 20 includes multiple structured light projectors 22 and multiple cameras 24, which can be coupled to a rigid structure 26 disposed within a probe 28 located at the distal end 30 of the intraoral scanner 20. In some applications, the probe 28 is inserted into the oral cavity of a subject or patient during an intraoral scanning procedure.

[0072] For some applications, the structured light projectors 22 are located within the probe 28, such that each structured light projector 22 faces the object 32 placed outside the intraoral scanner 20 within its field of view, rather than positioning the structured light projectors in the proximal region of the handheld bar and illuminating the object by reflecting light from a mirror and then onto the object. Alternatively, the structured light projectors can be positioned within the bar, such that they point towards a mirror that can be used to reflect the structured light generated by the structured light projectors onto the object 32. Similarly, for some applications, the cameras 24 are positioned within the probe 28, such that each camera 24 faces the object 32 placed outside the intraoral scanner 20 within its field of view, rather than positioning the camera in the proximal region of the intraoral scanner and viewing the object by reflecting light from a mirror into the camera. Alternatively, one or more cameras can be positioned within the bar, such that they are aligned with a mirror that can reflect light reflected from the object 32 back to the one or more cameras.

[0073] In some embodiments, the intraoral scanner 20 includes a plurality of image capture modules, each including one or more structured light projectors and one or more cameras. Each image capture module may be aligned with a reflector that reflects structured light onto an object 32 and returns light from the object 32 to one or more cameras. In some embodiments, different image capture modules may be associated with their respective individual reflectors. The positioning of the projectors and cameras within the probe 28 allows the scanner to have a large overall field of view while maintaining a low profile probe. In some embodiments, the cameras may be positioned in the proximal region of the handheld device.

[0074] In some applications, each camera 24 has a large field of view β (beta) of at least 45 degrees, at least 70 degrees, at least 80 degrees, 85 degrees, or other values. In some applications, the field of view may be less than 120 degrees, less than 100 degrees, less than 90 degrees, or other values. In one embodiment, the field of view β (beta) of each camera is between 80 and 90 degrees, which may be particularly useful because it provides a good balance between pixel size, field of view and camera overlap, optical quality, and cost. Camera 24 may include an image sensor 58 and an objective optics system 60 including one or more lenses. To achieve close-range focusing imaging, camera 24 may focus on an object focal plane 50 located at a distance of 1 mm to 30 mm, 4 mm to 24 mm, 5 mm to 11 mm, 9 mm to 10 mm, 10 mm to 50 mm, 10 mm to 30 mm, etc., from the lens furthest from the sensor. In some applications, the camera 24 can capture images at a frame rate of at least 30 frames per second (e.g., at least 75 frames per second, at least 100 frames per second, etc.). In other applications, the frame rate can be less than 200 frames per second.

[0075] A large field of view, achieved by combining the individual fields of view from all cameras, improves accuracy by reducing image stitching errors, particularly in edentulous areas where the gingival surface is smooth and may contain fewer sharp, high-resolution 3D features. The larger field of view allows large, smooth features (such as the overall curve of the teeth) to appear in each image frame, improving the accuracy of stitching individual surfaces obtained from multiple such image frames.

[0076] Similarly, each structured light projector 22 may have a large illumination field α (alpha) of at least 45 degrees (e.g., at least 70 degrees). In some applications, the illumination field α (alpha) may be less than 120 degrees, for example, less than 100 degrees. Each structured light pattern may be configured to generate the same or different structured light patterns and may use the same or different light wavelengths. Examples of projectable light patterns include speckles, checkerboard patterns, and patterns of repeating shapes (such as triangles, hexagons, octagons, squares, rectangles, etc.). In some embodiments, the repeating shapes may be arranged in a tessellation pattern. In some embodiments, the light pattern projected by the structured light projector does not change over time. However, in embodiments, the light pattern may be turned on and off (e.g., alternating between white light illumination for color 2D image capture and structured light projection for 3D intraoral scanning capture).

[0077] For some applications, to improve image capture, each camera 24 has multiple discrete preset focus positions, at each focus position the camera focuses on a corresponding object-focal plane 50. Each camera 24 may include an autofocus actuator that selects a focus position from the discrete preset focus positions to improve a given image capture. Additionally or alternatively, each camera 24 includes an optical aperture phase mask that extends the depth of focus of the camera, such that the image formed by each camera remains in focus at all object distances from the lens furthest from the sensor, ranging from 1 mm to 30 mm, 4 mm to 24 mm, 5 mm to 11 mm, 9 mm to 10 mm, 10 mm to 30 mm, 10 mm to 50 mm, etc.

[0078] In some applications, the structured light projector 22 and camera 24 are coupled to the rigid structure 26 in a closely spaced and / or alternating manner, such that (a) a majority of the field of view of each camera overlaps with the field of view of the adjacent camera, and (b) a majority of the field of view of each camera overlaps with the illumination field of the adjacent projector. Optionally, at least 20%, such as at least 50%, such as at least 75%, of the projected light pattern is in the field of view of at least one camera at a focal plane 50 located at least 4 mm away from the lens furthest from the sensor. Due to the different possible configurations of the projectors and cameras, some projected patterns may never be seen in the field of view of any camera, and some projected patterns may be obscured by the object 32 and not be visible as the scanner moves around during scanning.

[0079] The rigid structure 26 can be a non-flexible structure to which the structured light projector 22 and camera 24 are coupled to provide structural stability for the optics within the probe 28. Coupling all projectors and cameras to a common rigid structure helps maintain the geometric integrity of the optics of each structured light projector 22 and each camera 24 under varying environmental conditions, such as mechanical stresses that may be induced by the subject's mouth. Furthermore, the rigid structure 26 helps maintain the stable structural integrity and positioning of the structured light projectors 22 and cameras 24 relative to each other.

[0080] Now for reference Figure 2B-2C This includes schematic diagrams illustrating examples of positioning configurations for the camera 24 and the structured light projector 22, respectively, according to some applications of this disclosure. For some applications, to improve the overall field of view and illumination field of the intraoral scanner 20, the camera 24 and the structured light projector 22 are positioned such that they are not all facing the same direction. For some applications, such as... Figure 2BAs shown, multiple cameras 24 are coupled to a rigid structure 26 such that the angle θ (theta) between two corresponding optical axes 46 of at least two cameras 24 is 90 degrees or less, for example, 35 degrees or less. Similarly, for some applications, such as Figure 2C As shown, multiple structured light projectors 22 are coupled to a rigid structure 26 such that the angle between two corresponding optical axes 48 of at least two structured light projectors 22 is such that... (phi) is 90 degrees or less, such as 35 degrees or less.

[0081] Now for reference Figure 2D The figure depicts several different configurations of the structured light projector 22 and camera 24 within the probe 28 for some applications according to this disclosure. The structured light projector 22... Figure 2D The circle indicates that camera 24 is in Figure 2D Rectangles are used to represent the cameras. Note that rectangles are used to represent cameras because typically the field of view β (beta) of each image sensor 58 and each camera 24 has an aspect ratio of 1:2. Figure 2D Column (a) shows a bird's-eye view of various configurations of the structured light projector 22 and camera 24. The x-axis marked in the first row of column (a) corresponds to the central longitudinal axis of the probe 28. Column (b) shows a side view of various configurations of the camera 24 viewed from a line of sight coaxial with the central longitudinal axis of the probe 28 and substantially parallel to the viewing axis of the intraoral scanner. Similar to Figure 2B As shown, Figure 2D Column (b) shows a camera 24 positioned at an angle of 90 degrees or less relative to each other, such as 35 degrees or less, with the optical axis 46 in the same direction. Column (c) shows a side view of various configurations of the camera 24 as seen from a line of sight perpendicular to the central longitudinal axis of the probe 28.

[0082] In some instances, the farthest side (facing) Figure 2D (in the positive x direction) and the nearest side (towards) Figure 2D The negative x-direction cameras 24 can be positioned such that their optical axes 46 are slightly rotated inward relative to the next nearest camera 24, for example, by an angle of 90 degrees or less, such as 35 degrees or less. More centrally positioned cameras(one or more), i.e., neither the farthest nor the nearest camera 24, can be positioned directly facing outward from the probe, with their optical axes 46 substantially perpendicular to the central longitudinal axis of the probe 28. Note that in row (xi), the projector 22 can be located at the farthest point of the probe 28, so that the optical axis 48 of the projector 22 points inward, thereby allowing more cameras 24 to see the large number of light spots 33 projected from that particular projector 22.

[0083] In this embodiment, the number of structured light projectors 22 in probe 28 can range from, for example, Figure 2D The number of cameras 24 in probe 28 can range from two (as shown in rows (iv) and (v)) to seven or eight (as shown in row (ix)). Note that... Figure 2D The various configurations shown are exemplary and not limiting, and the scope of this disclosure includes additional configurations not shown. For example, the scope of this disclosure includes fewer or more than five projectors 22 located in probe 28 and fewer or more than seven cameras located in probe 28.

[0084] In one exemplary application, an apparatus for intraoral scanning (e.g., intraoral scanner 150) includes an elongated rod comprising a probe at its distal end, at least two light projectors disposed within the probe, and at least four cameras disposed within the probe. Each light projector may include at least one light source and a pattern-generating optics element, the light source being configured to generate light when activated, and the pattern-generating optics element being configured to generate a pattern of light as light is transmitted through it. Each of the at least four cameras may include a camera sensor (also referred to as an image sensor) and one or more lenses, wherein each of the at least four cameras is configured to capture multiple images depicting at least a portion of a projected pattern of light on an intraoral surface. The at least two light projectors and most of the at least four cameras may be arranged in at least two rows, each row being generally parallel to the longitudinal axis of the probe, the at least two rows including at least a first row and a second row.

[0085] In another application, the farthest and closest cameras along the longitudinal axis of at least four cameras are positioned such that their optical axes are at an angle of 90 degrees or less relative to each other when viewed from a line of sight perpendicular to the longitudinal axis. Cameras in the first and second rows can be positioned such that the optical axes of the cameras in the first row are at an angle of 90 degrees or less relative to the optical axes of the cameras in the second row when viewed from a line of sight coaxial with the longitudinal axis of the probe. Apart from the farthest and closest cameras, the remaining cameras in the at least four cameras have optical axes substantially parallel to the longitudinal axis of the probe. Each of the at least two rows may include an alternating sequence of a light projector and cameras.

[0086] In another application, at least four cameras include at least five cameras, at least two light projectors include at least five light projectors, the nearest component in the first row is a light projector, and the nearest component in the second row is a camera.

[0087] In another application, the farthest camera along the longitudinal axis and the nearest camera along the longitudinal axis are positioned such that their optical axes, viewed from a line of sight perpendicular to the longitudinal axis, form an angle of 35 degrees or less relative to each other. The cameras in the first row and the cameras in the second row can be positioned such that, viewed from a line of sight coaxial with the longitudinal axis of the probe, the optical axis of the camera in the first row forms an angle of 35 degrees or less relative to the optical axis of the camera in the second row.

[0088] In another application, at least four cameras can have a combined field of view of 25-45 mm along the longitudinal axis and a field of view of 20-40 mm along the z-axis corresponding to the distance from the probe.

[0089] Back Figure 2A For some applications, there is at least one uniform light projector 118 coupled to the rigid structure 26 (which may be an unstructured light projector that projects light within a certain wavelength range). The uniform light projector 118 can transmit white light onto the object 32 being scanned. At least one camera (e.g., one or all of the cameras 24) uses the illumination from the uniform light projector 118 to capture a two-dimensional color image of the object 32.

[0090] The processor 96 can run a surface reconstruction algorithm that can use detected patterns projected onto the object 32 (e.g., dot patterns, feature patterns such as checkerboard features, etc.) to generate a 3D surface of the object 32.

[0091] Figure 2E This is a flowchart outlining methods for generating digital 3D point clouds or images according to some applications of this disclosure. In blocks 62 and 64, respectively, each structured light projector driving the intraoral scanner projects a light pattern onto an intraoral 3D surface, and each camera driving the intraoral scanner captures an image including at least one feature of the light pattern. In block 66, a correspondence algorithm is run based on stored calibration values, which are referenced below. Figures 2F to 2I Further described, the stored calibration values ​​indicate: (a) the camera ray corresponding to each pixel on the camera sensor of each camera, and (b) the projector ray corresponding to each projection feature from each structured light projector. Once the correspondence is solved, the three-dimensional position on the inner surface of the mouth is calculated in box 68, and a digital three-dimensional image or point cloud of the inner surface of the mouth is generated using the three-dimensional position.

[0092] Now for reference Figure 2F This is an overview of some applications according to this disclosure. Figure 2E A flowchart illustrating the algorithm for the correspondence between the middle frame 66. Also refer to... Figures 2G to 2IBased on the stored calibration values, all projector rays 88 are mapped to all camera rays 86 (box 70) corresponding to all detected features of the projected light pattern, and all intersections 98 of at least one camera ray 86 and at least one projector ray 88 are identified. Figure 2H (Box 72). Figures 2G to 2H These are simplified illustrations of boxes 70 and 72, respectively. Figure 2G As shown, three projector rays 88 and eight camera rays 86 are mapped, these eight camera rays 86 corresponding to a total of eight detected features 33' on the camera sensor 58 of camera 24. Figure 2H As shown, sixteen intersection points 98 are marked.

[0093] exist Figure 2F In boxes 74 and 76, the processing logic determines the correspondence between the projected feature 33 and the detected feature 33', thereby identifying the three-dimensional position of each projected feature 33 on the surface. Figure 2I It uses the simplified example described in the preceding paragraph to depict... Figure 2F The diagrams of boxes 74 and 76 illustrate this. For a given projector ray i, the processing logic "looks at" the corresponding camera sensor path 90 on the camera sensor. Each detected feature j along the camera sensor path 90 will have a camera ray 86 that intersects the given projector ray i at an intersection point 98. The intersection point 98 defines a three-dimensional point in space. The processing logic further "looks at" the camera sensor paths 90' on the corresponding camera sensors of other cameras that correspond to the given projector ray i, and identifies how many other cameras also detected the corresponding feature k on their respective camera sensor paths 90' corresponding to the given projector ray i, whose camera ray 86' intersects the same three-dimensional point in space defined by the intersection point 98. This process is repeated for all detected features j along the camera sensor path 90, and the feature j that the most numerous cameras 24 "agree" to be identified as the feature projected onto the surface from the given projector ray i. That is, the projector ray i is identified as the specific projector ray 88 that produces the detected feature j for which the most numerous other cameras detected the corresponding feature k. Therefore, the three-dimensional position on the surface is calculated for this feature.

[0094] For example, such as Figure 2IAs shown, all four cameras detect corresponding features on their respective camera sensor paths corresponding to projector ray i. The camera rays of these features intersect projector ray i at intersection point 98, which is defined as the intersection of camera ray 86 corresponding to the detected feature j and projector ray i. Therefore, all four cameras are considered to "agree" that feature 33 projected by projector ray i exists at intersection point 98. However, when this process is repeated for the next feature j', none of the other cameras detect the corresponding feature on their respective camera sensor paths corresponding to projector ray i, where their camera rays intersect projector ray i at intersection point 98', which is defined as the intersection of camera ray 86'' (corresponding to the detected feature j') and projector ray i. Therefore, referring to... Figure 2J Only one camera is considered to "agree" that feature 33, projected by projector ray i, exists at intersection 98', while all four cameras "agree" that feature 33, projected by projector ray i, exists at intersection 98. Therefore, projector ray i is identified as the specific projector ray 88 that generates the detected feature j by projecting feature 33 onto the surface at intersection 98. Figure 2F The box 78 and such Figure 2I As shown, the three-dimensional position 35 on the inner surface of the opening is calculated at intersection 98.

[0095] In some embodiments, processor 96 may combine at least one 3D scan captured using illumination from structured light projector 22 with multiple intraoral 2D images captured using illumination from uniform light projector 118 to generate a digital 3D image of the intraoral 3D surface. The combination of structured light and uniform illumination enhances the overall capture of the intraoral scanner and helps reduce the number of options processor 96 needs to consider when running a correspondence algorithm for detecting depth values ​​of object 32. In one embodiment, the intraoral scanner and correspondence algorithm described in U.S. Patent No. 11,534,273, issued December 27, 2022, are used. The entire contents of U.S. Patent No. 11,534,273, issued December 27, 2022, are incorporated herein by reference. In one embodiment, the intraoral scanner and correspondence algorithm described in U.S. Patent No. 11,563,929, issued January 24, 2023, are used. The entire contents of U.S. Patent No. 11,563,929, issued January 24, 2023, are incorporated herein by reference. In this embodiment, the processor 92 may be Figure 1 The processor of the computing device 105. Alternatively, the processor 92 may be a processor integrated into the intraoral scanner 20.

[0096] For some applications, all data points acquired at a specific time are used as rigid point clouds, and multiple such point clouds are captured at a frame rate of more than 10 captures per second. Registration algorithms, such as Iterative Closest Point (ICP), are then used to stitch the multiple point clouds together to create a dense point cloud. Surface reconstruction algorithms can then be used to generate a representation of the surface of object 32.

[0097] For some applications, at least one temperature sensor 52 is coupled to the rigid structure 26 and measures the temperature of the rigid structure 26. A temperature control circuit 54 disposed within the intraoral scanner 20 (a) receives data from the temperature sensor 52 indicating the temperature of the rigid structure 26, and (b) activates a temperature control unit 56 in response to the received data. The temperature control unit 56 (e.g., a PID controller) maintains the probe 28 at a desired temperature (e.g., between 35 and 43 degrees Celsius, between 37 and 41 degrees Celsius, etc.). Maintaining the probe 28 above 35 degrees Celsius (e.g., above 37 degrees Celsius) reduces fogging of the glass surface of the intraoral scanner 20 through which the structured light projector 22 projects and the camera 24 views when the probe 28 enters the intraoral cavity, which is typically at or above 37 degrees Celsius. Maintaining the probe 28 below 43 degrees Celsius (e.g., below 41 degrees Celsius) prevents discomfort or pain.

[0098] In some embodiments, heat can be dissipated from the probe 28 via a heat-conducting element 94 (e.g., a heat pipe) disposed within the intraoral scanner 20, such that the distal end 95 of the heat-conducting element 94 contacts the rigid structure 26, and the proximal end 99 contacts the proximal end 100 of the intraoral scanner 20. Heat is thus transferred from the rigid structure 26 to the proximal end 100 of the intraoral scanner 20. Alternatively or additionally, a fan disposed in the handle region 174 of the intraoral scanner 20 can be used to dissipate heat from the probe 28.

[0099] Figures 2A-2D An intraoral scanner of one type that can be used in embodiments of this disclosure is shown. However, it should be understood that the embodiments are not limited to intraoral scanners of the type shown.

[0100] In some embodiments, an intraoral scanner performing confocal focusing can be used to determine depth information. Such an intraoral scanner may include a light source and / or illumination module that emits light (e.g., a focused beam or array of focused beams). The light passes through a polarizer and then through a one-way mirror or beamsplitter (e.g., a polarizing beamsplitter) that allows the light to pass through. The light may pass through a pattern before or after the beamsplitter to pattern the light. Along the optical path of the light after the one-way mirror or beamsplitter are optical devices, which may include one or more lens groups. Any lens group may include only a single lens or multiple lenses. One of the lens groups may include at least one movable lens.

[0101] Light can pass through an endoscopic probe member, which may include a rigid light-transmitting medium. This medium may be a hollow object defining a light transmission path within it, or an object made of a light-transmitting material, such as a glass body or tube. In one embodiment, the endoscopic probe member includes a prism, such as a folding prism. At its end, the endoscopic probe member may include a mirror of the type ensuring total internal reflection. This mirror can thus guide an array of light beams toward a dental segment or other object. The endoscopic probe member thus emits light, which optionally passes through one or more windows and then illuminates the surface of an intraoral object.

[0102] The light can comprise an array of beams arranged in the XY plane of a Cartesian coordinate system, propagating along the Z-axis, which corresponds to the imaging or viewing axis of the intraoral scanner. Because the surface on which the incident beam strikes is not flat, the illuminated spot may be at different (X... i Y i The positions are shifted relative to each other along the Z-axis. Therefore, while a spot at one position may be at the focal point of a confocal focusing optics, spots at other positions may be out of focus. Consequently, the intensity of the returning beam from the focused spot will be at its peak, while the intensity of other spots will deviate from the peak. Therefore, for each illuminated spot, multiple measurements of intensity are taken at different positions along the Z-axis. For each such (X... i Y i By determining the position, we can obtain the derivative of the intensity with respect to the distance (Z), and the maximum derivative of Z is obtained. i Z0 is the in-focus distance.

[0103] Light is reflected from an object inside the mouth, passes through a window (if present), reflects back from a mirror, passes through an optical system, and is reflected by a beam splitter onto a detector. The detector is an image sensor with a matrix of sensing elements, each sensing element representing a pixel of a scan or image. In one embodiment, the detector is a charge-coupled device (CCD) sensor. In another embodiment, the detector is a complementary metal-oxide-semiconductor (CMOS) type image sensor. Other types of image sensors may also be used as the detector. In one embodiment, the detector detects the light intensity at each pixel, which can be used to calculate height or depth.

[0104] Alternatively, in some embodiments, an intraoral scanner utilizing stereoscopic imaging is used to determine depth information.

[0105] Figure 3A view of a graphical user interface 300 for an intraoral scanning application according to an embodiment of the present disclosure is shown. The graphical user interface 300 includes a 3D surface 310 showing the current field of view (FOV) of the intraoral scanner and one or more 2D images 320-325. The 3D surface 310 is generated by registering and stitching together multiple intraoral scans captured during an intraoral scanning session. With each new intraoral scan generated, the scan is registered to the 3D surface and then stitched together. Thus, the 3D surface becomes increasingly accurate with each intraoral scan until the 3D surface is complete. A 3D model can then be generated based on the intraoral scans. In some embodiments, multiple images 320-325 are combined into a composite image. In some embodiments, a single image (e.g., an image generated by one of multiple cameras of the intraoral scanner) is used. In some embodiments, multiple images are used, each generated by a different camera. In some embodiments, processing logic determines which camera's image is presented in the user interface. The selected camera can be changed during intraoral scanning.

[0106] 2D color images 320 and 321 show blood 372A-B pooling at a portion of multiple teeth. Therefore, data from the obscured portions of the teeth in the intraoral scan were not used to generate the 3D surface 310. As shown, the 3D surface 310 includes gaps 350 corresponding to the areas of the teeth obscured by blood 372A-B. In an embodiment, processing logic can detect blood 372A-B, determine if the blood volume exceeds a threshold, output a notification 352 indicating that excessive blood (or saliva) has been detected, and / or suggest that a physician remove the blood.

[0107] In one embodiment, as shown, the scan segment indicator 330 may include an upper dental arch segment indicator 332, a lower dental arch segment indicator 334, and an occlusal segment indicator 336. When scanning the upper dental arch, the upper dental arch segment indicator 332 may be active (e.g., highlighted). Similarly, when scanning the lower dental arch, the lower dental arch segment indicator 334 may be active, and when scanning the patient's occlusion, the occlusal segment indicator 336 may be active. The user can select specific segment indicators 332, 334, and 336 to display the 3D surface associated with the selected segment. The user can also select specific segment indicators 332, 334, and 336 to indicate that a scan of that specific segment will be performed. Alternatively, the processing logic may automatically determine the segment to be scanned and may automatically select that segment to make it active.

[0108] The GUI may also include a taskbar with multiple operating modes or intraoral scanning stages. Selecting patient selection mode 340 allows the physician to input patient information and / or select patients already entered into the system. Selecting scan mode 342 enables an intraoral scan of the patient's oral cavity. After the scan is complete, selecting post-processing mode 344 prompts the intraoral scanning application to generate one or more 3D models based on the intraoral scan and / or 2D images generated during the intraoral scan, and optionally performs analysis of the 3D models. Examples of analyses that can be performed include analysis to detect areas of interest, evaluate the quality of the 3D models, etc. Once the physician is satisfied with the 3D models, they can generate orthodontic and / or restorative prescriptions. Selecting prescription fulfillment mode 346 allows the generated orthodontic and / or restorative prescriptions to be sent to a laboratory or other institution to generate restorative devices (e.g., crowns, bridges, dentures, etc.) or orthodontic devices (e.g., orthodontic appliances).

[0109] Figures 4 to 8B Methods associated with intraoral scanning and the identification and processing of bodily fluids generated during intraoral scanning are illustrated. These methods can be performed by processing logic including hardware (e.g., circuitry, dedicated logic, programmable logic, microcode, etc.), software (such as instructions running on a processing device), or a combination thereof. Various embodiments can be derived from... Figure 1 The processing logic executed on the computing device 105 is used to perform these methods. Additionally or alternatively, the operation of these methods can be performed by the processing device of the intraoral scanner 150. For example, before sending intraoral scan data to the computing device 105, the intraoral scanner 150 can determine 3D points from the image set, determine which 3D points represent body fluids, filter out points representing body fluids, etc. Alternatively, the computing device 105 can receive images from the intraoral scanner 150 and can perform one or more of these operations. In some embodiments, the intraoral scanner 150 and the processing device 105 can separate the operations performed for determining 3D points and / or identifying body fluid points, such that some operations are performed at the intraoral scanner 150 while others are performed at the computing device 105. These methods can be performed in real-time or near real-time as the intraoral scan is generated by and / or received from the scanner.

[0110] Figure 4 A flowchart of a method 400 for identifying and removing representations of bodily fluids from intraoral scan data according to an embodiment of the present disclosure is shown. At block 402 of method 400, an intraoral scanner captures intraoral scan data. In embodiments, the intraoral scanner can transmit the intraoral scan data to a computing device via a wired or wireless connection.

[0111] The captured intraoral scan data may include one or more first 2D image sets captured using structured light projection. Each image in the first image set may be generated by a different camera of the intraoral scanner. Each camera may have a unique field of view, a unique position within the intraoral scanner, a unique orientation, etc. The intraoral scanner may be configured such that the fields of view of multiple cameras overlap. This allows multiple cameras to generate images of the same part of the dental site from different perspectives. In an embodiment, one or more structured light projectors of the intraoral scanner project a structured light pattern onto the dental site. Each image in the image set may capture features of the projected structured light pattern. The processing logic of the intraoral scanner and / or computing device may then determine which captured features in different images correspond to the same projected features of the structured light pattern (e.g., a correspondence problem between images may be solved). By solving the correspondence problem, the three-dimensional positions of some or all of the captured features in the structured light pattern can be determined, thereby obtaining a 3D point cloud.

[0112] The captured intraoral scan data may additionally include one or more second image sets (e.g., 2D color images generated under white light, such as unstructured white light). The intraoral scanner may alternately capture images under structured light conditions to determine 3D data of the scanned surface and under unstructured light (e.g., white light) conditions to determine color information about the scanned surface. Therefore, each first image set generated under structured light conditions may correspond to a second image set generated under unstructured light and / or white light conditions. The first image set generated under structured light conditions may be captured temporally close to the second image set generated under unstructured light and / or white light conditions. Thus, the first and second image sets (e.g., white light images and structured light images) can be obtained from the same or approximately the same location of the intraoral scanner relative to the scanned surface.

[0113] At box 404, the processing logic of the intraoral scanner and / or computing device identifies suspected body fluid pixels in a second image set (e.g., a 2D color image set) of the generated intraoral scan data. In some embodiments, the second image set is not a color image set, but a monochrome image (e.g., a black and white image), an NIR image, an image generated under unstructured light at one or more wavelengths, etc. In some embodiments, suspected body fluid pixels are determined by processing each image in the second image set (e.g., a 2D color image or other images generated under unstructured light conditions) using a trained machine learning model (e.g., a neural network). Examples of neural networks that can be used include encoder-decoder architectures, u-net architectures, or other neural network architectures. The machine learning model can output a mask (e.g., a segmentation mask) for each pixel, which identifies the probability that the pixel depicts body fluid. The machine learning model (or one or more other machine learning models) can also output other dental classification probabilities for pixels, such as the probability that the pixel depicts teeth, gums, excess tissue, mobile tissue, foreign bodies, etc. Thus, in some embodiments, one or more trained machine learning models can output multiple masks, each mask providing a different dental object classification. In the example, one mask can identify pixels representing teeth and pixels representing gums, and another mask can identify pixels representing bodily fluids and pixels not representing bodily fluids. Therefore, in one example, pixels can be categorized as teeth with bodily fluids, gums with bodily fluids, teeth without bodily fluids, gums without bodily fluids, and so on.

[0114] In this embodiment, separate machine learning models can be used to detect saliva and blood. Saliva and blood may have different optical properties, so specialized machine learning models trained to identify specific categories of bodily fluids can be used to detect saliva and blood separately. Alternatively, a single machine learning model can be trained to identify both saliva and blood. It should be understood that the above and below discussion regarding the detection and processing of bodily fluid pixels, points, regions, etc., only covers blood, saliva, mixtures of blood and saliva, and / or other types of bodily fluids (such as pus).

[0115] At box 406, the processing logic of the intraoral scanner and / or computing device maps suspected bodily fluid pixels to points in a 3D point cloud generated from an image set of scan data generated using structured light projection. The processing logic may determine pixels in the 2D color image corresponding to pixels in the image generated using structured light projection (e.g., pixels depicting captured features of the projected light pattern). In some embodiments, pixels in the 2D color image are mapped to the same pixels in the image generated by the same camera using structured light projection. In some embodiments, the movement of the intraoral scanner relative to the dental site between the generation of the 2D color image and the image generated using structured light projection is determined or estimated, and pixels in the 2D color image are mapped to pixels in the image generated using structured light projection based on this relative movement.

[0116] By mapping a second image set (e.g., a set of color 2D images) to a first image set (e.g., an image set generated using structured light projection), the probability of pixels in the second image set depicting one or more dental object categories (e.g., the probability of depicting bodily fluids) can also be mapped to pixels in the corresponding first image set generated using structured light projection. Based on this mapping, one or more probabilities (e.g., the probability of depicting bodily fluids, the probability of depicting teeth, the probability of depicting gums, etc.) can be assigned to features of each captured light pattern in each image of the first image set. Since the correspondence problem has been solved for the first image set generated using structured light projection, the probabilities of pixels representing one or more dental object categories can be mapped to 3D points based on the solution to the determined correspondence problem.

[0117] As previously described, one or more 3D points in a generated 3D point cloud can be determined from multiple images in an image set. Each of these images may have one or more pixels mapped to one or more 3D points, and each of these pixels may have its own associated probability (i.e., the probability of depicting one or more dental object categories). Therefore, some 3D points may have multiple probabilities assigned to them. In the example, a 3D point determined from three images might have three different probabilities: that the 3D point represents blood, that the 3D point represents saliva, that the 3D point represents a tooth, that the 3D point represents gums, etc. The processing logic can combine multiple probabilities into a single combined probability score. This combination of multiple probabilities can be based on a weighted or unweighted average of the probabilities associated with the 3D point. Alternatively, other statistical techniques can be used to determine the combined probability, such as determining a median probability value. In some embodiments, a voting algorithm is used to generate the combined probability score, or suspected body fluid points are identified in other ways. In an embodiment, 3D points whose probability of depicting body fluid exceeds a probability threshold are identified as suspected body fluid points. Similarly, 3D points depicting teeth with a probability exceeding a threshold can be identified as tooth points, 3D points depicting gums with a probability exceeding a threshold can be identified as gum points, and so on. The probabilities determined for pixels by machine learning models may have a certain error rate. However, by using dental category probabilities from multiple images for 3D points, the individual errors associated with each image can be reduced, and the accuracy of the dental category probabilities for these 3D points can be improved.

[0118] At box 408, the processing logic of the intraoral scanner and / or computing device can update suspected body fluid points based on one or more criteria. In one embodiment, the criterion for confirming or denying a suspected body fluid point is the density or number of surrounding body fluid points. In some embodiments, the image generated by the intraoral scanner has sparsely distributed points. For example, if the projected light pattern captured in the image is a sparse point pattern, the image has sparse information about the captured features of the light pattern. Therefore, there may not be enough information to confirm or deny a suspected body fluid point from a single image set. In some embodiments, a moving window is determined for 3D point dental category probability information. This moving window may include a record (e.g., a mask) of 3D point dental category probabilities for one or more 3D point clouds generated before the current 3D point cloud being processed and one or more 3D point clouds generated after the current 3D point cloud being processed. For example, the moving window may extend into the future or into the past by a few milliseconds (e.g., 1-3 milliseconds). The moving window for the current 3D point cloud may be based in part on one or more 3D point clouds generated from an image set captured after the image set used to generate the current 3D point cloud. Therefore, the execution of the operation in box 408 may be delayed until these additional image sets are generated and processed.

[0119] For each suspected body fluid point in the 3D point cloud being processed, the processing logic can determine the quantity or density of suspected body fluid points in the vicinity or surrounding area. In one embodiment, the processing logic determines the quantity of other suspected body fluid points within a threshold distance (e.g., 0.5 mm, 1 mm, 1.5 mm, etc.) from the suspected body fluid point in question. The size of the threshold distance can be optimized for a specific domain and / or device; the values ​​provided are merely examples. If the quantity or density of suspected body fluid points in the vicinity or surrounding area meets or exceeds the threshold, the suspected body fluid point can be identified as a body fluid point. The threshold number of points can be optimized for a specific domain and / or device. Example threshold numbers of points include 5 points, 10 points, 15 points, 20 points, 25 points, 30 points, 100 points, etc. If the quantity or density of suspected body fluid points in the vicinity or surrounding area is below the threshold, the suspected body fluid point can be determined not to be a body fluid point. In some embodiments, statistical data on suspected body fluid points are collected over time. Clustering can then be performed to identify clusters of suspected body fluid points that meet one or more criteria (e.g., points with a probability greater than a threshold of being body fluid), confirming that these suspected body fluid points in the clusters are indeed body fluid points. In some embodiments, clustering algorithms such as density-based spatial clustering of applications with noise (DBSCAN) are used for this purpose.

[0120] When bodily fluids accumulate in a person's mouth, they can form pools, one or more bubbles, and / or other physical forms, which are represented in multiple pixels or points in a 3D point cloud. Therefore, bodily fluid points in a 3D point cloud should be adjacent to other bodily fluid points. Suspected bodily fluid points that are not closely adjacent to other bodily fluid points may not actually depict bodily fluid. The accuracy of bodily fluid detection can be improved by using the density or amount of suspected bodily fluid points around the suspected bodily fluid point in question as a criterion for confirming the suspected bodily fluid point.

[0121] At box 414, the processing logic of the intraoral scanner and / or computing device can perform one or more operations to update the body fluid point. In one embodiment, for each confirmed body fluid point, the processing logic draws a shape or boundary around the body fluid point in a 3D point cloud. The shape can be a 2D shape, such as a circle or ellipse, or a 3D shape, such as a sphere, hemisphere, or other circular shape. The shape can have a set size and can be centered on the body fluid point. For example, the size of the shape can be 0.5 mm, 1 mm, 1.5 mm, or other sizes. The size and / or shape of the boundary can be optimized for a specific area and / or device. In some embodiments, the shapes / boundaries drawn around the body fluid point are combined, and the superposition of shapes / boundaries can produce general shapes, organic shapes, or random shapes.

[0122] At box 416, the processing logic of the intraoral scanner and / or computing device can ultimately determine the fluid region (e.g., including multiple fluid points). In an embodiment, any point within a drawn shape can be updated so that it is also classified as a fluid point. By updating the fluid point classification using the drawn shape, the processing logic can ensure that the fluid region is generally continuous (e.g., avoiding situations where points classified as non-fluid exist between other points classified as fluid points), which corresponds to how these fluids form in a person's mouth in real life. Therefore, updating the fluid point classification using the shape surrounding the classified fluid points further improves the accuracy of fluid detection.

[0123] At box 418, the processing logic of the intraoral scanner and / or computing device may remove fluid points from the 3D point cloud. Alternatively, the processing logic may simply ignore or filter out fluid points without actually removing them from the 3D point cloud. If the operations of boxes 404-418 are performed by the intraoral scanner, the 3D point cloud may be provided to the computing device for further processing. Alternatively, captured intraoral scan data 402 may have been transmitted to the computing device, and the computing device may have performed the operations of boxes 404-418.

[0124] At box 420, the processing logic can register and stitch a 3D point cloud with other 3D point clouds and / or generated 3D surfaces (e.g., 3D surfaces generated by registering and stitching together multiple 3D point clouds, each associated with or constituting a discrete intraoral scan). Fluid points in the 3D point cloud may not be used for registering and / or stitching the 3D point cloud with other 3D point clouds and / or 3D surfaces (e.g., 3D meshes). Therefore, the 3D surface may include voids corresponding to detected fluid regions (e.g., fluid points).

[0125] At box 422, the processing logic can display the generated 3D surface. The displayed 3D surface may include one or more voids caused by the removed fluid spots. In some embodiments, these voids may be highlighted or emphasized on the display. This can draw the dentist's attention to these voids. When a threshold amount of fluid and / or suspected fluid is detected (e.g., when a threshold number of fluid spots is detected, when a fluid area with a threshold size is detected, etc.), the user interface can also output a notification or alert to the display. The alert may include a suggestion to the dentist to wipe or otherwise remove fluid from the patient's mouth.

[0126] Figure 5A flowchart 500 of a method for identifying and removing representations of bodily fluids from intraoral scan data according to an embodiment of the present disclosure is shown. At block 502 of method 500, a computing device receives intraoral scan data including a first image set generated using structured light projection and a corresponding second image set (e.g., a 2D color image set).

[0127] At box 504, the processing logic uses one or more trained machine learning models to process the second image set to generate a mask, such as a segmentation mask, for each image in the second image set. For each pixel in the image, the mask generated for the image may include the probability that the pixel is a bodily fluid pixel (e.g., a blood pixel or a saliva pixel).

[0128] At box 506, the processing logic generates a 3D point cloud from a first set of intra-aperture images generated using structured light projection. Each image in the first set may have been generated by a different camera at the same time. The 3D point cloud can be generated by solving a corresponding problem, where the solution provides triangulation data indicating the depth of each 3D point.

[0129] At box 508, the processing logic maps the probabilities from the mask associated with the images in the second image set to points in the 3D point cloud.

[0130] At box 510, the processing logic determines points in the 3D point cloud that are suspected body fluid points based on the mapping probabilities. This may include determining the average probability, median probability, or some other statistical combination of probabilities for each 3D point cloud based on the probability of the pixels mapped to these 3D points.

[0131] At block 512, the processing logic updates the suspected body fluid points based on the density of neighboring suspected body fluid points (e.g., optionally using a clustering algorithm). In an embodiment, neighboring suspected body fluid points may come from the current 3D point cloud and / or one or more additional 3D point clouds generated from intraoral scan data captured before and / or after the intraoral scan data used to generate the current 3D point cloud.

[0132] At box 514, the processing logic draws a shape around the fluid point in the 3D point cloud. This shape can be a 2D shape, such as a circle or ellipse, or a 3D shape, such as a sphere, hemisphere, or other circular shape.

[0133] At box 516, the processing logic makes a final determination of the body fluid region (e.g., including multiple body fluid points). In an embodiment, any points within the drawn shape can be updated so that these points are also classified as body fluid points.

[0134] At box 518, the processing logic removes fluid points from the 3D point cloud. Alternatively, the processing logic can simply ignore or filter out fluid points without actually removing them from the 3D point cloud.

[0135] At box 520, the 3D point cloud is registered and stitched together with other 3D point clouds and / or generated 3D surfaces or models (e.g., generated by registering and stitching together multiple 3D point clouds, each associated with or constituting a discrete intraoral scan). Fluid points in the 3D point cloud may not be used for registering and / or stitching the 3D point cloud with other 3D point clouds and / or 3D surfaces (e.g., 3D meshes). Therefore, the 3D surface may include voids corresponding to detected fluid regions (e.g., fluid points).

[0136] Figure 6 A flowchart of a method 600 for processing bodily fluids in intraoral scan data according to an embodiment of the present disclosure is shown. At block 602 of method 600, processing logic receives intraoral scan data of a portion of a dental site. The intraoral scan data may include a first image set generated using structured light projection (wherein, for each image set, each image may be generated by a different camera of the intraoral scanner) and a second image set (e.g., a color 2D image set, where each image may be generated by a different camera of the intraoral scanner).

[0137] At box 604, the processing logic uses intraoral scan data to generate a 3D point cloud of that portion of the dental site. In some embodiments, the 3D point cloud is determined by performing triangulation between corresponding points captured in different images of a first intraoral image set. As discussed above, such triangulation can be performed by solving a correspondence problem.

[0138] At box 605, the processing logic identifies one or more regions in the 3D point cloud that include bodily fluids or suspected bodily fluids. Suspected bodily fluid identification has been discussed in more detail above. In some embodiments, different operations are performed to identify blood and saliva (e.g., salivary vesicles). Alternatively, the same set of operations can be performed to identify both blood and saliva. Operations can also be performed to identify other bodily fluids, such as pus. In some embodiments, the detection of one or more regions including bodily fluids or suspected bodily fluids is achieved by feeding the 3D point cloud or data of the 3D point cloud into one or more trained machine learning models, which can output dental object classifications of 3D points in the 3D point cloud (e.g., blood classification, saliva classification, tooth classification, gum classification, etc.). In some embodiments, intraoral images from one or more sets of intraoral images are fed into one or more trained machine learning models, which output dental object classifications of pixels in the intraoral images, which can be mapped to points in the 3D point cloud. In some embodiments, 2D images from a second image set are input into one or more trained machine learning models that output dental object classifications of pixels in the images. The pixels in the images can then be mapped to pixels in intraoral images of a first image set generated using structured light projection, which can be mapped to 3D points in a 3D point cloud.

[0139] In some embodiments, additional operations may be performed to detect regions in a 3D point cloud that include or depict body fluids and / or suspected body fluids. For example, these operations may include generating statistical results or other combinations from multiple images having pixels mapped to the same 3D point for dental object classification, determining whether a 3D point meets one or more body fluid classification criteria, drawing boundaries or shapes around 3D points classified as body fluid points, and reclassifying 3D points near the boundaries or shapes as also body fluid points, etc. (See above reference) Figures 4 to 5 Examples of these operations are described.

[0140] At box 606, the processing logic removes one or more regions classified as body fluid regions from the 3D point cloud. Alternatively, without removing these body fluid 3D points, the processing logic can filter or ignore these 3D points only in one or more other operations.

[0141] At box 608, the processing logic uses the 3D point cloud to update the 3D surface of the dental site. In this embodiment, fluid regions from the 3D point cloud may not be added to the 3D surface (e.g., because these points have been removed or filtered out). Updating the 3D surface may include registering the 3D point cloud to the 3D surface and stitching the 3D point cloud to the 3D surface, as discussed in more detail above. If a 3D surface has not already been generated, it can be generated by registering and stitching the 3D point cloud to one or more other 3D point clouds for which operations 602-606 have been performed.

[0142] At box 610, the processing logic can output the 3D surface to a display. Since the fluid region is not included in the 3D surface, the 3D surface may have voids corresponding to the fluid region. These voids can be additional intraoral scan indications for the voided regions used by the dentist to generate the 3D surface. In some embodiments, voids caused by fluids can be highlighted or emphasized on the 3D surface and / or display to draw the dentist's attention to the voids. The voids may be accompanied by labels or markings indicating the cause of the void (e.g., blood, saliva, general bodily fluids, etc.). In embodiments, one or more viewfinder images may also be displayed. In embodiments, the viewfinder images may be or may be based on one or more color 2D images. Additionally or alternatively, identified fluid pixels may be highlighted or otherwise emphasized in the viewfinder image. In examples, blood may be highlighted in bright red, saliva may be emphasized by increasing intensity or brightness (e.g., making the saliva appear as if the saliva region is glowing), and / or other visual effects may be used to highlight fluid regions.

[0143] In some embodiments, at block 612, the processing logic determines the amount of detected and / or suspected bodily fluids (or multiple bodily fluids). In some embodiments, this may include separately determining the amount of detected and / or suspected blood and the amount of detected and / or suspected saliva. Alternatively, the determined amount of bodily fluid may be for a single type of bodily fluid or a combined total for a set of detected different types of bodily fluids. The amount of detected bodily fluid may be determined based on quantifying the number of 3D points classified as bodily fluid points in a 3D point cloud, measuring the size of voids in a 3D surface caused by bodily fluid regions, and / or other measurement or quantification techniques. In some embodiments, the amount of detected and / or suspected bodily fluid is determined for each 3D point cloud. The processing logic can then compare the amount of detected and / or suspected bodily fluid over time and can determine whether the amount of bodily fluid is stable, increasing, decreasing, etc. In some embodiments, the amount of detected and / or suspected body fluid in the 3D point cloud is integrated over time (e.g., based on timestamps of intraoral and / or color image sets used to generate the 3D point cloud and / or classify points on the 3D point cloud) to determine the rate of change of body fluid volume. In some embodiments, the amount of detected and / or suspected body fluid in a 3D surface is integrated over time to determine the rate of change of body fluid volume.

[0144] At box 616, the processing logic determines whether the amount of detected and / or suspected body fluid exceeds a threshold. This may also include determining whether the rate of change of body fluid exceeds a threshold, or using the determined rate of change of body fluid volume to predict a future time point at which the body fluid volume will exceed the threshold. If the amount of detected and / or suspected body fluid exceeds the body fluid volume threshold, the rate of change of body fluid exceeds the rate of change threshold, and / or the predicted amount of body fluid exceeds the body fluid volume threshold, then the method may proceed to box 618. Otherwise, the method proceeds to box 620.

[0145] At box 618, the processing logic can generate an alarm. This alarm can include audible alarms, visual alarms (e.g., prompts or pop-up notifications in the GUI), tactile alarms (e.g., tactile feedback using the scanner), and so on. The alarm may indicate that the amount of bodily fluid (e.g., blood and / or saliva) is excessive and / or is reducing scan accuracy, that the dentist should remove the fluid, that the amount of blood and / or saliva is rapidly accumulating, and / or other messages. In response to receiving an alarm, the dentist should wipe or otherwise remove the detected bodily fluid from the patient's mouth (e.g., from the dental site being scanned). The subsequent new intraoral scan data for the dental site will be the intraoral scan data of the cleaned dental site without bodily fluid, and therefore more accurate. This additional intraoral scan data can be registered and stitched onto the 3D model to fill gaps in the 3D surface caused by previously detected bodily fluid.

[0146] At box 620, the processing logic determines whether additional scan data has been received. If additional scan data has been received, the method returns to box 604, and the operation sequence can be repeated for new intraoral scan data. Method 600 can continue until the scan is complete. In some embodiments, the scan is complete once an accurate 3D model has been generated that has no voids caused by body fluids or that these voids are smaller than a threshold size.

[0147] Figure 7 A flowchart of a method 700 for identifying bodily fluids in intraoral scan data according to an embodiment of the present disclosure is shown. In some embodiments, the operation of method 700 is performed at block 605 of method 600. At block 702 of method 700, processing logic detects one or more suspected regions in one or more color 2D images or other 2D images. A suspected region is a region that is suspected to correspond to a specific dental object category. In an embodiment, a suspected region is a region that is suspected to depict bodily fluids (e.g., blood or saliva). Alternatively, a suspected region is a region that is suspected to depict a certain other dental category to be filtered out, such as moving tissue (e.g., tongue, cheek, etc.), foreign objects (e.g., dental tools, hands, fingers, etc.), excess tissue, etc. In an embodiment, suspected regions in one or more 2D color images or other 2D images are determined by processing each 2D color image or other 2D image from a set of 2D color images (or other 2D image sets) using a trained machine learning model. The trained machine learning model may output one or more masks, wherein the output mask may be a segmentation mask, and / or may provide a probability that a pixel represents one or more bodily fluids (e.g., a probability of representing blood, a probability of representing saliva, etc.). Images generated using structured light projection may contain less data than images generated using uniform white light (e.g., color 2D images). Therefore, processing color images using a trained machine learning model may be more accurate than processing images generated using structured light projection.

[0148] At box 703, the processing logic maps suspected regions in a color 2D image or other 2D image to an image generated using structured light projection. Each color 2D image (or other 2D image set) can correspond to an image in the image set generated using structured light projection. Based on this correspondence, a mask of a color 2D image or other 2D image can be converted into a mapping of the corresponding image generated using structured light projection.

[0149] At box 704, one or more suspected regions in the image generated using structured light projection are mapped to 3D points in the 3D point cloud. The 3D point cloud may have been generated using multiple images generated using structured light projection, where the correspondence between points in the different images provides the 3D coordinates of the points in the 3D point cloud. Suspected regions may consist of suspected body fluid pixels, and each suspected body fluid pixel may correspond to a 3D point in the 3D point cloud.

[0150] At box 705, the processing logic identifies one or more regions in the 3D point cloud that meet one or more criteria. These criteria can be body fluid point verification standards; if these standards are met, the suspected body fluid point is confirmed as a genuine body fluid point.

[0151] One or more points in a 3D point cloud can be associated with pixels in multiple images, and each such pixel from one image can have a different probability that the pixel depicts a bodily fluid associated with it. Therefore, a 3D point may have multiple probabilities of depicting a bodily fluid associated with it. At box 707, for each point in the 3D point cloud, the processing logic determines a combined probability score representing the probability that the 3D point depicts a bodily fluid. This combined probability score can be the mean, a weighted average, the median, or other statistical value generated from the multiple probabilities associated with the point.

[0152] At box 710, the processing logic can classify points with a combined probability exceeding a threshold as suspected body fluid points. At box 715, the processing logic can determine accumulated additional body fluid points from intraoral scan data collected within the movement time window. These additional body fluid points may include body fluid points from 3D point clouds generated from one or more image sets captured prior to the current image set used to generate the current 3D point cloud. These additional body fluid points may also include, or alternatively may include, body fluid points from 3D point clouds generated from one or more image sets captured after the current image set used to generate the current 3D point cloud. Each 3D point cloud before or after the current 3D point cloud may have already been processed in a similar manner to identify suspected body fluid points in these 3D point clouds. Body fluid points should not appear alone. Therefore, the processing logic can determine the density of points surrounding a suspected body fluid point and then determine whether to confirm or reject the suspected body fluid point based on the density of the surrounding body fluid points. Therefore, at box 720, the processing logic can update the body fluid classification of a point based on the classification of nearby points. In this embodiment, for each suspected body fluid spot, if the number of nearby body fluid spots exceeds a threshold, the suspected body fluid spot can be confirmed as a body fluid spot.

[0153] At box 725, the processing logic can draw a shape (e.g., a circular shape) around each confirmed body fluid point. Any other point within the drawn shape that is not currently classified as a body fluid point can be updated to be reclassified as a body fluid point.

[0154] Figure 8A A model training workflow 805 according to embodiments of the present disclosure is illustrated for training one or more machine learning models to identify the classification of body fluids and / or other dental objects in intraoral scan data. In embodiments, the model training workflow 805 may be performed at a server, which may or may not include an intraoral scanning application, and the trained model is provided to the intraoral scanning application (e.g., Figure 1 The intraoral scanning application on the computing device 105 can perform operations such as Figure 8B The model application workflow is described above. The model training workflow 805 can be executed by processing logic performed by the processor of a computing device.

[0155] The model training workflow 805 is to train one or more machine learning models (e.g., deep learning models) to perform one or more classification, segmentation, detection, recognition, image generation, prediction, parameter generation, and other tasks on intraoral scan data (e.g., 3D scans, 3D point clouds, height maps, 2D color images, NIRI images, etc.) and / or 3D surfaces generated based on intraoral scan data.

[0156] This document describes the outputs of several different machine learning models. The specific number and arrangement of these machine learning models are described and illustrated. However, it should be understood that the number and type of machine learning models used, as well as the arrangement of such models, can be modified to achieve the same or similar end results. Therefore, the machine learning model arrangements described and illustrated are merely examples and should not be construed as limiting. Furthermore, in some embodiments, rule-based logic may be used instead of trained machine learning models, or in addition to trained machine learning models, to perform one or more of the operations described herein.

[0157] In some embodiments, one or more machine learning models are trained to perform image classification to categorize 2D color images (or other 2D images) of dental sites into one or more dental object categories. In some embodiments, different machine learning models are trained to detect different dental object categories. In some embodiments, one or more combined machine learning models are trained that may share one or more layers but produce different classification results. In examples, one or more machine learning models may be trained where the trained ML model is a single shared neural network with multiple shared layers and multiple higher-level, different output layers, each outputting a different prediction, classification, label, etc.

[0158] One type of machine learning model that can be used to perform some or all of the tasks described above is an artificial neural network, such as a deep neural network. Artificial neural networks typically include feature representation components with classifier or regression layers that map features to a desired output space. For example, a convolutional neural network (CNN) has multiple convolutional filter layers. Pooling is performed at the lower layers, and non-linearity can be handled. Multiple perceptrons are typically attached above these lower layers, mapping the top-level features extracted by the convolutional layers to a decision (e.g., classification output). Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of non-linear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks can learn in a supervised (e.g., classification) and / or unsupervised (e.g., pattern analysis) manner. Deep neural networks consist of a hierarchical structure where different layers learn representations corresponding to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and complex representation. For example, in image recognition applications, the raw input can be a pixel matrix; the first representation layer can abstract the pixels and encode edges; the second layer can compose and encode the arrangement of edges; the third layer can encode higher-level shapes (e.g., teeth, lips, gums, blood, saliva, etc.). It's worth noting that the deep learning process can learn on its own which features are best placed at which level. The "depth" in "deep learning" refers to the number of layers that transform the data. More precisely, deep learning systems have a considerable credit assignment path (CAP) depth. A CAP is a chain of transformations from input to output. CAP describes the underlying causal relationship between the input and output. For feedforward neural networks, the CAP depth can be the depth of the network and can be the number of hidden layers plus one. For recurrent neural networks where a signal can propagate more than once through a single layer, the CAP depth can be infinite.

[0159] Training a neural network can be done in a supervised learning manner. This involves feeding the network a training dataset consisting of labeled inputs, observing its outputs, defining the error (by measuring the difference between the output and the labeled values), and using techniques such as deep gradient descent and backpropagation to adjust the network's weights across all its layers and nodes to minimize the error. In many applications, repeating this process with inputs from many labeled inputs in the training dataset produces a network that can generate the correct output when presented with inputs different from those present in the training dataset. This generalization can be achieved in high-dimensional settings, such as large images, when a sufficiently large and diverse training dataset is available.

[0160] In one embodiment, the model training workflow 805 includes multiple operation sequences 802-810. A first operation sequence 802 may be performed to train a first machine learning model to classify and / or segment an image into a first set of dental object categories (e.g., teeth and / or gums). A second operation sequence 804 may be performed to train a second machine learning model to classify and / or segment an image into a second set of dental object categories (e.g., one or more of bodily fluids, non-bodily fluids, blood, non-blood, saliva, non-saliva, etc.). A third operation sequence 806 may be performed to label the image using a first trained ML model 839A, which has been trained to classify and / or segment the image into teeth and / or gums. A fourth operation sequence 80 may be performed to label the image using a second trained ML model 839B, which has been trained to classify and / or segment the image into one or more bodily fluids. A fifth operation sequence 810 can be executed to train a combined ML model using images labeled at the third operation sequence 806 and / or the fourth operation sequence 808. This combined ML model can output multiple sets of classifications (e.g., classifying pixels in an image as gums or teeth, and also classifying pixels in an image as depicting or not depicting bodily fluids). Since pixels may be classified as both teeth and bodily fluids (e.g., representing bodily fluids covering teeth), standard ML models may not be able to generate such multifaceted classifications.

[0161] Training a single neural network for multiple segmentation tasks (e.g., ensemble machine learning model 852, etc.) requires annotations for all pixels across all tasks. Since this can make annotation prohibitively expensive, in some embodiments, a virtual annotation method is used in teacher-student training approaches. In this approach, for each task, a dedicated segmentation network is trained on all available labeled data already labeled for that task. Because these networks are not limited by real-time performance, they may produce better segmentation results than real-time multi-head networks. In embodiments, the trainer utilizes two data sources in a single training run by weighting the loss function towards manually labeled data.

[0162] In the first operation sequence 802, at box 836A, a first training dataset is generated, which contains hundreds, thousands, tens of thousands, hundreds of thousands, or more 2D images of dental sites. These images may be labeled with tags for teeth and gums (e.g., pixel-level tags). In some embodiments, each individual tooth may be labeled individually in the images (e.g., based on tooth number). In embodiments, cases of up to millions of patient dentitions can be used to form the training dataset, where each case may include images with various labels.

[0163] In one embodiment, generating one or more first training datasets 836A includes collecting one or more 2D images 812A labeled with teeth and / or gums. In embodiments, one or more images in the training dataset 812A, along with optional associated probability maps or pixel / tile-level labels, can be resized. For example, a machine learning model can be used with images within a certain range of pixel sizes, and if one or more images fall outside this range, their size can be adjusted. For example, methods such as nearest-neighbor interpolation or box sampling can be used to resize the images. The training dataset can be additionally or alternatively augmented. Training large-scale neural networks typically uses tens of thousands of images, which are not readily available in many real-world applications. Data augmentation can be used to artificially increase the effective sample size. Common techniques include randomly rotating, shifting, cropping, flipping, etc., existing images to increase the sample size.

[0164] To perform training, the processing logic inputs the training dataset 836A into one or more untrained machine learning models. The machine learning models can be initialized before the initial input is fed into them. The processing logic trains the untrained machine learning models based on the training dataset to generate one or more trained machine learning models that perform the various operations described above.

[0165] At box 838A, training can be performed by feeding one or more inputs from images one at a time into the machine learning model. Each input can include data from images in the training data items of the training dataset.

[0166] Machine learning models process inputs to generate outputs. Artificial neural networks consist of an input layer composed of values ​​from data points (e.g., intensity and / or height values ​​of pixels in a heightmap). The next layer is called a hidden layer, where each node receives one or more of the input values. Each node contains parameters (e.g., weights) applied to the input values. Therefore, each node essentially feeds the input values ​​into a multivariate function (e.g., a nonlinear mathematical transformation) to produce an output value. The next layer can be another hidden layer or an output layer. In either case, the nodes in the next layer receive output values ​​from the nodes in the previous layer, each node applies weights to these values, and then generates its own output value. This can be done in each layer. The final layer is the output layer.

[0167] The processing logic can then compare the generated output with the known labels included in the training data items. The processing logic determines the error (i.e., classification error) based on the difference between the output probability map and / or labels and the provided probability map and / or labels. The processing logic adjusts the weights of one or more nodes in the machine learning model based on this error. An error term or difference (delta) can be determined for each node in the artificial neural network. Based on this error, the artificial neural network adjusts one or more parameters (weights of one or more inputs to the node) of one or more of its nodes. Parameters can be updated in a backpropagation manner, such that the nodes in the highest layer are updated first, followed by the nodes in the next layer, and so on. An artificial neural network contains multiple layers of "neurons," where each layer receives values ​​from neurons in the previous layer as input. The parameters of each neuron include weights associated with the values ​​received from each neuron in the previous layer. Therefore, adjusting the parameters can include adjusting the weights assigned to each input of one or more neurons in one or more layers of the artificial neural network.

[0168] Once the model parameters have been optimized, model validation can be performed to determine if the model has improved and to determine the current accuracy of the deep learning model. After one or more rounds of training, processing logic can determine whether a stopping criterion has been met. The stopping criterion can be a target accuracy level, the target number of processed images from the training dataset, the target amount of change in parameters on one or more previous data points, a combination thereof, and / or other criteria. In one embodiment, the stopping criterion is met when at least a minimum number of data points have been processed and at least a threshold accuracy has been achieved. The threshold accuracy can be, for example, 70%, 80%, or 90% accuracy. In one embodiment, the stopping criterion is met if the accuracy of the machine learning model has stopped improving. If the stopping criterion has not yet been met, further training is performed. If the stopping criterion has been met, training can be completed. Once the machine learning model has been trained, a reserved portion of the training dataset can be used to test the model.

[0169] In one embodiment, at box 838A, a first ML model is trained to classify and / or segment the input image into teeth and / or gums. This segmentation and / or classification can be performed pixel-by-pixel or at the patch level, where each patch may include a group of pixels.

[0170] In the second sequence operation 804, at box 836B, a second training dataset is generated, containing hundreds, thousands, tens of thousands, hundreds of thousands, or more 2D images of dental sites. These images may be labeled with tags (e.g., pixel-level tags) for one or more bodily fluid types (e.g., blood, saliva, etc.). Some images may be labeled with bodily fluid information but without tooth or gum classification, while other images may be labeled with tooth or gum classification but without bodily fluid information. Therefore, the first ML model can be trained at box 838A and the second ML model can be trained at box 838B using different training datasets.

[0171] In one embodiment, generating one or more second training datasets 836B includes collecting one or more 2D images 812B labeled with bodily fluids. In another embodiment, the one or more images 812B in the training dataset, along with optional associated probability maps or pixel / tile-level labels, may be resized.

[0172] To perform training, the processing logic inputs one or more training datasets (836B) into one or more untrained machine learning models. The machine learning models can be initialized before the first input is fed into them. The processing logic trains the untrained machine learning models based on the training dataset to generate one or more trained machine learning models that perform the various operations described above.

[0173] At box 838B, training can be performed by feeding one or more images into the machine learning model one at a time. Each input can include data from an image in the training data item of the training dataset. The ML model can generate an output, the error can be determined based on the output and the known labels of the images, and the error can be used to update the ML model. This can be repeated once or multiple times.

[0174] In one embodiment, at box 838B, a second ML model is trained to classify and / or segment the input image into regions containing one or more types of bodily fluids and regions not containing bodily fluids. This segmentation and / or classification can be performed pixel-by-pixel or at the tile level.

[0175] In the third sequence operation 806, the first trained ML model 839A can be used to process the images from the training dataset containing 2D images 812B with body fluid labels to generate an updated training dataset 812D including labels for teeth and gums as well as labels for body fluids.

[0176] Similarly, in the fourth operation sequence 808, the second trained ML model 839B can be used to process the images from the training dataset containing 2D images 812A with labels for teeth and gums to generate an updated training dataset 812C including labels for teeth and gums as well as labels for bodily fluids.

[0177] In the fourth operation sequence 810, at box 836C, a new training dataset can be generated, which may contain the modified training dataset 812C and / or the modified training dataset 812D. Images from the new training dataset can be input into a new ensemble ML model, which comprises two independent heads, each of which may include different high-level layers but may share the same low-level layers. One head of the ensemble ML model 838A can classify the pixels of an image as gums and teeth, and the other head of the ensemble ML model 838A can classify the pixels of an image as one or more bodily fluids, or no bodily fluids.

[0178] Once one or more trained ML models have been generated, they can be stored in model storage 845.

[0179] Figure 8B A model application workflow 817 is illustrated according to embodiments of the present disclosure, which uses one or more machine learning models and / or other logic to classify bodily fluids and / or other dental objects in intraoral scan data. In embodiments, the model application workflow 817 may be executed at a server (which may or may not include an intraoral scanning application), and at the intraoral scanning application (e.g., Figure 1 The model application workflow 817 can be executed at the intraoral scanning application on the computing device 105, within the intraoral scanner, etc. It can be executed by processing logic executed by the processor of the computing device and / or the intraoral scanner. For example, the model application workflow 817 can be executed by… Figure 1 The intraoral scanner 150, by the intraoral scanning application 115 or Figure 9 This is achieved by other software and / or firmware, or a combination thereof, executed on the processing unit of the computing device 900 shown.

[0180] The model application workflow 817 is used to apply one or more trained machine learning models generated using the model training workflow 805 to perform tasks such as classification, segmentation, detection, recognition, image generation, prediction, and parameter generation on intraoral scan data (e.g., 3D scans, 3D point clouds, height maps, 2D color images, NIRI images, etc.) and / or 3D surfaces generated based on intraoral scan data. One or more of these machine learning models can receive and process 3D data (e.g., 3D point clouds, 3D surfaces, parts of 3D models, etc.). One or more of these machine learning models can receive and process 2D data (e.g., 2D images, height maps, projections of 3D surfaces onto a plane, etc.). In an embodiment, a combined ML model 852 is added from model storage 845 to the model application workflow 817.

[0181] According to one embodiment, for model application workflow 817, the intraoral scanner generates a sequence of intraoral scan data, each intraoral scan data may include one or more intraoral scans 848 (e.g., each intraoral scan includes a set of images captured simultaneously by different cameras of the intraoral scanner under structured light projection) and one or more 2D color image sets 850 (e.g., each 2D color image set includes color 2D images generated simultaneously by different cameras of the intraoral scanner).

[0182] Point cloud generator 860 can process each intraoral scan (e.g., each image set generated using structured light projection) to generate a 3D point cloud 862. Processing the image set can include performing triangulation and solving correspondence algorithms between features captured in the images.

[0183] The combined ML model 852 can receive each image from the image set 850. In this embodiment, a dual-head segmentation network (e.g., a U-net) is used to minimize the execution time of the neural network. By sharing some weights with other neural networks, the computational load of body fluid detection is reduced, enabling the system to process multiple segmentation tasks in real time.

[0184] For each image in the processed image set 850, the combined ML model 852 can output multiple masks 854, 856. In one example, the first mask 854 can be a segmentation mask that segments the input image into one or more teeth and / or gums. In another example, the second mask 856 can be a mask that identifies pixels depicting bodily fluids. Each mask can include one or more probabilities for each pixel of the input image, where each probability is the probability that the pixel belongs to a particular category. For example, each pixel of the first mask 854 can include the probability that the pixel is gum and the probability that the pixel is a tooth. In another example, each pixel of the second mask 856 can include the probability that the pixel is a bodily fluid pixel. Alternatively, each pixel of the second mask 856 can include the probability that the pixel is a blood pixel and the probability that the pixel is a saliva pixel.

[0185] The 3D point cloud fluid identifier 864 can receive a first mask 854 and a second mask 856 for each image in the image set, and can receive a 3D point cloud 862 generated from an intraoral scan associated with the image set (e.g., a 3D point cloud generated temporally close to the time when the image set was generated). The 3D point cloud fluid identifier can process the received data to identify fluid points in the 3D point cloud. In an embodiment, the 3D point cloud fluid identifier 864 performs one or more operations of method 700 to identify fluid points. The 3D point cloud fluid identifier can redact fluid points from the 3D point cloud and can output a redacted 3D point cloud 866.

[0186] A 3D model (or surface) generator 870 can register a culled 3D point cloud with one or more other culled 3D point clouds and / or a 3D surface or model generated by stitching together multiple 3D point clouds, and can stitch the culled 3D point cloud to other culled 3D point clouds and / or 3D surfaces or models to generate a 3D model 872. The 3D model 872 can be output to a display for viewing by the patient and / or dentist. The model application workflow 817 can continuously process intraoral scan data as it is generated during intraoral scanning. Fluid areas can be identified in real-time or near real-time during scanning, allowing the generated 3D surface and / or model 872 to be generated with the detected fluids taken into account. Therefore, during an intraoral scanning session, a dentist may see one or more areas of the 3D surface missing due to accumulated fluids in the patient's mouth, and can remove these fluids and continue scanning to fill in the missing information corresponding to these fluid-covered areas. In this way, the accuracy of intraoral scanning can be improved, and the quality of 3D models generated based on intraoral scanning can be enhanced.

[0187] Figure 9The illustration shows a machine in an example form of computing device 900, in which a set of instructions can be executed to cause the machine to perform any or more methods discussed herein. In some embodiments, the machine may be connected (e.g., networked) to other machines in a local area network (LAN), intranet, extranet, or the Internet. The machine may operate as a server or client machine in a client-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine may be a personal computer (PC), tablet computer, set-top box (STB), personal digital assistant (PDA), cellular phone, network device, server, network router, switch or bridge, computer operatively connected to an intraoral scanner (and parts of an intraoral scanning system), or any machine capable of executing a set of instructions (sequential or otherwise) specifying the actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" should also be considered to include any collection of machines (e.g., computers) that individually or jointly execute one or more sets of instructions to perform any or more methods discussed herein.

[0188] Example computing device 900 includes processing device 902, main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and auxiliary memory (e.g., data storage device 928), which communicate with each other via bus 908.

[0189] Processing device 902 represents one or more general-purpose processors, such as microprocessors, central processing units, etc. More specifically, processing device 902 may be a Complex Instruction Set Computing (CISC) microprocessor, a Reduced Instruction Set Computing (RISC) microprocessor, a Very Long Instruction Word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing combinations of instruction sets. Processing device 902 may also be one or more special-purpose processing devices, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), network processors, etc. Processing device 902 is configured to execute processing logic (instructions 926) for performing the operations and steps discussed herein.

[0190] The computing device 900 may also include a network interface device 922 for communicating with the network 964. The computing device 900 may also include a video display unit 910 (e.g., a liquid crystal display (LCD) or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), and a signal generation device 920 (e.g., a speaker).

[0191] Data storage device 928 may include machine-readable storage medium (or more specifically, non-transitory computer-readable storage medium) 924 thereon storing one or more instruction sets 926, which embody any one or more methods or functions described herein. Non-transitory storage medium refers to storage medium other than a carrier wave. Instructions 926 may also reside wholly or at least partially within main memory 904 and / or processing device 902 during execution by computer device 900, which also constitute computer-readable storage media.

[0192] Computer-readable storage medium 924 can also be used to store an in-port scanning application 950, which can be used with... Figure 1 The intraoral scanning application 950 corresponds to a similarly named intraoral scanning application 950. The computer-readable storage medium 924 may also store a software library containing methods for calling the intraoral scanning application 950. Although the computer-readable storage medium 924 is shown as a single medium in the example embodiment, the term "computer-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) storing one or more sets of instructions. The term "computer-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for machine execution, and enabling a machine to perform any one or more methods of this disclosure. Therefore, the term "computer-readable storage medium" should be considered to include, but is not limited to, solid-state memory and optical and magnetic media.

[0193] The following exemplary embodiments will now be described.

[0194] Example 1: A method comprising: receiving intraoral scan data, the intraoral scan data including one or more intraoral images of a portion of the dental site generated using structured light projection during an intraoral scan of the dental site; generating a three-dimensional (3D) point cloud of the portion of the dental site using the intraoral scan data; detecting one or more regions in the 3D point cloud that include bodily fluids; removing the one or more regions from the 3D point cloud; updating a 3D surface of the dental site using the 3D point cloud of the portion of the dental site; and outputting the 3D surface of the dental site to a display.

[0195] Example 2: The method according to Example 1 further includes: generating an alarm to perform corrective measures in response to detecting one or more regions in the 3D point cloud that include body fluid.

[0196] Example 3: The method according to Example 1 or 2 further includes: determining the amount of detected body fluid; determining whether the amount of detected body fluid exceeds a threshold; and generating an alarm to perform corrective measures in response to determining that the amount of body fluid exceeds the threshold.

[0197] Example 4: According to the method described in Example 3, the amount of the detected body fluid includes the amount of suspected body fluid.

[0198] Example 5: The method according to Example 3 or Example 4, wherein the alarm includes a suggestion to remove the body fluid from the dental site before continuing intraoral scanning of the dental site.

[0199] Example 6: The method according to any one of Examples 3-5, wherein the amount of the body fluid is determined based on the area covered by the one or more regions.

[0200] Example 7: The method according to any one of Examples 1-6, wherein the bodily fluid includes saliva.

[0201] Example 8: The method according to any one of Examples 1-7, wherein the intraoral scan data further includes one or more color two-dimensional (2D) images generated during an intraoral scan at the dental site, and wherein detecting the one or more regions comprising body fluid in the 3D point cloud comprises: detecting one or more suspected regions in the one or more color 2D images that are suspected of comprising body fluid; mapping the one or more suspected regions in the one or more color 2D images to the 3D point cloud to form one or more suspected regions in the 3D point cloud that are suspected of comprising body fluid; and determining one or more suspected regions in the 3D point cloud that meet one or more criteria, wherein the one or more suspected regions in the 3D point cloud that meet the one or more criteria correspond to one or more regions in the 3D point cloud that comprise body fluid.

[0202] Example 9: According to the method described in Example 8, wherein the one or more color 2D images are generated using unstructured white light illumination.

[0203] Example 10: The method according to Example 8 or 9, wherein detecting the one or more suspected regions in the one or more color 2D images includes performing the following operations on each of the one or more color 2D images: processing the color 2D image using a trained machine learning model, wherein the trained machine learning model outputs a segmentation mask for the color 2D image, and wherein at least one classification of the segmentation mask is a body fluid classification.

[0204] Example 11: The method according to Example 10 further includes: processing the color 2D image using a second trained machine learning model, wherein the second trained machine learning model outputs a second segmentation mask for the color 2D image, and wherein at least, a first classification of the second segmentation mask is a tooth classification and a second classification of the second segmentation mask is a gingival classification.

[0205] Example 12: According to the method of Example 11, wherein the trained machine learning model includes a first neural network, wherein the second trained machine learning model includes a second neural network, and wherein the first neural network and the second neural network share one or more layers.

[0206] Example 13: The method according to any one of Examples 8-12, wherein the one or more color 2D images comprise a plurality of color 2D images generated at the same time point, each of the plurality of color 2D images being generated by a different camera of an intraoral scanner for performing the intraoral scan.

[0207] Example 14: The method according to any one of Examples 1-13, wherein the one or more intraoral images of the portion of the dental site generated using the structured light projection include a plurality of color 2D images, each color 2D image generated by a different camera of an intraoral scanner at a first time point, and wherein the intraoral scan data further includes a plurality of color two-dimensional (2D) images, each color 2D image generated by a different camera of the intraoral scanner, each of the plurality of color 2D images being generated by the same camera that generated the images in the plurality of images, the method further comprising: using trained machine learning... The trained machine learning model processes each of the plurality of color 2D images, wherein, for each color 2D image, the trained machine learning model outputs a mask indicating the probability that a pixel in the color 2D image depicts bodily fluid; for each point in the 3D point cloud that maps to a pixel in one or more of the plurality of color 2D images, the probabilities of pixels in the one or more color 2D images depicting bodily fluid are combined to determine the probability that a point in the 3D point cloud depicts bodily fluid; and points whose probability of depicting bodily fluid exceeds a threshold are classified as depicting bodily fluid.

[0208] Example 15: According to the method of Example 14, combining the probabilities includes determining the average value of the probabilities.

[0209] Example 16: The method according to Example 14 or 15 further includes performing the following operations for each point on the 3D point cloud classified as depicting the body fluid: determining the density of surrounding points also classified as depicting the body fluid; and confirming that the point depicts the body fluid in response to determining that the density of surrounding points also classified as depicting the body fluid is higher than a threshold.

[0210] Example 17: The method according to Example 16, wherein the density of the surrounding points is determined using a first additional intraoral scan data captured before the intraoral scan data and a second additional intraoral scan data captured after the intraoral scan data is captured.

[0211] Example 18: The method according to Example 16 or 17 further includes: estimating the shape around a set of points classified as depicting the body fluid; and updating the points within the circular shape to depict the body fluid.

[0212] Example 19: The method according to Example 18, wherein estimating the shape includes: for each point classified as depicting the body fluid, determining a boundary around the point; and determining an overlay of boundaries around a plurality of points.

[0213] Example 20: The method according to Example 19, wherein the boundary is circular.

[0214] Example 21: The method according to any one of Examples 1-20, wherein, in receiving the intraoral scan data at a first time, the method further includes: receiving additional intraoral scan data of the dental site at a second time; generating a second 3D point cloud of the dental site using the additional intraoral scan data; detecting one or more second regions in the second 3D point cloud that include the body fluid; determining that the one or more second regions are larger than the one or more regions; and generating an alarm to perform corrective measures.

[0215] Example 22: The method according to any one of Examples 1-21, wherein the 3D surface includes one or more voids corresponding to the one or more regions including the body fluid, and the method further includes: highlighting the one or more voids in the 3D surface.

[0216] Example 23: The method according to Example 22 further includes: receiving additional intraoral scan data after removing the body fluid from the dental site; and updating the 3D surface based on the additional intraoral scan data, wherein the one or more voids are filled based on the additional intraoral scan data.

[0217] Example 24: The method according to any one of Examples 1-23, wherein the body fluid includes blood.

[0218] Example 25: A non-transitory computer-readable medium comprising instructions that, when executed by a processing device, cause the processing device to perform the method according to any one of Examples 1-24.

[0219] Example 26: A system comprising: a handheld scanner for performing intraoral scanning; and a non-transitory computer-readable medium including instructions that, when executed by a processor, cause the processor to perform the method according to any one of Examples 1-24.

[0220] Example 27: A system comprising: a handheld scanner for performing intraoral scanning; and a computing device for performing the method according to any one of Examples 1-24.

[0221] Example 28: A system comprising: a handheld scanner for performing one or more operations of the method according to any one of Examples 1-24; and a computing device for performing one or more remaining operations of the method according to any one of Examples 1-24.

[0222] Example 29: A system comprising: a handheld scanner for performing intraoral scanning; a first computing device for performing one or more operations of the method according to any one of Examples 1-24; and a second computing device for performing one or more remaining operations of the method according to any one of Examples 1-24.

[0223] It should be understood that the above description is intended to be illustrative and not limiting. Many other embodiments will become apparent upon reading and understanding the above description. Although embodiments of this disclosure have been described with reference to specific example embodiments, it will be recognized that this disclosure is not limited to the described embodiments but can be practiced with modifications and variations within the spirit and scope of the appended claims. Therefore, the specification and drawings should be considered illustrative and not restrictive. Consequently, the scope of this disclosure should be determined by reference to the appended claims and the full scope of their equivalents.

Claims

1. A method comprising: Receive intraoral scan data, which includes one or more intraoral images of a portion of the dental site generated using structured light projection during an intraoral scan of the dental site; The intraoral scan data is used to generate a three-dimensional 3D point cloud of the portion of the dental site; Detect one or more regions in the 3D point cloud that include body fluids; Remove one or more regions from the 3D point cloud; The 3D surface of the dental region is updated using the 3D point cloud of the portion of the dental region. as well as The 3D surface of the dental area is output to a display.

2. The method according to claim 1, further comprising: In response to the detection of one or more regions in the 3D point cloud that include the body fluid, an alarm is generated to perform corrective measures.

3. The method according to claim 1, further comprising: Determine the amount of the detected bodily fluid; Determine whether the amount of the detected bodily fluid exceeds a threshold; as well as In response to determining that the amount of the bodily fluid exceeds the threshold, an alarm is generated to perform corrective measures.

4. The method according to claim 3, wherein, The amount of bodily fluid detected includes suspected bodily fluid volume.

5. The method according to claim 3, wherein, The alert includes a recommendation to remove the bodily fluid from the dental site before continuing the intraoral scan of the dental site.

6. The method according to claim 3, wherein, The amount of the body fluid is determined based on the area covered by the one or more regions.

7. The method according to claim 1, wherein, The bodily fluids include saliva.

8. The method according to claim 1, wherein, The intraoral scan data also includes one or more color 2D images generated during intraoral scanning at the dental site, and wherein detecting the one or more regions comprising the bodily fluid in the 3D point cloud includes: Detect one or more suspected regions in the one or more color 2D images that may contain the body fluid; Mapping the one or more suspected regions in the one or more color 2D images onto the 3D point cloud to form one or more suspected regions in the 3D point cloud that may include the body fluid; and Identify one or more suspected regions in the 3D point cloud that meet one or more criteria, wherein the one or more suspected regions in the 3D point cloud that meet the one or more criteria correspond to the one or more regions in the 3D point cloud that include the body fluid.

9. The method according to claim 8, wherein, The one or more color 2D images are generated using unstructured white light illumination.

10. The method according to claim 8, wherein, Detecting the one or more suspected regions in the one or more color 2D images includes performing the following operations on each of the one or more color 2D images: The color 2D image is processed using a trained machine learning model, wherein the trained machine learning model outputs a segmentation mask for the color 2D image, and wherein at least one classification of the segmentation mask is a body fluid classification.

11. The method of claim 10, further comprising: The color 2D image is processed using a second trained machine learning model, wherein the second trained machine learning model outputs a second segmentation mask for the color 2D image, and wherein at least the first classification of the second segmentation mask is a tooth classification and the second classification of the second segmentation mask is a gum classification.

12. The method according to claim 11, wherein, The trained machine learning model includes a first neural network, wherein the second trained machine learning model includes a second neural network, and wherein the first neural network and the second neural network share one or more layers.

13. The method according to claim 8, wherein, The one or more color 2D images include multiple color 2D images generated at the same time point, each of the multiple color 2D images being generated by a different camera of an intraoral scanner used to perform the intraoral scan.

14. The method according to claim 1, wherein, The one or more intraoral images of the dental portion generated using the structured light projection include multiple color 2D images, each generated by a different camera of the intraoral scanner at a first time point, and wherein the intraoral scan data further includes multiple color 2D images, each generated by a different camera of the intraoral scanner, each of the multiple color 2D images being generated by the same camera that generated the images in the multiple images, the method further comprising: Each of the plurality of color 2D images is processed using a trained machine learning model, wherein, for each color 2D image, the trained machine learning model outputs a mask that indicates the probability that the pixel in the color 2D image depicts the body fluid for each pixel. For each point in the 3D point cloud that maps to a pixel in one or more of the plurality of color 2D images, the probabilities of pixels in the one or more color 2D images depicting the body fluid are combined to determine the probability that a point in the 3D point cloud depicts the body fluid; and Points whose probability of depicting the body fluid exceeds a threshold are classified as depicting the body fluid.

15. The method according to claim 14, wherein, Combining the probabilities includes determining the average value of the probabilities.

16. The method of claim 14, further comprising, for each point on the 3D point cloud classified as depicting the body fluid, performing the following operations: Determining the density of points surrounding the fluid, which are also classified as being described; and In response to determining that the density of a point also classified as depicting the surrounding fluid is higher than a threshold, it is confirmed that the point depicts the fluid.

17. The method according to claim 16, wherein, The density of the surrounding points is determined using a first additional intraoral scan data captured before the intraoral scan data and a second additional intraoral scan data captured after the intraoral scan data is captured.

18. The method of claim 16, further comprising: Estimate the shape around a set of points classified as depicting the bodily fluid; as well as Update the points within the circular shape to depict the bodily fluid.

19. The method according to claim 18, wherein, Estimating the shape includes: For each point classified as depicting the bodily fluid, determine the boundary surrounding that point; and Determine the superposition of the boundaries around the point.

20. The method according to claim 19, wherein, The boundary is circular in shape.

21. The method according to claim 1, wherein, The method further includes receiving the intraoral scan data at the first moment: Additional intraoral scan data of the dental site are received at a second time. A second 3D point cloud of the dental site is generated using the additional intraoral scan data. Detect one or more second regions in the second 3D point cloud that include the body fluid; Determine that the one or more second regions are greater than the one or more regions; and Generate an alert to implement corrective measures.

22. The method according to claim 1, wherein, The 3D surface includes one or more voids corresponding to the one or more regions including the bodily fluid, and the method further includes: Highlight the one or more voids in the 3D surface.

23. The method of claim 22, further comprising: Additional intraoral scan data are received after the body fluid is removed from the dental site; as well as The 3D surface is updated based on the additional intraoral scan data, wherein the one or more voids are filled based on the additional intraoral scan data.

24. The method according to claim 1, wherein, The bodily fluids include blood.

25. A non-transitory computer-readable medium comprising instructions that, when executed by a processing means, cause the processing means to perform the method according to any one of claims 1-24.

26. A system comprising: A handheld scanner used to perform intraoral scanning; as well as A non-transitory computer-readable medium includes instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1-24.

Images