Intraoral scanning with surface differentiation
In the 3D scan of the oral cavity through machine learning algorithm, the weighted splicing is used to use surface geometric data and nearby points for weighted splicing, which solves the problem of the impact of deformed surface on the scanning, and realizes a robust digital 3D representation and efficient splicing process.
Patent Information
- Application Number
- CN201980026060.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-02-16
- Filing Date
- 2019-02-08
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2039-02-08
AI Technical Summary
The accuracy of existing intraoral 3D scanners is difficult to guarantee when dealing with deformed or moving surfaces, and often require additional data or expensive computing resources, which cannot effectively solve the impact of non-rigid surfaces on scanning.
Using machine learning algorithms, the scores of surface points are determined through surface geometric data and inputs of nearby points, and used to weighted stitching views to generate robust digital 3D representations without directly detecting surface motion, and based on the assumption of deformation tendency of surface type distinguish.
Intraoral 3D scanning with robustness to deformed or moving surfaces is achieved, improving the accuracy and efficiency of splicing, reducing dependence on additional data, and simplifying the calculation process.
Smart Images

Figure CN112106109B_ABST
Abstract
Description
Technical Field
[0001] Scanner systems and methods for scanning the oral cavity of a patient are disclosed herein. In particular, the disclosure relates to stitching together a digital 3D representation of the oral cavity cavity taking into account tissue deformation during scanning. A score finding algorithm, such as a machine learning algorithm, can be used to train the system to distinguish between various types of surfaces and weight different surfaces to stitch recorded views into a combined digital 3D representation. Background Art
[0002] In the dental field, 3D topographic measurements of a patient's teeth and possibly other parts of the oral cavity are required as a basis for restorative or orthodontic treatment. Traditionally, such 3D measurements are performed by initially taking physical impressions. Because this process is often unsatisfactory for the patient, more recently, intraoral 3D scanners have been used to directly measure the topography of teeth or other parts of the oral cavity.
[0003] Due to size limitations, intraoral 3D scanners typically record small views at a time, each containing a distance map and possibly other information such as color. As the scanner moves, these views are gradually stitched together to make a combined 3D topography measurement. Such a recording, for example, of the teeth and surrounding gums of a single jaw as regions of interest, typically takes at least a minute and typically produces at least 100 views. The terms "registration" and "stitching" are often used interchangeably in this article. The stitched model, which is often also converted to a surface, is often referred to as a "virtual 3D model" or "digital 3D representation" of the 3D topography measured by the scanner.
[0004] Several surfaces of the oral cavity are not rigid. For example, the cheeks and tongue may deform noticeably during the recording process. The gums may also deform, but the deformation is usually smaller. In addition, foreign objects are encountered during recording with an intraoral scanner. Some foreign objects are rigid but very mobile and usually appear only in certain recording views, such as dental instruments. Other foreign objects, such as cotton pads, may move less and appear in more recording views, but deform more. The dentist's finger usually moves and deforms noticeably in only a few recording views.
[0005] Since the stitching of views is usually based on the assumption that the scanned surfaces are stable and rigid, any movable or deformable surfaces usually result in a loss of accuracy of the combined 3D topography. Non-rigid stitching algorithms exist, but they are computationally expensive and require additional information, such as landmarks, which is usually not available during intraoral scanning.
[0006] Several methods have been introduced to reduce the detrimental effects of movable or deformable surfaces on intraoral 3D scanning. One strategy is to move these surfaces away from the view, for example by using cheek retractors. Since cheek retractors are often considered uncomfortable by patients and since they only provide a partial solution to the problem, data processing methods have been introduced.
[0007] US7698068 describes a method for distinguishing teeth from other intraoral tissues based on color, and using only the portion of the view representing the tooth color for stitching during recording. Since teeth are rigid and whiter than other tissues, the quality of the combined 3D topography measurements can generally be improved. However, teeth may change color both naturally and due to restorations, so classification by color alone may be inaccurate. Similarly, non-white but more rigid tissues (such as palatal rugae) can be used for stitching, especially in edentulous cases, and therefore cannot be ignored. In addition, some deformable foreign body surfaces, such as cotton pads, can have a color similar to the tooth color, but should be ignored when stitching.
[0008] US9629551 describes a method for detecting movable objects by analyzing the consistency of multiple views during recording of the same part of the oral cavity. The method uses only geometric information and is therefore robust to color diversity.
[0009] There remains a need for an intraoral 3D scanner, and a method of using the same, that is generally robust to deformed or moving surfaces. Summary of the invention
[0010] In one aspect, disclosed herein is a method for generating a digital 3D representation of at least a portion of an oral cavity, the method comprising:
[0011] - recording a plurality of views containing surface data representing at least the geometry of surface points of the portion of the oral cavity using an intraoral scanner;
[0012] - determining a weight for each surface point based at least in part on a score that is a measure of confidence that the surface point represents a particular type of surface;
[0013] - executing a stitching algorithm that performs a weighted stitching of the surface points in the plurality of views to generate a digital 3D representation based on the determined weights;
[0014] Therein, the score of the surface point is found by at least one score finding algorithm which takes as input at least a geometric portion of the surface data of the surface point and surface data of points in the vicinity of the surface point.
[0015] One or more processing units may be configured to apply an algorithm trained on the data, such as a machine learning algorithm, to distinguish between various types of surfaces indicative of tissue types, other surfaces, and possibly erroneous data in the recorded views. Each view contains surface geometry data, at least some points z(x, y) defined in a coordinate system relative to the scanner. Scanners and methods according to embodiments of the present disclosure use weighting to stitch the recorded views into a combined representation of the 3D topography, also referred to as a digital 3D representation. The weight of a point in the stitching is determined at least in part by a score, which is a measure of confidence that the point represents at least one type of surface. The measure of confidence and therefore the score may be a heuristic measure or a probability.
[0016] The scanner and method according to the present disclosure may not necessarily detect movement or deformation of tissue or foreign matter directly. It typically distinguishes surface types based on the surface's assumed tendency to move or deform, regardless of whether the actual surface of such type has moved or deformed during the scan. Surface types can be based on histology, such as dentin or mucosa. They can also be based on location, for example, the gum between teeth has a smaller tendency to deform than the gum around prepared teeth. Surface types can also be heuristic, such as whether they are required in a digital 3D representation. Surface types with relatively small tendencies to deform or move are generally more desirable for stitching.
[0017] In the present disclosure, the distinction by surface type can be based at least in part on surface geometry data, whereas known techniques require additional data for distinction, such as surface color as in US7698068. In addition, a scanner or method according to an embodiment of the present disclosure can also provide and utilize additional surface data, such as surface color, in a view. A scanner or method according to the present disclosure can also provide and utilize the certainty of the surface data it records.
[0018] Weighted stitching can be performed using one of many variants of the Iterative Closest Point (ICP) algorithm or other suitable algorithms in the art. Pairwise weighting is described, see, for example, [1]. In another expression of weighted stitching, the surface data can be classified based on their weights, and then only some of the highest quantiles or some that exceed a certain threshold are used in the stitching. In the sense of the present disclosure, weighted stitching is some mathematical formula that expresses the distinction so that some data in the view have a relatively greater impact on the result than other data.
[0019] The common coordinate system of the stitched model may be the local coordinate system of the first view. Stitching views is understood to be stitching the surface geometric data of the views by transforming the surface geometric data z(x, y) of the views into the common coordinate system while applying the same geometric transformation to any other surface data in the views. Surface data points within the views that have zero weight or some small weight may be included in the transformation, or they may be removed.
[0020] The digital 3D representation can be represented in several ways. Stitching alone will at least provide a point cloud. It is often desirable to approximate the point cloud with a surface (e.g. a triangular mesh) to remove noise from the point data and provide a digital 3D representation, which is a better basis for dental CAD. Some algorithms build such a surface after all views have been stitched, e.g. [2]. Some algorithms progressively build some intermediate surface model for each view recorded and stitched, and may also use this intermediate model to improve the stitching, e.g. [3]. After all views have been recorded, a final surface is usually calculated to replace the intermediate surface. Surface data points with small weights (if they were not removed when stitching the corresponding views) are usually effectively removed in this step because they are detected as noise.
[0021] In some embodiments, the points z(x,y) are arranged as a distance map, i.e., the distances z(x,y) from some reference surface defined relative to the scanner to the scanned surface. In some embodiments, the coordinates (x,y) exist on a grid on a planar reference surface. A surface data point in the sense of the present disclosure contains at least geometric information, i.e., z(x,y). It may also be enhanced with other data recorded for the surface at (x,y), such as color, or some measure of the certainty of z(x,y), or some other data.
[0022] To distinguish a location (x, y) by surface type, the scanner of the present disclosure considers the value z(x, y) and additional values of z in the vicinity of (x, y). Considering the neighborhood can reveal geometric structure that is typical of certain surface types. A neighborhood can be an immediate neighborhood or a set of neighboring areas that extend beyond the immediate neighborhood. It can be useful to apply a kernel to reveal the geometry or a set of kernels. Considering the additional information contained in the neighborhood is another improvement over the prior art.
[0023] In some embodiments, the score finding algorithm is a machine learning algorithm. Any type of machine learning algorithm can be used. Some examples of machine learning algorithms include artificial neural networks, such as deep artificial neural networks, convolutional artificial neural networks, or recursive artificial neural networks. The machine learning method of the embodiments of the present disclosure can apply dimensionality reduction methods, such as principal component analysis or autoencoders.
[0024] In some embodiments, the machine learning algorithm comprises a neural network having at least one convolutional layer. Convolutional neural networks naturally provide for consideration of neighborhoods. For distance maps or colors defined or resampled on a grid (preferably an equidistant grid) of (x, y), many machine learning algorithms published for image analysis and image segmentation can be similarly applied. The algorithm used to distinguish between surface types can also be a more classical machine learning algorithm, such as using a support vector machine. The algorithm used to distinguish between surface types can also be an algorithm based on more classical statistical methods (e.g., Bayesian statistics or regression type). Various algorithms in the above categories of algorithms can also be used in combination.
[0025] In some embodiments, at least one machine learning algorithm is trained on multiple types of surfaces typically recorded in the oral cavity with a scanner. By annotating the training set images on various types of surfaces typically found in the oral cavity (e.g., teeth, gums, tongue, palate, etc.), the resulting weight determination will be more robust and consistent.
[0026] The machine learning algorithm for distinguishing surface types can be trained in a supervised, semi-supervised or unsupervised manner. For semi-supervised or supervised learning, the training can be based at least in part on annotated views or annotated digital 3D representations. The annotations on the digital 3D representation can be back-projected to each view that contributes to the digital 3D representation, since stitching can also achieve the transformation of each view to a common coordinate system. Thus, the annotations can be brought to the views and can be used to train the view-based machine learning algorithm. The annotations can be performed by a human and / or an algorithm.
[0027] During scanning, the machine learning algorithm runs in inference mode, detecting a score that is a measure of confidence that the surface data belongs to one or more surface types. Typically, the score can be represented as a vector with one value for each surface type in inference. In embodiments using neural networks, the score is typically obtained from the output layer, perhaps after applying some transformation such as a logarithmic transformation.
[0028] As used in this application, a confidence measure refers to some score that indicates a degree of certainty. The measure of confidence can be a probability, especially if the distribution of the underlying random variable is known or assumed to be known. If there is no such knowledge, or if no assumptions seem to be needed, or if it is preferred for other reasons, the measure of confidence can be some more subjective assessment and / or expression of the degree of certainty.
[0029] Mathematically, it may be convenient that a score of 1 represents confidence in the determination that the surface data belongs to a particular surface type (e.g., teeth or gums), while a score of zero represents confidence in the determination that the surface data does not belong to that particular surface type. Scores increasing from zero to one represent increasing confidence that the surface data belongs to that particular surface type.
[0030] Based on the score of the point, for example as a function of the score, a weight for the surface data point in the stitching is found. An embodiment in which the function is 1 for the surface type with the highest score and 0 otherwise is known in the art as classification. An example of a machine learning algorithm for classification is [4]. It may be advantageous to use a more sophisticated function, for example, returning a value of 1 only if the highest score is significantly greater than all other scores, for example greater than the sum of all other scores. It may also be advantageous for the function to return non-zero values for several surface types - for example if there is reason to believe that the surface data point may be any one of a plurality of surface types. The function may also return 0 for all surface types if the scores are not large or in similar situations of poor certainty.
[0031] In some embodiments, the type of surface represents more than one type of intraoral tissue. In some cases, it may be advantageous to group different intraoral tissue types together, such as grouping tooth surfaces with gum tops, as this is useful for stitching together digital 3D representations.
[0032] In some embodiments, the weight of each surface point in the stitching is also determined by the weight of the surface type. This means that the weight of a particular surface data point in the stitching is found from the score and from the surface type weight, for example a linear combination of all surface types that is the product of the surface type weight and the score. The surface type weights are preferably assigned a priori, with surface types required for the stitching receiving higher weights and other types receiving smaller weights. In some embodiments, some surface type weights are set to zero, so that surface data of those surface types are filtered out of view. There may be other considerations that affect the weight formula, such as the size of the surface patch represented by the surface data point, for example because it is the closest data point to all points within the patch.
[0033] In some embodiments, inference may be performed in real time or near real time while the views are recorded. This may allow stitching to also be real time or near real time. Performing stitching in real time is advantageous because a 3D representation of the site may be built and visualized as the user scans, thereby helping the user navigate the site. Thus, these embodiments are a significant improvement over other machine learning inference applications such as, for example, US7720267, US8170306, and US9349178.
[0034] Other embodiments of machine learning inference according to the present disclosure may perform more slowly - for example after two or more views have been recorded, but provide better accuracy. It is also possible to combine a limited degree of surface data weighting based on some inference from a single data with additional surface data weighting based on some inference from multiple views, potentially providing a good combination of speed and accuracy.
[0035] In some embodiments, the surface data also includes color information.Adding color information to the surface data can make tissue type determination more secure.
[0036] In some embodiments, at least one machine learning algorithm is trained at least in part using data recorded by an intraoral scanner. Because sensitivity and image quality may vary between scanners from different manufacturers, the closer the data used to train the machine learning algorithm matches the scans that will subsequently be acquired by the user, the more accurate the results will be.
[0037] In some embodiments, at least one machine learning algorithm is at least partially trained by an operator of the intraoral scanner. The scanner system of the present disclosure can be provided to a user with at least one pre-trained machine learning algorithm. In other embodiments, the user of the scanner performs at least some of the training after receiving the scanner. For example, additional training data may contain color images or surface geometry data where special types of gloves or cotton rolls used by dentists appear. Additional training data may also come from a patient race that is represented above average by dentists. Through additional training, the scanner can also adapt to the scanning style of the user. Additional training can be performed on one or more processing units of the scanner system or in the cloud. It may be advantageous to customize the machine learning algorithm with additional training because it may perform better.
[0038] In some embodiments, one score finding algorithm is selected for one type of application, and at least one other algorithm is selected for another type of application. It may be advantageous to train several machine learning algorithms for different types of applications for subsequent selection during inference. Selection of the appropriate algorithm can be made, for example, by a user of the scanner in a user interface.
[0039] The type of application may differ in the set of surface types that are trained on or inferred. For example, an algorithm with a set of surface types that represent interdental papillae and gingival pockets may be relevant to monitoring patients with gingivitis. In another example, an algorithm with a set of surface types that represent partial gingiva may be relevant to edentulous patients, where tooth surface data is sparse and often insufficient for stitching.
[0040] In other embodiments, the type of application is characterized, at least in part, by at least one of a particular patient age group, a particular patient race, a particular treatment type, a particular medical indication, a particular type of equipment used with the scanner, or a particular region of the oral cavity. For example, one algorithm may be best suited for children while another algorithm is best suited for adults, or certain races may differ from other races. The type of application may also represent different types of dental treatments, for example, as determined by organization- or region-specific standard operating procedures or equipment or the like.
[0041] In some embodiments, the scores are summed over multiple views. When the sub-scans are stitched together, a temporary digital 3D representation can be created. Each three-dimensional pixel in the temporary representation can then be imaged from multiple views, and the scores can then be summed over multiple views to make a more robust score determination.
[0042] In some embodiments, other algorithms or criteria for filtering data from the recorded views are applied.
[0043] In some embodiments, one of the other algorithms evaluates geometric consistency between multiple views. An example of this is movable object detection based on geometric consistency as disclosed in US9629551B1. During reasoning, filtering out surface data based on other criteria before finding the score simplifies the stitching problem, while filtering out surface data based on other criteria after finding the score can improve the overall result. However, during training, it may be advantageous not to filter out based on other criteria, in this way retaining relatively more training data.
[0044] Advantageous embodiments of the present disclosure use a combination of filtering based on geometric consistency and semantic segmentation. In this embodiment, an exclusion volume is established from all data in the same space as the digital 3D representation established by only those surface data of the fragments belonging to the desired surface type. Then, for example, after all views have been collected and therefore most of the information about the exclusion volume has been collected, the parts of the digital 3D representation in the exclusion volume can be removed. It is also feasible to stitch only based on the data that passed the filtering, but also retain the filtered out data for later analysis.
[0045] In some embodiments, the scanner also provides certain deterministic information of the measured surface data for the recorded view, and wherein the deterministic information at least partially determines the score. In some such example embodiments, the scanner is a focus scanner, such as the focus scanner disclosed in US8878905. The focus scanner can provide certainty of the measured z(x, y) data based on the uniqueness of the focus metric. Other kinds of 3D scanners can also provide deterministic information about the measured surface data. For example, a scanner using triangulation or projected light patterns recorded by at least two cameras can provide two simultaneous views and derive certainty from the degree of consistency between them. Other 3D scanners may infer certainty from image contrast or from other information. Still other scanners can provide certainty of other surface data (e.g., color).
[0046] The determinism of the surface data can be used to additionally modify their weights in stitching, or they can be used during training and inference. The determinism or other surface data in the view can be represented mathematically, for example, as additional channels in the enhanced distance map. Many machine learning algorithms published for multi-channel image analysis and image segmentation can then be applied similarly in the present disclosure.
[0047] In another aspect of the present disclosure, disclosed herein is a scanner system for generating a digital 3D representation of at least a portion of an oral cavity, the scanner system comprising:
[0048] -Handheld intraoral scanner;
[0049] a processing unit for executing a stitching algorithm for performing a weighted stitching of surface points of a plurality of views into a digital 3D representation, the weight of each surface point in the stitching being determined at least in part by a score being a measure of confidence that said surface point represents a particular type of surface; and
[0050] - finding a score for a surface point by at least one score finding algorithm taking as input at least a geometric part of the surface data for the surface point and the surface data for points in the vicinity of the surface point.
[0051] The data processing units running the machine learning algorithms can be part of the intraoral scanner, or they can also be contained in another peripheral device connected to the handheld scanner. Power requirements and regulatory requirements for handheld scanners to remain relatively cool make it advantageous to place the processing devices in a separate peripheral device. The one or more processing units can be a PC, FPGA, or similar, and can also include a GPU, and can also perform other data processing. The processing unit can be connected to a display, and the virtual model is displayed on the display as it is stitched during scanning.
[0052] In some embodiments, at least one score-finding algorithm is a machine learning algorithm.
[0053] In some embodiments, the scanner has an optical system that is at least near-telecentric. It is generally easier to train a machine learning algorithm and use for inference when the view is not affected by scale, that is, when a given surface type is imaged at the same resolution and size across the entire depth of field of the scanner. Scanners with telecentric optical systems provide this advantage by construction, while scanners with near-telecentric optical systems (e.g., scanners with a viewing angle greater than zero degrees but less than 10 degrees) provide an approximation thereof. For scanners with larger viewing angles, it may be advantageous to resample the views before using them for machine learning. For example, with knowledge of the optical system obtained from construction or calibration, the apparent orthogonal views can be calculated. Since resampling can compensate for scaling effects of size, but not resolution, a scanner with near-telecentric optical systems is preferred over a scanner that uses resampling.
[0054] In some embodiments of the present disclosure, the scanner is a confocal intraoral scanner.
[0055] In some embodiments of the present disclosure, the scanner may also provide and utilize certain deterministic information about the measured surface data. In some such example embodiments, the scanner is a focus scanner, such as the focus scanner disclosed in US8878905. The focus scanner may provide certainty of the measured z(x, y) data based on the uniqueness of the focus metric. Other kinds of 3D scanners may also provide information about the certainty of the measured surface data. For example, a scanner using triangulation of projected light patterns recorded by two cameras may provide two views simultaneously and derive certainty from the degree of consistency between them. Other 3D scanners may infer certainty from image contrast or other information. Other scanners may provide certainty of other surface data, such as color.
[0056] The determinism of the surface data can be used to additionally modify their weights in stitching, or they can be used during training and inference. The determinism or other surface data in the view can be represented mathematically, for example, as additional channels in the enhanced distance map. According to embodiments of the present disclosure, many machine learning algorithms published for multi-channel image analysis and image segmentation can then be similarly applied.
[0057] It is generally easier to train machine learning algorithms and use for inference when the views are not affected by scale, that is, when a given surface type is imaged at the same resolution and size across the entire depth of field of the scanner. Scanners with telecentric optical systems offer this advantage by construction, while scanners with near-telecentric optical systems (e.g., scanners with a viewing angle greater than zero degrees but less than 10 degrees) provide an approximation of it. For scanners with larger viewing angles, it may be advantageous to resample the views before using them for machine learning. For example, with knowledge of the optical system from construction or calibration, the apparent orthogonal views can be calculated. Since resampling can compensate for scaling effects of size, but not resolution, scanners with near-telecentric optical systems are preferred over scanners that use resampling.
[0058] In another aspect, a computer program product is disclosed herein, the computer program product comprising program code comprising instructions embodying any one or more of the methods or functions disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] The above and / or other objects, features and advantages of the embodiments of the present disclosure will be further described through the following illustrative and non-limiting detailed description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:
[0060] Figure 1 A scanner system according to an embodiment of the present disclosure is shown;
[0061] Figure 2 A schematic diagram showing an example handheld portion of a scanner system according to an embodiment of the present disclosure;
[0062] Figure 3 (a) and Figure 3 (b) is a diagram showing a spatially periodic checkerboard pattern for determining a distance map according to an embodiment of the present disclosure;
[0063] Figure 4 shows an exemplary curve of two pixel groups having their maximum values at the same position according to an embodiment of the present disclosure;
[0064] Figure 5a shows a cross-sectional view of a portion of the lower oral cavity according to an embodiment of the present disclosure;
[0065] Figure 5b A process for training a machine learning algorithm according to an embodiment of the present disclosure is shown; Figure 6 shows the architecture of a suitable convolutional neural network according to an embodiment of the present disclosure;
[0066] Figure 7(a) shows an example of how a trained machine learning algorithm may be applied in inference mode according to an embodiment of the present disclosure;
[0067] Figure 7 (b) illustrates another example of how a trained machine learning algorithm may be applied in inference mode according to an embodiment of the present disclosure; and
[0068] Figure 8 It shows how to convert Figure 7 The algorithm of (b) is extended to also use an example of exclusion volume based filtering. DETAILED DESCRIPTION
[0069] Figure 1 An example application of a scanner system according to an embodiment of the present disclosure is shown. A dentist holds a handheld scanner 100. The handheld scanner is typically connected to a laptop computer 50 having a screen 55 by a cable 60. In some cases, the handheld scanner may be wireless. As the scan is performed on the screen 55, a digital 3D representation of the stitched result is displayed. The display is updated over time as new views are recorded and stitched into a digital 3D representation. The laptop computer 50 is a peripheral device for one or more processing units that include a CPU and a GPU that execute the algorithms of an embodiment of the present disclosure. Some scanners communicate with the processing device via wireless transmission rather than via cables.
[0070] Figure 2 An example of a handheld portion 100 of an intraoral scanner according to an embodiment of the present disclosure is schematically shown. It includes an LED light source 110, a lens 111, a pattern 130 (a line in the true cross-sectional view, but shown here at an angle for clarity), a beam splitter 140, and an image sensor 180. The scanner is focused due to the movable lens 150. The reflector 170 folds the beam path toward a portion of the oral cavity being scanned, including the teeth 301 and gums 302. Also shown as dashed lines are some of the light rays emitted from the light source 110, which are transmitted through the optical system to the location on the tooth surface where they are focused, and then returned through the optical system and imaged onto the sensor 180, where they are also focused. For Figure 2 In the position of the focus lens 150 shown in , some gums 302 are visible, but not in focus. For other positions of the lens 150, it may come into focus.
[0071] Figure 3 (a) and Figure 3 (b) and Figure 4 It is shown how data from an exemplary handheld scanner 100 is processed to produce a range map. Figure 3(a) is a view of a spatially periodic checkerboard pattern 130 as seen by the image sensor 180 when the image of that spatial period on the scanned surface is in focus. In the extreme case where the image of the pattern is completely out of focus, the image sensor sees a uniform gray. It is advantageous if the image sensor pixels and the pattern are aligned so that the boundaries of the pattern are imaged onto the boundaries of the pixels. Figure 3 As shown in (b), pixels that are expected to be dark when the surface is focused are assigned a weight f(i)=-1, while pixels that are expected to be bright are assigned a weight f(i)=1, where i is the pixel index.
[0072] As the focus lens moves, images are captured. The position s of the focus lens for each image is found by the encoder and appropriate interpolation when necessary. For all pixel groups in the image, the relevant metric is calculated as:
[0073]
[0074] where I(i) is the intensity measured in the pixel. Figure 3 (b), i continues from 1 to n = (6*6) = 36. Note that in general, there are many spatial periods in a typical rectangular pattern 130 and its image on the sensor 180, for example N = 100*120 = 12,000 pixel groups, so in this example, the image will contain at least 36*12000 pixels.
[0075] A pixel group is in focus when A is at its maximum value over all s (i.e., in a series of images obtained within one pass of the focusing lens). The position of this maximum value will then determine the distance z(s) to the surface being measured, since the optics of the scanner are known by construction and, therefore, the position of the focus plane is known for all pixel groups and all positions of the lens 150. The distance z(s) can also be found from or perfected by calibration. The distance z(s) for a given pixel group results in a point z(x, y) where the plane of the image sensor and some point on it define a coordinate system relative to the scanner and each pixel group has (x, y) in that coordinate system - e.g., the center of the pixel group. Since all pixel groups are the same size, all positions (x, y) form a grid.
[0076] Note that if a pixel group is completely out of focus at some position s, i.e., all pixels have the same value, then A=0 at that position s. Note also that some pixel groups may never be in focus, e.g., when there is no surface to be imaged, or when a surface is present but out of focus. The distance of such pixel groups cannot be determined.
[0077] The focus lens should move quickly so that a given group of pixels at least approximately represents the same area of the scanned surface, even if there is some hand motion. For example, the focus lens can move back and forth through all s with a cycle frequency of 10 Hz, so 20 times per second. At the same time, the number of images during a pass should be high to produce a good resolution of the distance measurement, for example 200. This means that the image sensor must be quite fast, in this example it will need to acquire images at a rate of 10 Hz*2*200=4000 Hz.
[0078] Figure 4 Two exemplary curves A(s) for a group of pixels are shown. Both have their maximum at the same position s=10. Curve A1 has a more distinct maximum than A2 in the sense that both maximum values are higher and the width of the non-zero area for A1 is smaller than for A2. Therefore, the distance measurement based on curve A1 can be considered more certain than the measurement based on curve A2. There may be various reasons for less certain distance measurements. Some are due to the characteristics of the scanned surface, for example, the surface is tilted with respect to the viewing angle or has a varying reflectivity. Hand movements also typically result in less certain distance measurements. The certainty can be quantified, for example, as a maximum value, the logarithm of the maximum value, the inverse of the width of the area where A is greater than half its maximum value, or similar values.
[0079] In summary, the exemplary scanner provides a distance map where each pixel with coordinates (x, y) has a z value, with an associated deterministic q or undefined. The combination of (z, q)(x, y) can be called an enhanced distance map, similar to an image with two channels. Then, by stitching the enhanced distance maps obtained at various scanner poses, a complete 3D representation of the scanned portion of the oral cavity is obtained.
[0080] Other types of 3D scanners include triangulation 3D laser scanners and structured light 3D scanners. Triangulation 3D laser scanners use lasers to detect the environment or objects. Triangulation lasers shine a laser onto an object and use a camera to find the location of the laser dot. Depending on how far the laser strikes the surface, the laser dot appears at different locations in the camera's field of view. The technique is called triangulation because the laser dot, camera, and laser emitter form a triangle. Instead of a single laser dot, a laser bar can be used and then swept across the object to speed up the acquisition process.
[0081] A structured light 3D scanner projects a pattern of light on an object and observes the deformation of the pattern on the object. The pattern can be one-dimensional or two-dimensional. An example of a one-dimensional pattern is a line. The line is projected onto the object using, for example, an LCD projector or a swept laser. A camera slightly offset from the pattern projector observes the shape of the line and uses a technique similar to triangulation to calculate the distance of each point on the line. In the case of a single line pattern, the line is swept across the field of view to collect distance information one strip at a time. Other 3D scanner principles are well known in the art.
[0082] For an example of how to derive a set of surface types, Figure 5a for reference. The figure shows in cross-section the lower mouth with an incisor 600, its gingival margin 610, the gingiva 630 and the lower lip 650. A set of possible surface types will be based on the types of these three tissues only. However, for better training, it may be advantageous to reduce the number of surface types in the set. Taking the example of a generic type application with the goal of recording teeth and near gingiva, one could group "teeth" together with some near gingival tissue, for example up to point 620, and the rest being a second surface type. The former is "desirable" for stitching and the latter is "undesirable". Point 620 (3D line) does not have to have anatomical significance, but can be set (for example) from the extent of the scanner's field of view when the scanner images at least a portion of the tooth. In this example, any surface of a dental instrument, finger, cotton pad or other artifact is also grouped into the "undesirable" type.
[0083] exist Figure 5b An example of a process for training a machine learning algorithm according to an embodiment of the present disclosure is shown in FIG.
[0084] In step 501, a scanner is used to create a digital 3D representation of a portion of the oral cavity by recording a plurality of enhanced distance maps stitched together, such as for Figure 1-4 Some views and therefore some enhanced distance maps may contain dental instruments, fingers, cotton pads or other artifacts. These artifacts may be at least partially represented in the digital 3D representation, making the model a relatively poor representation of the oral cavity.
[0085] In step 502, the portion of the digital 3D representation is obtained by Figure 5a The defined surface types are annotated. Annotation can be performed, for example, by a human in a graphical user interface. Annotation can also be semi-automatic, where a human receives suggestions obtained using other methods, such as traditional image analysis algorithms.
[0086] In step 503, the annotated parts are projected back into separate enhanced distance maps using the inverse of the transformation found in the concatenation. This provides a class map c(x, y) for each enhanced distance map, where c is the class indicator. Because the annotations are assumed to provide perfect information, a one-hot encoding is used to derive a vector of scores p for the parts of the enhanced distance map of step 502 that belong to any surface type. Thus, in p, elements of class c are set to 1 and all others are set to zero. In the example used here, p has two elements, one for "desirable" and the other for "undesirable".
[0087] For many similar records R1, R2, ..., R m , for example at least m=2 records, steps 501-503 are performed. The number of records m may also be, for example, 10, 20, 50, 100, 200, 500 or 1000, or any number of records in between or higher. The records are similar with respect to the scanned portion of the oral cavity, and they may also be similar with respect to artifacts in certain views, affecting the m digital 3D representations created from the m records in a similar way.
[0088] In step 504, a neural network is trained to predict p(x, y) for (z, q)(x, y). Figure 4 As explained, the uncertainty q is taken as the logarithm of the maximum value of A. As the scanner outputs a distance map defined on a rectangular grid as described above, the enhanced distance map (z, q)(x, y) can be interpreted as a two-channel rectangular image that becomes the input to the neural network.
[0089] Note that steps 501-504 can also be performed on a single distance map, on a single enhanced distance map (neither of which results in stitching and trivial back-projection), or on multiple distance maps, although it is preferred to perform them on multiple enhanced distance maps because the constellation provides most of the information. For scanners that also provide color data, the training and prediction can be formulated as p(x,y) for (z,q,r,g,b)(x,y), where r,g,b are the red, green, and blue components of the color measurement; similarly for other or additional data provided by the scanner.
[0090] Figure 6 shows an example architecture of a suitable convolutional neural network where:
[0091] I: Input image, width is 120, height is 100 ( Figure 3 (a) and Figure 3 (b) and Figure 4 ), and two channels for z and q respectively.
[0092] P: Zero padding, 15 pixel border width around the image.
[0093] C: Convolutional layer, climate is rectified linear units.
[0094] CR: Cropping layer, used to reduce the size to fit the subsequent layer operations.
[0095] B: Upscaling layer using a factor of 2 and bilinear interpolation.
[0096] M: max pooling layer.
[0097] +: A layer that adds corresponding channels element-wise.
[0098] O: Output image of dimension with number of channels equal to p (2 in the example used here).
[0099] Figure 6 The dimensions of the data in the various layers and operations are also shown. Many network architectures are possible. Preferably, the neural network has convolutional layers with small kernels, such as 3×3×M, where M is the number of channels. The influence of the neighborhood of surface points comes from the use of convolutional layers. The influence of wider neighborhoods comes from the use of max pooling layers.
[0100] Because we define the same (x, y) as the input image, the output image has the same width and height as the input image, so for every input surface data point there is an output p.
[0101] Figure 7 (a) shows the first example of how a trained machine learning algorithm can be applied in inference mode.
[0102] In step 700, a set of surface types to be predicted is selected. In step 700, only the surface types defined during training in step 502 may be selected, but some of the training types may be grouped. For the current example, it is assumed that the training and inference groups are the same, with the two surface types being "desirable" and "undesirable." However, any number of surface types defined during training may be used.
[0103] In step 701, an enhanced distance map (z, q)(x, y) is recorded.
[0104] In step 702, a score vector p(x,y) about (z,q)(x,y) is inferred for all points (x,y) in the recorded enhanced distance map. The machine learning algorithm provides the inference. Figure 6 For an example convolutional neural network, p(x,y) is given by the value with coordinates (x,y) in the multi-channel output image O, i.e. each channel provides an element of the vector p.
[0105] In step 703, the weight w of p(x, y) in the stitching is found from a function of the surface type weight and the score (eg, as a dot product).
[0106] w = wg(p(x, y))
[0107] where w is a vector of surface type weights and g is an evaluation vector of the scoring function g. For example, w may be chosen to be [1, 0] (the first element representing a "desirable" surface type). The function g may be chosen, for example, as:
[0108]
[0109] Values other than 0.6 may be used as a matter of preference, preferably values greater than 0.5, but in principle values less than 0.5 may also be used. In step 704, the points z(x,y) are stitched into the digital 3D representation built from the previous view and weighted according to their weights found in step 703. For example, the standard ICP algorithm is used for stitching and all points with w>0 are included in the underlying minimization problem. For this first view, the digital 3D representation is set to the point z(x,y).
[0110] The process may then repeat from step 701 to acquire additional views, typically from different poses as the operator moves the scanner, or it may terminate (typically if the operator determines that the digital 3D representation is complete). The first digital 3D representation is then typically converted to a second digital 3D representation: a mesh.
[0111] Figure 7 (b) shows a second example of how to apply a trained machine learning classification algorithm in inference mode.
[0112] Steps 700 to 702 and Figure 7 The same as in the first example of (a).
[0113] In step 714, all points z(x,y) are stitched into the temporary digital 3D representation built from the previous view, as are the values of p(x,y), thereby receiving spatial coordinates of (X,Y,Z) and p(X,Y,Z), respectively, in the space of the digital 3D representation. For practical purposes, the 3D spatial representation of the temporary virtual model is represented as three-dimensional pixels, and the value of p(X,Y,Z) is assigned to the nearest three-dimensional pixel. In each three-dimensional pixel, the value of p is summed over the view and a count is kept of how many values have been added, so that, for example, an average value can be calculated later.
[0114] The process may then repeat from step 701 to acquire additional views, typically from different poses as the operator moves the scanner, or it may terminate (typically if the operator determines that the provisional digital 3D representation is complete).
[0115] In step 713, the weights are found in a similar manner to step 703, but for the average value p, the average value is found by dividing the sum by the number of values. Triplet pixels with a weight of 0 are filtered out, and triplet pixels without any value are also filtered out.
[0116] In step 715, a final virtual model is built from the points (X, Y, Z) in the remaining three pixels, for example, using a marching cubes algorithm.
[0117] Figure 8 Shows how to Figure 7 The algorithm of (b) is extended to also use an example of exclusion volume based filtering. First, the exclusion volume is empty.
[0118] Steps 700-702 and 714 are as described above for Figure 7 (a) and Figure 7 (b) as described above.
[0119] In step 814, the exclusion volume is updated with the stitching point (X, Y, Z) from step 714. The exclusion volume is, for example, the space from the visible surface up to the scanner and the body of the scanner. For practical purposes, the exclusion may be represented on the same three-bit pixel space as used in step 714. A binary flag may indicate whether a three-bit pixel is part of the exclusion space.
[0120] The process may then be repeated for additional views as described above from step 701. As more views are recorded, the number of tri-pixels marked as being in the exclusion space may increase, but never decrease.
[0121] Step 713 is as above for Figure 7 (a) and Figure 7 (b) explained.
[0122] At step 813, all tri-pixels that still contain the summed value of p but are in the excluded volume are deleted. This step is easiest to implement if the tri-pixel spaces are identical; otherwise, a nearest neighbor search can be used to find the closest tri-pixel in the excluded volume tri-pixel space.
[0123] Step 715 is as above for Figure 7 (a) and Figure 7 (b) explained.
[0124] Although some embodiments have been described and shown in detail, the present invention is not limited to them, but can also be implemented in other ways within the scope of the subject matter defined in the appended claims. In particular, it should be understood that other embodiments can be utilized and structural and functional modifications can be made without departing from the scope of the present invention.
[0125] In a device claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The mere fact that certain measures are recited in mutually different dependent claims or described in different embodiments does not indicate that a combination of these measures cannot be used to advantage.
[0126] A claim may refer to any one of the preceding claims, and "any" should be understood to refer to "any one or more" of the preceding claims.
[0127] It should be emphasized that when the term "include / comprises" is used in this specification, it should be regarded as specifying the existence of the stated features, integers, steps or components, but does not exclude the existence or addition of one or more other features, integers, steps, components or groups thereof.
[0128] literature :
[0129] [1] Khoshelham K, Dos Santos DR, Vosselman G; “Generation and weighting of 3D point correspondences for improved registration of RGB-D data”; Proc. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. 2013 Nov 11;5: W2.
[0130] [2] Kazhdan M, Hoppe H. “Screened Poisson surface reconstruction”; ACM Transactions on Graphics (TOG); June 1, 2013; 32(3): 29
[0131] [3] Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohi P, Shotton J, Hodges S, Fitzgibbon A; KinectFusion: “Real-time dense surface mapping and tracking”, 10th IEEE International Symposium on “Mixed and Augmented reality (SIMAR)”, 26 October 2011, (pp. 127–136), IEEE.
[0132] [4] Long J, Shelhamer E, Darrell T; “Fully convolutional networks for semantic segmentation”; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition 2015 (pp. 3431–3440).
Claims
1. A method for generating a digital 3D representation of at least a portion of an oral cavity, the method comprising: - recording a plurality of views containing surface data representing at least the geometry of surface points of the portion of the oral cavity using an intraoral scanner; - determining a weight for each surface point based at least in part on a score that is a measure of confidence that the surface point represents a particular type of surface; - executing a stitching algorithm that performs a weighted stitching of surface points in the plurality of views to generate the digital 3D representation based on the determined weights; wherein the score of the surface point is found by at least one score finding algorithm that takes as input at least a geometric portion of the surface data of the surface point and surface data of points near the surface point, wherein the scanner also provides certain deterministic information of the measured surface data for the recorded views, and wherein the deterministic information at least partially modifies the score, The deterministic information includes: the certainty of the measured geometrical part of the surface data obtained from the uniqueness of the focus metric, or Certainty is obtained from the degree of consistency between at least two cameras of the intraoral scanner.
2. The method according to claim 1, wherein: The at least one score finding algorithm is a machine learning algorithm.
3. The method according to claim 2, wherein: The at least one machine learning algorithm comprises a neural network having at least one convolutional layer.
4. The method according to claim 2 or 3, wherein: The at least one machine learning algorithm is trained on multiple types of surfaces recorded in the oral cavity using a scanner.
5. The method according to claim 1 or 2, wherein: The surface data also includes color information.
6. The method according to claim 1 or 2, wherein: Prior to generating the digital 3D representation, at least one machine learning algorithm is trained at least in part using data recorded by the intraoral scanner.
7. The method according to claim 1 or 2, wherein: At least one machine learning algorithm is trained at least in part by an operator of the intraoral scanner.
8. The method according to claim 1 or 2, wherein: Other algorithms evaluate geometric consistency over multiple views.
9. A scanner system for reconstructing a digital 3D representation of at least a portion of an oral cavity, the scanner system comprising: -Handheld intraoral scanner; - A processing unit configured to perform the steps of the method according to any of claims 1-8.
