A method for detecting segmentation results of three-dimensional digital models of teeth and jaws

Abstract

One aspect of the present application provides a method for detecting segmentation results of a three-dimensional digital model of a jaw, which comprises: obtaining a reference position segmentation result of a three-dimensional digital model of a jaw to be inspected, wherein the segmentation result is obtained by segmenting the three-dimensional digital model of the jaw positioned at the reference position using a first segmentation method; perturbing the three-dimensional digital model of the jaw from the reference position multiple times, and segmenting the three-dimensional digital model of the jaw located at multiple disturbance positions using the first segmentation method to obtain corresponding multiple disturbance position segmentation results; and judging whether the reference position segmentation result is credible based on the similarity between the reference position segmentation result and the multiple disturbance position segmentation results, wherein the segmentation of the three-dimensional digital model of the jaw is to segment it according to different teeth and gums, and to classify the face pieces of the three-dimensional digital model of the jaw according to tooth numbers and gums.

Description

Technical Field

[0001] The present application generally relates to a method for detecting segmentation results of a three-dimensional digital model of a tooth or jaw. Background Art

[0002] Nowadays, dental treatment is increasingly relying on computer technology. In many cases, it is necessary to segment the scanned three-dimensional digital model of the dental jaw including the dentition and at least part of the gums, so as to separate the crowns from the gums and also to separate the individual crowns.

[0003] Currently, there are semi-automatic and fully-automatic methods for segmenting three-dimensional digital models of teeth and jaws. However, there is no reliable method for automatically detecting the segmentation results of three-dimensional digital models of teeth and jaws. Therefore, it is necessary to provide a method for detecting the segmentation results of three-dimensional digital models of teeth and jaws. Summary of the invention

[0004] One aspect of the present application provides a method for detecting segmentation results of a three-dimensional digital model of a jaw, which comprises: obtaining a reference position segmentation result of a three-dimensional digital model of a jaw to be inspected, wherein the segmentation result is obtained by segmenting the three-dimensional digital model of the jaw positioned at the reference position using a first segmentation method; perturbing the three-dimensional digital model of the jaw from the reference position multiple times, and segmenting the three-dimensional digital model of the jaw located at multiple disturbance positions using the first segmentation method to obtain corresponding multiple disturbance position segmentation results; and judging whether the reference position segmentation result is credible based on the similarity between the reference position segmentation result and the multiple disturbance position segmentation results, wherein the segmentation of the three-dimensional digital model of the jaw is to segment it according to different teeth and gums, and to classify the face pieces of the three-dimensional digital model of the jaw according to tooth numbers and gums.

[0005] In some embodiments, the first segmentation method may be a segmentation method based on a deep learning artificial neural network, and the deep learning artificial neural network is trained with a plurality of three-dimensional digital models of the jaw positioned at the reference position.

[0006] In some embodiments, it is characterized in that the deep learning artificial neural network can be a DGCNN network.

[0007] In some embodiments, the method for detecting the segmentation results of the three-dimensional digital model of the jaw may further include: for each category, calculating the similarity between the reference position segmentation result and each of the disturbance position segmentation results; for each category, obtaining the confidence of the reference position segmentation result based on the corresponding similarity calculation; obtaining the representative confidence of the reference position segmentation result based on the confidence calculation of all categories; and judging whether the reference position segmentation result is credible based on the representative confidence and a preset threshold.

[0008] In some implementations, for each category, the similarity between the reference bit segmentation result and each of the perturbed bit segmentation results is calculated based on the number of patches that belong to the category in both.

[0009] In some implementations, the global confidence may be the minimum value of all classification confidences.

[0010] In some embodiments, the method for detecting the segmentation results of the three-dimensional digital model of the tooth and jaw may also include: if the reference position segmentation result is not credible, using the same method to determine whether there is a credible segmentation result among the multiple disturbance position segmentation results; if so, using the credible disturbance position segmentation result as the final segmentation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features of the present application will be further described below in conjunction with the accompanying drawings and detailed descriptions thereof. It should be understood that these drawings only illustrate several exemplary embodiments according to the present application and should not be considered as limiting the scope of protection of the present application. Unless otherwise specified, the drawings are not necessarily to scale and similar reference numerals represent similar components.

[0012] Figure 1 A schematic flow chart of a method for segmenting a three-dimensional digital model of a tooth or jaw in one embodiment of the present application;

[0013] Figure 2 A schematic module diagram of a DGCNN network in one embodiment of the present application; and

[0014] Figure 3 The present invention is a schematic flowchart of a method for detecting segmentation results of a three-dimensional digital model of a tooth or jaw in one embodiment of the present application. DETAILED DESCRIPTION

[0015] The following detailed description refers to the drawings that form part of this specification. The illustrative embodiments mentioned in the specification and drawings are for illustrative purposes only and are not intended to limit the scope of protection of the present application. Under the guidance of the present application, those skilled in the art will understand that many other embodiments can be adopted and various changes can be made to the described embodiments without departing from the subject matter and scope of protection of the present application. It should be understood that the various aspects of the present application described and illustrated herein can be arranged, replaced, combined, separated and designed according to many different configurations, and these different configurations are within the scope of protection of the present application.

[0016] In order to improve the segmentation accuracy and efficiency of the three-dimensional digital model of the tooth and jaw, and reduce the degree of human intervention, one aspect of the present application provides a new segmentation method of the three-dimensional digital model of the tooth and jaw.

[0017] Please refer to Figure 1 , which is a schematic flow chart of a method 100 for segmenting a three-dimensional digital model of a tooth or jaw in one embodiment of the present application.

[0018] In 101, a three-dimensional digital model of the jaw to be segmented is obtained.

[0019] In one embodiment, the three-dimensional digital model of the jaw to be segmented can be obtained by scanning the patient's oral cavity (for example, scanning with a laser oral scanning device). In another embodiment, the three-dimensional digital model of the jaw to be segmented can also be obtained by scanning the patient's jaw model (for example, a plaster model).

[0020] In one embodiment, the three-dimensional digital model of the jaw to be segmented is composed of triangular facets, and the high-precision three-dimensional digital model of the jaw includes a large number of triangular facets, for example, more than 100,000, or even more than 150,000.

[0021] In 103, the three-dimensional digital model of the tooth and jaw to be segmented is positioned at a reference position.

[0022] In order to ensure the robustness of the deep learning artificial neural network used to segment the three-dimensional digital model of the tooth and jaw, the three-dimensional digital model of the tooth and jaw is positioned to a predefined reference position when training the artificial neural network and when using it to segment the three-dimensional digital model of the tooth and jaw. The reference position includes direction and position.

[0023] In one embodiment, multiple representative reference three-dimensional digital models of the upper and lower jaws can be selected based on prior knowledge, for example, five reference three-dimensional digital models of the upper and lower jaws, corresponding to normal dental arches, short dental arches, long dental arches, wide dental arches, and narrow dental arches, respectively. In light of the present application, it can be understood that the selection of reference three-dimensional digital models of the upper and lower jaws is not limited to the above examples. For example, three reference three-dimensional digital models of the upper and lower jaws can be selected based on "Analysis of Dimensions and Shapes of Maxillary and Mandibular Dental Arch in Korean Young Adults" published by Su-Jung Park, Richard Leesungbok, Jae-Won Song, Se HunChang, Suk-Won Lee, and Su-Jin Ahn in The journal of advanced prosthodontics, 9(5):321–327, 2017, namely, oval, V-shaped, and U-shaped reference three-dimensional digital models of the upper and lower jaws.

[0024] In one embodiment, the reference three-dimensional digital model of the jaw is located at a reference position, and the three-dimensional digital model of the jaw to be segmented can be positioned at the reference position by registering with the reference three-dimensional digital model of the jaw.

[0025] In one embodiment, an iterative closest point (ICP) algorithm may be used to position the 3D digital model of the jaw to be segmented to the reference position by registering the 3D digital model of the jaw to be segmented with the reference 3D digital model of the jaw.

[0026] In one embodiment, the three-dimensional digital model of the jaw to be segmented can be registered with five reference three-dimensional digital models of the jaw, and the one with the highest matching degree is selected as the position after registration. Then, the average center (centroid) of a large number of previously registered three-dimensional digital models of the jaw is calculated and used as the center position. Finally, the center of the registered three-dimensional digital model of the jaw to be segmented is translated to the center position, and it is considered that the three-dimensional digital model of the jaw to be segmented has been positioned to the reference position.

[0027] In 105, the three-dimensional digital model of the to-be-segmented tooth and jaw is converted into a point cloud to obtain a point cloud.

[0028] Since the ICP algorithm is used in the above embodiment to register the three-dimensional digital model of the jaw to be segmented with the reference three-dimensional digital model of the jaw, the three-dimensional digital model of the jaw to be segmented has been converted into a point cloud, and the point cloud operation does not need to be performed. However, if the three-dimensional digital model of the jaw to be segmented is not converted into a point cloud during the process of positioning it to the reference position, then the point cloud operation needs to be performed at this time.

[0029] In 107 , feature extraction is performed on the points in the point cloud.

[0030] After a large number of experiments, the inventors of this application found that for each point, the following features are extracted and segmentation is performed based on this with high accuracy: the coordinates of the center point of the patch (each point in the point cloud can be the center point of the corresponding patch) (x, y, z, a total of 3 features), the patch normal vector (3 features) and the vector from the center point of the patch to the three vertices (9 features), a total of 15 features.

[0031] In 109, the point cloud is segmented based on the extracted features using a trained dynamic graph convolutional neural network.

[0032] In one embodiment, a dynamic graph convolutional neural network (DGCNN) may be used to classify (segment) the point cloud.

[0033] In one embodiment, 33 labels may be set, representing 32 teeth and gums, for classification (segmentation).

[0034] In one embodiment, in order to improve computational efficiency, the point cloud may be sampled, and only the sampled points may be classified using the trained DGCNN network.

[0035] In one embodiment, the sampling may be uniform sampling.

[0036] In one embodiment, the number of sampling points may be set according to the capability of the computing system, for example, 30,000 sampling points may be taken.

[0037] In one embodiment, in order to further improve the computing efficiency, the sampling points can be evenly grouped, and then each group of sampling points can be classified using the trained DGCNN network. In one embodiment, the number of groups can be set according to the capability of the computing system, for example, divided into 3 groups.

[0038] When the DGCNN network classifies a point, it will consider its adjacent points (i.e., calculate the feature relationship between the point to be classified and its adjacent points, and consider this feature relationship during classification). The inventors of the present application have found through a large number of experiments that, with a step value of 5, the accuracy of classification increases with the number of adjacent points considered, until 25. Since then, even if the number of adjacent points considered further increases, the accuracy of classification has not been significantly improved. Therefore, preferably, the number of adjacent points considered can be selected within the range of 20-30, and more preferably, the number of adjacent points considered can be selected within the range of 25-30.

[0039] For each sampling point, the DGCNN network outputs its probability distribution on the 33 categories (including 32 teeth and gums), and takes the category with the highest probability as the classification result of the sampling point.

[0040] After completing the classification of all sampling points, it is necessary to classify other points in the point cloud. In one embodiment, a K-nearest neighbor algorithm (KNN) can be used to classify other points in the point cloud based on the classification results of the sampling points.

[0041] The basic idea of ​​the KNN algorithm is that if most of the k most similar samples (i.e., the nearest samples in the feature space) of a sample (here, the point to be classified) in the feature space belong to a certain category, then the sample also belongs to this category. In the KNN algorithm, the selected neighboring samples are the classified samples.

[0042] In one embodiment, the average value of the probability distribution of the neighboring points of the point to be classified may be calculated, and the average value may be used as the probability distribution of the point to be classified, and the classification with the highest probability may be used as the classification result of the point to be classified.

[0043] In one embodiment, k may be set to 5, that is, for each non-sampling point, it is classified based on the classification of the five sampling points closest to it in the feature space.

[0044] The inventors of the present application have found through a large number of experiments that the setting of k has little effect on the classification accuracy and computational complexity of the network, and it only needs to be within a reasonable range.

[0045] Please refer to Figure 2 , which is a schematic module diagram of the DGCNN network 200 in one embodiment of the present application.

[0046] The input module 201 is used to input the extracted features. For the above embodiment, a set of features of 10,000 sampling points extracted may be input, wherein each point has 15 features.

[0047] The T-Net subnetwork 203 is used to realize automatic alignment of point clouds to reduce the spatial variation of points. The T-Net network is a network that predicts the feature space transformation matrix. It learns the transformation matrix consistent with the feature space dimension from the input data, and then multiplies the transformation matrix with the original data to realize the transformation operation of the input feature space, so that each subsequent point is related to each point in the input data. Through such data fusion, the step-by-step abstraction of the features contained in the original point cloud data is realized.

[0048] The DGCNN network 200 in this embodiment includes three edge convolution modules (Edge Convolution) 205, 207 and 209, wherein each edge convolution module has three layers. The inventors of the present application have found through a large number of experiments that setting a three-layer depth for the edge convolution module can ensure the prediction (classification) accuracy of the network. If the depth of the edge convolution module is further increased, the prediction accuracy of the network will be limited, and it may be more likely to cause overfitting and increase the amount of calculation.

[0049] The DGCNN network 200 in this embodiment can segment the upper and lower jaw three-dimensional digital models as a whole, and the one-hot encoded category module 213 is used to distinguish whether a point belongs to the upper jaw or the lower jaw.

[0050] The DGCNN network 200 in this embodiment also includes three two-dimensional convolution modules 211, 215 and 217.

[0051] in, Figure 2 In The symbol represents feature concatenation.

[0052] The output module 219 outputs the probability distribution of all points on 33 categories.

[0053] In 111 , the segmentation result is optimized.

[0054] In order to eliminate possible minor local unevenness in the segmentation result (for example, burrs on the boundary, or the classification of a certain face patch is different from that of surrounding faces), the segmentation result may be smoothed.

[0055] In one embodiment, the Graph-Cut algorithm may be used to smooth the segmentation result based on the geometric relationship between the facets of the three-dimensional digital model of the tooth and jaw.

[0056] In one embodiment, two losses may be set, namely, classification loss and geometric loss. The weighted sum of the classification loss and the geometric loss is minimized through the Graph-Cut algorithm to achieve smooth classification results.

[0057] The classification loss can be defined as: the loss of smoothing the predicted tooth number of the current patch (i.e., the classification result) to other tooth numbers is equal to the value of the current patch on the probability distribution output by the automatic segmentation system, that is, the greater the probability, the greater the loss of smoothing. When calculating the classification loss, all possibilities can be calculated, that is, the loss when the current patch is smoothed to other 32 categories is calculated.

[0058] The geometric loss can be defined as: the loss of smoothing the predicted tooth number of the current patch to the predicted tooth number of its adjacent patch, which is equal to the product of the distance from the center point of the current patch to the center point of its adjacent patch and the dihedral angle.

[0059] Then, the Graph-Cut algorithm is used to minimize the weighted sum of the annotation loss and the geometric loss to achieve smoothing.

[0060] In one embodiment, in the process of minimizing the loss, a non-negative constant λ can be set as the weight of the geometric loss to balance the impact of the annotation loss and the geometric loss on the total loss. Since the boundary between the teeth and the gums is less obvious, the segmentation result is more likely to be believed at this time. Therefore, the geometric loss weight between the gums and the teeth can be set smaller than the geometric loss weight in other cases. That is to say, when the classification of the current point and the adjacent point is one tooth and the other gum, when the classification of the current point is smoothed to the classification of the adjacent point, its geometric loss weight is smaller than other cases. After a large number of experiments, the inventors of the present application found that when the geometric loss weight λ between the gums and the teeth is set to 50 and the geometric loss weight λ between the teeth is set to 250, the optimization effect is better.

[0061] In one embodiment, the minimization loss can be expressed by the following expression (1):

[0062]

[0063] Among them, the first term is the classification loss, the second term is the geometric loss, F represents the face set of the three-dimensional digital model of the jaw, i represents the i-th face, l i Represents the classification of the i-th patch, and its corresponding probability is p i ,in,

[0064] ξ U (p i ,l i )=-log(p i (l i )) Expression (2)

[0065]

[0066] Among them, θ ij represents the dihedral angle between facet i and facet j, φ ij Represents the distance between the centers of patch i and patch j.

[0067] In light of the present application, it can be understood that although the above embodiment segments the upper and lower jaw three-dimensional digital models as a whole, the method of the present application is also applicable to segmenting the upper and lower jaw three-dimensional digital models separately (correspondingly, the number of annotations can be changed to 17, representing 16 teeth and gums respectively).

[0068] Although the segmentation method of the above-mentioned three-dimensional digital model of the dental jaw in the present application has a very high accuracy rate, it still cannot ensure 100% segmentation accuracy for some extreme cases. Therefore, it is necessary to provide a method for detecting the credibility of the classification results.

[0069] Currently, the industry lacks a method that can detect the credibility of the segmentation results of the three-dimensional digital model of the tooth and jaw by the artificial neural network. After extensive research, the inventor of this application has developed a method for evaluating the credibility of the segmentation results based on disturbance. The basic idea of ​​this method is to detect whether the segmentation method is sensitive to the position disturbance of the current three-dimensional digital model of the tooth and jaw. If so, the current segmentation result is considered unreliable.

[0070] Please refer to Figure 3 , which is a method 300 for detecting the credibility of the segmentation result of the three-dimensional digital model of the tooth and jaw by an artificial neural network in one embodiment of the present application.

[0071] From the above, it can be seen that in the three-dimensional digital model segmentation method of the present application, before using the DGCNN network to segment the three-dimensional digital model of the jaw, it is necessary to position it to the reference position, because the training of the network also uses the three-dimensional digital model of the jaw positioned at the reference position, thereby ensuring the accuracy of the segmentation.

[0072] In 301, a reference bit segmentation result is obtained.

[0073] In one embodiment, the reference position segmentation result may be a result obtained by segmenting the three-dimensional digital model of the tooth jaw positioned at the reference position using the segmentation method of the present application.

[0074] In 303, a plurality of disturbed bit segmentation results are generated.

[0075] In one embodiment, the three-dimensional digital model of the dental jaw positioned at the reference position can be rotated along the x, y, and z axes by plus or minus 5 degrees and plus or minus 10 degrees, respectively, to obtain 12 groups of disturbances.

[0076] Next, the three-dimensional digital model of the tooth and jaw located at the 12 disturbance positions is segmented using the same segmentation method to obtain 12 groups of corresponding disturbance position segmentation results.

[0077] In light of the present application, it can be understood that the disturbance mode and the number of disturbance positions can be determined according to the specific circumstances and are not limited to the above specific examples. For example, the disturbance mode can include translation, rotation, and a combination of the two, or the disturbance can be performed along any axis or direction.

[0078] In 305 , whether the reference bit segmentation result is credible is determined based on the similarity between the reference bit segmentation result and the plurality of disturbance bit segmentation results.

[0079] In one embodiment, for each tooth number X, the number of patches with tooth number X in the reference bit segmentation result is A, the number of patches with tooth number X in the first perturbation bit segmentation result is B, and the number of patches with tooth number X in both the reference bit segmentation result and the first perturbation bit segmentation result is C (C<=A, C<=B). The minimum value of C / A and C / B can be taken as the similarity of the tooth number X prediction (annotation) between the reference bit and the first perturbation bit. If the denominator of C / A or C / B is 0, this fraction can be assigned a value of 1.

[0080] For each tooth number X, repeat the above operation 12 times (corresponding to the number of perturbation position segmentation results), and 12 similarities will be obtained. For each tooth number X, take the first quartile (Q1) of these 12 similarities as the confidence of the segmentation method for tooth number X. Take the minimum value of all tooth number confidences as the representative confidence of the segmentation result of the segmentation method for the three-dimensional digital model of the jaw located at the reference position.

[0081] In one embodiment, a threshold may be set. If the representative confidence of the segmentation result is greater than the threshold, the segmentation result is considered to be credible, otherwise the segmentation result is considered to be unreliable.

[0082] In one embodiment, based on current parameter settings, the threshold may be set to 0.85.

[0083] In light of this application, it is understood that, for each tooth number, the confidence level can be replaced by other values ​​besides the first quartile of the 12 similarities, such as the second quartile or the average value, etc. Accordingly, the threshold may need to be modified.

[0084] Although the tooth three-dimensional digital model segmentation method in the embodiment of the present application is a point-based segmentation method, it can be understood from the inspiration of the present application that the segmentation result detection method of the present application is also applicable to the facet-based segmentation method.

[0085] Inspired by the present application, in addition to the segmentation method of the present application, the segmentation result detection method of the present application is applicable to any neural network three-dimensional digital model segmentation method of the tooth and jaw that is based on positioning to a reference position for training and segmentation.

[0086] When a segmentation result is determined to be unreliable, there are two options: one is to remind the user that manual intervention is required for verification and / or adjustment; the other is to try to filter out a reliable segmentation result as the final output from the multiple perturbation bit segmentation results. If a reliable segmentation result cannot be found among the multiple perturbation bit segmentation results, the user is notified to manually intervene for verification and / or adjustment.

[0087] In one embodiment, the 12 disturbed bit segmentation results can be traversed to find the disturbed bit segmentation result with the highest confidence and output it as the final segmentation result. In this case, even if the segmentation result needs to be repaired manually, the workload of manual repair can be reduced.

[0088] Although various aspects and embodiments of the present application are disclosed herein, other aspects and embodiments of the present application will be apparent to those skilled in the art in light of the present application. The various aspects and embodiments disclosed herein are for illustrative purposes only and not for limiting purposes. The scope and subject matter of the present application are determined solely by the appended claims.

[0089] Likewise, various diagrams may illustrate exemplary architectures or other configurations of the disclosed methods and systems that aid in understanding the features and functions that may be included in the disclosed methods and systems. The claimed content is not limited to the exemplary architectures or configurations shown, and the desired features may be implemented with various alternative architectures and configurations. In addition, for flow charts, functional descriptions, and method claims, the order of blocks presented herein should not be limited to various embodiments that are implemented in the same order to perform the described functions, unless otherwise clearly indicated in the context.

[0090] Unless otherwise expressly noted, the terms and phrases used herein and their variations should be interpreted as open ended, not restrictive. In some instances, the appearance of broad words and phrases such as "one or more", "at least", "but not limited to", or other similar terms should not be understood as an intent or need to indicate a narrowing of the example where such broad terms may not be present.

Claims

1. A method for detecting segmentation results of a three-dimensional digital model of a tooth and jaw, comprising: Acquire a reference position segmentation result of the three-dimensional digital model of the tooth and jaw to be inspected, wherein the segmentation result is obtained by segmenting the three-dimensional digital model of the tooth and jaw positioned at the reference position using a first segmentation method; Disturbing the three-dimensional digital model of the jaw from the reference position for multiple times, and segmenting the three-dimensional digital model of the jaw at multiple disturbance positions using the first segmentation method to obtain corresponding multiple disturbance position segmentation results; and Based on the similarity between the reference bit segmentation result and the plurality of disturbance bit segmentation results, determining whether the reference bit segmentation result is credible, The segmentation of the three-dimensional digital model of the jaw is to segment it according to different teeth and gums, and to classify the surface of the three-dimensional digital model of the jaw according to the tooth number and gums.

2. The method for detecting the segmentation result of the three-dimensional digital model of the tooth and jaw according to claim 1, characterized in that: The first segmentation method is a segmentation method based on a deep learning artificial neural network, and the deep learning artificial neural network is trained with a plurality of three-dimensional digital models of the jaws positioned at the reference positions.

3. The method for detecting the segmentation result of the three-dimensional digital model of the tooth and jaw according to claim 2, characterized in that: The deep learning artificial neural network is a DGCNN network.

4. The method for detecting the segmentation result of the three-dimensional digital model of the tooth and jaw according to claim 1, characterized in that: It also includes: For each classification, calculating the similarity between the reference bit segmentation result and each of the perturbation bit segmentation results; For each classification, the confidence of the benchmark segmentation result is obtained based on the corresponding similarity calculation; Calculating the representative confidence of the reference bit segmentation result based on the confidence of all classifications; and Whether the reference bit segmentation result is credible is determined based on the representative confidence and a preset threshold.

5. The method for detecting the segmentation result of the three-dimensional digital model of the tooth and jaw according to claim 4, characterized in that: For each category, the similarity between the reference bit segmentation result and each of the perturbed bit segmentation results is calculated based on the number of patches of the category in both.

6. The method for detecting the segmentation result of the three-dimensional digital model of the tooth and jaw according to claim 4, characterized in that: The global confidence is the minimum value of all classification confidences.

7. The method for detecting the segmentation result of the three-dimensional digital model of the tooth and jaw according to claim 1, characterized in that: It also includes: if the reference bit segmentation result is not credible, using the same method to determine whether there is a credible segmentation result among the multiple disturbance bit segmentation results, and if so, taking the credible disturbance bit segmentation result as the final segmentation result.

Images