Method for detecting key points of a three-dimensional digital model of teeth

Abstract

The application provides a computer-implemented method for detecting key points of a three-dimensional digital model of teeth, which includes: obtaining a first three-dimensional digital model representing teeth of a patient; sampling the first three-dimensional digital model to obtain N sampling points; extracting features from the N sampling points; and detecting key points of the first three-dimensional digital model based on the extracted features using a trained deep neural network, the deep neural network including a backbone network, a first head network, a second head network, and a post-processing module, the backbone network generating high-dimensional features based on the extracted features, the first head network generating a coarse detection result of key points based on the high-dimensional features, the coarse detection result being a heat map, the second head network generating residual error vectors between each sampling point and the nearest feature point on the first three-dimensional digital model based on the high-dimensional features, and the post-processing module adjusting the coarse detection result using the residual error vectors and performing a non-maximum suppression operation to obtain a fine detection result of key points.

Description

Technical Field

[0001] This application generally relates to a method for detecting key points in a three-dimensional digital model of teeth. Background Technology

[0002] With the development of computer technology, dental treatment is becoming increasingly reliant on computer technology.

[0003] Accurate detection of key dental points is crucial for orthodontic treatment planning and clinical evaluation. These key points play a vital role in various clinical applications, including aligning teeth using three-dimensional digital models, determining occlusal relationships, and locating attachments.

[0004] However, due to the varied anatomical structure of human teeth, crowding, and overlapping, accurate detection of key dental points is extremely difficult. In clinical practice, even slight deviations in the results of key dental point detection can lead to incorrect diagnoses or ineffective treatment plans.

[0005] Currently, existing methods for detecting key points in teeth have limited accuracy, making it difficult to meet the sub-millimeter precision requirements of clinical practice. Therefore, it is necessary to provide a new method for detecting key points in three-dimensional digital models of teeth. Summary of the Invention

[0006] One aspect of this application provides a computer-executed method for detecting key points in a three-dimensional digital model of teeth. The method includes: acquiring a first three-dimensional digital model representing a patient's teeth; sampling the first three-dimensional digital model to obtain N sampling points, where N is a predetermined number; extracting features from the N sampling points; and detecting key points in the first three-dimensional digital model based on the extracted features using a trained deep neural network. The deep neural network includes a backbone network, a first head network, a second head network, and a post-processing module. The backbone network generates high-dimensional features based on the extracted features. The first head network generates a coarse detection result for the key points based on the high-dimensional features; this result is a heatmap characterizing whether each sampling point in the first three-dimensional digital model is a key point. The second head network generates a residual vector between each sampling point in the first three-dimensional digital model and its nearest feature point based on the high-dimensional features. The post-processing module adjusts the coarse detection result using the residual vector and performs non-maximum suppression on the adjusted coarse detection result to obtain a fine detection result for the key points. The fine detection result is the coordinates of the selected sampling points after adjustment using the corresponding residual vector.

[0007] In some implementations, the non-maximum suppression operation includes: for each heatmap, sorting the sampling points by confidence level; adding the sampling point with the highest confidence level to the candidate set; calculating the distance between the sampling point with the second highest confidence level and every sampling point in the candidate set; if the distance between the sampling point and any sampling point in the candidate set is less than a preset threshold, deleting the sampling point; otherwise, adding the sampling point to the candidate set, wherein the distance is calculated based on the coordinates of the sampling point after adjustment by the corresponding residual vector; and so on, until all sampling points have been filtered.

[0008] In some embodiments, the method for detecting key points in the three-dimensional digital model of teeth further includes: filtering the heatmap using a preset confidence threshold before the non-maximum suppression operation to filter out sampling points with low confidence.

[0009] In some implementations, the sampling is the farthest point sampling.

[0010] In some implementations, the first three-dimensional digital model is a triangular mesh model, and the extracted features include the center point of the patch, the patch normal vector, and the three vertices of the patch.

[0011] In some implementations, the deep neural network is a deep neural network capable of processing point clouds or voxels.

[0012] In some implementations, the deep neural network is DGCNN.

[0013] In some implementations, during the training of the deep neural network, the second head network uses the vector between the calibrated keypoints and each sampling point as the ground truth.

[0014] In some implementations, the key points are divided into multiple categories, and the first head network generates a heatmap for each key point category.

[0015] Another aspect of this application provides a computer system for detecting key points of a three-dimensional digital model of teeth, comprising a storage device and a processor. The storage device stores a computer program for detecting key points of a three-dimensional digital model of teeth, and when the program is run, the processor executes the key point detection method for a three-dimensional digital model of teeth as described in claim 1. Attached Figure Description

[0016] The above and other features of this application will be further described below with reference to the accompanying drawings and their detailed description. It should be understood that these drawings only illustrate several exemplary embodiments according to this application and should not be considered as limiting the scope of protection of this application. Unless otherwise specified, the drawings are not necessarily to scale, and similar reference numerals denote similar parts.

[0017] Figure 1 This is a schematic flowchart illustrating a computer-executed method for detecting key points in a three-dimensional digital model of teeth, according to one embodiment of this application.

[0018] Figure 2A A three-dimensional digital model representing a patient's maxillary dentition is shown in an example.

[0019] Figure 2B This demonstrates the use of deep neural networks in [the following context]: Figure 2A Key points detected on the 3D digital model of the teeth shown; and

[0020] Figure 3 This is a schematic block diagram of a deep neural network used to detect key points of a three-dimensional digital model of teeth in one embodiment of this application. Detailed Implementation

[0021] The following detailed description incorporates the accompanying drawings, which form part of this specification. The illustrative embodiments mentioned in the specification and drawings are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art will understand, based on the teachings of this application, that many other embodiments can be employed and various changes can be made to the described embodiments without departing from the spirit and scope of this application. It should be understood that the various aspects of this application illustrated herein can be arranged, substituted, combined, separated, and designed in many different configurations, all of which are within the scope of this application.

[0022] One aspect of this application provides a computer-executed method for detecting key points on a 3D digital model of teeth. It utilizes a coarse detection module of a trained deep neural network to detect key points on the 3D digital model of teeth, obtaining coarse detection results. Then, it uses a regression module of the deep neural network to calculate a residual vector. Finally, it uses the residual vector to adjust the coarse detection results to obtain the final detection result. Experimental verification shows that the method of this application can effectively improve the detection accuracy of key points on a 3D digital model of teeth.

[0023] Another aspect of this application provides a computer system for detecting key points of a three-dimensional digital model of a tooth, comprising a storage device and a processor. The storage device stores a computer program for detecting key points of a three-dimensional digital model of a tooth, and when the program is run, the processor executes a method for detecting key points of the three-dimensional digital model of the tooth.

[0024] Please refer to Figure 1 This is a schematic flowchart of a computer-executed method 100 for detecting key points of a three-dimensional digital model of teeth in one embodiment of this application.

[0025] In step 101, a first three-dimensional digital model representing the patient's teeth is obtained.

[0026] The first three-dimensional digital model can be a three-dimensional digital model representing a single tooth of the patient, or it can be a three-dimensional digital model representing multiple teeth of the patient, for example, a three-dimensional digital model representing the patient's maxillary or mandibular dentition.

[0027] The first three-dimensional digital model is a mesh three-dimensional digital model, for example, a triangular mesh three-dimensional digital model.

[0028] Please refer to Figure 2A This example demonstrates a three-dimensional digital model representing a patient's maxillary dentition.

[0029] In step 103, the first three-dimensional digital model is sampled.

[0030] N points are sampled on the first three-dimensional digital model, where N is a predetermined number.

[0031] In one embodiment, the sampling can be performed using the Farthest Point Sampling (FPS) method. It is understood that other applicable sampling methods besides FPS can also be used.

[0032] In step 105, feature extraction is performed on the sampling points.

[0033] In one embodiment, for each sampling point, the following features are extracted as input to the subsequent deep neural network: the center point of the patch containing the sampling point, the patch normal vector, and the three vertices of the patch.

[0034] In step 107, a trained deep neural network is used to detect key points of the first three-dimensional digital model based on the extracted features.

[0035] Key points (landmarks) of teeth, also known as feature points, can be broadly categorized into the following four types based on their location:

[0036] (1) Mesial Points and Distal Points: These points are located on the adjacent surfaces of the teeth. The mesial point is located on the side of the tooth closer to the dental midline, while the distal point is located on the side of the tooth further away from the dental midline. These points are very important for determining the alignment and positioning of the teeth on the dental arch.

[0037] (2) Cusp Point: A cusp is a sharp or rounded protrusion on the chewing surface of a tooth. The cusp point is the highest point on a tooth. The cusp point is crucial for determining the occlusal relationship between the upper and lower teeth.

[0038] (3) Inner Points and Outer Points: These points are located at the junction of the teeth and gums. Inner points are located inside the teeth, near the tongue or palate, while outer points are located outside the teeth, near the cheek or lip. These points are crucial for determining the position of the teeth.

[0039] (4) FA point (Facial Axis Point): The FA point is located at the center of the labial or buccal surface of the tooth.

[0040] The FA point is very important for determining tooth angulation and inclination.

[0041] Please refer to Figure 3 This is a schematic block diagram of a deep neural network 200 used to detect key points of a three-dimensional digital model of teeth in one embodiment of this application.

[0042] The deep neural network 200 includes a backbone network 201, a first head network 203, a second head network 205, and a post-processing module 207.

[0043] Backbone network 201 generates high-dimensional features based on the features extracted from 105, and outputs them to the first head network 203 and the second head network 205 respectively.

[0044] The first head network 203 predicts coarse detection results of key points based on the high-dimensional features, wherein the coarse detection results are heatmaps, which represent the confidence level of whether each sampled point is a key point. In one embodiment, during training, the loss function of the first head network 203 can be L2 loss.

[0045] In one embodiment, a heatmap can be generated for each type of key point, so that for a three-dimensional digital model including multiple teeth, each heatmap has multiple key points.

[0046] Based on the classification method of key points described above, as inspired by this application, key points can be further subdivided according to their orientation. For example, near midpoint and far midpoint can be classified as two categories, and in-point and out-point as two categories. The classification methods for key points are not limited to the examples above.

[0047] The second head network 205 predicts a residual vector based on the high-dimensional features. This residual vector represents the offset of each sampling point to the nearest keypoint. Each sampling point in each heatmap has a residual vector. The second head network 205 is a regression network used to regress the residual vector. In one embodiment, it can be a two-layer multilayer perceptron (MLP). In one embodiment, the loss function of the second head network 205 during training can be L1 loss.

[0048] On the one hand, since the input of the deep neural network 200 is based on the features extracted from the sampling results of the three-dimensional digital model of the teeth, the real key points may not be in the sampling points. The fine inspection result obtained by adjusting the coarse inspection result using the residual vector can significantly improve the detection accuracy of the key points. That is, compared with the coarse inspection result, the fine inspection result is closer to the real key points.

[0049] On the other hand, since each heatmap contains multiple key points, when the detection results are subsequently processed using the Non-Maximum Suppression (NMS) method, some closely spaced key points may be suppressed. Adjusting the coarse detection results using residual vectors narrows down the range of candidate coordinates for each key point, thus avoiding this situation and significantly improving the robustness of NMS.

[0050] The post-processing module 207 performs post-processing based on the heatmap and the residual vector to obtain the fine inspection result.

[0051] In one embodiment, the post-processing includes the following operations:

[0052] First, the heatmap is filtered using a pre-set confidence threshold to remove points with low confidence, thereby reducing false positives.

[0053] Next, the filtered detection result is adjusted using the residual vector, that is, the coarse detection result is added to the corresponding residual vector.

[0054] Finally, the adjusted detection results are processed using a class-specific nonmaximum suppression method to eliminate redundant detection results and ensure that only the points with the highest confidence and the most accurate spatial location are selected as key points.

[0055] The process of processing a heatmap adjusted using the corresponding residual vector using the nonmaximum suppression method is roughly as follows: First, the sampling points are sorted by confidence level; then, the sampling point with the highest confidence level is added to the candidate point set; next, the point with the second highest confidence level is selected, and the distance between this point and every point in the candidate point set is calculated. If the distance between this point and any point in the candidate point set is less than a preset threshold, the point is deleted; otherwise, it is added to the candidate point set. The distance is calculated based on the coordinates adjusted by the corresponding residual vector. This process continues until all sampling points have been filtered. In one embodiment, the threshold can be set based on statistical results; for example, the threshold can be 0.5 mm.

[0056] After the non-maximum suppression process, the coordinates of multiple sampling points are obtained after adjustment by the corresponding residual vectors. These coordinates are the coordinates of the key points corresponding to the heatmap, i.e., the fine inspection results.

[0057] In one embodiment, the deep neural network 200 may be based on a dynamic graph convolutional neural network (DGCNN).

[0058] In one embodiment, each layer of the deep neural network 200 performs a K-Nearest Neighbor search, based solely on the coordinates of the points input to the deep neural network 200. Thus, the K-Nearest Neighbor search is calculated only once, and the result is used in each layer, significantly improving computational efficiency.

[0059] It is understandable that, in addition to the DGCNN network, Deep Neural Network 200 can also be based on any other applicable deep neural network, such as a deep neural network capable of processing point clouds, such as Point Transformer, or a deep neural network capable of processing voxels, such as 3DUnet.

[0060] During the training of the deep neural network 200, the difference between the labeled key points and the key points in the coarse detection results is used as the ground truth of the second head network 205.

[0061] Please refer to Figure 2B This demonstrates the capabilities of deep neural networks 200. Figure 2A Key points detected on the three-dimensional digital model of the teeth shown.

[0062] Please refer to Table 1 below, which shows the test results for each scheme.

[0063] plan mAP@0.5mm (%) Basic scheme (N=5000) 25.7 +N＝10000 31.6 +NMS 34.3 +FPS 40.0 +Residual Vector Regression 49.3 +N＝20000 53

[0064] Where mAP@0.5mm(%) represents the average accuracy (mean Average Precision) of the detected feature points being less than 0.5mm from the true feature points, and N is the number of samples.

[0065] With the basic sampling scheme (5000 samples), mAP@0.5mm was 25.7%. Increasing the sampling number to 10000, mAP@0.5mm was 31.6%. Based on this, using NMS, mAP@0.5mm was 34.3%. Using FPS, mAP@0.5mm was 40.0%. Using residual vector regression, mAP@0.5mm was 49.3%. Further increasing the sampling number to 20000, mAP@0.5mm reached 53%.

[0066] As can be seen from the test results above, residual vector regression significantly improves detection accuracy, reaching 9.3%.

[0067] Although various aspects and embodiments of this application have been disclosed herein, other aspects and embodiments of this application will be apparent to those skilled in the art upon inspiration from this application. The various aspects and embodiments disclosed herein are for illustrative purposes only and not for limiting purposes. The scope and spirit of this application are determined solely by the appended claims.

[0068] Similarly, the diagrams may illustrate exemplary architectures or other configurations of the disclosed methods and systems, which aid in understanding the features and functions that may be included in the disclosed methods and systems. The claims are not limited to the exemplary architectures or configurations shown, and the desired features may be implemented with various alternative architectures and configurations. Furthermore, the order of the blocks given herein with respect to flowcharts, functional descriptions, and method claims should not be limited to various embodiments implemented in the same order to perform the said functions, unless explicitly indicated in the context.

[0069] Unless otherwise expressly stated, the terms and phrases used herein, and their variations thereof, should be interpreted as open-ended rather than restrictive. In some instances, the appearance of extended words and phrases such as “one or more,” “at least,” “but not limited to,” or other similar expressions should not be construed as an intention or necessity to indicate a narrower scope in examples where such extended expressions might not exist.

Claims

1. A method for detecting key points in a computer-executed three-dimensional digital model of teeth, comprising: Obtain a first three-dimensional digital model representing the patient's teeth; The first three-dimensional digital model is sampled to obtain N sampling points, where N is a predetermined number; Feature extraction is performed on the N sampling points; and The key points of the first 3D digital model are detected using a trained deep neural network based on the extracted features. The deep neural network includes a backbone network, a first head network, a second head network, and a post-processing module. The backbone network generates high-dimensional features based on the extracted features. The first head network generates coarse detection results for key points based on the high-dimensional features. This result is a heatmap characterizing whether each sampled point on the first three-dimensional digital model is a key point. The second head network generates residual vectors between each sampling point and the nearest feature point on the first three-dimensional digital model based on the high-dimensional features. The post-processing module uses the residual vector to adjust the coarse inspection result and performs non-maximum suppression operation on the coarse inspection result adjusted by the residual vector to obtain the fine inspection result of the key point. The fine inspection result is the coordinate of the selected sampling point after adjustment by the corresponding residual vector.

2. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, The non-maximum suppression operation includes: For each heatmap, the sampling points are sorted according to confidence level; The sampling point with the highest confidence level is added to the candidate set; Calculate the distance between the sampling point with the second highest confidence and each sampling point in the candidate set. If the distance between the sampling point and any sampling point in the candidate set is less than a preset threshold, delete the sampling point; otherwise, add the sampling point to the candidate set. The distance is calculated based on the coordinates of the sampling point after adjustment by the corresponding residual vector. This process continues until all sampling points have been screened.

3. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, It also includes: before the non-maximum suppression operation, filtering the heatmap using a preset confidence threshold to filter out sampling points with low confidence.

4. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, The sampling mentioned is the sampling at the farthest point.

5. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, The first three-dimensional digital model is a triangular mesh model, and the extracted features include the center point of the patch, the normal vector of the patch, and the three vertices of the patch.

6. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, The deep neural network is a deep neural network capable of processing point clouds or voxels.

7. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 6, characterized in that, The deep neural network in question is DGCNN.

8. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, In the training of the deep neural network, the second head network uses the vector between the calibrated key points and each sampling point as the ground truth.

9. The method for detecting key points in a three-dimensional digital model of teeth as described in claim 1, characterized in that, The key points are divided into multiple categories, and the first head network generates a heatmap for each key point category.

10. A computer system for detecting key points of a three-dimensional digital model of teeth, comprising a storage device and a processor, wherein the storage device stores a computer program for detecting key points of a three-dimensional digital model of teeth, and when the program is run, the processor executes the key point detection method of a three-dimensional digital model of teeth as described in claim 1.

Images