Method and system for grading tooth wear based on deep learning

Abstract

The present invention provides a method and system for grading and judging the degree of tooth wear based on deep learning. The present invention proposes for the first time a method for grading and judging the degree of tooth wear based on deep learning, which can effectively solve the problems of low accuracy in the existing tooth image segmentation technology, such as high segmentation reflection, blurred edges, and relatively small structure details, as well as the existing image classification method's overfitting for small sample data and lack of effective data enhancement mode for small sample data.

Description

Technical Field

[0001] The present invention relates to an intelligent grading method for tooth wear degree, and in particular to a grading judgment method and system for tooth wear degree based on deep learning. Background Art

[0002] The current application of deep learning in the field of stomatology is mainly to perform intelligent diagnosis of diseases such as dental caries in the oral cavity based on image data. However, there are no reports on the use of deep learning technology to assist in the diagnosis of tooth wear.

[0003] The existing deep learning-based tooth image segmentation technology has the problem of low accuracy in segmenting detailed images with high reflection, blurred edges, and relatively small structures. At the same time, image segmentation also suffers from overfitting for small sample data. Among image classification methods, existing image classification methods suffer from overfitting for small sample data and lack effective data enhancement modes for small sample data. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a method and system for grading the degree of tooth wear based on deep learning.

[0005] Specifically, the method for grading the degree of tooth wear based on deep learning includes the following steps:

[0006] Detect teeth in the input image and perform image segmentation;

[0007] The image of a single tooth output after image segmentation is classified and input into the classification judgment model, the degree of wear of each tooth surface is graded, and then the bounding box information generated by the segmentation task is used to locate the original image. The classification results are collected in the original input image and returned, and the graded judgment results of the degree of wear of each tooth surface are output to realize the judgment of the degree of wear of the teeth in the input image.

[0008] Preferably, the image segmentation of the tooth image includes two parts: image segmentation fine structure recognition and image segmentation edge segmentation.

[0009] Preferably, in the recognition of small structures in image segmentation, a segmentation network model based on Mask R-CNN is selected to realize single tooth segmentation. On this basis, the SGE attention mechanism module is introduced into the Backbone structure to enhance the accuracy of small sample tooth recognition.

[0010] Preferably, in the Backbone structure, an SGE attention mechanism module is inserted between its ResNet module and FCN module.

[0011] Preferably, the number of segmentations of the SGE attention mechanism module in the image segmentation stage is 8, which can help the model better recognize small tooth structures.

[0012] In the above scheme, in order to solve the problem of fine structure recognition in image segmentation, the SGE attention mechanism module is added to the Backbone structure according to the overall structure of Mask R-CNN to highlight the correct features, enhance the accuracy of tooth recognition, and the precision of tooth recognition for small samples. The SGE attention mechanism module is a lightweight attention module. By dividing the features into different sub-features, SGE can highlight the correct sub-features more specifically. In the Backbone structure, inserting the SGE layer between the ResNet module and the FCN module can improve the mPA of the segmentation results to a certain extent.

[0013] Preferably, when performing image segmentation and edge segmentation, the FCN module is replaced by a U-Net module to further improve the model's recognition accuracy for fine structures and edges.

[0014] Preferably, an SGE attention mechanism module is further added in the splicing process of the U-Net module to improve the accuracy of mask output by strengthening effective features.

[0015] As a preferred choice, the U-Net module selects a structure of 3 downsampling layers, 3 upsampling layers, a total of 6 layers, and a learning rate of 10 -5 .

[0016] In the above scheme, the FCN module is replaced by the U-Net module to solve the problem of edge segmentation in image segmentation. The U-Net network is a high-precision network for small sample image segmentation tasks. By splicing the data in the downsampling process with the data in the upsampling process, a thicker feature that incorporates contextual information is formed to avoid losing the upper-level feature details during the training process. Replacing the FCN module in the mask generation module with the U-Net module can optimize the mask edge segmentation and improve the accuracy of mask segmentation. Adding the SGE attention mechanism module in the splicing process of the U-Net network can improve the accuracy of mask output by strengthening effective features. Specifically, the feature vector generated in the encoding process is input into the SGE attention mechanism module. In the SGE attention mechanism, the features are first grouped, and the features of each group are spatially dot-producted with the features after global pooling and the features are normalized. Then the output features of the SGE module are spliced ​​with the feature vector generated in the decoding process to strengthen the effective features in the upper-level information of the U-Net.

[0017] Preferably, the image of a single tooth output after image segmentation is classified and input into a subsequent classification judgment model to grade the degree of wear of each tooth surface, including:

[0018] First, the ResNet18 image classification network is used to classify the labial and buccal surfaces and occlusal surfaces in the input single tooth data, and then a network model based on Vision Transformer is selected to classify the wear level of each tooth surface. Among them, a mask mechanism is introduced in the network model based on Vision Transformer.

[0019] Preferably, the mask pixel generation ratio is 20%, and the mean generation range is 2*2 pixels.

[0020] In the above scheme, in the image classification task, a network based on Vision Transformer (ViT) is selected. By adding a mask mechanism to the ViT model, the size of the training set can be increased to a certain extent to avoid overfitting. The Mask mechanism comes from the idea of ​​BERT. Since the training data set is small, the pixels of the input image are replaced with a mask generated by the mean of the surrounding pixels with a certain probability during model training. This operation can optimize the overfitting problem caused by a small data set to a certain extent.

[0021] Preferably, the degree of tooth wear is divided into four wear grades of 0-3 according to the modified TWI index, depending on whether there is dentin exposure and the proportion of the dentin exposure area.

[0022] Preferably, the four wear grades of 0-3 are no dentin exposure, dentin exposure less than 1 / 3 of the tooth surface, dentin exposure greater than 1 / 3 of the tooth surface, and presence or absence of pulp exposure or secondary dentin formation.

[0023] The present invention also provides an artificial intelligence identification system for tooth wear degree, which is embedded with a deep learning-based grading method for tooth wear degree as described in any of the above technical solutions.

[0024] Compared with the prior art, the advantages and positive effects of the present invention are:

[0025] 1. The method for grading and judging the degree of tooth wear based on deep learning proposed in the present invention can solve the problem of small structures being difficult to identify, such as a large amount of reflections in high-resolution oral cavity images and small structures being difficult to distinguish from the reflective parts. By adding an SGE attention mechanism module to the Mask R-CNN network for feature extraction, the extracted features can be input into the SGE module to obtain a feature vector with enhanced effective features, thereby enhancing the accuracy of tooth recognition; on this basis, for the segmentation problem of blurred edges of images, the U-Net network with the SGE module added is used for image mask generation, which can improve the model's edge recognition accuracy. Furthermore, for the overfitting phenomenon that occurs in classification tasks for smaller data sets, a mask mechanism is added to the ViT model, and the pixels of the input image are selected to be masked according to a certain ratio, thereby optimizing the overfitting problem caused by the small data set to a certain extent.

[0026] 2. The present invention proposes for the first time a method for grading the degree of tooth wear based on deep learning. This method can effectively solve the problems of low accuracy in existing tooth image segmentation technology, such as high segmentation reflections, blurred edges, and relatively small structures in detail images, as well as the problem of overfitting of existing image classification methods for small sample data and the lack of effective data enhancement modes for small sample data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Angles for collecting intraoral photos for clinical use provided by the embodiments of the present invention;

[0028] Figure 2 Examples of various wear levels in the classification input provided in the embodiment of the present invention (0 to 3 from left to right);

[0029] Figure 3-4 A comparison diagram of the Mask R-CNN model structure provided in an embodiment of the present invention and the optimized Mask R-CNN model structure combining the attention mechanism and the U-Net algorithm;

[0030] Figure 5 A schematic diagram of the structure of a U-Net model with SGE added provided in an embodiment of the present invention;

[0031] Figure 6 The segmentation output results of different models for the same image in the test set provided by the embodiment of the present invention;

[0032] Figure 7-8 This is a comparison chart of the Vision Transformer model structure and the Vision Transformer model structure after optimization with the mask mechanism added;

[0033] Fig. 9 The overall flow chart for the present invention;

[0034] Fig.10 This is an example diagram of the results output by the present invention in an application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0036] Example 1 Data Collection and Annotation

[0037] 1.1 Data Collection

[0038] Intraoral photographs of the upper and lower permanent dental arches were collected by dental professionals using a digital camera (3216×2136 pixels, Canon EOS 60D, Canon Co., Ltd., Japan) with the aid of a plastic retractor and a reflector. The sensitivity was 200, the aperture value was F32, and the shutter speed was 1 / 400-1 / 200. A total of 6 photos were collected for each patient, including frontal small mouth opening photos, protrusive and left and right lateral occlusal photos, and upper and lower dental arch occlusal surface photos ( Figure 1 ). Inclusion criteria for photos: ① The image is moderately exposed, not out of focus, and the tooth surface of a single tooth is clear; ② The tooth surface is not obstructed by obvious soft plaque, tartar, blood stains, etc.; ③ The tooth surface is not obstructed by auxiliary photographing equipment, mirror stains, or soft tissue; ④ The angle between the camera and the tooth surface is moderate, and the photo is not excessively distorted. Exclusion criteria: ① The photo is underexposed or overexposed; ② The photo is obviously out of focus; ③ The tooth surface is obstructed by foreign objects; ④ The shooting angle is too tilted. A total of - photos of the upper and lower jaws of - patients who visited the Peking University School of Stomatology from October 2013 to June 2020 were obtained, and the patients were - years old.

[0039] 1.2 Data Annotation

[0040] In the classification task, the degree of tooth wear is divided into four wear levels from 0 to 3 according to the modified TWI index, whether there is dentin exposure, and the proportion of dentin exposure area. The image annotation is based on the tooth surface, and each tooth surface is labeled according to four categories: no dentin exposure, dentin exposure less than 1 / 3 of the tooth surface, dentin exposure greater than 1 / 3 of the tooth surface, and whether there is pulp exposure or secondary dentin formation. Examples of the various wear levels entered in the classification task are as follows: Figure 2 shown.

[0041] VoTT software (Windows 2.1.0, Microsoft, USA) was used to classify and label each tooth and tooth surface in the entire dentition on the image data. VoTT is a labeling tool developed based on javascript for image target detection. It supports labeling and annotating images and videos, and exports labels in different formats, which can be used for neural network training. The software is characterized by labeling by region, that is, single tooth images and interproximal space images are classified and labeled according to the confirmed diagnosis.

[0042] In the process of machine learning, it is generally believed that the ratio of each data set is between 7.0:2.0:1.0 and 6.0:2.0:2.0. According to the number of tooth wear images, in this study, each label group data was divided into training set, validation set and test set in a ratio of 6:2:2 using a random number table method. The training and validation data sets were used together to build the network model, and the images in the test data set were used to evaluate the diagnostic performance of the network model and ultrasound physicians.

[0043] To ensure the accuracy of the annotation, each case was annotated by two dental residents, and then the consistency test was performed. After the formal annotation was completed, a chief physician with 25 years of clinical work experience randomly checked the annotation accuracy by drawing lots. A total of 489 teeth and 186 adjacent areas were randomly checked, and the consistency rate of the annotation classification was 100%.

[0044] Example 2: Construction of a grading method for determining the degree of tooth wear based on deep learning and an artificial intelligence recognition system

[0045] The artificial intelligence recognition system for tooth wear degree based on deep learning is embedded with a grading judgment method for tooth wear degree based on deep learning, and the grading judgment method includes the following steps:

[0046] Detect teeth in the input image and perform image segmentation;

[0047] The image of a single tooth output after image segmentation is classified and input into the classification judgment model, the degree of wear of each tooth surface is graded, and then the bounding box information generated by the segmentation task is used to locate the original image. The classification results are collected in the original input image and returned, and the graded judgment results of the degree of wear of each tooth surface are output to realize the judgment of the degree of wear of the teeth in the input image.

[0048] As can be seen from the above, the method / system includes two parts: tooth image segmentation and tooth wear degree grading. Among them, the tooth image segmentation part chooses to use a model that combines Mask R-CNN, U-Net and SGE attention mechanism, while the ViT model with mask mechanism is selected for the tooth wear grading task.

[0049] Evaluation and statistical analysis of tooth wear degree recognition accuracy: SPSS Statistics 26.0 software (IBM, USA) was used for statistical processing. In the evaluation of tooth image segmentation tasks, mPA (average pixel accuracy of categories) was mainly selected to judge the accuracy. In the task of grading the degree of tooth wear, precision (Precision, the proportion of correct prediction results in all events predicted as positive), recall (Recall, the proportion of correct prediction results in all positive events), F1-Score (the harmonic mean of precision and recall), and accuracy (Accuracy, the proportion of correctly predicted samples in the total number of samples) were used as evaluation indicators for the accuracy of artificial intelligence recognition and grading.

[0050] 2.1 Image Segmentation

[0051] The tooth image segmentation task includes two parts: image segmentation fine structure recognition and image segmentation edge segmentation. In order to better achieve the segmentation of a single tooth and provide a data source for the next step of tooth wear determination, the segmentation network based on Mask R-CNN is selected to achieve single tooth segmentation for the problem of image segmentation fine structure recognition. The model is based on the Mask R-CNN architecture ( Figure 3 ), an SGE attention mechanism module is inserted between the ResNet-50 module and the FCN module in the Backbone structure. Specifically, an SGE attention mechanism module is inserted between the ResNet-50 and FCN modules to enhance the accuracy of small sample tooth recognition. Furthermore, for the problem of edge segmentation in image segmentation, the FCN module is replaced with a U-Net module to improve the model's recognition accuracy for small structures and edges ( Figure 4 ). In the process of splicing the U-Net network, the SGE attention mechanism can also be added to improve the accuracy of mask output by strengthening effective features ( Figure 5 ).

[0052] In the above task, in the image segmentation stage, the number of SGE module segmentations was set to 8. After adding the SGE module, the model had better recognition of fine tooth structures. The U-Net structure selected a structure with 3 downsampling layers and 3 upsampling layers, a total of 6 layers, and a learning rate of 10 -5 In the image classification task, the mask pixel generation ratio is 20%, and the mean generation range is 2*2 pixels.

[0053] In the segmentation task, the training set contains 64 images and the test set contains 14 images. The segmentation output results of each model for the same image in the test set are as follows: Figure 6As shown in Table 1, the accuracy is shown in Table 1. In the model evaluation, the mPA indicator is mainly selected to judge the accuracy.

[0054] Table 1 mPA indicators of several image segmentation algorithms

[0055]

[0056] From the mPA accuracy results in Table 1, we can see that after the introduction of the attention mechanism, since it is more sensitive to fine structures, it can mark the areas that the original model did not recognize during recognition, and the mPA index is improved more significantly. After the introduction of U-Net, the model is more sensitive to the recognition of masked areas and can better recognize the noise caused by light in the picture. The introduction of the two structures can significantly improve the accuracy of the segmentation results.

[0057] 2.2 Image Classification

[0058] In the image classification task, the wear level of a single tooth is determined by classifying the single tooth image output in the segmentation task.

[0059] First, the classification task is divided into two subtasks. First, the ResNet18 image classification network is used to classify the labial and buccal surfaces and occlusal surfaces of the input single tooth data. Then, the VisionTransformer model with the mask mechanism is used ( Figure 8-9 ) Classify the wear grade of each tooth surface.

[0060] In the classification task, the segmentation results output in the segmentation task are selected as the data set for training. In the output of the segmentation task, a total of 2774 single tooth images are selected as the input data for the classification task, including 778 occlusal surface images and 1996 lip and cheek surface images. In the result evaluation, four parameters, F1-Score, Precision, Recall, and Accuracy, are selected to evaluate the classification accuracy. The test results are shown in Table 2. In the classification of the lip and cheek surface and the occlusal surface, the classification accuracy of ResNet18 is 99.7%, which basically meets the subsequent classification requirements.

[0061] Table 2 Accuracy of different classification models

[0062] F1-Score Precision Recall Accuracy ViT occlusal surface 0.91 0.92 0.86 0.90 ViT+mask occlusal surface 0.92 0.93 0.91 0.92 ViT Lip and Cheek 0.86 0.82 0.91 0.90 ViT+mask lips and cheeks 0.87 0.93 0.82 0.93

[0063] As shown in Table 2, the accuracy of the ViT model in occlusal surface classification without the introduction of the mask mechanism is 90%, and the accuracy is 92% after the introduction of the mask mechanism. The accuracy of the ViT model in lip and cheek surface classification without the introduction of the mask mechanism is 90%, and the accuracy is 93% after the introduction of the mask mechanism.

[0064] 2.3 Labeling of classification results

[0065] In the process of determining and labeling the degree of wear of a single tooth in an oral image ( Fig. 9 ), locate the position information of each tooth to be judged through the bounding box information generated in the segmentation task, input the segmented tooth image into the classification network for wear category identification, combine the result returned by the classification network with the position information of the bounding box, and realize the annotation of the tooth wear grade identification in the original input image. Fig.10 shown.

[0066] Combined with the above examples and analysis, it can be seen that the method for grading the degree of tooth wear based on deep learning of the present invention can effectively solve the problems of low accuracy in the existing tooth image segmentation technology, such as the segmentation of detail images with a lot of reflections, blurred edges, and relatively small structures, as well as the problem that the existing image classification method has overfitting for small sample data and lacks an effective data enhancement mode for small sample data. It can realize the effective grading judgment of the degree of tooth wear.

Claims

1. A method for grading tooth wear based on deep learning. It is characterized in that The following steps are involved: Detect teeth in the input image and perform image segmentation on them. The image segmentation includes two parts: image segmentation fine structure recognition and image segmentation edge segmentation, where: In the recognition of fine structures in image segmentation, a segmentation network model based on Mask R-CNN is selected to achieve single tooth segmentation. On this basis, an SGE attention mechanism module is introduced between the ResNet module and the FCN module of the Backbone structure to enhance the accuracy of small sample tooth recognition. When segmenting the edges of an image, the FCN module is replaced by the U-Net module, and the SGE attention mechanism module is further added during the splicing process of the U-Net module to further improve the model's edge recognition accuracy by strengthening effective features. Then, the image of a single tooth output after image segmentation is classified and input into the classification judgment model, the degree of wear of each tooth surface is graded, and then the bounding box information generated by the segmentation task is used to locate the original image, the classification results are collected in the original input image and returned, and the graded judgment results of the degree of wear of each tooth surface are output to realize the judgment of the degree of wear of the teeth in the input image.

2. The classification determination method according to claim 1, It is characterized in that The image of a single tooth output after image segmentation is classified and input into the classification judgment model to grade the degree of wear of each tooth surface, including: First, the ResNet18 image classification network is used to classify the labial and buccal surfaces and occlusal surfaces in the input single tooth data, and then a network model based on Vision Transformer is selected to classify the wear level of each tooth surface. Among them, a mask mechanism is introduced in the network model based on Vision Transformer.

3. The classification determination method according to claim 2, It is characterized in that According to the modified TWI index, the degree of tooth wear is divided into four wear grades from 0 to 3 according to whether there is dentin exposure and the proportion of dentin exposure area.

4. The classification determination method according to claim 3, It is characterized in that The four wear grades of 0-3 are no dentin exposure, dentin exposure less than 1 / 3 of the tooth surface, dentin exposure greater than 1 / 3 of the tooth surface, and whether there is pulp exposure or secondary dentin formation.

Images