Deep learning-based color image early caries diagnosis model

Abstract

The invention relates to the technical field of computer science and oral medical treatment, and discloses a color image early caries diagnosis model based on deep learning, which comprises an image collection stage and a model construction stage, and the model construction stage comprises training of a deep learning network and operation of a caries diagnosis model. The method comprises the following steps: S1, screening caries-containing images: training an image classification network to divide an intraoral image into a healthy intraoral image group and a caries-containing intraoral image group; s2, dental caries detection: training a target detection network to mark dental caries in the image in the oral cavity containing caries; and S3, dental caries segmentation and numbering: marking and segmenting the teeth in the dental caries-containing intraoral image and numbering the teeth. According to the scheme, a large amount of intraoral image data are collected and marked and are respectively used for training and testing of different classification, target detection and segmentation networks, and then the network with the best performance in each task is selected for construction of a dental caries diagnosis model. And finally, performing overall operation and analysis on the constructed diagnosis model so as to realize complete diagnosis of caries in the intraoral image.

Description

Technical Field

[0001] The present invention relates to the fields of computer science and oral medical technology, and in particular to a color image early caries diagnosis model based on deep learning. Background Art

[0002] Artificial intelligence is developing rapidly, and deep learning, as an important branch, has also been explored in the rapid and automatic diagnosis of caries. Early applications mainly focused on the use of deep convolutional neural networks (CNNs) to detect and segment caries lesions in periapical films, U-Net models to detect caries on bitewing films, ResNet models to classify caries on bitewing films, and CNN models to analyze panoramic films and CBCT images. However, the existing technologies all recognize and process oral radiographic images obtained by X-ray transmission, but cannot detect directly taken intraoral images, and the practical scope is relatively narrow; oral radiographic images are difficult to use to diagnose small changes in the enamel surface, and the accuracy is low. They still need to rely on naked eye observation and cannot achieve rapid recognition and detection.

[0003] In recent years, the YOLOv3 model, U-Net structure, ResNet-18 and Faster R-CNN have also been used to automatically detect caries in intraoral images taken by mobile phones. This has further reduced the requirements for caries screening facilities, making it possible to promote this model in low- and middle-income countries. However, the existing technology still has the following technical problems: On the one hand, most deep learning solutions for auxiliary caries diagnosis focus on the detection and classification of caries lesion areas. In order to complete a complete diagnosis, dentists still need to identify the specific location of the affected teeth and manually number these teeth, which has a very limited improvement in diagnostic efficiency. On the other hand, it is worth noting that most existing caries diagnosis systems use a supervised learning framework, which means that doctors need to manually label all training and test data, which is a very time-consuming and labor-intensive task, thereby reducing diagnostic efficiency.

[0004] In view of this, developing a caries diagnosis model that can automatically segment and number diseased teeth can not only effectively make up for the shortcomings of existing technologies, but also effectively improve the efficiency of caries diagnosis and effectively reduce the repetitive work of dentists, which is of great significance for the rapid identification and diagnosis of caries. Summary of the invention

[0005] The present invention aims to provide a color image early caries diagnosis model based on deep learning to solve the technical problem that the existing caries diagnosis system still requires doctors to manually mark or number caries in intraoral images, thereby reducing the diagnostic efficiency.

[0006] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a color image early caries diagnosis model based on deep learning, including an image collection stage and a model construction stage. The image collection stage includes collecting intraoral occlusal surface images of patients; the model construction stage includes training a deep learning network and running a caries diagnosis model.

[0007] Preferably, as an improvement, the training of the deep learning network comprises the following steps:

[0008] S1. Screening of images containing caries: training an image classification network to classify intraoral images into two groups: healthy intraoral images and intraoral images containing caries;

[0009] S2. Caries detection: Train the object detection network to annotate caries in intraoral images containing caries and classify them into two categories: early stage caries and mid- to late stage caries.

[0010] S3. Tooth segmentation and numbering: Train the image segmentation network to annotate and segment teeth in images of caries inside the mouth and annotate them with numbers.

[0011] Preferably, as an improvement, in S1, the image classification network is trained as follows: ResNet34, ResNet50 and ResNet101 networks are selected to screen the upper and lower mandibular intraoral images for caries images, and the various indicators of the three in the test set are compared, and the ResNet34 network with the best effect is selected to screen and group the upper and lower mandibular intraoral images for caries images.

[0012] Preferably, as an improvement, in S2, the target detection network is trained as follows: YOLOv5s, YOLOv5m and YOLOv5x networks are selected to detect and classify caries in images containing caries in the mouth, and various indicators of the three in the test set are compared, and the YOLOv5s network with better effect is selected to automatically identify and locate early caries and mid-to-late stage caries.

[0013] Preferably, as an improvement, in S2, the image segmentation network is trained as follows: U-Net, U-Net++ and MT-Net networks are selected to segment and number caries in images of oral cavity containing caries, various indicators of the three in the test set are compared, and the MT-Net network with better effect is selected to segment and number the diseased teeth in images of oral cavity containing caries.

[0014] Preferably, as an improvement, an evaluation stage of the deep learning model is also included, and the diagnostic performance of the caries diagnosis model of this scheme is evaluated with reference to the classification and labeling results of dentists.

[0015] Preferably, as an improvement, the method for obtaining the classification and labeling results of the dentist includes the dentist classifying, labeling and segmenting the images, including the following steps:

[0016] Step 1: Collect the patient's intraoral occlusal surface image;

[0017] Step 2: Classification of teeth in the image: The dentist classifies each tooth in the intraoral image into healthy teeth, teeth with early caries changes, and teeth with mid- to late-stage caries changes according to the International Caries Detection and Assessment System (ICDAS);

[0018] Step 3: Segmentation and numbering of teeth in the image: The dentist segments each tooth in the intraoral image except the third molar in turn, and numbers the segmented teeth according to the International Dental Federation numbering system (FDI).

[0019] Preferably, as an improvement, the overall operation steps of the caries diagnosis model are as follows: first, the intraoral images are classified using the ResNet34 network. Healthy intraoral images are diagnosed as "healthy" after classification, and intraoral images containing caries are further detected and segmented and numbered using YOLO v5s and MT-Net structures; the model extracts the pixel coordinates of the caries detection results and the pixel coordinates of each tooth that has completed the segmentation and numbering, and then overlaps and compares them in turn, calculates the pixel overlap rate of the two, and extracts the numbering result of the tooth with the highest overlap rate with the detected caries, thereby diagnosing the severity and tooth position numbering of all diseased teeth at the same time.

[0020] Preferably, as an improvement, the calculation formula of the pixel overlap rate is: TP is a true positive pixel, FP is a false positive pixel, and FN is a false negative pixel.

[0021] The principles and advantages of this solution are:

[0022] 1. Compared with the existing caries diagnosis system, which can only handle part of the caries diagnosis work in intraoral images (such as the detection and classification of caries lesion areas), resulting in other parts of the work (such as numbering, etc.) still needing to be completed by doctors, thus reducing the diagnostic efficiency, this solution collects and annotates a large amount of intraoral image data, which is used for training and testing of different classification, target detection and segmentation networks, and then selects the network with the best performance in each task for the construction of the caries diagnosis model. Finally, the constructed diagnostic model is run and analyzed as a whole, thereby achieving a complete diagnosis of caries in intraoral images.

[0023] 2. In the screening process of this scheme, the ResNet34 network framework uses shortcut connections to alleviate the gradient vanishing problem. Its prediction layer is located after the convolution layer, and this layer uses the sigmoid activation function for binary classification (healthy intraoral images are 0, and intraoral images with caries are 1). In order to improve the difference between the predicted labels and the true labels, the cross entropy loss is used to provide a specific goal for this optimization process, so that it can infer the correct category in noisy samples. Then, the adaptive moment estimation algorithm is combined with the StepLR strategy to optimize the neural network parameters and effectively improve the accuracy of intraoral image screening. The applicant found through long-term experiments that, referring to the doctor's marking results, the accuracy of the screening stage of this scheme is as high as 100%.

[0024] 3. In the process of training the target detection network, this scheme uses the non-maximum suppression algorithm to post-process the generated prediction results. By sorting the confidence, the non-maximum suppression algorithm can minimize the overlap of the target detection box, thereby improving the detection accuracy. Unlike the optimizer used in the caries image screening stage, this scheme uses stochastic gradient descent to adjust the network parameters in the caries detection stage, while retaining the cross entropy loss as the loss function, effectively improving the detection accuracy. In the process of caries segmentation and numbering, the selected MT-Net network framework trains both labeled and unlabeled samples, and the content of unlabeled samples in the data set is significantly more than that of labeled samples. The parameters of its teacher model are updated in real time through exponential moving average. Its loss function consists of two parts, namely supervised loss and unsupervised consistency loss. The supervised loss includes cross entropy loss and Dice loss, with ratios of 0.999 and 0.001 respectively. The unsupervised consistency loss is the mean square error loss. This scheme calculates the consistency loss of the teacher model after adding noise and the student model without noise to enhance the robustness of the entire framework.

[0025] 4. This scheme effectively evaluates the accuracy and feasibility of the caries diagnosis model in this scheme by comparing the model diagnosis results with the doctor's annotation results, thereby improving the credibility of the model diagnosis results and facilitating the widespread application of the deep learning-based color image early caries diagnosis model in this scheme to assist dentists in diagnosing patients' conditions, improve the efficiency of dentists' diagnosis, and thus quickly provide treatment plans to alleviate patients' pain. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a process flow chart for constructing a color image early caries diagnosis model based on deep learning in an embodiment of the present invention.

[0027] Figure 2 This is the diagnostic operation process and diagnostic results of the color image early caries diagnosis model based on deep learning for intraoral images in an embodiment of the present invention.

[0028] Figure 3 This is a comparison chart of the mean average precision (mAP) of three target detection networks (YOLO v5s, YOLO v5m, and YOLO v5x networks) at a 50% interaction ratio threshold in Experimental Example 2 of the present invention.

[0029] Figure 4 This is a comparison chart of the precision-recall curves of YOLO v5s on the mandibular and maxillary datasets in Experimental Example 2 of the present invention.

[0030] Figure 5 These are the sensitivity and specificity curves of the three networks, U-Net, U-Net++ and MT-Net, in Experimental Example 3 of the present invention.

[0031] Figure 6 This is a diagram showing the process and results of diagnosing some images using the deep learning-based color image early caries diagnosis model in Experimental Example 4 of the present invention.

[0032] Figure 7 The results of the dental doctor's diagnosis process in Experimental Example 4 of the present invention are compared with those of the dental caries diagnosis model of the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with examples, but the embodiments of the present invention are not limited thereto. Unless otherwise specified, the technical means used in the following examples and experimental examples are conventional means well known to those skilled in the art, and the materials, reagents, etc. used can be obtained from commercial sources.

[0034] Example

[0035] This scheme provides a color image early caries diagnosis model based on deep learning, including the following steps:

[0036] 1. Image Collection Stage

[0037] Step 1: Collect the patient's intraoral occlusal surface images.

[0038] All images used in this protocol were taken by dental clinicians or their assistants using professional cameras in the clinic. In order to improve the stability of the deep learning system, the protocol does not make specific requirements on the camera model, lighting environment, and shutter speed during shooting. The intraoral occlusal images taken were numbered in the order of collection without any processing such as cropping or compression. All images were then saved in the corresponding numbered folders in jpg file format. All included intraoral images were required to be clear, without abnormal shadows or occlusions, and to be able to fully display the remaining permanent dentition except the third molars.

[0039] (ii) Image classification and labeling stage: This includes the dentist classifying, labeling, and segmenting and numbering the images.

[0040] Step 2: Image classification: Dentists classify each tooth in the intraoral image into three different categories based on the International Caries Detection and Assessment System (ICDAS).

[0041] As a reference, among the 1436 collected intraoral occlusal images (including healthy intraoral images and some intraoral images with caries), a total of 455 healthy intraoral images and some intraoral images with caries were selected for subsequent training and testing of caries image screening. For the other 981 intraoral images with caries, a recorder used a labeling tool (labelme3.16.2) to frame the carious teeth in the intraoral images with rectangular frames and accurately labeled them as "classification 1" or "classification 2" based on the expert classification results. The labeled data were used for subsequent training and testing of caries detection.

[0042] The classification and annotation method is as follows: During the caries screening and caries detection stages, the collected intraoral images were handed over to three experts in the field of endodontics to identify and classify caries, and the three experts discussed and decided on any disagreements. The plan requires each expert to classify each tooth in the intraoral image into three different categories based on the International Caries Detection and Assessment System (ICDAS). This classification method can provide a reference for the formulation of clinical treatment plans and has important clinical significance. The classification standards of the International Caries Detection and Assessment System (ICDAS) and the classification and annotation results of intraoral images are detailed in Table 1.

[0043] Table 1 Classification criteria of the International Caries Detection and Assessment System (ICDAS)

[0044]

[0045] Step 3: Labeling, segmenting and numbering caries in the image: The dentist segments each tooth in the intraoral image except the third molar in turn, and accurately numbers the segmented teeth according to the International Dental Federation numbering system (FDI).

[0046] Specifically, the dentist first used the labelme 3.16.2 tool to annotate 400 of the other 1,280 intraoral occlusal images. During the annotation process, the doctor segmented each tooth in the intraoral image except the third molars (i.e., "terminal teeth" or "wisdom teeth") in turn, and used the International Dental Federation numbering system (FDI) to accurately number the segmented teeth. Due to the high missing rate of third molars and limited clinical significance, this plan did not segment and number the teeth separately. In addition, during the annotation process, the doctor is required to draw the annotation points close to the morphological edges of the teeth themselves to ensure that the segmented tooth morphology is as consistent as possible with the actual morphology of the teeth. The remaining 880 intraoral images have not been annotated and will be directly used for training the semi-supervised segmentation network.

[0047] 3. Construction of deep learning model

[0048] The process includes the training of the deep learning network and the overall operation of the caries diagnosis model, as follows:

[0049] Step 4: Training of deep learning networks. By testing the performance of different networks and making adjustments, the network with the best performance in each task is selected to build a caries diagnosis model. The process flow chart is as follows: Figure 1 As shown, including the following:

[0050] S1. Screening of images containing caries: The image classification network is trained to classify intraoral images into two groups: healthy intraoral images and intraoral images containing caries. This scheme uses the ResNet34 network to screen the intraoral images of the upper and lower jaws for images containing caries.

[0051] Before the formal screening, ResNet34, ResNet50 and ResNet101 networks were used for training and adjustment. Then the various indicators of the three in the test set were compared, and finally the ResNet34 network with the best effect was selected to screen the upper and lower jaw intraoral images for caries.

[0052] During the screening process, the ResNet34 network framework uses shortcut connections to alleviate the gradient vanishing problem. Its prediction layer is located after the convolution layer, which uses the sigmoid activation function for binary classification (healthy intraoral images are 0, and intraoral images with caries are 1). In order to improve the difference between the predicted labels and the true labels, the cross entropy loss is used to provide a specific goal for this optimization process, so that it can infer the correct category in noisy samples. Then, the adaptive moment estimation algorithm is combined with the StepLR strategy to optimize the neural network parameters. The specific parameters for image screening are as follows: the initial learning rate is 0.0001, the learning rate adjustment step is 10, the gamma value is 0.1, and the batch size is 4.

[0053] S2. Caries detection: The target detection network is trained to annotate the caries in the intraoral images and classify them into two categories: early-stage caries and mid-stage caries. This scheme uses the YOLO v5s network to automatically identify and locate early-stage caries and mid-stage caries.

[0054] In order to select the most suitable detection network, this scheme evaluates the performance of three networks with different parameter sizes in this task, including YOLOv5s, YOLOv5m and YOLOv5x networks. On this basis, this scheme uses the non-maximum suppression algorithm to post-process the generated prediction results. By sorting the confidence, the non-maximum suppression algorithm can minimize the overlap of the target detection box, thereby improving the detection accuracy. The caries detection network at this stage can realize the automatic recognition and positioning of early caries and mid-to-late caries. Unlike the optimizer used in the caries image screening stage, this scheme uses stochastic gradient descent to adjust the network parameters in the caries detection stage, while retaining the cross entropy loss as the loss function. The parameter settings for this stage are as follows: the initial learning rate is 0.001, the batch size is 4, the momentum is 0.9, and the decay is 1e-5.

[0055] S3. Tooth segmentation and numbering: The image segmentation network is trained to annotate and segment the teeth in the intra-oral images and divide them into 28 tooth position number labels. This sub-scheme uses the MT-Net network to segment and number the teeth in the above intra-oral images containing caries.

[0056] Prior to this, it is necessary to train a suitable segmentation network to complete the task of segmenting and numbering teeth in caries-containing intraoral images. This scheme trained and tested three networks, U-Net, U-Net++ and MT-Net, and finally selected the MT-Net network for model construction. This framework trains both labeled and unlabeled samples, and the content of unlabeled samples in the data set is significantly more than that of labeled samples. The parameters of its teacher model are updated in real time through exponential moving average. Its loss function consists of two parts, namely supervised loss and unsupervised consistency loss. The supervised loss includes cross entropy loss and Dice loss, with ratios of 0.999 and 0.001 respectively. The unsupervised consistency loss is the mean square error loss. This scheme calculates the consistency loss between the teacher model after adding noise and the student model without noise to enhance the robustness of the entire framework. The optimizer in this stage still uses stochastic gradient descent. The additional parameters of the tooth segmentation and numbering stage are as follows: the initial learning rate is 0.0001, the momentum is 0.99, the weight decay is 0.0001, and the batch size of the upper and lower jaws is 14 and 20 respectively.

[0057] All networks were trained on a cloud server using the PyTorch architecture, an RTX 4090 GPU with 24GB of video memory (Nvidia) and a 16vCPU Intel(R) Xeon(R).

[0058] Step 5: Overall operation of the caries diagnosis model: After training, the best performing networks are used together to build the caries diagnosis model. The diagnostic operation process and diagnostic results of the caries diagnosis model of this solution for intraoral images are shown in the following figure. Figure 2 As shown in the figure, the model first uses the ResNet 34 network to classify the intraoral images. The healthy intraoral images are diagnosed as "healthy" after classification, and the intraoral images with caries are further detected and segmented using the YOLO v5s and MT-Net structures. Subsequently, the model extracts the pixel coordinates of the caries detection results and the pixel coordinates of each tooth that has been segmented and numbered, and then compares them in turn, calculates the pixel overlap rate of the two, and extracts the numbering result of the tooth with the highest overlap rate with the detected caries, thereby diagnosing the severity and tooth position numbering of all teeth at the same time.

[0059] (IV) In the evaluation stage of the deep learning model, the diagnostic performance of the caries diagnosis model of this scheme is evaluated with reference to the classification and labeling results of dentists.

[0060] Specifically, this scheme recognizes the classification and labeling results of dentists as the gold standard for evaluating the performance of caries diagnosis models. In a specific classification, pixels that are correctly labeled as this category by the deep learning model are called true positives (TP). On the contrary, pixels that are incorrectly labeled as other categories are called false negatives (FN). If a pixel does not belong to the category but is incorrectly labeled as this category, it is called a false positive (FP). Pixels that do not belong to this category and also do not belong to this category in the system segmentation results are called true negatives (TN). The statistical indicators and results of each stage in this scheme are detailed below.

[0061] Step 6: Testing the Deep Learning Network

[0062] First, caries-containing image screening is a binary classification task, the purpose of which is to correctly distinguish healthy images from caries-containing images. This study uses accuracy, precision, recall, and F1-score to evaluate the classification network we use. Accuracy (Equation 1) represents the proportion of correctly classified samples to all samples. Precision (Equation 2) represents the proportion of true positive samples to positive samples identified by the classification network. Recall (Equation 3) represents the ratio of true positive samples to actual positive samples. The F1-score (Equation 4) comprehensively considers precision and recall to evaluate the performance of the classification network under imbalanced categories.

[0063] Accuracy = (TP + TN) / (TP + TN + FP + FN + e-7) (Equation 1)

[0064] Precision = TP / (TP + FP + e-7) (Equation 2)

[0065] Recall = TP / (TP + FN + e-7) (Equation 3)

[0066] F1-score = 2 × Precision × Recall / (Precision + Recall) (Equation 4)

[0067] In the above equation, TP is the true positive pixel, TN is the true negative pixel, FP is the false positive pixel, and FN is the false negative pixel.

[0068] Secondly, in the caries detection stage, the goal of this study is to detect the specific location of caries and the classification of their severity, so as to assist dentists in making clinical decisions. In order to evaluate the effect of caries detection, we used a series of statistical indicators, including precision, recall, mAP@50, and mAP@50:90. The average precision (AP) indicator represents the area under the curve of precision-recall (PR), while mAP represents the average area under the curve of all categories.

[0069] mAP@50: A more tolerant target detection metric, representing the mAP when the Intersection over Union (IoU) threshold is set to 50%. Only target detection boxes with an IoU greater than or equal to 50% are considered correct.

[0070] mAP@50:90: Compared with mAP@50, this metric is more stringent and represents the mAP for IoU thresholds ranging from 50% to 90%.

[0071] Subsequently, in the tooth segmentation and numbering stage, the teeth in the intraoral images were classified into 28 different categories without considering the background. In order to evaluate the segmentation effect of the model, we applied multiple indicators for statistical evaluation, including precision, recall, F1 score, mean intersection over union (MIoU), and specificity. MIoU (Equation 5) represents the average ratio of pixels overlapping between the predicted and true results to all pixels in the predicted and true results, and is often used to measure the similarity between the predicted and correct results. On the other hand, specificity (Equation 6) quantifies the ratio of true negative samples to all actual negative samples. This indicator is often used to analyze the misdiagnosis rate of deep learning networks. The higher the specificity value, the lower the misdiagnosis rate of the model.

[0072]

[0073] Specificity = TN / (TN + FP + e-7) (Equation 6)

[0074] As above, in the above equation, TP is a true positive pixel, TN is a true negative pixel, FP is a false positive pixel, and FN is a false negative pixel.

[0075] Step 7. Evaluation of caries diagnostic model

[0076] The study re-selected 30 intraoral images for an overall evaluation of the caries diagnosis model. This part of the data set includes 15 maxillary images and 15 mandibular images. Three dentists with more than ten years of clinical experience first performed diagnostic analysis on these images, of which 10 images were diagnosed as healthy images, and the remaining 20 images were diagnosed as caries images and completed further detection and numbering. Subsequently, the 30 intraoral images were input into the caries diagnosis model for diagnosis, and the diagnosis results were compared with the dentists' diagnosis results to evaluate the predictive performance of the caries diagnosis model.

[0077] This solution collects and annotates a large amount of intraoral image data, which are used for training and testing different classification, object detection and segmentation networks, and then selects the best performing network in each task for the construction of the caries diagnosis model. Finally, the constructed diagnostic model is run and analyzed as a whole, thus achieving a complete diagnosis of caries in intraoral images.

[0078] Experimental Example 1: Performance of different classification networks in the task of screening images containing caries

[0079] In this experiment, different classification networks (such as ResNet34, ResNet50 and ResNet101 networks) were first used to screen caries images, and the results are shown in Table 2.

[0080] Table 2 Performance of different classification networks in the task of screening caries images

[0081]

[0082]

[0083] Experimental data show that among the three selected base networks, ResNet 34 has the highest classification performance. Specifically, the classification accuracy of the network in mandibular intraoral images reached 93.1%, and the classification accuracy in maxillary intraoral images reached 92.1%. In addition, ResNet 34 also performed well in precision, recall, and F1 scores, which were 93.8% vs 93.2% (mandibular vs maxillary), 93.3% vs 92.1%, and 93.1% vs 92.1%, respectively. Compared with ResNet 101, the accuracy of ResNet34 increased by 10.3% and 13.1%, respectively.

[0084] Experimental Example 2: Performance of different object detection networks in the caries detection task

[0085] Subsequently, this scheme compares the performance of three networks based on the YOLO-v5 structure (such as YOLO v5s, YOLO v5m, and YOLO v5x) in caries detection. The specific results can be seen in Tables 3 and Figure 3-4 .

[0086] Table 3 Performance of different object detection networks in caries detection tasks

[0087]

[0088] Figure 3 The mAP@50 curves of the three when predicting all categories are described. As shown in the figure, YOLO v5s shows obvious advantages in caries detection, especially in "detecting slight enamel changes in early caries", with mAP@50 values ​​of 91.5% and 89.7% in mandibular and maxillary intraoral images, respectively.

[0089] However, the network has not yet achieved convincing results in detecting advanced caries, with mAP@50 values ​​of 68.2% and 77.4% for the mandibular and maxillary, respectively. Ideally, target detection should not have any redundant or leaky areas, but in actual conditions, achieving this ideal state is often quite challenging.

[0090] Figure 4 It shows that the YOLO v5s network trades off between precision and recall to select the appropriate threshold. By analyzing the PR curve, we came to the same conclusion as the mAP@50 result, that is, YOLO v5s has a high accuracy in identifying early caries and can effectively assist dentists in early caries diagnosis.

[0091] Experimental Example 3: Performance of different segmentation networks in the tooth segmentation and numbering task

[0092] This scheme compares the performance of three networks, U-Net, U-Net++ and MT-Net, in caries segmentation and numbering. The specific results can be seen in Table 4 and Figure 5 .

[0093] Table 4 Performance of different segmentation networks in the tooth segmentation and numbering task

[0094]

[0095] Experimental data show that compared with other segmentation networks, MT-Net based on the semi-supervised framework performs significantly better in segmentation tasks. This conclusion is particularly evident in the MIoU and F1 scores. In the mandibular and maxillary intraoral images, MT-Net's MIoU and F1 scores are 95.8% vs 95.4% and 97.8% vs 97.6%, respectively. For details, see Table 4. In addition, in order to confirm that the segmentation network can accurately and comprehensively segment the tooth area and minimize segmentation errors or omissions, it is usually necessary to consider sensitivity and specificity indicators comprehensively. Figure 3 The specific details of the sensitivity and specificity curves of the three networks are presented, and the results show that MT-Net has better sensitivity and specificity than the other two models, reducing errors and omissions in the tooth segmentation and numbering tasks in intraoral images. The improved performance of the network can be attributed to its ability to learn from a large number of unlabeled intraoral photos, thereby improving its feature capture ability.

[0096] Experimental Example 4: Diagnostic Performance of Caries Diagnosis Model

[0097] 30 intraoral images were selected for diagnostic evaluation of the diagnostic model (including 15 maxillary images and 15 mandibular images). The diagnostic performance of the caries diagnosis model of this scheme was evaluated by comparing the model diagnosis results with the dentist's diagnosis results. The following steps were included:

[0098] First, three dentists with more than ten years of clinical experience conducted diagnostic analysis on these images with reference to the above “(II) Image processing and labeling: including dentists’ classification, labeling, segmentation and numbering of images”. Among them, 10 images were diagnosed as healthy images, and the remaining 20 images were diagnosed as caries images and completed further testing and numbering.

[0099] Subsequently, 30 intraoral images were input into the caries diagnosis model of this scheme for diagnosis, and the diagnosis results were compared with those of dentists to evaluate the predictive performance of the caries diagnosis model.

[0100] The results show that the dental caries diagnosis model of this scheme can accurately classify all intraoral images, of which 10 intraoral images were diagnosed as healthy images and 20 intraoral images were diagnosed as images with caries, which is completely consistent with the diagnosis results of dentists. Subsequently, the 20 intraoral images with caries were further diagnosed in the diagnosis model. The diagnosis process and results of some images are detailed in Figure 6 , the comparative analysis found that 16 of the images were exactly the same as those of the dentists in terms of caries detection, classification and numbering (see Figure 7 ), the accuracy of the model diagnosis is completely correct is 86.7% (calculated according to the above equation). Through careful analysis of the remaining 4 intraoral images where the model detection results are not completely consistent with the doctor's annotation results, the inventors analyzed that the reason is that there may be errors in the segmentation and numbering of the dentist, resulting in the model detection results and the doctor's annotation results are not completely consistent, and the actual detection accuracy of the model detection results should be higher, close to 95%.

[0101] This solution collects and annotates a large amount of intraoral image data, which are used for training and testing of different classification, target detection and segmentation networks, and then selects the best performing network in each task for the construction of the caries diagnosis model. Finally, the constructed diagnostic model is run and analyzed as a whole, and by segmenting and numbering the affected teeth, a complete diagnosis of caries in the intraoral images is achieved, with a diagnostic accuracy of up to 86.7%, which effectively realizes the rapid diagnosis of caries in intraoral images, facilitates the use of semi-supervised structures to reduce the annotation burden of dentists, and improves the richness of research data, thereby improving diagnostic accuracy and efficiency.

[0102] The above is only an embodiment of the present invention, and the common knowledge such as the known specific technical solutions and / or characteristics in the solution is not described in detail here. It should be pointed out that for those skilled in the art, without departing from the technical solution of the present invention, several modifications and improvements can be made, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A color image early caries diagnosis model based on deep learning, characterized by: The method includes an image collection stage and a model building stage. The image collection stage includes collecting intraoral occlusal surface images of the patient; the model building stage includes training a deep learning network and running a caries diagnosis model.

2. The color image early caries diagnosis model based on deep learning according to claim 1, characterized in that: The training of the deep learning network includes the following steps: S1. Screening of images containing caries: training an image classification network to classify intraoral images into two groups: healthy intraoral images and intraoral images containing caries; S2. Caries detection: Train the object detection network to annotate caries in intraoral images containing caries and classify them into two categories: early stage caries and mid- to late stage caries. S3. Teeth segmentation and numbering: Train the image segmentation network to annotate and segment the teeth in the images containing caries and annotate the caries by number.

3. The color image early caries diagnosis model based on deep learning according to claim 2, characterized in that: In S1, the training of the image classification network is as follows: ResNet34, ResNet50 and ResNet101 networks are selected to screen the upper and lower mandibular intraoral images for caries images, and the various indicators of the three in the test set are compared. The ResNet34 network with the best effect is selected to screen and group the upper and lower mandibular intraoral images for caries images.

4. The color image early caries diagnosis model based on deep learning according to claim 2, characterized in that: In S2, the training of the target detection network is as follows: YOLOv5s, YOLOv5m and YOLOv5x networks are selected to detect and classify caries in images containing caries in the mouth, and the various indicators of the three in the test set are compared. The YOLOv5s network with better results is selected for automatic identification and positioning of early caries and mid-to-late stage caries.

5. The color image early caries diagnosis model based on deep learning according to claim 2, characterized in that: In S2, the training of the image segmentation network is as follows: U-Net, U-Net++ and MT-Net networks are selected to segment and number caries in caries-containing intraoral images, and the various indicators of the three in the test set are compared, and the MT-Net network with better effect is selected to segment and number the diseased teeth in caries-containing intraoral images.

6. The color image early caries diagnosis model based on deep learning according to claim 1, characterized in that: It also includes the evaluation stage of the deep learning model, which uses the classification and labeling results of dentists as a reference to evaluate the diagnostic performance of the caries diagnosis model of this scheme.

7. The color image early caries diagnosis model based on deep learning according to claim 6, characterized in that: The method for obtaining the classification and labeling results of the dentist includes the dentist classifying, labeling and segmenting the images, and includes the following steps: Step 1: Collect the patient's intraoral occlusal surface image; Step 2: Classification of teeth in the image: The dentist classifies each tooth in the intraoral image into healthy teeth, teeth with early caries changes, and teeth with mid- to late-stage caries changes according to the International Caries Detection and Assessment System (ICDAS); Step 3: Segmentation and numbering of teeth in the image: The dentist segments each tooth in the intraoral image except the third molar in turn, and numbers the segmented teeth according to the International Dental Federation numbering system (FDI).

8. The color image early caries diagnosis model based on deep learning according to any one of claims 3 to 5, characterized in that: The overall operation steps of the caries diagnosis model are as follows: first, the intraoral images are classified using the ResNet 34 network. Healthy intraoral images are diagnosed as "healthy" after classification, and intraoral images with caries are further detected and segmented using the YOLO v5s and MT-Net structures. The model extracts the pixel coordinates of the caries detection results and the pixel coordinates of each tooth that has completed the segmentation and numbering, and then compares them in turn, calculates the pixel overlap rate of the two, and extracts the numbering result of the tooth with the highest overlap rate with the detected caries, thereby diagnosing the severity and tooth position numbering of all diseased teeth at the same time.

9. The color image early caries diagnosis model based on deep learning according to claim 8, characterized in that: The calculation formula of the pixel overlap rate is: Among them, TP is a true positive pixel, FP is a false positive pixel, and FN is a false negative pixel.

Images