Semi-supervised small-scale tooth disease automatic detection method and device based on feature fusion and pseudo label mining

Abstract

The invention discloses a semi-supervised small-scale automatic tooth disease detection method and device based on feature fusion and pseudo label mining, and the method comprises the steps: constructing a multi-modal feature pyramid through employing an EIA module, combining a panoramic X-ray film and an entropy graph mode, and enabling a model to extract small target foreground features more accurately under the complex background of the panoramic X-ray film; and meanwhile, designing a dynamic self-adaptive threshold strategy, screening false labels with low confidence and high quality, and supplementing supervision information. According to the method, under the condition that only a small amount of labeled data and a large amount of unlabeled data are needed, the detection precision of low-contrast and small-scale tooth diseases in the panoramic X-ray film is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of dental detection, and particularly relates to a semi-supervised small-scale dental disease automatic detection method and device based on feature fusion and pseudo-label mining. Background Art

[0002] Tooth diseases are one of the most common oral diseases in humans, affecting the health of many people. Dental panoramic X-ray films, also known as panoramic X-ray photographs, are important diagnostic tools widely used in dental practice. With the rapid development of deep learning, deep learning-based automatic detection can automatically detect tooth diseases in panoramic X-ray films and improve the accuracy and efficiency of doctor diagnosis.

[0003] With the rapid development of deep learning, convolutional neural network-based automatic tooth disease detection methods have made many breakthroughs. However, due to the complexity of the data distribution of dental oral X-ray films, large scale differences between different lesions, and the existence of a large number of small-scale diseases, the existing technologies have unsatisfactory effects on the detection of small-scale tooth diseases. In addition, existing semi-supervised medical detection methods usually filter pseudo-labels generated by the teacher network by setting a relatively high classification score threshold. However, small-scale tooth diseases usually have low confidence, resulting in insufficient supervision of these diseases by the model and the inability to utilize low-confidence but high-quality labels. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a semi-supervised small-scale dental disease automatic detection method and device based on feature fusion and pseudo-label mining, which can adapt to the complex background and data distribution characteristics of dental panoramic X-ray films and achieve more accurate detection of small-scale tooth diseases.

[0005] To achieve the above object, the technical solution provided by the present invention is as follows:

[0006] A semi-supervised small-scale dental disease automatic detection method based on feature fusion and pseudo-label mining, comprising the following steps:

[0007] Step 1: Collect clinical oral panoramic X-ray films, label some of the collected oral panoramic X-ray films, and make a tooth disease detection data set;

[0008] Step 2: Construct an improved model - MLCM based on the mean teacher framework. This model includes a teacher model and a student model with the same network structure, both using ResNet-50+FPN as the backbone network, and fusing the entropy map and panoramic X-ray film features through the EIA module to enhance the attention to small-scale tooth diseases;

[0009] Step 3: Train the improved MLCM model with the labeled data and unlabeled data;

[0010] Step 4: Input the panoramic dental X-ray to be detected into the trained improved best MLCM model for detection.

[0011] In one embodiment of the present invention, the step 1 specifically includes:

[0012] Collect a large number of clinical panoramic dental X-ray images. All images are saved in JPG format to ensure the consistency of data format and facilitate subsequent processing work. Next, invite professional dentists with rich clinical experience to manually label a part of the X-rays, mainly labeling four common dental diseases such as high-density shadows, low-density shadows, apical shadows, and impacted teeth. The labeling process will strictly follow the format of COCO (Common Objects in Context) object detection, including accurately bounding box labeling of the regions of interest in the image and attaching corresponding class labels, and constructing a data set from all the labeled images and unlabeled images.

[0013] In one embodiment of the present invention, the step 2 specifically includes:

[0014] Step 2.1: Initially, use the ResNet-50 network to extract the preliminary features of the panoramic dental X-ray image and the calculated entropy map. The image size is 224*224*1. This network consists of multiple convolutional layers, pooling layers, fully connected layers, and residual blocks, and outputs feature maps of sizes 56, 28, 14, and 7; and on this basis, construct an FPN feature pyramid, align the channels through 1×1 convolution and perform 2-fold upsampling from top to bottom for fusion to generate {P3:28×28, P4:14×14, P5:7×7}; at the same time, perform 3×3 convolution with a stride of 2 on C5 for downsampling to obtain P6:3×3, and then perform the same operation on P6 to generate P7:1×1, finally forming a {P3-P7} multi-scale feature pyramid;

[0015] Step 2.2: Since alignment errors may occur between feature maps during recursive downsampling in FPN, an EIA module is designed based on FPN. This module passes the FPN feature pyramid of the entropy map in step 2.1 through global averaging and a convolutional kernel of size 1*1, stride 1, and padding 0, followed by the ReLu function and then another convolutional kernel with the same settings, and after the Sigmoid function, performs convolutional operation with the feature pyramid of the panoramic X-ray image to obtain the final result P X . The panoramic X-ray image is also processed in the same way to obtain P + . After that, based on P X and P + generate corresponding predictions.

[0016] Step 2.3: The teacher model uses the same framework to generate predictions to supervise the results of the student model. The overall loss is as follows:

[0017]

[0018] where λ1 and λ2 are the weights for balancing the loss, L s is the student supervision loss, and is the unsupervised loss.

[0019] In one embodiment of the present invention, step 3 specifically includes:

[0020] Step 3.1: For the labeled data, only weak data augmentation is performed and then it is fed into the student model, and the supervised loss is calculated using the true annotation information to obtain L s ;

[0021] Step 3.2: For the unlabeled data, first weak data augmentation and strong data augmentation are respectively applied to the original image to generate two different augmented views; meanwhile, the corresponding entropy map modality is generated by calculating the local Shannon entropy to highlight the regions containing lesion information in the image;

[0022] Step 3.3: The augmented views and the entropy map modality are used to construct a multi-modal feature pyramid, and the features of each modality are weighted and fused through the EIA module to obtain a more discriminative feature representation.

[0023] Step 3.4: The processed unlabeled data is respectively fed into the teacher model and the student model for prediction, and the low-confidence pseudo-label mining strategy is adopted to mine high-quality low-confidence pseudo-labels to provide additional supervision signals for the student model, obtaining the losses and where the parameters of the teacher model are updated by exponential moving average (EMA).

[0024] In one embodiment of the present invention, step 4 specifically includes: removing redundant modules (such as the EIA module) in the training stage, retaining the lightweight student model, inputting the panoramic dental X-ray film to be detected into the improved MLCM best student model trained in step 3, and outputting the detection result image. The disease detection result image includes rectangular box annotations for the disease location and displays the disease name information corresponding to the rectangular box.

[0025] The second aspect of the present invention relates to a semi-supervised small-scale dental disease automatic detection device based on feature fusion and pseudo-label mining, which is characterized by including a memory and one or more processors. Executable code is stored in the memory, and when the one or more processors execute the executable code, it is used to implement a semi-supervised small-scale dental disease automatic detection method according to any one of claims 1-5.

[0026] The advantages of the present invention are as follows: The present invention improves the ResNet50 network and semi-supervised methods, can more effectively extract small-scale dental disease features, uses EIA to fuse the features of panoramic X-ray films and corresponding entropy maps, which is more conducive to obtaining more accurate position feature information, improves the detection accuracy of the model, the adaptive threshold strategy alleviates the missed detection of small-scale targets by fixed thresholds, and the classification score improvement Δq effectively screens low-confidence positive samples. Brief Description of the Drawings

[0027] Figure 1 It is the core architecture diagram of the MLCM provided by the present invention.

[0028] Figure 2 It is the schematic diagram of the overall training process of the present invention.

[0029] Figure 3 It is the core architecture diagram of the EIA module provided by the present invention.

[0030] Figure 4 It is the device schematic diagram of the present invention. Detailed Description of the Invention

[0031] The technical solution of the present invention will be further described below with reference to the drawings.

[0032] Embodiment 1

[0033] This embodiment provides a semi-supervised small-scale dental disease automatic detection method based on feature fusion and pseudo-label mining, as Figure 1 shown, including the following steps:

[0034] Step 1: Collect a large number of clinical dental panoramic X-ray images. All images are saved in JPEG format to ensure the consistency of data format and facilitate subsequent processing work. Next, invite professional dentists with rich clinical experience to manually label some X-ray films, mainly labeling four common dental diseases, namely high-density shadows, low-density shadows, apical shadows, and impacted teeth. The labeling process will strictly follow the format of COCO (Common Objects in Context) object detection, including accurately labeling the bounding boxes of the regions of interest in the images and attaching corresponding class labels.

[0035] Step 2: Construct an improved model - MLCM based on the mean teacher framework. This model consists of a teacher model and a student model with the same network structure, both using ResNet - 50 + FPN as the backbone network, and enhancing the attention to small - scale dental diseases by fusing the entropy map with panoramic X - ray features; specifically including the following steps:

[0036] The feature extraction module consists of a ResNet50 network, an FPN feature pyramid + EIA module, which extracts small - scale dental disease features. Dental diseases in oral panoramic X - rays have the characteristics of complex data distribution, similar lesion features to the background, and small scale. The entropy map can effectively highlight the information - rich regions in the image by measuring the complexity or information content of local regions of the image. The calculation method is as follows:

[0037] H=-∑ (i) P i log(p i ) (2)

[0038] where pi represents the normalized probability that the pixel value i appears in a certain local window.

[0039] By fusing the information in the entropy map, it can help the model enhance the attention to lesions, especially when dealing with low - contrast and small - scale lesions, optimize the localization and boundary recognition of the target, thereby improving the overall detection accuracy.

[0040] Therefore, use the ResNet - 50 network to perform preliminary feature extraction on the dental panoramic X - ray and its corresponding entropy map, and the input image size is 224×224×1. This network consists of multiple convolutional layers, pooling layers, fully - connected layers, and residual blocks, and finally outputs feature maps with sizes of 56×56, 28×28, 14×14, and 7×7 respectively. On this basis, an FPN feature pyramid is constructed: First, align the channels of each layer of features through 1×1 convolution, and then upsample by a factor of 2 from top to bottom and fuse to generate P3(28×28), P4(14×14), and P5(7×7); at the same time, perform downsampling on the C5 layer using a 3×3 convolution with a stride of 2 to obtain P6(3×3), and generate P7(1×1) from P6 in the same way, finally forming a multi - scale feature pyramid containing P3 to P7.

[0041] Since the FPN may cause alignment errors between feature maps during recursive downsampling, an EIA module is designed based on the FPN. Based on this, this paper proposes a multi - modal feature pyramid, and effectively fuses the feature information of the panoramic X - ray and the entropy map through the EIA module to construct a feature representation with richer information, thereby improving the quality of pseudo - labels. The detailed structure of the EIA is as Figure 3 shown. Given an unlabeled image First, feature pyramids are extracted from the enhanced view X of the input image + and the entropy map modality X E respectively and P + and P E are input into the EIA module to construct a multi-modal feature pyramid. The purpose of the EIA module is to obtain the spatial attention weight SA through P E to guide P + to focus on the feature parts that contain lesion information in the entropy map. This module performs global average pooling on the FPN feature pyramid generated from the entropy map, then uses a convolutional kernel with a size of 1×1, a stride of 1, no padding, and the ReLU activation function for preliminary feature transformation, and then further processes it with a convolutional kernel and the Sigmoid activation function with the same configuration, and finally obtains the feature representation P X of the entropy map. The feature pyramid of the panoramic X-ray image is also processed as above to generate the feature representation P + . The calculation process of the EIA module can be expressed by the following formula:

[0042]

[0043] where σ(·), δ(·), * represent the Sigmoid function, the ReLU function, and the convolutional operation respectively. W and b represent the weight parameters and bias values of the convolutional operation

[0044] After calculation, the SA spatial attention weight with a value in the range of 0 to 1 is obtained. The multi-modal feature pyramid P x is the weighted sum of P + and P E :

[0045]

[0046] After that, based on P x and P + the corresponding prediction results for unsupervised training are generated

[0047] Step 3: The labeled image is weakly data-augmented for supervised training to obtain the supervised loss L s , and the formula is as follows:

[0048] L s = L cls + L reg (5)

[0049] where the classification loss L cls adopts the cross-entropy loss, and its formula is:

[0050]

[0051] Regression loss L reg Use Smooth L1 loss, and its formula is:

[0052]

[0053] where x represents the difference between the predicted bounding box and the ground truth bounding box.

[0054] Meanwhile, for unlabeled images, strong and weak data augmentations and local Shannon entropy are used to generate entropy maps, thereby constructing a multi-modal feature pyramid; the processed unlabeled data are respectively fed into the teacher model and the student model for prediction, and the teacher model additionally adopts a low-confidence pseudo-label mining strategy to obtain pseudo-labels. The low-confidence pseudo-label mining strategy mainly solves the problem that in most unsupervised training, a very high classification score threshold τ is usually set for the predicted result boxes h to filter pseudo-labels from the predicted boxes output by the model. However, this setting largely ignores small-scale objects. In object detection, due to their small size and the difficulty of accurately extracting features, small-scale objects are very unlikely to have a high classification score, resulting in many predicted boxes of small-scale objects being misjudged as the background. Therefore, a low-confidence pseudo-label mining strategy is proposed.

[0055] For the predicted boxes based on the multi-modal feature pyramid, first, it is determined that the classification score s is within τ l <= s < τ h are low-confidence candidate boxes. For each low-confidence candidate box, according to the overlap degree (Intersection over Union, IoU) >= 0.7, a group of proposal boxes and their corresponding classification scores that match it in the predicted boxes based on the multi-modal feature pyramid are selected, denoted as where N and M are the numbers of the two groups of proposal boxes respectively. Then, according to the two groups of candidate boxes and to calculate the classification score improvement Δq:

[0056]

[0057] A higher Δq indicates that in the multi-modal feature pyramid, the classification scores of the proposal boxes matching the low-confidence candidate box corresponding to Δq have a significant improvement. In other words, this means that this low-confidence candidate box is relatively difficult to be detected in the enhanced view feature pyramid, but is relatively easy to be detected in the more informative multi-modal feature pyramid. Therefore, this low-confidence candidate box can be regarded as a high-quality pseudo-label.

[0058] In addition, through the classification score thresholds τ l and τ hto select low-confidence candidate bounding boxes. In previous studies, τ h τ was usually set to a fixed value, such as 0.9. If the classification score of a predicted bounding box is greater than τ h , then this bounding box is regarded as a pseudo-label. However, this strategy of fixed threshold is not friendly to small-scale diseases because it is usually difficult for small-scale diseases to obtain high classification scores. Although a low-confidence pseudo-label mining strategy has been proposed to mine those predicted bounding boxes with classification scores lower than hτ, there may still be omissions. To address the problem of missed detections caused by low classification confidence for small-scale diseases, an adaptive threshold and classification score improvement amplitude joint screening mechanism is proposed:

[0059] For each class c, there are corresponding predicted bounding boxes The threshold for this class c is dynamically adjusted according to the classification scores of the predicted bounding boxes of this class, and the threshold for this class c is defined as

[0060]

[0061] τ l = τ h - 0.2(10)

[0062] where N bg is the number of background classes among all predicted bounding boxes, used to control the intensity of the adaptive threshold, γ is set to 0.05, denotes rounding down (e.g., 0.94 will be set to 0.9). and represent the classification scores of the i-th and j-th foreground predicted bounding boxes and background predicted bounding boxes. At the initial stage of training, since the network parameters have not been fully learned, most of the predictions of the teacher network are incorrect, mainly manifested as predictions of background classes rather than foreground classes. Therefore, the adaptive threshold will output a relatively small value. This strategy dynamically adjusts the threshold according to the training stage: the threshold is lower in the initial stage to cover more potential positive samples, and gradually tightened in the later stage to filter out noise and thus filter out higher-quality pseudo-labels.

[0063] Subsequently, the teacher predictions (including the mined pseudo-labels) are compared with the student model predictions, and the unsupervised loss is calculated, including the original-size unsupervised loss multi-modal learning unsupervised loss They can be expressed as:

[0064]

[0065] Finally, the model is jointly optimized by combining the supervised and unsupervised losses, and the parameters of the teacher model are updated by exponential moving average (EMA). The model is iteratively learned a total of 40K times.

[0066] Step 4: Remove redundant modules (such as the EIA module) in the training stage, retain the lightweight student model, and input the panoramic dental X-ray to be detected into the improved best student model of MLCM trained in Step 3 to output the detection result image. The disease detection result image includes rectangular box annotations for the disease location and displays the disease name information corresponding to the rectangular box.

[0067] The present invention mainly aims at the problem of automatic detection of small-scale dental diseases, and proposes an automatic detection of small-scale dental diseases based on a multi-modal feature pyramid and a low-confidence pseudo-label mining strategy. The model utilizes the highly discriminative disease information provided by the entropy map, combines the feature information of the enhanced views, constructs a multi-modal feature pyramid through the EIA module, extracts multi-modal disease features from different scales, and enhances the model's perception ability of small-scale lesions. In addition, based on the prediction results of the multi-modal feature pyramid, the improvement amplitude of the classification score of the enhanced view feature pyramid is measured, so as to mine high-quality low-confidence pseudo-labels, strengthen the supervision of the model for small-scale diseases, and improve the detection accuracy.

[0068] Embodiment 2

[0069] Refer to Figure 2 , this embodiment provides a semi-supervised automatic detection device for small-scale dental diseases based on feature fusion and pseudo-label mining, including a memory and one or more processors. Executable code is stored in the memory. When the one or more processors execute the executable code, it is used to implement the semi-supervised automatic detection method for small-scale dental diseases based on feature fusion and pseudo-label mining in Embodiment 1.

[0070] The present invention has been illustrated by the above examples. However, it is obvious that these examples only serve to explain and illustrate, rather than limit the present invention within the scope of these examples. Researchers in the technical field of the present invention can make non-substantive changes in form and content without departing from the scope of the present invention's essential protection according to the above steps. Therefore, the present invention is not limited to the above specific forms and details.

Claims

1. A semi - supervised small - scale dental disease automatic detection method based on feature fusion and pseudo - label mining, comprising the following steps: Step 1: Collect clinical oral panoramic X - ray images, label some of the collected oral panoramic X - ray images, and make a dental disease detection dataset; Step 2: Construct an improved MLCM model based on the mean teacher framework. This model includes a teacher model and a student model with the same network structure. Both use ResNet - 50+FPN as the backbone network, and fuse the entropy map and panoramic X - ray image features through the EIA module to enhance the attention to small - scale dental diseases; Step 3: Train the improved MLCM model with the labeled data and unlabeled data; Step 4: Input the dental panoramic X - ray image to be detected into the trained best improved MLCM model for detection.

2. According to the semi - supervised small - scale dental disease automatic detection method based on feature fusion and pseudo - label mining described in claim 1, the specific content of step 1 includes: Collect a large number of clinical oral panoramic X - ray images, and save all images in JPG format; Manually label a part of the X - ray images, including labeling four common dental diseases such as high - density shadows, low - density shadows, apical shadows, and impacted teeth. The labeling process follows the format of COCO object detection, including accurately bounding - box labeling the regions of interest in the image and attaching corresponding class labels, and constructing a dataset with all the labeled images and unlabeled images.

3. According to the semi - supervised small - scale dental disease automatic detection method based on feature fusion and pseudo - label mining described in claim 1, the specific content of step 2 includes: Step 2.1: Initially, use the ResNet - 50 network to extract the preliminary features of the dental panoramic X - ray images and the calculated entropy map. The size of both images is 224*224*1. This ResNet - 50 network consists of multiple convolutional layers, pooling layers, fully - connected layers, and residual blocks, and outputs feature maps of sizes 56, 28, 14, and 7. On this basis, construct an FPN feature pyramid, align channels through 1×1 convolution and fuse them by top - down 2 - fold upsampling to generate {P3:28×28, P4:14×14, P5:7×7}; at the same time, perform 3×3 convolution with a stride of 2 on C5 for downsampling to get P6:3×3, and perform the same operation on P6 to generate P7:1×1, finally forming a multi - scale feature pyramid of {P3 - P7}; Step 2.2: Since the Feature Pyramid Network (FPN) may cause alignment errors between feature maps during recursive downsampling, an Error Invariant Alignment (EIA) module is designed based on the FPN. The EIA module subjects the FPN feature pyramid of the entropy map in Step 2.1 to global averaging and then convolution with a convolutional kernel of size 1×1, stride 1, and padding 0, followed by the ReLu function, another convolution with the same settings, the Sigmoid function, and finally convolution with the feature pyramid of the panoramic X-ray image to obtain the final result P X ; The panoramic X-ray image is also processed in the same way to obtain P + ; After that, based on P X and P + corresponding predictions are generated; Step 2.3: The teacher model uses the same framework as the student model to generate predictions, and the teacher model supervises the results of the student model. The overall loss is shown in formula (1): Among them, λ1 and λ2 are the weights for balancing the losses, L s is the student supervision loss, and is the unsupervised loss.

4. According to the semi - supervised small - scale dental disease automatic detection method based on feature fusion and pseudo - label mining described in claim 1, the specific content of step 3 includes: Step 3.1: For the labeled data, only perform weak data augmentation and then send it into the student model, and calculate the supervised loss using the true labeling information to obtain L s ; Step 3.2: For unlabeled data, first apply weak data augmentation and strong data augmentation to the original image respectively to generate two different augmented views; at the same time, calculate the corresponding entropy map modality through local Shannon entropy to highlight the regions containing lesion information in the image; Step 3.3: Construct a multi-modal feature pyramid using the enhanced view and entropy map modalities, and perform weighted fusion on the features of each modality through the EIA module to obtain a more discriminative feature representation; Step 3.4: The processed unlabeled data are fed into the teacher model and the student model for prediction respectively, and a low-confidence pseudo-label mining strategy is adopted to mine high-quality low-confidence pseudo-labels to provide additional supervision signals for the student model, obtaining the loss and where the parameters of the teacher model are updated by exponential moving average (EMA).

5. For a semi-supervised small-scale dental disease automated detection method based on feature fusion and pseudo-label mining according to claim 1, the specific steps of step 4 are as follows: Remove redundant modules in the training phase, retain the lightweight student model, and input the panoramic dental X-ray film to be detected into the improved MLCM best student model trained in step 3 to output a detection result image. The disease detection result image includes rectangular box annotations for the disease location and displays the disease name information corresponding to the rectangular box.

6. A semi-supervised small-scale dental disease automated detection device based on feature fusion and pseudo-label mining, characterized in that, Comprising a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, it is used to implement a semi-supervised small-scale dental disease automated detection method according to any one of claims 1-5.

Images