Tooth crack detection and segmentation method based on improved Mask R-CNN
Through the improved Mask R-CNN network, combined with data augmentation and deep supervision technology, the challenging problems of dental crack diagnosis are solved, achieving higher accuracy detection and segmentation, and reducing diagnostic complexity.
Patent Information
- Application Number
- CN202411495494.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-21
AI Technical Summary
Diagnosis of dental cracks is challenging, and existing methods rely on the experience of dentists and a good clinical environment, and are prone to misjudgment due to visual fatigue.
Using an improved Mask R-CNN network, the model is trained to detect and segment tooth cracks through data augmentation and deep supervision techniques. This method includes data acquisition, label production, network training and model testing, and uses ResNeXt-50 structure and crack feature enhancement module to improve model expression capabilities.
It improves the detection and segmentation accuracy of tooth cracks, reduces the dependence on dentist's experience, and reduces the complexity and inconvenience of diagnosis.
Smart Images
Figure CN119494817A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of dental diagnosis and medical technology, and in particular to a tooth crack detection and segmentation method based on an improved Mask R-CNN. Background Art
[0002] Currently, problems related to tooth cracks are the third leading cause of tooth loss after caries and periodontal disease. The presence of tooth cracks brings a lot of inconvenience and pain to people. Discomfort in chewing is the most common symptom of cracked teeth, and they are also sensitive to temperature changes, especially cold. On the other hand, cracks are a pathway for bacteria to invade the dental pulp. And the bacteria colonize deeper and deeper in the cracks, and the inflammatory response of the dental pulp becomes more and more serious. Therefore, early diagnosis and intervention are necessary to prevent further tooth damage. However, in clinical trials, the diagnosis of cracked teeth is very challenging due to the complexity of symptoms and the difficulty of locating cracks. Currently commonly used detection methods include occlusal testing, periodontal probing, staining, cold stimulation, etc.
[0003] Based on the above methods, the identification and judgment of tooth cracks by dentists alone requires a certain level of experience from dentists, and due to inevitable visual fatigue, misjudgment of dental symptoms may occur. In addition, many medical imaging modes require a good clinical environment and medical equipment, which also brings many inconveniences to the inspection of tooth cracks.
[0004] To this end, we proposed a tooth crack detection and segmentation method based on an improved Mask R-CNN to solve the problem. Summary of the invention
[0005] The purpose of the present invention is to provide a tooth crack detection and segmentation method based on an improved Mask R-CNN, which solves the problems in the above-mentioned background technology.
[0006] To achieve the above object, the present invention provides the following technical solution: a tooth crack detection and segmentation method based on an improved Mask R-CNN, comprising the following steps:
[0007] S1. Collecting Datasets
[0008] A tooth model including cracks is made, and multi-directional images of the tooth model are collected using an electron microscope to obtain an original data set;
[0009] S2. Expanding the dataset
[0010] Use data augmentation methods to process the original data set to obtain an expanded data set;
[0011] S3, labeling and dividing the data set
[0012] The images with cracks in the original data set and the expanded data set were manually annotated using the annotation software Labelme to generate corresponding image labels to obtain an annotated data set, which was then randomly divided into a training set, a validation set, and a test set with a ratio of 8:1:1.
[0013] S4. Establish and train the Mask R-CNN network structure
[0014] Use the training set in S3 to train the improved Mask R-CNN network structure, using a deep supervision method to balance the weights of multiple loss functions, and use the validation set in S3 to monitor network performance to obtain the optimal weight model;
[0015] S5. Test model
[0016] Use the trained model to detect the test set in S3, obtain the detection and segmentation accuracy of the model, and obtain the predicted image.
[0017] Preferably, the data enhancement methods in S2 include up-down flipping, left-right flipping, random brightness, and uniformly distributed noise.
[0018] Preferably, the image labels in S3 are divided into longitudinal cracks, transverse cracks, oblique cracks, bifurcated cracks and occlusal cracks according to the relative positions of the cracks on the teeth.
[0019] Preferably, the process of using the training set in S3 to train the improved Mask R-CNN network structure in S4 includes:
[0020] S41, obtaining a labeled image in the training set of S3, and inputting the labeled image into the backbone network, the backbone network extracts features from the labeled image to obtain a feature map;
[0021] S42, using RPN to generate anchor frames, and performing binary classification and bounding box regression on the feature map in S41 according to the RPN network, and selecting a predetermined number of anchor frames as suggestion frames;
[0022] S43, using ROI Align to adjust all the suggestion boxes in S42 to ROI feature maps of the same size;
[0023] S44. Use the detection head of the fully connected layer to perform binary classification and bounding box regression on the ROI feature map. The suggested box after binary classification and bounding box regression is the predicted box. Use the encoder-decoder segmentation head to generate a mask for each ROI feature map, use the mask to segment the crack contour, and mark the corresponding label on the image.
[0024] Preferably, the backbone network in S41 uses ResNeXt50 (32×4) as the basic network and adds a feature pyramid network to further extract features. In the feature pyramid network, the feature map extracted by the basic network is merged with the top-level feature map, and the 1×1 convolution is replaced with a crack feature enhancement module.
[0025] Preferably, the encoding-decoding segmentation head network structure of S44 includes an encoder and a decoder. In the encoder, the feature map undergoes two processes of 9×9 snake convolution block-downsampling-3×3 convolution block, the 9×9 snake convolution block includes a 9×9 snake convolution-batch normalization layer-ReLU activation function, and the 3×3 convolution block includes a 3×3 convolution-batch normalization-ReLU activation function. In the decoder, the feature map extracted from the encoder undergoes two processes of 3×3 convolution block-upsampling-3×3 convolution block, and finally the segmentation mask is obtained through prediction.
[0026] Preferably, the loss function in S4 is:
[0027] L=L cls +L bbox +L mask
[0028] Where L cls , L bbox and L mask Represent classification loss, regression box loss and segmentation mask loss respectively; for classification loss L cls
[0029]
[0030] Where P i and P i * Represent the predicted probability and true value of anchor i respectively, and the bounding box regression loss L bbox ,
[0031]
[0032] Where t=(t x ,t y ,t w ,t h ) are the predicted bounding box parameters, is the true bounding box parameter. For the segmentation mask loss, deep supervision is used to add auxiliary losses in multiple places in the encoder-decoder segmentation head.
[0033]
[0034] Among them, M m.ij and They represent the predicted value and true value of the jth pixel of the ith sample, respectively. k represents the height or width of the mask. In particular, when m=5, the output mask prediction value is the final value, and the final mask loss is,
[0035]
[0036] Among them, λ m is the weight corresponding to the mth loss.
[0037] The present invention provides a tooth crack detection and segmentation method based on an improved Mask R-CNN. The tooth crack detection and segmentation method based on an improved Mask R-CNN has the following beneficial effects:
[0038] (1) The present invention uses the ResNeXt-50 (32×4) structure as the backbone network to improve the expressiveness of the model while maintaining low computational complexity and avoid the exponential growth of computation and memory;
[0039] (2) The present invention uses a crack feature enhancement module (CFEM) to replace the 1×1 convolution, which can help the feature pyramid network enhance crack features in different directions in feature fusion and better capture the spatial and texture information of the crack.
[0040] (3) The present invention obtains the global information of the entire image and the local information of the crack by using an encoding-decoding segmentation head, thereby obtaining a crack mask with higher quality.
[0041] (4) The present invention uses deep supervision to balance and optimize the weights between multiple loss functions, which enables the network to gradually optimize feature representation at different levels, thereby improving segmentation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 This is a flow chart of the tooth crack detection and segmentation method based on the improved Mask R-CNN of the present invention;
[0043] Figure 2 The original image of the present invention, the effect image of cracks marked on the image by the software Lebelme, and the network prediction effect image;
[0044] Figure 3 This is a diagram of the improved Mask R-CNN network structure of the present invention;
[0045] Figure 4 It is a network flow chart of the crack feature enhancement module (CFEM) of the present invention;
[0046] Figure 5 This is a network structure diagram of the encoding-decoding segmentation head of the present invention. DETAILED DESCRIPTION
[0047] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described with reference to the accompanying drawings.
[0048] A preferred embodiment of a tooth crack detection and segmentation method based on an improved Mask R-CNN provided by the present invention is as follows: Figures 1 to 5 As shown: A tooth crack detection and segmentation method based on an improved Mask R-CNN comprises the following steps:
[0049] S1. Dataset collection: Use the temperature difference method to make a tooth model with cracks, and use an electron microscope with a magnification of ×16 to collect multi-directional images of the tooth model to obtain the original data set. The collected images can overlap but cannot be collected repeatedly;
[0050] S2, expanded data set: The cracks in the original data set are processed by left-right flipping, up-down flipping, random brightness and uniformly distributed noise data enhancement methods. The processed original data set is the expanded data set;
[0051] S3, make labels and divide the data set: Figure 2 The top and middle figures shown are sample images of the test set and the corresponding label images, respectively. The images with cracks in the original data set and the expanded data set are manually annotated using the labeling software Labelme. The annotated information includes the center coordinates of the rectangular box where the crack is located, the size of the rectangular box, and the pixel position of the crack. The annotated objects are classified according to the relative position of the crack on the tooth, including longitudinal cracks, transverse cracks, oblique cracks, bifurcation cracks, and occlusal cracks. The corresponding annotated images are generated to obtain the annotated data set, and the annotated data set is randomly divided into a training set, a validation set, and a test set in a ratio of 8:1:1.
[0052] S4, build and train the improved Mask R-CNN network structure: use the training set in S3 to train the Mask R-CNN network structure and train the optimal weight model. Figure 3 As shown in the figure, the training process of using the training set in S3 to train the improved Mask R-CNN network structure includes:
[0053] S41. Obtain a labeled image from the training set of S3 and input the labeled image into the backbone network. The backbone network extracts features from the labeled image to obtain a feature map. The backbone network uses the basic network ResNeXt50 (32×4) combined with the feature pyramid network for feature extraction. The 1×1 convolution in the feature pyramid network is replaced with the crack feature enhancement module (CFEM). The network flow of the crack feature enhancement module is as follows: Figure 4As shown. This module needs to set two hyperparameters: width threshold w and pixel threshold a. For a single two-dimensional feature map, first move the original feature map along four directions (up / down / left / right) by a distance of w pixels to generate a rolling feature map; set the last w eigenvalues of the opposite position (down / up / right / left) to zero to generate a zeroed feature map; subtract the eigenvalues of the original feature map from the corresponding position of the zeroed feature map and take the absolute value. If the value is greater than the set pixel threshold p, the pixel of the original feature map at that position is retained; if the value is less than the set pixel threshold p, the eigenvalue at that position is set to zero to obtain the subtracted feature map. Perform a logical AND operation on the subtracted feature map and the corresponding eigenvalues of the feature map in the opposite direction (i.e., up and down, left and right) to obtain the transverse crack feature map and the longitudinal crack feature map; perform a logical OR operation on the eigenvalues of the corresponding positions of the two feature maps to obtain the crack enhancement feature map. Finally, the unified batch of feature maps are used to adjust the number of channels using 1×1 convolution. The best combination of width threshold and pixel threshold is searched using Bayesian optimization algorithm.
[0054] S42, using RPN to generate anchor frames, and performing binary classification and bounding box regression on the feature map in S41 according to the RPN network, and selecting a predetermined number of anchor frames as suggestion frames;
[0055] S43, using ROI Align to adjust all the suggestion boxes in S42 to ROI feature maps of the same size;
[0056] S44, using the detection head of the fully connected layer to perform binary classification and bounding box regression on the ROI feature map, and the proposed box after binary classification and bounding box regression is the predicted box; using the encoder-decoder segmentation head to generate a mask for each ROI feature map, using the mask to segment the crack contour, and marking the corresponding label on the image; Figure 5 As shown in the figure, the encoding-decoding segmentation head network structure includes an encoder and a decoder. In the encoder, the feature map goes through two processes of 9×9 snake convolution block-downsampling-3×3 convolution block. The 9×9 snake convolution block includes 9×9 snake convolution-batch normalization layer-ReLU activation function, and the 3×3 convolution block includes 3×3 convolution-batch normalization-ReLU activation function. In the decoder, the feature map extracted from the encoder goes through two processes of 3×3 convolution block-upsampling-3×3 convolution block, and finally the segmentation mask is obtained through prediction.
[0057] S45. The loss function calculation formula of the improved Mask R-CNN model is:
[0058] L=L cls +L bbox +L mask
[0059] L cls, L bbox and L mask Represent classification loss, regression box loss and segmentation mask loss respectively; for classification loss L cls , the formula is:
[0060]
[0061] Where P i and P i * Represent the predicted probability and true value of anchor i respectively. For the bounding box regression loss L bbox , the formula is:
[0062]
[0063] Where t=(t x ,t y ,t w ,t h ) are the predicted bounding box parameters, is the true bounding box parameter. For the segmentation mask loss, deep supervision is used to add auxiliary losses in multiple places of the encoder-decoder segmentation head.
[0064]
[0065] Among them, M m.ij and Respectively represent the predicted value and true value of the jth pixel of the ith sample, and k represents the height or width of the mask. In particular, when m=5, the output mask prediction value is the final value. The final mask loss is,
[0066]
[0067] Among them, λ m is the weight corresponding to the mth loss.
[0068] S5, test model: Use the trained model to test the test set in S3, obtain the detection and segmentation accuracy of the model, and obtain the predicted image. The prediction effect is as follows Figure 2 As shown in the figure below.
[0069] The above description is only an illustrative specific implementation mode of the present invention and is not intended to limit the scope of the present invention. Any equivalent changes and modifications made by any technician in the field without departing from the concept and principle of the present invention should fall within the scope of protection of the present invention. It should also be noted that the various components of the present invention are not limited to the above-mentioned overall application. The various technical features described in the specification of the present invention can be selected one by one or multiple ones can be selected and used in combination according to actual needs. Therefore, the present invention should naturally cover other combinations and specific applications related to the invention point of this case.
Claims
1. A tooth crack detection and segmentation method based on improved Mask R-CNN, characterized by: The following steps are involved: S1. Collecting Datasets A tooth model including cracks is made, and multi-directional images of the tooth model are collected using an electron microscope to obtain an original data set; S2. Expanding the dataset Use data augmentation methods to process the original data set to obtain an expanded data set; S3, labeling and dividing the data set The images with cracks in the original data set and the expanded data set were manually annotated using the annotation software Labelme to generate corresponding image labels to obtain an annotated data set, which was then randomly divided into a training set, a validation set, and a test set with a ratio of 8:1:
1. S4. Establish and train the Mask R-CNN network structure Use the training set in S3 to train the improved Mask R-CNN network structure, using a deep supervision method to balance the weights of multiple loss functions, and use the validation set in S3 to monitor network performance to obtain the optimal weight model; S5. Test model Use the trained model to detect the test set in S3, obtain the detection and segmentation accuracy of the model, and obtain the predicted image.
2. The tooth crack detection and segmentation method based on improved Mask R-CNN according to claim 1, characterized in that: The data enhancement methods in S2 include up-down flipping, left-right flipping, random brightness, and uniformly distributed noise.
3. The tooth crack detection and segmentation method based on improved Mask R-CNN according to claim 2, characterized in that: The image labels in S3 are divided into longitudinal cracks, transverse cracks, oblique cracks, bifurcation cracks and occlusal cracks according to the relative positions of the cracks on the teeth.
4. The tooth crack detection and segmentation method based on improved Mask R-CNN according to claim 3, characterized in that: The process of using the training set in S3 to train the improved Mask R-CNN network structure in S4 includes: S41, obtaining a labeled image in the training set of S3, and inputting the labeled image into the backbone network, the backbone network extracts features from the labeled image to obtain a feature map; S42, using RPN to generate anchor frames, and performing binary classification and bounding box regression on the feature map in S41 according to the RPN network, and selecting a predetermined number of anchor frames as suggestion frames; S43, using ROI Align to adjust all the suggestion boxes in S42 to ROI feature maps of the same size; S44. Use the detection head of the fully connected layer to perform binary classification and bounding box regression on the ROI feature map. The suggested box after binary classification and bounding box regression is the predicted box. Use the encoder-decoder segmentation head to generate a mask for each ROI feature map, use the mask to segment the crack contour, and mark the corresponding label on the image.
5. The tooth crack detection and segmentation method based on improved Mask R-CNN according to claim 4, characterized in that: The backbone network in S41 uses ResNeXt50 (32×4) as the basic network and adds a feature pyramid network to further extract features. In the feature pyramid network, the feature map extracted by the basic network is integrated with the top-level feature map, and the 1×1 convolution is replaced with a crack feature enhancement module.
6. The tooth crack detection and segmentation method based on improved Mask R-CNN according to claim 5, characterized in that: The encoding-decoding segmentation head network structure of S44 includes an encoder and a decoder. In the encoder, the feature map undergoes two processes of 9×9 snake convolution block-downsampling-3×3 convolution block. The 9×9 snake convolution block includes 9×9 snake convolution-batch normalization layer-ReLU activation function, and the 3×3 convolution block includes 3×3 convolution-batch normalization-ReLU activation function. In the decoder, the feature map extracted from the encoder undergoes two processes of 3×3 convolution block-upsampling-3×3 convolution block, and finally the segmentation mask is obtained through prediction.
7. The tooth crack detection and segmentation method based on improved Mask R-CNN according to claim 6, characterized in that: The loss function in S4 is: L=L cls +L bbox +L mask Where L cls , L bbox and L mask Represent classification loss, regression box loss and segmentation mask loss respectively; for classification loss L cls Where P i and P i * Represent the predicted probability and true value of anchor i respectively, and the bounding box regression loss L bbox , Where t=(t x ,t y ,t w ,t h ) are the predicted bounding box parameters, is the true bounding box parameter. For the segmentation mask loss, deep supervision is used to add auxiliary losses in multiple places in the encoder-decoder segmentation head. Among them, M m.ij and They represent the predicted value and true value of the jth pixel of the ith sample, respectively. k represents the height or width of the mask. In particular, when m=5, the output mask prediction value is the final value, and the final mask loss is, Among them, λ m is the weight corresponding to the mth loss.