Man-machine fusion tooth segmentation method based on deep learning

Abstract

The invention discloses a human-machine fusion tooth segmentation method based on deep learning. An existing automatic segmentation algorithm has deficiencies in accuracy and stability. The method comprises the following steps: segmenting teeth on a panoramic X-ray film by using a Mask R-CNN model to obtain feature maps of different types of teeth; a segmentation result is evaluated through feature extraction, and a visualization technology is used for displaying; providing an interaction tool to support an expert to correct the segmentation result of the model; and feeding back the corrected high-quality tooth sample to the model, and carrying out adaptive iterative optimization to improve the segmentation performance of the model. According to the method, an interactive correction and feedback optimization mechanism of the anatomical features of the teeth is introduced, so that the tooth segmentation result is more accurate, and the accuracy and efficiency of tooth segmentation are remarkably improved through comprehensive evaluation and optimization tools.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and in particular relates to a human-machine fusion tooth segmentation method based on deep learning. Background Art

[0002] Computer-automated diagnosis of dental diseases has become a current trend. In this process, tooth segmentation plays a vital role, and it is one of the necessary steps in computer disease diagnosis. Through segmentation, the computer can accurately identify the position, shape, and structure of teeth, providing reliable data support for subsequent disease diagnosis and treatment. At the same time, the development of this technology will greatly improve the efficiency and accuracy of dental diagnosis and provide better medical services for patients. However, manual annotation and segmentation of teeth is a time-consuming and tedious task, especially when there are a large number of overlapping shadows of teeth in the panoramic X-ray, or the contrast of the line itself is low, the shortcomings of manual annotation become more obvious. In this case, the advantages of automatic segmentation, which is fast and convenient, show obvious advantages.

[0003] In recent years, domain experts have proposed many deep learning models for tooth segmentation in panoramic radiographs, including unsupervised and supervised methods. Unsupervised methods include threshold-based segmentation, edge detection, and graph-theoretic segmentation, while supervised methods rely on labeled data for training.

[0004] In the unsupervised category, various strategies have been developed. For example, Modi et al. proposed a region-based method for identifying regions of interest for gap valleys and tooth isolations using binary edge intensity integral curves (Modi CK, Desai NP, A simple and novel algorithm for automatic selection of ROI for dental radiograph segmentation, 2011 24th Canadian Conference on Electrical and Computer Engineering (CCECE), 2011: 504–507). Indraswari et al. adopted a three-step process involving directional image formation using DDFBT, edge enhancement for reinforcement and denoising, and segmentation using MAT with Sauvola local threshold (Indraswari R, Ariffn AZ, Navastara DA, Jawas N, Teethsegmentation on dental panoramic radiographs using decimation-freedirectional filter bank thresholding and multistage adaptive thresholding, 2015 International Conference on Information & Communication Technology and Systems (ICTS), 2015: 49–54). Hasan et al. focused on using gradient information for jaw segmentation, using a four-step approach including k-means clustering, point detection around the jaw, gradient vector flow snakes, and shape correction of the segmented region (Hasan M, Ismail WB, Hassan R, Yoshitaka A, Automatic segmentation of jawfrom panoramic dental X-ray images using GVF snakes, 2016 World AutomationCongress (WAC), 2016: 1–6). However, due to the significant changes in tooth shape and type, these strategies still cannot fundamentally solve the problem of accuracy and robustness of automatic segmentation algorithms, which brings uncertainty to the subsequent diagnosis of dental diseases.

[0005] In contrast, supervised methods use deep learning models trained with annotated data to improve segmentation accuracy and stability. For example, Almalki et al. applied two self-supervised learning methods to Swin Transformer on dental panoramic radiographs: SimMIM and UM-MAE (Almalki A, Latecki L, Self-supervised learning with masked image modeling for teeth numbering, detection of dental restorations, and instance segmentation in dental panoramic radiographs, Jan. 2023: 5583–5592). Zhang et al. proposed a new method that combines multiple key strategies for tooth recognition using label tree and cascade network structure, which can handle many complex situations (Zhang K, Wu J, Chen H, Lyu P, An effective teeth recognition method using label tree with cascade network structure, Computerized Medical Imaging and Graphics: The Official Journal of the Computerized Medical Imaging Society, vol. 68, 2018: 61–70). Helli et al. adopted a two-step approach where they used U-Net for prediction and then performed post-processing operations to achieve segmentation (Helli S, Hamamcı A, Tooth instance segmentation on panoramic dental radiographs usingU-Nets and morphological processing, Düzce Üniversitesi Bilim ve TeknolojiDergisi, vol. 10, no. 1, 2022: 39–50).

[0006] However, both supervised and unsupervised methods inevitably have problems with segmentation accuracy. The complexity and diversity of dental conditions make it impossible for various deep learning models to guarantee 100% accuracy. The analysis and diagnosis of dental diseases require consideration of various factors, such as oral anatomical structure, local inflammation, degree of damage, etc., which will interfere with automatic segmentation to a certain extent. Therefore, it is very necessary to introduce expert advice and judgment in a timely manner. Experts can evaluate and correct the results of automatic segmentation based on their clinical experience and knowledge, thereby improving the accuracy of diagnosis and treatment. At the same time, experts can provide more training data and annotation information for the automatic segmentation model, thereby continuously optimizing the performance and effect of the model. Summary of the invention

[0007] The purpose of the present invention is to provide a human-machine fusion tooth segmentation method based on deep learning.

[0008] The present invention adopts the following technical solution: Step (1) Use the trained Mask R-CNN model to segment teeth on panoramic X-rays; Use the annotated panoramic images to train the Mask R-CNN model to obtain feature maps of different categories of teeth; The Mask R-CNN model adopts a two-stage instance segmentation framework: Phase 1: Divide candidate tooth bounding boxes: First, the panoramic X-ray film is input into the backbone network of the Mask R-CNN model to extract tooth image features. Then, the extracted tooth image features are fused through the pyramid network to generate multiple candidate regions containing tooth structures. The second stage: Use the RoIAlign method to classify each candidate area, match each pixel on the original panoramic X-ray film with the preset fixed features in the feature map of different categories of teeth, and use the bounding box regression method to match. After classification, five tooth categories are obtained: incisors, canines, first molars, second molars, and third molars; according to the borders and classification categories of the candidate areas, a mask containing the tooth segmentation boundaries and category labels is generated to complete the segmentation; when the features of the X-ray film image are blurred and difficult to identify, the tooth category is determined according to the order of tooth arrangement.

[0009] Step (2) evaluating the tooth segmentation results; (2-1) Use the LDA method to map the multidimensional feature values ​​into a two-dimensional coordinate system for visualization, and use the HU moment as the shape feature of the segmentation mask to describe a single tooth: ;m pq is the (p,q)-order moment of a single tooth image; x and y represent the abscissa and ordinate in the image coordinate system; f(x,y) represents the gray value of the image at (x,y); p and q represent the order of the moment, x p and x qand represent the powers of the x and y coordinates, respectively, and are used to calculate moments of specific orders; through moments m of different orders pq Describe the shape characteristics of the image; use a scatter plot as an overview of the distribution of tooth clusters of each category; (2-2) Visualize the multi-dimensional features of the segmentation mask using glyph view; (2-3) View historical tags through similar example views; (2-4) View detailed feature requirements by zooming in on the view; (2-5) The consistency between the feature distribution in the segmentation result and the real structure is evaluated by using quantifiable evaluation indicators. If the consistency exceeds the set normal range, it is corrected according to the anatomical characteristics of the tooth to obtain an updated mask, and the scatter plot is updated according to the updated mask.

[0010] Step (3) feeds the updated mask back to the Mask R-CNN model for adaptive iterative optimization to improve the segmentation performance of the model.

[0011] Furthermore, the backbone network of the Mask R-CNN model adopts the ResNet50 network.

[0012] The present invention provides a visual interface to display the preliminary segmentation results of the model, and corrects errors through interactive tools, so that high-quality tooth samples are fed back into the model for adaptive iterative optimization to ensure the accuracy and efficiency of tooth segmentation. Compared with the prior art, the present invention has the following beneficial effects: the present invention integrates automatic segmentation, index evaluation and visual analysis systems, provides the ability to fully compare tooth segmentation results; combines deep learning algorithms with the anatomical characteristics of teeth, and ensures the accuracy and efficiency of tooth segmentation.

[0013] The present invention introduces an interactive correction and feedback optimization mechanism of the anatomical features of teeth, making the tooth segmentation results more accurate, and provides a comprehensive set of evaluation and optimization tools, which significantly improves the quality and efficiency of segmentation. Using the method of the present invention, preliminary tooth segmentation can be automatically performed, and various supporting information can be provided through a visual interface to help experts learn the segmentation results and detect abnormalities. Compared with traditional tooth segmentation methods, the present invention achieves higher quality correction data and iterative optimization of the segmentation model, thereby increasing the accuracy of segmentation and providing users with a more convenient and easy-to-use tooth segmentation tool. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an example diagram of the Mask R-CNN two-stage instance segmentation framework; Figure 2 It is a scatter view of the two-dimensional mapping of multi-dimensional eigenvalues ​​of the tooth image; Figure 3is a glyph view of multi-dimensional feature values ​​of a single tooth image; Figure 4 A similar example view of a single tooth image; Figure 5 It is a magnified view of a single tooth image; Figure 6 This is a system interface diagram of the present invention. DETAILED DESCRIPTION

[0015] The following is a further description of the human-machine fusion teeth segmentation method based on deep learning of the present invention in conjunction with the accompanying drawings. The specific steps are as follows: Step (1) Use the trained ResNet50 network Mask R-CNN model to segment teeth on panoramic X-rays. Use the annotated panoramic images to train the Mask R-CNN model to obtain feature maps of different categories of teeth. The Mask R-CNN model uses a two-stage instance segmentation framework, such as Figure 1 shown.

[0016] The first stage: divide the candidate tooth bounding boxes. Specifically, the panoramic X-ray film is first input into the backbone network of the Mask R-CNN model to extract the tooth image features. Then, the extracted tooth image features are fused through the pyramid network (FPN) to generate multiple candidate areas containing tooth structures.

[0017] Since Mask R-CNN is a flexible framework, the backbone network selection includes 50, 101 and 152 layers of ResNet network and VGG16 network. The present invention selects ResNet50 to analyze the panoramic X-ray film because it has fewer layers, which can avoid overfitting, and its overall IoU score reaches 75.14%, as shown in the following table.

[0018] Model Backbone IoU(%) Precision(%) Recall(%) F1-score(%) Mask R-CNN ResNet-50 75.1 75.7 83.5 79.4 Mask R-CNN ResNet-101 65.3 65.9 73.7 69.6 Mask R-CNN ResNet-152 53.4 53.9 58.1 55.9 Mask R-CNN VGG16 71.2 71.8 81.1 76.2 The second stage: Use the RoIAlign (Region of Interest Alignment) method to classify each candidate region, match each pixel on the original panoramic X-ray film with the preset fixed features in the feature map of different categories of teeth, and use the bounding box regression method to classify and obtain five tooth categories: incisors, canines, first molars, second molars, and third molars. According to the borders and classification categories of the candidate regions, a mask containing the tooth segmentation boundaries and category labels is generated to complete the segmentation. When the features of the X-ray film image are blurred and difficult to identify, the tooth category is determined according to the order in which the teeth are arranged.

[0019] Step (2) evaluates the tooth segmentation results.

[0020] (2-1) Use the LDA method to map the multidimensional feature values ​​into a two-dimensional coordinate system for visualization, and use the HU moment (MK Hu. Visual pattern recognition by moment invariants. IRE transactions on information theory, 1962, 8(2):179–187&&F. JS Doerr and AJ Florence, Amicro-xrt image analysis and machine learning methodology for the characterisation of multi-particulate capsule formulations, International Journal of Pharmaceutics: X, 2:100041, 2020. 4) as the shape feature of the segmentation mask to describe a single tooth: m pq is the (p,q)-order moment of a single tooth image; x and y represent the abscissa and ordinate in the image coordinate system; f(x,y) represents the gray value of the image at (x,y); p and q represent the order of the moment, x p and x q and represent the powers of the x and y coordinates, respectively, and are used to calculate moments of specific orders. By weighted summation, combined with the grayscale value f(x,y), moments m of different orders are obtained pq , used to describe the shape characteristics of an image.

[0021] The scatter plot is used as an overview of the distribution of tooth clusters in each category, such as Figure 2 As shown. Dimensionality reduction and mapping are used to obtain the standard range and distribution of multidimensional features for each tooth category. By comparing the newly generated segmentation mask with the sample set distribution, segmentation results that deviate from the pattern are identified, which have a higher probability of error. In the scatter plot view, each point represents a tooth sample, and different colors represent different categories. The radius of the point representing the loaded tooth sample is larger than the radius of the point representing the annotated panoramic image teeth, and the transparency of the point representing the loaded tooth sample is smaller than the transparency of the point representing the annotated panoramic image teeth, so as to distinguish them.

[0022] The shape features are extracted from a single tooth slice using the HU moment, and the centerline angle of the shape features is combined with the corresponding position information in the original panoramic X-ray film to form a set of high-dimensional feature vectors. The high-dimensional feature vectors are projected onto a two-dimensional plane using LDA (David M Blei, Andrew Y Ng, and Michael I Jordan. Latent dirichlet allocation. Journal of machine Learning research, 2003. Jan: 993–1022) to generate a scatter plot so that samples with similar features are closer to each other. In this embodiment, the training samples are displayed as solid circles, the newly loaded samples are displayed as circles with black outlines, and the samples specified by the experts are displayed as crosses.

[0023] (2-2) Use the glyph view to visualize the multi-dimensional features of the segmentation mask, such as Figure 3 As shown in Figure 2, the HU moment values ​​are encoded using radial bar graphs (e.g. Figure 3 At the same time, in the glyph, the two-dimensional coordinates of the teeth are encoded using the dashboard method (such as Figure 3 ③-④) and centerline angle (such as Figure 3 In order to visually show the difference between the segmentation results and the traditional training samples, the average value of each feature is calculated, and the features close to the average value, the features significantly higher than the average value, and the features significantly lower than the average value are encoded into three distinct colors (such as Figure 3 Different colors of icons will be used according to the recognized categories, which will help to observe the tooth classification more clearly (such as Figure 3 (6)

[0024] (2-3) View the history tags through similar example views, such as Figure 4 .

[0025] (2-4) View detailed feature requirements by zooming in on the view, such as Figure 5 .

[0026] (2-5) The consistency between the feature distribution in the segmentation result and the real structure is evaluated by quantifiable evaluation indicators, including tooth shape, tooth position and tooth angle. If the consistency exceeds the set normal range, it is corrected according to the anatomical characteristics of the tooth to obtain an updated mask, and the scatter plot is updated according to the updated mask.

[0027] Step (3) feeds the updated mask back to the Mask R-CNN model for adaptive iterative optimization to improve the segmentation performance of the model. Figure 6As shown; where (A) is a control panel for controlling parameter values ​​such as the number of training times and learning rate in the model, (A1) is a line graph showing the model optimization process, and (A2) is a bar graph showing the manual correction time for each panoramic image; (B) is a panoramic image view, (B1) is a segmentation view for displaying the segmentation mask on the panoramic image, and (B2) is a glyph view for displaying the features of tooth segmentation; (C) is a feature view, (C1) is a scatter view for outlining the relationship between tooth samples, (C2) is a zoom view for displaying more detailed features, and (C3) is a similar example view for illustrating the properties of similar instances.

[0028] The present invention sets up a series of interactive tools. The intelligent model and the anatomical features of the teeth are integrated into the tooth segmentation process. First, the automatic tooth segmentation result is obtained by loading the panoramic X-ray film in the control panel. Among them, the scatter plot view provides a concise overview of the standard range and distribution of the multi-dimensional features of each tooth category. For more detailed features, similar samples are viewed by observing the glyphs in the enlarged view and clicking the corresponding scatter points in the similar example view. By comparing the non-similarity, the consistency of the feature distribution in the segmentation result with the real structure is evaluated. When the result of the automatic segmentation deviates significantly from the normal range, it needs to be corrected by the anatomical features of the teeth.

[0029] During this process, the tooth boundaries are corrected by clicking anchor points on the teeth. When the corrected segmentation result is satisfactory, the scatter plot view will be updated to show an overview of the corrected result. Then, high-quality labeled data is selected, which is regarded as expert feedback and helps to improve the accuracy of the segmentation result. This iterative feedback loop helps to improve the performance of the model over time.

Claims

1. A human-machine fusion tooth segmentation method based on deep learning, characterized by: Step (1) Use the trained Mask R-CNN model to segment teeth on panoramic X-rays; Use the annotated panoramic images to train the Mask R-CNN model to obtain feature maps of different categories of teeth; The Mask R-CNN model adopts a two-stage instance segmentation framework: Phase 1: Delineating candidate tooth bounding boxes: First, the panoramic X-ray film is input into the backbone network of the Mask R-CNN model to extract tooth image features; Then, the extracted tooth image features are fused through a pyramid network to generate multiple candidate regions containing tooth structures; The second stage: Use the RoIAlign method to classify each candidate region, match each pixel on the original panoramic X-ray film with the preset fixed features in the feature map of different categories of teeth, and use the bounding box regression method to classify and obtain five tooth categories: incisors, canines, first molars, second molars, and third molars. According to the borders and classification categories of the candidate regions, a mask containing the tooth segmentation boundaries and category labels is generated to complete the segmentation. When the features of the X-ray image are blurred and difficult to identify, the tooth type is determined according to the order in which the teeth are arranged; Step (2) evaluating the tooth segmentation results; (2-1) Use the LDA method to map the multidimensional feature values ​​into a two-dimensional coordinate system for visualization, and use the HU moment as the shape feature of the segmentation mask to describe a single tooth: ;m pq is the (p,q)-order moment of a single tooth image; x and y represent the abscissa and ordinate in the image coordinate system; f(x,y) represents the gray value of the image at (x,y); p and q represent the order of the moment, x p and x q and represent the powers of the x and y coordinates, respectively, and are used to calculate moments of specific orders; through moments m of different orders pq Describe the shape characteristics of the image; use a scatter plot as an overview of the distribution of tooth clusters of each category; (2-2) Visualize the multi-dimensional features of the segmentation mask using glyph view; (2-3) View historical tags through similar example views; (2-4) View detailed feature requirements by zooming in on the view; (2-5) evaluating the consistency between the feature distribution in the segmentation result and the real structure through quantifiable evaluation indicators. If the consistency exceeds the set normal range, the result is corrected according to the anatomical characteristics of the tooth to obtain an updated mask, and the scatter plot is updated according to the updated mask; Step (3) feeds the updated mask back to the Mask R-CNN model for adaptive iterative optimization.

2. The human-machine fusion tooth segmentation method based on deep learning as claimed in claim 1, characterized in that: The backbone network of the Mask R-CNN model adopts the ResNet50 network.

3. The human-machine fusion tooth segmentation method based on deep learning as claimed in claim 1, characterized in that: In (2-1), the HU moment is used to extract shape features from a single tooth slice, and the centerline angle of the shape feature is combined with the corresponding position information in the original panoramic X-ray film to form a set of high-dimensional feature vectors; the LDA method is used to project the high-dimensional feature vectors onto a two-dimensional plane to generate a scatter plot.

4. The human-machine fusion tooth segmentation method based on deep learning as claimed in claim 3, characterized in that: In (2-1), dimension reduction and mapping are used to obtain the standard range and distribution of multidimensional features of each tooth category; by comparing the newly generated segmentation mask with the sample set distribution, the segmentation results that deviate from the pattern are identified; in the scatter plot, each point represents a tooth sample, and different colors represent different categories; the radius of the point representing the loaded tooth sample is larger than the radius of the point representing the annotated panoramic image teeth, and the transparency of the point representing the loaded tooth sample is smaller than the transparency of the point representing the annotated panoramic image teeth, so as to distinguish them.

5. The human-machine fusion tooth segmentation method based on deep learning as claimed in claim 1, characterized in that: In (2-2), the HU moment value is encoded using a radial bar graph; in the glyph, the two-dimensional coordinates of the tooth are encoded using a dashboard method; the average value of each feature is calculated, and the features close to the average value, the features significantly higher than the average value, and the features significantly lower than the average value are encoded into three clearly distinguishable colors; different color icons will be used according to the identified category.

6. The human-machine fusion tooth segmentation method based on deep learning as claimed in claim 1, characterized in that: In (2-5), the evaluation indicators include tooth shape, tooth position and tooth angle.

Images