A tooth segmentation and numbering method based on the derivation of constraint relations between adjacent teeth

Abstract

The present invention belongs to the field of oral endoscopy technology, and in particular, relates to a tooth segmentation and numbering method based on the derivation of constraint relationships between adjacent teeth. The method segments teeth in an RGB image using computer image processing instance segmentation techniques, and uses image classification techniques to identify a reference tooth, first determining its position number. Next, the non-reference teeth adjacent to the reference tooth are identified based on tooth coordinate overlap, and the position numbers of the adjacent teeth of the reference tooth are inferred based on the constraint relationships between the tooth position numbers of the adjacent teeth. Finally, the adjacent teeth of the numbered tooth are continuously identified and their numbers inferred until all teeth are assigned a position number.

Description

Technical Field

[0001] The present invention relates to the technical field of oral endoscopy, and in particular to a tooth segmentation and numbering method based on derivation of constraint relationships between adjacent teeth. Background Art

[0002] As an important branch of medicine, dentistry is of vital importance to human health. Its fundamental needs include the diagnosis, care, and prevention of oral diseases. However, due to the complexity and diversity of oral diseases, traditional dentistry diagnostic methods often require dentists to expend considerable time and effort, and are costly. To address these issues, researchers are exploring solutions based on artificial intelligence (AI) technology to enable intelligent monitoring of oral health. This research direction aims to free dental professionals from their daily burdens and improve population health at a lower cost.

[0003] The preliminary analysis of teeth requires segmenting and numbering the teeth according to the standard method of tooth numbering. When dentists use photos to diagnose patient problems, the shape, number and position of the teeth are the most important factors to consider. In the past, the International Dental Federation (FDI) tooth position representation and the Universal Tooth Numbering (UTN) were the two most commonly used tooth numbering system methods. Most of the existing tooth segmentation and numbering methods in RGB images are based on end-to-end deep learning instance segmentation models, such as MaskR-CNN, which extracts image features through its backbone network, further finds the possible locations of the teeth, and marks them with candidate boxes. Finally, these candidate boxes are segmented at the mask level in the head network, and the classifier is used to calculate the possibility of the numbers to which these teeth belong. However, this method has the following problems:

[0004] (1) There are 32 categories of teeth, and the characteristics of teeth in each category are relatively similar, which puts a great burden on the classifier and may not be able to classify them accurately.

[0005] (2) Teeth in camera photos are often obscured or interfered with by dental diseases, resulting in the loss of tooth features in the image, making it impossible for the model to determine the category to which it belongs.

[0006] Therefore, we proposed a tooth segmentation and numbering method based on the derivation of the constraint relationship between adjacent teeth to solve the above problem. Summary of the Invention

[0007] (1) Technical problems solved

[0008] In view of the shortcomings of the existing technology, the present invention provides a tooth segmentation and numbering method based on the derivation of the constraint relationship between adjacent teeth, which solves the problems raised in the above background technology.

[0009] (2) Technical solution

[0010] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0011] A tooth segmentation and numbering method based on the constraint relationship between adjacent teeth.

[0012] S1. Use a camera to capture a video image of the dental area, obtaining real-time images that meet user needs;

[0013] S2. Build a deep learning instance segmentation model to pre-segment the teeth in the tooth region image obtained in S1, perform tooth segmentation on the image, and obtain an image processed by the tooth segmentation module, thereby obtaining an accurate and precise tooth segmentation image using deep learning.

[0014] S3. Build a deep convolutional neural network classifier and apply all the tooth images segmented in S2 to the classifier for recognition, obtaining the FDI tooth position number for each tooth. The upper tooth with the highest confidence and the lower tooth with the highest confidence are retained as reference teeth.

[0015] S4. Complete the numbering of all teeth according to the adjacent relationship between other teeth and the reference tooth, perform the tooth numbering operation on all teeth and obtain the result map, and realize the FDI numbering of the teeth under the RGB image;

[0016] S5. Visually label the teeth in the image. First, color-code the tooth image according to the tooth mask information and digitally label the assigned FDI number on the tooth.

[0017] Furthermore, the specific construction method of the instance segmentation model is:

[0018] S21. Use the polygon drawing function of the image annotation tool labelme or labelimg to annotate all teeth in each image of the public oral image dataset, labeling them as "teeth" and obtaining a segmented dataset file for future use.

[0019] S22. Use the train.py file in YOLOv8-seg to perform multiple rounds of training on the segmentation dataset file labeled in S21, and obtain the best performing pt-format segmentation model in all rounds.

[0020] S23. Based on the best-performing segmentation model obtained in S22, use the predict.py file in YOLOv8-seg to predict the sample image obtained in S21. In the backbone network of YOLOv8-seg, the Conv module is used to reduce the size of the feature map and enhance the nonlinear representation capability; the C2f module achieves richer feature fusion through skip-layer connections and feature connection operations; the SPPF module uses a combination of serial and parallel feature pooling to enhance the ability to capture information at different scales; in the neck part, YOLOv8 uses the PAN-FPN method for feature fusion; the head network uses the category branch and the bounding box branch to predict the target category and bounding box respectively; finally, YOLOv8 outputs N target candidate box information, including the category, location and size of each target; YOLOv8-seg is an instance segmentation variant of YOLOv8. Compared with YOLOv8, YOLOv8-seg adds a mask branch in the head network;

[0021] S24. Final output N: N is the number of teeth in the sample image predicted by the model, N dimensions of tooth information, each dimension of information includes the position and size of the tooth candidate box, and a binary mask matrix of the same size as the original image, where 1 represents the part of the image belonging to the tooth and 0 represents the background part.

[0022] Furthermore, the specific method of S3 is:

[0023] S31. The instance segmentation model in S22 predicts the positions and sizes of the candidate frames of the teeth in the sample image, and crops all the candidate frames into images. The number of images is the same as the number of teeth predicted by the segmentation model, which is N.

[0024] S32. The publicly available single tooth image dataset is manually divided into 32 categories according to its FDI number, and different categories are grouped into different folder directories to obtain the classification dataset file for backup;

[0025] S33. Use the train.py file in the YOLOv8-cls image classification network to train the classification dataset obtained in S32 and obtain the best-performing classification model in pt format across all rounds.

[0026] S34. Based on the classification model with the highest comprehensive index obtained in S33, use the predict.py file in YOLOv8-cls to predict the N cropped tooth images in S31. The final output is N dimensions of tooth instance number information. Each dimension of the number information includes the FDI number to which the image belongs and its confidence level. There are 32 categories in total: 11-18 for the upper left, 21-28 for the upper right, 31-38 for the lower right, and 41-48 for the lower left.

[0027] S35. Based on the numbering results of all tooth instances predicted in S34, among the predicted results of 11-18 or 21-28, select a tooth with the highest classification confidence as the upper reference tooth; among the predicted results of 31-38 or 41-48, select a tooth with the highest classification confidence as the lower reference tooth.

[0028] Furthermore, the specific method of S4 is:

[0029] S41. Calculate the average of the center coordinates of the two reference teeth obtained in S35 in the sample image as the sorting center coordinates. The center coordinates of the first reference tooth are (x1, y1), and the center coordinates of the second reference tooth are (x2, y2). Then the sorting center coordinates are (x3, y3) = ((x1 + x2) / 2, ((y1 + y2) / 2)).

[0030] S41. Calculate the Euclidean distance between the center coordinates of all non-reference teeth and the sorting center coordinates, and take the center coordinates of a non-reference tooth (x i ,y i ), the calculation formula is: After all non-reference teeth are calculated, d i ; i=1,2,3,…,n; n is the number of non-reference teeth;

[0031] S42. Calculate the priority of all non-reference teeth according to the Euclidean distance d between the non-reference teeth and the sorting center coordinates. i Assign priority in ascending order of size;

[0032] S43. According to the priority order, a tooth is selected as the tooth to be sorted. The sorting process is as follows:

[0033] S431. Set two scores, x_score and y_score. The smaller the difference between the x-axis coordinates of the center points of any two tooth detection frames, the higher the x_score score. The greater the y-axis overlap of the detection frames, the higher the y_score score. A y_score threshold y_thresholds is preset.

[0034] S432. Calculate the x_score and y_score of the teeth to be sorted and all sorted teeth;

[0035] S433. The tooth combination with a y_score score greater than the threshold y_thresholds is selected as the adjacent tooth combination, and the tooth combination with the highest x_score score is selected as the adjacent tooth combination;

[0036] S434. Determine whether the tooth to be sorted is adjacent to the left or right. If the horizontal coordinate of the center coordinate of the tooth to be sorted is greater than the horizontal coordinate of the center coordinate of the adjacent tooth, it is determined to be right adjacent, otherwise it is determined to be left adjacent.

[0037] S435. According to the judgment result and the FDI numbering system, a unique number is assigned. For example, if the adjacent tooth is adjacent to the tooth to be sorted on the left, and the adjacent tooth FDI number is 34, the number assigned to the tooth to be sorted is 35;

[0038] S44. Repeat the above step S43 until all teeth are assigned FDI numbers.

[0039] (3) Beneficial effects

[0040] Compared with the existing technology, the present invention provides a tooth segmentation and numbering method based on the derivation of the constraint relationship between adjacent teeth, which has the following beneficial effects:

[0041] The present invention finds all tooth instances through an instance segmentation model, uses an image classification model to confirm the reference teeth, and derives the FDI numbers of all teeth based on the reference teeth. This can alleviate the problem of excessive classifier burden in the end-to-end numbering method. For dental diseases, occlusions and other phenomena that often appear in camera photos, as long as the segmentation model can recognize the tooth, its number can be further confirmed. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a structural schematic diagram of a specific embodiment of the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the specific numbering derivation method of the present invention;

[0044] Figure 3 Schematic diagram of the FDI numbering system adopted in the present invention;

[0045] Figure 4 This is an input and output schematic diagram of the present invention. DETAILED DESCRIPTION

[0046] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0047] Example

[0048] like Figure 1-4As shown, one embodiment of the present invention proposes a tooth segmentation and numbering method based on the derivation of the constraint relationship between adjacent teeth, including:

[0049] S1. The user holds a camera to obtain a video image of the tooth area, and obtains the following Figure 4 The input image shown is used to obtain real-time images that meet user needs;

[0050] S2. Build a deep learning instance segmentation model to pre-segment the teeth in the tooth area image obtained in S1, and perform the following operations on the image: Figure 4 The tooth segmentation operation shown is performed and the result image of the tooth segmentation module is obtained, and an accurate and precise tooth segmentation image is obtained in a deep learning manner;

[0051] The specific construction method of the instance segmentation model is:

[0052] S21. Use the polygon drawing function of the image annotation tool labelme or labelimg to annotate all teeth in each image of the public oral image dataset, labeling them as "teeth" and obtaining a segmented dataset file for future use.

[0053] S22. Use the train.py file in YOLOv8-seg to perform multiple rounds of training on the segmentation dataset file labeled in S21, and obtain the best performing pt-format segmentation model in all rounds.

[0054] S23. Based on the best-performing segmentation model obtained in S22, use the predict.py file in YOLOv8-seg to predict the sample image obtained in S21. In the backbone network of YOLOv8-seg, the Conv module is used to reduce the size of the feature map and enhance the nonlinear representation capability; the C2f module achieves richer feature fusion through skip-layer connections and feature connection operations; the SPPF module uses a combination of serial and parallel feature pooling to enhance the ability to capture information at different scales; in the neck part, YOLOv8 uses the PAN-FPN method for feature fusion; the head network uses the category branch and the bounding box branch to predict the target category and bounding box respectively; finally, YOLOv8 outputs N target candidate box information, including the category, location and size of each target; YOLOv8-seg is an instance segmentation variant of YOLOv8. Compared with YOLOv8, YOLOv8-seg adds a mask branch in the head network;

[0055] S24. Finally, output N dimensions of tooth information (N is the number of teeth in the sample image predicted by the model). Each dimension of information includes the position and size of the tooth candidate box and a binary mask matrix of the same size as the original image (1 represents the part of the image belonging to the tooth, and 0 represents the background).

[0056] S3. Build a deep convolutional neural network classifier and apply all the tooth images segmented in S2 to the classifier for recognition, obtaining the FDI tooth position number for each tooth. The upper tooth with the highest confidence and the lower tooth with the highest confidence are retained as reference teeth.

[0057] The specific method of S3 is:

[0058] S31. The instance segmentation model in S22 predicts the positions and sizes of the candidate frames of the teeth in the sample image, and crops all the candidate frames into images. The number of images is the same as the number of teeth predicted by the segmentation model, which is N.

[0059] S32. The publicly available single tooth image dataset is manually divided into 32 categories according to its FDI number, and different categories are grouped into different folder directories to obtain the classification dataset file for backup;

[0060] S33. Use the train.py file in the YOLOv8-cls image classification network to train the classification dataset obtained in S32 and obtain the best-performing classification model in pt format across all rounds.

[0061] S34. Based on the classification model with the highest comprehensive index obtained in S33, use the predict.py file in YOLOv8-cls to predict the N cropped tooth images in S31, and ultimately output N dimensions of tooth instance number information. The number information in each dimension includes the FDI number to which the image belongs and its confidence level. There are 32 categories in total (11-18 for the upper left, 21-28 for the upper right, 31-38 for the lower right, and 41-48 for the lower left).

[0062] S35. Based on the numbering results of all tooth instances predicted in S34, among the predicted results of 11-18 or 21-28, select a tooth with the highest classification confidence as the upper reference tooth; among the predicted results of 31-38 or 41-48, select a tooth with the highest classification confidence as the lower reference tooth.

[0063] S4. Figure 2 As shown, all teeth are numbered according to the adjacent relationship between other teeth and the reference teeth, and all teeth are numbered as follows. Figure 4 The tooth numbering operation shown in the figure is performed to obtain the result image, and the FDI numbering of the teeth under the RGB image is realized. The specific method shown is:

[0064] The specific method of S4 is:

[0065] S41. Calculate the average of the center coordinates of the two reference teeth obtained in S35 in the sample image as the sorting center coordinates. The center coordinates of the first reference tooth are (x1, y1), and the center coordinates of the second reference tooth are (x2, y2). Then the sorting center coordinates are (x3, y3) = ((x1 + x2) / 2, ((y1 + y2) / 2)).

[0066] S41. Calculate the Euclidean distance between the center coordinates of all non-reference teeth and the sorting center coordinates, and take the center coordinates of a non-reference tooth (x i ,y i ), the calculation formula is: After all non-reference teeth are calculated, d i i=1,2,3,…,n (n is the number of non-reference teeth);

[0067] S42. Calculate the priority of all non-reference teeth according to the Euclidean distance d between the non-reference teeth and the sorting center coordinates. i Assign priority in ascending order of size;

[0068] S43. According to the priority order, a tooth is selected as the tooth to be sorted. The sorting process is as follows:

[0069] S431. Set two scores, x_score and y_score. The smaller the difference between the x-axis coordinates of the center points of any two tooth detection frames, the higher the x_score score. The greater the y-axis overlap of the detection frames, the higher the y_score score. A y_score threshold y_thresholds is preset.

[0070] S432. Calculate the x_score and y_score of the teeth to be sorted and all sorted teeth;

[0071] S433. The tooth combination with a y_score score greater than the threshold y_thresholds is selected as the adjacent tooth combination, and the tooth combination with the highest x_score score is selected as the adjacent tooth combination;

[0072] S434. Determine whether the tooth to be sorted is adjacent to the left or right. If the horizontal coordinate of the center coordinate of the tooth to be sorted is greater than the horizontal coordinate of the center coordinate of the adjacent tooth, it is determined to be right adjacent, otherwise it is determined to be left adjacent.

[0073] S435. According to the judgment result and the FDI numbering system, a unique number is assigned. For example, if the adjacent tooth is adjacent to the tooth to be sorted on the left, and the adjacent tooth FDI number is 34, the number assigned to the tooth to be sorted is 35;

[0074] S44. Repeat the above step S43 until all teeth are assigned FDI numbers;

[0075] S5. Visually annotate the teeth in the image. First, color-code the tooth image according to the tooth mask information, and digitally mark the assigned FDI number on the teeth. Provide the position information and FDI number information of the teeth in the sample image. Subsequently, combine it with dental disease detection to build a tooth-disease association model to achieve precise positioning of dental diseases, help users detect and prevent oral health problems as early as possible, and realize automated preventive monitoring of oral health.

[0076] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A tooth segmentation and numbering method based on the derivation of constraint relationships between adjacent teeth, characterized by: S1. Use a camera to capture a video image of the dental area, obtaining real-time images that meet user needs; S2. Build a deep learning instance segmentation model to pre-segment the teeth in the tooth region image obtained in S1, perform tooth segmentation on the image, and obtain an image processed by the tooth segmentation module, thereby obtaining an accurate and precise tooth segmentation image using deep learning. The deep learning instance segmentation model is implemented based on YOLOv8-seg. In the backbone network of YOLOv8-seg, the Conv module is used to reduce the size of the feature map and enhance the nonlinear representation capability; the C2f module achieves richer feature fusion through skip-layer connection and feature connection operations; the SPPF module uses a combination of serial and parallel feature pooling to enhance the ability to capture information at different scales; in the neck part, YOLOv8 uses the PAN-FPN method for feature fusion; the head network uses the category branch and the bounding box branch to predict the target category and bounding box respectively; finally, YOLOv8 outputs N target candidate box information, including the category, candidate box position and size of each target; YOLOv8-seg is an instance segmentation variant of YOLOv8. Compared with YOLOv8, YOLOv8-seg adds a mask branch in the head network; S3. Build a deep convolutional neural network classifier and apply all the tooth images segmented in S2 to the classifier for recognition, obtaining the FDI tooth position number for each tooth. The upper tooth with the highest confidence and the lower tooth with the highest confidence are retained as reference teeth. S4. Complete the numbering of all teeth according to the adjacent relationship between other teeth and the reference tooth, perform the tooth numbering operation on all teeth and obtain the result map, and realize the FDI numbering of the teeth under the RGB image; The specific method of S4 is: S41. Calculate the average of the center coordinates of the two reference teeth obtained in S35 in the sample image as the sorting center coordinates. The center coordinates of the first reference tooth are (x1, y1), and the center coordinates of the second reference tooth are (x2, y2). Then the sorting center coordinates are (x3, y3) = ((x1 + x2) / 2, ((y1 + y2) / 2)). S42. Calculate the Euclidean distance between the center coordinates of all non-reference teeth and the sort center coordinates, and take the center coordinates of a non-reference tooth (x i ,y i ), the calculation formula is: After all non-reference teeth are calculated, d i ; i=1,2,3,…,n; n is the number of non-reference teeth; S43. Calculate the priority of all non-reference teeth according to the Euclidean distance d between the non-reference teeth and the sorting center coordinates. i Assign priority in ascending order of size; S44. According to the priority order, select a tooth as the tooth to be sorted; S45. Repeat the above S44 step until all teeth are assigned FDI numbers; S5. Visually label the teeth in the image. First, color-code the tooth image according to the tooth mask information and digitally label the assigned FDI number on the tooth.

2. The tooth segmentation and numbering method based on the derivation of the constraint relationship between adjacent teeth according to claim 1, characterized in that: The specific construction method of the instance segmentation model is: S21. Use the polygon drawing function of the image annotation tool labelme or labelimg to annotate all teeth in each image of the public oral image dataset, labeling them as "teeth" and obtaining a segmented dataset file for future use. S22. Use the train.py file in YOLOv8-seg to perform multiple rounds of training on the segmentation dataset file labeled in S21, and obtain the best performing pt-format segmentation model in all rounds. S23. Based on the best performing segmentation model obtained in S22, use the predict.py file in YOLOv8-seg to predict the sample image obtained in S21. S24. Final output N: N is the number of teeth in the sample image predicted by the model, N dimensions of tooth information, each dimension of information includes the position and size of the tooth candidate box, and a binary mask matrix of the same size as the original image, where 1 represents the part of the image belonging to the tooth and 0 represents the background part.

3. The tooth segmentation and numbering method based on derivation of constraint relationships between adjacent teeth according to claim 1, characterized in that: The specific method of S3 is: S31. The instance segmentation model in S22 predicts the positions and sizes of the candidate frames of the teeth in the sample image, and crops all the candidate frames into images. The number of images is the same as the number of teeth predicted by the segmentation model, which is N. S32. The publicly available single tooth image dataset is manually divided into 32 categories according to its FDI number, and different categories are grouped into different folder directories to obtain the classification dataset file for backup; S33. Use the train.py file in the YOLOv8-cls image classification network to train the classification dataset obtained in S32 and obtain the best-performing classification model in pt format across all rounds. S34. Based on the classification model with the highest comprehensive index obtained in S33, use the predict.py file in YOLOv8-cls to predict the N cropped tooth images in S31. The final output is N dimensions of tooth instance number information. Each dimension of the number information includes the FDI number to which the image belongs and its confidence level. There are 32 categories in total: 11-18 for the upper left, 21-28 for the upper right, 31-38 for the lower right, and 41-48 for the lower left. S35. Based on the numbering results of all tooth instances predicted in S34, among the predicted results of 11-18 or 21-28, select a tooth with the highest classification confidence as the upper reference tooth; among the predicted results of 31-38 or 41-48, select a tooth with the highest classification confidence as the lower reference tooth.

4. The tooth segmentation and numbering method based on derivation of constraint relationships between adjacent teeth according to claim 1, characterized in that: The specific sorting process of S44 is as follows: S441. Set two scores, x_score and y_score; the smaller the difference in the x-axis coordinates of the center points of any two tooth detection frames, the higher the x_score score, and the greater the y-axis overlap of the detection frames, the higher the y_score score. A y_score threshold y_thresholds is preset; S442. Calculate the x_score and y_score of the teeth to be sorted and all sorted teeth; S443. The tooth combination with a y_score score greater than the threshold y_thresholds is selected as the adjacent tooth combination, and the tooth combination with the highest x_score score is selected as the adjacent tooth combination; S444. Determine whether the tooth to be sorted is adjacent to the left or right of the adjacent tooth. If the horizontal coordinate of the center coordinate of the tooth to be sorted is greater than the horizontal coordinate of the center coordinate of the adjacent tooth, it is determined to be right adjacent, otherwise it is determined to be left adjacent; S445. Assign a unique number based on the judgment result and the FDI numbering system. Assuming that the adjacent tooth is adjacent to the tooth to be sorted on the left, and the adjacent tooth FDI number is 34, the number assigned to the tooth to be sorted is 35.

Images