Segmentation method, system, device and storage medium for third molar and mandibular canal

Abstract

The application discloses a third molar and mandibular canal segmentation method, system, device and storage medium, and comprises the following steps: acquiring a panoramic X-ray image of a tooth to be segmented; inputting the panoramic X-ray image of the tooth to be segmented into a trained region of interest detection model, and outputting a region of interest containing a third molar and a mandibular canal; inputting the region of interest containing the third molar and the mandibular canal into a trained image enhancement model, and outputting an enhanced region of interest; inputting the enhanced region of interest into a trained image segmentation model, and outputting a segmented third molar image and a mandibular canal image. The third molar and the mandibular canal are segmented.

Description

Technical Field

[0001] This invention relates to the field of image segmentation technology, and in particular to methods, systems, devices, and storage media for segmenting the third molar and mandibular canal. Background Technology

[0002] The statements in this section merely refer to the background art related to this invention and do not necessarily constitute prior art.

[0003] The lower third molars (M3) are the last teeth to erupt in a person's mouth, also known as mandibular wisdom teeth, and they often fail to erupt properly due to insufficient space. Panoramic radiographs (PRs) can provide precise location and angular information about the M3, helping dentists determine whether they need to be extracted. By segmenting the M3, dentists can better understand its impact on surrounding teeth and alveolar bone, as well as potential complications such as crowding and infection.

[0004] The mandibular canal (MC) is an important neural structure located within the mandible, responsible for sensation and movement of the lips, jaw, and tongue. Understanding the location of the third molar (M3) relative to the MC before extraction surgery can effectively prevent nerve damage. Prescriptions (PRs) can provide precise location and morphological information of the MC, helping dentists plan the extraction procedure and reduce potential risks. By segmenting the MC, dentists can better assess its relationship to the M3, thus determining whether alternative treatments, such as surgical resection or traction, are necessary.

[0005] Furthermore, segmenting M3 and MC can assist doctors in diagnosing pericoronitis based on imaging findings and in the preliminary detection of MC disease. This is because pericoronitis presents as an area of ​​increased density, while MC disease presents as an area of ​​abnormal density or expansion.

[0006] However, because M3 grows relatively far back, it is often obscured by other teeth or bone tissue, and usually appears as a blurry line on PRs, making it difficult to determine its exact location and shape. Furthermore, differences in the equipment and parameters used by different doctors in different clinics lead to significant variations in the image quality of PRs, which are often accompanied by considerable noise. This poses a significant challenge for professional doctors to visually observe M3 and MC.

[0007] While existing methods have achieved some success in segmenting M3 and MC, several problems remain to be addressed:

[0008] 1) Existing methods still use manual cropping when it comes to cropping regions of interest (ROIs). This is not only time-consuming and labor-intensive, but the cropping criteria are also affected by the doctor's subjective experience. The consistency of ROIs cannot be guaranteed, which is detrimental to feature extraction by neural networks.

[0009] 2) Existing segmentation methods all use the general medical image segmentation model UNet based on convolutional neural networks (CNN), and have not attempted targeted improvements. Therefore, the actual segmentation results cannot meet the needs of M3 and MC segmentation in clinical diagnosis.

[0010] 3) Because the quality of PRs collected from different hospitals varies greatly, existing methods have introduced transfer learning to train in two stages. This not only requires a large amount of training data, but also increases the complexity of training the target model, which is not conducive to clinical auxiliary diagnosis in hospitals. Summary of the Invention

[0011] To address the shortcomings of existing technologies, this invention provides a method, system, device, and storage medium for segmenting the third molar and mandibular canal. Furthermore, this invention develops a deep learning-based framework for automatically segmenting the M3 and MC on PRs (Profiles, Sequences, and Orthogonals). This framework employs a network specifically designed for fuzzy contour segmentation tasks, achieving excellent M3 and MC segmentation results, a feat not achieved by previous segmentation methods using general semantic segmentation models.

[0012] On the one hand, it provides a method for dividing the third molar and the mandibular canal;

[0013] Methods for dividing the third molar and mandibular canal include:

[0014] Obtain panoramic X-ray images of the teeth to be segmented;

[0015] The panoramic X-ray image of the teeth to be segmented is input into the trained region of interest detection model, and the output is a region of interest containing the third molar and the mandibular canal.

[0016] The region of interest containing the third molar and mandibular canal is input into the trained image augmentation model, and the augmented region of interest is output.

[0017] The enhanced region of interest is input into the trained image segmentation model, which outputs the segmented images of the third molar and the mandibular canal.

[0018] On the other hand, it provides a segmentation system for the third molar and mandibular canal;

[0019] The segmentation system of the third molar and mandibular canal includes:

[0020] The acquisition module is configured to acquire panoramic X-ray images of the teeth to be segmented.

[0021] The region detection module is configured to: input the panoramic X-ray image of the teeth to be segmented into the trained region of interest detection model, and output the region of interest containing the third molar and the mandibular canal;

[0022] The enhancement module is configured to input the region of interest containing the third molar and mandibular canal into the trained image enhancement model and output the enhanced region of interest.

[0023] The segmentation module is configured to input the enhanced region of interest into the trained image segmentation model and output the segmented images of the third molar and mandibular canal.

[0024] Furthermore, an electronic device is also provided, including:

[0025] Memory, used for non-transitory storage of computer-readable instructions; and

[0026] Processor, for executing the computer-readable instructions,

[0027] When the computer-readable instructions are executed by the processor, they perform the method described in the first aspect above.

[0028] In another aspect, a storage medium is also provided for non-transitory storage of computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0029] In another aspect, a computer program product is also provided, including a computer program that, when run on one or more processors, is used to implement the method described in the first aspect above.

[0030] The above technical solution has the following advantages or beneficial effects:

[0031] This invention proposes a fully automated method for detecting M3 and MC on PRs. First, a lightweight object detection network is used to detect ROIs from PRs; then, an image enhancement method based on generative adversarial networks is used to improve the quality of images from other data sources, aiming to improve the segmentation performance in actual tests; finally, a semantic segmentation network based on the TransFormer architecture is used to segment M3 and MC, thereby achieving segmentation.

[0032] Using the YOLOv7-tiny object detection model to prune ROIs not only saves time and effort, but also ensures the consistency of ROIs, which is more conducive to feature extraction from neural networks.

[0033] For segmentation tasks involving fuzzy boundaries such as M3 and MC on PRs, a segmentation model RPIFormer based on the TransFormer architecture is proposed, which achieves better segmentation results in testing.

[0034] By training CycleGAN to obtain an image enhancement model, a new approach is provided to solve the problem of large differences in image quality caused by different data sources.

[0035] This invention introduces deep learning algorithms into the preprocessing and postprocessing stages of segmenting M3 and MC tasks on PRs, thereby achieving fully automatic segmentation, which has never been achieved by previous methods. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is the overall flowchart of the automatic segmentation of M3 and MC on PRs according to the present invention;

[0038] Figure 2(a) is the architecture diagram of the YOLOv7-tiny network;

[0039] Figure 2(b) is a diagram of the internal structure of the CBL module;

[0040] Figure 2(c) is a diagram of the internal structure of the MCB module;

[0041] Figure 2(d) is a diagram of the internal structure of the SPPCSP module;

[0042] Figure 3(a) is the architecture diagram of the RPIFormer network;

[0043] Figure 3(b) shows the specific architecture of the Transformer coding block;

[0044] Figure 4 These are the splitting results of RPIFormer and SegFormer on the D2 test set for M3;

[0045] Figure 5 These are the splitting results of MC by RPIFormer and SegFormer on the D2 test set;

[0046] Figure 6 This is the architecture diagram of the CycleGAN network;

[0047] Figures 7(a) and 7(b) are examples of the results of a split test performed on randomly selected patients in a clinical setting. Detailed Implementation

[0048] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0049] Example 1

[0050] This embodiment provides a method for segmenting the third molar and the mandibular canal;

[0051] like Figure 1 As shown, the method of dividing the third molar and mandibular canal includes:

[0052] S101: Obtain panoramic X-ray images of the teeth to be segmented;

[0053] S102: Input the panoramic X-ray image of the teeth to be segmented into the trained region of interest detection model, and output the region of interest containing the third molar and mandibular canal;

[0054] S103: Input the region of interest containing the third molar and mandibular canal into the trained image augmentation model and output the augmented region of interest;

[0055] S104: Input the enhanced region of interest into the trained image segmentation model and output the segmented images of the third molar and mandibular canal.

[0056] Furthermore, the process of obtaining the trained region of interest detection model includes:

[0057] Construct the first dataset, which consists of panoramic X-ray images of the third molar and the region of interest of the mandibular canal that have been annotated;

[0058] The first dataset is input into the object detection network YOLOv7-tiny. The object detection network YOLOv7-tiny is trained. When the total loss function value no longer decreases, or the number of iterations exceeds the set number, the training is stopped, and the trained object detection network YOLOv7-tiny is obtained. The trained object detection network YOLOv7-tiny is used as the trained region of interest detection model.

[0059] The total loss function is the sum of the rectangular box loss function and the confidence loss function.

[0060] Furthermore, the rectangular frame loss is calculated as follows:

[0061]

[0062] Among them, B pred B represents the area of ​​the prediction box.true B represents the actual bounding box area, while B c L represents the area of ​​the frame corresponding to the longest diagonal of the two frames. giou This represents the rectangular box loss function.

[0063] Furthermore, the confidence loss is calculated as follows:

[0064]

[0065] Among them, y i p represents the true label of the sample, which is 0 or 1. i L represents the predicted probability (between 0 and 1). bce This represents the confidence loss function.

[0066] For example, the first dataset initially selected PRs (Properties of Interest) from 100 patients photographed at a hospital. After discussion among senior dentists, the inclusion criteria were determined as follows: at least one lower third molar (M3) was present in the patient's PRs, and cases with significant bone resorption and odontoid cysts around impacted teeth were excluded from further analysis. Then, the LabelImg tool was used to label Regions of Interest (ROIs) containing M3 and MC, obtaining rectangular bounding boxes (coordinates of the top-left point, width, and height), thus forming the first dataset. This dataset was then divided into training and testing sets in a 7:3 ratio. The lightweight object detection network YOLOv7-tiny was trained and validated on the first dataset to achieve automatic ROI detection and localization, resulting in a trained Region of Interest detection model.

[0067] During training, the batch size was set to 64, the Adam optimizer was used, the initial learning rate was 0.0001, the L2 decay coefficient was 0, and a cosine annealing learning strategy was employed for 200 epochs. During testing, Non-Maximum Suppression (DIOUNMS) was used for bounding box selection, with a confidence threshold of 0.55 and an NMS threshold of 0.45. The final test accuracy reached 99%.

[0068] Further, as shown in Figure 2(a), the target detection network YOLOv7-tiny has the following network structure:

[0069] The input layer, backbone network, feature fusion layer, and detection head are connected in sequence.

[0070] The input layer is used to input panoramic X-ray images and to perform size scaling on the input panoramic X-ray images;

[0071] The backbone network is used to extract features from scaled images to obtain feature maps at different scales;

[0072] The feature fusion layer stacks and connects feature maps of different scales to achieve feature fusion;

[0073] The detection head predicts a set of bounding boxes for each feature map location and provides the confidence score for each bounding box, thereby converting the feature map into the location of the target box.

[0074] Further, as shown in Figure 2(a), the backbone network has the following network structure:

[0075] The first CBL module, the second CBL module, the first MCB module, the first MX module, the second MCB module, the second MX module, the third MCB module, the third MX module, and the fourth MCB module are connected in sequence.

[0076] Furthermore, the feature fusion layer has the following network structure:

[0077] The SPPCSP module has its input terminal connected to the output terminal of the fourth MCB module.

[0078] The output of the SPPCSP module is connected to the input of the third CBL module;

[0079] The output of the third MCB module is connected to the input of the fourth CBL module;

[0080] The output of the second MCB module is connected to the input of the fifth CBL module;

[0081] The output of the third CBL module is connected to the input of the first upsampling module;

[0082] The output terminals of the first upsampling module and the fourth CBL module are both connected to the input terminal of the first parallel splicing unit; the output terminal of the first parallel splicing unit is connected to the input terminal of the fifth MCB module.

[0083] The output of the fifth MCB module is connected to the input of the sixth CBL module;

[0084] The output of the sixth CBL module is connected to the input of the second upsampling module;

[0085] The output terminals of the second upsampling module and the fifth CBL module are both connected to the input terminal of the second parallel splicing unit, and the output terminal of the second parallel splicing unit is connected to the input terminal of the sixth MCB module.

[0086] The output of the sixth MCB module is connected to the input of the seventh CBL module;

[0087] The output terminals of the seventh CBL module and the fifth MCB module are both connected to the input terminal of the third parallel splicing unit, and the output terminal of the third parallel splicing unit is connected to the input terminal of the seventh MCB module.

[0088] The output of the seventh MCB module is connected to the input of the eighth CBL module;

[0089] The output terminals of the eighth CBL module and the SPPCSP module are both connected to the input terminal of the fourth parallel splicing unit, and the output terminal of the fourth parallel splicing unit is connected to the input terminal of the eighth MCB module.

[0090] Furthermore, the detection head has a network structure including:

[0091] CBL Module 9, CBL Module 10, CBL Module 11, CBL Module 12, CBL Module 13, and CBL Module 14;

[0092] The input terminal of the ninth CBL module is connected to the output terminal of the sixth MCB module;

[0093] The output of the ninth CBL module is connected to the input of the tenth CBL module, and the output of the tenth CBL module outputs the first category label.

[0094] The input terminal of the eleventh CBL module is connected to the output terminal of the seventh MCB module;

[0095] The output of the eleventh CBL module is connected to the input of the twelfth CBL module, and the output of the twelfth CBL module outputs the second category label;

[0096] The input terminal of the thirteenth CBL module is connected to the output terminal of the eighth MCB module;

[0097] The output of the thirteenth CBL module is connected to the input of the fourteenth CBL module, and the output of the fourteenth CBL module outputs the third category label.

[0098] Furthermore, as shown in Figure 2(c), the internal structures of the first, second, third, fourth, fifth, sixth, seventh, and eighth MCB modules are the same. The internal structure of the first MCB module includes:

[0099] Two parallel branches: the first branch and the second branch;

[0100] The input terminal of the first branch is connected to the input terminal of the first MCB module; the first branch is equipped with a fifteenth CBL module; the output terminal of the first branch is connected to the input terminal of the fifth parallel splicing unit.

[0101] The input terminal of the second branch is connected to the input terminal of the first MCB module; the second branch is provided with a sixteenth CBL module, a seventeenth CBL module and an eighteenth CBL module connected in sequence;

[0102] The output of the eighteenth CBL module is connected to the input of the fifth parallel splicing unit;

[0103] The output of the fifth parallel splicing unit is connected to the input of the nineteenth CBL module;

[0104] The output of the nineteenth CBL module is connected to the output of the first MCB module.

[0105] Further, as shown in Figure 2(d), the SPPCSP module includes:

[0106] The two branches in parallel: the third branch and the fourth branch;

[0107] The third branch includes: a twentieth CBL module, the input of which is connected to the input of the SPPCSP module, and the output of which is connected to the input of the sixth parallel splicing unit; the output of the twentieth CBL module is also connected to the input of the sixth parallel splicing unit through a first MAX module; the output of the twentieth CBL module is also connected to the input of the sixth parallel splicing unit through a second MAX module; and the output of the twentieth CBL module is also connected to the input of the sixth parallel splicing unit through a third MAX module.

[0108] The output of the sixth parallel splicing unit is connected to the input of the twenty-first CBL module;

[0109] The output of the 21st CBL module is connected to the input of the 7th parallel splicing unit;

[0110] The fourth branch is provided with a twenty-second CBL module; the input terminal of the twenty-second CBL module is connected to the input terminal of the SPPCSP module, and the output terminal of the twenty-second CBL module is connected to the input terminal of the seventh parallel splicing unit.

[0111] The output of the seventh parallel splicing unit is connected to the input of the twenty-third CBL module.

[0112] Furthermore, the internal structures of the first CBL module to the twenty-third CBL module are the same. The first CBL module includes a convolutional layer, a batch normalization layer and a linear rectifier unit with leakage, as shown in Figure 2(b).

[0113] Furthermore, the internal structures of the first MX module to the third MX module are the same. The internal structure of the first MX module includes: MX is a pooling layer, a common operation in convolutional neural networks;

[0114] Furthermore, the internal structures of the first MAX module to the third MAX module are identical. The internal structure of the first MAX module includes: MAX is a max pooling layer, a common pooling operation; it reduces the spatial size of the feature map and extracts important features by selecting the maximum value within the pooling window. It is widely used in convolutional neural networks to help the network achieve features such as feature compression, translation invariance, and feature extraction.

[0115] For example, this embodiment uses the YOLOv7-tiny network, whose overall network structure consists of four parts: input layer, backbone, feature fusion layer, and head, as shown in Figure 2(a). First, the input image size is scaled to 640*640*3; then, the backbone extracts the feature representation of the input image, gradually reducing the image resolution through multiple convolution and pooling layers while increasing the number of channels to capture features at different scales; then, the neck layer stacks and connects the feature maps from different layers to achieve feature fusion; finally, the head layer predicts a set of bounding boxes for each feature map location and provides the confidence score of each bounding box, thereby converting the feature map into the location of the target box.

[0116] Furthermore, the training process of the trained image segmentation model includes:

[0117] An image segmentation model is constructed, which is implemented using the segmentation network RPIFormer.

[0118] Construct a second training set; the second training set is obtained by inputting newly collected panoramic X-ray images into the trained region of interest detection model, outputting regions of interest containing the third molar and mandibular canal, and annotating the regions of interest of the third molar and mandibular canal; perform data augmentation processing on the second training set;

[0119] The augmented second training set is input into the image segmentation model to train the model. Training is stopped when the model's loss function value no longer decreases, or when the training exceeds the set number of iterations, and the trained image segmentation model is obtained.

[0120] For example, newly collected PRs are input into the trained YOLOv7-tiny network to detect ROIs containing M3 and MC, the cropped ROIs are labeled to form a second dataset, and then it is split and data augmented.

[0121] The newly collected 250 PRs were input into the trained YOLOv7-tiny network for object detection to obtain ROIs that meet the segmentation requirements. Then, M3 and MC in the ROI were labeled, and the labeling standard for M3 was determined to be all visible contours, while the labeling standard for MC was the visible contour between the mandibular foramen and the buccal foramen.

[0122] The specific annotation process was as follows: Before the annotation task, all dentists received training on the use of annotation tools. First, two professional dentists with more than 15 years of clinical experience used the open-source data annotation tool LabelMe to annotate M3 and MC in duplicate. Then, a senior dentist with more than 20 years of clinical experience was responsible for reviewing and revising all labels.

[0123] After completing the data labeling, a second dataset was obtained and divided into training, validation, and test sets in a 7:2:1 ratio. Data augmentation operations such as random rotation, contrast enhancement, and contrast reduction were performed on the training and validation images in the second dataset using the Augmentor library. This effectively increased sample diversity and aimed to improve the model's generalization ability during training.

[0124] Further, as shown in Figure 3(a), the segmentation network RPIFormer has the following network structure:

[0125] Interconnected encoders and decoders;

[0126] The encoder includes: Overlap Patch Embeddings, a first Transformer encoding block, a second Transformer encoding block, a third Transformer encoding block, and a fourth Transformer encoding block connected in sequence;

[0127] The decoder includes: a first multilayer perceptron layer and a second multilayer perceptron layer;

[0128] The input terminal of the first multilayer perceptron layer is connected to the output terminal of the first Transformer coding block, the output terminal of the second Transformer coding block, the output terminal of the third Transformer coding block, and the output terminal of the fourth Transformer coding block, respectively.

[0129] The output of the first multilayer sensor is connected to the input of the second multilayer sensor;

[0130] The output of the second multilayer perceptron is used to output the image segmentation result.

[0131] Further, as shown in Figure 3(b), the internal structures of the first Transformer coding block, the second Transformer coding block, the third Transformer coding block, and the fourth Transformer coding block are the same; the first Transformer coding block includes:

[0132] The multi-head depthwise separable convolutional transposed attention (MDSCTA) layer, the depthwise separable convolution feedforward network (DSCFFN) and the overlap patch merging layer are connected in sequence.

[0133] The multi-head depthwise separable convolutional transpose attention mechanism layer MDSCTA includes:

[0134] The first normalized layer has its input as the input of the multi-head depthwise separable convolutional transpose attention mechanism layer MDSCTA; the output of the first normalized layer is connected to the input of three parallel branches, which include the fifth branch, the sixth branch, and the seventh branch.

[0135] The fifth branch includes: a 1*1 convolutional layer C1, a 3*3 depth-separable convolutional layer C2, and a 1*1 convolutional layer C3 connected in sequence;

[0136] The sixth branch includes: a 1*1 convolutional layer C4, a 3*3 depth-separable convolutional layer C5, and a 1*1 convolutional layer C6 connected in sequence;

[0137] The seventh branch includes: a 1*1 convolutional layer C7, a 3*3 depth-separable convolutional layer C8, and a 1*1 convolutional layer C9 connected in sequence;

[0138] The input terminals of convolutional layer C1, convolutional layer C4, and convolutional layer C7 are all connected to the output terminal of the first normalization layer.

[0139] The outputs of convolutional layer C3 and convolutional layer C6 are both connected to the input of the first multiplier.

[0140] The output of convolutional layer C9 and the output of the first multiplier are both connected to the input of the second multiplier;

[0141] The output of the second multiplier is connected to the input of the convolutional layer C10;

[0142] The output of the convolutional layer C10 and the input of the first normalization layer are both connected to the input of the first adder. The output of the first adder serves as the output of the multi-head depth-separable convolutional transpose attention mechanism layer MDSCTA.

[0143] Furthermore, the depthlessly separable convolutional feedforward network (DSCFFN) includes:

[0144] The second normalized layer, convolutional layer C10, convolutional layer C11, convolutional layer C12, convolutional layer C13, and second adder are connected in sequence.

[0145] The input of the second normalization layer is connected to the input of the second adder;

[0146] The input of the second normalization layer is the input of the depthwise separable convolutional feedforward network (DSCFFN).

[0147] The output of the second adder is the output of the depthwise separable convolutional feedforward network (DSCFFN).

[0148] Furthermore, the Overlap Patch Merging layer is an image representation method used to divide the input image into multiple overlapping small blocks, convert each small block into a vector representation, and obtain the final segmentation result by merging adjacent small blocks.

[0149] Furthermore, the specific working process of the Overlap Patch Merging layer is as follows:

[0150] (1) Divide the input image into multiple small blocks of the same size, which are allowed to overlap. Typically, the size of a small block is p×p, and the size of the overlapping part is s×s, where p>s;

[0151] (2) For each small block, features are extracted using a pre-trained convolutional neural network (such as ResNet). These features typically come from the last convolutional layer or a global average pooling layer of the network;

[0152] (3) For each small block, flatten the extracted features into a vector and use it as the vector representation of that small block;

[0153] (4) Piece together the vector representations of all the small blocks to form the vector representation of the entire image.

[0154] Furthermore, the encoder is used to extract feature representations of the input image;

[0155] Furthermore, the decoder is used to convert the feature maps extracted by the encoder into pixel-level semantic segmentation results;

[0156] Furthermore, the multi-head depthwise separable convolutional transpose attention mechanism layer MDSCTA generates an attention graph of the global context by applying self-attention across channels;

[0157] Multi-head Depthwise Separable Convolutional Transpose Attention Mechanism Layer (MDSCTA) Flowchart:

[0158]

[0159]

[0160] Among them, X and These are the input and output feature maps, respectively; the original size is... The tensor is reshaped to obtain a matrix. and W p It is a 1×1 pointwise convolution; α is a learnable scaling parameter used to control the scaling before applying the softmax function. and The size of the dot product; It is a transposed attention graph.

[0161] Furthermore, the depthwise separable convolutional feedforward network (DSCFFN) first filters the feature map obtained after expanding the feature channels through a 3×3 depthwise convolution, and then enriches the local context through a 1×1 pointwise convolution to achieve feature integration.

[0162] Given input tensor The DSCFFN process is defined as follows:

[0163]

[0164] Dsc(X)=φW d W p (LN(X))

[0165] Among them, X and These are the input and output feature maps, respectively. φ represents the GELU nonlinear transformation, W p It is a 1×1 pointwise convolution, W d It is a 3×3 depthwise convolution, LN is layer normalization, and Dsc(X) represents the feature map obtained after depthwise separable convolution.

[0166] In summary, DSCFFN enriches the contextual information of the feature map by controlling the information flow of each channel, allowing each channel to focus on details that complement those of other channels. Since the proposed DSCFFN performs more operations compared to conventional FN, we reduce the expansion ratio γ to achieve similar parameters and computational costs.

[0167] For example, the segmentation network RPIFormer based on the TransFormer structure is trained and validated on the second dataset to achieve automatic segmentation of M3 and MC on PRs, and the trained segmentation network model is obtained.

[0168] This invention proposes RPIFormer by improving the Transformer encoding block of SegFormer, and its overall architecture is shown in Figure 3(a). RPIFormer adopts an Encoder-Decoder structure. The Encoder is composed of stacked Transformer encoding blocks, and the Decoder adopts a U structure and adds a context aggregation module. It achieves efficient end-to-end image segmentation through positional encoding and multi-head self-attention.

[0169] This invention introduces a multi-head depthwise separable convolutional transposed attention (MDSCTA) mechanism into the Transformer coding block of the network, the structure of which is shown in Figure 3(b).

[0170] The Multi-Head Depthwise Separable Convolutional Transposed Attention (MDSCTA) layer generates a global context attention map by applying self-attention across channels. It first projects the input tensor into queries, keys, and values, then enriches the local context sequentially through 1×1 convolutions, 3×3 depthwise separable convolutions, and 1×1 pointwise convolutions. The initial 1×1 convolution is used for cross-channel context aggregation. In the Depthwise Separable Convolutional (DSC) module, because the 3×3 depthwise convolution only filters each channel, and the pointwise convolution only integrates features, the number of convolution kernels can be significantly reduced, thus reducing the number of parameters and computational cost. This is highly beneficial for resource-constrained devices and scenarios, improving computational efficiency, reducing memory usage, and accelerating model inference and training. Similar to traditional multi-head self-attention, we divide the number of channels into "heads" and then learn individual attention maps in parallel.

[0171] Pointwise convolution is an operation in convolutional neural networks, also known as 1×1 convolution. It can be used to change the number of channels in a feature map or to perform information exchange between channels. Specifically, pointwise convolution uses a 1×1 kernel that only performs a weighted sum operation on the channels of each pixel, without changing the pixel position or the spatial relationships between pixels. Therefore, pointwise convolution can be seen as an alternative to fully connected layers, but it preserves spatial structure information, making it more suitable for image processing tasks.

[0172] This invention introduces a Depthwise Separable Convolution Feedforward Network (DSCFFN) into the Transformer encoding block of the network, the structure of which is shown in Figure 3(b). The DSCFFN body employs two 1×1 convolutional layers, one of which is used to expand the feature channels (typically multiplied by a factor γ = 4), and the other is used to reduce the number of channels to the dimension of the original input.

[0173] During training, the input image resolution was 512*512, the Adamw optimizer was used, the initial learning rate was 0.0001, the L2 decay coefficient was 0.01, the cosine annealing strategy was used, the batch size was 16, Dice loss and Focal loss were used, and the training was conducted for 200 epochs. Figure 4 , Figure 5 The results of RPIFormer and SegFormer on the test set for M3 and MC segmentation are shown. The evaluation metrics used are Dice Coefficient, Intersection over Union (IoU), Precision, and Recall, and their calculation formulas are as follows:

[0174]

[0175]

[0176]

[0177]

[0178] In the formula, TP, FP, TN, and FN represent the number of correctly predicted positive sample pixels, the number of incorrectly predicted negative sample pixels, the number of correctly predicted negative sample pixels, and the number of incorrectly predicted positive sample pixels, respectively.

[0179] Furthermore, the training process of the trained image enhancement model includes:

[0180] An image enhancement model is constructed, which is implemented using a CycleGAN generative adversarial network.

[0181] A third training set is constructed by inputting newly collected panoramic X-ray images into the trained region of interest detection model, outputting regions of interest containing the third molar and mandibular canal, and annotating the regions of interest of the third molar and mandibular canal; data augmentation processing is then performed on the third dataset.

[0182] The augmented second and third training sets are input together into the image augmentation model to train the model. Training is stopped when the model's loss function value no longer decreases, or when the training exceeds the set number of iterations, resulting in the trained image augmentation model.

[0183] For example, PRs from other data sources (images acquired from different models of equipment in other hospitals, with varying image quality) are processed to form a third dataset, which is then divided.

[0184] The performance of the proposed RPIFormer was further evaluated using 332 images from other data sources (images collected from different models of equipment in other hospitals, with varying image quality). First, 255 eligible ROIs were obtained using the same processing method as the PRs collected from hospitals. These were then labeled by clinicians and finally divided into training and testing sets in a 5:2 ratio to obtain the D3 dataset.

[0185] Furthermore, the CycleGAN recurrent generative adversarial network has the following network structure:

[0186] The first generator G(AB) and the second generator G(BA) are connected in sequence; G(AB) is responsible for converting the source domain image style into the target domain image style, and G(BA) is responsible for restoring the target domain image style back to the source domain image style.

[0187] The output of the first generator is connected to the input of the first discriminator;

[0188] The output of the second generator is connected to the input of the second discriminator.

[0189] Furthermore, the CycleGAN generative adversarial network is used to enhance images from other data sources to further improve the actual test performance of the segmentation model RPIFormer proposed in this invention.

[0190] It should be understood that the generative adversarial network CycleGAN is used to train and test on the second and third datasets to achieve image augmentation on the third dataset, resulting in a trained image augmentation network model.

[0191] CycleGAN is a generative adversarial network that improves image quality by learning a mapping from low-quality to high-quality images. Its architecture is as follows: Figure 6 As shown, it consists of two GAN networks, left and right. G(AB) is responsible for converting the style of the source domain image into the style of the target domain image, and G(BA) is responsible for restoring the style of the target domain image back to the style of the source domain image. This design can effectively prevent the generator G(AB) from arbitrarily generating images in the style of the target domain and passing them off as genuine.

[0192] During training, the generator and decision generator are continuously updated based on GAN loss and Cycle-consistency loss, thereby achieving style transfer of images from the source domain to the target domain. In the specific training settings, the training sets in the second and third datasets are used as the target domain dataset and the source domain dataset, respectively. The input image resolution is 256*256, the batch size is 8, and the iteration is 100 epochs.

[0193] Patient PR images were randomly selected, ensuring that M3 was present on at least one side of each PR. To test the clinical performance of this invention, 50 patient PRs containing M3 were randomly selected from hospital samples for testing. These PRs were not used in the training and testing of the network at any stage of this invention. Figures 7(a) and 7(b) show examples of the results of the segmentation test performed on randomly selected patients in clinical settings.

[0194] Example 2

[0195] This embodiment provides a segmentation system for the third molar and mandibular canal;

[0196] The segmentation system of the third molar and mandibular canal includes:

[0197] The acquisition module is configured to acquire panoramic X-ray images of the teeth to be segmented.

[0198] The region detection module is configured to: input the panoramic X-ray image of the teeth to be segmented into the trained region of interest detection model, and output the region of interest containing the third molar and the mandibular canal;

[0199] The enhancement module is configured to input the region of interest containing the third molar and mandibular canal into the trained image enhancement model and output the enhanced region of interest.

[0200] The segmentation module is configured to input the enhanced region of interest into the trained image segmentation model and output the segmented images of the third molar and mandibular canal.

[0201] It should be noted that the acquisition module, region detection module, enhancement module, and segmentation module described above correspond to steps S101 to S104 in Embodiment 1. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in Embodiment 1. It should also be noted that these modules, as part of the system, can be executed in a computer system, such as a set of computer-executable instructions.

[0202] The descriptions of each embodiment in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0203] The proposed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of modules described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed.

[0204] Example 3

[0205] This embodiment also provides an electronic device, including: one or more processors, one or more memories, and one or more computer programs; wherein, the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory to cause the electronic device to perform the method described in Embodiment 1.

[0206] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0207] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0208] In the implementation process, each step of the above method can be completed by the integrated logic circuits in the processor hardware or by software instructions.

[0209] The method in Embodiment 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0210] Those skilled in the art will recognize that the units and algorithm steps described in connection with the various examples of this embodiment can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0211] Example 4

[0212] This embodiment also provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, complete the method described in Embodiment 1.

[0213] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for dividing the third molar and mandibular canal, characterized by: include: Obtain panoramic X-ray images of the teeth to be segmented; The panoramic X-ray image of the teeth to be segmented is input into the trained region of interest detection model, and the output is a region of interest containing the third molar and the mandibular canal. The region of interest containing the third molar and mandibular canal is input into the trained image augmentation model, and the augmented region of interest is output. The training process of the trained image segmentation model includes: constructing an image segmentation model, wherein the image segmentation model is implemented using the segmentation network RPIFormer; The segmentation network RPIFormer has a network structure including: an encoder and a decoder connected to each other; the encoder includes: an Overlap Patch Embeddings, a first Transformer encoding block, a second Transformer encoding block, a third Transformer encoding block and a fourth Transformer encoding block connected in sequence; The decoder includes: a first multilayer perceptron layer and a second multilayer perceptron layer; the input of the first multilayer perceptron layer is connected to the output of the first Transformer coding block, the output of the second Transformer coding block, the output of the third Transformer coding block and the output of the fourth Transformer coding block, respectively. The output of the first multilayer perceptron is connected to the input of the second multilayer perceptron; the output of the second multilayer perceptron is used to output the image segmentation result. The first Transformer coding block includes: a multi-head depthwise separable convolutional transpose attention mechanism layer (MDSCTA), a depthwise separable convolutional feedforward network (DSCFFN), and an overlap patch merging layer connected in sequence. The Multi-Head Depthically Separable Convolutional Transposed Attention Mechanism (MDSCTA) layer generates an attention graph of the global context by applying self-attention across channels. Multi-head Depthwise Separable Convolutional Transpose Attention Mechanism Layer (MDSCTA) Flowchart: in, and These are the input and output feature maps, respectively; the original size is... The tensor is reshaped to obtain a matrix. , and ; It is a 1×1 pointwise convolution; It is a learnable scaling parameter used to control the scaling before applying the softmax function. and The size of the dot product; It is a transposed attention graph; The enhanced region of interest is input into the trained image segmentation model, which outputs the segmented images of the third molar and the mandibular canal.

2. The method for separating the third molar and mandibular canal as described in claim 1, characterized in that, The process of obtaining the trained region of interest detection model includes: Construct the first dataset, which consists of panoramic X-ray images of the third molar and the region of interest of the mandibular canal that have been annotated; The first dataset is input into the object detection network YOLOv7-tiny. The object detection network YOLOv7-tiny is trained. When the total loss function value no longer decreases, or the number of iterations exceeds the set number, the training is stopped, and the trained object detection network YOLOv7-tiny is obtained. The trained object detection network YOLOv7-tiny is used as the trained region of interest detection model. The total loss function is the sum of the rectangular box loss function and the confidence loss function.

3. The method for separating the third molar and mandibular canal as described in claim 1, characterized in that, The training process of the trained image segmentation model includes: Construct a second training set; the second training set is obtained by inputting newly collected panoramic X-ray images into the trained region of interest detection model, outputting regions of interest containing the third molar and mandibular canal, and annotating the regions of interest of the third molar and mandibular canal; perform data augmentation processing on the second training set; The augmented second training set is input into the image segmentation model to train the model. Training is stopped when the model's loss function value no longer decreases, or when the training exceeds the set number of iterations, and the trained image segmentation model is obtained.

4. The method for separating the third molar and mandibular canal as described in claim 1, characterized in that, The depthwise separable convolutional feedforward network (DSCFFN) first filters the feature map obtained after expanding the feature channels through a 3×3 depthwise convolution, and then enriches the local context through a 1×1 pointwise convolution to achieve feature integration. Given input tensor The DSCFFN process is defined as follows: in, and These are the input and output feature maps, respectively. , W represents the GELU nonlinear transformation. p It is a 1x1 pointwise convolution, W d It is a 3×3 depth-direction convolution, and LN is layer normalization. This represents the feature map obtained after depthwise separable convolution.

5. The method for separating the third molar and mandibular canal as described in claim 1, characterized in that, The training process of the trained image enhancement model includes: An image enhancement model is constructed, which is implemented using a CycleGAN generative adversarial network. A third training set is constructed by inputting newly collected panoramic X-ray images into the trained region of interest detection model, outputting regions of interest containing the third molar and mandibular canal, and annotating the regions of interest of the third molar and mandibular canal; data augmentation processing is then performed on the third dataset. The augmented second and third training sets are input together into the image augmentation model to train the model. Training is stopped when the model's loss function value no longer decreases, or when the training exceeds the set number of iterations, resulting in the trained image augmentation model.

6. A segmentation system for the third molar and mandibular canal, employing the segmentation method for the third molar and mandibular canal as described in any one of claims 1-5, characterized in that, include: The acquisition module is configured to acquire panoramic X-ray images of the teeth to be segmented. The region detection module is configured to: input the panoramic X-ray image of the teeth to be segmented into the trained region of interest detection model, and output the region of interest containing the third molar and the mandibular canal; The enhancement module is configured to input the region of interest containing the third molar and mandibular canal into the trained image enhancement model and output the enhanced region of interest. The segmentation module is configured to input the enhanced region of interest into the trained image segmentation model and output the segmented images of the third molar and mandibular canal.

7. An electronic device, characterized in that it comprises: Memory is used to store computer-readable instructions in a non-transitory manner. as well as Processor, for executing the computer-readable instructions, When the computer-readable instructions are executed by the processor, they perform the method described in any one of claims 1-5.

8. A storage medium characterized by being non-transitory. The system stores computer-readable instructions, wherein, when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1-5 are executed.

Images