Ear canal CT image segmentation method based on improved nn-UNet

Abstract

The embodiment of the invention discloses an ear canal CT image segmentation method based on improved nn-UNet, and the method comprises the steps: collecting and marking the ear canal CT data of a plurality of patients, and making a data set; preprocessing the data; training on a training set by using a model improved based on nn-UNet, dividing a data set for multiple times by using a five-fold cross validation method, then training and verifying each fold, and storing a training model after training is completed; reasoning and testing the trained model on the test set data which are not seen; post-processing is carried out, and an output result of the model is optimized; and formulating an evaluation scheme to evaluate the prediction result. According to the method, the segmentation efficiency and accuracy can be greatly improved, and subjective deviation and human resource waste possibly generated in the manual segmentation process can be effectively avoided.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an ear canal CT image segmentation method based on an improved nn-UNet. Background Art

[0002] CT image segmentation is a research hotspot in the field of computer vision and medical image processing. With the continuous development of medical imaging technology, CT technology is increasingly used in the diagnosis of ear diseases. The complex and delicate anatomical structure of the ear canal places higher demands on the segmentation and modeling of ear canal CT images. Traditional CT image segmentation methods usually require doctors or technicians to perform manually. Due to the small and complex structure of the ear canal and the high requirements for segmentation accuracy, this method is time-consuming, laborious, and easily affected by subjective factors, making it difficult to ensure the consistency and accuracy of the segmentation results. Summary of the invention

[0003] The purpose of the present invention is to solve the problems raised in the above-mentioned background technology. The present invention provides an ear canal CT image segmentation method based on improved nn-UNet, which can greatly improve the efficiency and accuracy of segmentation and effectively avoid subjective bias and waste of human resources that may occur in the manual segmentation process.

[0004] In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions:

[0005] The ear canal CT image segmentation method based on the improved nn-UNet includes:

[0006] S1: Collect and annotate the ear canal CT data of multiple patients to create a dataset;

[0007] S2: preprocess the data;

[0008] S3: Use the improved model based on nn-UNet to train on the training set, use the five-fold cross-validation method to divide the data set multiple times, then train and validate each fold, and save the training model after training is completed;

[0009] S4: Perform inference tests on the trained model on unseen test set data;

[0010] S5: perform post-processing to optimize the output results of the model;

[0011] S6: Visualize the output results and evaluate the segmentation results.

[0012] Beneficial effects:

[0013] After the model is trained and evaluated using the data set, the present invention only needs to input the CT image, and the model can automatically segment the desired ear canal structure; the present invention greatly simplifies the segmentation process, avoids the subjective influence and inefficiency problems existing in manual medical image annotation, reduces the difficulty of manually segmenting the tiny structures of the ear canal in the field of medical images, and greatly improves the segmentation efficiency. The present invention helps to get rid of personnel restrictions and saves a lot of manpower and material resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 This is a flow chart of an ear canal CT image segmentation method based on an improved nn-UNet according to an embodiment of the present invention;

[0016] Figure 2 It is a structural schematic diagram of a segmentation model according to an embodiment of the present invention;

[0017] Figure 3 The figure is a schematic diagram of the structure of an improved Transformer module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0019] As shown in the figure, an embodiment of the present invention discloses an ear canal CT image segmentation method based on an improved nn-UNet, comprising:

[0020] S1: Collect and annotate the ear canal CT data of multiple patients to create a dataset;

[0021] S2: preprocess the data;

[0022] S3: Use the improved model based on nn-UNet to train on the training set, use the five-fold cross-validation method to divide the data set multiple times, select DSC and IoU as the evaluation indicators of segmentation performance, and then train and verify each fold. After the training is completed, save the training model;

[0023] S4: Perform inference tests on the trained model on unseen test set data;

[0024] S5: perform post-processing to optimize the output results of the model;

[0025] S6: Visualize the output results and evaluate the segmentation effect.

[0026] In step S1, the ear canal CT data of 50 patients were collected, and the external auditory canal, internal auditory canal, tympanic membrane, round window membrane, membranous labyrinth, bony labyrinth, facial nerve, and auditory ossicles were manually labeled using 3D-Slicer. After the labeling was completed, the images and labels were generated into NIFTI files and placed in the images and labels folders respectively to construct the dataset.

[0027] The dataset is divided into two parts: training set and test set, with the training set accounting for 90% and the test set accounting for 10%. A Python script is written to batch process the image names into the corresponding format, with _0000 added to the end of the image name to distinguish the modality. _0000 indicates CT image, and the image and label are consistent in the file name before _0000.nii.gz.

[0028] Write a Python script to create the dataset.json file, which contains a description of the dataset, modal information, and definitions of multiple labels in the label file.

[0029] In step S2, all data are cropped to the non-zero value area to reduce the image size, avoid useless calculations, and improve calculation efficiency, and then all data are resampled.

[0030] In step S2, the data is normalized. The specific steps are:

[0031] The grayscale values ​​of CT images are unified by intensity normalization, reducing the differences between different scanners, scanning settings and patients, thereby improving the stability and accuracy of the model. The target space size for resampling is determined, and then the target size of each image is determined based on the target space. The image is resampled using the trilinear interpolation method, and the voxel spacing is adjusted to a target spacing of 1×1×1, so that images between different scanners and data sets have the same spatial resolution, which can improve the versatility of the model, reduce model deviation, and simplify the next image processing operations (such as cropping, rotation, etc.).

[0032] The mean and variance of the pixel intensity within the segmentation mask of the training dataset are zscore normalized. Where x is the original data value, μ is the mean of the dataset, σ is the standard deviation of the dataset, and these intensity values ​​are clipped to [0.5, 0.95].

[0033] In step S2, the data is enhanced by rotation, the x-axis rotation range is [-180, 180], Gaussian noise is added to the image to perform Gaussian blur processing on the image, and the brightness and contrast are adjusted;

[0034] When using Gaussian noise transformation, the probability of applying this transformation to each sample is set to 0.1. When using Gaussian blur transformation, the standard deviation of the Gaussian kernel ranges from 0.5-1.0, the probability of applying Gaussian blur to each sample is 20%, and the probability of applying Gaussian blur to each channel is 50%. Use gamma transformation to control the brightness and contrast of the image. The range of brightness transformation is set between 0.75-1.25. Each sample has a 15% probability of applying brightness adjustment transformation, and the probability of contrast transformation is set to 15%. Through the above steps, the diversity of samples is increased.

[0035] The improved model based on nn-UNet includes the nn-UNet main network framework and the embedded Transformer module to obtain fused local and global features. The improvement method is as follows:

[0036] First, the encoder of nnUNet is used to effectively generate a series of feature maps by extracting multi-scale spatial features. In this way, the rich local 3D context features from the encoder are gradually and effectively embedded into high-level features, which are then fed into the Transformer module to further learn long-range dependencies and supplement the local context features with global receptive fields. After that, the splicing operation on the two corresponding branches of upsampling and skip connections is repeatedly applied, followed by convolutional layers to gradually produce high-resolution segmentation maps. Since shallow features tend to generate more detailed spatial information, while deep features extract more discriminative semantic information, the original nn-UNet architecture is modified by using a deeper encoder. This modification aims to increase the network capacity to better model greater data diversity. Six convolutional layers of the same resolution and five convolution operations with a stride of 2 are used in the encoder for downsampling to extract more discriminative semantic information. The decoder adopts the same structure, including one splicing operation and two convolution operations. It should be noted that the splicing operation is used to splice the upsampling of the transposed convolution operation with the skip features in the encoder branch at the same resolution level. More specifically, batch normalization and leaky ReLU are applied after each convolution operation; the number of channels starts from 64 at the initial level and doubles at each downsampling.

[0037] Compared with the original nn-UNet model, the improvement of the present invention is mainly focused on the construction of the model. Taking nn-UNet as the main body, the Transformer module is integrated between the encoder and the decoder, and the traditional Tokenization method is replaced by the pooling layer module in the Transformer module. By using convolution and maximum pooling operations, the spatial information of the input is maintained and the size of the feature map is reduced. At the same time, the self-attention matrix is ​​used to construct long-range dependencies to enhance the performance of the Transformer in processing medical images.

[0038] Due to GPU memory limitations, it is not advisable to apply Transformer modules to high-resolution features. Therefore, three Transformer modules are fused into the three lowest resolution scales of the modified nnUNet, namely level 4, level 5, and level 6, while keeping the skip connections unchanged.

[0039] In step S3, a combination of Dice loss and cross entropy loss is used as the loss function of the network, L = L Dice +L CE ,The combined loss function can evaluate the prediction results more comprehensively and enhance the training effect of the model. where p i represents the predicted probability value, g i represents the true label, which is used to maximize the overlap between the predicted segmentation and the true segmentation. Where N is the sample size, p i is the predicted probability, g i is the true label, which is used to measure the difference between the predicted probability distribution and the true distribution in classification problems.

[0040] In step S4, data from at least three different manufacturers' CT devices are collected as a test set, and the CT in the test set is used to perform segmentation tests on the trained model to obtain prediction values. The specific steps are: read the test image and the corresponding segmentation label, perform the same preprocessing operations on the test image as the training data, including resizing and standardization, then load the model weight file, input the test image into the model, and obtain the model's prediction output.

[0041] In step S5, the probability map output by the model test is converted into a binary segmentation map for further analysis and processing. The threshold is determined, and the pixels with probability values ​​greater than the threshold are set as the foreground, and the pixels with probability values ​​less than the threshold are set as the background. Then, the main connected components are identified and retained through connected domain analysis, and small, isolated noise areas are removed, and then edge smoothing and hole filling are performed. Edge smoothing is specifically to smooth the boundaries of the segmentation results, reduce the jagged effect, and make the boundaries more natural; hole filling is specifically to fill the small holes in the segmentation results to make the foreground area more complete; based on the prior knowledge of the anatomical structure or region of interest (ROI), the segmented area is cropped or filtered. This series of methods ensures the accuracy and reliability of the segmentation results.

[0042] In step S6, the segmentation result, that is, the predicted label output by the model inference, is combined with the original image for visualization, and the segmented area is displayed on each slice of the original image.

[0043] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An ear canal CT image segmentation method based on improved nn-UNet, characterized in that: include: S1: Collect and annotate the ear canal CT data of multiple patients to create a dataset; S2: preprocess the data; S3: Use the improved model based on nn-UNet to train on the training set, use the five-fold cross-validation method to divide the training set data multiple times, then train and validate each fold, and save the training model after training is completed; S4: Perform inference tests on the trained model on unseen test set data; S5: perform post-processing to optimize the output results of the model; S6: Visualize the output results and evaluate the segmentation results.

2. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: In step S1, the ear canal CT data of 50 patients were collected, and the external auditory canal, internal auditory canal, tympanic membrane, round window membrane, membranous labyrinth, bony labyrinth, facial nerve, and auditory ossicles were manually labeled using 3D-Slicer. After the labeling was completed, the images and labels were generated into NIFTI files and placed in the images and labels folders respectively to construct the dataset.

3. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: The dataset is divided into two parts: training set and test set, with the training set accounting for 90% and the test set accounting for 10%. A Python script is written to batch process the image names into the corresponding format, with _0000 added to the end of the image name to distinguish the modality. _0000 indicates CT image, and the image and label are consistent in the file name before _0000.nii.gz. Write a Python script to create the dataset.json file of the training set data. The json file contains a description of the dataset, modal information, and definitions of multiple labels in the label file.

4. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: In step S2, all data are clipped to the non-zero value area, and then all data are preprocessed; The grayscale values ​​of CT images are unified by intensity normalization to reduce the differences between different scanners, scanning settings and patients, and the target space size of resampling is determined. Then, the target size of each image is determined based on the target space, and the image is resampled by applying the trilinear interpolation method to adjust the voxel spacing to the target spacing. The mean and variance of the pixel intensity within the segmentation mask of the training dataset are zscore normalized. Where x is the original data value, μ is the mean of the dataset, σ is the standard deviation of the dataset, and these intensity values ​​are clipped to [0.5, 0.95].

5. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: In step S3, the improved model based on nn-UNet includes the nn-UNet main network framework and the embedded Transformer module, and the improvement method is as follows: The encoder of nnUNet is used to extract multi-scale spatial features to generate a series of feature maps. The local 3D context features from the encoder are gradually embedded into high-level features and then fed into the Transformer module to learn long-distance dependencies and supplement local context features with global receptive fields. The original nn-UNet architecture is modified. Six convolutional layers with the same resolution and five convolution operations with a stride of 2 are used in the encoder for downsampling to extract more discriminative semantic information. The decoder adopts the same structure including a splicing operation and two convolution operations. The splicing operation splices the upsampling of the transposed convolution operation with the skipped features in the encoder branch at the same resolution level. Batch normalization and LeakyReLU are applied after each convolution operation. The number of channels starts from 64 at the initial level and doubles at each downsampling.

6. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: The three transformer modules are fused into the three lowest resolution scales of the modified nnUNet, namely level 4, level 5 and level 6, while keeping the skip connections unchanged.

7. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: The manually labeled results are taken as the true values, and DSC and IoU are calculated as evaluation indicators, where: Used to measure the similarity between two sample sets. It is the ratio of the intersection and union between the predicted result and the true label, where TP is a true positive example, FP is a false positive example, and FN is a false negative example; After each verification, the model’s performance indicators DSC and IoU were recorded to obtain five sets of results. The mean and standard deviation of all verification results were calculated. The mean was used as the overall performance evaluation of the model, and the standard deviation reflected the volatility of the model’s performance.

8. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: In step S4, the model is used to perform inference tests on the test set data, and the labels output after model inference are combined with the original image for visualization operations to show the image segmentation effect.

9. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: In step S5, the probability map output by the model test is converted into a binary segmentation map, the threshold is determined, and the pixels with probability values ​​greater than the threshold are set as the foreground, and the pixels with probability values ​​less than the threshold are set as the background. Then, the main connected components are identified and retained through connected domain analysis, and small, isolated noise areas are removed. Finally, the segmentation results are optimized and the accuracy is improved through edge smoothing and hole filling.

10. The ear canal CT image segmentation method based on improved nn-UNet according to claim 1, characterized in that: In step S2, the data is enhanced by rotation, the x-axis rotation range is [-180, 180], Gaussian noise is added to the image to perform Gaussian blur processing on the image, and the brightness and contrast are adjusted; When using Gaussian noise transformation, the probability of applying this transformation to each sample is set to 0.

1. When using Gaussian blur transformation, the standard deviation of the Gaussian kernel ranges from 0.5-1.0, the probability of applying Gaussian blur to each sample is 20%, and the probability of applying Gaussian blur to each channel is 50%. Use gamma transformation to control the brightness and contrast of the image. The range of brightness transformation is set between 0.75-1.

25. Each sample has a 15% probability of applying brightness adjustment transformation, and the probability of contrast transformation is set to 15%.

Images