A Medical Image Segmentation Method Based on Semi-Supervised Learning

Abstract

This invention discloses a semi-supervised learning method for medical image segmentation: Step 1: Pre-training; First, a fine-tuning network is pre-trained on a first set of original medical images to obtain a trained fine-tuning network; Step 2 (fine-tuning): The encoder in the trained fine-tuning network is combined with a randomly initialized decoder to obtain a segmentation network; A second set of original medical images is manually labeled to obtain a segmentation dataset; The segmentation network is trained using the segmentation dataset to obtain a trained segmentation network; Step 3: The trained segmentation network is used as a teacher model for self-training to obtain a trained student model; Step 4: The image to be segmented is input into the trained student model to obtain the segmentation result. This invention utilizes readily available and abundant raw data, solving the technical problem of low segmentation accuracy caused by insufficient labeled data for training deep learning segmentation models.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image segmentation and relates to a medical image segmentation method based on semi-supervised learning. BACKGROUND

[0002] The application relates to two parts of medical image segmentation and semi-supervised learning algorithm:

[0003] 1) Medical image segmentation

[0004] Recent researches on medical image segmentation mainly focus on the improvement of segmentation models, and CS-Net, CE-Net and MDACN all focus on the design of multi-scale information fusion modules to improve the performance of medical image segmentation. However, the problem of insufficient medical data sets still restricts the further improvement of model capacity, and under this background, the image segmentation method using unlabeled data has not been proposed.

[0005] CS-Net is a general unified curve structure segmentation network suitable for different medical imaging methods: optical coherence tomography angiography (OCT-A), color fundus image and corneal confocal microscopic image (CCM). The network adds a self-attention mechanism in the encoder and decoder, replacing the convolutional neural network based on U-net. Two types of attention modules, spatial attention and channel attention, are used to further adaptively combine local features with their global dependencies.

[0006] CE-Net is a context encoder network that can capture more high-level information and preserve spatial information for two-dimensional medical image segmentation. CE-Net mainly includes three main components: a feature encoder module, a context extractor and a feature decoder module. The context extraction module is composed of a newly proposed dense atrous convolution (DAC) block and a residual multi-kernel pooling (RMP) block.

[0007] MDACN is a new multi-discriminator adversarial convolutional network, in which the generator and the two discriminators both emphasize multi-scale feature representation. The generator is a U-shaped fully convolutional network with multi-scale splitting and concatenation blocks, and the two discriminators have different effective receptive fields and are sensitive to features of different scales.

[0008] 2) Semi-supervised learning algorithm

[0009] In real medical scenarios, unlabeled data is relatively easy to obtain, while labeled data is usually difficult to collect, and annotation is time-consuming and laborious. In this case, semi-supervised learning is more suitable for application in real scenarios, and recently it has become a new direction in the field of deep learning. This method only needs a small amount of labeled samples and a large amount of unlabeled samples as training data.

[0010] Mean Teacher is a method of averaging model weights, rather than averaging prediction label weights. Additionally, Mean Teacher improves the accuracy of the test and makes the training less label-intensive than temporal ensembling.

[0011] Uncertainty-Aware Self-Ensembling is a new uncertainty-aware semi-supervised framework for left atrium segmentation from 3D MR images. This method can effectively utilize unlabeled data by encouraging consistent predictions on the same input under different perturbations. Specifically, the framework consists of a student model and a teacher model, and the student model learns the teacher model by minimizing the segmentation loss and consistency loss of the teacher model. This method designs a new uncertainty-aware scheme, which enables the student model to gradually learn meaningful and reliable targets using uncertainty information.

[0012] Cross Pseudo Supervision is a new consistency regularization method called Cross Pseudo Supervision. This method uses different initialization perturbations for the same input image, so that the two segmentation networks have consistency. The pseudo one-hot label map output by one perturbation segmentation network is used to supervise the other segmentation network with standard cross-entropy loss, and vice versa.

[0013] Deep Adversarial Networks is a new deep adversarial network biomedical image segmentation model, aiming to obtain consistent good segmentation results on annotated and unannotated images. The model consists of two networks: (1) a segmentation network (SN) for segmentation; (2) an evaluation network (EN) to evaluate the segmentation quality. During training, EN is encouraged to distinguish the segmentation results of unannotated images from the segmentation results of annotated images (give different scores), and SN is encouraged to generate segmentation results of unannotated images so that EN cannot distinguish unannotated images from annotated images. Through an iterative adversarial training process, SN can be trained to produce more and more accurate segmentation of unannotated samples because EN constantly "criticizes" the segmentation results of unannotated images.

[0014] In summary, there has been some progress in the research on medical image segmentation problems at home and abroad, but there is a technical problem of insufficient labeled data for training deep learning segmentation models, which restricts the further improvement of model performance. However, it takes professional doctors ten minutes to half an hour to draw an accurate labeled data, and there are strong subjectivity and other shortcomings, so it is very difficult to solve the problem of insufficient labeled data by relying on doctor labeling. SUMMARY

[0015] The present application aims to provide a medical image segmentation method based on semi-supervised learning to solve the technical problem of low segmentation accuracy caused by insufficient labeled data for training deep learning segmentation model.

[0016] To achieve the above-mentioned purpose, the present application adopts the following technical solutions to solve it:

[0017] A medical image segmentation method based on semi-supervised learning comprises the following steps:

[0018] Step one: pre-training; comprising the following sub-steps:

[0019] Step 1: Take the first original medical image set as the repair data set, randomly cover each image in the repair data set and input it into the rough repair network for training to obtain the rough repair feature map;

[0020] Step 2: Input the rough repair feature map obtained in step 1 into the encoder of the fine repair network, and obtain the fine repair feature map and the trained fine repair network after training the fine repair network;

[0021] Step 3: Input the fine repair feature map and the position data of randomly covering the images in the repair data set in step 1 into the discriminator to obtain the output result of the discriminator.

[0022] Step 4: Input the output result of the discriminator into the fine repair network in reverse to obtain the trained fine repair network;

[0023] Step two: fine-tuning: combine the encoder in the trained fine repair network obtained in step one with the randomly initialized decoder to obtain the segmentation network; obtain the segmentation data set by manually labeling the second original medical image set; input the segmentation data set into the segmentation network to output the segmentation result, and input the segmentation result and the label in the segmentation data set into the segmentation network in reverse to train, to obtain the trained segmentation network;

[0024] Step three: self-training using semi-supervised learning algorithm: take the trained segmentation network obtained in step two as the teacher model, generate pseudo-labels for the first original medical image set in step one using the teacher model, and combine the pseudo-labels with the segmentation data set to obtain a new training set; perform data augmentation on the new training set and input it into the student model for training to obtain the trained student model.

[0025] Step four: input the medical image to be segmented into the trained student model in step three to obtain the image segmentation result.

[0026] Further, in step one, the rough repair network adopts a gated convolution.

[0027] Further, in the step one, the fine restoration network uses a typical deep learning segmentation network.

[0028] Further, in the step two, when training the segmentation network, take the weighted sum L = al dice +(1- a)l BCE as the final loss function, where a = 0.5.

[0029] Further, in the step three, the student model adopts a typical deep learning segmentation network.

[0030] Further, in the step three, the semi-supervised learning algorithm adopts ST++ algorithm, Mean Teacher or Noisy Student.

[0031] Compared with the prior art, the present application has the following technical effects:

[0032] 1) Since it is relatively easy to obtain the original data of medical images, the method of the present application uses easily obtained and large amounts of original data to improve the performance of the model, solving the technical problem of low segmentation accuracy caused by insufficient labeled data for training deep learning segmentation models in the prior art. At the same time, the doctor's time is liberated.

[0033] 2) It can effectively utilize unlabeled data to improve the performance of the model. The more unlabeled data we have, the better the performance of our model, until a performance improvement boundary is reached.

[0034] 3) The requirement for computing resources is not high, only a normal GPU graphics card is needed.

[0035] 4) The coupling between pre-training, self-training and segmentation network is not strong, one of which can be replaced by a more efficient method to further improve the performance.

[0036] 5) The method of the present application is versatile and can be used on other medical image datasets. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is a flowchart of the method of the present application;

[0038] Figure 2 is a schematic diagram of the pre-training model in the method of the present application.

[0039] The present application is further explained in conjunction with the accompanying drawings and specific embodiments. DETAILED DESCRIPTION

[0040] The medical image segmentation method based on semi-supervised learning given by the present application, like Figure 1As shown, comprising the following steps:

[0041] Step one: pre-training; as Figure 2 As shown, comprising the following sub-steps:

[0042] Step 1: the first original medical image set (no label data) as the repair data set, each image in the repair data set is randomly covered and input into the rough repair network for training, and the rough repair feature map is obtained; preferably, the rough repair network adopts the gated convolution.

[0043] This step trains the rough repair network to predict the low-frequency information (such as the color and texture of the background) of the covered pixels;

[0044] Step 2: input the rough repair feature map obtained in step 1 into the encoder of the fine repair network, and obtain the fine repair feature map and the trained fine repair network after training the fine repair network; preferably, the fine repair network uses typical deep learning segmentation network such as U-net, Unet++, CE-Net or CS-Net.

[0045] The fine repair network is used to reconstruct more detailed information of the missing pixels on the basis of the rough repair network.

[0046] Step 3: input the fine repair feature map and the position data of the randomly covered images in the repair data set in step 1 into the discriminator to obtain the output result of the discriminator.

[0047] The discriminator is used to judge whether the predicted image is reasonable, and the loss of the confrontation is propagated in the opposite direction.

[0048] Step 4: input the output result of the discriminator into the fine repair network (i.e. input the decoder of the fine repair network) to obtain the trained fine repair network.

[0049] From the above, through step one, the network model learns about the information in the medical image before formal training, so that the model has a good initial value.

[0050] Step two: fine-tuning: combine the encoder of the trained fine repair network obtained in step one with the randomly initialized decoder to obtain the segmentation network; obtain the segmentation data set by manually labeling the second original medical image set; input the segmentation data set into the segmentation network, output the segmentation result, and input the segmentation result and the label in the segmentation data set into the segmentation network in the opposite direction to train, and obtain the trained segmentation network.

[0051] Step three: self-training using a semi-supervised learning algorithm: use the trained segmentation network obtained in step two as a teacher model, generate pseudo-labels for the first set of original medical images in step one using the teacher model, and combine the pseudo-labels with the segmentation dataset to obtain a new training set; perform data augmentation on the new training set and input it into the student model for training to obtain a trained student model.

[0052] Preferably, the student model uses a typical deep learning segmentation network such as U-net, Unet++, CE-Net, or CS-Net. Data augmentation is performed using strong data augmentation. The semi-supervised learning algorithm uses the ST++ algorithm, Mean Teacher, or NoisyStudent.

[0053] In this step, data augmentation on the new dataset can help the student model surpass the teacher model.

[0054] Step four: input the medical image to be segmented into the trained student model in step three to obtain the image segmentation result.

[0055] Embodiment:

[0056] 1) Algorithm settings

[0057] The algorithm of the present application is implemented using PyTorch. The coarse repair network uses gated convolution. The fine repair network is flexible and can use U-net and Unet++. The self-training stage uses the ST++ algorithm. It is run on an NVIDIA GeForce GTX1080Ti GPU.

[0058] 2) Model training

[0059] As shown in Figure 1 , taking a common corneal confocal microscopic image in a medical image as an example, the corneal confocal microscopic image and the corresponding small-scale artificial annotation information are used as input. In the pre-training stage, masks of random shapes are used, with a mean number of 20. 1500 unannotated data and 50 annotated data are used. When training the segmentation network, take the weighted sum of the Dice loss and the two-dimensional cross-entropy loss L = a l dice + (1-a) l BCE as the final loss function, where a = 0.5. In the pre-training process, the batch size is 4 and the training is performed for 200 rounds. In the training process, the SGD optimizer is used with a batch size of 4 and the training is performed for 1000 rounds.

[0060] 3) Model usage

[0061] After the model is trained, the corneal confocal microscopic image to be segmented is input into the model for testing, and the image segmentation result can be obtained.

[0062] In order to prove the feasibility and effectiveness of the method of the present application, the present application carries out the following experiments:

[0063] 1. Verification experiment of effectiveness of image reconstruction pre-training

[0064] In order to verify the effectiveness of introducing pre-training, an ablation experiment is used to verify each module. In the experiment, the first original image dataset is part of the CORN dataset, which is 1500 images (CORN: CORneal Nerve Database - Medical Imaging / Ophthalmology Imaging Team - imed.nimte.ac.cn); the second original image dataset is part of the CORN dataset after removing the first original dataset, which is 50 images, and is pixel-level labeled.

[0065] Table 1 shows the performance of the model on the CCM30 dataset after adding the coarse repair network and the discriminator, respectively, where UNet is used for comparison, and the pros and cons of the pre-training method using super-resolution reconstruction as the proxy task are compared.

[0066] Table 1

[0067]

[0068] From the experimental results, it can be seen that after using pre-training, even if the pre-training is missing some modules, the segmentation performance is still better than that without pre-training. At the same time, after using all the modules proposed by us, the pre-training effect is the best, which is improved by 2 percentage points compared with the baseline. This is because the information about the image is learned in advance during the pre-training process, and there is a good initial value, which is conducive to model training, and ultimately better performance is obtained.

[0069] 2. Verification experiment of effectiveness of self-training

[0070] In order to verify the effectiveness of introducing self-training, an ablation experiment is used to verify each module. Table 2 shows the performance of the model on the CCM30 (http: / / bioimlab.dei.unipd.it / Corneal%20Nerve%20Tortuosity%20Data%20Set.htm) after using the ST method and the ST++ method as the self-training method, respectively, where U-net is used for comparison.

[0071] Table 2

[0072]

[0073] Through the experimental results, it can be seen that after using self-training, the performance is improved compared with the baseline. After using both pre-training and ST++ methods, the performance is improved by 6 percentage points compared with the baseline. This is because self-training corrects the model classification boundary through unlabeled data, making it meet the clustering assumption and smoothing assumption, that is, the classification boundary is more reasonable, thus achieving the effect of improving the segmentation performance.

Claims

1. A method for medical image segmentation based on semi-supervised learning, characterized in that, The method comprises the following steps: Step one: pre-training; comprising the following sub-steps: Step 1: taking the first original medical image set as a repair data set, randomly covering each image in the repair data set and inputting the rough repair network for training to obtain a rough repair feature map; Step 2: inputting the rough repair feature map obtained in step 1 into the encoder of the fine repair network, obtaining a fine repair feature map and a trained fine repair network after training of the fine repair network; Step 3: inputting the fine repair feature map and the position data of randomly covering the images in the repair data set in step 1 into the discriminator to obtain the output result of the discriminator; Step 4: inputting the output result of the discriminator into the fine repair network in reverse to obtain the trained fine repair network; Step two: Fine-tuning: combining the encoder in the trained fine repair network obtained in step one with a randomly initialized decoder to obtain a segmentation network; obtaining a segmentation data set by manually labeling the second original medical image set; inputting the segmentation data set into the segmentation network to output a segmentation result, and inputting the segmentation result and the label in the segmentation data set into the segmentation network in reverse to train, thereby obtaining a trained segmentation network; Step three: self-training by using a semi-supervised learning algorithm: taking the trained segmentation network obtained in step two as a teacher model, generating pseudo-labels by using the teacher model on the first original medical image set in step one, and combining the pseudo-labels with the segmentation data set to obtain a new training set; taking a randomly initialized segmentation network as a student model, performing data augmentation on the new training set and inputting the new training set into the student model for training to obtain a trained student model; Step four: inputting a medical image to be segmented into the trained student model in step three to obtain an image segmentation result; In the step one, the rough repair network adopts a gated convolution; In the step one, the fine repair network uses a deep learning segmentation network. 2.The method of claim 1, wherein, In the second step, when training the segmentation network, take the weighted sum of Dice loss and two-dimensional cross-entropy loss as the final loss function L = a +(1−a) , where a = 0.

5. 3.The medical image segmentation method based on semi-supervised learning according to claim 1, wherein, In the step three, the student model uses a deep learning segmentation network. 4.The method of claim 1, wherein, In the step three, the semi-supervised learning algorithm uses an ST++ algorithm, a Mean Teacher or a Noisy Student.

Images