Strong generalization method for fundus image segmentation based on semi-supervised learning

Abstract

The present invention provides a strongly generalized fundus image segmentation method based on semi-supervised learning. First, a labeled pair of segmentation main networks is preheated for training. Further, an unsupervised method is used to allow the two segmentation main networks to compare with each other after segmentation. Because the two segmentation networks segment the original fundus image and the perturbed image respectively without mutual communication, the original image and the perturbed image have structural consistency. Therefore, with this consistency criterion as a constraint, the loss error is calculated for the boundary map and entropy map of the two segmentation main networks each time, and it is continuously iterated until the difference between the segmentation results of the two is less than the specified iteration condition, and the structures of the two segmentation networks reach approximately the same level. Through such iterative training, the model's recognition effect on the segmentation area of ​​interest in the fundus image can be effectively improved, thereby obtaining a high-precision segmentation model for the segmentation of the fundus image area of ​​interest.

Description

Technical Field

[0001] The present invention belongs to the fields of computer vision technology and medical imaging technology, and particularly relates to a strongly generalized fundus image segmentation method based on semi-supervised learning. Background Art

[0002] Many systemic diseases, such as hypertension and diabetes, can cause fundus lesions, making fundus images a crucial diagnostic tool. Structural variation in retinal fundus images is a key indicator of certain diseases and crucial for clinical diagnosis. For example, the ratio of the optic cup (OC) to the optic disc (OD) can be used to detect glaucoma. Segmentation of microaneurysms (MA), hemorrhages (HE), soft exudates (SE), and hard exudates (EX) can be used to grade and identify diabetic retinopathy and macular edema.

[0003] In recent years, the rapid development of deep learning, especially the outstanding performance of convolutional neural networks in pattern classification, has enabled more and more image classification and segmentation tasks to be efficiently automated. Deep learning methods have achieved remarkable success in medical image classification and segmentation. When the test dataset has the same distribution as the training dataset, deep convolutional neural networks significantly improve the performance of fundus image segmentation.

[0004] However, current lesion segmentation faces two major challenges. The first is its strong reliance on domain knowledge of the data, and its strong recognition ability is only strong for fundus images of the main field of view with good image quality. In clinical practice, medical images often show changes in appearance due to various external factors, such as different scanner vendors and image quality. These distribution differences may cause deep networks to overfit on training datasets and lack generalization ability on unseen test datasets. The second is that these high-precision detection results rely on detailed, high-quality pixel-level labels. In medical image analysis, obtaining high-quality labels for data is both laborious and expensive, because accurately annotating medical images requires the expertise of clinicians. Summary of the Invention

[0005] The present invention is made to solve the above problems and aims to provide a method for automatically segmenting the region of interest in fundus images without manual annotation. The present invention adopts the following technical solutions:

[0006] The present invention provides a strong generalization fundus image segmentation method based on semi-supervised learning, which is characterized by comprising the following steps:

[0007] Step S1, preprocessing a preheating image of a preheating data set, wherein the preheating data set includes a label corresponding to the preheating image, that is, includes labeled data;

[0008] Step S2, inputting the labeled data into two preheating network models with the same structure for synchronous preheating training;

[0009] Step S3, extracting the encoder of the preheating network model after preheating training as the encoder of two segmentation main network models with the same structure;

[0010] Step S4, preprocessing and performing image perturbation on the training images of the training data set, wherein the training images are unlabeled data;

[0011] In step S5, the training image and the corresponding perturbed training image are respectively input into the two segmentation main network models to perform segmentation consistency training. When the difference in segmentation results between the two segmentation main network models is less than a predetermined difference threshold, the trained segmentation main network model is obtained.

[0012] The strongly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention may also have the following technical features, wherein step S5 includes the following sub-steps:

[0013] Step S5-1, dividing the plurality of training images into N batches;

[0014] Step S5-2, sequentially inputting the k-th batch of training images into one of the segmentation main network models for iterative training to obtain corresponding segmentation mask images, segmentation boundary images, and entropy images;

[0015] Step S5-3, sequentially inputting the k-th batch of training images into another segmentation main network model for iterative training to obtain the corresponding segmentation mask map, the segmentation boundary map, and the entropy map;

[0016] Step S5-4, calculating the loss error of the segmentation boundary graphs of the two segmentation main network models to obtain a first loss;

[0017] Step S5-5, calculating the loss error of the entropy graphs of the two segmentation main network models to obtain a second loss;

[0018] Step S5-6, calculating a weighted sum of the first loss and the second loss as a total loss error;

[0019] Step S5-7, back-propagating the total loss error to update the model parameters of the two segmentation main network models;

[0020] Step S5-8, determine whether the model parameters of each layer have converged. If it is judged to be so, obtain the trained segmentation main network model. If it is judged to be not, set k=k+1 and repeat steps S5-2 to S5-7.

[0021] The strongly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention may also have such a technical feature, wherein, in step S5-2, the process of performing the iterative training on one of the segmentation main network models once includes the following steps:

[0022] Step S5-2-1, downsampling the training image through the encoder of the segmentation main network model to obtain a feature map;

[0023] Step S5-2-2, based on the feature map, obtaining the segmentation mask map with rough edges and the corresponding entropy map through the decoder of the segmentation main network model;

[0024] Step S5-2-3, amplifying the rough edges of the segmentation mask image through the segmentation magnification module of the segmentation main network model, and refining the edges of the segmentation mask image in the magnified spatial dimension to obtain the segmentation mask image and the corresponding entropy image.

[0025] The strongly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention may also have the following technical features, wherein step S5 further includes:

[0026] Step S5-9, use the verification data set to verify the trained fundus image segmentation model, and determine whether the segmentation accuracy reaches the predetermined accuracy threshold. When the judgment is yes, the trained segmentation main network model is obtained. When the judgment is no, repeat steps S5-1 to S5-8.

[0027] The strongly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention may also have the following technical features: the segmentation accuracy is measured by the Dice index, and the accuracy threshold is 85%.

[0028] The strongly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention may also have such technical features, wherein the main segmentation network model includes an input layer, a convolution layer, a maximum pooling layer, a residual module C1, a residual module C2, a residual module C3, a residual module C4, an average pooling layer, a boundary upsampling module, a segmentation mask upsampling module and an entropy map acquisition module arranged in sequence, and the entropy map acquisition module is used to obtain an entropy map according to the segmentation mask map.

[0029] Functions and effects of the invention

[0030] According to the semi-supervised learning-based, strongly generalized fundus image segmentation method of the present invention, the main segmentation network is first pre-trained with labeled data. The purpose is not to make the model achieve the corresponding accuracy, but to use the labeled data as a guide to give the main segmentation network a preliminary impression. Furthermore, an unsupervised method is used to compare the two main segmentation network models after segmentation. Because the two main segmentation network models segment the original fundus image and the perturbed image respectively without mutual communication, the original image and the perturbed image should have structural consistency. Therefore, using this consistency criterion as a constraint, the loss error is calculated for the segmentation boundary map and entropy map of the two main segmentation network models each time, and it is continuously iterated until the difference between the two segmentation results is less than a predetermined iteration condition. The structures of the two main segmentation network models reach approximately the same level, that is, the recognition of the fundus image region of interest is relatively consistent. Through this iterative training, the model's recognition effect on the segmented region of interest in the fundus image can be effectively improved, thereby obtaining a high-precision segmentation model for the segmentation of the fundus image region of interest. The highly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention has two advantages. On the one hand, it only requires a small amount of labeled data for pre-training, and then relies on two main segmentation networks for comparative learning and improvement. In other words, it relies solely on the characteristics of the fundus image for automatic segmentation, effectively solving the problem of medical image annotation. On the other hand, the model trained by the method of the present invention does not rely on the domain knowledge of a certain data set, and can self-segment. It can quickly adapt to the characteristics of the fundus image obtained by different shooting instruments, shooting angles, shooting personnel skills, external lighting conditions, etc., solving the problem of domain generalization and having the advantages of strong generalization and high precision. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flowchart of a strongly generalized fundus image segmentation method based on semi-supervised learning in an embodiment of the present invention;

[0032] Figure 2 is a simplified flowchart of the fundus image segmentation model training process in an embodiment of the present invention;

[0033] Figure 3 is a flow chart of the fundus image segmentation model training process in an embodiment of the present invention;

[0034] Figure 4 2 is a schematic diagram of the structure of the residual module of the main network for fundus image segmentation in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the technical means, creative features, objectives and effects achieved by the present invention easier to understand, the following is a detailed description of the strongly generalized fundus image segmentation method based on semi-supervised learning of the present invention in combination with embodiments and drawings.

[0036] In the description of the present invention, the terms “first” and “second” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0037] <Example>

[0038] In this embodiment, a strongly generalized fundus image segmentation method based on semi-supervised learning is run on a computer, which requires a graphics card for GPU acceleration to complete the model training process. The trained fundus image segmentation model, fundus image segmentation process, and results are all stored in the computer.

[0039] In this embodiment, the datasets used include: the labeled IDRID public dataset to warm up the main segmentation network, and the unlabeled fundus dataset collected by the community hospital used as the training set and test set for testing the model effect.

[0040] Figure 1 4 is a flowchart of the strongly generalized fundus image segmentation method based on semi-supervised learning in this embodiment.

[0041] like Figure 1 As shown in FIG, the strong generalization fundus image segmentation method based on semi-supervised learning specifically includes the following steps:

[0042] Step S1, preprocessing the preheating images of the preheating data set, where the preheating data set contains labels corresponding to the preheating images, that is, contains labeled data.

[0043] In this embodiment, a preheating data set, a training data set and a verification data set are used to preheat, train and verify the model respectively. The three data sets contain multiple fundus images. For the convenience of description, the fundus images in the preheating data set are recorded as preheating images, the fundus images in the training data set are recorded as training images, and the fundus images in the verification data set are recorded as verification images.

[0044] The preprocessing of the warmed-up images includes resizing, center cropping, and normalization.

[0045] In step S2, a small amount of labeled data in the preheating dataset is input into two preheating network models with the same structure for synchronous preheating training, thereby injecting preliminary knowledge of the structure to be segmented.

[0046] In this embodiment, the model is first preheated and trained with a small amount of labeled data. The preheated network model after preheating training is saved in a computer. The computer can call the model through executable code and batch process multiple fundus images at the same time.

[0047] In step S3, the encoder of the preheating network model after preheating training is extracted and used as the encoder of two segmentation main network models with the same structure.

[0048] In this example, the preheating network model uses a ResNet101 residual network model as the encoder and decoder backbone. After preheating training, the encoder module is removed and used as the encoder for the main segmentation network model. The decoder for the main segmentation network model includes two types: a decoder for generating segmentation masks and a decoder for generating segmentation boundaries. The main segmentation network model was built using the existing deep learning framework PyTorch.

[0049] The specific structure of the segmentation main network model includes the input layer, convolution layer, maximum pooling layer, residual module C1, residual module C2, residual module C3, residual module C4, average pooling layer, boundary upsampling module, segmentation mask upsampling module, and entropy map acquisition module, which are arranged in sequence. The entropy map acquisition module is used to obtain the entropy map based on the segmentation mask. The specific structure and corresponding parameters of the segmentation main network model are as follows:

[0050] (1) Input layer, which is used to input the original fundus image. After image preprocessing operations such as resizing, center cropping, and normalization, a normalized image with a size of 512×512×3 is obtained;

[0051] (2) Convolutional layer, the convolution kernel size is 7×7, the sliding step is 2, and the output is 256×256×64;

[0052] (3) Maximum pooling layer, with a pooling size of 3×3, a sliding step of 2, and an output of 128×128×64;

[0053] (4) Multiple residual modules, including residual module C1 (convolution kernel size is 3×3, sliding step is 2, output is 128×128×64), residual module C2 (convolution kernel size is 3×3, sliding step is 2, output is 64×64×128), residual module C3 (convolution kernel size is 3×3, sliding step is 2, output is 32×32×256), residual module C4 (convolution kernel size is 3×3, sliding step is 2, output is 16×16×512);

[0054] (5) Average pooling layer, averaging in each dimension, the output is 1×1×512;

[0055] (6) Boundary upsampling module, including three convolution operations, with an output of 512×512×1;

[0056] (7) Segmentation mask upsampling module, including one convolution operation, with an output of 512×512×2;

[0057] (8) Entropy map acquisition module.

[0058] Figure 4 Schematic diagram of the structure of the residual module of the segmentation main network model in this embodiment.

[0059] like Figure 4 As shown, the structures of multiple residual modules in this embodiment are consistent.

[0060] Step S4: preprocessing and image perturbation are performed on the training images of the training data set. The training images are unlabeled data, that is, they have no corresponding labels.

[0061] In this embodiment, the training images are perturbed, including rotation, scaling, random cropping, and brightness and contrast changes. The preprocessing of the training images is consistent with the preprocessing of the warm-up images, so it will not be repeated here.

[0062] Step S5: Use the training data set to train the segmentation main network model. Specifically, the original training image and the corresponding perturbed training image are respectively input into the two segmentation main network models for segmentation consistency training. When the difference in the segmentation results of the two segmentation main network models is less than the predetermined difference threshold, the trained segmentation main network model is obtained.

[0063] The trained segmentation main network model can be used to automatically segment fundus images.

[0064] For the convenience of description, the two split main network models are respectively recorded as the first split main network model and the second split main network model.

[0065] Figure 2 1 is a simplified flowchart of the training process of the segmentation main network model in this embodiment.

[0066] Figure 3 Flowchart of the main network model training process for segmentation in this embodiment.

[0067] like Figure 2 and Figure 3 As shown, in step S5, the training of the segmentation main network model specifically includes the following steps:

[0068] Step S5-1: Divide the multiple training images in the training data set into N batches.

[0069] In this embodiment, the training images in the training data set are divided into 20 batches, each batch contains 32 training images, which are used for iterative training of the model.

[0070] In step S5-1a, k is set to 1, that is, 32 training images of the first batch are specified.

[0071] In step S5-2, the k-th batch of training images is sequentially input into the first segmentation main network model for iterative training to obtain the corresponding segmentation mask map, segmentation boundary map and entropy map.

[0072] In step S5-2, the process of iteratively training one of the segmentation main network models specifically includes the following steps:

[0073] Step S5-2-1, downsample the initial training image through the encoder of the segmentation main network model to obtain a feature map.

[0074] In step S5-2-2, based on the obtained feature map, a segmentation mask map and a corresponding entropy map are obtained by segmenting the decoder of the main network model. The obtained segmentation mask map is in a rough state and the edge features need to be further refined.

[0075] Step S5-2-3, amplify the edges of the rough segmentation mask map through the segmentation magnification module of the segmentation main network model, and refine the edges of the segmentation mask map in the magnified spatial dimension to obtain a refined segmentation mask map and a corresponding entropy map.

[0076] In step S5-3, the perturbed training images of the kth batch are sequentially input into the second segmentation main network model for iterative training to obtain the corresponding segmentation mask map, segmentation boundary map and entropy map.

[0077] The training process is consistent with steps S5-2-1 to S5-2-3, so it will not be repeated here.

[0078] In step S5-4, a loss error is calculated for the segmentation boundary graph of the first segmentation main network model and the segmentation boundary graph of the second segmentation main network model, and the calculated loss error is recorded as the first loss.

[0079] Step S5-5: Calculate the loss error of the entropy graph of the first segmented main network model and the entropy graph of the second segmented main network model, and record the calculated loss error as the second loss.

[0080] Step S5-6, calculating the weighted sum of the first loss and the second loss as the total loss error.

[0081] In step S5-7, the total loss error is back-propagated to update the model parameters of the two segmentation main network models.

[0082] Because the entropy map can represent the uncertainty of the segmentation boundary, the model parameters can be optimized by calculating the loss error through the entropy map and performing backpropagation.

[0083] Step S5-8, determines whether the model parameters of each layer have converged. If it is judged to be so, proceed to step S5-8a; if it is judged to be not, proceed to step S5-8b.

[0084] Step S5-8a, stop iteration and obtain the trained segmentation main network model.

[0085] In step S5-8b, k=k+1 is set, that is, the next batch of training images is specified, and then the process returns to step S4-2, that is, steps S5-2 to S5-7 are repeated with the next batch of training images to further optimize the model parameters.

[0086] In the above steps, after each iteration (i.e., passing a training image through the model), the model parameters of the last layer of the model calculate the loss error (i.e., cross-entropy loss) of the boundary map and entropy map, respectively. These two losses are then weighted and summed to obtain the total loss error. The calculated loss error is backpropagated and parameter optimization is performed using the stochastic gradient descent algorithm with a learning rate of 0.001 to update the model parameters. The training completion conditions for the model are the same as those for conventional convolutional neural network models: training is completed when the model parameters of each layer converge.

[0087] Step S5-9: Use the verification data set to verify the trained segmentation main network model and determine whether the segmentation accuracy reaches the predetermined accuracy threshold. If it is judged to be yes, the final, trained fundus image segmentation model is obtained. If it is judged to be no, repeat the above steps S5-1 to S5-8 to further optimize the model parameters.

[0088] It has been verified that for fundus images collected by the community, the accuracy of the model of this embodiment is improved compared with the original supervised learning model.

[0089] Example Function and Effect

[0090] According to the present embodiment, the segmentation main network is first preheated with labeled data for training. The purpose is not to make the model reach the corresponding accuracy, but to let the labeled data serve as a guide to give the segmentation main network a preliminary impression. Further, an unsupervised method is used to allow the two segmentation main networks to compare with each other after segmentation, because the two segmentation networks segment the original fundus image and the perturbed image respectively without mutual communication, but the original image and the perturbed image should have structural consistency. Therefore, with such a consistency criterion as a constraint, the loss error of the boundary map and entropy map of the two segmentation main networks is calculated each time, and it is continuously iterated until the difference between the segmentation results of the two is less than the specified iteration condition, and the structures of the two segmentation networks reach approximate consistency (i.e., the cognition of the fundus image region of interest is relatively consistent). Through such iterative training, the model's recognition effect on the segmentation region of interest of the fundus image can be effectively improved, thereby obtaining a high-precision segmentation model for the segmentation of the fundus image region of interest. The strongly generalized fundus image segmentation method based on semi-supervised learning provided by the present invention has two advantages. On the one hand, it only requires a small amount of labeled data for pre-training, and then relies on two main segmentation networks for comparative learning and improvement, that is, it relies solely on the characteristics of the fundus image for automatic segmentation, effectively solving the problem of medical image annotation. On the other hand, the model trained by the method of the embodiment of the present invention does not rely on the domain knowledge of a certain data set, and can self-segment. It can quickly adapt to the characteristics of the fundus image obtained by different shooting instruments, shooting angles, shooting personnel skills, external lighting conditions, etc., solving the problem of domain generalization and having the advantages of strong generalization and high precision.

[0091] Furthermore, due to the relatively simple structure of the model in the embodiment of the present invention, it can also be easily applied to other fields of medical image segmentation and can be embedded as a plug-and-play model in other segmentation tasks. Thanks to the introduction of the amplifier mechanism, it can effectively capture some tiny areas in medical images or areas that are highly integrated with the background, such as the segmentation of lesions in fundus images of diabetic retinopathy. It also achieves a relatively balanced data distribution, solving the overfitting problem caused by extremely uneven proportions in some segmented areas.

[0092] The above embodiments are only used to illustrate specific implementations of the present invention, and the present invention is not limited to the description scope of the above embodiments.

Claims

1. A strong generalized fundus image segmentation method based on semi-supervised learning, characterized in that: The following steps are involved: Step S1, preprocessing a preheating image of a preheating data set, wherein the preheating data set includes a label corresponding to the preheating image, that is, includes labeled data; Step S2: inputting the labeled data into two preheating network models with the same structure for synchronous preheating training, so as to inject preliminary knowledge of the structure to be segmented into the preheating network models; Step S3, extracting the encoder of the preheating network model after preheating training as the encoder of two segmentation main network models with the same structure; Step S4, preprocessing and performing image perturbation on the training images of the training data set, wherein the training images are unlabeled data; Step S5: input the training image and the corresponding perturbed training image into the two segmentation main network models respectively to perform segmentation consistency training. When the difference in segmentation results of the two segmentation main network models is less than a predetermined difference threshold, the trained segmentation main network model is obtained. The preheating network model uses the residual network ResNet101 model as the backbone of the encoder and decoder. The decoder of the segmentation main network model includes a decoder for generating a segmentation mask and a decoder for generating a segmentation boundary. Wherein, step S5 includes the following sub-steps: Step S5-1, dividing the plurality of training images into N batches; Step S5-2, sequentially inputting the k-th batch of training images into one of the segmentation main network models for iterative training to obtain corresponding segmentation mask images, segmentation boundary images, and entropy images; Step S5-3, sequentially inputting the k-th batch of training images into another segmentation main network model for iterative training to obtain the corresponding segmentation mask map, the segmentation boundary map, and the entropy map; Step S5-4, calculating the loss error of the segmentation boundary graphs of the two segmentation main network models to obtain a first loss; Step S5-5, calculating the loss error of the entropy graphs of the two segmentation main network models to obtain a second loss; Step S5-6, calculating a weighted sum of the first loss and the second loss as a total loss error; Step S5-7, back-propagating the total loss error to update the model parameters of the two segmentation main network models; Step S5-8, judging whether the model parameters of each layer have converged, if it is judged to be so, obtaining the trained main segmentation network model, if it is judged to be not, setting k=k+1, and repeating steps S5-2 to S5-7, In step S5-2, the process of performing the iterative training on one of the segmentation main network models includes the following steps: Step S5-2-1, downsampling the training image through the encoder of the segmentation main network model to obtain a feature map; Step S5-2-2, based on the feature map, obtain the segmentation mask map with rough edges and the corresponding entropy map through the decoder of the segmentation main network model; Step S5-2-3, amplifying the rough edges of the segmentation mask image through the segmentation magnification module of the segmentation main network model, and refining the edges of the segmentation mask image in the magnified spatial dimension to obtain the segmentation mask image and the corresponding entropy image.

2. The method for strongly generalized fundus image segmentation based on semi-supervised learning according to claim 1, characterized in that: in, Step S5 further includes: Step S5-9, use the verification data set to verify the trained segmentation main network model, and determine whether the segmentation accuracy reaches the predetermined accuracy threshold. When the judgment is yes, the trained segmentation main network model is obtained. When the judgment is no, repeat steps S5-1 to S5-8.

3. The method for strongly generalized fundus image segmentation based on semi-supervised learning according to claim 2, characterized in that: in, The segmentation accuracy is measured using the Dice indicator. The accuracy threshold is 85%.

4. The method for strongly generalized fundus image segmentation based on semi-supervised learning according to claim 1, characterized in that: in, The segmentation main network model includes an input layer, a convolution layer, a maximum pooling layer, a residual module C1, a residual module C2, a residual module C3, a residual module C4, an average pooling layer, a boundary upsampling module, a segmentation mask upsampling module and an entropy map acquisition module, which are arranged in sequence. The entropy map acquisition module is used to obtain an entropy map according to the segmentation mask map.

Images