A Disc Segmentation Method and Device Based on Circular Cropping Segmentation and Multi-Model Fusion

Abstract

The present invention discloses a method and device for retinal optic disc segmentation by cyclic cropping and multi-model fusion. The method includes: collecting an optic disc segmentation data set; selecting a deep learning neural network suitable for optic disc segmentation; cyclically cropping and then segmenting the optic disc; comprehensively analyzing the advantages of each neural network model, and performing model fusion. The present invention uses a variety of different neural network models and a variety of model fusion schemes, trains on multiple optic disc segmentation data sets, and evaluates the segmentation effects of each model and model fusion. The experimental results show that the model fusion scheme achieves better segmentation effects compared with any single sub-network model.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly relates to a method and device for retinal optic disc segmentation by cyclic cropping segmentation and multi-model fusion. Background Art

[0002] The optic disc is the most important physiological structure in the fundus image of the retina, where rich blood vessels are distributed. The position of the optic disc can locate the macula, and the optic disc plays an important role in the detection of glaucoma and the diagnosis of diabetic retinopathy. In the statistics of the causes of blindness in our country, once glaucoma causes damage to visual function and then leads to vision loss, the vision cannot be restored, causing permanent damage to the eyes. Clinically, glaucoma is diagnosed by observing symptoms such as the ratio of the size of the optic cup to the optic disc in the fundus color picture, whether there is a notch on the edge of the optic disc, and whether there is bleeding in the optic disc. The cup-to-disc ratio is the ratio of the size of the optic cup to the optic disc in the fundus image, which can effectively reflect the morphology of the optic nerve in the fundus. Then, the cup-to-disc ratio can be estimated by segmenting the optic disc in the fundus image, and a preliminary judgment can be made on diabetic complications. Therefore, reliable optic disc segmentation is very important in the automatic diagnosis of many fundus diseases.

[0003] Traditional optic disc segmentation is completed manually by experienced doctors. Since the optic disc is an approximately circular bright yellow area, according to clinical principles, some automatic optic disc segmentation methods have been designed using manual features. For example, local prior knowledge is used to remove blood vessels in the optic disc area to obtain the segmentation result. The representation ability of these manual features is limited, which may affect the generalization performance of the model and result in poor robustness. Deep learning technology has been widely applied to computer vision tasks, such as medical image segmentation, detection, and classification. The great success of convolutional neural networks has guided researchers to design deep neural network structures for optic disc segmentation. However, for optic disc segmentation methods using deep learning, previous studies have only been limited to a single neural network, and the accuracy, generalization, and applicability of the model are not very ideal. Therefore, it is very crucial to improve the segmentation accuracy, generalization, and robustness of the model.

[0004] The Fully Convolutional Networks (FCN) opened the chapter of semantic segmentation research based on deep learning. By classifying each pixel of the image, a prediction is generated for each pixel at the semantic level. FCN established the basic framework of the segmentation network, and there is no restriction on the input image. FCN uses a deconvolution layer for upsampling, so that the size of the output image is the same as that of the input image, ensuring that each pixel point of the input image has a corresponding pixel point in the output image, thereby realizing the prediction of each pixel point.

[0005] UNet originated from medical image segmentation. Compared with other networks, UNet has very few parameters, and the corresponding computational cost is also very small. It has strong adaptability and is widely used in the field of medical images. The UNet network is very simple and consists of only two parts. The first part is the encoder responsible for feature extraction, and the second part is the decoder responsible for upsampling. The first part and the second part expand symmetrically in two directions, so it is named UNet. The first part and the second part achieve precise positioning, and skip connections are used between them. This helps the decoder better restore the detailed features of the target. The MNet network is improved based on UNet. Similar to the symmetric structure of UNet, MNet has five parts and presents a symmetric structure. The network structure of MNet is in the shape of a symmetric "M". Screening glaucoma by the ratio of the optic cup to the optic disc is a commonly used method. Therefore, both optic cup segmentation and optic disc segmentation are very important. EfficientNet+UNet++ proposes an improved UNet++ network structure, using the EfficientNetB4 model as the backbone. After extracting features from EfficientNet-B4, precise segmentation is performed using the improved U-Net skip connections. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method and device for retinal optic disc segmentation by circular cropping segmentation and multi-model fusion, including collecting an optic disc segmentation data set; selecting a network model suitable for optic disc segmentation from numerous neural networks; first cropping the optic disc area of the fundus image and then segmenting the optic disc; comprehensively analyzing the characteristics of each network, integrating the advantages of different neural network models, and performing model fusion.

[0007] In the first aspect of the present invention, a method for retinal optic disc segmentation by circular cropping segmentation and multi-model fusion is provided. The method includes the following steps:

[0008] Performing optic disc area detection on the fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation;

[0009] Selecting an optic disc segmentation model and training the optic disc segmentation model;

[0010] Using the trained optic disc segmentation model to segment the optic disc of the first fundus image to obtain a first optic disc segmentation image. Taking the centroid of the first optic disc segmentation image as the center point, expanding the same size in the four directions of up, down, left, and right, and cropping out a second optic disc segmentation image of 512×512;

[0011] Retraining the optic disc segmentation model with the new data set composed of the second optic disc segmentation images to obtain a trained new optic disc segmentation model, and using the new optic disc segmentation model to segment the optic disc of the second optic disc segmentation images to obtain a third optic disc segmentation image;

[0012] Restore the third optic disc segmentation image to its original size to obtain the final optic disc segmentation image;

[0013] Among them, for fundus images, perform optic disc region detection based on a lightweight model to obtain the first fundus image for optic disc segmentation, specifically including:

[0014] According to the annotation results of the optic disc segmentation training set, use graphics to extract the centroid and border to construct the object detection dataset;

[0015] Scale the data in the object detection dataset using linear interpolation;

[0016] Construct a lightweight optic disc region detection model and train the lightweight optic disc region detection model;

[0017] Use the trained lightweight optic disc region detection model to crop the images in the scaled object detection dataset;

[0018] Restore the cropped images to visualization to obtain the first fundus image for optic disc segmentation.

[0019] A further technical solution of the present invention is that the method further includes fusing the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel.

[0020] A further technical solution of the present invention is that fusing the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel specifically includes:

[0021] Select several optic disc segmentation models to train on the dataset to obtain several trained optic disc segmentation models;

[0022] Use the several trained optic disc segmentation models to perform optic disc segmentation on the fundus images after optic disc region detection respectively;

[0023] Perform weight fusion on the results of the several optic disc segmentation models obtained to obtain the weight fusion result;

[0024] Adopt the principle of the minority obeying the majority with different weights to align the obtained weight fusion result.

[0025] A further technical solution of the present invention is that before training the optic disc segmentation model, perform data augmentation on the training dataset, and the data augmentation includes horizontal flipping, vertical flipping and image enhancement of the images in the training dataset.

[0026] A further technical solution of the present invention is that during the training process of the optic disc segmentation model, use the mean intersection over union miou coefficient and dice coefficient to evaluate the model performance, and the specific expressions are: Where X and Y respectively represent the segmented optic disc and the optic disc manually marked by experts, |X∩Y| is the number of pixels in the intersection between X and Y, and |X| and |Y| respectively represent the number of pixels of X and Y.

[0027] In the second aspect of the present invention, a retinal optic disc segmentation device for cyclic cropping segmentation and multi-model fusion includes:

[0028] An optic disc area detection unit, configured to perform optic disc area detection on a fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation;

[0029] An optic disc segmentation model acquisition unit, configured to select an optic disc segmentation model and train the optic disc segmentation model;

[0030] An optic disc segmentation unit, configured to use the trained optic disc segmentation model to perform optic disc segmentation on the first fundus image to obtain a first optic disc segmentation image, and with the centroid of the first optic disc segmentation image as the center point, expand the same size in the up, down, left, and right directions to crop out a second optic disc segmentation image of 512×512;

[0031] A model re-training and re-optic disc segmentation unit, configured to re-train the optic disc segmentation model with a new data set composed of the second optic disc segmentation images to obtain a trained new optic disc segmentation model, and use the new optic disc segmentation model to perform optic disc segmentation on the second optic disc segmentation images to obtain a third optic disc segmentation image;

[0032] An image restoration unit, configured to restore the third optic disc segmentation image to the original size to obtain the final optic disc segmentation image;

[0033] Among them, the optic disc area detection unit includes:

[0034] A target detection data set construction module, configured to construct a target detection data set by extracting the centroid and the border using graphics according to the annotation results of the optic disc segmentation training set;

[0035] A target detection module, configured to scale the data in the target detection data set using linear interpolation;

[0036] A model construction and training module, configured to construct a lightweight optic disc area detection model and train the lightweight optic disc area detection model;

[0037] An execution cropping module, configured to crop the images in the scaled target detection data set using the trained lightweight optic disc area detection model;

[0038] An image restoration module, configured to restore the cropped image into a visualization to obtain a first fundus image for optic disc segmentation.

[0039] A further technical solution of the present invention is that the device further includes a segmentation result fusion unit for fusing the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel.

[0040] A further technical solution of the present invention is that the segmentation result fusion unit includes:

[0041] An optic disc segmentation model training module for selecting several optic disc segmentation models to train on a dataset to obtain several trained optic disc segmentation models;

[0042] Several optic disc segmentation modules for using the several trained optic disc segmentation models to respectively perform optic disc segmentation on the fundus images after optic disc region detection;

[0043] A weight fusion module for performing weight fusion on the results of the several optic disc segmentation models obtained to obtain a weight fusion result;

[0044] A segmentation result processing module for aligning the obtained weight fusion result by adopting the principle of the minority obeying the majority with different weights.

[0045] A method and device for retinal optic disc segmentation by cyclic cropping segmentation and multi-model fusion provided by the present invention adopt two schemes of multiple cyclic cropping segmentation and multi-model fusion. The specific process includes: collecting an optic disc segmentation dataset; selecting a network suitable for optic disc segmentation from many neural networks; first cropping the optic disc region of the fundus image and then segmenting the optic disc; comprehensively analyzing the characteristics of each network, integrating the advantages of different neural network models, and performing model fusion. The beneficial effects obtained by the present invention are:

[0046] Segmenting after cropping can improve the model training effect and make the retinal optic disc segmentation effect better;

[0047] When the accuracy of each network has reached the limit, the model fusion scheme can still improve the accuracy. The model fusion is also effective for special fundus images that are difficult to segment. The model fusion method in the field of optic disc segmentation is effective and feasible, and the weighted fusion scheme has a better effect. When selecting a network model, not only the accuracy of a single sub-network model is considered, but also the differences between different sub-network models are considered. The differences between sub-network models will directly affect the effect of model fusion. Therefore, using multiple different neural network models and multiple schemes for fusion, training on the combined optic disc segmentation dataset, and finally evaluating the segmentation effects of each model and model fusion. Compared with any single sub-network model, the model fusion scheme can achieve a better segmentation effect. The more scientific the selected sub-network model and the more scientific the weight ratio, the better the effect of the fusion model. Description of the Drawings

[0048] Figure 1 It is a schematic flowchart of the retinal optic disc segmentation method with high-cycle cropping segmentation and multi-model fusion in the embodiments of the present invention;

[0049] Figure 2 It is a schematic flowchart of the optic disc area detection method based on a lightweight model for fundus images in the embodiments of the present invention;

[0050] Figure 3 It is the original fundus image and the effect diagram after cropping by the trained lightweight model deep learning network model in the embodiments of the present invention;

[0051] Figure 4 It is the effect diagram of segmentation, cropping and re-segmentation in the embodiments of the present invention;

[0052] Figure 5 It is a schematic flowchart of the method for fusing the segmentation results of several optic disc segmentation models in the embodiments of the present invention;

[0053] Figure 6 It is the visualization segmentation effect diagram in the embodiments of the present invention;

[0054] Figure 7 It is a schematic structural diagram of the retinal optic disc segmentation device with cyclic cropping segmentation and multi-model fusion in the embodiments of the present invention;

[0055] Figure 8 It is a schematic structural diagram of the optic disc area detection unit in the embodiments of the present invention;

[0056] Figure 9 It is a schematic structural diagram of the segmentation result fusion unit in the embodiments of the present invention. Detailed implementation manners

[0057] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only for explaining the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of convenience of description, only parts related to the present invention rather than all structures are shown in the drawings.

[0058] Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of the steps can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0059] In addition, terms such as "first", "second", etc. may be used herein to describe various directions, actions, steps, or elements, etc., but these directions, actions, steps, or elements are not limited by these terms. These terms are only used to distinguish a first direction, action, step, or element from another direction, action, step, or element. For example, without departing from the scope of the present application, a first disc segmentation image may be referred to as a second disc segmentation image, and similarly, a second disc segmentation image may be referred to as a first disc segmentation image. Both the first disc segmentation image and the second disc segmentation image are speed differences, but they are not the same disc segmentation image. Terms such as "first" and "second" should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the present invention, the meaning of "a plurality" is at least two, such as two, three, etc., unless otherwise specifically defined.

[0060] An embodiment of the present invention provides the following embodiments for a method and device for retinal optic disc segmentation with circular cropping segmentation and multi-model fusion:

[0061] Based on Embodiment 1 of the present invention

[0062] This embodiment is used to illustrate the method for retinal optic disc segmentation with circular cropping segmentation and multi-model fusion. Refer to Figure 1 As shown, the method includes the following steps:

[0063] S110. Perform optic disc area detection on the fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation;

[0064] In the specific implementation process, when segmenting the optic disc of the fundus image, the most interesting area is the area where the optic disc is located, and the area of interest of the optic disc needs to be extracted. In the field of image segmentation, the extraction of the area of interest is a common and very effective method. After extraction, it can not only eliminate the interference of noise, but also reduce the computational amount of image processing and accelerate the training speed. Due to the inconsistent standards for collecting fundus images in different datasets, the fundus images are different in size and integrity. When the training set consists of relatively complete fundus images, the segmentation effect of the optic disc of the complete fundus images in other datasets is relatively ideal; however, when segmenting only the part of the optic disc (such as the RIM-ONE dataset) that is captured, there are great problems and the segmentation effect is very unsatisfactory. To increase the generalization and robustness of the model, it is very necessary to crop the area of interest of the optic disc and then perform model training and test segmentation. The training results show that since the fundus images in the RIM-ONE dataset only have the part of the optic disc area, compared with other complete fundus images, RIM-ONE has achieved better training results. It shows that segmentation after cropping can improve the model training effect.

[0065] Further, perform optic disc region detection on fundus images based on a lightweight model to obtain a first fundus image for optic disc segmentation. Refer to Figure 2 , which specifically includes:

[0066] S1101. According to the annotation results of the optic disc segmentation training set, use graphics to extract the centroid and bounding box to construct a target detection data set;

[0067] S1102. Scale the data in the target detection data set using linear interpolation;

[0068] S1103. Construct a lightweight optic disc region detection model and train the lightweight optic disc region detection model;

[0069] S1104. Use the trained lightweight optic disc region detection model to crop the images in the scaled target detection data set;

[0070] S1105. Restore the cropped images to visualization to obtain a first fundus image for optic disc segmentation.

[0071] Specifically, the steps for implementing optic disc region detection based on a lightweight model are as follows:

[0072] Construct a target detection data set: Construct a target detection data set according to the annotation results of the optic disc segmentation training set. Among them, the optic disc segmentation training set is the hospital annotation training set and the PALM competition data set, specifically the PALM competition data set, Drishti-GS, REFUGE and other data sets. Create a new file directory for detection training and copy the training images over first. Another implemented algorithm is to use graphics to extract the centroid and bounding box, that is, smooth the image brightness through filtering, and then extract the coordinates of the brightest point to obtain the centroid. Finally, obtain the optic disc region detection data set.

[0073] Target detection process. Construct a data augmentation method. Using linear interpolation for scaling is faster for training because a 512×512 size region is cropped, so the accuracy requirement for the intersection over union does not need to be too high.

[0074] Cluster the bounding boxes to construct a PP-TOLO Tiny model. Use image normalization, random rotation, vertical flipping and image augmentation, adopt the Warmup+Piecewise-decay strategy, and adopt a mixed loss function of weighted CELoss and DiceLoss and other appropriate training strategies. Set the number of training epochs and save the trained detection model.

[0075] Call the trained model, perform cropping, and finally restore it for visualization.

[0076] After renaming the cropped images, a training set for disc segmentation can be obtained.

[0077] See Figure 3 , which are the original fundus images and the images obtained after cropping by a trained lightweight deep learning network model. It can be seen that the lightweight model can accurately crop the disc area of the color fundus image.

[0078] S120: Select a disc segmentation model and train the disc segmentation model;

[0079] S130: Use the trained disc segmentation model to perform disc segmentation on the first fundus image to obtain a first disc segmentation image. Taking the centroid of the first disc segmentation image as the center point, expand the same size in the up, down, left, and right directions, and crop out a second disc segmentation image of 512×512;

[0080] S140: Retrain the disc segmentation model with the new data set composed of the second disc segmentation images to obtain a trained new disc segmentation model, and use the new disc segmentation model to perform disc segmentation on the second disc segmentation images to obtain a third disc segmentation image;

[0081] S150: Restore the third disc segmentation image to its original size to obtain the final disc segmentation image.

[0082] In the specific implementation process, a deep learning segmentation model can be used for rough segmentation of the disc area for the first time. Then, expand and crop the same-sized disc area with the centroid of the predicted disc as the center point. The cropped fundus images are used as a new data set for the second model training to obtain a more accurate training model on the new data set. Then repeat the cycle, but each time use a more accurate model to segment and locate the disc.

[0083] The effect of disc cropping by this method depends on the quality of disc segmentation, so the metrics of disc segmentation can be used to evaluate the effect of disc cropping. As long as the disc segmentation effect is good, the effect of disc area cropping will definitely be good. When the cropping effect is good, the original image and the annotated image obtained by cropping can be used as the data set. To make the segmentation effect better, the process of segmentation, cropping, segmentation, and finally restoration is as follows:

[0084] The first step: Use a model with good segmentation effect, including but not limited to MNet, FCN, HRNet18, HRNet48, UNet, OCRNet, EMANet, ETNet, EfficientNet+Unet++, to perform disc segmentation on the color fundus image. Then, taking the centroid of the segmented disc area as the center point, expand the same size in the up, down, left, and right directions, and crop out a disc area of 512×512. SeeFigure 4 As shown Figure 4 (a) is the original color fundus image, Figure 4 (b) is the optic disc area map obtained by first segmenting and then cropping.

[0085] Step 2: On the obtained new dataset (only containing a partial area of the optic disc), retrain the optic disc segmentation model. Then, use the newly trained model to perform optic disc segmentation on the cropped map of the optic disc area. After segmentation, the optic disc segmentation overlay map as shown in Figure 4 (c) is obtained.

[0086] Step 3: Restore the optic disc segmentation overlay Figure 4 (c) obtained in Step 2 to its original size, obtaining Figure 4 (d).

[0087] In summary, the optic disc in the circle of Figure 4 (e) can be segmented to obtain Figure 4 (f).

[0088] Furthermore, the method of the embodiment further includes fusing the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel.

[0089] In the specific implementation process, by fusing different models, the performance of the model can be improved. There are three directions for fusing models: the first is the model result, the second is the model itself, and the third is the sample set. Different models have different characteristics and focuses. In order to achieve better prediction results, models with different focuses can be fused. So far, for the binary classification problem of optic disc segmentation, at least five different characteristic segmentation models have been proposed. The method of the present invention selects typical and highly accurate segmentation models for fusion to improve the segmentation effect of the optic disc.

[0090] Use mathematical probability methods to calculate the theoretical effect of fusing three or five characteristic models. For the convenience of calculation, the following assumptions are made: 1. For each pixel point, the accuracy of each model is P (0 < P < 1), 2. Different models are independent of each other. The voting method of the minority obeying the majority is adopted. Calculate and compare the relative magnitudes of the accuracy of a single model and the accuracy of model fusion.

[0091] As described above, the accuracy of a single model is P. The accuracy of fusing three models is denoted as P3, and P3 is obtained by the following formula (1). To make the accuracy P3 of fusing three network models greater than the accuracy P of a single model, that is, P3 > P, the following formula (2) can be obtained:

[0092]

[0093] 3*p 2 *(1 - p) + p3 > P (2)

[0094] Calculating the above formula (2) gives 0.5 < P < 1. Then theoretically, as long as the accuracy rate P of a single independent model is greater than 0.5, the fusion of the three network models is better than that of a single model.

[0095] The accuracy rate of the fusion of five network models is denoted as P5, and P5 is obtained from the following formulas (3) and (4). Similarly, to make the accuracy rate P5 of the fusion of five network models greater than the accuracy rate P of a single model, that is, P5 > P, the following formula (5) can be obtained:

[0096]

[0097] P5 = 10 * p 3 *(1 - p) 2 + 5 * p 4 *(1 - p)+ p 5 (4)

[0098] 10 * p 3 *(1 - p) 2 + 5 * p 4 *(1 - p)+ p 5 > P (5)

[0099] Calculating the above formula (5) gives 0.500 < P < 1. As long as the accuracy rate P of a single model is greater than 0.500, then the fusion of five network models is better than that of a single model. By analogy, the effects of the fusion of more models can be calculated.

[0100] To sum up, as long as the accuracy rates of each independent model are greater than 0.500, then theoretically the accuracy rate of the fusion of three or five models is higher than that of a single model. In specific practice, it is found that the accuracy rates of different neural networks are not the same and sometimes vary greatly. Then for different neural network models, the higher the accuracy rate, the greater the weight assigned to it.

[0101] Furthermore, fusing the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel, see Figure 5 , specifically including:

[0102] S210. Select several optic disc segmentation models to be trained on the dataset to obtain several trained optic disc segmentation models;

[0103] S220. Use the several trained optic disc segmentation models to perform optic disc segmentation on the fundus images after optic disc area detection respectively;

[0104] S230. Perform weight fusion on the results of the several obtained optic disc segmentation models to obtain a weight fusion result;

[0105] S240. Adopt the principle of the minority obeying the majority after taking different weights to align the obtained weight fusion result, that is, convert the segmentation result into a PNG format image with the optic disc being black and the rest of the area being white. In fact, it is to obtain an image with the same format as the annotation map.

[0106] In the specific implementation process, the rationality of model fusion lies in that different models have different advantages, disadvantages and focuses in image segmentation, and model fusion can integrate the advantages of each model. The specific idea of model fusion is to fuse the segmentation results of different models and finally determine the prediction result of each pixel through voting. Specifically, the implementation process of model fusion is as follows:

[0107] The first step: First, fully train the selected network models on the mixed dataset to obtain the best trained models and parameters. The mixed dataset includes but is not limited to Drishti - GS, REFUGE, RIM - ONE - R1, ORDS, PALM, MESSIDOR.

[0108] The second step: Load the trained models and perform optic disc segmentation on the fundus images in the validation set.

[0109] The third step: Select the model result with the best segmentation effect, perform appropriate weight fusion to obtain the final fusion result.

[0110] The fourth step: We adopt the principle of the minority obeying the majority after taking different weights to obtain the final model fusion segmentation result.

[0111] The fifth step: In order to comprehensively analyze and compare the segmentation results of each sub - model and after model fusion, calculate indicators such as the dice coefficient and accuracy rate between each segmentation result and the annotation map, and comprehensively analyze and compare the segmentation results of each sub - model and model fusion.

[0112] Conduct various model fusion schemes. When the accuracy rate of each network has reached the limit, the model fusion scheme can still improve the accuracy rate. Model fusion is also effective for special fundus images that are difficult to segment. The effectiveness and feasibility of the model fusion method in the field of optic disc segmentation; The weighted fusion scheme has a better effect. When selecting network models, not only the accuracy rate of a single sub - network model should be considered, but also the differences between different sub - network models should be considered. The differences between sub - network models will directly affect the effect of model fusion. Not all model fusion schemes will have a better segmentation effect than a single model. Therefore, when selecting a specific single model for the model fusion scheme, the characteristics of the single model should be analyzed, and the segmentation effect of the single model itself should be considered.

[0113] Further, before training the optic disc segmentation model, data augmentation is performed on the training dataset. The data augmentation includes horizontal flipping, vertical flipping, and image enhancement of the images in the training dataset. Specifically, since the number of images in a single optic disc segmentation dataset is limited, in order to reduce the risk of overfitting, the dataset is augmented. First, different optic disc segmentation datasets are selected online. Training is not only carried out on a single dataset, but also different datasets are combined to increase the number of images in the training set. In this way, the generalization of the trained model is better. Second, data augmentation is performed, including horizontal flipping, vertical flipping, and image enhancement, which can improve the robustness of the segmentation method.

[0114] Further, the model fusion method requires training multiple segmentation network models. The implementation of model training should adopt a GPU training environment. During the training process, set the batch size, momentum, learning rate, decay function, and loss function. Specifically, during the training process of the optic disc segmentation model, the mean intersection over union (mIoU) coefficient and the Dice coefficient are used to evaluate the model performance. The specific expressions are as follows: where X and Y represent the segmented optic disc and the expert manually annotated optic disc respectively, |X∩Y| is the number of pixels in the intersection between X and Y, and |X| and |Y| represent the number of pixels in X and Y respectively.

[0115] It should be noted that for the training results of a single segmentation model, in order to further improve the model effect, the following multiple measures can also be taken:

[0116] Change the ratio of the training set and the test set to better train the model. Re-divide the dataset and retrain the model multiple times. Adjust the ratio of the two loss functions multiple times. Reduce the model saving interval step size multiple times; increase the number of training epochs multiple times, and a model is trained multiple times.

[0117] Analyze the characteristics of each model, and scientifically select sub-models. Try multiple sub-models, fully train the sub-models, and fuse the sub-models in multiple ways. Visualize the training process and select appropriate parameters for each sub-model. After cropping the optic disc area and then performing segmentation, the segmentation effect can be improved.

[0118] This embodiment gives a specific example. Refer to Figure 6 , and use the trained model to segment a single image. Figure 6 To visualize the segmentation effect, the pseudo-color fundus image and the segmented image are superimposed, and the mean intersection over union (mIoU) and Dice coefficient indexes after segmenting a single image are calculated. The mIoU of the segmentation prediction of this fundus image reaches 0.9852, and the Dice coefficient even reaches 0.9925, and the segmentation effect is extremely ideal.

[0119] Based on Embodiment 2 of the present invention

[0120] The retinal optic disc segmentation device 700 provided in the second embodiment of the present invention, which can perform the retinal optic disc segmentation method of cyclic cropping segmentation and multi-model fusion provided in any embodiment of the present invention, has corresponding functional modules and beneficial effects for executing the method. The device can be implemented in the form of software and / or hardware (integrated circuit), and is generally integrated in a server or a terminal device. Figure 7 It is a schematic structural diagram of a retinal optic disc segmentation device 700 in the second embodiment of the present invention. Referring to Figure 7 , the retinal optic disc segmentation device 700 of the cyclic cropping segmentation and multi-model fusion in the embodiment of the present invention specifically may include:

[0121] An optic disc area detection unit 710, configured to perform optic disc area detection on a fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation;

[0122] An optic disc segmentation model acquisition unit 720, configured to select an optic disc segmentation model and train the optic disc segmentation model;

[0123] An optic disc segmentation unit 730, configured to perform optic disc segmentation on the first fundus image using the trained optic disc segmentation model to obtain a first optic disc segmentation image, and taking the centroid of the first optic disc segmentation image as the center point, expanding the same size in the four directions of up, down, left, and right, and cropping out a second optic disc segmentation image of 512×512;

[0124] A model re-training and then optic disc segmentation unit 740, configured to re-train the optic disc segmentation model with a new data set composed of the second optic disc segmentation images to obtain a trained new optic disc segmentation model, and perform optic disc segmentation on the second optic disc segmentation images using the new optic disc segmentation model to obtain a third optic disc segmentation image;

[0125] An image restoration unit 750, configured to restore the third optic disc segmentation image to its original size, and thus obtain the final optic disc segmentation image;

[0126] Among them, referring to Figure 8 , the optic disc area detection unit 710 includes:

[0127] A target detection data set construction module 7101, configured to construct a target detection data set by using graphics to extract the centroid and the border according to the annotation results of the optic disc segmentation training set;

[0128] A target detection module 7102, configured to scale the data in the target detection data set using linear interpolation;

[0129] The model building and training module 7103 is configured to build a lightweight optic disc region detection model and train the lightweight optic disc region detection model;

[0130] The execution and cropping module 7104 is configured to crop the images in the scaled target detection dataset by using the trained lightweight optic disc region detection model;

[0131] The image restoration module 7105 is configured to restore the cropped images into a visual form to obtain the first fundus image for optic disc segmentation.

[0132] Furthermore, the apparatus 700 further includes a segmentation result fusion unit 760 configured to fuse the segmentation results of a plurality of optic disc segmentation models to obtain the prediction result of each pixel.

[0133] See Figure 9 , the segmentation result fusion unit 760 includes:

[0134] The optic disc segmentation model training module 7601 is configured to select a plurality of optic disc segmentation models for training on a dataset to obtain a plurality of trained optic disc segmentation models;

[0135] A plurality of optic disc segmentation modules 7602 are configured to respectively perform optic disc segmentation on the fundus images after optic disc region detection by using the plurality of trained optic disc segmentation models;

[0136] The weight fusion module 7603 is configured to perform weight fusion on the results of the plurality of obtained optic disc segmentation models to obtain a weight fusion result;

[0137] The segmentation result processing module 7604 is configured to align the obtained weight fusion result by adopting the principle of the minority obeying the majority with different weights.

[0138] In addition to the above units and modules, the retinal optic disc segmentation apparatus 700 with cyclic cropping segmentation and multi-model fusion may further include other components. However, since these components are not related to the content of the embodiments of the present disclosure, their illustrations and descriptions are omitted here.

[0139] The specific working process of a retinal optic disc segmentation apparatus 700 with cyclic cropping segmentation and multi-model fusion refers to the description of Embodiment 1 of the above-mentioned retinal optic disc segmentation method with cyclic cropping segmentation and multi-model fusion, and will not be elaborated herein.

[0140] A method and device for retinal optic disc segmentation based on cyclic cropping segmentation and multi-model fusion provided by the present invention adopt two schemes of multiple cyclic cropping segmentation and multi-model fusion. The specific process includes: collecting an optic disc segmentation data set; selecting a network suitable for optic disc segmentation from numerous neural networks; first cropping the optic disc area of the fundus image and then segmenting the optic disc; comprehensively analyzing the characteristics of each network, integrating the advantages of different neural network models, and performing model fusion. The beneficial effects obtained by the present invention are as follows:

[0141] Segmenting after cropping can improve the model training effect and make the retinal optic disc segmentation effect better;

[0142] When the accuracy of each network has reached the limit, the model fusion scheme can still improve the accuracy. The model fusion is also effective for special fundus images that are difficult to segment. The model fusion method in the field of optic disc segmentation is effective and feasible, and the weighted fusion scheme has a better effect. When selecting network models, not only the accuracy of a single sub-network model is considered, but also the differences between different sub-network models are considered. The differences between sub-network models will directly affect the effect of model fusion. Therefore, multiple different neural network models and multiple schemes are used for fusion, and training is carried out on the combined optic disc segmentation data set. Finally, the segmentation effects of each model and the model fusion are evaluated. Compared with any single sub-network model, the model fusion scheme can achieve a better segmentation effect. The more scientific the selected sub-network models and the weight ratios are, the better the effect of the fusion model will be.

[0143] Note that the above is only a preferred embodiment of the present invention and the applied technical principle. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described here. Various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments can be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for segmenting the optic disc of the retina by cyclic cutting and multi-model fusion, characterized in that, The method includes the following steps: Perform optic disc region detection on the fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation; Select an optic disc segmentation model and train the optic disc segmentation model; Use the trained optic disc segmentation model to segment the optic disc of the first fundus image to obtain a first optic disc segmentation image. Taking the centroid of the first optic disc segmentation image as the center point, expand the same size in the four directions of up, down, left, and right, and crop out a second optic disc segmentation image; Retrain the optic disc segmentation model with the new dataset composed of the second optic disc segmentation images to obtain a trained new optic disc segmentation model, and use the new optic disc segmentation model to segment the second optic disc segmentation image to obtain a third optic disc segmentation image; Restore the third optic disc segmentation image to its original size to obtain the final optic disc segmentation image; Among them, performing optic disc region detection on the fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation specifically includes: According to the annotation results of the optic disc segmentation training set, use graphics to extract the centroid and border to construct a target detection dataset; Scale the data in the target detection dataset using linear interpolation; Construct a lightweight optic disc region detection model and train the lightweight optic disc region detection model; Use the trained lightweight optic disc region detection model to crop the images in the scaled target detection dataset; Restore and visualize the cropped images to obtain a first fundus image for optic disc segmentation.

2. The retinal optic disc segmentation method based on cyclic cropping segmentation and multi-model fusion according to claim 1, wherein The method further includes fusing the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel.

3. The retinal optic disc segmentation method of cyclic cropping segmentation and multi-model fusion according to claim 2, characterized in that, The fusing of the segmentation results of several optic disc segmentation models to obtain the prediction result of each pixel specifically includes: Select several optic disc segmentation models to train on the dataset to obtain several trained optic disc segmentation models; Use the several trained optic disc segmentation models to segment the optic disc of the fundus image after optic disc region detection respectively; Perform weight fusion on the results of the several obtained optic disc segmentation models to obtain a weight fusion result; Adopt the principle of the minority obeying the majority with different weights to align the obtained weight fusion result.

4. The retinal optic disc segmentation method based on cyclic cropping segmentation and multi-model fusion according to claim 1, characterized in that Before training the optic disc segmentation model, perform data augmentation on the training dataset. The data augmentation includes horizontal flipping, vertical flipping, and image enhancement of the images in the training dataset.

5. The retinal optic disc segmentation method of cyclic cropping segmentation and multi-model fusion according to claim 1, characterized in that During the training process of the optic disc segmentation model, the mean intersection over union (mIoU) coefficient and the Dice coefficient are used to evaluate the model performance. The specific expressions are as follows: where X and Y represent the segmented optic disc and the manually annotated optic disc by experts respectively, |X∩Y| is the number of pixels in the intersection between X and Y, and |X| and |Y| represent the number of pixels in X and Y respectively.

6. A retinal optic disc segmentation device with cyclic cutting segmentation and multi-model fusion, characterized in that, Including: An optic disc region detection unit for performing optic disc region detection on the fundus image based on a lightweight model to obtain a first fundus image for optic disc segmentation; An optic disc segmentation model acquisition unit for selecting an optic disc segmentation model and training the optic disc segmentation model; An optic disc segmentation unit for using the trained optic disc segmentation model to segment the optic disc of the first fundus image to obtain a first optic disc segmentation image. Taking the centroid of the first optic disc segmentation image as the center point, expand the same size in the four directions of up, down, left, and right, and crop out a second optic disc segmentation image; The post-model retraining disc segmentation unit is used to retrain the disc segmentation model with a new dataset composed of the second disc segmentation images to obtain a trained new disc segmentation model, and use the new disc segmentation model to perform disc segmentation on the second disc segmentation images to obtain the third disc segmentation images; The image restoration unit is used to restore the third disc segmentation images to the original size, and the final disc segmentation images can be obtained; Among them, the disc region detection unit includes: The target detection dataset construction module is used to construct a target detection dataset by using graphics to extract the centroid and the border according to the annotation results of the disc segmentation training set; The target detection module is used to scale the data in the target detection dataset by using linear interpolation; The model construction and training module is used to construct a lightweight disc region detection model and train the lightweight disc region detection model; The execution cropping module is used to crop the images in the scaled target detection dataset by using the trained lightweight disc region detection model; The image restoration module is used to restore the cropped images into visualizations to obtain the first fundus images for disc segmentation.

7. The retinal optic disc segmentation device for cyclic cropping segmentation and multi-model fusion according to claim 6, characterized in that, The device further includes a segmentation result fusion unit, which is used to fuse the segmentation results of several disc segmentation models to obtain the prediction results of each pixel.

8. The retinal optic disc segmentation device for cyclic cropping segmentation and multi-model fusion according to claim 6, characterized in that, The segmentation result fusion unit includes: The disc segmentation model training module is used to select several disc segmentation models for training on the dataset to obtain several trained disc segmentation models; Several disc segmentation modules are used to perform disc segmentation on the fundus images after disc region detection by using the several trained disc segmentation models respectively; The weight fusion module is used to perform weight fusion on the results of the several disc segmentation models obtained to obtain a weight fusion result; The segmentation result processing module is used to align the obtained weight fusion result by adopting the principle of the minority obeying the majority with different weights.

Images