An eye fundus image lesion segmentation method, system, device and storage medium

Abstract

The application discloses an eye fundus image lesion segmentation method, system, device and storage medium, which comprises the following steps: constructing a first classification network model, inputting historical eye fundus images and real classification labels into the first classification network model for training, and obtaining a second classification network model; constructing a first lesion segmentation network model, training the first lesion segmentation network model according to the second classification network model, historical eye fundus images and segmentation mask images corresponding to the historical eye fundus images, and obtaining a second lesion segmentation network model; inputting a to-be-predicted eye fundus image into the second lesion segmentation network model for prediction, obtaining lesion segmentation information of the to-be-predicted eye fundus image, associating segmentation information features on a classification task, and providing a supervision signal and prior knowledge for the training of the segmentation model through the classification network model, so that the eye fundus image lesion segmentation accuracy is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of fundus image lesion segmentation, in particular to a fundus image lesion segmentation method, system, device and storage medium. BACKGROUND

[0002] In recent years, artificial intelligence technology has been widely applied in various fields of daily life, such as smart cities, intelligent retail and smart homes, and provides various services and conveniences for daily life. In addition, artificial intelligence has gradually developed in the medical field. The current artificial intelligence technology highly depends on the quantity and quality of data, and the more training data, the better the effect of the algorithm model trained. Since medical data involves patient privacy information compared with natural data, the acquisition of medical data is strictly controlled, and the acquisition of medical data is difficult. And artificial intelligence technology needs labeled images or text information when training the model, but the labeling of medical data is too professional, and the data labeling is difficult. The most difficult part of medical data labeling is the labeling of segmentation data. At present, the amount of open-source available fundus image lesion segmentation data is very limited, and the accuracy of fundus image lesion segmentation is low. SUMMARY

[0003] The present application aims to at least solve the technical problems existing in the prior art. To this end, the present application provides a fundus image lesion segmentation method, system, device and storage medium, which can improve the accuracy of fundus image lesion segmentation.

[0004] In a first aspect of the present application, a fundus image lesion segmentation method is provided, comprising the following steps:

[0005] Obtaining a historical fundus image dataset and a to-be-predicted fundus image, wherein the historical fundus image dataset comprises a historical fundus image, a true classification label corresponding to the historical fundus image, and a segmentation mask corresponding to the historical fundus image;

[0006] Building a first classification network model, inputting the historical fundus image and the true classification label into the first classification network model for training to obtain a second classification network model;

[0007] Building a first lesion segmentation network model, training the first lesion segmentation network model according to the second classification network model, the historical fundus image and the segmentation mask corresponding to the historical fundus image to obtain a second lesion segmentation network model;

[0008] Inputting the to-be-predicted fundus image into the second lesion segmentation network model for prediction to obtain lesion segmentation information of the to-be-predicted fundus image.

[0009] According to the control method of the embodiment of the present application, at least the following beneficial effects are achieved:

[0010] The method comprises the following steps: acquiring a historical fundus image dataset and a fundus image to be predicted, wherein the historical fundus image dataset comprises historical fundus images, real classification labels corresponding to the historical fundus images, and segmentation mask images corresponding to the historical fundus images; constructing a first classification network model; inputting the historical fundus images and the real classification labels into the first classification network model for training to obtain a second classification network model; constructing a first lesion segmentation network model; training the first lesion segmentation network model according to the second classification network model, the historical fundus images and the segmentation mask images corresponding to the historical fundus images to obtain a second lesion segmentation network model; inputting the fundus image to be predicted into the second lesion segmentation network model for prediction to obtain lesion segmentation information of the fundus image to be predicted; and correlating segmentation information features on a classification task, and providing a supervision signal and prior knowledge for training of the segmentation model through the classification network model, so as to improve the lesion segmentation accuracy of the fundus image.

[0011] According to some embodiments of the present application, the first lesion segmentation network model comprises a VGG-19 encoder, and the step of constructing the first lesion segmentation network model and training the first lesion segmentation network model according to the second classification network model, the historical fundus images and the segmentation mask images corresponding to the historical fundus images to obtain a second lesion segmentation network model comprises the following steps:

[0012] inputting the historical fundus images and the segmentation mask images into the first lesion segmentation network model for training to obtain a first segmentation model and a first segmentation prediction value output by the first segmentation model;

[0013] extracting high-dimensional feature information through the VGG-19 encoder according to the first segmentation prediction value and the segmentation mask image;

[0014] calculating a first segmentation feature loss value through a segmentation feature perception loss function according to the high-dimensional feature information and the segmentation mask image;

[0015] calculating a second segmentation feature loss value according to the first segmentation prediction value and the segmentation mask image;

[0016] calculating a first total loss value according to the first segmentation feature loss value and the second segmentation feature loss value;

[0017] when the first total loss value is less than a preset total loss value, the first segmentation model is the second lesion segmentation network model;

[0018] When the first total loss value is greater than the preset total loss value, the first segmentation model is updated according to the first total loss value, the second classification network model, the historical fundus image and the segmentation mask graph, to obtain a second segmentation model and a second segmentation prediction value output by the second segmentation model, loss value calculation is performed according to the second segmentation prediction value and the segmentation mask graph to obtain a second total loss value; when the second total loss value is less than the preset total loss value, the second segmentation model is the second lesion segmentation network model; when the second total loss value is greater than the preset total loss value, the second segmentation model is updated according to the second total loss value, the second classification network model, the historical fundus image and the segmentation mask graph to obtain a third segmentation model and a third segmentation prediction value output by the third segmentation model, and the process is repeated until the qth total loss value is less than the preset total loss value, to obtain the second lesion segmentation network model, wherein q is the number of iterations.

[0019] According to some embodiments of the application, the loss value calculation according to the high-dimensional feature information and the segmentation mask graph through the segmentation feature perception loss function obtains a first segmentation feature loss value, and the calculation formula is:

[0020]

[0021] L1=λ1L sp is the first segmentation feature loss value, W is the dimension of the feature map output by the feature extraction layer of VGG-19, H is the dimension of the feature map output by the feature extraction layer of VGG-19, φ i is the i-th layer feature extraction layer of VGG-19, n is the n-th point in the high-dimensional feature map, x n is the true label of the segmentation mask graph, p n is the segmentation prediction value of the n-th pixel point.

[0022] According to some embodiments of the application, the loss value calculation according to the first segmentation prediction value and the segmentation mask graph obtains a second segmentation feature loss value, and the calculation formula is:

[0023]

[0024]

[0025] L2=λ2L MAE +λ3L dice

[0026] L1=λ1L MAE is the MAELoss loss value, L dice is the DiceLoss loss value, L2 is the segmentation feature loss value, N is the total number of pixel points of the fundus image cup-disc segmentation graph, r na true label of an nth pixel in the optic disc segmentation map, p n a segmentation prediction value of the nth pixel, ε is a preset constant, ∈ is a preset constant, λ2 is a preset second hyperparameter, and λ3 is a preset third hyperparameter.

[0027] According to some embodiments of the present application, the total loss calculation is performed according to the first segmentation feature loss value and the second segmentation feature loss value to obtain a first total loss value, which includes:

[0028] The first segmentation feature loss value is multiplied by a preset first hyperparameter to obtain a first loss value.

[0029] The first loss value is added to the second segmentation feature loss value to obtain the first total loss value.

[0030] According to some embodiments of the present application, the historical fundus images and the true classification labels are input into the first classification network model for training to obtain a second classification network model, which includes:

[0031] The historical fundus images and the true classification labels are input into the first classification network model for training to obtain a first training model and a first classification prediction value output by the first training model.

[0032] A first cross-entropy loss value is obtained by performing loss value calculation on the first classification prediction value and the true classification label through a cross-entropy loss function.

[0033] When the first cross-entropy loss value is less than a preset cross-entropy loss value, the first training model is the second classification network model.

[0034] When the first cross-entropy loss value is greater than the preset cross-entropy loss value, a second training model and a second classification prediction value output by the second training model are obtained by updating the first training model according to the first cross-entropy loss value, the historical fundus images, and the true classification labels, a second cross-entropy loss value is obtained by performing loss value calculation on the second classification prediction value and the true classification label through a cross-entropy loss function, when the second cross-entropy loss value is less than a preset cross-entropy loss value, the second training model is the second classification network model, when the second cross-entropy loss value is greater than the preset cross-entropy loss value, a third training model and a third classification prediction value output by the third training model are obtained by updating the second training model according to the second cross-entropy loss value, the historical fundus images, and the true classification labels, and so on, until a kth cross-entropy loss value is less than a preset cross-entropy loss value, to obtain the second classification network model, where k is the number of iterations.

[0035] According to some embodiments of the present application, the loss value calculation is performed by a cross-entropy loss function according to the first classification prediction value and the true classification label, and the calculation formula of the first cross-entropy loss value is as follows:

[0036]

[0037] wherein L ce is the cross-entropy loss value, N is the total number of fundus images, M is the total number of fundus image classification categories, y ic is the true value of the i-th fundus image, and 0 otherwise, p ic is the prediction probability of the i-th fundus image belonging to category c.

[0038] In a second aspect of the present application, a fundus image lesion segmentation system is provided, which comprises:

[0039] A data acquisition module is configured to acquire historical project defect data and to-be-predicted project data.

[0040] An internal feature calculation module is configured to construct a first graph attention network model, to input the historical project defect data into the first graph attention network model for training, to obtain a second graph attention network model and first internal features output by the second graph attention network model, and to input the to-be-predicted project data into the second graph attention network model for prediction, to obtain second internal features.

[0041] An external feature calculation module is configured to construct a first scalable graph network model, to input the historical project defect data into the first scalable graph network model for training, to obtain a second scalable graph network model and first external features output by the second scalable graph network model, and to input the to-be-predicted project data into the second scalable graph network model for prediction, to obtain second external features.

[0042] A sample acquisition module is configured to fuse the first internal features and the first external features to obtain first fusion features, to match the first fusion features with defect information of the historical project defect data to obtain training samples, and to fuse the second internal features and the second external features to obtain to-be-predicted samples.

[0043] A multi-prediction model training module is configured to construct a first multi-prediction model, to input the training samples into the first multi-prediction model for training, and to obtain a second multi-prediction model.

[0044] A defect information prediction module is configured to input the to-be-predicted samples into the second multi-prediction model for prediction, and to obtain defect information of the to-be-predicted project data.

[0045] The system acquires a historical fundus image data set and a to-be-predicted fundus image, wherein the historical fundus image data set comprises historical fundus images, real classification labels corresponding to the historical fundus images, and segmentation mask images corresponding to the historical fundus images, a first classification network model is constructed, the historical fundus images and the real classification labels are input into the first classification network model for training, and a second classification network model is obtained; a first lesion segmentation network model is constructed, and the first lesion segmentation network model is trained according to the second classification network model, the historical fundus images and the segmentation mask images corresponding to the historical fundus images, and a second lesion segmentation network model is obtained; the to-be-predicted fundus image is input into the second lesion segmentation network model for prediction, and lesion segmentation information of the to-be-predicted fundus image is obtained, the segmentation information features on the classification task are associated, and the classification network model is used to provide a supervision signal and prior knowledge for training of the segmentation model, so that the lesion segmentation accuracy of the fundus image is improved.

[0046] In a third aspect, the present application provides an electronic device for fundus image lesion segmentation, comprising at least one control processor and a memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the fundus image lesion segmentation method described above.

[0047] In a fourth aspect, the present application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the fundus image lesion segmentation method described above.

[0048] It should be noted that the beneficial effects of the second to fourth aspects of the present application and the prior art are the same as those of the above-mentioned fundus image lesion segmentation system and the prior art, which will not be described here.

[0049] Additional aspects and advantages of the present application will be partially given in the following description, partially will become obvious from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0050] The above and / or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

[0051] Figure 1 is a flowchart of a fundus image lesion segmentation method according to an embodiment of the present application;

[0052] Figure 2 is a classification network model structure schematic diagram of a fundus image lesion segmentation method according to an embodiment of the present application;

[0053] Figure 3 Fig. 1 is a structure schematic diagram of a lesion segmentation network model of an eye fundus image lesion segmentation method provided by an embodiment of the present application;

[0054] Figure 4 Fig. 2 is a segmentation effect schematic diagram of an eye fundus image lesion segmentation method provided by an embodiment of the present application;

[0055] Figure 5 Fig. 3 is a structure schematic diagram of an eye fundus image lesion segmentation system according to an embodiment of the present application. DETAILED DESCRIPTION

[0056] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary, only for explaining the present application, and cannot be understood as limiting the present application.

[0057] In the description of the present application, if the first, second, etc. are described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features or the sequence of the indicated technical features.

[0058] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc. is based on the orientation or position relationship shown in the drawings, only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0059] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and the person skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0060] In recent years, artificial intelligence technology has been widely applied in various fields of daily life, such as smart city, intelligent retail and smart home, etc., providing various services and conveniences for daily life. In addition, artificial intelligence has gradually developed in the medical field. The current artificial intelligence technology highly depends on the quantity and quality of data, and the more training data, the better the effect of the algorithm model trained. Since medical data involves patient privacy information compared with natural data, the acquisition of medical data is strictly controlled, and it is difficult to obtain medical data. Moreover, artificial intelligence technology requires labeled images or text information for model training, but the labeling of medical data is too professional, and the data labeling is difficult. The most difficult labeling of medical data is the labeling of segmentation data. At present, the amount of open-source available fundus image lesion segmentation data is very limited, and the fundus image lesion segmentation accuracy is low.

[0061] In order to solve the above technical defects, with reference to Figure 1 The application provides a fundus image lesion segmentation method, comprising the following steps:

[0062] Step S101, acquiring a historical fundus image data set and a to-be-predicted fundus image, wherein the historical fundus image data set comprises a historical fundus image, a real classification label corresponding to the historical fundus image and a segmentation mask corresponding to the historical fundus image;

[0063] Step S102, constructing a first classification network model, inputting the historical fundus image and the real classification label into the first classification network model for training to obtain a second classification network model;

[0064] Step S103, constructing a first lesion segmentation network model, training the first lesion segmentation network model according to the second classification network model, the historical fundus image and the segmentation mask corresponding to the historical fundus image to obtain a second lesion segmentation network model;

[0065] Step S104, inputting the to-be-predicted fundus image into the second lesion segmentation network model for prediction to obtain lesion segmentation information of the to-be-predicted fundus image.

[0066] The method comprises the following steps: acquiring a historical fundus image dataset and a fundus image to be predicted, wherein the historical fundus image dataset comprises historical fundus images, real classification labels corresponding to the historical fundus images and segmentation mask images corresponding to the historical fundus images; constructing a first classification network model; inputting the historical fundus images and the real classification labels into the first classification network model for training to obtain a second classification network model; constructing a first lesion segmentation network model; training the first lesion segmentation network model according to the second classification network model, the historical fundus images and the segmentation mask images corresponding to the historical fundus images to obtain a second lesion segmentation network model; inputting the fundus image to be predicted into the second lesion segmentation network model for prediction to obtain lesion segmentation information of the fundus image to be predicted; correlating segmentation information features on a classification task; and providing a supervision signal and prior knowledge for training of the segmentation model through the classification network model, so as to improve the lesion segmentation accuracy of the fundus image.

[0067] With reference to Figure 3 In some embodiments, the first lesion segmentation network model comprises a VGG-19 encoder, and the construction of the first lesion segmentation network model, the training of the first lesion segmentation network model according to the second classification network model, the historical fundus images and the segmentation mask images corresponding to the historical fundus images to obtain the second lesion segmentation network model comprises the following steps:

[0068] inputting the historical fundus images and the segmentation mask images into the first lesion segmentation network model for training to obtain a first segmentation model and a first segmentation prediction value output by the first segmentation model;

[0069] extracting high-dimensional feature information through the VGG-19 encoder according to the first segmentation prediction value and the segmentation mask image;

[0070] calculating a first segmentation feature loss value through a segmentation feature perception loss function according to the high-dimensional feature information and the segmentation mask image;

[0071] calculating a second segmentation feature loss value according to the first segmentation prediction value and the segmentation mask image;

[0072] calculating a first total loss value according to the first segmentation feature loss value and the second segmentation feature loss value;

[0073] when the first total loss value is less than a preset total loss value, the first segmentation model is the second lesion segmentation network model;

[0074] When the first total loss value is greater than the preset total loss value, the first segmentation model is updated according to the first total loss value, the second classification network model, the historical fundus image and the segmentation mask map, to obtain a second segmentation model and a second segmentation prediction value output by the second segmentation model, loss value calculation is performed according to the second segmentation prediction value and the segmentation mask map, to obtain a second total loss value; when the second total loss value is less than the preset total loss value, the second segmentation model is the second lesion segmentation network model; when the second total loss value is greater than the preset total loss value, the second segmentation model is updated according to the second total loss value, the second classification network model, the historical fundus image and the segmentation mask map, to obtain a third segmentation model and a third segmentation prediction value output by the third segmentation model, and the process is repeated until the qth total loss value is less than the preset total loss value, to obtain the second lesion segmentation network model, wherein q is the number of iterations.

[0075] In some embodiments, loss value calculation is performed according to the high-dimensional feature information and the segmentation mask map through a segmentation feature perception loss function, and the calculation formula of the first segmentation feature loss value is:

[0076]

[0077] wherein, L sp is the first segmentation feature loss value, W is the dimension of the feature map output by the feature extraction layer of VGG-19, H is the dimension of the feature map output by the feature extraction layer of VGG-19, φ i is the i-th layer feature extraction layer of VGG-19, n is the n-th point in the high-dimensional feature map, x n is the true label of the segmentation mask map, p n is the segmentation prediction value of the n-th pixel.

[0078] In some embodiments, loss value calculation is performed according to the first segmentation prediction value and the segmentation mask map, and the calculation formula of the second segmentation feature loss value is:

[0079]

[0080]

[0081] L2=λ2L MAE +λ3L dice

[0082] wherein, L MAE is the MAELoss loss value, L dice is the DiceLoss loss value, L2 is the segmentation feature loss value, N is the total number of pixel points of the optic cup and disc segmentation map of the fundus image, r n is the true label of the n-th pixel in the optic cup and disc segmentation map, p nis a preset constant, ∈ is a preset constant, λ2 is a preset second hyperparameter, and λ3 is a preset third hyperparameter.

[0083] In some embodiments, the total loss calculation is performed according to the first segmentation feature loss value and the second segmentation feature loss value to obtain a first total loss value, including:

[0084] The first segmentation feature loss value is multiplied by a preset first hyperparameter to obtain a first loss value.

[0085] The first loss value is added to the second segmentation feature loss value to obtain the first total loss value.

[0086] Reference Figure 2 In some embodiments, the historical fundus image and the real classification label are input into the first classification network model for training to obtain a second classification network model, including:

[0087] The historical fundus image and the real classification label are input into the first classification network model for training to obtain a first training model and a first classification prediction value output by the first training model.

[0088] The first classification prediction value and the real classification label are used to calculate a loss value by a cross-entropy loss function to obtain a first cross-entropy loss value.

[0089] When the first cross-entropy loss value is less than a preset cross-entropy loss value, the first training model is the second classification network model.

[0090] When the first cross-entropy loss value is greater than the preset cross-entropy loss value, the first training model is updated according to the first cross-entropy loss value, the historical fundus image, and the real classification label to obtain a second training model and a second classification prediction value output by the second training model. The second classification prediction value and the real classification label are used to calculate a loss value by a cross-entropy loss function to obtain a second cross-entropy loss value. When the second cross-entropy loss value is less than the preset cross-entropy loss value, the second training model is the second classification network model. When the second cross-entropy loss value is greater than the preset cross-entropy loss value, the second training model is updated according to the second cross-entropy loss value, the historical fundus image, and the real classification label to obtain a third training model and a third classification prediction value output by the third training model. This process is repeated until the kth cross-entropy loss value is less than the preset cross-entropy loss value, and the second classification network model is obtained, where k is the number of iterations.

[0091] In some embodiments, the calculation formula of the first cross-entropy loss value calculated according to the first classification prediction value and the real classification label by the cross-entropy loss function is:

[0092]

[0093] where Lce is the cross-entropy loss value, N is the total number of fundus images, M is the total number of fundus image classification categories, y ic is the true value of the i-th fundus image, otherwise 0, p ic is the predicted probability of the i-th fundus image belonging to category c.

[0094] Specifically, a set of best embodiments is provided below:

[0095] Step one: select lesion classification pre-training training data, and divide the training data and test data. The present application takes diabetic retinal lesion segmentation as an example for description, and other lesion segmentation can be similarly applied to the scene. The segmentation task is diabetic retinal lesion segmentation, so the lesion classification pre-training data set needs to select the diabetic retinal lesion classification data set. The diabetic retinal lesion classification data set selects a private annotation data set, that is, based on the open source DRD (kaggle) diabetic retinopathy classification data set, the fundus image is labeled at the image level whether there is hard exudation, soft exudation, hemorrhage and microaneurysm, and finally the diabetic retinal lesion classification data set is obtained. The diabetic retinal lesion classification data set contains 30000 fundus images, each of which has hard exudation, soft exudation, hemorrhage and microaneurysm annotation label. The diabetic retinal lesion classification data set is divided according to 80%:20%, 80% is divided into training set, and 20% is divided into test set, that is, the training set data amount is 24000, and the test set data amount is 6000.

[0096] Step two: refer to Figure 2 , build a lesion classification pre-training network. The lesion classification pre-training network in the present application is composed of an encoder and an output projection layer, wherein the encoder adopts VGG-19, and the output projection layer is composed of three fully connected layers in series, and finally outputs a one-dimensional vector (1x4).

[0097] Step three: lesion classification training data preprocessing. The present application transforms the fundus image into 224x224 from the original width height through histogram scaling operation, and the training stage will have a random horizontal flip data enhancement strategy.

[0098] Step four: training the lesion classification pre-training network. The loss function of the lesion classification pre-training network in the present application in the training stage is composed of cross-entropy loss function (CELoss), and the specific expression of CELoss is as follows:

[0099]

[0100] Wherein, L ce is the cross-entropy loss value, N is the total number of fundus images, M is the total number of fundus image classification categories, y icpi is the predicted probability that the i-th fundus image belongs to class c. ic pi is the predicted probability that the i-th fundus image belongs to class c.

[0101] The present application carries out lesion classification pre-training network training on a workbench installed with an NVIDIA-A100 GPU, the programming language is Python 3.8, and the PyTorch 1.13 deep learning framework is adopted. In the training stage, the BatchSize is set to 64, the optimizer adopts AdamW, a total of 100 rounds of training, the initial learning rate is set to 10 -4 , the learning rate is adjusted in the cosine annealing manner, and the terminal learning rate is 10 -6 . The model with the minimum loss value of the lesion classification pre-training network on the test data is obtained as the final lesion classification pre-training network.

[0102] Step five: select the lesion segmentation dataset and divide the training data and test data. The present application takes diabetic retinal lesion segmentation as an example for description, and other dense lesions can be similarly applied to the scene. The diabetic retinal lesion segmentation dataset selects the currently published DDR dataset, which contains 757 fundus images and annotations of hard exudation, soft exudation, microaneurysm and hemorrhage lesions in the fundus images. In the data preparation stage, the annotation data is converted, that is, the annotated lesions are replaced by different pixel values, wherein the background is represented by pixel value 0, the hard exudation lesion is represented by pixel value 1, the soft exudation lesion is represented by pixel value 2, the microaneurysm lesion is represented by pixel value 3, and the hemorrhage lesion is represented by pixel value 4. The mask information of the lesion annotation is obtained through the above conversion. Then, the DDR dataset is divided to obtain 680 training data and 77 test data.

[0103] Step six: refer to Figure 3 , and build the lesion segmentation network. The lesion segmentation network in the present application is composed of an encoder, a decoder and a segmentation head. The lesion segmentation network structure diagram is shown in Figure 3 . The encoder in the lesion segmentation network is ResNet50, the decoder is UNet++, and the segmentation head is composed of a convolution layer with a kernel size of 3x3 and an up-sampling layer. The segmentation head can restore the width and height of the output feature map to the original input image width and height, and the channel number of the output feature map is consistent with the class number, which is 5 here.

[0104] Step seven: use the lesion classification pre-training model to construct a segmentation feature perception loss function. Take the encoder VGG-19 in the lesion classification pre-training network trained in step four and freeze the weight parameters of VGG19, use the encoder of the lesion classification pre-training network to extract high-dimensional features from the predicted image and the mask image of the lesion segmentation respectively, and then perform supervised training. The specific expression of the segmentation feature perception loss function (SPLoss) is as follows:

[0105]

[0106] wherein, L sp is the first segmentation feature loss value, W is the dimension of the feature map output by the feature extraction layer of VGG-19, H is the dimension of the feature map output by the feature extraction layer of VGG-19, and φ i is the i-th layer feature extraction layer of VGG-19, n is the n-th point in the high-dimensional feature map, x n is the true label of the segmentation mask image, p n is the segmentation prediction value of the n-th pixel.

[0107] Step eight: online processing of lesion segmentation data. The present application will perform online scaling of the fundus image and the corresponding lesion mask information, and will change the original width and height to 512x512. At the same time, there will be an online random horizontal flip data augmentation strategy in the training stage.

[0108] Step nine: training the lesion segmentation network. In the present application, SPLoss is introduced in the loss function in the training stage of the lesion segmentation network. In addition to SPLoss, the loss function also uses MAELoss and DiceLoss. The expression of MAELoss is as follows:

[0109]

[0110] The expression of DiceLoss is as follows:

[0111]

[0112] wherein, L MAE is the MAELoss loss value, L dice is the DiceLoss loss value, N is the total number of pixel points of the fundus image cup-disc segmentation image, r n is the true label of the n-th pixel in the cup-disc segmentation image, p n is the segmentation prediction value of the n-th pixel, and ε is a preset constant.

[0113] In summary, the loss function of the lesion segmentation network model is composed of SPLoss, MAELoss and DiceLoss three loss functions, and the expression of the loss function is as follows:

[0114] L Loc = λ1L sp + λ2L MAE + λ3L dice

[0115] Wherein, λ1, λ2, λ3 are hyperparameters for controlling the weight of different losses.

[0116] The application trains a diabetic retinopathy lesion segmentation network on a workbench installed with an NVIDIA-A100 GPU, the programming language is Python 3.8, and the Pytorch 1.13 deep learning framework is adopted. In the training stage, the Batchsize is set to 16, the optimizer adopts AdamW, the training is performed for 50 rounds, the initial learning rate is set to 10 -4 , the learning rate is adjusted in the cosine annealing manner, and the final learning rate is 10 -6 . The model with the minimum loss value of the lesion segmentation network on the test data is obtained as the final lesion segmentation network.

[0117] The segmentation effect of the diabetic retinopathy lesion segmentation network on the fundus image is as shown in Figure 4 .

[0118] The results of the lesion segmentation network on the diabetic retinopathy lesion segmentation test data are shown in Table 1, and the index in the table is Dice:

[0119] Table 1

[0120] Loss Hard Exudation Soft Exudation Microaneurysm Hemorrhage Point MAELoss + DiceLoss 0.8342 0.8019 0.6732 0.7992 MAELoss + DiceLoss + SPLoss 0.8543 0.8125 0.6993 0.8054

[0121] As can be seen from Table 1, the lesion segmentation network trained on the diabetic retinopathy segmentation data set has obvious improvement in the Dice index of the segmentation of the four lesions after adding the segmentation feature perception loss function in the training, and the segmentation effect is obviously improved.

[0122] In addition, according to Figure 5 , one embodiment of the application provides a fundus image lesion segmentation system, comprising a data acquisition module 1100, an image classification module 1200, an image lesion segmentation module 1300 and an image prediction module 1400, wherein:

[0123] The data acquisition module 1100 is used for acquiring historical fundus image data set and to-be-predicted fundus image, wherein the historical fundus image data set comprises historical fundus image, real classification label corresponding to the historical fundus image and segmentation mask corresponding to the historical fundus image.

[0124] The image classification module 1200 is used for constructing a first classification network model, inputting the historical fundus image and the real classification label into the first classification network model for training, and obtaining a second classification network model;

[0125] The image lesion segmentation module 1300 is configured to construct a first lesion segmentation network model, train the first lesion segmentation network model according to the second classification network model, the historical fundus image and the segmentation mask corresponding to the historical fundus image, and obtain a second lesion segmentation network model.

[0126] The image prediction module 1400 is configured to input the to-be-predicted fundus image into the second lesion segmentation network model for prediction to obtain lesion segmentation information of the to-be-predicted fundus image.

[0127] The system is configured to obtain a historical fundus image dataset and a to-be-predicted fundus image, wherein the historical fundus image dataset includes historical fundus images, real classification labels corresponding to the historical fundus images and segmentation mask images corresponding to the historical fundus images, construct a first classification network model, input the historical fundus images and the real classification labels into the first classification network model for training to obtain a second classification network model, construct a first lesion segmentation network model, train the first lesion segmentation network model according to the second classification network model, the historical fundus images and the segmentation mask images corresponding to the historical fundus images to obtain a second lesion segmentation network model, input the to-be-predicted fundus image into the second lesion segmentation network model for prediction to obtain lesion segmentation information of the to-be-predicted fundus image, associate segmentation information features on a classification task, and provide a supervision signal and prior knowledge for training of the segmentation model through the classification network model, thereby improving the fundus image lesion segmentation accuracy.

[0128] It should be noted that the system embodiment and the above-mentioned system embodiment are based on the same inventive concept, and therefore the related content of the above-mentioned method embodiments is also applicable to the system embodiment, which will not be described here again.

[0129] The application also provides an electronic device for fundus image lesion segmentation, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the fundus image lesion segmentation method as described above.

[0130] The processor and the memory can be connected through a bus or other means.

[0131] The memory, as a non-transitory computer readable storage medium, can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0132] The non-transitory software program and instructions required to implement the fundus image lesion segmentation method of the above embodiments are stored in the memory, and when executed by the processor, the fundus image lesion segmentation method in the above embodiments is executed, for example, the method steps S101 to S104 in the above description are executed. Figure 1

[0133] The present application also provides a computer readable storage medium storing computer executable instructions for executing the fundus image lesion segmentation method as described above.

[0134] The computer readable storage medium stores computer executable instructions, which are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the above processor executes the fundus image lesion segmentation method in the above embodiments, for example, the method steps S101 to S104 in the above description are executed. Figure 1

[0135] Those of ordinary skill in the art can understand that all or some steps in the above disclosed method and system can be implemented as software, firmware, hardware and appropriate combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program units or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. In addition, as known to those of ordinary skill in the art, communication media generally includes computer readable instructions, data structures, program units or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery medium.

[0136] The embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the above embodiments, and various changes can be made within the knowledge of those of ordinary skill in the art without departing from the purpose of the present application.​​

Claims

1. A method for segmenting lesions in fundus images, characterized in that, The fundus image lesion segmentation method comprises: obtain a historical fundus image dataset and a fundus image to be predicted, wherein the historical fundus image dataset comprises a historical fundus image, a true classification label corresponding to the historical fundus image, and a segmentation mask corresponding to the historical fundus image; a first classification network model is constructed, the historical fundus image and the true classification label are input into the first classification network model for training, and a second classification network model is obtained; a first lesion segmentation network model is constructed, the second classification network model, the historical fundus image and the segmentation mask corresponding to the historical fundus image are used to train the first lesion segmentation network model, and a second lesion segmentation network model is obtained, wherein the first lesion segmentation network model comprises a VGG-19 encoder, and specifically comprises: the historical fundus image and the segmentation mask are input into the first lesion segmentation network model for training, and a first segmentation model and a first segmentation prediction value output by the first segmentation model are obtained; high-dimensional feature information is obtained by high-dimensional feature extraction of the VGG-19 encoder based on the first segmentation prediction value and the segmentation mask; a first segmentation feature loss value is obtained by loss value calculation of the segmentation feature perception loss function based on the high-dimensional feature information and the segmentation mask; a second segmentation feature loss value is obtained by loss value calculation of the first segmentation prediction value and the segmentation mask; a first total loss value is obtained by total loss calculation of the first segmentation feature loss value and the second segmentation feature loss value; when the first total loss value is less than a preset total loss value, the first segmentation model is the second lesion segmentation network model; when the first total loss value is greater than the preset total loss value, the second segmentation model and a second segmentation prediction value output by the second segmentation model are obtained by updating the first segmentation model based on the first total loss value, the second classification network model, the historical fundus image and the segmentation mask, loss value calculation is performed based on the second segmentation prediction value and the segmentation mask, and a second total loss value is obtained; when the second total loss value is less than the preset total loss value, the second segmentation model is the second lesion segmentation network model; when the second total loss value is greater than the preset total loss value, a third segmentation model and a third segmentation prediction value output by the third segmentation model are obtained by updating the second segmentation model based on the second total loss value, the second classification network model, the historical fundus image and the segmentation mask, and so on, until a qth total loss value is less than the preset total loss value, and the second lesion segmentation network model is obtained, wherein q is the number of iterations; the fundus image to be predicted is input into the second lesion segmentation network model for prediction, and lesion segmentation information of the fundus image to be predicted is obtained.

2. The fundus image lesion segmentation method of claim 1, wherein, The calculation formula of the first segmentation feature loss value obtained by loss value calculation of the high-dimensional feature information and the segmentation mask through the segmentation feature perception loss function is: wherein L sp is the first segmentation feature loss value, W is the dimension of the feature map output by the feature extraction layer of VGG-19, H is the dimension of the feature map output by the feature extraction layer of VGG-19, φ i is the i-th layer feature extraction layer of VGG-19, n is the n-th point in the high-dimensional feature map, x n is the true label of the segmentation mask map, p n is the segmentation prediction value of the n-th pixel point.

3. The fundus image lesion segmentation method of claim 2, wherein, The calculation formula of the second segmentation feature loss value obtained by loss value calculation according to the first segmentation prediction value and the segmentation mask graph is: L2 = λ2L MAE + λ3L dice wherein, L MAE is a MAELoss loss value, L dice is a DiceLoss loss value, L2 is a segmentation feature loss value, N is a total number of pixel points of the optic cup and optic disc segmentation map of the fundus image, r n is a true label of the nth pixel in the optic cup and optic disc segmentation map, p n is a segmentation prediction value of the nth pixel point, ε is a preset constant, ∈ is a preset constant, λ2 is a preset second hyperparameter, and λ3 is a preset third hyperparameter.

4. The fundus image lesion segmentation method of claim 3, wherein, The first total loss value is obtained by total loss calculation according to the first segmentation feature loss value and the second segmentation feature loss value, including: The first loss value is obtained by multiplying the first segmentation feature loss value by a preset first hyperparameter; The first total loss value is obtained by adding the first loss value and the second segmentation feature loss value.

5. The fundus image lesion segmentation method of claim 1, wherein, The second classification network model is obtained by inputting the historical fundus image and the real classification label into the first classification network model for training, including: The first training model and the first classification prediction value output by the first training model are obtained by inputting the historical fundus image and the real classification label into the first classification network model for training; The first cross-entropy loss value is obtained by loss value calculation according to the first classification prediction value and the real classification label through a cross-entropy loss function; When the first cross-entropy loss value is less than a preset cross-entropy loss value, the first training model is the second classification network model; When the first cross-entropy loss value is greater than the preset cross-entropy loss value, the second training model and the second classification prediction value output by the second training model are obtained by updating the first training model according to the first cross-entropy loss value, the historical fundus image and the real classification label, and the second cross-entropy loss value is obtained by loss value calculation according to the second classification prediction value and the real classification label through a cross-entropy loss function; when the second cross-entropy loss value is less than a preset cross-entropy loss value, the second training model is the second classification network model; when the second cross-entropy loss value is greater than the preset cross-entropy loss value, the third training model and the third classification prediction value output by the third training model are obtained by updating the second training model according to the second cross-entropy loss value, the historical fundus image and the real classification label, and so on, until the kth cross-entropy loss value is less than a preset cross-entropy loss value, to obtain the second classification network model, wherein k is the number of iterations.

6. The fundus image lesion segmentation method of claim 5, wherein, The calculation formula of the first cross-entropy loss value obtained by loss value calculation according to the first classification prediction value and the real classification label through a cross-entropy loss function is: wherein L ce is the cross-entropy loss value, N is the total number of fundus images, M is the total number of fundus image classification categories, y ic is the true value of the i-th fundus image, otherwise 0, p ic is the predicted probability of the i-th fundus image belonging to category c.

7. An ocular fundus image lesion segmentation system characterized by comprising: The fundus image lesion segmentation system comprises: A data acquisition module is configured to acquire historical fundus image data sets and to-be-predicted fundus images, wherein the historical fundus image data sets include historical fundus images, real classification labels corresponding to the historical fundus images, and segmentation mask graphs corresponding to the historical fundus images; An image classification module is configured to construct a first classification network model, to input the historical fundus images and the real classification labels into the first classification network model for training, and to obtain a second classification network model. The image lesion segmentation module is configured to construct a first lesion segmentation network model, train the first lesion segmentation network model according to the second classification network model, the historical fundus image and a segmentation mask corresponding to the historical fundus image, and obtain a second lesion segmentation network model. The historical fundus image and the segmentation mask are input into the first lesion segmentation network model for training, and a first segmentation model and a first segmentation prediction value output by the first segmentation model are obtained. High-dimensional feature information is obtained by performing high-dimensional feature extraction on the first segmentation prediction value and the segmentation mask through the VGG-19 encoder. A first segmentation feature loss value is obtained by performing loss value calculation on the first segmentation prediction value and the segmentation mask. A second segmentation feature loss value is obtained by performing loss value calculation on the first segmentation prediction value and the segmentation mask. A first total loss value is obtained by performing total loss calculation on the first segmentation feature loss value and the second segmentation feature loss value. When the first total loss value is less than a preset total loss value, the first segmentation model is the second lesion segmentation network model. When the first total loss value is greater than the preset total loss value, a second segmentation model and a second segmentation prediction value output by the second segmentation model are obtained by updating the first segmentation model according to the first total loss value, the second classification network model, the historical fundus image and the segmentation mask, loss value calculation is performed on the second segmentation prediction value and the segmentation mask to obtain a second total loss value, and when the second total loss value is less than the preset total loss value, the second segmentation model is the second lesion segmentation network model. When the second total loss value is greater than the preset total loss value, a third segmentation model and a third segmentation prediction value output by the third segmentation model are obtained by updating the second segmentation model according to the second total loss value, the second classification network model, the historical fundus image and the segmentation mask, and the process is repeated until a qth total loss value is less than the preset total loss value, and the second lesion segmentation network model is obtained, where q is the number of iterations. The image prediction module is configured to input the to-be-predicted fundus image into the second lesion segmentation network model for prediction, and obtain lesion segmentation information of the to-be-predicted fundus image.

8. A device for segmenting lesions in fundus images, characterized in that, The at least one control processor and the memory connected in communication with the at least one control processor; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the fundus image lesion segmentation method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform the fundus image lesion segmentation method of any one of claims 1 to 6.

Images