A method for locating a lesion region of an uwf fundus image based on weakly supervised learning

Abstract

The application provides a kind of UWF fundus image lesion area positioning method based on weakly supervised learning, comprising: obtaining UWF fundus image;ResNet convolutional neural network model is constructed;Obtain multi-scale feature data;Obtain global attention feature data;The multi-scale feature data and global attention feature data are added to obtain feature saliency data;Filtering;Positioning of lesion area;Each of the positioned data is marked;Probabilistic mapping is carried out on the data with marks;Classifier is constructed, and the mapped data and the data with marks are trained;The lesion area in test image set is positioned and predicted by the trained classifier;The application only needs to carry out simple classification label to mark the model, form weakly supervised learning, and can effectively position the lesion area at last, and artificial marking is not needed on the scale of fundus image pixel level, and the artificial cost is greatly reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ophthalmology, in particular to a UWF fundus image lesion region positioning method based on weakly supervised learning. BACKGROUND

[0002] The occurrence of visual impairment is highly related to fundus diseases such as diabetic retinopathy, age-related macular degeneration, retinal detachment, etc. Patients may experience sudden and painless blurred vision, loss of monocular vision, flashes and floaters in the visual field, etc. Related studies have shown that diabetic retinopathy is a microvascular complication caused by diabetes and has become the leading cause of blindness and visual impairment worldwide. Age-related macular degeneration is a serious, irreversible eye disease that damages vision. In the early and middle stages, there is little or only subtle visual change, and eventually the patient will experience central visual impairment. Retinal detachment can cause the retina to separate from the fundus, and if not treated promptly, it can lead to permanent vision loss. Therefore, positioning the lesion area of these types of eye diseases can help doctors better understand the extent and range of the patient's fundus lesions, so as to develop targeted treatment plans.

[0003] Clinically, fundus color photographs are mainly used to diagnose diseases, which are mainly divided into traditional fundus images and ultra-wide field (UWF) fundus images. Traditional fundus images can only observe the 30°-60° range of the fundus, while UWF images can observe the 200° or so range of the fundus, providing more clinically relevant pathological information about the retina periphery. However, during the processing of UWF images, manual feature extraction is often required, which usually presents low reproducibility and high variability, with low accuracy, and cannot provide high-confidence auxiliary diagnostic information for doctors. In addition, although recent methods have achieved high diagnostic accuracy, doctors cannot trust the results of these algorithms for diagnostic decisions due to the lack of explainability. One potential solution to this problem is to directly train a model to locate the lesion area through a segmentation algorithm. However, accurately doing this requires a large amount of disease positioning annotations by clinical experts, which is time-consuming and labor-intensive. To address these issues, the present application proposes a new neural network based on weakly supervised learning to locate the lesion area of UWF images. SUMMARY

[0004] The problem solved by the present application is to provide a UWF fundus image lesion region positioning method based on weakly supervised learning, which can focus on more small-sized lesion regions in the fundus image while considering the overall lesion region. Only simple classification labels are needed for labeling, and the model is trained to predict and locate the lesion area, which does not require the large amount of manual labeling required by general supervised learning, reducing labor costs.

[0005] To solve the above problems, the application provides a UWF fundus image lesion region positioning method based on weakly supervised learning, comprising the following steps:

[0006] S1, obtaining a UWF fundus image, dividing the UWF fundus image into a training image set and a test image set;

[0007] S2, constructing a ResNet convolutional neural network model, inputting the training image set into the ResNet convolutional neural network model; presetting a first path and a second path according to the number of layers of the ResNet convolutional neural network model, the first path and the second path are both used for outputting feature data of each image in the training image set;

[0008] S3, constructing a feature fusion module based on the ResNet convolutional neural network model, used for receiving the feature data in the first path and processing the feature data in the first path into multi-scale feature data; constructing a global attention module based on the ResNet convolutional neural network model, used for receiving the feature data in the second path and processing the feature data in the second path into global attention feature data;

[0009] S4, adding the multi-scale feature data and the global attention feature data to obtain feature saliency data of each image in the training image set;

[0010] S5, sequentially performing convolution and activation operations on the feature saliency data to obtain feature saliency filtering data;

[0011] S6, constructing a lesion positioning module based on the ResNet convolutional neural network model, used for positioning the lesion region in the feature saliency filtering data to obtain positioned feature saliency filtering data of each image in the training image set; according to different lesion types, each positioned feature saliency filtering data is labeled, the type of the label corresponds to the type of the lesion, and labeled feature saliency filtering data of each image in the training image set is obtained;

[0012] S7, constructing an aggregation function, and performing probability mapping on the labeled feature saliency filtering data through the aggregation function to obtain mapped feature saliency filtering data of each image in the training image set;

[0013] S8, constructing a classifier based on the ResNet convolutional neural network model, training the mapped feature saliency filtering data and the labeled feature saliency filtering data to obtain a trained classifier;

[0014] S9, positioning the lesion region in the test image set through the trained classifier to obtain a positioning result.

[0015] Further, in step S2, the ResNet convolutional neural network model can obtain different features of the UWF fundus image from shallow to deep layers in sequence.

[0016] Further, in step S2, the feature data output is specifically: according to different resolutions of each image in the training image set, four groups of feature data F are generated through four ResBlocks of the ResNet convolutional neural network model, which are A1, A2, A3 and A4 respectively, wherein F∈R C×H×W , C is the number of channels, and HxW is the size of the A1, A2, A3 or A4 image.

[0017] Further, in step S3, the feature data in the first path is processed into multi-scale feature data, which includes the following steps:

[0018] S31, the down-sampling step of A3 is set to 1;

[0019] S32, A1, A2, A3 and A4 are transformed through convolution operation and pooling operation of 1x1 convolution kernel, and the transformed A1, A2, A3 and A4 are marked as and wherein N=HxW, HxW is the size of the A1, A2, A3 or A4 image; i=1, 2, 3, j=4; lowercase c is the transformation mark; uppercase C represents the number of channels;

[0020] S33, the and and are respectively multiplied by matrix, T represents matrix transposition, to obtain three corresponding product matrices;

[0021] S34, according to the three product matrices, the relationship matrix M∈R N×N is calculated by the softmax function;

[0022] S35, the and are respectively multiplied by matrix M and output as A f ∈R Cn×Hn , the formula is as follows:

[0023]

[0024]

[0025] wherein φ represents convolution operation using 1x1 convolution kernel; F softwherein, represents the softmax activation function, G represents the concatenation operation, A represents the multi-scale feature data, and CFFM represents the feature fusion operation. The down-sampling step of A3 is set to 1 because the resolution is sequentially reduced according to A1, A2, A3, A4, so that the resolutions of A3 and A4 are approximately the same, and more features can be collected.

[0026] Further, in step S3, the global attention module includes a channel attention module and a spatial attention module, and the global attention feature data F s The calculation formula is as follows:

[0027] F s = F i + F j + F.

[0028] wherein, F i represents the feature data obtained by the channel attention module, F j represents the feature data obtained by the spatial attention module.

[0029] Further, the channel attention module obtains the feature data F i by the following formula:

[0030] F i = σ(Ac(GAP(F))) x F,

[0031] wherein, F i ∈ R C×H×W , C is the number of channels, and H x W is the size of the feature data F i image; σ represents a sigmoid function; GAP represents a global average pooling layer; and Ac represents two 1 x 1 fully connected layer operations using a ReLU activation function and a Sigmoid activation function, respectively.

[0032] Further, the spatial attention module obtains the feature data F j by the following formula:

[0033] F j = σ(CONV(F)) x F,

[0034] wherein, F j ∈ R C×H×W , C is the number of channels, and H x W is the size of the feature data F j image; σ represents a sigmoid function; CONV represents a convolution operation using two 1 x 1 convolution kernels, the first 1 x 1 convolution kernel outputs C / r channels after convolution operation, r represents a reduction rate of input channels, and the other 1 x 1 convolution kernel outputs one channel and a ReLU activation function after convolution operation.

[0035] Further, in step S5, the feature saliency filtered data D is obtained by the following formula:

[0036] D = σ(conv(X)), X = F s +A

[0037] wherein X is the feature saliency data, D ∈ R C×H×W , C is the number of channels, HxW is the size of the feature saliency filtered data D image; σ represents the sigmoid activation function; conv represents the convolution operation using a 1x1 convolution kernel.

[0038] Further, step S6 comprises:

[0039] S61, the positioned feature saliency map is divided into DR, AMD, RD and normal four categories according to the lesions, wherein DR represents diabetic retinopathy, AMD represents age-related macular degeneration, and RD represents retinal degeneration;

[0040] S62, set the classification label y as 0, 1, 2 and 3;

[0041] S63, mark 0, 1, 2 and 3 one by one with DR, AMD, RD and normal;

[0042] S64, obtain the labeled feature saliency filtered data D'.

[0043] Further, the formula for predicting in step S8 is:

[0044]

[0045] represents the predicted value, F g is the aggregation function, the feature saliency map F is trained by backpropagating the classification loss gradient between y and , and y represents the classification label.

[0046] Further, the expression of the aggregation function F g in step S7 is:

[0047]

[0048] wherein F g is the aggregation function, represents the predicted probability value, H + represents the position set of the top t% values in D', wherein t is a hyperparameter, HxW is the size of the labeled feature saliency filtered data D' image, k represents the horizontal coordinate axis, and n represents the vertical coordinate axis.

[0049] Compared with the traditional method, the present application has the following beneficial effects:

[0050] The present application only needs to perform simple classification labels to mark the model, forms weak supervision learning, and can effectively locate the lesion area at the end, and does not need to manually mark the pixel level scale of the fundus image, so that the artificial cost is greatly reduced. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 The flowchart of the present application is shown in the figure;

[0052] Figure 2 The schematic diagram of the lesion area positioning in the present application is shown in the figure; DETAILED DESCRIPTION

[0053] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0054] The present application comprises the following steps:

[0055] S1, obtaining a UWF fundus image, dividing the UWF fundus image into a training image set and a test image set;

[0056] S2, constructing a ResNet convolutional neural network model, inputting the training image set into the ResNet convolutional neural network model; presetting a first path and a second path according to the number of layers of the ResNet convolutional neural network model, and the first path and the second path are both used for outputting feature data of each image in the training image set;

[0057] S3, constructing a feature fusion module based on the ResNet convolutional neural network model, for receiving the feature data in the first path and processing the feature data in the first path into multi-scale feature data; constructing a global attention module based on the ResNet convolutional neural network model, for receiving the feature data in the second path and processing the feature data in the second path into global attention feature data;

[0058] S4, adding the multi-scale feature data and the global attention feature data to obtain feature saliency data of each image in the training image set;

[0059] S5, sequentially performing convolution and activation operations on the feature saliency data to obtain feature saliency filtering data;

[0060] S6, a lesion positioning module based on a ResNet convolutional neural network model is constructed, which is used for positioning the lesion area in the feature saliency filtered data, and feature saliency filtered data after positioning of each image in the training image set is obtained; according to different lesion types, each feature saliency filtered data after positioning is labeled, and the type of the label corresponds to the type of the lesion one by one, and labeled feature saliency filtered data of each image in the training image set is obtained;

[0061] S7, an aggregation function is constructed, and the labeled feature saliency filtered data is mapped by the aggregation function, and the mapped feature saliency filtered data of each image in the training image set is obtained;

[0062] S8, a classifier based on a ResNet convolutional neural network model is constructed, and the mapped feature saliency filtered data and the labeled feature saliency filtered data are trained to obtain a trained classifier;

[0063] S9, the lesion area in the test image set is positioned by the trained classifier, and a positioning result is obtained.

[0064] Specifically, the following embodiment collects 416 age-related macular degeneration images, 477 diabetic retinopathy images, 228 retinal detachment images and 449 normal images. Among them, AMD represents age-related macular degeneration, mainly showing lesions in the macular area, and the pathological characteristics mainly include drusen; DR is diabetic retinal degeneration, mainly including four kinds of lesions: microaneurysm, hemorrhage, hard exudation and soft exudation; RD represents retinal degeneration, which shows the separation of the retina and retinal pigment epithelium. The data set contains a total of 1565 UWF fundus images, which are randomly divided into training and test data according to a ratio of 4:1. Therefore, finally, 1252 training images for training and 313 test images for testing are obtained. Since the size of the original UWF fundus image is 3900x3072, which is too large for a deep neural network, the size of the original image is adjusted to 448x448. In addition, the color of the training image is enhanced by randomly changing the brightness and contrast.

[0065] The ResNet convolutional neural network model is a ResNet34 convolutional neural network model, and the lower layers of the ResNet34 convolutional neural network model can capture the detailed features of the UWF fundus image, and the deeper layers can capture the global features of the UWF fundus image.

[0066] According to the different resolutions of each image in the 1252 training images, four groups of feature data F are generated by four ResBlocks of the ResNet34 convolutional neural network model, which are A1, A2, A3 and A4 and output, wherein F∈R C×H×W, C is the number of channels, and HxW is the size of the feature data F image.

[0067] The feature fusion module processes the output feature data A1, A2, A3, and A4 in the first path into multi-scale feature data, specifically including the following steps:

[0068] S31, the down-sampling step of A3 is set to 1;

[0069] S32, A1, A2, A3, and A4 are transformed through convolution operation and pooling operation of a 1x1 convolution kernel, and the transformed A1, A2, A3, and A4 are marked as and Where N = HxW, HxW is the size of the A1, A2, A3, or A4 image; i = 1, 2, 3, and j = 4; lowercase c is the transformation mark; and uppercase C represents the number of channels;

[0070] S33, and and respectively, to obtain three corresponding product matrices;

[0071] S34, according to the three product matrices, a relationship matrix M ∈ R N×N is calculated through a softmax function;

[0072] S35, and and M are respectively multiplied by the matrix to output Af ∈ R Cn×Hn , and the formula is as follows:

[0073]

[0074]

[0075] Where φ represents convolution operation using a 1x1 convolution kernel; Fsoft represents a softmax activation function, G represents a splicing operation, A is multi-scale feature data, and CFFM represents a feature fusion operation.

[0076] The global attention module includes a channel attention module and a spatial attention module, and the global attention feature data F s The calculation formula is as follows:

[0077] F s = F i + F j + F.

[0078] Where F i represents feature data obtained by the channel attention module, and F jF represents the feature data obtained by the spatial attention module, F represents the feature data, F ∈ R C×H×W , C is the number of channels, and H × W is the size of the feature data F s of the image.

[0079] The channel attention module obtains the feature data F i by the following formula:

[0080] F i = σ(Ac(GAP(F))) × F,

[0081] where F i ∈ R C×H×W , C is the number of channels, and H × W is the size of the feature data F i of the image; σ represents a sigmoid function; GAP represents a global average pooling layer; and Ac represents two 1 × 1 fully connected layer operations using ReLU and Sigmoid activation functions, respectively. In the first fully connected layer, the number of neurons is set to rC, where r is a dimension reduction ratio and is set to 1 / 16 according to experience. In the second fully connected layer, the number of neurons is set to C. For the spatial attention module, it indicates the importance of each spatial position by learning spatial attention weights, which is complementary to the channel attention module.

[0082] The spatial attention module obtains the feature data F j by the following formula:

[0083] F j = σ(CONV(F)) × F,

[0084] where F j ∈ R C×H×W , C is the number of channels, and H × W is the size of the feature data F j of the image; σ represents a sigmoid function; and CONV represents a convolution operation using two 1 × 1 convolution kernels, the first 1 × 1 convolution kernel outputs C / r channels after convolution operation, r represents a reduction rate of input channels, and the other 1 × 1 convolution kernel outputs one channel and a ReLU activation function after convolution operation.

[0085] The multi-scale feature data A and the global attention feature data F s are added to obtain the feature saliency data X of each image in the training image set. That is, X = F s +A.

[0086] The feature saliency filtering data D is obtained by the following formula:

[0087] D = σ(conv(X)),

[0088] Wherein, X is the above characteristic saliency data, D is R C×H×W , C is the number of channels, HxW is the size of the characteristic saliency filtering data D image; Sigma represents a sigmoid activation function; Conv represents a convolution operation using a 1x1 convolution kernel.

[0089] The lesion positioning module selects different thresholds of the characteristics in the characteristic saliency filtering data D, thereby completing positioning, and obtaining the positioned characteristic saliency filtering data D".

[0090] The marking operation in step S6 includes:

[0091] S61, the positioned characteristic saliency filtering data D" is divided into DR, AMD, RD and normal four categories according to the lesions, wherein DR represents diabetic retinopathy, AMD represents age-related macular degeneration, and RD represents retinal degeneration;

[0092] S62, set the classification label y as 0, 1, 2 and 3;

[0093] S63, mark 0, 1, 2 and 3 one by one with DR, AMD, RD and normal;

[0094] S64, obtain the labeled characteristic saliency filtering data D'.

[0095] In order to obtain the lesion area information, it is difficult to directly optimize the loss function of the characteristic saliency filtering data D' and the classification label y, because the classification label y does not contain positioning information, and the present application uses an aggregation function F g to map the probability of the characteristic saliency filtering data D'. g The expression of the aggregation function F

[0096]

[0097] Wherein, F g is the aggregation function, represents the predicted probability value, H + represents the position set of the first t% value in D', wherein t is a hyperparameter, HxW is the size of the labeled characteristic saliency filtering data D' image, k represents the horizontal coordinate axis, and n represents the vertical coordinate axis. The loss gradient between and the classification label y can be completely optimized by the back propagation algorithm.

[0098] A classifier based on a ResNet convolutional neural network model is constructed, and the mapped feature saliency filtering data and the labeled feature saliency filtering data are trained to obtain a trained classifier. The lesion regions in a test image set are positioned by using the trained classifier to obtain a positioning result.

[0099] The final prediction performance is shown in the following table, and F1 represents the harmonic mean of precision and recall;

[0100] Recall represents the proportion of samples with a predicted value of 1 and a true value of 1 in all samples with a true value of 1; precision represents the proportion of samples with a predicted value of 1 and a true value of 1 in all samples with a predicted value of 1; accuracy represents the proportion of all correctly predicted samples (including positive samples and negative samples) in all samples; and Model represents the model type. It can be seen that the present application does not need to manually mark the pixel level scale of the fundus image, so that the artificial cost is greatly reduced, and the accuracy of positioning the lesion region is excellent.

[0101]

[0102] Although the present disclosure is disclosed as above, the protection scope of the present disclosure is not limited to this. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present disclosure, and these changes and modifications will fall within the protection scope of the present disclosure.

Claims

1. A method for locating a lesion region of an UWF fundus image based on weakly supervised learning, characterized in that, The method comprises the following steps: S1, obtaining a UWF fundus image, and dividing the UWF fundus image into a training image set and a test image set; S2, constructing a ResNet convolutional neural network model, inputting the training image set into the ResNet convolutional neural network model, and presetting a first path and a second path according to the number of layers of the ResNet convolutional neural network model, wherein the first path and the second path are both used to output feature data of each image in the training image set; S3, constructing a feature fusion module based on the ResNet convolutional neural network model, receiving the feature data in the first path, and processing the feature data in the first path into multi-scale feature data; constructing a global attention module based on the ResNet convolutional neural network model, receiving the feature data in the second path, and processing the feature data in the second path into global attention feature data; S4, adding the multi-scale feature data and the global attention feature data to obtain feature saliency data of each image in the training image set; S5, sequentially performing convolution and activation operations on the feature saliency data to obtain feature saliency filtering data; S6, constructing a lesion positioning module based on the ResNet convolutional neural network model, positioning a lesion region in the feature saliency filtering data to obtain positioned feature saliency filtering data of each image in the training image set, and marking each positioned feature saliency filtering data according to different lesion types, so that the types of the marks correspond to the types of the lesions one by one to obtain labeled feature saliency filtering data of each image in the training image set; S7, constructing an aggregation function, and performing probability mapping on the labeled feature saliency filtering data through the aggregation function to obtain mapped feature saliency filtering data of each image in the training image set; S8, constructing a classifier based on the ResNet convolutional neural network model, training the mapped feature saliency filtering data and the labeled feature saliency filtering data to obtain the trained classifier; S9, positioning a lesion region in the test image set through the trained classifier to obtain a positioning result.

2. The method of claim 1, wherein the method is based on weakly supervised learning. In the step S2, different features of the UWF fundus image can be obtained through the layers of the ResNet convolutional neural network model from shallow to deep.

3. The method of claim 2, wherein the method is based on weakly supervised learning. In the step S2, the feature data output is specifically as follows: According to different resolutions of each image in the training image set, four groups of feature data are generated through four layers of ResBlocks of the ResNet convolutional neural network model , , , , wherein, , is the number of channels, , , , or the size of the image.

4. The UWF fundus image lesion region positioning method based on weakly supervised learning according to claim 3, characterized in that, In the step S3, processing the feature data in the first path into multi-scale feature data specifically comprises the following steps: S31, will The downsampling step size is set to 1; S32, the convolution operation and the pooling operation are performed on the image by a 1x1 convolution kernel , , , The transformed , , and are marked as and , wherein , is , , or the size of the image , , and lowercase c is the transformation mark; uppercase C represents the number of channels; S33, will and , as well as Perform matrix multiplication operations separately. The transpose of the matrix represents the three corresponding product matrices. S34, calculating the relation matrix by a softmax function according to the three product matrices ; S35, will , as well as and Perform matrix multiplication operations respectively and output the following: The formula is as follows: ； ； wherein, denotes a convolution operation using a 1x1 convolution kernel; denotes a softmax activation function, denotes a concatenation operation, is multi-scale feature data, and CFFM denotes a feature fusion operation.

5. The UWF fundus image lesion region positioning method based on weakly supervised learning according to claim 4, characterized in that, In the step S3, the global attention module includes a channel attention module and a spatial attention module, and the global attention feature data The calculation formula is as follows: ； wherein, denotes feature data obtained by the channel attention module, denotes feature data obtained by the spatial attention module.

6. The UWF fundus image lesion region positioning method based on weakly supervised learning according to claim 5, characterized in that, The channel attention module obtains the feature data by the following formula : ； wherein, , is the number of channels, is the feature data size of the image; denotes a sigmoid function; denotes a global average pooling layer; denotes two 1 x 1 fully connected layer operations using a ReLU activation function and a Sigmoid activation function, respectively.

7. The method of claim 6, wherein the method is a weakly supervised learning based UWF fundus image lesion region positioning method. The spatial attention module obtains the feature data by the following formula : ； wherein, , is the number of channels, is the feature data the size of the image; denotes a sigmoid function; denotes a convolution operation using two 1x1 convolution kernels, the first 1x1 convolution kernel outputs C / r channels after the convolution operation, r represents the reduction rate of the input channel, and the other 1x1 convolution kernel outputs 1 channel after the convolution operation and a ReLU activation function.

8. The method of claim 7, wherein the method is a weakly supervised learning based UWF fundus image lesion region positioning method. In the step S5, the feature saliency filtered data is obtained by the following formula : ； wherein, is feature saliency data, , is a number of channels, is feature saliency filter data a size of the image; denotes a sigmoid activation function; conv denotes a convolution operation using a 1 x 1 convolution kernel.

9. The method of claim 8, wherein the method is based on weakly supervised learning. The step S6 comprises: S61, dividing the positioned feature saliency filtering data into four categories of DR, AMD, RD and normal according to the lesions, wherein DR represents diabetic retinopathy, AMD represents age-related macular degeneration, and RD represents retinal degeneration; S62, setting a classification label y as 0, 1, 2 and 3; S63, marking 0, 1, 2 and 3 and DR, AMD, RD and normal one by one. S64, obtaining labeled feature saliency filtering data .

10. The method of claim 9, wherein the method is based on weakly supervised learning. The expression of the aggregation function in the step S7 is: ； wherein, is a polymeric function, denotes a predicted probability value, denotes middle front a set of positions of values, wherein is a hyperparameter, , is a labeled feature saliency filtered data size of the image, represents an abscissa axis, represents an ordinate axis.

Images