A method for segmenting hard exudates from a retinal fundus image

Abstract

The application relates to a hard exudate segmentation method of a retinal fundus image, wherein a pretreated retinal fundus image is input into a pre-constructed and trained RMCA U-net model to obtain a hard exudate segmentation result; the RMCA U-net model comprises an encoder and a decoder; the encoder comprises multiple encoding stages, each encoding stage comprises a convolution layer, a DropBlock module, a maximum pooling layer and a double residual module which are sequentially connected, each decoder comprises multiple decoding stages, each decoding stage comprises a channel attention module, a deconvolution layer and a multi-scale feature fusion module which are sequentially connected, the channel attention module is used to utilize the most useful feature channel, and the multi-scale feature fusion module is used to guide the model to fuse multi-scale semantic information and global context features. Compared with the prior art, the application can accurately segment the fundus image under different field angles.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method for segmenting hard exudate in retinal fundus images. Background Technology

[0002] In clinical practice, hard exudates are often an important basis for diagnosing diabetic retinopathy (DR). Therefore, accurate segmentation of hard exudates is crucial for early DR diagnosis and prevention of vision loss. However, hard exudates account for a relatively small proportion of the entire retinal image, their shape is often irregular, and their contrast is usually not high enough, making them susceptible to interference from the optic disc. This can easily lead to low segmentation accuracy and efficiency.

[0003] Segmentation methods for hard exudates can be mainly divided into three categories: unsupervised, coarse-to-fine supervised, and end-to-end supervised. Unsupervised methods mainly utilize the brightness characteristics of hard exudates, relying on morphological and thresholding methods. The advantage of this type of method is that it does not require labels, but the disadvantages are low accuracy and the need to remove blood vessels and optic discs, which places high demands on preprocessing.

[0004] Coarse-to-fine supervised methods require expert physician labeling. It consists of two stages: (1) a coarse detection stage, extracting candidate detection regions for hard exudate; and (2) a fine detection stage, segmenting the hard exudate from the candidate regions. These methods typically first remove blood vessels and the optic disc from the fundus image, then use morphological operations or classification networks to select candidate regions for hard exudate, and finally use support vector machines or thresholding methods to segment the hard exudate. Compared to unsupervised methods, these methods significantly improve accuracy, but still place extremely high demands on image preprocessing. Furthermore, how to efficiently select candidate blocks of hard exudate is a problem worth considering.

[0005] End-to-end supervised methods primarily utilize deep learning models, such as variants of U-net, Fully Convolutional Residual Network (FCRN), and HED-net. However, these models only segment hard exudates in fundus images with a 45-degree or 50-degree field of view. Fundus images with a 200-degree field of view have more complex background environments and lighting conditions, therefore, current techniques are insufficient for segmenting hard exudates in fundus images with a 200-degree field of view. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art by providing a method for segmenting hard exudate in retinal fundus images.

[0007] The objective of this invention can be achieved through the following technical solutions:

[0008] This invention provides a method for segmenting hard exudate in retinal fundus images. The method involves inputting a pre-processed retinal fundus image into a pre-constructed and trained RMCAU-net model to obtain the segmentation result of hard exudate.

[0009] The RMCAU-net model includes an encoder and a decoder. The encoder includes multiple encoding stages, each of which includes a convolutional layer, a DropBlock module, a max-pooling layer, and a dual residual module connected in sequence. Each decoder includes multiple decoding stages, each of which includes a channel attention module, a deconvolutional layer, and a multi-scale feature fusion module connected in sequence. The channel attention module is used to utilize the most useful feature channels, and the multi-scale feature fusion module is used to guide the model to fuse multi-scale semantic information and global contextual features. The input of the multi-scale feature fusion module in each decoding stage includes the outputs of all encoding stages and the output of the previous decoding stage.

[0010] Preferably, the dual residual module includes two identical residual blocks, each residual block including two residual block layers connected in sequence, and each residual block layer including a normalization layer, a ReLU activation function, a 3×3 convolutional layer and a DropBlock module connected in sequence.

[0011] Preferably, the calculation formula describing the structure of each residual block is as follows:

[0012] X1 H,W,C =DB(Conv3×3(Relu(BN(X)) H,W,C )))

[0013] X2 H,W,C =DB(Conv3×3(Relu(BN(X1)) H,W,C )))

[0014] X3 H,W,C =Conv1×1(X H,W,C )+X2 H,W,C

[0015] In the formula, X H,W,C The input to this residual block is H, W, and C, which represent the height, width, and number of channels of the feature map, respectively. DB represents the DropBlock layer, and Conv3×3 and Conv1×1 represent 3×3 and 1×1 convolutions, respectively. H,W,C For the output of this residual block, X1 H,W,C and X2 H,E,C These are the outputs of the previous residual block layer and the next residual block layer, respectively.

[0016] Preferably, the channel attention module includes parallel average pooling branches and max pooling branches to obtain more feature information. The calculation formula describing the structure of the channel attention module is as follows:

[0017]

[0018]

[0019] CA(F)=σ(Con1D(F ap )+Con1D(F mp ))

[0020] F = F c +F c ×CA(F)

[0021] In the formula, F c For the input feature layer, and These represent max pooling and average pooling, respectively. σ is the sigmoid function, F is the output feature layer, and H, W, and C represent the height, width, and number of channels of the feature layer, respectively.

[0022] Preferably, the processing procedure of the multi-scale feature fusion module in the current decoding stage is as follows:

[0023] The feature layers output from all encoding stages are processed, including 1×1 convolution and DropBlock regularization operations. If the feature layer of the processed encoding stage is different in size from the feature layer output from the previous decoding stage, it is processed into a feature layer of the same size. All feature layers of the same size from the five stages are concatenated, and then the channels are adjusted by 3×3 convolution, DropBlock module and residual module.

[0024] Preferably, the calculations describing the operation of the multi-scale feature fusion module are as follows:

[0025] x a H,W,C =Pool2(DB(Conv1×1(x1)) H,W,C )))

[0026] x b H,W,C =DB(Conv1×1(x2) H,W,C ))

[0027] x c H,W,C =Up2(DB(Conv1×1(x3)) H,W,C )))

[0028] x d H,W,C=Up4(DB(Conv1×1(x4) H,W,C )))

[0029] x f H,W,C =Concat(x) a H,W,C ,x b H,W,C ,x c H,W,C ,x c H,W,C x5 H,W,C )

[0030] x c H,W,C =RS(DB(Conv3×3(x f H,W,C )))

[0031] In the formula, x1 H,W,C x2 H,W,C x3 H,W,C and x4 H,W,C These are the feature layers output from each encoding stage, x5 H,W,C This is the feature layer output from the previous decoding stage. DB is the DropBlock module, Pool2 is 2×2 pooling, Conv1×1 is 1×1 convolution, Up2 is 2x deconvolution, Up4 is 4x deconvolution, Concat is the concatenation operation, Conv3×3 is 3×3 convolution, RS is the residual module, and H, W, and C represent the height, width, and number of channels of the feature layer, respectively.

[0032] Preferably, the loss function used in the RMCA U-net model is the binary cross-entropy loss function, and the calculation formula is as follows:

[0033]

[0034] Where X and Y represent the segmentation result image and label, respectively, and h and w represent the coordinates of the pixel in X and Y, respectively.

[0035] Preferably, for a retinal fundus image with a field of view of 200 degrees, the preprocessing procedure includes:

[0036] The U-net model was used to extract the mask and region of interest from retinal fundus images with a field of view of 200 degrees.

[0037] Preferably, for retinal fundus images at 45-degree and 50-degree field of view, the image preprocessing process includes random reversal, addition of Gaussian noise and contrast enhancement to expand the dataset, and zero-padding of the image.

[0038] Preferably, the segmentation results of the trained model are measured by PR curves and ROC curves.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] This invention provides a method for segmenting hard exudate in retinal fundus images. It incorporates a dual residual module into the U-net encoder, replaces the original skip connections with a multi-scale feature fusion module, and integrates an improved channel attention module into the U-net decoder, thereby achieving end-to-end segmentation of hard exudate. This method effectively avoids interference from eyelashes, device borders, and uneven illumination in the image, achieving accurate segmentation of hard exudate and exhibiting strong robustness across fundus images at different viewing angles. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the RMCA U-net model for a method of segmenting hard exudate in retinal fundus images provided in this embodiment.

[0042] Figure 2 The dataset contains a subset of images. The first row contains fundus images from the Kaggle dataset with a 45-degree field of view, the second row contains fundus images from the IDRID dataset with a 50-degree field of view, and the third row contains fundus images from the ultra-wide-angle fundus image dataset with a 200-degree field of view.

[0043] Figure 3 Data augmentation for the Kaggle dataset, where (a) is the original image, (b) is a random rotation, (c) is Gaussian noise added, and (d) is contrast enhancement.

[0044] Figure 4 Data augmentation for the IDRID dataset, where (a) is the original image, (b) is a random rotation, (c) is Gaussian noise added, and (d) is contrast enhancement.

[0045] Figure 5 This is a dataset of fundus images with a field of view of 200 degrees, where (a) is the original image, (b) is the image extracted using U-net masking, and (c) is the region of interest.

[0046] Figure 6 This is a schematic diagram of the dual residual module.

[0047] Figure 7 This is a schematic diagram of the multi-scale input module.

[0048] Figure 8 This is a schematic diagram of the channel attention module.

[0049] Figure 9The graph shows the PR and ROC curves of RMCAU-net compared with two other algorithms on the Kaggle dataset.

[0050] Figure 10 The graph shows the PR and ROC curves of RMCA U-net compared with two other algorithms on the IDRID dataset.

[0051] Figure 11 The graph shows the PR and ROC curves of RMCAU-net compared with two other algorithms on the ultra-wide-angle fundus image dataset.

[0052] Figure 12 A visualization of the segmentation results on the Kaggle dataset.

[0053] Figure 13 A visualization of the segmentation results on the IDRID dataset.

[0054] Figure 14 This is a visualization of the segmentation results on an ultra-wide-angle fundus image dataset. Detailed Implementation

[0055] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0056] This embodiment provides a method for segmenting hard exudate in retinal fundus images, including the following steps:

[0057] S1: Preprocess fundus images at 45°, 50° and 200° field of view.

[0058] The datasets used in this embodiment include the Kaggle dataset, the IDRID dataset, and the ultra-wide-angle fundus image dataset, for reference. Figure 2 As shown, the images in the first row belong to the Kaggle dataset, which comes from the 2015 Diabetic Retinopathy Detection Competition. The entire dataset contains over 10,000 images; this example selects 120 fundus images showing the entire retinal region and resizes them to 448*488 pixels. Of these, 80 are used for training, 20 for validation, and 20 for testing. (Reference) Figure 3 As shown, images in the Kaggle dataset are augmented by random rotation, addition of Gaussian noise, and contrast enhancement, respectively.

[0059] The images in the second row belong to the IDRID dataset, which comes from the 2018 International Biomedical Retinal Imaging Challenge. The IDRID dataset includes 81 fundus images, each with a resolution of 4288×2848, and all provide pixel-level annotations. These 81 images are officially divided into two parts: a training set of 54 images and a test set of 27 images. For ease of training, this embodiment uses 608×608 images obtained through cropping and zero-padding as input to the model. (Reference) Figure 4 As shown, images in the IDRID dataset are augmented by random rotation, addition of Gaussian noise, and contrast enhancement, respectively.

[0060] The images in the third row belong to the Ultra-wide-angle fundus image dataset, a private dataset from Xinhua Hospital affiliated with Shanghai Jiao Tong University School of Medicine. These images were captured at a 200-degree field of view and include 260 fundus images with a resolution of 3900×3072. For ease of training, the image size was adjusted to 1300×1024 in this embodiment, with 163 images used for training and 97 for testing. During training, 800 random 128×128 image patches were selected from each image as input, for a total of 132,000 image patches. (Reference) Figure 5 As shown, the U-net model is used to extract the mask and region of interest from the ultra-wide-angle fundus image dataset.

[0061] S2: Input the preprocessed fundus images at 45°, 50° and 200° field of view into the pre-built RMCAU-net model and train it on the training set to obtain three trained models. The trained model is the one with the smallest loss when trained on the training set.

[0062] The pre-built RMCAU-net model uses an encoder-decoder architecture as its basic framework, referencing... Figure 1 As shown, the encoder includes Stage 1 to Stage 4, which are downsampling sections, and the decoder includes Stage 5 to Stage 7, which are upsampling sections.

[0063] For the encoder, each stage includes a 3×3 convolutional layer, a DropBlock module, a max pooling layer, and a dual residual module connected in sequence. After the input image is processed by the encoder, a feature map containing semantic information is generated.

[0064] refer to Figure 6As shown, the dual residual module includes two identical first residual blocks and second residual blocks. Each residual block includes two residual block layers connected in sequence. Each residual block layer includes a normalization layer, a ReLU activation function, a 3×3 convolutional layer, and a DropBlock module connected in sequence.

[0065] The calculation formula describing the structure of each residual block is shown below:

[0066] X1 H,W,C =DB(Conv3×3(Relu(BN(X)) H,W,C )))

[0067] X2 H,W,C =DB(Conv3×3(Relu(BN(X1)) H,W,C )))

[0068] X3 H,W,C =Conv1×1(X H,W,C )+X2 H,W,C

[0069] In the formula, X H,W,C The input to this residual block is H, W, and C, which represent the height, width, and number of channels of the feature map, respectively. DB represents the DropBlock layer, and Conv3×3 and Conv1×1 represent 3×3 and 1×1 convolutions, respectively. H,W,c For the output of this residual block, X1 H,W,C and X2 H,E,C These are the outputs of the previous residual block layer and the next residual block layer, respectively.

[0070] By adding a dual residual module, the residual output of the network is not only related to the previous layer but also retains network information from skip connections. This structure can effectively alleviate gradient vanishing or exploding and exhibits good adaptability to features of different dimensions, especially for hard exudates of different sizes and complex shapes. Therefore, the dual residual module plays an important role in capturing the detailed features of exudates. The DropBlock module can effectively alleviate overfitting of the model.

[0071] For the decoder, each stage includes a channel attention module, a deconvolution layer, and a multi-scale feature fusion module connected in sequence. The channel attention module is used to better utilize the most useful feature channels, and the multi-scale feature fusion module is used to replace the skip connections from the encoder to the decoder, guide the model to fuse multi-scale semantic information and global contextual features, improve the model's learning of globally salient features and suppress irrelevant local features. The input of the multi-scale feature fusion module in each stage includes the output of each stage of the encoder and the output of the previous stage of the decoder.

[0072] refer to Figure 7 As shown, with Figure 5 Taking the second multi-scale feature fusion module MSFF2 as an example, the operation of the multi-scale feature fusion module is explained:

[0073] First, the output features of Stage 1 to Stage 4 are processed using a first processing method, which includes 1×1 convolution and DropBlock regularization. Then, a 2×2 Pooling operation is performed on the output features of Stage 1 after the first processing, a 4×4 deconvolution operation is performed on the output features of Stage 4 after the first processing, and a 2×2 deconvolution operation is performed on the output features of Stage 3 after the first processing. Finally, a feature layer of the same size as Stage 2 and Stage 6 is obtained. All the features of the five stages are concatenated together to obtain a feature layer with 160 channels. Then, the channels are adjusted by a 3×3 convolution, a DropBlock module, and a residual module to make the number of channels 32.

[0074] The computation of the second multi-scale feature fusion module MSFF2 is described below:

[0075] x a H,W,C =Pool2(DB(Conv1×1(x1)) H,W,C )))

[0076] x b H,W,C =DB(Conv1×1(x2) H,W,C ))

[0077] x c H,W,C =Up2(DB(Conv1×1(x3)) H,W,C )))

[0078] x d H,W,c=Up4(DB(Conv1×1(x4) H,W,C )))

[0079] x f H,W,C =Concat(x) a H,W,C ,x b H,W,C ,x c H,W,C ,x c H,W,C x5 H,W,C )

[0080] x c H,W,C =RS(DB(Conv3×3(x f H,W,C )))

[0081] In the formula, x1 H,W,C x2 H,W,C x3 H,W,C x4 H,W,C and x5 H,W,C These are the feature layers for Stage 1, Stage 2, Stage 3, Stage 4, and Stage 6, respectively. DB is the DropBlock module, Pool2 is 2×2 pooling, Conv1×1 is 1×1 convolution, Up2 is 2x deconvolution, Up4 is 4x deconvolution, Concat is the concatenation operation, Conv3×3 is 3×3 convolution, RS is the residual module, and H, W, and C represent the height, width, and number of channels of the feature layer, respectively.

[0082] refer to Figure 8 As shown, the channel attention module includes parallel average pooling and max pooling branches, which can obtain more feature information. The addition of the channel attention module can effectively suppress the interference of irrelevant channels.

[0083] The calculation formula describing the channel attention module structure is shown below:

[0084]

[0085]

[0086] CA(F)=σ(Con1D(F ap )+Con1D(F mp ))

[0087] F = F c +F c ×CA(F)

[0088] In the formula, F cFor the input feature layer, and These represent max pooling and average pooling, respectively. σ is the sigmoid function, and F is the output feature layer.

[0089] The loss function used by RMCA U-net is the binary cross-entropy loss function, and the calculation formula is shown below:

[0090]

[0091] Where X and Y represent the segmentation result image and label, respectively, and h and w represent the coordinates of the pixel in X and Y, respectively. Adaptive moment estimation (Adam) is used to optimize the model parameters, with an initial learning rate of 0.001.

[0092] refer to Figure 9 As shown, the PR and ROC curves of Unet, Drnet, and the RMCA U-net proposed in this embodiment on the Kaggle dataset are compared respectively. Figure 10 As shown, the PR curves and ROC curves of Unet, Drnet, and the RMCA U-net proposed in this embodiment on the IDRID dataset are compared respectively. Figure 11 As shown, the PR curves and ROC curves of Unet, Drnet, and the RMCA U-net proposed in this embodiment on the ultra-wide-angle dataset are compared respectively.

[0093] Figure 12 This is a partial detection result from the Kaggle dataset. The fundus images contain significant interference, such as soft exudate and laser scars in the first two rows. The first row shows that U-net exhibited large false positive regions when segmenting laser scars. The second row indicates that our proposed method effectively suppresses the interference from soft exudate and can detect more accurate regions. The third and fourth rows present the detection results for fundus images of different qualities. Our method achieves the best results among the three techniques, while U-net and DR-net both exhibited false positives to some extent. In summary, the RMCA U-net proposed in this paper can more robustly segment hard exudate.

[0094] Figure 13This is a visualization of partial segmentation results from the IDRID dataset. Looking at the first row of segmentation results from the IDRID dataset, when hard and soft exudates are mixed, our proposed method can effectively suppress the interference of soft exudates, achieving more accurate segmentation. For the fourth row, U-net exhibits significant false negatives. Although DR-net can accurately segment hard exudates, its brightness value is significantly lower than that of RMCA U-net, meaning our model is more sensitive to exudates and more likely to detect them.

[0095] Figure 14 This is a visualization of partial segmentation results from an ultra-wide-angle fundus image dataset. Segmentation of hard exudate in ultra-wide-angle fundus images is challenging due to their low contrast and sparse lesions. In the first row, red boxes represent faint hard exudates in the original image, which our method clearly segments (marked with green boxes), while the other two methods fail to do so. The second and fourth rows contain many interfering points in the original images that are slightly similar to hard exudates. In these cases, our method provides the clearest detection results among the three detection techniques, with orange circles marking false positive regions. In the third row, the region of interest occupies only a small portion, becoming blurred, and there is interference from multiple factors such as eyelids and eyelashes in the original image. Here, our RMCA U-net provides the segmentation results with the most complete region among the techniques discussed.

[0096] In summary, the hard exudate segmentation method for retinal fundus images provided by this invention has strong robustness to fundus images under different viewing angles. Compared with traditional unsupervised methods or coarse-to-fine supervised methods, the hard exudate segmentation method provided by this invention can effectively avoid interference from the optic disc, illumination, etc. Compared with the other two end-to-end deep learning models, the hard exudate segmentation method provided by this invention has more obvious advantages.

[0097] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for segmenting hard exudate in retinal fundus images, characterized in that, The preprocessed retinal fundus image is input into the pre-built and trained RMCA U-net model to obtain the segmentation results of hard exudate. The RMCA U-net model includes an encoder and a decoder. The encoder includes multiple encoding stages, each of which includes a convolutional layer, a DropBlock module, a max-pooling layer, and a dual residual module connected in sequence. Each decoder includes multiple decoding stages, each of which includes a channel attention module, a deconvolutional layer, and a multi-scale feature fusion module connected in sequence. The channel attention module is used to utilize the most useful feature channels, and the multi-scale feature fusion module is used to guide the model to fuse multi-scale semantic information and global contextual features. The input of the multi-scale feature fusion module in each decoding stage includes the outputs of all encoding stages and the output of the previous decoding stage. The dual residual module includes two identical residual blocks. Each residual block includes two residual block layers connected in sequence. Each residual block layer includes a normalization layer, a ReLU activation function, a 3×3 convolutional layer, and a DropBlock module connected in sequence. The calculation formula describing the structure of each residual block is shown below: In the formula, This is the input for the residual block. Represents the output of the DropBlock layer and These represent 3×3 and 1×1 convolutions, respectively. This is the output of the residual block. and These are the outputs of the previous residual block layer and the next residual block layer, respectively. The channel attention module includes parallel average pooling and max pooling branches to obtain more feature information. The calculation formula describing the structure of the channel attention module is as follows: In the formula, For the input feature layer, and These represent max pooling and average pooling, respectively. For the sigmoid function, For the output feature layer, These represent the height and width of the feature layer, respectively. The current multi-scale feature fusion module in the decoding stage processes the following steps: The feature layers output from all encoding stages are processed, including 1×1 convolution and DropBlock regularization operations. If the feature layer of the processed encoding stage is different in size from the feature layer output from the previous decoding stage, it is processed into a feature layer of the same size. All feature layers of the same size from the five stages are concatenated, and then the channels are adjusted by 3×3 convolution, DropBlock module and residual module. The computation describing the operation of the multi-scale feature fusion module is shown below: In the formula, , , and These are the feature layers output from each encoding stage. This is the feature layer output from the previous decoding stage. This is the output of the DropBlock module. For 2×2 pooling, For 1×1 convolution, It is a 2x deconvolution. It is a 4x deconvolution. For splicing operations, It is a 3×3 convolution. This is the output of the residual module.

2. The method for segmenting hard exudate in a retinal fundus image according to claim 1, characterized in that, The RMCA U-net model uses the binary cross-entropy loss function, calculated as follows: in, , These represent the segmentation result image and the labels, respectively. , These represent the pixel points in... , The coordinates in the diagram.

3. The method for segmenting hard exudate in a retinal fundus image according to claim 1, characterized in that, For a retinal fundus image with a field of view of 200 degrees, the preprocessing procedure includes: The U-net model was used to extract the mask and region of interest from retinal fundus images with a field of view of 200 degrees.

4. The method for segmenting hard exudate in a retinal fundus image according to claim 1, characterized in that, For retinal fundus images at 45-degree and 50-degree field of view, the preprocessing of the images includes random inversion, addition of Gaussian noise and contrast enhancement to expand the dataset, and zero padding of the images.

5. The method for segmenting hard exudate in a retinal fundus image according to claim 1, characterized in that, The segmentation results of the trained model are measured using PR curves and ROC curves.

Images