Neural network-based fundus microaneurysm image automatic segmentation method, device and system
Through the neural network-based automatic segmentation method of fundus microarthrosm images, combined with preprocessing and feature extraction technology, the problem of low detection accuracy of microarthrosm in diabetic retinopathy is solved, achieving higher segmentation accuracy and more accurate microarterials Tumor recognition.
Patent Information
- Application Number
- CN202410937912.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-12
- Publication Date
- 2024-11-26
AI Technical Summary
The prior art has low detection accuracy of microarthrosm in diabetic retinopathy, making it difficult to segment tiny lesions composed of relatively macroscopic structures, resulting in poor detection results.
The automatic segmentation method of fundus microaret aneurysm images based on neural networks, including image preprocessing and feature extraction, and the Tuned-PSP-UNet model is designed using preprocessing and preprocessing technologies such as Otsu threshold algorithm, CLAHE, gamma correction and non-local mean denoising. , combined with the PSP module and NAM attention mechanism, the segmentation accuracy of the model is improved by combining the loss function.
The semantic segmentation accuracy of microarthrosses is significantly improved, and the most advanced segmentation results are achieved, which can more accurately identify microarthrosses in complex fundus images, reducing the false positive rate.
Smart Images

Figure CN119027431A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of neural network image segmentation, and in particular to a method, device and system for automatic segmentation of fundus microaneurysm images based on a neural network. Background Art
[0002] Diabetes mellitus (DM) is a growing disease in the world. According to the World Health Organization (WHO), the number of DM patients is expected to reach 439 million by 2030. Diabetic retinopathy (DR), one of the major complications of diabetes, is one of the most serious eye diseases in the world and one of the leading causes of blindness. Fortunately, early diagnosis and treatment of DR can effectively reduce the risk of blindness in diabetic patients. Retinal microaneurysms (MA), small swellings caused by leakage and bleeding in the retinal capillaries, are considered to be the earliest clinical signs of DR. Figure 1Several examples of MA in fundus images are shown. They usually appear as small red dots in fundus images, varying in intensity, shape, size, and location. Undoubtedly, MA segmentation is crucial for the early diagnosis of DR and is of great significance for controlling and delaying visual impairment. Traditional MA detection mainly relies on ophthalmologists, which makes large-scale screening of DR almost impossible. Therefore, computer-aided detection of MA came into being. So far, many microaneurysm detection methods have been proposed, which can generally be divided into three categories: physical model-based methods, classifier-based methods, and deep learning-based detection methods. Physical model-based microaneurysm detection methods are mainly based on the physical characteristics of retinal microaneurysms. Joshi et al. enhanced fundus images and removed blood vessels through morphological processing in "Mathematical morphology for microaneurysm detection in fundus images", and then extracted microaneurysms. Zhang et al. proposed a method for detecting microaneurysms based on feature transfer network and local background suppression in "Feature-transfer network and local background suppression for microaneurysm detection". The similarity matrix of feature distance is calculated to measure the difference between background noise and retinal objects and feature transfer network to detect microaneurysms with unbalanced data. Ding et al. proposed a dynamic multi-parameter template (DMPT) matching scheme in "An Accurate Approach for Microaneurysm Detection in Digital Fundus Images". The matching degree is measured by the sum of errors combined with the correlation coefficient. This scheme can balance the matching degree of template and MA between correlation and relative error. Kamble and Kokare proposed a method for extracting blood vessels using gradient-guided filters in "DETECTION OF MICROANEURYSM USING LOCAL RANKTRANSFORM IN COLOR FUNDUS IMAGES". However, there are still many false positives, such as thin vascular connections, punctate hemorrhages, and dark plaques.
[0003] Classifier-based methods dominate the mainstream of microaneurysm detection. The main processes of this method include candidate extraction, feature extraction, and candidate classification. Akram et al. in "Identification and classification of microaneurysms for early detection of diabetic retinopathy" extracted all possible candidate regions with microaneurysms in retinal images through mathematical morphological operations, filter banks, and vascular segmentation, and then formulated feature vectors for each candidate region based on shape, color, intensity, and statistics. Finally, the Gaussian mixture model (GMM), support vector machine (SVM), and the extension of the modeling method based on multi-model medium were combined in an ensemble to improve the accuracy of classification. Orlando et al. in "An Ensemble Deep Learning Based Approach for Red Lesion Detection in Fundus Images" used morphological reconstruction to extract microaneurysm candidate regions. In the feature extraction process, they built a convolutional neural network (CNN) model to extract deep features and combined them with color features, texture features, and geometric features for candidate classification. Melo et al. proposed a sliding band filter for microaneurysm enhancement in "Microaneurysm detection in coloreye fundus images for diabetic retinopathy screening" to obtain initial microaneurysm candidates. Then, the combination of filter response with color, contrast and shape information is used by a classifier ensemble for final candidate classification. Sopharak et al. first eliminated exudates and blood vessels in "Simple hybrid method for fine microaneurysm detection from non-dilated diabetic retinopathy retinal images" and used morphology for rough segmentation to identify microaneurysm candidate regions. Then 18 microvascular features were extracted as input to a naive Bayes classifier to classify each candidate pixel. Shah et al. removed blood vessels from the preprocessed green channel in "Automated microaneurysm detection in diabetic retinopathy using curvelet transform" and extracted preliminary microaneurysm candidate regions by local thresholding technique. The candidate regions were then divided into microaneurysms and non-microaneurysms based on statistical features.Deepa et al. proposed a method based on discrete normal Stockwell transform and statistical features to distinguish normal and diseased retinal images in "Automated detection of microaneurysms using Stockwell transform andstatistical features". They analyzed the results of various classifiers on their own data, but did not conduct experiments on any public database. Wang et al. used spectral analysis to measure the similarity of cross-sectional profiles in "Localising Microaneurysms in Fundus Images Through Singular Spectrum Analysis". Then, a set of coefficients was calculated to identify true microaneurysms. They selected three classifiers for testing and selected the best classifier for final classification.
[0004] In recent years, deep learning technology has made breakthroughs in various fields. Its development has received extensive attention in the medical field, and medical image segmentation has become a hot topic. Detection methods based on deep learning mainly regard microaneurysm detection as a segmentation task. Budak et al. proposed a microaneurysm detection method based on convolutional neural networks with reinforcement sample learning algorithm in A novel microaneurysms detection approach based on convolutional neural networks with reinforcement sample learning algorithm. Chudzik et al. proposed a novel patch-based fully convolutional neural network with batch normalization layer and Dice loss function in Microaneurysm Detection using Fully Convolutional Neural Networks, which achieved the best performance on the e-ophta-MA database. Eftekhari et al. proposed a new MA detection method based on two-stage neural network in Microaneurysm detection in fundus images using a two-step convolutional neural network to solve the problem of unbalanced data. They used a low-cost neural network to remove basic false positives and used a more complex network to classify the remaining pixels. Dai et al. proposed a clinical report-guided multi-sieving convolutional neural network in "Clinical Report Guided Retinal Microaneurysm Detection With Multi-Sieving Deep Learning", which uses a small amount of supervised information in clinical reports to identify potential microaneurysm areas through image-to-text mapping in feature space. These potential microaneurysm areas are then interwoven with fundus image information to perform multi-sieving deep mining in highly unbalanced classification problems. Tan et al. used a 10-layer convolutional neural network in "Automated Segmentation of Exudates, Haemorrhages, Microaneurysms using Single Convolutional Neural Network" to automatically and simultaneously segment and distinguish exudates, hemorrhages, and microaneurysms.
[0005] Computer-aided diagnosis (CAD) systems can detect DR accurately and quickly, thereby providing timely treatment for patients. In CAD systems for DR, segmentation of lesions is a prerequisite for accurately defining the region of interest, which can subsequently be used to assess the severity of the disease. In the early years, researchers focused on traditional image processing methods such as morphological operations and threshold segmentation. Due to the high dependence on the design level, traditional MA segmentation methods are relatively infeasible in practical applications. Nowadays, traditional MA segmentation methods that only rely on simple cues have been replaced by machine learning methods due to the significant improvement in the accuracy of machine learning methods. In recent years, deep learning-based methods have shown significant performance compared to traditional machine learning-based methods, and many researchers have adopted deep learning methods to segment diabetic retinopathy.
[0006] Due to the complex structure of the retina, such as the presence of small blood clots, vascular structures, background noise, etc., and the uneven color and brightness of fundus images collected by different devices, the detection effect of methods based on physical models is often unstable and prone to false detection. Classifier-based methods often use traditional methods such as filtering, background estimation, and morphology to extract features from candidate areas. These traditional methods are easily limited by parameters. At the same time, the fundus images of diabetic retinopathy are complex, resulting in low accuracy in microaneurysm detection.
[0007] Although deep learning models can avoid manually crafting complex image features, it is difficult to segment tiny lesions composed of relatively macroscopic structures, such as microaneurysms, which has become a technical problem that the industry urgently needs to solve. Summary of the invention
[0008] The present invention provides a method for automatic segmentation of fundus microaneurysm images based on a neural network, so as to solve the technical problem in the prior art that diabetic retinopathy fundus images are complex, resulting in low detection accuracy and difficulty in segmentation of microaneurysms.
[0009] To this end, the specific technical solution adopted by the present invention is as follows:
[0010] An automatic segmentation method for fundus microaneurysm images based on neural network includes image preprocessing, which preprocesses the original image to eliminate the interference caused by color unevenness; feature extraction, which designs an end-to-end feature extraction network to extract features of retinal microaneurysms.
[0011] In a possible design, the Otsu threshold algorithm is used to process redundant information, the green channel is selected from the RGB fundus image to detect retinal microaneurysms, the CLAHE algorithm is used to enhance the contrast of the green channel, the image brightness is gamma corrected, and the image is denoised by non-local means.
[0012] In a possible design, the end-to-end feature extraction network is a Tuned-PSP-UNet model, which at least includes:
[0013] Encoder: The encoder is a VGG16 architecture, which is pre-trained on ImageNet to effectively encode and classify images;
[0014] The decoder consists of several U-Net blocks, each of which receives the output and activation features of the last block to be upsampled from the middle layer of the encoder, and obtains the final pixel-by-pixel segmentation result by obtaining the results of the transfer convolution and cross connection of the input image;
[0015] The PSP module is located on the jump connection between the encoder and decoder to improve the performance of the network;
[0016] The NAM module suppresses insignificant weights by applying weight sparsity penalties, thereby improving feature extraction capabilities and computational efficiency;
[0017] The loss function evaluates the error between the model's predicted value and the true value.
[0018] In a possible design, the loss function is a combined loss function as follows:
[0019]
[0020] where y is the basic fact, is the predicted mask; FL and DL are focal loss and dice loss respectively.
[0021] In one possible design, the NAM module uses the scaling factor γ in batch normalization BN i and λ i To measure the variance of each channel and space and indicate their importance, the larger the scale factor, the more important the spatial or channel features are. During back propagation, a regularization penalty term is added to the loss function to suppress the weights of insignificant features, so that the network can pay more attention to important channels and spatial features with larger weights. The BN calculation is shown in formula (1):
[0022]
[0023] Among them, μ B , σ B They represent the mean and variance of the mini-batch respectively, and γ and β are trainable parameters.
[0024] In one possible design, the scaling factor γ i The weight W iIt reflects the size of the change in each channel and indicates the importance of the channel. The channel attention mechanism formula is shown in formula (2):
[0025] M c =sigmoid(W γ (BN(F1))) (2)
[0026] Among them, W γ represents the weight of each channel, F1 represents the input feature map of the channel attention mechanism, M c Represents the output feature map.
[0027] In a possible design, in spatial attention, the BN formula (1) is applied to the spatial dimension, and the scaling factor γ i The weight W i Reflects the degree of spatial variation and indicates the importance of space. The scaling factor γ i The weight W i The larger the weight is, the greater the variance of the pixels in the space is, indicating that the space is more transformed, the spatial information is richer, and the spatial significance is more important. The smaller the weight is, the smaller the spatial change is, the simpler the spatial feature information is, and the smaller the spatial significance is. The calculation formula of the spatial attention mechanism can be expressed as:
[0028] M s =sigmoid(W λ (BN(F2))) (3)
[0029] Among them, W λ represents the weight of the spatial pixel, F2 represents the input feature map of the spatial attention mechanism, and M s Represents the output feature map.
[0030] The present invention also provides an automatic segmentation device for fundus microaneurysm images based on a neural network, comprising a memory, a control processor, and a computer program stored in the memory and executable on the control processor. The control processor executes the program to implement the aforementioned automatic segmentation method for fundus microaneurysm images based on a neural network.
[0031] The present invention also provides a control system, including the aforementioned neural network-based automatic segmentation device for fundus microaneurysm images.
[0032] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the aforementioned neural network-based automatic segmentation method for fundus microaneurysm images.
[0033] Compared with the prior art, the present invention has the following beneficial effects:
[0034] 1. The present invention proposes to use an improved U-Net architecture to perform semantic segmentation on MA images. The model uses a pre-trained VGG16 as an encoder, adopts three 3×3 convolution kernels, and passes through a ReLU (non-linear activation function) layer and a BN (regularization layer) layer, which not only ensures the same receptive field, increases the non-linear expression ability of the model, but also prevents gradient explosion and overfitting during training.
[0035] 2. This paper introduces a new mechanism on skip connections to extract multi-scale features and contextual associations in fixed-width neural networks, thereby improving the segmentation accuracy of complex (blurred and low-contrast) MA images.
[0036] 3. The purpose of introducing the NAM attention mechanism in the decoder of the present invention is to improve the attention mechanism by utilizing the contribution factor of the weight. By applying sparse weight penalties to reduce the weights of less significant features, the model can not only be more efficient in weight calculation, but also maintain the same segmentation performance.
[0037] 4. The present invention performs comprehensive preprocessing of fundus images and uses a combined loss training model, emphasizing the impact of loss function and preprocessing on network segmentation accuracy.
[0038] 5. The technology proposed in this paper achieves state-of-the-art results in semantic segmentation of MA. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0040] Figure 1 The fundus image microaneurysm image in the background technology of the present invention;
[0041] Figure 2 Schematic diagram of the MA automatic segmentation method in an embodiment of the present invention, wherein (a) is image preprocessing and (b) is feature extraction;
[0042] Figure 3 This is an Otsu threshold algorithm processing diagram in an embodiment of the present invention;
[0043] Figure 4 1 are different color channel images of MA in the embodiment of the present invention, where (a) is the original color image, (b) is the red channel, (c) is the green channel, and (d) is the blue channel;
[0044] Figure 5 1 is a comparison diagram before and after non-local mean denoising in an embodiment of the present invention, where (a) is before denoising and (b) is after denoising;
[0045] Figure 6 1 is a diagram of image processing steps for MA extraction in an embodiment of the present invention, wherein (a) is green channel extraction, (b) is a histogram equalization image, (c) is a gamma correction image, and (d) is non-local mean denoising;
[0046] Figure 7 The overall architecture diagram of the Tuned-PSP-UNet model proposed in the embodiment of the present invention;
[0047] Figure 8 It is a diagram of the improved PSP module in the embodiment of the present invention;
[0048] Fig. 9 : The NAM diagram of the attention mechanism in the embodiment of the present invention, where (a) is the channel attention module diagram and (b) is the spatial attention module diagram;
[0049] Fig.10 1 is a visual comparison diagram of MA images with and without preprocessing in an embodiment of the present invention, wherein (a) is an MA image with preprocessing, and (b) is an MA image without preprocessing;
[0050] Fig.11 Schematic diagram of MA segmentation results in an embodiment of the present invention. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] To further illustrate each embodiment, the present invention provides drawings, which are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, ordinary technicians in the field should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.
[0054] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.
[0055] A neural network-based automatic segmentation method for fundus microaneurysm images based on multi-feature matching, the MA segmentation method proposed in the present invention is as follows Figure 2 As shown. It mainly includes two parts: image preprocessing and feature extraction. First, the original image is preprocessed to eliminate the interference caused by color unevenness. Then, the present invention designs an end-to-end feature extraction network to extract features of MA. The proposed method will be further described in detail below.
[0056] Step 1: Image preprocessing
[0057] Due to the interference of environment and equipment in the process of retinal image acquisition, as well as the differences in age and race, the fundus images captured usually have uneven lighting and different colors. In order to eliminate these effects as much as possible, the Otsu threshold algorithm, green channel extraction, contrast-limited adaptive histogram equalization method, gamma correction and non-local mean denoising are used to preprocess the fundus images.
[0058] Step 1.1: Otsu threshold algorithm processes redundant information
[0059] By observing the images in the e-ophta-MA dataset, we can see that outside the circular boundary of the fundus image is a large black area, which does not contain any information related to the eye image, so the first thing to do is to remove this redundant information. Here, the Otsu threshold algorithm is used. The Otsu threshold algorithm calculates a threshold based on the grayscale value of the image, divides the image into foreground and background, and then obtains the coordinates of the foreground boundary and cuts the background on the original image. The effect is as follows Figure 3 As shown, the white part is the foreground, the black part is the background, and the last picture is the effect after part of the background is cut off.
[0060] Step 1.2: Green channel extraction
[0061] MA appears as tiny red dots in color fundus images. Figure 4An example of MA in a color fundus image is shown, with the red channel, green channel, and blue channel extracted from an RGB fundus image. Walter et al. in Automatic detection of microaneurysms in color fundus images explain the imaging principles of color retinal images and why the green channel is usually chosen to detect MAs. As shown in the figure, MAs usually show higher contrast in the green channel than in the red and blue channels.
[0062] Step 1.3: Contrast Limited Adaptive Histogram Equalization (CLAHE)
[0063] CLAHE is an adaptive histogram equalization technique for contrast enhancement. In the preprocessing of MA detection methods, CLAHE is widely used to enhance the contrast of the green channel. This step is performed to eliminate the effects of low contrast between microaneurysms and the background.
[0064] Step 1.4: Gamma Correction
[0065] Gamma correction is usually applied when the illumination in an image is uneven. This method is used to increase or decrease the brightness in an image. In the example of the present invention, the present invention increases the brightness by applying gamma correction, where the value of gamma is set to 3.
[0066] Step 1.5: Non-local means denoising
[0067] After the above processing, the lesion features in the image are more obvious than the initial image. However, when zooming in on the local area, it is found that the image is full of noise points, such as Figure 5 As shown in (a), non-local mean denoising can utilize the redundant information in the image and retain the detailed features of the image to the greatest extent while removing noise. Figure 5 (b) is the processed image. By observation, it can be found that the noise in the image disappears significantly after denoising, and the edges of blood vessels and lesions become clearer.
[0068] After preprocessing the fundus image, the output after each step is Figure 6 This figure gives a graphical representation of all the image processing steps used for MA extraction in the present invention.
[0069] Figure 6 In the figure, (a) green channel extraction (b) histogram equalization image (c) gamma correction image (d) non-local means denoising.
[0070] Step 2: Model architecture design
[0071] This paper proposes an improved U-Net structure for microaneurysm segmentation. The modified architecture is as follows Figure 7 As shown. It consists of a contraction path (encoder) on the left and a symmetrical expansion path (decoder) on the right, giving it a U-shape. The encoder of the modified U-Net model is replaced by the VGG16 architecture, which has been pre-trained on ImageNet and effectively encodes and classifies images. The decoder consists of several U-Net blocks. Each U-Net block receives the output and activation features of the last block to be upsampled from the middle layer of the encoder. The final pixel-by-pixel segmentation result is obtained by taking the results of the transfer convolution and cross connection of the input image. The modified PSP module is placed on the skip connection to improve the performance of the network. The NAM module is added in the enhanced feature extraction stage to suppress insignificant weights by applying weight sparsity penalties, thereby improving the feature extraction ability and computational efficiency of the model. In addition, the network is trained using a combined loss function.
[0072] Step 2.1 Network architecture
[0073] U-Net was proposed by Ronneberger et al. in 2015 and is one of the most popular architectures for biomedical image segmentation. Its architecture is widely considered to be an encoder network followed by a decoder network. For the encoder network, it is usually a pre-trained classification network with downsampling pooling layers attached at multiple different levels. For the decoder network, it consists of upsampling and concatenation followed by regular convolution operations. The discriminative features obtained by the encoder are projected onto the pixel space to predict the pixel-by-pixel classification. U-Net overcomes the limitations of the previously proposed FCN by minimizing the spatial information loss between the encoder and decoder using skip connections. In addition, repeated pooling operations allow the network to generate rich feature representations that contain features of different scales.
[0074] Our preliminary evaluation of the U-net model found that some segmentation errors were due to poor contextual relationships between microaneurysms and background and lack of sufficient global information. Therefore, we proposed an improved U-Net structure for microaneurysm segmentation in fundus images. A modified PSP module was added to the skip connection to correct this problem as an influential global context prior and multi-scale feature aggregator. The addition of the NAM module in the enhanced feature extraction stage can reduce the weights of less significant features. This method applies a sparse weight penalty on the attention module, which makes these weights more computationally efficient while maintaining the same performance.
[0075] Step 2.2 Improved pyramid scene parsing module
[0076] The PSP module is commonly used in semantic segmentation to mitigate errors due to poor contextual relationships and as a global context prior. The present invention carefully tunes the PSP module to extract influential contextual associations between microaneurysms and background to create glottal feature representations at each encoder stage. These rich features are passed to the corresponding decoder modules via skip connections to provide the necessary multi-scale spatial information for effectively constructing microaneurysm segmentation.
[0077] A key component of the PSP module is the global average pooling (GAP), which provides a concise vector representation of the input feature map. Independently, GAP may not be suitable for low-contrast fundus images as it may lead to a significant loss of spatial correlation and may not help the network distinguish microaneurysms from the background. Therefore, additional multi-scale information from different sub-regions is crucial to distinguish regions with weaker contextual relationships. Multi-scale features can be obtained by using pooling operations with different kernel sizes, which are then fused with the GAP output to create a rich feature representation.
[0078] The present invention modifies the PSP module by changing the number and size of multi-scale kernels to improve the segmentation of microaneurysms in low-contrast fundus images. The present invention considers three different kernel sizes (i.e., initial powers of two) to capture significantly different multi-scale features to identify blurred segmentation boundaries. In addition, the present invention fixes the number of kernels in the 1×1 convolution after multi-scale pooling to 64 to ensure limited parameter usage. And avoids modifying the channel width of the input feature map to ensure that the decoder can access the original feature map of the encoder. Figure 8 The modified PSP module and its components and connections are shown. The improved PSP module is used to capture multi-scale features to mitigate errors caused by poor context and contrast in fundus microaneurysm images. Max pooling is highlighted in green (8×8), blue (4×4), and orange (2×2). Next, a 1×1 convolution (with 64 kernels) is applied to the output of the pooling layer to ensure equal weights for each pooling operation in the global feature. Subsequently, the convolution result is upsampled using bilinear interpolation to create a feature map with the exact dimensions as the input. Finally, the multi-scale features are concatenated to produce the output of the modified pyramid pooling operation.
[0079] Step 2.3 Attention Mechanism NAM
[0080] The attention mechanism can enhance the network's ability to extract small features. In essence, it uses a series of weight parameters to enhance the important information of the image so that the network can focus on important and useful feature information. The deeper the network, the higher the level of important feature information captured by the attention mechanism. Using the attention mechanism can make the model focus on microaneurysms and discard irrelevant areas. Fig. 9As shown in Figure 2, the attention mechanism NAM uses the scaling factor γ in batch normalization BN. i and λ i To measure the variance of each channel and space and indicate their importance, the larger the scale factor, the more important the spatial or channel feature is. This module adds a regularization penalty term to the loss function during back propagation to suppress the weights of insignificant features, so that the network can pay more attention to important channels and spatial features with larger weights. BN calculation is shown in formula (1):
[0081]
[0082] Among them, μ B , σ B They represent the mean and variance of the mini-batch respectively, and γ and β are trainable parameters.
[0083] Channel attention module such as Fig. 9 As shown in (a), the scaling factor γ i The weight W i It reflects the size of each channel change and indicates the importance of the channel. It can be understood that the channel scale factor γ i The weight W i The larger the value, the greater the variance of the channel, which means that the channel changes more strongly, then the information contained in the channel will be richer, and the greater the significance and importance. For channels with little change, the information is single and the importance is small. The formula of the channel attention mechanism is shown in formula (2):
[0084] M c =sigmoid(W γ (BN(F1))) (2)
[0085] Among them, W γ represents the weight of each channel, F1 represents the input feature map of the channel attention mechanism, M c Represents the output feature map.
[0086] In spatial attention, applying BN formula (1) to the spatial dimension is called pixel normalization. Fig. 9 As shown in (b), the scaling factor γ i The weight W i It reflects the degree of spatial change and indicates the importance of space. i The weight W iThe larger the weight is, the greater the variance of the pixels in the space is, indicating that the space is more transformed, the spatial information is richer, and the spatial significance is more important. The smaller the weight is, the smaller the spatial change is, the simpler the spatial feature information is, and the smaller the spatial significance is. The calculation formula of the spatial attention mechanism can be expressed as:
[0087] M s =sigmoid(W λ (BM(F2))) (3)
[0088] Among them, W λ represents the weight of the spatial pixel, F2 represents the input feature map of the spatial attention mechanism, and M s Represents the output feature map.
[0089] Although traditional attention mechanism modules such as CBAM and SENET capture features in space and channels, they lack consideration of weight influencing factors, so they cannot further suppress the weights of insignificant features. In order to suppress insignificant channel and spatial features, the attention mechanism module NAM used in this paper adds a regularization penalty term to the loss function, as shown in formula (4). By adding a regularization penalty term to weaken the degree of insignificant channel and spatial feature information with small weights, the weights of insignificant features are suppressed to a certain extent, so that the network can focus more attention on important spatial and channel feature information with large weights during back propagation, improve the efficiency of back propagation, extract important and useful detail feature information in channels and spaces, and also improve the accuracy of network recognition.
[0090] Loss=∑ (x,y) l(f(x,W),y)+p∑g(γ)+p∑g(λ) (4)
[0091] Among them, x and y represent input and output, W represents the network weight, l represents the loss function, g is the l1 norm penalty function, and p represents the penalty factor of g(γ) and g(λ).
[0092] Step 2.4 Loss Function
[0093] The loss function plays a key role in neural network based MA segmentation because the update of gradients and parameters directly depends on the output of the loss function. A well-balanced loss function ensures that the network learns to generate segmentation masks while giving importance to the ground truth distribution, regions and boundaries. The present invention trains the model with well-known loss functions and proposes an improved combined loss to ensure that the model is trained for optimal performance. The present invention uses the following loss function along with focal loss and dice loss to train the network:
[0094] Combo loss: The combined loss is a weighted average of Dice loss and cross entropy loss, allowing the network to learn the distribution and area of the ground truth. Taghanaki et al. showed that the combined loss can converge faster than distribution-based loss functions (e.g., cross entropy loss alone). The present invention proposes to modify the combined loss to be the sum of focal loss and dice loss. Adjusting the hyperparameters α and γ parameters within the focal loss enables the combined loss to manage the distribution of imbalanced classes and penalize incorrectly segmented pixels. In addition, the dice loss provides relevant information about the area overlap between the ground truth and the prediction. The modified combined loss function can be described as:
[0095]
[0096] where y is the basic fact, is the predicted mask; FL and DL are focal loss and dice loss respectively.
[0097] Step 3: Experimental dataset
[0098] To evaluate the performance of the proposed method, experiments were conducted on two color fundus image databases, namely, e-ophta-MA and Indian Diabetic Retinopathy Image Dataset (IDRiD), all of which are publicly available.
[0099] e-ophta-MA: e-ophth-MA is a sub-database of the public fundus image database e-ophtha, used for scientific research on diabetic retinopathy. e-ophta-MA contains 148 images with MA, which are manually annotated by ophthalmologists. The images in e-ophta-MA are mainly available in two different resolutions, 1440×960 and 2544×1696 pixels. All images have a 45° field of view (FOV).
[0100] IDRID: IDRiD images were acquired using a Kowa VX-10 Alpha digital fundus camera (Nand, Maharashtra, India) with a field of view of 50 degrees and all images were focused near the macula. The resolution of fundus images is 4288 × 2848 and the images are in JPEG format. The dataset used here consists of 81 images for MA detection and contains the corresponding binary MA label map for each image.
[0101] Step 4: Evaluation Metrics
[0102] Evaluation indicators play an important role in measuring the performance of compared models. In order to quantitatively analyze the experimental results, the present invention uses several specific indicators to evaluate the performance of microaneurysm segmentation tasks, including sensitivity (SEN), intersection over union (IOU), F-1 score and precision-recall curve (PRC). These four indicators all reflect the precision and recall performance of the model on binary segmentation tasks from different angles. In order to achieve them, the present invention first calculates the true positive (TP), false positive (FP), true negative (TN) and false negative (FN). The TP here refers to the intersection of the real lesion area and the predicted lesion area, FP is the intersection of the real normal area and the predicted lesion area, and TN is the intersection of the real normal area and the predicted normal area. Based on the above concepts, the present invention introduces the following indicators:
[0103]
[0104] Let the present invention take a closer look at these three indicators. SEN can be regarded as the misdiagnosis rate of the disease. In the work of the present invention, it refers to the ratio of TP to the true lesion area, which is a key and primary factor for patients and doctors. In practical applications, the present invention tries its best to reduce the misdiagnosis rate. IOU is an evaluation metric for measuring the accuracy of semantic segmentation models, which specifies the amount of overlap between the predicted results and the basic facts. F1 is a measure of the similarity between the prediction and the basic facts. It is not only a measure of the number of positives predicted by the model, but also penalizes the false positives of the model. It considers both precision and recall, allowing both to reach the highest level at the same time and achieve a balance. Regarding the above commonly used metrics, the closer they are to 1, the better the segmentation performance.
[0105] The impact of the loss function and preprocessing is as follows:
[0106] The present invention trains the Tuned-PSP-UNet model with three different loss functions to infer the optimal loss for MA segmentation. Table 1 shows that compared with using Focal loss and Dice loss alone, the combined loss achieves the highest values in all three indicators.
[0107] Table 1
[0108] The segmentation performance of Tuned-PSP-UNet using different loss functions and preprocessing shows that the combined loss maximizes the segmentation accuracy.
[0109] Loss function SEN(%) IOU(%) F1(%) Focalloss 26.68 24.45 39.29 Diceloss 42.53 34.11 50.88 Comboloss 48.01 36.21 53.17
[0110] We train the Tuned-PSP-UNet model with and without preprocessing to understand its quantitative impact on performance. This comparative analysis is crucial because MA images contain significant speckle noise. Interestingly, some deep learning methods in the literature do not employ a preprocessing stage to handle the widespread noise, which is described as:
[0111] f'(x,y)=f(x,y)+f(x,y) γ Gauss(x,y) (9)
[0112] Where f is the noise-free image, f' is the noisy image, Gauss is an additive noise function with Gaussian distribution and fixed variance, γ is a hyperparameter in the range [0, 1], and x, y represent the horizontal and vertical coordinates of the pixel in the image, respectively. Table 2 shows that preprocessing improves the segmentation performance by providing the model with images with higher local and global contrast and clearer anatomical boundaries. In addition, the present invention visually compares the MA images of six patients in the training set with and without preprocessing. Fig.10 It is emphasized that image preprocessing significantly improves the visual quality of MA images by enhancing the contrast of MA edges. In addition, the preprocessed images clearly highlight the MA, providing useful supplementary information for the model to reduce segmentation errors. Based on these observations, the present invention trains all models after applying preprocessing to MA images.
[0113] Table 2 Segmentation performance of Tuned-PSP-UNet with / without preprocessing, highlighting the performance advantages of contrast and edge enhancement.
[0114]
[0115] The effects of improving the PSP module and NAM attention mechanism on segmentation performance are as follows:
[0116] In order to conduct a comparative study between the proposed models, the present invention takes the performance of the ordinary U-Net model as a baseline. The U-Net model only achieved 26.01% and 41.28% IoU and F1, respectively. Therefore, the U-Net model has low accuracy for the microaneurysm segmentation task. The present invention proposes a Tuned-PSP-UNet model to improve the microaneurysm segmentation accuracy. Table 3 shows that adding the modified PSP module to the proposed model can increase the sensitivity value by 9.77%, the IoU by 8.02%, and the F1 by 9.5%. Adding the NAM attention mechanism to the proposed model can increase the sensitivity value by 4.78%, the IoU by 2.18%, and the F1 by 2.39% on the above basis.
[0117] The original U-Net has skip connections between the encoder and decoder to improve the information flow between corresponding blocks. However, the same feature map flows through the backbone and skip connections of the U-Net, resulting in blurred segmentation boundaries in the feature maps. The modified PSP module replaces the repeated encoder features with multi-scale features and acts as an influential global context prior, allowing the construction of rich multi-scale feature representations in each level of the architecture. In the initial layers of the encoder, the PSP module extracts low-level features (i.e., edges and curves), while in deeper layers, it learns to detect the shape of microaneurysms. In the upsampling stage, high-level information is first used to generate a coarse mask of the microaneurysm, which is refined by low-level multi-scale features passed by skip connections. The NAM attention mechanism has the feature of "plug and play", which improves the detection accuracy of the model without increasing the number of model parameters. Therefore, the present invention observes the largest performance improvement in the case of Tuned-PSP-UNet.
[0118] Table 3 shows the segmentation performance of the models with / without the modified PSP block and the addition of the NAM attention mechanism, indicating that the segmentation accuracy improves with the use of the adjusted PSP module and the NAM attention mechanism.
[0119] Modelname SEN(%) IOU(%) F1(%) UNet 33.46 26.01 41.28 UNet+PSP 43.23 34.03 50.78 UNet+PSP+NAM 48.01 36.21 53.17
[0120] Table 4 shows the segmentation accuracy of prominent architectures from the literature.
[0121] The present invention has the following observations: (1) The Tuned-PSP-UNet of the present invention consistently outperforms all competitors in terms of IOU and F1; (2) It makes a significant contribution to MA segmentation and outperforms HRNetV2 by more than 10.68% in F-score and more than 9.23% in IoU, which indicates that the Tuned-PSP-UNet of the present invention is able to maintain more subtle activations for minor lesions;
[0122] (3) Among them, the Tuned-PSP-UNet of the present invention achieves the best performance, surpassing the latest transformer-based segmentation method Swin-base by 9.08% and 7.93% in terms of F-score and IoU respectively. In summary, the present invention can infer that among the tested models, the Tuned-PSP-UNet model achieves the highest segmentation accuracy.
[0123] Table 4 Results on IDRiD
[0124]
[0125]
[0126] The present invention proposes to use an improved U-Net architecture to perform semantic segmentation on MA images. The model uses a pre-trained VGG16 as an encoder, adopts three 3×3 convolution kernels, and passes through a ReLU (non-linear activation function) layer and a BN (regularization layer) layer, which not only ensures the same receptive field and increases the non-linear expression ability of the model, but also prevents gradient explosion and overfitting during training. The present invention introduces a new mechanism on the skip connection to extract multi-scale features and contextual associations in fixed-width neural networks, such as Figure 8 As shown, the segmentation accuracy of complex (blurred and low-contrast) MA images is improved.
[0127] The purpose of introducing the NAM attention mechanism in the decoder is to improve the attention mechanism by using the contribution factor of the weight. By applying sparse weight penalties to reduce the weights of less significant features, the model can be more efficient in weight calculation while maintaining the same segmentation performance. The present invention performs comprehensive preprocessing on fundus images and uses a combined loss training model, emphasizing the impact of loss functions and preprocessing on network segmentation accuracy. The technology proposed in the present invention has achieved state-of-the-art results in semantic segmentation of MA. The MA segmentation results are as follows: Fig.11 shown.
[0128] The present invention also provides an automatic segmentation device for fundus microaneurysm images based on a neural network, comprising a memory, a control processor, and a computer program stored in the memory and executable on the control processor. The control processor executes the program to implement the aforementioned automatic segmentation method for fundus microaneurysm images based on a neural network.
[0129] The present invention also provides a control system, including the aforementioned neural network-based automatic segmentation device for fundus microaneurysm images.
[0130] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the aforementioned neural network-based automatic segmentation method for fundus microaneurysm images.
[0131] The method for automatically segmenting fundus microaneurysm images based on a neural network according to an embodiment disclosed in the present invention can be written as a computer program and stored on a computer-readable storage medium. When the computer program is executed by a processor, the method for automatically segmenting fundus microaneurysm images based on a neural network as described above can be implemented.
[0132] Examples of computer-readable storage media include read-only memory (ROM), random-access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random-access memory (RAM), dynamic random-access memory (DRAM), static random-access memory (SRAM), flash memory, nonvolatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. In one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.
[0133] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in different orders and / or concurrently with other actions from the illustrations and descriptions herein or not illustrated and described herein but understandable to those skilled in the art. It will be further appreciated by those skilled in the art that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly explain this interchangeability of hardware and software, various illustrative components, frames, modules, circuits, and steps are generally described in the form of their functionality above. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technical staff can implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention. The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, a battery compartment control board, a micro-battery compartment control board, or a state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration. The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. In an alternative, the storage medium can be integrated into the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In an alternative, the processor and the storage medium can reside in a user terminal as discrete components. In one or more exemplary embodiments, the functions described can be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function can be stored on a computer-readable medium or transmitted therefrom as one or more instructions or codes.Computer-readable media include both computer storage media and communication media, including any media that facilitates a computer program to be transferred from one place to another. Storage media can be any available media that can be accessed by a computer. As an example and not limitation, such computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage devices, or any other media that can be used to carry or store the agreed program code in the form of instructions or data structures and can be accessed by a computer. Any connection is also properly referred to as computer-readable media. For example, if software is transmitted from a web site, a central control computer, or other remote sources using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.
[0134] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A method for automatic segmentation of fundus microaneurysm images based on neural network, characterized by: Including, image preprocessing, preprocessing the original image to eliminate the interference caused by color unevenness; Feature extraction,An end-to-end feature extraction network is designed to extract features of retinal microaneurysms.
2. The method for automatic segmentation of fundus microaneurysm images based on neural network according to claim 1, characterized in that: The Otsu threshold algorithm is used to process redundant information, the green channel is selected from the RGB fundus image to detect retinal microaneurysms, the CLAHE algorithm is used to enhance the contrast of the green channel, the image brightness is gamma corrected, and the image is denoised by non-local means.
3. The method for automatic segmentation of fundus microaneurysm images based on neural network according to claim 1 or 2, characterized in that: The end-to-end feature extraction network is a Tuned-PSP-UNet model, which at least includes: Encoder: The encoder is a VGG16 architecture, which is pre-trained on ImageNet to effectively encode and classify images; The decoder consists of several U-Net blocks, each of which receives the output and activation features of the last block to be upsampled from the middle layer of the encoder, and obtains the final pixel-by-pixel segmentation result by obtaining the results of the transfer convolution and cross connection of the input image; The PSP module is located on the jump connection between the encoder and decoder to improve the performance of the network; The NAM module suppresses insignificant weights by applying weight sparsity penalties, thereby improving feature extraction capabilities and computational efficiency; The loss function evaluates the error between the model's predicted value and the true value.
4. The method for automatic segmentation of fundus microaneurysm images based on neural network according to claim 3, characterized in that: The loss function is a combined loss function, specifically as follows: where y is the basic fact, is the predicted mask; FL and DL are focal loss and dice loss respectively.
5. The method for automatic segmentation of fundus microaneurysm images based on neural network according to claim 3, characterized in that: The NAM module uses the scaling factor γ in batch normalization BN. i and λ i To measure the variance of each channel and space and indicate their importance, the larger the scale factor, the more important the spatial or channel features are. During back propagation, a regularization penalty term is added to the loss function to suppress the weights of insignificant features, so that the network can pay more attention to important channels and spatial features with larger weights. The BN calculation is shown in formula (1): Among them, μ B , σ B They represent the mean and variance of the mini-batch respectively, and γ and β are trainable parameters.
6. The method for automatic segmentation of fundus microaneurysm images based on neural network according to claim 5, characterized in that: Scaling factor γ i The weight W i It reflects the size of the change in each channel and indicates the importance of the channel. The channel attention mechanism formula is shown in formula (2): M c =sigmoid(W γ (BN(F1))) (2) Among them, W γ represents the weight of each channel, F1 represents the input feature map of the channel attention mechanism, M c Represents the output feature map.
7. The method for automatic segmentation of fundus microaneurysm images based on neural network according to claim 5 or 6, characterized in that: In spatial attention, the BN formula (1) is applied to the spatial dimension, and the scaling factor γ i The weight W i Reflects the degree of spatial variation and indicates the importance of space. The scaling factor γ i The weight W i The larger the weight is, the greater the variance of the pixels in the space is, indicating that the space is more transformed, the spatial information is richer, and the spatial significance is more important. The smaller the weight is, the smaller the spatial change is, the simpler the spatial feature information is, and the smaller the spatial significance is. The calculation formula of the spatial attention mechanism can be expressed as: M s =sigmoid(W λ (BN(F2))) (3) Among them, W λ represents the weight of the spatial pixel, F2 represents the input feature map of the spatial attention mechanism, and M s Represents the output feature map.
8. A neural network-based automatic segmentation device for fundus microaneurysm images, characterized in that: It comprises a memory, a control processor and a computer program stored in the memory and executable on the control processor, wherein the control processor executes the program to implement the neural network-based automatic segmentation method for fundus microaneurysm images as described in any one of claims 1 to 7.
9. A control system, characterized in that: It includes the neural network-based automatic segmentation device for fundus microaneurysm images as described in claim 8.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the neural network-based automatic segmentation method for fundus microaneurysm images as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Fundus image microaneurysm identification method based on R-CNN
CN110276356A
Fundus image optic cup and optic disk segmentation method and system for assisting glaucoma screening
CN110992382A
Method and system for segmenting choroidal neovascularization from fundus OCT image
CN111292338A
Automatic detection method and imaging method for retinal microaneurysm
CN112700409A
Brain tumor multi-mode MRI image segmentation method based on deep learning
CN114926477A
