A deep learning-based method for segmenting fundus hard exudates
By adding dual attention module and multi-scale feature fusion to the Unet model, combined with improved loss function and hollow convolutional structure, the problem of low detection accuracy and recall in fundus lesions detection is solved, and more efficient detection effect.
Patent Information
- Application Number
- CN202111465678.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-03
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-12-03
AI Technical Summary
The prior art has problems such as time-consuming, labor-intensive, misdiagnosis and misdiagnosis in fundus lesions detection. Especially in diabetic retinopathy, the detection of hard exudates is difficult, and traditional methods are difficult to effectively improve detection accuracy and recall. Rate.
Based on the Unet model, the dual attention module and multi-scale feature fusion are added, combined with the improved Dice Loss loss function and multi-level hollow convolution structure, and the pyramid-type image fusion method is used to improve the detection ability of hard exudate lesions. .
It significantly improves the accuracy and recall rate of hard exudate detection, enhances the ability to segment lesions of different scales, reduces misdiagnosis and misdiagnosis, and improves detection efficiency.
Smart Images

Figure CN114140437B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning image segmentation, and in particular to a method for segmenting fundus hard exudates based on a multi-scale convolutional neural network. Background Art
[0002] Diabetes is a common disease characterized by high blood sugar, and retinopathy is a complication that is easy to occur after suffering from diabetes. It is also one of the main lesions that cause blindness. The clinical manifestations of diabetic retinopathy include microaneurysms, exudates, and hemorrhage areas. In order to prevent the further development of fundus lesions, it is of great significance to detect retinopathy in advance. Hard exudates are a lesion with symptoms of fundus hemorrhage, and detecting them is an important step in retinopathy detection. Due to the large number of people who need to be tested, it is time-consuming and labor-intensive for doctors to conduct manual testing, and due to subjective factors, it is easy to cause misdiagnosis and missed diagnosis. Therefore, it is particularly important to develop an auxiliary system that can automatically identify lesions, which can greatly improve the efficiency of detection and reduce the number of misdiagnoses.
[0003] The main methods currently used in fundus lesion detection are: supervised learning method, first mark all areas where lesions may exist, then use classifiers to filter and remove areas that are not lesions, and gradually improve the accuracy of the algorithm. With the development of deep learning in image processing, Long et al. proposed the fully convolutional neural network FCN (Fully Convolution Network) in 2015. By changing the fully connected layer at the end of CNN to a convolutional layer, it can be used in tasks such as image classification and segmentation. Later, Olaf Ronneberger et al. proposed Unet on this basis. Unet is now often used in the field of medical image segmentation. It is a symmetrical encoding and decoding structure. At the same time, the jump connection module added to it improves the accuracy of segmentation to a certain extent. Summary of the invention
[0004] The present invention proposes a method for detecting and segmenting fundus hard exudate lesions based on an improved Unet model. In order to improve the detection accuracy of hard exudates, a dual attention module is added to the traditional Unet model to increase the attention to the lesion area; at the same time, features of more scales are integrated, which is conducive to the expression of detailed information. The color fundus lesion image is input into the convolutional neural network, and the network can automatically mark the corresponding lesion area and output it, saving manpower and material resources for marking and improving the accuracy of lesion marking.
[0005] The method for detecting fundus hard exudate lesions based on improved U-NET shown in the present invention can be implemented by the following technical solutions:
[0006] Step 1: taking the dataset image containing fundus lesions as the original data sample and performing data preprocessing on it;
[0007] Step 2: manually mark the hard exudate lesions in the fundus image to obtain a data set with labeled information required for training the network model, and divide the obtained data set into a training set and a test set;
[0008] Step 3: Input the obtained training set into the improved Unet model, supervise the network training through the improved Dice Loss loss function, and obtain the hard exudate lesion detection model; one limitation of the Dice Loss function is that it does not distinguish between false positives (FP) and false negatives (FN). When training the model, it is necessary to give higher weights to false negatives (FN) to improve the recall rate. At the same time, another problem with the Dice Loss loss function is that it is difficult to segment small objects because they do not contribute much to the loss.
[0009] The improved Unet model adds a dual attention module to the skip connection layer, including a channel attention module and a spatial attention module, which can pay more attention to the information of the hard exudate lesion area and improve the detection effect; the feature map obtained by the attention module for each multi-scale input feature of the encoding layer is used as the input of the previous attention feature on the one hand, and as the input of the decoding feature on the other hand, so that the multi-scale information can be fused into the final output feature to improve the segmentation effect; the last step of the downsampling process uses a multi-level hole convolution module to fuse the feature map results obtained by the convolution and use it as the input of the decoding process, so that the model's detection effect on multi-scale lesions will be effectively enhanced;
[0010] The improved Dice Loss loss function is specifically:
[0011]
[0012] DC c Represents the loss result, α and β are adjustable variables, where p kc is 1, indicating that pixel k in the output result belongs to category c. is 0, indicating that pixel k in the output result does not belong to category c, g kc is 1, indicating that the pixel k of the original labeled image belongs to category c. is 0, indicating that the pixel k in the original labeled image does not belong to category c; ε is a constant, and N represents the number of sample pixels;
[0013]
[0014] DM c represents the loss result after processing, σ is the adjustment parameter;
[0015] Step 4: Input the test set obtained in step 2 into the hard exudate detection model obtained in step 3 for image segmentation, and further fine-tune the model based on the segmentation results.
[0016] Preferably, the data preprocessing in step 1 is specifically as follows:
[0017] The image is padded and scaled to a fixed size with equal length and width, and then flipped horizontally and upside down to expand the number of data sets. The brightness of the image is adjusted and the image is subjected to Mosaic data enhancement. Mosaic enhancement is to combine four images into one image, which can increase the number of target objects, especially the number of small targets, thereby improving the training effect of the model.
[0018] Preferably, the position attention mechanism module is as follows: input A is convolved to obtain features B, C, and D respectively, and the resolution of features B, C, and D are the same as that of A. Then, features B, C, and D are reshaped to C×N, where N=H×W; then feature C is transposed, and the result is multiplied by B to obtain an N×N matrix S, and each point of S is softmax normalized by formula (3);
[0019]
[0020] where s ij represents the influence of the i-th position on the j-th position, B i represents the i-th value in feature B, C j Represents the j-th value in feature C.
[0021] The result is multiplied by D and reshaped into a feature map of size C×H×W, which is then added to A to obtain the final result, as shown in formula (4):
[0022]
[0023] E j represents the value in feature E, the parameter gradually learned after α is initialized to 0, and D i represents the i-th value in feature D, A j Represents the jth value in feature A.
[0024] The channel attention module described above is as follows: reshape the input A (C×N), where N=H×W, and C is the feature obtained by convolution of the input A; multiply the two feature map results of reshape and transpose (N×C) of the input feature A, and then obtain the feature map X of size C×C through the following softmax normalization formula:
[0025]
[0026] where x ji represents the influence of the ith channel on the jth channel, A i , A j are the i-th and j-th values in feature A.
[0027] Then transpose X to get the transposed feature of size C×C, multiply the transposed feature with the reshaped A(C×N) as the feature map, multiply the result by the scale factor β, reshape it to the feature map of the original resolution, and finally add it to A as the matrix to get the output result feature map E;
[0028]
[0029] E j Represents the value in feature E, and the parameter that is gradually learned after β is initialized to 0.
[0030] It also includes the use of a pyramid-like image fusion method. Since the low-level feature map includes more semantic information, while the high-level feature map includes more detail information, the feature outputs of different resolutions generated by the previous network are aggregated from high to low levels, which is conducive to strengthening the expression of detail information in the input and improving the detection and segmentation capabilities of hard exudate lesions.
[0031] Beneficial results of the present invention:
[0032] 1. This technology adds a dual attention module to the traditional Unet model, which is conducive to focusing on important lesion areas; and adds more multi-scale fusion to extract more features and semantic information; uses a multi-level hole convolution structure to improve the model's ability to segment lesions of different sizes; finally, uses a pyramid fusion method, so that the improved model can effectively improve the detection ability of hard exudates.
[0033] 2. Aiming at the class imbalance problem of fundus hard exudates, a loss function based on Dice Loss is proposed, which can improve the recall rate of target segmentation and the weight of wrong segmentation. Applying this loss function to the improved Unet model above is conducive to improving the segmentation effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is the overall flow chart of the method of the present invention;
[0035] Figure 2 It is the structural diagram of the improved Unet network model;
[0036] Figure 3 is the flowchart of the spatial attention module;
[0037] Figure 4 is the flow chart of the channel attention module;
[0038] Figure 5 Schematic diagram of multi-level dilated convolution. DETAILED DESCRIPTION
[0039] The present invention is described in more detail below based on the accompanying drawings and data sets. Figure 1 , specifically including the following steps:
[0040] The method for detecting fundus lesions based on improved U-NET shown in the present invention can be implemented by the following technical solutions:
[0041] Step 1: taking the dataset image containing fundus lesions as the original data sample and performing data preprocessing on it;
[0042] Step 2: Professionals then manually mark the hard exudate lesions in the fundus image to obtain a data set with labeled information required for training the network model, and divide the obtained data set into a training set and a test set;
[0043] Step 3: Input the training set obtained above into the improved Unet model, train the network parameters of the model, and obtain the hard exudate lesion detection model
[0044] Step 4: Input the test set obtained in step 2 into the hard exudate detection model obtained in step 3 for image segmentation, and further fine-tune the model based on the segmentation results.
[0045] Furthermore, the image preprocessor in step 1 preprocesses the collected image as follows: first fill the image, scale it to a fixed size with equal length and width, then flip it horizontally and upside down to expand the number of data sets, then adjust the image brightness and perform Mosaic data enhancement on the image. Mosaic enhancement is to combine four images into one image, which can increase the number of target objects, especially the number of small targets, thereby improving the training effect of the model. At the same time, the system can read four images at a time to improve the running efficiency of the model.
[0046] Furthermore, the above-mentioned improved network is a corresponding improvement made on the basis of the Unet model for the problem in the field of fundus hard exudate lesions. Specifically, first, an attention mechanism module is added at multiple scales. The attention mechanism is a method that allows the network to learn a set of weight coefficients and dynamically weight each target area to focus more on the information of the area of interest, that is, the information of the lesion area. A dual attention module is added to the jump connection layer, including channel attention and spatial attention modules, which can pay more attention to the information of the hard exudate lesion area and improve the detection effect. Then, the feature map obtained by the attention module for each input feature of the multi-scale of the encoding layer is used as the input of the previous attention feature on the one hand, and as the input of the decoding feature on the other hand, so that the multi-scale information can be integrated into the final output feature to improve the segmentation effect.
[0047] Furthermore, the CNN network uses the local receptive field to extract features from the image, which may lead to inconsistent expressions of elements at the same position in the final output feature map, resulting in inconsistency within the class and thus causing segmentation errors; Figure 2 As shown in the figure, the Unet network improves spatial features by fusing the previous and next scales, but does not take into account the importance and connectivity of each position. In order to increase the detection accuracy of fundus hard exudate lesions, this paper proposes an efficient pyramid attention module based on the Unet model. The spatial attention module and the channel attention module are used at multiple scales to adaptively learn local features and global dependencies. The attention module introduces a dual attention network (DANet), which aggregates two results:
[0048] (1) Spatial Attention Module:
[0049] The spatial attention mechanism module is as follows Figure 2 As shown, features B, C, and D are obtained by convolution of input A, and the size is the same as A. Then B, C, and D are reshaped to C×N (N=H×W); then C is transposed, and the result is multiplied by B to obtain an N×N matrix S. Softmax is performed on each point of S. The calculation formula of softmax is as follows. The result obtained above is multiplied by D and reshaped to a feature map of size C×H×W, and it is added to A to obtain the final result. The above operation is shown in the following formula:
[0050]
[0051]
[0052] From the above process, we can know that each position pixel of the final result has a position weight relationship with other pixels.
[0053] (2) Channel Attention Module:
[0054] Channel attention module such as Figure 3 As shown, first reshape A (C×N), and multiply the two feature map results of reshape and transpose (N×C) of A, and then obtain the feature map X of size C×C through softmax, then transpose X to obtain the transposed map of size C×C, multiply the transposed map with the reshaped A (C×N) feature map, multiply the result by the scale factor α, and then reshape it to the feature map of the original image size, and finally add the matrix with A to obtain the output result feature map. The α initialization value is 0, and it is learned through the network to obtain a larger weight. The formula for softmax and matrix addition is as follows:
[0055]
[0056]
[0057] Furthermore, hard exudates have different sizes due to their different development cycles. In Unet, a single feature extraction network is used for hard exudates of different sizes, which cannot better capture the information of lesions of different sizes. In order to solve this problem, Figure 5 As shown in the figure, the present invention adopts a multi-level dilated convolution structure, which can increase the receptive field without increasing the amount of calculation, and is conducive to obtaining information about hard exudates of different sizes. By using a multi-level dilated convolution module in the last step of the Unet network downsampling process, the feature map results obtained by convolution are fused and used as the input of the decoding process, the model's detection effect on lesions with multi-scale changes will be effectively enhanced.
[0058] Furthermore, the final output module uses a pyramid feature fusion structure. In this structure, the feature maps of different resolutions extracted by the previous network, the upper high-resolution map contains more detailed features, while the lower low-resolution map has more semantic information. Therefore, the output method is to fuse them from high-level to low-level, which is conducive to enhancing the representation of detailed features and improving the accuracy of the network in segmenting microvascular tumors; through this pyramid feature extraction result, more detailed information can be obtained, which can greatly improve the segmentation accuracy of the model for hard exudates.
[0059] Furthermore, the segmentation accuracy of the model still has room for improvement. In some fundus lesion images of the present invention, the pixel area of hard exudates in the entire fundus lesion image is very small, which makes it difficult to identify and segment. At the same time, the number of samples of hard exudate lesions in the image is not large, which makes it easy for the model to classify all pixels as background or other categories, resulting in a serious class imbalance problem. The present invention proposes a loss function based on the improved DiceLoss to address the class imbalance problem of hard exudate lesions, which can effectively improve the detection of hard exudate lesions.
[0060] The definition of the Dice Loss function is as follows:
[0061]
[0062] where p kc represents the probability that pixel k belongs to category c, represents the probability that pixel k does not belong to category c.
[0063] In fundus lesion detection, it is necessary to correctly segment as many lesion detections as possible, that is, to improve the recall rate. However, a limitation of the Dice Loss function is that it does not distinguish between false positives (FP) and false negatives (FN). When training the model, it is necessary to give higher weights to false negatives (FN) to improve the recall rate:
[0064]
[0065] Among them, α and β are adjustable variables.
[0066] Another problem with the Dice Loss function is that for small targets, segmentation errors will not have much impact on the loss. In fundus lesion images, some hard exudates are small in area and are easily affected by image quality such as image brightness and noise, which makes some lesions difficult to detect. These samples are called difficult-to-segment samples. To solve this problem, the following loss function is proposed based on the above:
[0067]
[0068] Where σ is an adjustable variable.
[0069] Use this improved loss function to supervise the training model to obtain better segmentation effect.
Claims
1. A deep learning-based method for segmenting fundus hard exudates, characterized in that: The method specifically comprises the following steps: Step 1: taking the dataset image containing fundus lesions as the original data sample and performing data preprocessing on it; Step 2: manually mark the hard exudate lesions in the fundus image to obtain a data set with labeled information required for training the network model, and divide the obtained data set into a training set and a test set; Step 3, input the obtained training set into the improved Unet model, supervise the network training through the improved Dice Loss function, and obtain the hard exudate lesion detection model; The improved Unet model adds a dual attention module to the skip connection layer, including a channel attention module and a spatial attention module; the feature map obtained by passing each multi-scale input feature of the encoding layer through the attention module is used as the input of the previous attention feature on the one hand, and as the input of the decoding feature on the other hand; the last step of the downsampling process uses a multi-level hole convolution module to fuse the feature map results obtained by the convolution and use them as the input of the decoding process; the improved Dice Loss loss function is specifically: DC c Represents the loss result, α and β are adjustable variables, where p kc is 1, indicating that pixel k in the output result belongs to category c. is 0, indicating that pixel k in the output result does not belong to category c, g kc is 1, indicating that the pixel k of the original labeled image belongs to category c. is 0, indicating that the pixel k in the original labeled image does not belong to category c; ε is a constant, and N represents the number of sample pixels; DM c represents the loss result after processing, σ is the adjustment parameter; Step 4: Input the test set obtained in step 2 into the hard exudate detection model obtained in step 3 for image segmentation, and further fine-tune the model based on the segmentation results.
2. The deep learning-based method for segmenting fundus hard exudates according to claim 1, characterized in that: The data preprocessing in step 1 is as follows: The image is padded and scaled to a fixed size with equal length and width, and then flipped horizontally and upside down to expand the number of data sets. The brightness of the image is adjusted and the image is subjected to Mosaic data enhancement. Mosaic enhancement is to combine four images into one image, which can increase the number of target objects, especially the number of small targets, thereby improving the training effect of the model.
3. The deep learning-based method for segmenting fundus hard exudates according to claim 1, characterized in that: The spatial attention module is as follows: input A is convolved to obtain features B, C, and D, and the resolution of features B, C, and D are the same as that of A. Then, features B, C, and D are reshaped to C×N, where N=H×W; then feature C is transposed, and the result is multiplied with B to obtain an N×N matrix S, and each point of S is softmax normalized using formula (3); where s ij represents the influence of the i-th position on the j-th position, B i represents the i-th value in feature B, C j represents the jth value in feature C; The result is multiplied by D and reshaped into a feature map of size C×H×W, which is then added to A to obtain the final result, as shown in formula (4): E j represents the value in feature E, the parameter gradually learned after α is initialized to 0, and D i represents the i-th value in feature D, A j Represents the jth value in feature A.
4. The deep learning-based method for segmenting fundus hard exudates according to claim 1, characterized in that: The channel attention module described above is as follows: reshape the input A (C×N), where N=H×W, and C is the feature obtained by convolution of the input A; multiply the two feature map results of reshape and transpose (N×C) of the input feature A, and then obtain the feature map X of size C×C through the following softmax normalization formula: where x ji represents the influence of the ith channel on the jth channel, A i , A j are the i-th and j-th values in feature A; Then transpose X to get the transposed feature of size C×C, multiply the transposed feature with the reshaped A(C×N) as the feature map, multiply the result by the scale factor β, reshape it to the feature map of the original resolution, and finally add it to A as the matrix to get the output result feature map E; Where E j Represents the value in feature E, and the parameters gradually learned after β is initialized to 0.
