Multi-scale small target detection method in retinal fundus images
A multi-scale neural network model addresses the challenge of detecting microaneurysms in retinal fundus images by enhancing feature expression and fusing multi-scale features, achieving improved detection accuracy and efficiency.
Patent Information
- Application Number
- CN202110262435.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-11
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2041-03-11
AI Technical Summary
Existing methods for detecting microaneurysms in retinal fundus images face challenges due to their small size and varying scales, leading to inefficiencies in manual inspection and limited accuracy in computer-assisted detection, particularly with traditional and deep learning techniques.
A multi-scale neural network model is designed to detect and classify microaneurysms using a novel neural network architecture that includes adaptive sampling, a Feature Pyramid Network (FPN) with a Convolutional Block Attention Module (CBAM) to enhance feature expression and fuse multi-scale features, and a Spatial Pyramid Pooling (SPP) framework to handle images of varying scales.
The method achieves accurate and efficient classification of microaneurysms across different scales, improving detection accuracy and reducing computational costs by effectively extracting and fusing semantic features.
Smart Images

Figure CN115082372B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a multi-scale small target detection method in retinal fundus images, and belongs to the fields of computer vision and medical image processing. Background Art
[0002] Microaneurysms (MAs) are small targets in retinal fundus images and are also considered to be the earliest lesions of diabetic retinopathy (DR). In color fundus images, MAs appear as round, dark red spots of varying scales. Manual inspection by ophthalmologists is time-consuming and labor-intensive. Therefore, accurate computer-assisted MA detection technology is crucial for large-scale screening of DR. Many methods have been published to detect MAs on public and clinical databases. Before deep learning was widely used in computer vision tasks, traditional methods aimed to design a two-stage framework and extract manual features for classification. In addition, deep learning techniques have been introduced based on an end-to-end architecture to extract contextual features and detect MAs.
[0003] Many studies have proposed a two-stage framework for detecting MAs separately. Mathematical morphology techniques are used for candidate extraction, resulting in the emergence of a few MAs and a large number of non-MAs. Visual features are extracted using supervised classifiers (e.g., k-nearest neighbor (KNN)) for accurate classification. The imaging principles of RGB retinal fundus images were studied, and it was concluded that the green channel contains the most retinal information and has the highest contrast. Therefore, the green channel is consistently used for MA detection. Furthermore, this approach proposes a nonlinear polynomial operator for contrast enhancement and shading correction. To improve candidate extraction performance, filter-based template matching is introduced instead of morphology to eliminate false candidates (e.g., blood vessels and hemorrhages). Although more MAs can be roughly detected, a large number of non-MAs remain in the candidate set. To distinguish these, various features (e.g., statistics, shape, and texture) have been investigated for classification. One statistical feature scans the intensity along different directions and calculates a cross-sectional profile. MAs are characterized by a distinct Gaussian peak at the center of the profile. Consequently, this feature has been widely used in the classification stage, achieving significant results. Based on the circular shape of MA, a Gaussian filter is used to extract circular candidates along multiple directions, and a set of local convergence index features from the shape adaptive filter are found, and the fragments of fine blood vessels can be accurately removed. In addition, many methods encode the local area of MA and generate a sparse texture feature that can detect the true MA. Depending on the rotation and brightness invariance of the local binary pattern (LBP), the LBP operation is used to represent the local texture features of MA. However, due to the low representation power of the hand-crafted features, these mentioned methods still suffer from limited detection accuracy.
[0004] Recently, deep convolutional neural network (DCNN) technology has been widely used for MA detection. A pixel-level MA classification method based on a convolutional neural network (CNN) was previously proposed. In RGB color fundus images, each pixel can be classified as MA or non-MA. The basic CNN computes a probability map for the training image and generates informative patches resembling MA. To achieve more accurate classification, a deeper final CNN obtains a set of candidate patches for each MA and smoothes the probability map using a disk kernel to remove noise. Morphological algorithms and Gaussian filters are applied to suppress background noise. For convenience, the resolution of all images is normalized using bicubic interpolation. To improve representativeness and the capabilities of the CNN, the network was retrained using another dataset from the same domain. Furthermore, only layers that extract specific features are fine-tuned, while the weights and biases of other layers are frozen. This staggered freezing approach reduces computational cost. The network focuses on network correction and reclassification, similar to reinforcement learning. The reclassification network is driven by the proneness of misclassification results. The purpose of the reclassification network is to minimize the difference between the refined classification results and the ground truth. To improve accuracy, a validation network is introduced to determine whether to accept the reclassification results. Compared with the two-stage framework-based method, the deep learning-based method achieves higher accuracy for MA classification.
[0005] MAs, as small targets in color fundus images, are small and of varying sizes. Using only a single patchsize to construct a sample set makes it difficult to accurately include the local and global information of MAs of different sizes. If the sample size is too large, smaller MAs will be lost in the background, making it impossible to extract local features. Conversely, if the sample size is too small, the network will have difficulty learning effective features of the target's local neighborhood or background. Therefore, this network first calculates the optimal receptive field of each MA and negative sample based on the Laplacian scale space theory to obtain the patchsize of each MA and construct a training sample set. Then, based on the Spatial Pyramid Pooling (SPP) framework, which is suitable for the excellent properties of multi-scale patches, patches of different scales are unified to the same dimension through the SPP layer after the network input layer, realizing multi-scale fusion of visual features.
[0006] In addition, for small and weak targets in images such as MA, traditional deep networks will lose a lot of target information after multiple convolutions and pooling, making it difficult to extract rich and effective semantic features (semantic features can better represent the intrinsic characteristics of the target). Therefore, after a lightweight conv+pooling operation, this network introduces the Feature Pyramid Network (FPN), which not only avoids the large-scale loss of features, but also enables multi-scale fusion of semantic features. At the same time, in the Feature Pyramid Network, replacing up-sampling with deconv can effectively make up for the defect of feature detail loss caused by conv&pooling. In addition, a visual attention mechanism (Convolutional Block Attention Module) is added to FPN to optimize and fuse features, thereby better improving the ability of feature expression. Summary of the Invention
[0007] The present invention aims to design a multi-scale neural network model to detect small objects (MA) in fundus images.
[0008] In order to achieve the above object, the scheme of the present invention is:
[0009] Design a multi-scale neural network model to detect and classify small target areas in fundus images. A novel neural network architecture is proposed that can accept input images of any scale, enhance feature expression, focus on salient features, and accurately classify images. The current steps are as follows:
[0010] (1) Eight public datasets are selected, including ROC, DB1, e-ophtha, RC-RGB-MA1, RC-RGB-MA2, RC-SLO-G, RC-SLO-I, and IDRiD. The optimal receptive field of each MA and negative sample is calculated based on the Laplacian scale space theory to obtain the patch size of each MA.
[0011] (2) For the calculation of positive and negative sample patches, the positive sample (MA) is centered on the centroid of the connected domain (gold standard) of the MA area, and the negative sample is a patch area randomly cut out of the image except the MA area.
[0012] (3) MA detection is a binary classification task, and the number of negative samples in fundus images is much larger than the number of positive samples, resulting in uneven training samples. In order to ensure that the network can fully learn the characteristics of positive and negative samples, this network constructs a training sample set with a positive-to-negative sample ratio of 1:2;
[0013] (4) Design a neural network that can train samples of different scales and achieve the classification function of positive and negative samples;
[0014] (5) Conduct multiple experiments to explore the optimal network parameters and evaluate the model using multiple evaluation metrics, including accuracy, precision, sensitivity, and F1 score;
[0015] (6) Design ablation experiments to explore the specific problems solved by each module;
[0016] (7) Prove the feasibility and superiority of this method, that is, this method can achieve correct and accurate classification of various fundus images.
[0017] The beneficial effects of the present invention are: this method can extract features from fundus images of various scales and use shared parameters from the same part to make positive and negative predictions for test samples of different scales. Based on neural networks, we manually captured positive and negative sample images of various sizes using various public datasets to train a suitable network model. On top of completing the basic classification problem, we further solved a challenging problem: multi-scale image classification. To solve this problem, we constructed the ASPP module, adaptively pooled the images, used a special feature fusion mechanism, and introduced the attention mechanism and feature pyramid fusion method to obtain effective features that are more conducive to classification tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 This is the flow chart of dataset production.
[0019] Figure 2 is a flowchart of the hierarchical pyramid network algorithm for microaneurysm detection in retinal fundus images. DETAILED DESCRIPTION
[0020] A classification algorithm for processing multi-scale fundus images overcomes the limitation of traditional deep learning algorithms that can only process a single scale, and makes a small contribution to medical research in the field of fundus lesion area classification.
[0021] Eight public datasets are selected, including ROC, DB1, e-ophtha, RC-RGB-MA1, RC-RGB-MA2, RC-SLO-G, RC-SLO-I, and IDRiD. The optimal receptive field of each MA and negative sample is calculated based on the Laplacian scale space theory, so as to obtain the patch size of each MA as the input of the neural network, such as Figure 1 shown.
[0022] Based on the labeled fundus image patches, convolutional neural networks (CNNs) and deep learning methods are used to achieve binary classification and learn the features of the lesion areas in the fundus images. Convolutional neural networks are a type of feedforward neural network with a deep structure that includes convolution calculations and are one of the representative algorithms of deep learning.
[0023] We selected eight scales of input images: 19×19, 21×21, 23×23, 25×25, 27×27, 29×29, 31×31, and 33×33, and used a new T-type network framework. First, after the first stage of feature extraction to increase channel correlation, the ASPP structure is used to perform adaptive pooling for input images of different scales. After the visual features are over-scaled and fused, feature maps of the same size are obtained for feature extraction in the second stage. The feature extraction in the second stage includes two stages: downsampling and upsampling. The cbam attention mechanism is introduced in the downsampling stage, and the pyramid fusion (multi-scale fusion of semantic features) mechanism is introduced in the upsampling stage. There are tens of thousands of sample images for each scale, and the experiment is conducted using a ratio of training set: validation set: test set = 7:2:1. The algorithm flow is as follows: Figure 2 shown.
[0024] In our neural network algorithm, the ASPP architecture uses adaptive pooling to generate 5×5, 7×7, and 9×9 feature maps. Padding and adding are used to fuse these feature maps, allowing input images of different scales to generate feature maps of the same scale, 9×9. The fused feature maps undergo three layers of downsampling for feature extraction. The output of each downsampling layer is then subjected to a channel-wise and spatial-wise attention mechanism, respectively. This allows for more accurate emphasis on valid features and prevents interference from background and other noise. After extracting valid features, the feature maps are restored through three layers of upsampling. These features are then fused using a feature pyramid with the feature maps of different scales from the downsampling. Finally, the feature maps from each layer of the pyramid are pixel-wise summed to form the feature map used for classification, ultimately yielding a binary classification prediction.
[0025] Based on the detected confusion matrix, calculate the accuracy, precision, sensitivity, and F1 score to comprehensively evaluate the model performance:
[0026]
[0027]
[0028]
[0029]
[0030] TP (True Positive) represents true positives, FN (False Negative) represents false negatives, and FP (False Positive) represents false positives. They represent different prediction results. The higher the result, the better the model performance.
[0031] It should be noted that the above is only an embodiment of the present invention and is only for explaining the present invention, and does not limit the scope of the present invention. Any obvious changes that fall within the technical concept of the present invention are also within the scope of protection of the present invention.
Claims
1. A multi-scale small target detection method in retinal fundus images, comprising the following steps: (1) Eight public datasets are selected, and the optimal receptive field of each MA and negative sample is calculated based on the Laplacian scale space theory to obtain the patch size of each MA; (2) Manually calculate the positive and negative sample patches. The positive sample is centered on the centroid of the connected domain of the MA region and the patch is cut. The negative sample is a patch area randomly cut from the image except the MA region as the negative sample. (3) Create labels corresponding to the data set and standardize and normalize the images using preset values; (4) Design a neural network applicable to the existing dataset to achieve the classification of microaneurysms in multi-scale retinal fundus images; (5) Evaluating the performance of the neural network by calculating accuracy, precision, sensitivity, and F1 score; The neural network uses the ASPP method to perform adaptive pooling on fundus images of different scales, and then performs multi-scale feature fusion to obtain a fused feature map, so that images of different sizes can obtain feature maps of the same scale for subsequent training of shared parameters, and introduces an attention mechanism and a feature pyramid fusion method to obtain effective features; The multi-scale feature fusion uses a pyramid fusion mechanism. After extracting effective features, the detail information of the feature map is restored through a three-layer upsampling operation, and the feature pyramid fusion is performed with the feature maps of different scales in the downsampling. Finally, the feature map of each layer of the pyramid is subjected to pixel accumulation operation as the feature map for classification, and finally a binary classification prediction result is obtained.
2. The multi-scale small target detection method in retinal fundus images according to claim 1, characterized in that: The neural network in step (4) is a feature extraction network, which is used to extract effective feature information of the lesion area to complete the subsequent classification task.
3. According to the multi-scale small target detection method in retinal fundus images as described in claim 1, the attention mechanism performs three-layer downsampling feature extraction on the fused feature map, and the output of each layer of downsampling passes through the channel attention mechanism and the spatial attention mechanism respectively, which can more accurately emphasize the effective features and avoid interference of background and other noise on the effective feature information.
