Eye fundus image detection method based on visual dual-channel cognitive mechanism

Through a parallel multi-path eye disease image detection network based on visual dual-path cognitive mechanism, the problem of difficulty in detecting multiple eye diseases at the same time in the existing technology is solved, and the disease such as diabetic retinopathy, cataract, glaucoma and pathological myopia is realized. Efficient testing improves the efficiency and accuracy of early screening.

CN119048828APending Publication Date: 2024-11-29SOUTHWEST JIAOTONG UNIV
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411172634.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-26
Publication Date
2024-11-29

Smart Images

  • Figure CN119048828A_ABST
    Figure CN119048828A_ABST
Patent Text Reader

Abstract

The invention discloses a fundus image detection method based on a visual dual-channel cognitive mechanism, which is applied to the field of image detection and aims at solving the problems of single detection result and low detection accuracy of the existing fundus image detection method. The invention provides a parallel multi-path network for image detection of various eye diseases, which is called PMP-OD and integrates detection of various common eye disease images, including cataract, glaucoma, diabetic retinopathy and pathological myopia. The specific detection process is as follows: bottom-up (low-level) features of an eye fundus image are extracted by a universal convolution module (eye fundus feature extraction) simulating a non-selective channel; meanwhile, top-down (advanced) retinal blood vessel features are extracted by a blood vessel feature extraction module simulating a selective channel; secondly, fusing the features through a feature fusion module based on an attention mechanism; and finally, the disease classifier gives a prediction result according to the integrated multiple features.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of image detection, and in particular relates to a fundus image detection technology. Background Art

[0002] In recent years, the number of people who have become blind due to various eye diseases has been on the rise. In many places, more than 60% of adult blindness is mainly caused by five eye diseases: diabetic retinopathy, cataracts, glaucoma, high myopia, and age-related macular degeneration. Through early screening and treatment, most of the irreversible vision loss can be avoided, and some patients with eye diseases can even regain their sight. Retinal fundus imaging is a common screening technology that requires experts with ophthalmic expertise to manually identify possible disease conditions on fundus images. However, in many areas, the screening base is large and there are relatively few ophthalmologists, making it difficult to achieve large-scale early screening of diseases. Therefore, computer-aided diagnosis (CAD) to automatically detect eye diseases has become a trend, which will help popularize eye disease screening and effectively reduce the workload of doctors.

[0003] However, most of the existing eye image detection methods target a certain disease, such as diabetic retinopathy, glaucoma, cataracts, myopia, etc. Such a system may not be suitable for the actual eye disease screening scenario of the general public, because ordinary people suffer from more than one eye disease, so it is necessary to achieve simultaneous detection of multiple diseases. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a fundus image detection method based on a visual dual-pathway cognitive mechanism. Based on a parallel multi-path eye disease image detection network (PMP-OD, Parallel Multi-Path Networkfor Ocular Disease Detection), it can realize the detection of common eye disease images such as diabetic retinopathy, cataract, glaucoma and pathological myopia.

[0005] The technical solution adopted by the present invention is: a fundus image detection method based on a visual dual-pathway cognitive mechanism, comprising:

[0006] S1. Construct a training dataset including images of various eye diseases;

[0007] S2, build a parallel multi-path eye disease image detection network;

[0008] S3, based on the training data set constructed in step S1, training the parallel multi-path eye disease image detection network constructed in step S2;

[0009] S4. Input the fundus image to be detected into the trained parallel multi-path eye disease image detection network to obtain the detection result.

[0010] Beneficial effects of the present invention: In order to assist in the diagnosis of eye diseases, inspired by the dual-path theory and the theory of visual attention mechanism, the present invention proposes a multi-disease image detection model for the eye based on a parallel multi-path network, which can simultaneously detect cataracts, glaucoma, diabetic retinopathy and pathological myopia. The fundus feature extraction and vascular feature extraction modules extract bottom-up fundus image features and global retinal vascular features. Then, these features are fused by a feature fusion module based on an attention mechanism. Finally, the classification result is obtained through a disease classifier module. Experimental results show that the model of the present invention achieves competitive performance compared with other classification models in the task of disease image detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a schematic diagram of the human visual information processing mechanism;

[0012] Figure 2 A structural diagram of the PMP-OD model provided by the present invention;

[0013] Figure 3 A structural diagram of a fundus feature extraction module provided by the present invention;

[0014] Figure 4 A structural diagram of a blood vessel feature extraction module provided by the present invention;

[0015] Among them, (a) is the structural diagram of the vascular feature extraction module, and (b) is the structure of the basic block;

[0016] Figure 5 A structural diagram of the feature fusion module provided by the present invention;

[0017] Figure 6 This is a structural diagram of the disease classifier provided by the present invention. DETAILED DESCRIPTION

[0018] To facilitate those skilled in the art to understand the technical content of the present invention, the present invention is further explained below with reference to the accompanying drawings.

[0019] This paper proposes an eye image detection model that integrates the detection of several common eye disease images such as diabetic retinopathy, cataract, glaucoma and pathological myopia, called Parallel Multi-Path Eye Disease Image Detection Network (PMP-OD). The eye disease image detection model proposed in this paper is a hybrid architecture based on convolutional neural network and attention mechanism, which is inspired by the human visual information processing mechanism, such as Figure 1 shown.

[0020] The human visual system is a complex perceptual system. Studies have shown that information entering the visual search system follows two different pathways: the non-selective pathway extracts global spatial information, and the selective pathway processes local and high-level information images. Similarly, human visual attention consists of bottom-up and top-down models. The bottom-up model is driven by low-level image features such as color, material, shape, etc. The top-down model is driven by specific tasks, goals, and experience.

[0021] From the perspective of ophthalmologists, the results of vascular information in fundus images can be used as prior information (top-down) to assist in the diagnosis of diseases, while other image details other than blood vessels may contain focal information reflecting the disease condition. Therefore, the present invention considers combining them to construct a model that can perform intelligent disease image detection on fundus images more comprehensively.

[0022] Inspired by these two information processing pathways in visual search theory and combining the bottom-up and top-down models in visual attention mechanism, we designed a parallel multi-path network based eye disease image detection model (PMP-OD). The structure of the proposed model is shown in Figure 2 As shown, the bottom-up pathway represents a non-selective pathway for extracting bottom-up features of the fundus image, and the top-down pathway represents a selective pathway aimed at obtaining top-down fundus image information.

[0023] Figure 2 In it, Bottom-Up means top-down, Fundus Feature Extract means fundus feature extraction, Top-Down means top-down, Vessel Feature Extract means vessel feature extraction, Feature fusion means feature fusion, Disease Classifier means disease classifier, Stage1, Stage2, Stage3, Stage4 means 4 stages, Preprocessing means preprocessing, Down Sampling means down sampling, Up Sampling means up sampling, Patch Embedding means block embedding, Transformer Encoder means self-attention encoder, MLP Head means multi-layer perceptron detection head, Non-Selective pathway means non-selective pathway, and Selective pathway means selective pathway.

[0024] The model has two feature extraction branch networks: the low-level feature extraction module (LFE) simulates the non-selective pathway, and the high-level feature extraction vascular feature extraction module (HFE) simulates the selective pathway.

[0025] First, the input fundus image is sent to the LFE module to extract the basic features of the fundus image. At the same time, the input image is sent to the HFE module to extract the retinal vascular features as prior information. Then, these two sets of features are fused through a feature fusion module based on the attention mechanism. Finally, the comprehensive multi-features are input to the Transformer classifier, and the predicted eye disease category of the input image is output, thereby realizing intelligent diagnosis of fundus imaging diseases.

[0026] Fundus Feature Extraction Module (LFE)

[0027] In order to obtain information about the fundus of retinal vascular images, the present invention designs a fundus feature extraction module (LFE). It uses ResNet as the backbone model to extract the overall features of the fundus image. The architecture of LFE is as follows Figure 3 shown.

[0028] LFE adapts the ResNet-50 model and re-customizes the layer distribution in the Vision Transformer to verify the performance of the hybrid model. The hybrid model still has 50 weighted layers, but the number of layers of each type of convolution kernel is changed to meet the conditions of the subsequent ViT embedding input.

[0029] The LFE module converts the original input 224×224 pixels into a 14×14 pixel output and consists of four stages. The first stage consists of a transposed convolution (i.e. Figure 3 Std Conv in), group normalization (i.e. Figure 3 GN in ), ReLU and MaxPool (maximum pooling) operations. The second to fourth stages consist of convolution blocks (i.e. Figure 3 Conv Block in ) and different numbers of identification blocks (i.e. Figure 3 The primary image features F of the fundus image can be obtained through the LFE module. FFE , it is possible to extract rich primary features of fundus images containing useful vascular information and background information.

[0030] Vascular Feature Extraction Module (HFE)

[0031] In this work, the vascular feature extraction (HFE) module is used to extract retinal vascular features to assist in the subsequent extraction and classification of deeper features. In the HFE module, we apply an encoder-decoder based structure, and use skip connections and residual modules on this basis to obtain more information flow paths.

[0032] The structure of HFE is as follows Figure 4 (a). The network consists of two downsampling (i.e. Figure 4 Down-sampling) branch and an up-sampling (i.e. Figure 4 The Up-sampling) branch in the .

[0033] The basic blocks of the HFE model (i.e. Figure 4 The Basic block in is a common residual block with shared weights, such as Figure 4 As shown in (b), the structure of the basic block includes the first 3×3 convolutional layer (i.e. Figure 4 3×3Conv in ), the first ReLU layer, and the random dropout layer (i.e. Figure 4 The data of the input basic block is processed by the first 3×3 convolution layer, the first ReLU layer, the random dropout layer, and the second 3×3 convolution layer in sequence. The data is added to the input through the Add layer as the input of the second ReLU layer. The output of the second ReLU layer is the output of the basic block.

[0034] Both upsampling and downsampling branches use this basic module and then use convolution or transposed convolution (i.e. Figure 4 Conv Transpose in ) replaces pooling to achieve upsampling or downsampling, and finally obtains the feature F containing retinal vascular information. VFE .like Figure 4 As shown in the figure, the first downsampling module includes five layers of structures in sequence, specifically: the first layer consists of 3×3 inConv and ReLU, and the second to fifth layers consist of basic blocks and convolutions; the second downsampling module includes four layers of structures consisting of basic blocks and convolutions in sequence; the upsampling module includes four layers of structures consisting of transposed convolutions and basic blocks in sequence; and there is also a basic block between the first downsampling module and the upsampling module.

[0035] The HFE module can extract rich retinal vascular features from fundus images, effectively encode low-level and high-level features of the vessels, and obtain useful vascular feature information.

[0036] Figure 4In the above figure, HFE Module represents HFE module, input represents input, and output represents output.

[0037] Feature Fusion

[0038] The feature F generated by the previous module LFE and F HFE Both can be used as the basis for disease image detection, so the present invention uses the fusion features of the two as the input of the final disease image detection classifier. Since the attention mechanism can better fuse features with inconsistent semantics and scales. In this embodiment, the attention gate (AG) will be used as a feature fusion technology. The attention gate (AG) is a spatial attention mechanism. The feature fusion diagram is shown in Figure 5 shown.

[0039] Input feature F LFE Scaled by the attention factor calculated in AG. By analyzing another input feature F HFE The activation and background information provided are used to select the spatial region. HFE (F H ×H H ×W H ×D H ) is convolved with a size of 1×1×1 to obtain In the feature map F LFE (F L ×H L ×W L ×D L ) is convolved with a size of 1×1×1 to obtain The feature maps obtained in the first two steps and After adding, ReLU is performed, and then a 1×1×1 convolution operation is performed to obtain The calculation is as follows:

[0040]

[0041] Where σ1 represents the ReLU function.

[0042] Then perform the Sigmoid activation function to get the attention coefficient It can be expressed as:

[0043]

[0044] Where σ2 represents the Sigmoid function, Θ att Represents all the parameters in the previous operation.

[0045] Finally, the attention coefficient is grid-resampled using trilinear interpolation to obtain the final attention coefficient α. The attention coefficient α is multiplied by the input feature map to produce the final output of the attention gate

[0046]

[0047] Disease Image Detection Classifier

[0048] In this module, the present invention uses Vision Transformer (ViT, a deep learning model based on attention mechanism) as the final disease classifier. Compared with the original Transformer, the ViT model needs to perform block segmentation and data transformation on the original image because the input is a two-dimensional image instead of a one-dimensional sequence. The schematic diagram of the Transformers classifier is shown in Figure 6 As shown. In general, this classifier consists of three parts: Patch Embedding, Transformer Encoder, and MLP Head. For image data, the data format is a three-dimensional matrix, so the data needs to be transformed through the Patch Embedding module, and the input features of size 1024×14×14 are convolved and mapped into a one-dimensional vector. The length of the 196 vector is obtained through the linear mapping of each patch. Then the class label dedicated to classification and the Position Embedding representing the position information are added to the vector. Finally, the vector is input to the Transformer Encoder, which consists of Encoder Blocks repeatedly stacked L times. Since ViT-B / 16 is used as the pre-training model, L is 12. After extracting the corresponding result generated by the class token from the output, the classification result is finally obtained in the format of a 1×5 probability vector through the MLP Head.

[0049] The effect of the detection method of the present invention is described below in combination with the existing data set:

[0050] Since many datasets contain only one disease, this embodiment collects fundus images from four datasets to synthesize a comprehensive dataset for multi-classification of eye diseases. This embodiment uses four datasets: ODIR-5K, APTOS-2019, ORIGA-light, and Kaggle. This embodiment uses five categories from ODIR-5K: Normal (N, Normal), Cataract (C, Cataract), Glaucoma (G, Glaucoma), Diabetic Retinopathy (DR, Diabetic Retinopathy), and Myopia (M, Myopia); Normal and proliferative DR images from APTOS-2019; Glaucoma images from ORIGA-light; Cataract and glaucoma fundus images from Kaggle to form the final dataset. The final dataset is divided into five categories: normal, diabetic retinopathy, glaucoma, cataract, and pathological myopia, with 3020, 1916, 562, 386, and 253 images, respectively, for a total of 6137 images. In this embodiment, the training set and the test set are randomly divided into a ratio of 3:1, and 20% of the training set is divided as a validation set.

[0051] The evaluation strategy used in this invention is divided into two parts: overall effect and each category. First, the overall Accuracy (ACC), Precision (PRE), Recall (RE) and F1 index of the model are evaluated based on all samples. They can be expressed as:

[0052]

[0053] Among them, TP, TN, FP, and FN represent the true positive, true negative, false positive, and false negative in the confusion matrix respectively.

[0054] When the proportion of samples in different categories is very unbalanced, the category with a large proportion often becomes the most important factor affecting the accuracy. In our data set, the number of samples in the normal and DR categories is much larger than that in other categories. Therefore, this embodiment also introduces the Balanced Accuracy Score (b-ACC) to measure the accuracy of the model. The balanced accuracy score is defined as the arithmetic mean of the sample accuracy in each category. For binary classification problems, it is half the sum of recall and specificity. For multi-class classification problems, it can be defined as follows:

[0055]

[0056] Where n represents the number of sample types, N i represents the number of samples in the i-th category, T i Represents the number of samples correctly classified into the i-th category.

[0057] This example is implemented based on the Pytorch framework. The experiment was conducted on a Tesla P100 GPU. During the training process, the Dice function was used as the loss function L Dice , using the AdamW optimizer with an initial learning rate of 5×e -5 , and restart the planner with warmed cosine annealing for 40 rounds. The R50+ViT-B_16 pre-trained model weights were used on Imagenet-21k. 224×224 was selected as the uniform image size. Each image was first scaled equally, with the shortest side scaled to 224 pixels and the long side cropped to the midpoint of each side exceeding 112 pixels.

[0058]

[0059] In order to comprehensively evaluate the performance of the method of the present invention, the model of the present invention is compared with other existing effective image classification networks, including models based on convolutional networks and models based on Vision Transformer, and trained using the same strategy and data. The results are shown in Table 1. The model of the present invention performs better than models of convolutional networks such as ResNet, DenseNet, MobileNetV2, ShufflenetV2, and EfficientNet-B0, and the ACC, PRE, and F1 indicators are all higher than these models.

[0060] Some recent Transformer-based models: ViT-B / 16, MobileViT, Swin-Transformer, compared with the convolutional network-based models, the models of the present invention generally show better performance. Interestingly, the model proposed in the present invention achieved better results in ACC, PRE, and F1, which are 93.32%, 93.25%, and 93.18%, respectively, which shows that the method proposed in the present invention can effectively detect images of different diseases. In general, the model of the present invention is superior to other methods, including the Transformer-based models based on convolutional networks. In addition, the inference speed of the PMP-OD proposed in the present invention reaches 28.1 frames per second (FPS), which can meet practical needs.

[0061] Table 1 Comparison of the proposed model with other methods

[0062]

[0063] This embodiment also provides a Kappa score, which is used to measure the consistency between the prediction result and the label, thereby penalizing the deviation of the model. The Kappa indicator calculation formula is:

[0064]

[0065] Among them, p0 represents the prediction accuracy, p e represents accidental consistency, C represents the number of categories, M ii represents the data in the i-th row and i-th column of the confusion matrix, N represents the sum of the confusion matrix, S i+ represents the sum of the i-th row in the confusion matrix, S +i represents the sum of the i-th column in the confusion matrix.

[0066] In addition, Table 2 shows the precision, recall and F1 score of the model of the present invention on each disease category, where only images of the corresponding disease are considered for each category. When testing our model with data of a certain disease, it can be seen that the model of the present invention works best on diabetic retinopathy and normal categories, followed by myopia and cataract.

[0067] Table 2 Indicators of the proposed model on each disease category

[0068]

[0069] In order to explore the role of each module in the network proposed by the present invention, the present invention conducted an ablation experiment. In the method of the present invention, Vison Transformer is used as the backbone model. The ViT model in Table 3 refers to the original ViT without any additional modules, using the original image as input. Hybrid ViT refers to introducing the LFE branch in the embedding module of ViT to extract features, which are then used for Position embedding.

[0070] The PMP-OD model proposed in the present invention introduces a parallel vascular feature extraction branch HFE on the basis of LFE. After HFE extracts the retinal vascular features, the attention gating mechanism is introduced to fuse the features extracted from the two branches. Table 3 shows the performance indicators of the three models under the same experimental conditions. Among them, Hybrid ViT is 0.9% and 0.44% higher than ViT in accuracy and balanced accuracy, and 0.91%, 0.9% and 0.87% higher than ViT in precision, recall and F1 metrics, respectively. Compared with Hybrid ViT, PMP-OD has an improvement of 0.32% and 0.93% in ACC and balanced accuracy, and an improvement of 0.29%, 0.32% and 0.34% in precision, recall and F1 metrics, respectively. The introduction of LFE and HFE modules improves model performance.

[0071] Table 3 Indicators of the proposed model on each disease category

[0072]

[0073] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the scope of protection of the present invention is not limited to such specific statements and embodiments. For those skilled in the art, the present invention may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A fundus image detection method based on a visual dual-pathway cognitive mechanism, characterized in that: include: S1. Construct a training dataset including images of various eye diseases; S2, build a parallel multi-path eye disease image detection network; S3, based on the training data set constructed in step S1, training the parallel multi-path eye disease image detection network constructed in step S2; S4. Input the fundus image to be detected into the trained parallel multi-path eye disease image detection network to obtain the detection result.

2. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 1 is characterized in that: The training data set described in step S1 at least includes: normal fundus images, cataract fundus images, glaucoma fundus images, diabetic fundus images and myopia fundus images.

3. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 2 is characterized in that: The parallel multi-path eye disease image detection network structure in step S2 includes: a fundus feature extraction branch, a vascular feature extraction branch, a feature fusion module, and a classifier; the input fundus image is respectively subjected to the fundus feature extraction branch and the vascular feature extraction branch to obtain fundus features and vascular features; the fundus features and vascular features are subjected to the feature fusion module to obtain fused features, and the fused features are input into the classifier to obtain the classification results.

4. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 3 is characterized in that: The fundus feature extraction branch includes four stages in sequence. The first stage includes transposed convolution, group normalization, ReLU and MaxPool operations in sequence; the second, third and fourth stages include convolution blocks and different numbers of identification blocks respectively.

5. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 4 is characterized in that: The vascular feature extraction branch includes a first downsampling module, an upsampling module and a second downsampling module in sequence; the first downsampling module includes a five-layer structure in sequence, specifically: the first layer composed of 3×3 inConv and ReLU, and the second to fifth layers composed of basic blocks and convolutions; the second downsampling module includes a four-layer structure composed of basic blocks and convolutions in sequence; the upsampling module includes a four-layer structure composed of transposed convolutions and basic blocks in sequence; and there is also a basic block between the first downsampling module and the upsampling module.

6. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 5 is characterized in that: The structure of the basic block includes, in sequence: a first 3×3 convolution layer, a first ReLU layer, a random dropout layer, a second 3×3 convolution layer, an Add layer, and a second ReLU layer. The data obtained after processing by the first 3×3 convolution layer, the first ReLU layer, the random dropout layer, and the second 3×3 convolution layer and the input data of the basic block are used as input of the Add layer.

7. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 6 is characterized in that: The feature fusion module is implemented using attention gate.

8. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 7 is characterized in that: The classifier is implemented using a deep learning model based on the attention mechanism.

9. The fundus image detection method based on the visual dual-pathway cognitive mechanism according to claim 8, characterized in that: The Dice function is used as the loss function during the training process of step S3.

Citation Information

Cited By

  • Eye fundus disease type identification method and system based on visual converter model, medium, terminal and program product

    CN119339168A