Diabetic retinopathy image classification method and system

Abstract

This invention discloses a method and system for classifying diabetic retinopathy images, belonging to the field of medical image processing technology. The technical problem this invention aims to solve is the low accuracy of diabetic retinopathy image classification and the localization issue of CNN networks' insufficient global understanding of retinal images. The technical solution adopted is as follows: The method specifically includes: Dataset acquisition and image preprocessing: Preprocessing the original dataset of diabetic retinopathy images and the patch dataset; Dataset feature extraction; Channel feature and global feature extraction; Channel-global feature and global-channel feature extraction; Feature fusion; Category feature extraction; Feature classification: Classifying the category features Q... cout With fusion features RC out The images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a method and system for classifying images of diabetic retinopathy. Background Technology

[0002] Current statistics show that 10.5% of adults aged 20-79 worldwide have diabetes, totaling 573 million, and this number is rapidly increasing. More than 30% of diabetic patients suffer from diabetic retinopathy (DR). DR is classified into five categories based on different stages of the disease: no DR, mild DR, moderate DR, severe DR, and proliferative DR. Diagnosis is primarily based on identifying minute features in retinal images, such as microaneurysms, hemorrhages, and exudates. Due to uneven distribution of medical resources and limited conditions in remote areas, patients often fail to recognize their condition in its early stages. Therefore, early diagnosis is extremely important in current medical treatments.

[0003] Manual examination of diabetic retinopathy is a tedious and demanding process. With the development of artificial intelligence in the medical field, machine learning and deep learning have been applied to the detection of diabetic retinopathy. Disease classification can assist doctors in early diagnosis, but its accuracy still needs improvement. Meanwhile, CNN models have limitations in extracting features from fundus images, mainly because convolution operations can only extract features from local neighborhoods, while accurate classification requires the network to pay attention to numerous small lesions unevenly distributed in the image. This makes it difficult for the model to retain all the features of small lesions in the retina. Therefore, improving the accuracy of diabetic retinopathy image classification, while simultaneously improving the accuracy of global understanding of retinal images, is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] The technical objective of this invention is to provide a method and system for classifying diabetic retinopathy images, in order to solve the problems of low accuracy in classifying diabetic retinopathy images and the lack of global understanding of retinal images by CNN networks.

[0005] The technical objective of this invention is achieved as follows: a method for classifying images of diabetic retinopathy, the specific method of which is as follows:

[0006] Dataset acquisition and image preprocessing: Image preprocessing was performed on the original dataset of diabetic retinopathy images and on the patch dataset;

[0007] Dataset feature extraction: Using Xception as the backbone network, feature extraction is performed on the original dataset to obtain the original dataset features BS1. Using ResNet50 as the backbone network, feature extraction is performed on the patch dataset to obtain the patch dataset features BS2.

[0008] Channel feature and global feature extraction: 20% of the original dataset features BS1 extracted from the backbone network Xception are randomly discarded to obtain random features B of the original dataset. Ein , the random features B of the original dataset Ein Channel features B are obtained by inputting into the channel attention module ECA. out ; 20% of the features BS2 extracted from the feature patch dataset of the backbone network ResNet50 are randomly discarded to obtain random features M of the patch dataset. nin The random features M of the patch dataset nin The input is fed into a non-local attention module (NAM) to obtain global features M. out ;

[0009] Channel-global feature extraction and global-channel feature extraction: extracting channel features B out The input is fed into the nonlocal attention module (NAM) to process channel features B. out Perform global location feature extraction to obtain channel-global feature MB. out ; global feature M out The input is fed into the channel attention module (ECA) to process the global feature M. out Perform channel feature extraction to obtain global-channel features BM out ;

[0010] Feature fusion: The original dataset features BS1 and the patch dataset features BS2, obtained through backbone network 1 (Xception) and backbone network 2 (ResNet50), are randomly discarded by 50% each, and then combined with the channel features B. out Global features M out Channel-Global Feature MB out and global-channel features BM out The fusion is performed to obtain the fusion feature RC out ;

[0011] Category feature extraction: fusing features RC out The category feature Q is obtained by feeding it into the category attention module CAB. cout ;

[0012] Feature classification: classifying the categorical features Q cout With fusion features RC outThe images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

[0013] As a preferred option, the dataset acquisition and image preprocessing are as follows:

[0014] The DDR dataset and the Messidor dataset were used as the datasets for five-class classification and two-class classification, respectively.

[0015] The image preprocessing is as follows:

[0016] Since the grading of diabetic retinopathy is achieved through microaneurysms, hemorrhages, and exudates in the retina, in order to better extract retinal image features from conventional retinal images, we remove background noise from the dataset images and retain the retinal area to the maximum extent.

[0017] Enriching image data using data augmentation techniques, specifically:

[0018] The classification of diabetic retinopathy is heavily based on microaneurysms in retinal images and circular localized capillary dilations in fundus images. The original dataset is segmented, and common lesion local areas are segmented as patches. A new patch dataset is created using the patch images. All black edge noise in the patch dataset is ignored, so that it focuses on more subtle lesion features.

[0019] In the training set, images are scaled and enlarged: the scaling factor is set to 1 / 255. The purpose of scaling is to enlarge or reduce the length and width of the image, changing the overall size of the image, not to crop the image; during the image enhancement process, the size is randomly enlarged or reduced by 0.45 times to enhance the image, and finally the image is flipped horizontally again.

[0020] Here, rotation range represents the angle by which the image is randomly rotated during training (rotating around the center of the image); height displacement represents a 0.2% shift in the vertical direction of the image; width displacement is similar to height displacement, except that the angle changes from the vertical direction to the horizontal direction by 0.2%.

[0021] More specifically, the extraction of channel features and global features is as follows:

[0022] By adding different attention modules (ECA), we can focus on different directions, extract more dimensional and targeted features, and leverage the local cross-channel interaction mechanism of ECA without dimensionality reduction to extract more channel information from the backbone network's xception features, thus improving the random features B of the original dataset. EinPerform channel-wise global average pooling;

[0023] After the GAP layer generates 1×1×C feature vectors, it is then... The adaptive convolutional kernel size is used to perform cross-channel information interaction in a one-dimensional convolutional layer to obtain the second 1×1×C feature vector.

[0024] Where C represents the channel dimension, which determines the size of the one-dimensional convolution kernel;

[0025] The mapping process of the attention module ECA is used with δ e This indicates that the channel feature B output by the attention module (ECA module) is... out The formula is as follows:

[0026]

[0027] in, This indicates element-wise operations, which involve multiplying the corresponding elements one by one.

[0028] By introducing a nonlocal attention module (NAM) to collect contextual information and enhance pixel-level representation capabilities, NAM considers all feature points and regions for weighted calculations, overcoming the limitation of convolutional neural networks that focus on local features, and obtaining global features M for the input image. out ;

[0029] The patch dataset's random features M are obtained by inputting the patch dataset into the backbone network of the non-local attention module NAM, specifically ResNet50. nin , through θ and Transform into θ(M) nin ) and φ(M nin ), θ and It is a 1x1 convolutional layer, and the feature map is passed through θ or Afterwards, the width and height remain unchanged, the channel size is halved, and θ(M) is... nin ) and φ(M nin Multiply the results and perform a softmax operation to obtain the weighted correlation matrix, thus obtaining the pixel correlation weight map M′. nin The formula is as follows:

[0030]

[0031] in, Represent θ and The similarity weight between these two positions;

[0032] M′ nin With the feature of the calculated position, g(M) nin Matrix multiplication is performed to fuse the correlations into the feature map, resulting in the attention map M″.nin The formula is as follows:

[0033]

[0034] Where g is a 1*1 convolutional layer;

[0035] The obtained attention map M″ nin After passing through a 1x1 convolution, and then combining it with the random features M of the patch dataset... nin By performing matrix addition, we obtain the final output global feature M of the nonlocal attention module. out The formula is as follows:

[0036]

[0037] in, This represents element-wise multiplication. This indicates an element addition operation.

[0038] More specifically, the extraction of channel-global features and global-channel features is as follows:

[0039] Channel feature B out The channel features are then placed into a nonlocal attention module (NAM) and nonlocal feature extraction is performed again to obtain channel-global features (MB) with the extracted channel features. out The formula is as follows:

[0040]

[0041] global feature M out The image is fed into the Channel Attention Module (ECA) to extract channel features, resulting in a patch image with a vector BM containing both spatial and channel features extracted later. out The formula is as follows:

[0042]

[0043] More preferably, the feature fusion is as follows:

[0044] The original dataset features BS1 obtained from the backbone network Xception and the patch dataset features BS2 obtained from the backbone network ResNet50 are randomly discarded by 50% using Dropout, resulting in the original dataset retained features BS1′ and the patch dataset retained features BS2′. The original dataset retained features BS1′, the patch dataset retained features BS2′, and the channel features B are then combined. out Global features M out Global-channel features BM out and channel-global feature MB outPerform concat fusion to obtain the fused feature RC out The formula is as follows:

[0045] RC out =concat((BS1′),(BS2′),(B out ),(M out ),(BM out ),(MB out )).

[0046] More preferably, categorical feature extraction is achieved by adding a Category Attention (CAB) module to obtain features from different categories by treating each DR category equally; specifically as follows:

[0047] RC fusion feature out The input feature Q is obtained by randomly dropping 20% ​​of the input data using Dropout. cin The Category Attention Module (CAB) processes the input features Q. cin Perform a 1×1 convolutional layer to obtain a new feature Q′, and perform a dropout operation on the new feature Q′ to randomly delete half of the features to obtain Q″;

[0048] pass The weights S = {S1, S2, ..., S...} are obtained to determine the importance of feature maps for different categories in class L. L}; where GMP represents the global max pool; q″ i,j This represents the j-th feature map of the i-th class in Q″;

[0049] pass Perform a class-based cross-channel average pooling operation on Q′ to obtain the average feature map for each class; where q′ i,j Q′ represents the j-th feature map of the i-th class in Q′; i_avg ∈R H×W×1 Represents the semantic feature map of the i-th class;

[0050] pass Q was ultimately enhanced. cin The feature Q used to distinguish classification regions cout .

[0051] More preferably, the feature classification is as follows:

[0052] RC fusion feature out Feature Q after categorical feature extraction cout Again, the fusion feature RC obtained earlier outThe final fused features are obtained by fusing the data. Flattening is used to transform these features into a one-dimensional form, which is then input into a fully connected layer. Classification is performed using a softmax activation function, yielding the classification result XRA. out The formula is as follows:

[0053] XRA out =softmax(Flatten(concat((RC) out ),(Q cout )))).

[0054] A diabetic retinopathy image classification system, the system comprising,

[0055] The preprocessing module is used to preprocess the raw dataset of diabetic retinopathy images and the patch dataset.

[0056] Feature extraction module one is used to extract features from the original dataset using Xception as the backbone network to obtain the original dataset features BS1, and to extract features from the patch dataset using ResNet50 as the backbone network to obtain the patch dataset features BS2.

[0057] Feature extraction module two is used to randomly discard 20% of the original dataset features BS1 extracted from the backbone network xception to obtain random features B of the original dataset. Ein , the random features B of the original dataset Ein Channel features B are obtained by inputting into the channel attention module ECA. out ; 20% of the features BS2 extracted from the feature patch dataset of the backbone network ResNet50 are randomly discarded to obtain random features M of the patch dataset. nin The random features M of the patch dataset nin The input is fed into a non-local attention module (NAM) to obtain global features M. out ;

[0058] Feature extraction module three is used to extract channel features B out The input is fed into the nonlocal attention module (NAM) to process channel features B. out Perform global location feature extraction to obtain channel-global feature MB. out ; global feature M out The input is fed into the channel attention module (ECA) to process the global feature M. out Perform channel feature extraction to obtain global-channel features BM out ;

[0059] The feature fusion module is used to combine the original dataset features BS1 and patch dataset features BS2 obtained through the backbone network Xception and the backbone network ResNet50 with the channel features B after randomly discarding 50% of each feature. out Global features M out Channel-Global Feature MB out and global-channel features BM out The fusion is performed to obtain the fusion feature RC out ;

[0060] The categorical feature extraction module is used to extract fused features (RC). out The category feature Q is obtained by feeding it into the category attention module CAB. cout ;

[0061] The feature classification module is used to classify the categorical features Q. cout With fusion features RC out The images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

[0062] An electronic device includes: a memory and at least one processor;

[0063] The memory contains computer programs;

[0064] The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the diabetic retinopathy image classification method as described above.

[0065] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the image classification method for diabetic retinopathy as described above.

[0066] The image classification method and system for diabetic retinopathy of the present invention have the following advantages:

[0067] (I) This invention uses two backbone deep networks, Xception and ResNet50, to extract features from retinal images. The different features obtained from the two different networks are combined through a cross-attention mechanism to obtain features that coexist with global and needle channels, while preserving the original feature details. These features are then fused together. Furthermore, while preserving the fused features, category features are extracted through category attention and fused with the fused features. This approach takes into account attention in all directions and preserves a wide range of different features more comprehensively. Classification is completed with a full range of features, which improves the accuracy and stability of retinal image classification. By adding a non-local attention module, the limitations of CNN models in extracting features of numerous small lesions that are unevenly distributed in fundus images are addressed.

[0068] (II) This invention provides a deep learning model for detecting and classifying diabetic retinopathy, capable of classifying retinal fundus images. First, a dual-backbone network is used to perform multiple deep feature extractions on the data, and then a cross-attention mechanism is used to fuse multiple features. Experimental results show that this invention achieves an accuracy of 92% and an AUC accuracy of 95% on the Messidor dataset, and a five-class classification accuracy of up to 87% on the DDR dataset. Compared with existing techniques, this invention achieves the expected results on currently available datasets and provides higher reliability for doctors in classifying diabetic retinopathy.

[0069] (iii) The category attention module CAB of the present invention allocates the same number of feature channels to each category, ensuring that each category is treated the same, which effectively alleviates the problem of category features being ignored due to the small number of samples, while increasing the distance between different categories. The category attention module CAB can play an important role in datasets with imbalanced category distribution.

[0070] (iv) When it comes to classification tasks that require identifying minute lesions, such as locating hemorrhage areas in retinal images, existing CNN-based models often fail to achieve excellent results. The main reason is that convolution operations can only extract features from local neighborhoods, while accurate classification requires the network to pay attention to numerous minute lesions that are unevenly distributed in the image. Pooling layers commonly used in CNNs downsample feature sizes and aggregate global information, which makes it difficult for the model to retain all the features of minute lesions in the retina. However, this invention models long-range dependencies through the channel attention module ECA and the non-local attention module NAM. By calculating the relative importance of all pixels, long-range dependencies can be easily captured, thereby providing a global understanding of the visual scene.

[0071] (v) This invention enriches the image data in the dataset through image enhancement technology, thereby improving the accuracy of subsequent classification;

[0072] (vi) This invention effectively prevents overfitting by randomly discarding features through Dropout. Attached Figure Description

[0073] The invention will be further described below with reference to the accompanying drawings.

[0074] Appendix Figure 1 A flowchart of an image classification method for diabetic retinopathy. Detailed Implementation

[0075] The image classification method and system for diabetic retinopathy of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0076] Example 1:

[0077] As attached Figure 1 As shown in the figure, this embodiment provides an image classification method for diabetic retinopathy, which is as follows:

[0078] S1. Dataset Acquisition and Image Preprocessing: Image preprocessing is performed on the original dataset of diabetic retinopathy images and on the patch dataset.

[0079] S2. Dataset feature extraction: Using Xception as the backbone network, feature extraction is performed on the original dataset to obtain the original dataset features BS1. Using ResNet50 as the backbone network, feature extraction is performed on the patch dataset to obtain the patch dataset features BS2.

[0080] S3. Channel Feature and Global Feature Extraction: The original dataset features BS1 extracted from the backbone network Xception are randomly discarded by 20% using Dropout to obtain random features B from the original dataset. Ein , the random features B of the original dataset Ein Channel features B are obtained by inputting into the channel attention module ECA. out The features BS2 extracted from the backbone network ResNet50 are used to randomly discard 20% of the patch dataset features M, resulting in random features M of the patch dataset. nin The random features M of the patch dataset nin The input is fed into a non-local attention module (NAM) to obtain global features M. out ;

[0081] S4. Channel-Global Feature Extraction and Global-Channel Feature Extraction: Extracting channel features B out The input is fed into the nonlocal attention module (NAM) to process channel features B. outPerform global location feature extraction to obtain channel-global feature MB. out ; global feature M out The input is fed into the channel attention module (ECA) to process the global feature M. out Perform channel feature extraction to obtain global-channel features BM out ;

[0082] S5. Feature Fusion: The original dataset features BS1 and the patch dataset features BS2, obtained through backbone network Xception and backbone network ResNet50 respectively, are randomly discarded by 50% each using Dropout, and then combined with the channel features B. out Global features M out Channel-Global Feature MB out and global-channel features BM out The fusion is performed to obtain the fusion feature RC out ;

[0083] S6. Category Feature Extraction: Extracting RC fusion features out The category feature Q is obtained by feeding it into the category attention module CAB. cout ;

[0084] S7. Feature Classification: Classifying category features Q cout With fusion features RC out The images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

[0085] The specific details of dataset acquisition and image preprocessing in step S1 of this embodiment are as follows:

[0086] S101. Use the DDR dataset and the Messidor dataset as the five-class and two-class datasets, respectively.

[0087] S102. Image preprocessing is as follows:

[0088] S10201. Since the grading of diabetic retinopathy is achieved through microaneurysms, hemorrhages, and exudates in the retina, in order to better extract retinal image features from conventional retinal images, we remove background noise from the dataset images and retain the retinal area to the maximum extent.

[0089] S10202. Enriching image data using data augmentation techniques, specifically:

[0090] ① The classification of diabetic retinopathy is largely based on microaneurysms in retinal images and circular localized capillary dilations in fundus images. The original dataset is segmented, and common lesion local areas are segmented as patches. A new patch dataset is created using the patch images. All black edge noise is ignored in the patch dataset, so that it focuses on more subtle lesion features.

[0091] ② In the training set, the images are scaled and enlarged: the scaling factor is set to 1 / 255. The purpose of scaling is to enlarge or reduce the length and width of the image, changing the overall size of the image, not to crop the image; during the image enhancement process, the size is randomly enlarged or reduced by 0.45 times to enhance the image, and finally the image is flipped horizontally again.

[0092] Here, rotation range represents the angle by which the image is randomly rotated during training (rotating around the center of the image); height displacement represents a 0.2% shift in the vertical direction of the image; width displacement is similar to height displacement, except that the angle changes from the vertical direction to the horizontal direction by 0.2%.

[0093] The specific extraction of channel features and global features in step S3 of this embodiment is as follows:

[0094] S301. By adding different attention modules (ECA), we can focus on different directions, extract more dimensional features and more targeted features. Through the local cross-channel interaction mechanism of the attention module (ECA) without dimensionality reduction, we can extract more channel information from the backbone network's xception features, and transform the random features B of the original dataset. Ein Perform channel-wise global average pooling;

[0095] After the S302 and GAP layers generate 1×1×C feature vectors, through... The adaptive convolutional kernel size is used to perform cross-channel information interaction in a one-dimensional convolutional layer to obtain the second 1×1×C feature vector.

[0096] Where C represents the channel dimension, which determines the size of the one-dimensional convolution kernel;

[0097] S303, The mapping process of the attention module ECA is implemented using δ e This indicates that the channel feature B output by the attention module (ECA module) is... out The formula is as follows:

[0098]

[0099] in, This indicates element-wise operations, which involve multiplying the corresponding elements one by one.

[0100] S304. By introducing a non-local attention module (NAM) to collect contextual information, pixel-level representation capabilities are enhanced. The NAM module considers all feature points and regions for weighted calculation, overcoming the shortcomings of convolutional neural networks that focus on local features, and obtaining global features M for the input image. out ;

[0101] S305, The random features M of the patch dataset obtained by inputting the patch dataset into the backbone network of the non-local attention module NAM, ResNet50. nin , through θ and Transform into θ(M) nin ) and φ(M nin ), θ and It is a 1x1 convolutional layer, and the feature map is passed through θ or Afterwards, the width and height remain unchanged, the channel size is halved, and θ(M) is... nin ) and φ(M nin Multiply the results and perform a softmax operation to obtain the weighted correlation matrix, thus obtaining the pixel correlation weight map M′. nin The formula is as follows:

[0102]

[0103] in, Represent θ and The similarity weight between these two positions;

[0104] S306, M′ nin With the feature of the calculated position, g(M) nin Matrix multiplication is performed to fuse the correlations into the feature map, resulting in the attention map M″. nin The formula is as follows:

[0105]

[0106] Where g is a 1*1 convolutional layer;

[0107] S307, Obtain the attention map M″ nin After passing through a 1x1 convolution, and then combining it with the random features M of the patch dataset... nin By performing matrix addition, we obtain the final output global feature M of the nonlocal attention module. out The formula is as follows:

[0108] in, This represents element-wise multiplication. This indicates an element addition operation.

[0109] The extraction of channel-global features and global-channel features in step S4 of this embodiment is as follows:

[0110] S401, Channel Feature B out The channel features are then placed into a nonlocal attention module (NAM) and nonlocal feature extraction is performed again to obtain channel-global features (MB) with the extracted channel features. out The formula is as follows:

[0111]

[0112] S402, Transfer global feature M out The image is fed into the Channel Attention Module (ECA) to extract channel features, resulting in a patch image with a vector BM containing both spatial and channel features extracted later. out The formula is as follows:

[0113]

[0114] The feature fusion in step S5 of this embodiment is as follows:

[0115] The original dataset features BS1 obtained from the backbone network Xception and the patch dataset features BS2 obtained from the backbone network ResNet50 are randomly discarded by 50% using Dropout, resulting in the original dataset retained features BS1′ and the patch dataset retained features BS2′. The original dataset retained features BS1′, the patch dataset retained features BS2′, and the channel features B are then combined. out Global features M out Global-channel features BM out and channel-global feature MB out Perform concat fusion to obtain the fused feature RC out The formula is as follows:

[0116] RC out =concat((BS1′),(BS2′),(B out ),(M out ),(BM out ),(MB out )).

[0117] In this embodiment, the category feature extraction in step S6 is achieved by adding a category attention module (CAB) to obtain features of different categories by treating each DR category equally; specifically as follows:

[0118] S601, RC fusion feature out The input feature Q is obtained by randomly dropping 20% ​​of the input data using Dropout. cinThe Category Attention Module (CAB) processes the input features Q. cin Perform a 1×1 convolutional layer to obtain a new feature Q′, and perform a dropout operation on the new feature Q′ to randomly delete half of the features to obtain Q″;

[0119] S602, Through The weights S = {S1, S2, ..., S...} are obtained to determine the importance of feature maps for different categories in class L. L}; where GMP represents the global max pool; q″ i,j This represents the j-th feature map of the i-th class in Q″;

[0120] S603, through Perform a class-based cross-channel average pooling operation on Q′ to obtain the average feature map for each class; where q′ i,j Q′ represents the j-th feature map of the i-th class in Q′; i_avg ∈R H×W×1 Represents the semantic feature map of the i-th class;

[0121] S604, Through Q was ultimately enhanced. cin The feature Q used to distinguish classification regions cout .

[0122] The feature classification in step S7 of this embodiment is as follows:

[0123] RC fusion feature out Feature Q after categorical feature extraction cout Again, the fusion feature RC obtained earlier out The final fused features are obtained by fusing the data. Flattening is used to transform these features into a one-dimensional form, which is then input into a fully connected layer. Classification is performed using a softmax activation function, yielding the classification result XRA. out The formula is as follows:

[0124] XRA out =softmax(Flatten(concat((RC) out ),(Q cout )))).

[0125] The classification categories of images for diabetic retinopathy include asymptomatic, mild, moderate, severe, and proliferative.

[0126] Example 2:

[0127] This embodiment provides an image classification system for diabetic retinopathy, the system comprising,

[0128] The preprocessing module is used to preprocess the raw dataset of diabetic retinopathy images and the patch dataset.

[0129] Feature extraction module one is used to extract features from the original dataset using Xception as the backbone network to obtain the original dataset features BS1, and to extract features from the patch dataset using ResNet50 as the backbone network to obtain the patch dataset features BS2.

[0130] Feature extraction module two is used to randomly discard 20% of the original dataset features BS1 extracted from the backbone network xception to obtain random features B of the original dataset. Ein , the random features B of the original dataset Ein Channel features B are obtained by inputting into the channel attention module ECA. out ; 20% of the features BS2 extracted from the feature patch dataset of the backbone network ResNet50 are randomly discarded to obtain random features M of the patch dataset. nin The random features M of the patch dataset nin The input is fed into a non-local attention module (NAM) to obtain global features M. out ;

[0131] Feature extraction module three is used to extract channel features B out The input is fed into the nonlocal attention module (NAM) to process channel features B. out Perform global location feature extraction to obtain channel-global feature MB. out ; global feature M out The input is fed into the channel attention module (ECA) to process the global feature M. out Perform channel feature extraction to obtain global-channel features BM out ;

[0132] The feature fusion module is used to combine the original dataset features BS1 and patch dataset features BS2 obtained through the backbone network Xception and the backbone network ResNet50 with the channel features B after randomly discarding 50% of each feature. out Global features M out Channel-Global Feature MB out and global-channel features BM out The fusion is performed to obtain the fusion feature RC out ;

[0133] The categorical feature extraction module is used to extract fused features (RC). out The category feature Q is obtained by feeding it into the category attention module CAB. cout ;

[0134] The feature classification module is used to classify the categorical features Q. cout With fusion features RC out The images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

[0135] Example 3:

[0136] This embodiment also provides an electronic device, including: a memory and a processor;

[0137] The memory stores the instructions executed by the computer.

[0138] The processor executes computer execution instructions stored in the memory, causing the processor to perform the diabetic retinopathy image classification method in any embodiment of the present invention.

[0139] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0140] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0141] Example 4:

[0142] This invention also provides a computer-readable storage medium storing a plurality of instructions, which are loaded by a processor to cause the processor to execute the diabetic retinopathy image classification method according to any embodiment of this invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0143] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0144] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0145] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0146] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0147] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for classifying images of diabetic retinopathy, characterized in that, The method is as follows: Dataset acquisition and image preprocessing: Image preprocessing was performed on the original dataset of diabetic retinopathy images and on the patch dataset; Dataset feature extraction: Using Xception as the backbone network, feature extraction is performed on the original dataset to obtain the original dataset features BS1. Using ResNet50 as the backbone network, feature extraction is performed on the patch dataset to obtain the patch dataset features BS2. Channel feature and global feature extraction: 20% of the original dataset features BS1 extracted from the backbone network Xception are randomly discarded to obtain random features B of the original dataset. Ein , the random features B of the original dataset Ein Channel features B are obtained by inputting into the channel attention module ECA. out ; 20% of the features BS2 extracted from the feature patch dataset of the backbone network ResNet50 are randomly discarded to obtain random features M of the patch dataset. nin The random features M of the patch dataset nin The input is fed into the nonlocal attention module (NAM) to obtain global features M. out ; Channel-global feature extraction and global-channel feature extraction: extracting channel features B out The input is fed into the nonlocal attention module (NAM) to process channel features B. out Perform global location feature extraction to obtain channel-global feature MB. out ; global feature M out The input is fed into the channel attention module (ECA) to process the global feature M. out Perform channel feature extraction to obtain global-channel features BM out ; feature Fusion: The original dataset features BS1 and the patch dataset features BS2, obtained through backbone network 1 (Xception) and backbone network 2 (ResNet50), are randomly discarded by 50% each, and then combined with the channel features B. out Global features M out Channel-Global Feature MB out and global-channel features BM out The fusion is performed to obtain the fusion feature RC out ; Category feature extraction: fusing features RC out The category feature Q is obtained by feeding it into the category attention module CAB. cout ; feature Classification: Classification features Q cout With fusion features RC out The images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

2. The image classification method for diabetic retinopathy according to claim 1, characterized in that, The data acquisition and image preprocessing are detailed below: The DDR dataset and the Messidor dataset were used as the datasets for five-class classification and two-class classification, respectively. The image preprocessing is as follows: Remove background noise from the dataset images while preserving the maximum retinal area; Enriching image data using data augmentation techniques, specifically: The original dataset is segmented, and common lesion local areas are segmented as patches. A new patch dataset is created using the patch images. The patch dataset ignores all black edge noise, so that it focuses on more subtle lesion features. In the training set, images are scaled and enlarged: the scaling factor is set to 1 / 255. The purpose of scaling is to enlarge or reduce the length and width of the image, changing the overall size of the image. During the image enhancement process, the size is randomly enlarged or reduced by 0.45 times to enhance the image. Finally, the image is flipped horizontally again. Here, rotation range represents the angle by which the image is randomly rotated during training; height displacement represents a 0.2% shift in the vertical direction; width displacement is similar to height displacement, except that the angle changes from the vertical direction to the horizontal direction by 0.2%.

3. The image classification method for diabetic retinopathy according to claim 1 or 2, characterized in that, The specific extraction of channel features and global features is as follows: By leveraging the local cross-channel interaction mechanism of the attention module ECA without dimensionality reduction, channel information from the backbone network's xception features is extracted, and the random features B of the original dataset are then processed. Ein Perform channel-wise global average pooling; After the GAP layer generates 1×1×C feature vectors, it is then... The adaptive convolutional kernel size is used to perform cross-channel information interaction in a one-dimensional convolutional layer to obtain the second 1×1×C feature vector. Where C represents the channel dimension, which determines the size of the one-dimensional convolution kernel; The mapping process of the attention module ECA is used with δ e This indicates that the channel feature B output by the attention module (ECA module) is... out The formula is as follows: in, This indicates element-wise operations, which involve multiplying the corresponding elements one by one. By introducing a nonlocal attention module (NAM) to collect contextual information and enhance pixel-level representation capabilities, the NAM module considers all feature points and regions for weighted calculation, resulting in global features M for the input image. out ; The patch dataset's random features M are obtained by inputting the patch dataset into the backbone network of the non-local attention module NAM, specifically ResNet50. nin , through θ and Transform into θ(M) nin ) and φ(M nin ), θ and It is a 1x1 convolutional layer, and the feature map is passed through θ or Afterwards, the width and height remain unchanged, the channel size is halved, and θ(M) is... nin )and Multiply and perform a softmax operation to obtain the weighted correlation matrix, thus obtaining the pixel correlation weight map M. n ′ in The formula is as follows: in, Represent θ and The similarity weight between these two positions; M′ nin With the feature of the calculated position, g(M) nin Matrix multiplication is performed to fuse the correlations into the feature map, resulting in the attention map M″. nin The formula is as follows: Where g is a 1*1 convolutional layer; The obtained attention map M″ nin After passing through a 1x1 convolution, and then combining it with the random features M of the patch dataset... nin By performing matrix addition, we obtain the final output global feature M of the nonlocal attention module. out The formula is as follows: in, This represents element-wise multiplication. This indicates an element addition operation.

4. The image classification method for diabetic retinopathy according to claim 3, characterized in that, The extraction of channel-global features and global-channel features are as follows: Channel feature B out The channel features are then placed into a nonlocal attention module (NAM) and nonlocal feature extraction is performed again to obtain channel-global features (MB) with the extracted channel features. out The formula is as follows: global feature M out The image is fed into the Channel Attention Module (ECA) to extract channel features, resulting in a patch image with a vector BM containing both spatial and channel features extracted later. out The formula is as follows:

5. The image classification method for diabetic retinopathy according to claim 4, characterized in that, Feature fusion is detailed below: The original dataset features BS1 obtained from the backbone network Xception and the patch dataset features BS2 obtained from the backbone network ResNet50 are randomly discarded by 50% using Dropout, resulting in the original dataset retained features BS1′ and the patch dataset retained features BS2′. The original dataset retained features BS1′, the patch dataset retained features BS2′, and the channel features B are then combined. out Global features M out Global-channel features BM out and channel-global feature MB out Perform concat fusion to obtain the fused feature RC out The formula is as follows: RC out =concat((BS1′),(BS2′),(B out ),(M out ),(BM out ),(MB out ))。 6. The image classification method for diabetic retinopathy according to claim 5, characterized in that, Category feature extraction is achieved by adding a Category Attention (CAB) module, which treats each DR category equally to obtain features from different categories; specifically as follows: RC fusion feature out The input feature Q is obtained by randomly dropping 20% ​​of the input data using Dropout. cin The Category Attention Module (CAB) processes the input features Q. cin Perform a 1×1 convolutional layer to obtain a new feature Q′, and perform a dropout operation on the new feature Q′ to randomly delete half of the features to obtain Q″; pass The weights S = {S1, S2, ..., S...} are obtained to determine the importance of feature maps for different categories in class L. L }; where GMP represents the global max pool; q″ i,j This represents the j-th feature map of the i-th class in Q″; pass Perform a class-based cross-channel average pooling operation on Q′ to obtain the average feature map for each class; where q′ i,j Q′ represents the j-th feature map of the i-th class in Q′; i_avg ∈R H×W×1 Represents the semantic feature map of the i-th class; pass Q was ultimately enhanced. cin The feature Q used to distinguish classification regions cout .

7. The image classification method for diabetic retinopathy according to claim 6, characterized in that, The feature classification is as follows: RC fusion feature out Feature Q after categorical feature extraction cout Again, the fusion feature RC obtained earlier out The final fused features are obtained by fusing the data. Flattening is used to transform these features into a one-dimensional form, which is then input into a fully connected layer. Classification is performed using a softmax activation function, yielding the classification result XRA. out The formula is as follows: XRA out =softmax(Flatten(concat((RC out ),(Q cout ))))。 8. An image classification system for diabetic retinopathy, characterized in that, The system includes, The preprocessing module is used to preprocess the raw dataset of diabetic retinopathy images and the patch dataset. Feature extraction module one is used to extract features from the original dataset using Xception as the backbone network to obtain the original dataset features BS1, and to extract features from the patch dataset using ResNet50 as the backbone network to obtain the patch dataset features BS2. Feature extraction module two is used to randomly discard 20% of the original dataset features BS1 extracted from the backbone network xception to obtain random features B of the original dataset. Ein , the random features B of the original dataset Ein Channel features B are obtained by inputting into the channel attention module ECA. out ; 20% of the features BS2 extracted from the feature patch dataset of the backbone network ResNet50 are randomly discarded to obtain random features M of the patch dataset. nin The random features M of the patch dataset nin The input is fed into the nonlocal attention module (NAM) to obtain global features M. out ; Feature extraction module three is used to extract channel features B out The input is fed into the nonlocal attention module (NAM) to process channel features B. out Perform global location feature extraction to obtain channel-global feature MB. out ; global feature M out The input is fed into the channel attention module (ECA) to process the global feature M. out Perform channel feature extraction to obtain global-channel features BM out ; The feature fusion module is used to combine the original dataset features BS1 and patch dataset features BS2 obtained through the backbone network Xception and the backbone network ResNet50 with the channel features B after randomly discarding 50% of each feature. out Global features M out Channel-Global Feature MB out and global-channel features BM out The fusion is performed to obtain the fusion feature RC out ; The categorical feature extraction module is used to extract fused features (RC). out The category feature Q is obtained by feeding it into the category attention module CAB. cout ; The feature classification module is used to classify the categorical features Q. cout With fusion features RC out The images are then fused again to obtain the final fused features. These features are then one-dimensionalized and input into a fully connected layer. The softmax activation function is used to classify the images of diabetic retinopathy.

9. An electronic device, characterized in that, include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the diabetic retinopathy image classification method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the image classification method for diabetic retinopathy as described in any one of claims 1 to 7.

Images