Diabetic retinopathy grading model system and method using frequency domain attention mechanism

Abstract

The invention belongs to the technical field of medical image analysis, and discloses a diabetic retinopathy grading model system using a frequency domain attention mechanism, which comprises a data set construction module, a data preprocessing module, an evaluation index module, a feature extraction module, a frequency domain channel attention module and a frequency domain space attention module. The invention provides a DR grading model based on deep learning, and aims to more accurately identify and classify lesions in fundus images. By introducing a frequency domain channel attention module (FCAM) and a frequency domain space attention module (FSAM), the model can more effectively capture weak lesion features. FCAM improves the sensitivity of a model to a lesion area by fusing frequency domain and channel domain information; according to the FSAM, key pixels are emphasized in a spatial domain and a frequency domain, so that the extraction capability of the model on fine texture features is enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image analysis, and in particular relates to a diabetic retinopathy grading model system and method using a frequency domain attention mechanism. Background Art

[0002] Diabetic retinopathy (DR) is one of the most serious microvascular complications of diabetes and the leading cause of irreversible vision loss worldwide. According to the severity of the lesions, DR can be divided into non-proliferative diabetic retinopathy (NPDR) and proliferative diabetic retinopathy (PDR). Non-proliferative diabetic retinopathy (NPDR) can be further divided into mild non-proliferative (MildNPDR), moderate non-proliferative (Moderate NPDR), and severe non-proliferative (Severe NPDR), mainly manifested as microaneurysms, hard exudates, cotton-wool spots, etc. Proliferative diabetic retinopathy (PDR) is characterized by neovascularization, which can lead to serious complications such as vitreous hemorrhage and traction retinal detachment, which seriously threatens vision. DR grading is of great significance for clinical diagnosis and treatment. Doctors evaluate the type, number, and distribution of DR lesions to determine the severity of the disease and formulate corresponding treatment plans. With the rapid development of artificial intelligence technology, deep learning and other technologies have been widely used in DR image analysis, providing new means for early diagnosis and precise treatment of DR.

[0003] The severity of diabetic retinopathy (DR) is mainly determined by analyzing color fundus images (CFP). Although these images can clearly show the fine structure of the retina, the characteristics of DR lesions such as microaneurysms and hemorrhages are often subtle and diverse, which poses a challenge to automatic classification.

[0004] In recent years, convolutional neural networks (CNNs) have shown excellent performance in DR grading tasks. For example, the Inceptionv3 model proposed by Gulshan et al. has achieved remarkable results in DR detection. In order to further improve the model's feature extraction capabilities, researchers have introduced an attention mechanism that enables the model to focus on key areas in the image, such as the Category Attention Block proposed by He et al., which effectively solves the problem of minor lesion features being ignored. In addition, Zhou et al. combined lesion segmentation with DR grading for collaborative learning, and through pixel-level annotation, they were able to provide more detailed lesion information and improve grading accuracy.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) Insufficient high-quality labeled data: The limited amount of high-quality labeled fundus image data limits the generalization ability of the model;

[0007] (2) Difficulty in feature extraction: The features of DR lesions are diverse and subtle, and it is difficult for the model to extract sufficiently discriminative features;

[0008] (3) Insufficient model robustness: In complex and changing clinical environments, the robustness of the model still needs to be further improved. Summary of the invention

[0009] In view of the problems existing in the prior art, the present invention provides a diabetic retinopathy grading model system using a frequency domain attention mechanism.

[0010] The present invention is implemented as follows: a diabetic retinopathy grading model system using a frequency domain attention mechanism, comprising:

[0011] Dataset construction module, data preprocessing module, evaluation index module, feature extraction module, frequency domain channel attention module, frequency domain spatial attention module.

[0012] The data set construction module is connected with the data preprocessing module to construct the training data set and the test data set;

[0013] The data preprocessing module is connected with the data set construction module, the evaluation index module, and the feature extraction module to perform various image enhancement processes on the training data set, including horizontal flipping, vertical flipping, rotation, cropping, and random adjustment of the image brightness and contrast;

[0014] The evaluation index module is connected to the data preprocessing module and is used to select accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score, and joint accuracy as evaluation indicators to analyze and compare the performance of different classification models;

[0015] The feature extraction module is connected with the data preprocessing module, the frequency domain channel attention module, and the frequency domain spatial attention module, and uses ResNet50 as the backbone network to extract the deep features of the CFP image;

[0016] The frequency domain channel attention module is connected to the feature extraction module and is used to extract the frequency domain features of the image through the improved multi-domain attention layer and fuse them with the spatial domain features extracted by the channel attention module to give higher weights to the key channels, thereby improving the model's sensitivity to the lesion area;

[0017] The frequency domain spatial attention module is connected to the feature extraction module to extract the attention maps of the spatial domain and frequency domain of the image, and enhance the subtle texture features in the image and emphasize the key pixels by multiplying them pixel by pixel.

[0018] Furthermore, the dataset:

[0019] The EAM dataset was constructed by integrating the EyePacs, Messidor, and Aptos datasets; the EAM dataset contains a total of 92,501 images;

[0020] EyePacs: As the largest DR grading dataset, EyePacs contains 35,126 images with large differences in image quality, which can effectively test the robustness of the model to different image qualities; Messidor: This dataset contains 1,200 high-quality color fundus images, which can be used to verify the performance of the model on a small-scale dataset; Aptos: This dataset provides 3,662 high-quality fundus images, which can be used as supplementary training data for the model.

[0021] IDRiD: This dataset provides 516 high-quality fundus images and two labels, DR and DME, which can be used to evaluate the multi-task learning ability of the model; DDR: This dataset contains 13,673 images covering multiple DR levels and can be used to evaluate the performance of the model for fine-grained classification.

[0022] Furthermore, the evaluation index module:

[0023] The accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score, and joint accuracy were selected as evaluation indicators to analyze and compare the performance of different classification models;

[0024] Accuracy is used to measure the accuracy of model predictions, that is, the proportion of samples predicted correctly by the model to the total samples.

[0025]

[0026] Among them, TP, TN, FP, and FN are the number of samples of true positive examples, true negative examples, false positive examples, and false negative examples, respectively.

[0027] The consistency test coefficient (Kappa) further evaluates the consistency between the model prediction results and the true label. The Kappa value range is 1 to 1. It is generally believed that when the Kappa value is above 0.8, the model has good consistency.

[0028]

[0029] Among them, W is the weight matrix, W i.jrepresents the penalty between the predicted category i and the true category j. O is the observation matrix, that is, the confusion matrix, O i.j Represents the number of samples of predicted category i and true category j. E is the expectation matrix, E i.j It represents the probability that a class i is predicted to be class j in the case of random prediction.

[0030] Indicators such as AUC, Recall, and F1 are mainly used in binary classification tasks. AUC represents the area under the ROC curve, ranging from 0 to 1. 0.5 means that the model performance is close to random, and 1 means that the model is the best. The formulas for Recall and F1 are as follows:

[0031]

[0032] Joint Accuracy is used for the joint classification task on the IDRiD dataset and is defined as follows: if the prediction result of the sample model matches the true label of both its DR and DME, it is counted as a valid sample. The joint accuracy is obtained by dividing the number of all valid samples by the total number of samples.

[0033] Furthermore, the frequency domain channel attention module:

[0034] (1) Discrete Cosine Transform

[0035] Discrete cosine transform is an orthogonal transform that converts time domain or spatial domain signals into frequency domain representation; DCT has a good energy concentration characteristic, that is, most of the signal energy is concentrated in the low-frequency part; in the field of image compression, DCT is widely used in the JPEG image compression standard; using this characteristic of DCT, the image is converted from the spatial domain to the frequency domain to extract the frequency domain features of the image;

[0036] For a two-dimensional image x 2d ∈R H×W , which corresponds to the two-dimensional DCT transformed image f 2d ∈R H×W The formula is as follows:

[0037]

[0038] sth∈{0,1,…,H-1},w∈{0,1,…,W-1}.

[0039] Among them, H and W represent x respectively. 2d The height and width of B h,w ∈R H×W The basis function representing the DCT transform is the key to extracting the frequency information of the image; Respectively represent x 2d , B h,w The pixel at (i,j) in represents f 2d The pixel at (h,w) in The formula can be expressed as:

[0040]

[0041] When h = 0, w = 0 in formula (8), the DCT transform is equivalent to global average pooling (GAP), as shown in formula (9):

[0042]

[0043] That is, using the basis function to perform DCT transformation on the image can be regarded as a pooling operation;

[0044] (2) Multi-spectral Attention Layer

[0045] Given an input feature map F∈R C×H×W , through drMSA, we can get the DCT frequency domain channel attention map F dct_att ∈R C ×H×W :

[0046]

[0047] F is the input feature map, F MSA ∈R C×1×1 Represents the attention weight vector obtained by DCT basis function transformation, W 0 With W 1 Represents the weight of MLP, and the final output of drMSA is F dct_att ;

[0048] For the CAM part, given the input feature map F∈R H×W×C , the channel attention map F att ∈R C×H×W :

[0049]

[0050] In the above formula, F gap ∈R C×1×1 With F gmp ∈R C×1×1 is the weight vector obtained by GAP and GMP, which is multiplied by F to get the final result F att ;

[0051] Finally, drMSA output F dct_att With CAM output F att Add element by element to allow the model to capture feature information from both the spatial domain and the frequency domain, increase model complexity and data diversity, and improve generalization capabilities;

[0052] Get the output F of FCAM DFCAM ∈R C×H×W ;

[0053]

[0054] Furthermore, the frequency domain spatial attention module:

[0055] Given an input feature map F∈R C×H×W , and its corresponding frequency domain image is F dct ∈R C×H×W First, the two feature maps are stacked and concatenated through GAP and GMP operations to obtain two intermediate feature maps F concat ∈R 2×H×W ;

[0056] After the concatenation operation, the intermediate feature maps in the spatial domain and the frequency domain are fused by element-by-element multiplication to obtain F mix ∈R 2×H×W ,The fusion feature information can enrich and highlight the most ,noticeable details in the image;

[0057] Finally, F mix Send it to the convolution layer and activation function to get the attention map F att ∈R 1×H×W , output F FSAM ∈R C×H×W It can be expressed as:

[0058] F concat =[GAP(F),GMP(F)]

[0059] F dct_concat =[GAP(F dct ),GMP(F dct )]

[0060]

[0061] Among them, GAP and GMP are global average pooling and global maximum pooling, W is the convolution filter weight, and the softmax operation obtains the spatial attention map, which is finally multiplied with the original image to obtain the result F FSAM .

[0062] Another object of the present invention is to provide a method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism, comprising:

[0063] Step 1: Build training data set and test data set through data set building module;

[0064] Step 2: Perform various image enhancement processes on the training data set through the data preprocessing module, including horizontal flipping, vertical flipping, rotation, cropping, and random adjustment of image brightness and contrast;

[0065] Step 3, the accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score, and joint accuracy are selected as evaluation indicators through the evaluation indicator module to compare and analyze the performance of different classification models;

[0066] Step 4, extract the deep features of the CFP image through the feature extraction module using ResNet50 as the backbone network;

[0067] Step 5: The frequency domain features of the image are extracted by using the improved multi-domain attention layer through the frequency domain channel attention module, and are integrated with the spatial domain features extracted by the channel attention module to give higher weights to the key channels, thereby improving the model's sensitivity to the lesion area.

[0068] Step 6: The frequency-domain spatial attention module is used to extract the attention maps of the image in the spatial and frequency domains, and the subtle texture features in the image are enhanced by multiplying them pixel by pixel to emphasize the key pixels.

[0069] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism.

[0070] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism.

[0071] Another object of the present invention is to provide an information data processing terminal, which is used to implement the diabetic retinopathy grading model system using the frequency domain attention mechanism.

[0072] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0073] First, the present invention proposes a diabetic retinopathy grading model system based on the frequency domain attention mechanism. The model effectively enhances the model's ability to extract and grade DR lesion features by introducing the frequency domain channel attention module (FCAM) ​​and the frequency domain spatial attention module (FSAM). The present invention proposes a DR grading model based on deep learning, which aims to more accurately identify and classify lesions in fundus images. By introducing the frequency domain channel attention module (FCAM) ​​and the frequency domain spatial attention module (FSAM), the model can more effectively capture weak lesion features. FCAM improves the model's sensitivity to lesion areas by fusing frequency domain and channel domain information; FSAM enhances the model's ability to extract subtle texture features by emphasizing key pixels in the spatial domain and frequency domain. Experimental results on multiple public datasets show that the model has achieved better performance than existing methods in DR grading tasks, has high accuracy and generalization ability, and provides strong support for clinical auxiliary diagnosis.

[0074] This paper proposes a convolutional neural network based on the frequency domain attention mechanism for diabetic retinopathy (DR) grading. By introducing the frequency domain channel attention module (FCAM) ​​and the frequency domain spatial attention module (FSAM), the model can more effectively extract the subtle features of DR lesions and improve the classification accuracy. Experimental results show that the model has achieved better performance than existing methods on multiple public datasets. The innovations are: 1) A new frequency domain attention mechanism is proposed to enhance the model's ability to express DR lesion characteristics; 2) Through the data enhancement strategy, the generalization ability of the model is improved.

[0075] Second, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0076] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0077] In terms of expected benefits, the "Study on the Prevalence and Treatment of Diabetes in China from 2013 to 2018" shows that the prevalence of diabetes in adults in my country increased from 10.9% in 2013 to 12.4% in 2018, but the diabetes treatment rate in 2018 was only 32.9%, and the treatment control rate was 50.1%, which is still at a low level overall. During 2013-2018, the prevalence of prediabetes in adults in my country increased from 35.7% to 38.1%, and the prevention and control situation is not optimistic.

[0078] This data shows that although the prevalence of diabetes continues to rise, the treatment and control effects are still not ideal, especially in the prevention of diabetic complications. This provides broad prospects and important value for the application of the present invention, namely the DR grading deep learning model. The current low diabetes treatment rate means that most patients cannot get timely diagnosis and treatment. The introduction of an automated diabetic retinopathy grading model can greatly improve the screening efficiency, which can help the medical system to automatically screen patients faster and on a larger scale, improve the early detection rate of diabetic retinopathy, especially in grassroots and remote areas with limited medical resources, and effectively alleviate the problem of insufficient doctor resources.

[0079] At the commercial level, the present invention also shows great potential value. With the increase in the prevalence of prediabetes, the demand for early screening tools is also growing. The deep learning model can be combined with existing medical equipment to become part of the medical diagnostic tool, providing hospitals and clinics with efficient diabetic retinopathy screening solutions. Especially in the field of telemedicine, the model can be promoted through cloud service platforms, enabling primary hospitals to conduct preliminary screening and greatly broadening the coverage of high-quality medical resources.

[0080] (2) The technical solution of the present invention fills the technical gap in the industry at home and abroad:

[0081] The present invention proposes a new frequency domain attention mechanism, including a frequency domain channel attention module (FCAM) ​​and a frequency domain space attention module (FSAM). The frequency domain information of the image is obtained through DCT transformation, and the mixed features of the frequency domain and spatial domain of the image are used to perform image classification tasks. This is the first model application at home and abroad to apply the frequency domain attention mechanism to the field of DR grading and synchronous lesion segmentation, which effectively improves the accuracy and generalization of the DR grading model, and fills the technical gap in the frequency domain attention mechanism for DR grading and synchronous lesion segmentation tasks at home and abroad.

[0082] (3) The technical solution of the present invention solves the technical problems that people have been eager to solve but have never been able to solve successfully:

[0083] The present invention proposes a computer-assisted deep learning DR grading model for DR grading diagnosis, which is used to process automated DR diagnosis tasks. The accuracy of the automated grading model and its stability in complex environments are improved by introducing a frequency domain attention mechanism.

[0084] (4) The technical solution of the present invention overcomes technical prejudice:

[0085] This paper introduces the frequency domain attention mechanism into the DR grading model, demonstrates the good performance of the frequency domain attention mechanism in the DR grading task, and proves the effectiveness of the frequency domain attention mechanism in the DR grading task.

[0086] Third, the present invention solves the problem of insufficient accuracy and sensitivity to the lesion area in the prior art diabetic retinopathy grading model. In industrial applications, traditional grading models usually rely on spatial domain features and are difficult to effectively identify subtle lesion information, resulting in low classification accuracy and failure to meet actual diagnostic needs. The present invention introduces a frequency domain attention mechanism and utilizes the extraction and optimization of frequency domain features to significantly improve the model's sensitivity to the lesion area, effectively solving the defects of traditional models in identifying lesions.

[0087] In terms of data processing, the present invention constructs a more comprehensive EAM dataset by integrating multiple public datasets and competition datasets, and adopts multiple data enhancement methods to improve the robustness of the model and the performance under different image quality conditions. In industrial applications, this comprehensive dataset and enhancement method greatly enhances the generalization ability of the model, enabling it to adapt to different image qualities and meet a wider range of application scenarios.

[0088] The present invention further optimizes the evaluation criteria of the model, using multiple evaluation indicators such as accuracy, consistency test coefficient, and area under the ROC curve to comprehensively measure the performance of the model. At the same time, by introducing the joint accuracy indicator, the performance of the model in a multi-task environment can be better evaluated. These improvements make the actual effect of the model more stable in an industrial environment and can meet the requirements of high-precision classification tasks.

[0089] By innovatively introducing the frequency domain channel attention and frequency domain spatial attention modules, the present invention has achieved significant technical progress in industrial applications, enabling the model to not only capture spatial domain features, but also deeply extract frequency domain features, effectively improving the recognition accuracy of the lesion area. This improvement has brought higher diagnostic accuracy and more reliable grading results in industrial applications, providing effective technical support for the automated detection and auxiliary diagnosis of diabetic retinopathy. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] Figure 1 It is a structural block diagram of a diabetic retinopathy grading model system using a frequency domain attention mechanism provided in an embodiment of the present invention.

[0091] Figure 2 It is a flow chart of a method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism provided in an embodiment of the present invention.

[0092] Figure 3 It is a schematic diagram of DR classification provided by an embodiment of the present invention.

[0093] Figure 4 It is a system architecture diagram of a diabetic retinopathy grading model provided in an embodiment of the present invention.

[0094] Figure 5 It is a basis function selection scheme for the multi-spectral attention layer (MSA) provided in an embodiment of the present invention; (a) visualization of 64 basis functions generated by the discrete cosine transform (DCT) of an 8×8 image; (b) classification accuracy graph of different DCT basis functions when performing a diabetic retinopathy grading task on the Messidor dataset.

[0095] Figure 6 It is a ROC curve diagram of the model provided by the embodiment of the present invention on the Messidor validation set.

[0096] Figure 7 It is the classification confusion matrix of DR and DME on the IDRiD dataset provided by an embodiment of the present invention.

[0097] Figure 1 In: 1. Dataset construction module; 2. Data preprocessing module; 3. Evaluation index module; 4. Feature extraction module; 5. Frequency domain channel attention module; 6. Frequency domain spatial attention module. DETAILED DESCRIPTION

[0098] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0099] like Figure 1 As shown, an embodiment of the present invention provides a diabetic retinopathy grading model system using a frequency domain attention mechanism, comprising:

[0100] Dataset construction module 1, data preprocessing module 2, evaluation index module 3, feature extraction module 4, frequency domain channel attention module 5, frequency domain spatial attention module 6;

[0101] The data set construction module 1 is connected with the data preprocessing module 2 and is used to construct a training data set and a test data set;

[0102] The data preprocessing module 2 is connected with the data set construction module 1, the evaluation index module 3 and the feature extraction module 4, and is used to perform various image enhancement processes on the training data set, including: horizontal flipping, vertical flipping, rotation, cropping and random adjustment of the brightness and contrast of the image;

[0103] Evaluation index module 3, connected to data preprocessing module 2, is used to select accuracy, consistency test coefficient, area under ROC curve, recall rate, F1 score, and joint accuracy as evaluation indicators to analyze and compare the performance of different classification models;

[0104] The feature extraction module 4 is connected with the data preprocessing module 2, the frequency domain channel attention module 5, and the frequency domain spatial attention module 6, and is used to extract the deep features of the CFP image using ResNet50 as the backbone network;

[0105] The frequency domain channel attention module 5 is connected to the feature extraction module 4 and is used to extract the frequency domain features of the image through the improved multi-domain attention layer, and fuse them with the spatial domain features extracted by the channel attention module to give higher weights to the key channels, thereby improving the sensitivity of the model to the lesion area;

[0106] The frequency domain spatial attention module 6 is connected to the feature extraction module 4 and is used to extract the attention map of the spatial domain and frequency domain of the image, and enhance the subtle texture features in the image and emphasize the key pixels by multiplying the pixels one by one.

[0107] The diabetic retinopathy grading model system using the frequency domain attention mechanism of the present invention realizes accurate grading of diabetic retinopathy by combining data set construction, preprocessing, feature extraction and attention mechanism. The specific working principle is as follows:

[0108] First, in the dataset construction module (module 1), the system constructs training and test datasets by collecting a large amount of fundus color image data. This module is responsible for dividing the original data to ensure that the model can obtain enough samples for training and testing, thereby improving the generalization performance of the classification model. When dividing the data, the module will consider the diversity and balance of the data to ensure that the images of each category are reasonably distributed in the training and test sets.

[0109] In the data preprocessing module (module 2), the system performs a variety of image enhancement operations on the training data set, including horizontal flipping, vertical flipping, rotation, cropping, and random adjustment of image brightness and contrast. Through these enhancement techniques, the system can effectively expand the data set and generate diverse samples. This preprocessing method helps to improve the robustness of the model, reduce the risk of overfitting, and improve the model's ability to recognize different lesion features, so that the model can adapt to different lighting conditions, image resolution, and other influences.

[0110] The feature extraction module (module 4) is the core part of the model. It uses ResNet50 as the backbone network to extract deep features of fundus images. Through this feature extraction process, the model can capture the complex structure and texture information in diabetic retinopathy. The multi-layer residual structure of the ResNet50 network effectively solves the gradient vanishing problem in deep neural network training, so that the extracted features contain rich information about the lesion area, providing input for the subsequent attention mechanism module.

[0111] The frequency domain channel attention module (module 5) introduces an improved multi-domain attention layer to extract the frequency domain features of the image and fuse them with the spatial domain features of the channel attention module. This module first converts the image into the frequency domain, extracts the key frequency features from it, and then assigns higher weights to the channels containing lesion information through the attention mechanism. This channel weighting strategy improves the model's sensitivity to retinal lesion areas, allowing the model to focus more on key image areas containing lesions, which helps improve the accuracy of grading.

[0112] The frequency domain spatial attention module (module 6) further fuses the spatial domain and frequency domain features of the image, and generates a frequency domain spatial attention map by multiplying each pixel. This method enhances the subtle texture features of the lesion area, allowing the model to identify tiny lesions that may be overlooked in the image. In addition, this module gives higher weights to key pixels in the spatial distribution of the image, highlighting the edge features and details of the lesion area, making the model more accurate in judging the degree of lesions.

[0113] Finally, the evaluation index module (module 3) plays a key role in the training and testing phase of the entire model. This module evaluates and monitors the model performance in real time through evaluation indicators such as accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score and joint accuracy. Through these multi-dimensional evaluations, the system can comprehensively compare the performance of different grading models to ensure that the final model has high accuracy and stability in the diabetic retinopathy grading task, thereby providing a reliable reference for actual diagnosis.

[0114] The public dataset provided by the embodiment of the present invention is:

[0115] The EAM dataset was constructed by integrating the EyePacs, Messidor, and Aptos datasets; the EAM dataset contains a total of 92,501 images;

[0116] EyePacs: As the largest DR grading dataset, EyePacs contains 35,126 images with large differences in image quality, which can effectively test the robustness of the model to different image qualities; Messidor: This dataset contains 1,200 high-quality color fundus images, which can be used to verify the performance of the model on a small-scale dataset; Aptos: This dataset provides 3,662 high-quality fundus images, which can be used as supplementary training data for the model.

[0117] The competition data set provided by the embodiment of the present invention is:

[0118] IDRiD: This dataset provides 516 high-quality fundus images and two labels, DR and DME, which can be used to evaluate the multi-task learning ability of the model; DDR: This dataset contains 13,673 images covering multiple DR levels and can be used to evaluate the performance of the model for fine-grained classification.

[0119] The evaluation index module provided by the embodiment of the present invention is:

[0120] The accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score, and joint accuracy were selected as evaluation indicators to analyze and compare the performance of different classification models;

[0121] Accuracy is used to measure the accuracy of model predictions, that is, the proportion of samples predicted correctly by the model to the total samples.

[0122]

[0123] Among them, TP, TN, FP, and FN are the number of samples of true positive examples, true negative examples, false positive examples, and false negative examples, respectively.

[0124] The consistency test coefficient (Kappa) further evaluates the consistency between the model prediction results and the true label. The Kappa value range is 1 to 1. It is generally believed that when the Kappa value is above 0.8, the model has good consistency.

[0125]

[0126] Among them, W is the weight matrix, W i.j represents the penalty between the predicted category i and the true category j. O is the observation matrix, that is, the confusion matrix, O i.j Represents the number of samples of predicted category i and true category j. E is the expectation matrix, E i.j It represents the probability that a class i is predicted to be class j in the case of random prediction.

[0127] Indicators such as AUC, Recall, and F1 are mainly used in binary classification tasks. AUC represents the area under the ROC curve, ranging from 0 to 1. 0.5 means that the model performance is close to random, and 1 means that the model is the best. The formulas for Recall and F1 are as follows:

[0128]

[0129] Joint Accuracy is used for the joint classification task on the IDRiD dataset and is defined as follows: if the prediction result of the sample model matches the true label of both its DR and DME, it is counted as a valid sample. The joint accuracy is obtained by dividing the number of all valid samples by the total number of samples.

[0130] The frequency domain channel attention module provided by the embodiment of the present invention:

[0131] (1) Discrete Cosine Transform

[0132] Discrete cosine transform is an orthogonal transform that converts time domain or spatial domain signals into frequency domain representation; DCT has a good energy concentration characteristic, that is, most of the signal energy is concentrated in the low-frequency part; in the field of image compression, DCT is widely used in the JPEG image compression standard; using this characteristic of DCT, the image is converted from the spatial domain to the frequency domain to extract the frequency domain features of the image;

[0133] For a two-dimensional image x 2d ∈R H×W , which corresponds to the two-dimensional DCT transformed image f 2d ∈R H×W The formula is as follows:

[0134]

[0135] sth∈{0,1,…,H-1},w∈{0,1,…,W-1}.

[0136] Among them, H and W represent x respectively. 2d The height and width of B h,w ∈R H×W The basis function representing the DCT transform is the key to extracting the frequency information of the image; Respectively represent x 2d , B h,w The pixel at (i,j) in represents f 2d The pixel at (h,w) in The formula can be expressed as:

[0137]

[0138] When h = 0, w = 0 in formula (8), the DCT transform is equivalent to global average pooling (GAP), as shown in formula (9):

[0139]

[0140] That is, using the basis function to perform DCT transformation on the image can be regarded as a pooling operation;

[0141] (2) Multi-spectral Attention Layer

[0142] Given an input feature map F∈R C×H×W , through drMSA, we can get the DCT frequency domain channel attention map F dct_att ∈R C ×H×W:

[0143]

[0144] F is the input feature map, F MSA ∈R C×1×1 Represents the attention weight vector obtained by DCT basis function transformation, W 0 With W 1 Represents the weight of MLP, and the final output of drMSA is F dct_att ;

[0145] For the CAM part, given the input feature map F∈R H×W×C , the channel attention map F att ∈R C×H×W :

[0146]

[0147] In the above formula, F gap ∈R C×1×1 With F gmp ∈R C×1×1 is the weight vector obtained by GAP and GMP, which is multiplied by F to get the final result F att ;

[0148] Finally, drMSA output F dct_att With CAM output F att Add element by element to allow the model to capture feature information from both the spatial domain and the frequency domain, increase model complexity and data diversity, and improve generalization capabilities;

[0149] Get the output F of FCAM DFCAM ∈R C×H×W ;

[0150]

[0151] The frequency domain spatial attention module provided by the embodiment of the present invention:

[0152] Given an input feature map F∈R C×H×W , and its corresponding frequency domain image is F dct ∈R C×H×W First, the two feature maps are stacked and concatenated through GAP and GMP operations to obtain two intermediate feature maps F concat ∈R 2×H×W ;

[0153] After the concatenation operation, the intermediate feature maps in the spatial domain and the frequency domain are fused by element-by-element multiplication to obtain F mix ∈R 2×H×W ,The fusion feature information can enrich and highlight the most ,noticeable details in the image;

[0154] Finally, F mix Send it to the convolution layer and activation function to get the attention map F att ∈R 1×H×W , output F FSAM ∈R C×H×W It can be expressed as:

[0155] F concat =[GAP(F),GMP(F)]

[0156] F dct_concat =[GAP(F dct ),GMP(F dct )]

[0157]

[0158] Among them, GAP and GMP are global average pooling and global maximum pooling, W is the convolution filter weight, and the softmax operation obtains the spatial attention map, which is finally multiplied with the original image to obtain the result F FSAM .

[0159] like Figure 2 As shown, a method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism provided by an embodiment of the present invention includes:

[0160] S101, constructing a training data set and a test data set through a data set construction module;

[0161] S102, performing various image enhancement processes on the training data set through a data preprocessing module, including: horizontal flipping, vertical flipping, rotation, cropping, and randomly adjusting the brightness and contrast of the image;

[0162] S103, selecting accuracy, consistency test coefficient, area under ROC curve, recall rate, F1 score, and joint accuracy as evaluation indicators through an evaluation indicator module to compare and analyze the performance of different classification models;

[0163] S104, extracting deep features of the CFP image using a feature extraction module with ResNet50 as the backbone network;

[0164] S105, extracting the frequency domain features of the image by using the improved multi-domain attention layer through the frequency domain channel attention module, and fusing them with the spatial domain features extracted by the channel attention module, giving higher weights to the key channels, thereby improving the sensitivity of the model to the lesion area;

[0165] S106, extracting the attention maps of the image in the spatial domain and frequency domain through the frequency domain spatial attention module, and enhancing the subtle texture features in the image and emphasizing the key pixels by multiplying them pixel by pixel.

[0166] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism.

[0167] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism.

[0168] Another object of the present invention is to provide an information data processing terminal, which is used to implement the diabetic retinopathy grading model system using the frequency domain attention mechanism.

[0169] The present invention is specifically implemented:

[0170] 1. Figure 3 The figure shows a schematic diagram of DR grading. DR grading is of great significance for clinical diagnosis and treatment. Doctors evaluate the type, number and distribution of DR lesions to determine the severity of the disease and formulate corresponding treatment plans. With the rapid development of artificial intelligence technology, deep learning and other technologies have been widely used in DR image analysis, providing new means for early diagnosis and precise treatment of DR.

[0171] This paper proposes a diabetic retinopathy grading model system based on frequency domain attention mechanism. The model effectively enhances the model's ability to extract and grade DR lesion features by introducing frequency domain channel attention module (FCAM) ​​and frequency domain spatial attention module (FSAM).

[0172] 2. Related Work

[0173] 2.1 Deep Learning Model

[0174] The rise of deep learning has injected new vitality into automated diagnostic technology. In the field of ophthalmology, deep convolutional neural networks (CNNs) have become the first choice for DR grading tasks due to their powerful feature extraction capabilities. The classic CNN architecture usually consists of convolutional layers, pooling layers, fully connected layers, and activation functions. This structure significantly reduces the number of model parameters, making it perform well in tasks such as image classification and target detection. For example, Pratt et al. combined image enhancement technology and used the classic CNN model to grade DR in color fundus images; Wang et al. proposed a two-stream CNN for the multimodal age-related macular degeneration grading problem, which effectively integrated color fundus images (CFP) and optical coherence tomography (OCT) information. Although CNN has achieved excellent performance in DR grading tasks, it still has certain limitations when dealing with subtle features (such as microaneurysms).

[0175] 2.2 Attention Mechanism

[0176] In order to improve the model's ability to capture subtle features, researchers introduced the attention mechanism into CNN. The attention mechanism enables the model to focus on key information by assigning different weights to different features. The attention mechanism has been widely used in image classification, object detection and other fields. In the DR grading task, Yang et al. designed a two-stage network that can not only locate the lesion but also grade DR. Li et al. simultaneously evaluated DR and diabetic macular edema by a joint grading method, and used disease-specific attention modules to learn disease-specific features. The spatial domain attention mechanism improves the performance of the model to a certain extent, but its global perception ability still needs to be improved. To solve this problem, the frequency domain attention mechanism came into being. By converting the data to the frequency domain, the model can better capture global correlations.

[0177] 2.3 Frequency Domain Attention Mechanism

[0178] The frequency domain attention mechanism transforms spatial domain features into frequency domain features, allowing the model to capture richer feature information from a global perspective. Qin et al. proved that discrete cosine transform (DCT) is equivalent to global average pooling (GAP) and integrated it into the channel attention mechanism. Zhou et al. embedded the fast Fourier transform (FFT) into Transformer and constructed a frequency domain enhancement module to replace the self-attention and cross-attention modules. Compared with these studies, FCAM+FSAM can consider the characteristics of both frequency domain and spatial domain at the same time, and it is expected to further improve the global perception and robustness of the model through dual-domain fusion.

[0179] 3. Materials and methods

[0180] 3.1 Dataset

[0181] In order to comprehensively evaluate the performance of the proposed model and improve the generalization ability of the model, the present invention integrates the three datasets of EyePacs, Messidor, and Aptos to construct the EAM dataset. The EAM dataset contains a total of 92,501 images.

[0182] EyePacs: As the largest DR grading dataset, EyePacs contains 35,126 images with large differences in image quality, which can effectively test the robustness of the model to different image qualities. Messidor: This dataset contains 1,200 high-quality color fundus images, which can be used to verify the performance of the model on a small-scale dataset. Aptos: This dataset provides 3,662 high-quality fundus images, which can be used as supplementary training data for the model.

[0183] IDRiD: This dataset provides 516 high-quality fundus images and two labels, DR and DME, which can be used to evaluate the multi-task learning ability of the model. DDR: This dataset contains 13,673 images covering multiple DR levels and can be used to evaluate the performance of the model for fine-grained classification.

[0184] 3.2 Data Preprocessing and Experimental Setup

[0185] In order to improve the generalization ability of the model, a variety of image enhancement processes were performed on the training data set. Specific methods include: horizontal flipping, vertical flipping, rotation, cropping, and random adjustment of image brightness and contrast. Through these operations, the diversity of the data set was expanded, thereby enhancing the model's adaptability to different types of images, and improving the model's robustness and classification accuracy.

[0186] The experiment of the present invention uses ResNet50 as the basic network and loads the ImageNet pre-trained model. The input image size is unified to 512x512 pixels. The initial learning rate is set to 0.001, the batch size is 32, and the cosine annealing strategy is used to adjust the learning rate to improve the stability of model training. All models are trained for 20 epochs, using the stochastic gradient descent (SGD) optimizer and cross entropy loss as the objective function for optimization. Detailed parameters are shown in Table 1.

[0187] All experimental models are built in Python based on the PyTorch deep learning framework. The experiments are run on the Kaggle platform and GPU P100 (16GB) is used to accelerate the training process.

[0188] Table 1 DR classification model experimental parameters

[0189]

[0190] 3.3 Evaluation Metrics

[0191] The accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score, and joint accuracy were selected as evaluation indicators to analyze and compare the performance of different classification models;

[0192] Accuracy is used to measure the accuracy of model predictions, that is, the proportion of samples predicted correctly by the model to the total samples.

[0193]

[0194] Among them, TP, TN, FP, and FN are the number of samples of true positive examples, true negative examples, false positive examples, and false negative examples, respectively.

[0195] The consistency test coefficient (Kappa) further evaluates the consistency between the model prediction results and the true label. The Kappa value range is 1 to 1. It is generally believed that when the Kappa value is above 0.8, the model has good consistency.

[0196]

[0197] Among them, W is the weight matrix, W i.j represents the penalty between the predicted category i and the true category j. O is the observation matrix, that is, the confusion matrix, O i.j Represents the number of samples of predicted category i and true category j. E is the expectation matrix, E i.j It represents the probability that a class i is predicted to be class j in the case of random prediction.

[0198] Indicators such as AUC, Recall, and F1 are mainly used in binary classification tasks. AUC represents the area under the ROC curve, ranging from 0 to 1. 0.5 means that the model performance is close to random, and 1 means that the model is the best. The formulas for Recall and F1 are as follows:

[0199]

[0200] Joint Accuracy is used for the joint classification task on the IDRiD dataset and is defined as follows: if the prediction result of the sample model matches the true label of both its DR and DME, it is counted as a valid sample. The joint accuracy is obtained by dividing the number of all valid samples by the total number of samples.

[0201] 3.4 Diabetic retinopathy grading model system

[0202] Figure 4The figure shows the system architecture diagram of the diabetic retinopathy grading model based on the frequency domain attention mechanism proposed by the present invention. The model uses ResNet50 as the backbone network to extract the deep features of the CFP image. In order to enhance the model's attention to the lesion area, the frequency channel attention module (FCAM) ​​and the frequency spatial attention module (FSAM) are introduced. Among them: (1) FCAM: The frequency domain features of the image are extracted through the improved multi-domain attention layer (MSA), and are integrated with the spatial domain features extracted by the channel attention module (CAM), giving the key channel a higher weight, thereby improving the model's sensitivity to the lesion area; (2) FSAM: The attention maps of the spatial domain and frequency domain of the image are extracted simultaneously, and the subtle texture features in the image are enhanced by pixel-by-pixel multiplication, emphasizing the key pixels.

[0203] The FCAM and FSAM modules introduce frequency domain information from the channel and spatial dimensions respectively to enhance the model's perception of lesion features. The features processed by the attention module are fused with the original features. The fused features are classified using a fully connected layer to achieve the classification of diabetic retinopathy.

[0204] Figure 4 In the figure, the first column gives the overall architecture of the entire network, visually showing the connection relationship between each module. The second column deeply analyzes the internal structure of the FCAM and FSAM modules, and explains in detail the working principle of each module.

[0205] 3.4.1 Frequency Channel Attention Module (FCAM)

[0206] As an effective feature calibration method, the channel attention mechanism has been widely used in convolutional neural networks. The classic channel attention module (CAM) generates channel weights through global pooling operations and multi-layer perceptrons (MLPs) to emphasize key features. However, CAM only focuses on spatial domain information and ignores the rich information that the image may contain in the frequency domain. In order to make up for the shortcomings of CAM, the present invention introduces the frequency domain channel attention module (FCAM).

[0207] FCAM extracts frequency domain features of images through an improved multi-spectral attention layer (MSA). The MSA layer first converts the image from the spatial domain to the frequency domain using discrete cosine transform (DCT), and then captures the correlation between frequency domain features through the self-attention mechanism. Finally, the obtained frequency domain features are fused with the spatial domain features extracted by CAM to generate a more comprehensive channel attention. This design enables FCAM to focus on feature information in both the spatial and frequency domains, thereby giving higher weights to key channels and improving the model's sensitivity to lesion areas.

[0208] (1) Discrete Cosine Transform (DCT)

[0209] Discrete cosine transform (DCT) is an orthogonal transform that converts time domain or spatial domain signals into frequency domain representation. DCT has a good energy concentration characteristic, that is, most of the signal energy is concentrated in the low-frequency part. In the field of image compression, DCT is widely used in the JPEG image compression standard. The present invention utilizes this characteristic of DCT to convert the image from the spatial domain to the frequency domain to extract the frequency domain features of the image.

[0210] For a two-dimensional image x 2d ∈R H×W , which corresponds to the two-dimensional DCT transformed image f 2d ∈R H×W The formula is as follows:

[0211]

[0212] sth∈{0,1,…,H-1},w∈{0,1,…,W-1}.

[0213] Among them, H and W represent x respectively. 2d The height and width of B h,w ∈R H×W The basis function representing DCT transform is the key to extracting image frequency information. Respectively represent x 2d , B h,w The pixel at (i,j) in represents f 2d The pixel at (h,w) in The formula can be expressed as:

[0214]

[0215] When h = 0 and w = 0 in formula (8), the DCT transform is equivalent to global average pooling (GAP) [1], as shown in formula (9):

[0216]

[0217] That is, using basis functions to perform DCT transformation on an image can be considered a pooling operation.

[0218] (2) Multispectral Attention Layer (MSA)

[0219] MSA decomposes the image into multiple frequency bands by using discrete cosine transform (DCT) basis functions of different frequencies, thereby realizing the extraction of multi-scale features. The DCT basis function has good energy concentration characteristics. The low-frequency component usually contains the global information of the image, while the high-frequency component contains the detailed information of the image. In this way, MSA can capture the rich features of the image from the frequency domain perspective, thereby improving the expressive power of the model. However, different combinations of DCT basis functions will have different effects on channel attention. FcaNet proposes a basis function selection scheme based on classification effect, but this scheme is not necessarily suitable for diabetic retinopathy (DR) grading tasks. The scheme will be referred to as "fca" in the following text.

[0220] In order to better adapt to the diabetic retinopathy (DR) grading task, the present invention improves the basis function selection scheme of the multi-spectral attention layer (MSA): (1) Increasing the number of basis functions: In order to more comprehensively capture the high-frequency detail information of the image, the number of discrete cosine transform (DCT) basis functions is increased from the original 49 to 64, such as Figure 5 As shown in (a), 15 new high-frequency basis functions are added. (2) Basis function screening based on DR dataset: By conducting comparative experiments on DR public datasets, the basis function combination that is most suitable for DR grading tasks is screened out.

[0221] The improved MSA module is called “drMSA”. The basis function structure of drMSA will be dynamically adjusted according to the classification accuracy of each basis function on the DR dataset. Figure 5 (b) shows the classification accuracy of 64 basis functions, taking the Messidor dataset as an example. Subsequent research can select different DR datasets according to different task scenarios and refer to Figure 5 The result in (b) flexibly selects the optimal combination of basis functions.

[0222] After determining the main basis functions of drMSA, the channels of the input features are grouped. Each group of channels is assigned a specific basis function to extract the corresponding frequency domain features. In this way, a set of channel vectors containing different frequency features is obtained. Subsequently, these channel vectors are processed using a multi-layer perceptron (MLP) to generate a weight vector representing the importance of each channel in the frequency domain. Given an input feature map F∈R C×H×W , through drMSA, we can get the DCT frequency domain channel attention map Fdct_att ∈R C×H×W :

[0223]

[0224] F is the input feature map, F MSA ∈R C×1×1 Represents the attention weight vector obtained by DCT basis function transformation, W 0 With W 1 Represents the weight of MLP, and the final output of drMSA is F dct_att .

[0225] For the CAM part, given the input feature map F∈R H×W×C , the channel attention map F att ∈R C×H×W :

[0226]

[0227] In the above formula, F gap ∈R C×1×1 With F gmp ∈R C×1×1 is the weight vector obtained by GAP and GMP, which is multiplied by F to get the final result F att .

[0228] Finally, drMSA output F dct_att With CAM output F att Add element by element to allow the model to capture feature information from both the spatial domain and the frequency domain, increase model complexity and data diversity, and improve generalization capabilities.

[0229] Get the output F of FCAM DFCAM ∈R C×H×W .

[0230]

[0231] 3.4.2 Frequency Spatial Attention Module (FSAM)

[0232] The spatial attention module (SAM) in CBAM can effectively capture the spatial information of the image and help the model better understand the global structure and semantic information of the image. However, SAM only focuses on the spatial domain features and ignores the rich information contained in the image in the frequency domain.

[0233] In order to make full use of frequency domain information, the present invention proposes a frequency domain spatial attention module (FSAM). FSAM introduces the DCT frequency domain spatial attention mechanism based on SAM. Through DCT transformation, the image is converted from the spatial domain to the frequency domain to extract the frequency domain features of the image. These frequency domain features contain the detailed texture information of the image, and combined with the spatial domain features, they can enhance the model's attention to key pixels and improve the model's feature extraction ability and classification accuracy. In addition, the fusion of frequency domain spatial features also makes the model more robust to noise.

[0234] FSAM structure is as follows Figure 4 As shown. Given an input feature map F∈R C×H×W , and its corresponding frequency domain image is F dct ∈R C×H×W First, the two feature maps are stacked and concatenated through GAP and GMP operations to obtain two intermediate feature maps F concat ∈R 2 ×H×W .

[0235] After the concatenation operation, the intermediate feature maps in the spatial domain and the frequency domain are fused by element-by-element multiplication to obtain F mix ∈R 2×H×W ,The fusion feature information can enrich and highlight the most ,noticeable details in the image.

[0236] Finally, F mix Send it to the convolution layer and activation function to get the attention map F att ∈R 1×H×W , output F FSAM ∈R C×H×W It can be expressed as:

[0237] F concat =[GAP(F),GMP(F)]

[0238] F dct_concat =[GAP(F dct ),GMP(F dct )]

[0239]

[0240] Among them, GAP and GMP are global average pooling and global maximum pooling, W is the convolution filter weight, and the softmax operation obtains the spatial attention map, which is finally multiplied with the original image to obtain the result F FSAM .

[0241] 4. Experimental analysis and results comparison

[0242] In order to evaluate the performance of the proposed diabetic retinopathy grading model system, the present invention designed two sets of experiments. Experiment 1 is an ablation experiment, which evaluates the contribution of each improved module of the diabetic retinopathy grading model system proposed in the present invention to the grading performance by comparing with grading models with different attention mechanisms. Experiment 2 is a comparative experiment, which compares with other end-to-end models to evaluate the grading advantage of the proposed diabetic retinopathy grading model system.

[0243] 4.1 Ablation Experiment

[0244] 4.1.1 Comparison and Analysis of Multi-frequency Attention Layer (MSA)

[0245] In order to comprehensively evaluate the effectiveness of drMSA, comparative experiments were conducted. In the experiment, the drMSA and MSA modules were respectively embedded in the frequency domain channel attention module (FCAM), and comparative experiments were conducted on three public diabetic retinopathy (DR) datasets: EAM dataset, IDRiD, and DDR. In order to verify the impact of different numbers of basis functions on model performance, four different basis function combinations were set, namely "dr4", "dr8", "dr16", and "dr32", and the numbers represent the number of selected basis functions. In order to make a fair comparison, MSA also uses the same number of basis functions.

[0246] The experimental results are shown in Table 2. Table 2 shows the performance of different basis function selection schemes on the EAM dataset. The results show that the dr32 scheme achieves the best performance on the EAM dataset, and the performance differences of different dr schemes are small, all within 0.01. Overall, drMSA performs better than MSA on the EAM dataset.

[0247] Tables 3 and 4 show the generalization ability of each module on the IDRiD and DDR test sets, respectively. Among them, Table 4 calculates the performance improvement of the drMSA and MSA modules compared with CBAM. The experimental results show that on the IDRiD dataset, the dr8 scheme performs best, with a Kappa index improvement of 2.22% and an Acc index improvement of 1.55%. On the DDR dataset, the fca16 scheme performs best, with a Kappa index improvement of 2.57% and an Acc index improvement of 2.24%. The experimental results show that drMSA performs better than MSA on multiple DR datasets, especially for the IDRiD dataset, the performance improvement of drMSA is more significant. This fully demonstrates the effectiveness of drMSA in DR classification tasks.

[0248] Table 2 Comparison and analysis of different settings of the improved multi-spectral attention layer (drMSA) (taking the EAM dataset as an example)

[0249]

[0250] Table 3 Comparison and analysis of the improved multi-spectral attention layer (drMSA) on different test sets

[0251]

[0252] Table 4 Performance improvement of MSA, improved multi-spectral attention layer (drMSA) compared with CBAM model (“” represents no improvement)

[0253]

[0254]

[0255] 4.1.2 Comparison and analysis of different attention modules

[0256] In order to deeply evaluate the role of frequency domain attention modules FCAM and FSAM in the diabetic retinopathy (DR) grading model, this section uses ResNet50 as the backbone network to compare and analyze the performance of different attention module combinations on three public datasets: EAM dataset, IDRiD, and DDR.

[0257] The experimental results are shown in Tables 5 and 6. Table 5 shows the performance of different attention module combinations on the EAM dataset. The results show that the combination strategy of FCAM+FSAM achieves the best performance, with a Kappa coefficient of 84.21% and an accuracy of 88.36%.

[0258] By comparing with other combination strategies, we can find that: (1) The spatial attention module is fixed: When the spatial attention module remains unchanged (SAM), the channel attention module is replaced from CAM to MSA, and then to FCAM, the model performance continues to improve. (2) Introducing frequency domain spatial attention: On the basis of replacing the channel attention module with FCAM, FSAM is further introduced, and the model performance is significantly improved. The Kappa coefficient is greatly improved by 0.64%, and the accuracy remains stable. Compared with the traditional CBAM, the combination strategy of FCAM+FSAM improves the Kappa coefficient by 1.96% and the accuracy by 0.8%, which fully proves the effectiveness of this combination strategy.

[0259] Table 6 shows the generalization capabilities of different attention module combinations on multiple datasets. The results show that the FCAM(dr8)+FSAM combination exhibits the best generalization performance on multiple datasets, which indicates that introducing frequency domain information in channel attention and spatial attention at the same time can effectively improve the feature extraction and generalization capabilities of the model.

[0260] Experimental results show that the combination of FCAM and FSAM can effectively improve the performance of the DR classification model. Among them, FCAM focuses on the features in the channel dimension, and FSAM focuses on the features in the spatial dimension. The combination of the two is more comprehensive.

[0261] Table 5 Analysis and comparison of DR grading models with different attention mechanisms (taking the EAM dataset as an example)

[0262]

[0263] Table 6 Analysis and comparison of DR grading models with different attention mechanisms on multi-source datasets

[0264]

[0265]

[0266] 4.2 Comparative Experiments

[0267] In order to comprehensively evaluate the performance of the proposed model in the diabetic retinopathy (DR) grading task, this section will conduct comparative experiments with the current state-of-the-art (SOTA) DR grading model.

[0268] 4.2.1 Comparative analysis of model performance based on binary classification tasks

[0269] This experiment uses Messidor as experimental data. The Messidor dataset is a commonly used diabetic retinopathy (DR) dataset. It is a two-level classification dataset. Level 01 is considered normal and level 23 is considered abnormal. It contains 540 normal images and 660 abnormal images. In order to compare with other methods more accurately, the present invention is compared with CANet and CABNet. Figure 6 The ROC curves of the model of the present invention, CANet, and CABNet on the Messidor validation set are shown. Figure 4 It can be seen that the ROC curve of the present invention is obviously higher than that of other methods. The AUC of the model of the present invention is 98.1%, which is 1.5% higher than that of CABNet and CANet methods.

[0270] Table 7 shows the experimental results of different models on the IDRiD dataset. As shown in Table 7, the model of the present invention has achieved a leading AUC in the Messidor binary classification task. Compared with the current SOTA model CABNet, our model has improved AUC, accuracy and F1 value by 1.2%, 0.7%, 2.0% and 1.9% respectively. This shows that our model can more accurately distinguish normal and abnormal DR fundus images, showing significant advantages in DR grading tasks.

[0271] Table 7 DR hierarchical ablation experiment results on Messidor validation set (data units are %, R, F1 represent Recall, F1Score, respectively)

[0272]

[0273] 4.2.2 Comparative Experiments on Multi-Classification Tasks on IDRiD Dataset

[0274] This experiment uses the IDRiD dataset as experimental data, and classifies diabetic retinopathy (DR) and diabetic maculopathy (DME) at the same time. DR is divided into five categories, namely normal (Normal), non-proliferative diabetic retinopathy (NPDR), mild non-proliferative diabetic retinopathy (Mild NPDR), moderate non-proliferative diabetic retinopathy (Moderate NPDR), severe non-proliferative diabetic retinopathy (Severe NPDR), and proliferative diabetic retinopathy (PDR). Diabetic macular edema (DME) is divided into three levels according to the shortest distance d from the location of hard exudates to the macula, namely 0 (no hard exudates), 1 (d is greater than the optic disc diameter), and 2 (d is less than or equal to the optic disc diameter). The experimental results are shown in Table 8. The joint grading accuracy of our method on the IDRiD dataset reached 64.1%, which is 1% higher than that of the lzyuncc method. And it also shows good performance in the classification accuracy of DME. The classification confusion matrix of DR and DME of the method of the present invention on the IDRiD dataset is shown as follows. Figure 7 shown.

[0275] Table 8 IDRi D combined grading results

[0276]

[0277] This paper proposes a convolutional neural network based on the frequency domain attention mechanism for diabetic retinopathy (DR) grading. By introducing the frequency domain channel attention module (FCAM) ​​and the frequency domain spatial attention module (FSAM), the model can more effectively extract the subtle features of DR lesions and improve the classification accuracy. Experimental results show that the model has achieved better performance than the existing technology on multiple public datasets. The innovations are: 1) a new frequency domain attention mechanism is proposed to enhance the model's ability to express DR lesion characteristics; 2) the generalization ability of the model is improved through data enhancement strategies.

[0278] The present invention is specifically applied to the field of diabetic retinopathy (DR) grading diagnosis and lesion segmentation, embedding the frequency domain attention mechanism into the deep learning model to perform automated DR grading and lesion segmentation.

[0279] The present invention has achieved good results in two tasks. In the binary classification task, the ROC curve AUC of the model of the present invention on the Messidor validation set is 97.6%. Compared with the results of similar methods such as the CANet method with an AUC of 96.3% and the CABNet method with an AUC of 96.9%, the model of the present invention has taken the lead in AUC. Compared with the current SOTA model CABNet, the model has improved by 0.7%, 1.5%, 0.4% and 1.3% in AUC, accuracy, precision and F1 value respectively. In the multi-classification task, the joint grading accuracy of DR and DME of the model of the present invention on the IDRiD dataset reached 64.1%, which is 1% higher than the lzyuncc method and ranks second.

[0280] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. A person of ordinary skill in the art will appreciate that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVDROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0281] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A diabetic retinopathy grading model system using frequency domain attention mechanism, characterized in that: include: Dataset building module, used to integrate public datasets and competition datasets; Data preprocessing module, used to perform image enhancement on training data, including horizontal flipping, vertical flipping, rotation, cropping, and random adjustment of brightness and contrast; Evaluation indicator module, which is used to evaluate the model performance using indicators such as accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score and joint accuracy; Feature extraction module, which extracts deep features of CFP images based on the ResNet50 backbone network; The frequency domain channel attention module extracts the frequency domain features of the image through the improved multi-domain attention layer and integrates the spatial domain features of the channel attention module; The frequency domain spatial attention module enhances the subtle texture features in the image and highlights the key pixels.

2. The system according to claim 1, characterized in that The public datasets include EyePacs, Messidor, and Aptos datasets. The EAM dataset is constructed through integration, which contains a total of 92,501 images. EyePacs is used to test the robustness of the model under different image qualities, and Messidor and Aptos provide high-quality images to verify the performance of small-scale datasets.

3. The system according to claim 1, characterized in that The competition datasets include IDRiD and DDR datasets, where the IDRiD dataset is used to evaluate the multi-task learning ability of the model, and the DDR dataset covers multiple levels of diabetic retinopathy and can be used to evaluate the performance of the model for fine-grained classification.

4. The system according to claim 1, characterized in that The evaluation index module adopts joint accuracy for the joint classification task of IDRiD dataset, calculates the model performance by matching the DR and DME labels of samples, and improves the accuracy of the system in a multi-task environment.

5. The system according to claim 1, characterized in that The frequency domain channel attention module includes: Use discrete cosine transform (DCT) to convert the image from the spatial domain to the frequency domain and extract the low-frequency information of the image; Based on the multi-spectral attention layer, the attention map is obtained through the weight vector after DCT transformation and the MLP weight to capture the feature information in the spatial and frequency domains and improve the generalization ability of the model.

6. The system according to claim 1, wherein: The frequency-domain spatial attention module performs global average pooling (GAP) and global maximum pooling (GMP) on the feature map, stacks and concatenates them, generates a spatial attention map through convolution and activation functions, and finally multiplies it with the original image to improve the detail recognition ability of the image.

7. A method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism according to any one of claims 1 to 6, characterized in that: The method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism comprises: Step 1: Build training data set and test data set through data set building module; Step 2: Perform various image enhancement processes on the training data set through the data preprocessing module, including horizontal flipping, vertical flipping, rotation, cropping, and random adjustment of image brightness and contrast; Step 3, the accuracy, consistency test coefficient, area under the ROC curve, recall rate, F1 score, and joint accuracy are selected as evaluation indicators through the evaluation indicator module to compare and analyze the performance of different classification models; Step 4, extract the deep features of the CFP image through the feature extraction module using ResNet50 as the backbone network; Step 5: The frequency domain features of the image are extracted by using the improved multi-domain attention layer through the frequency domain channel attention module, and are integrated with the spatial domain features extracted by the channel attention module to give higher weights to the key channels, thereby improving the model's sensitivity to the lesion area. Step 6: The frequency-domain spatial attention module is used to extract the attention maps of the image in the spatial and frequency domains, and the subtle texture features in the image are enhanced by multiplying them pixel by pixel to emphasize the key pixels.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism as described in claim 7.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the method for constructing a diabetic retinopathy grading model system using a frequency domain attention mechanism as described in claim 7.

10. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the diabetic retinopathy grading model system using the frequency domain attention mechanism as described in any one of claim 16.

Images