A diabetic retinopathy grading method and system based on CNN and transformer

Abstract

The application discloses a diabetic retinopathy grading method based on CNN and a Transformer, and comprises the following steps: importing an original image dataset, dividing the original image dataset into an image training set and an image test set; dividing each original image in the image training set into several image blocks, and generating disordered images of different granularity versions; establishing a diabetic retinopathy grading network through a residual network and a Swin-Transformer, training the diabetic retinopathy grading network by adopting a progressive strategy, and obtaining a multi-scale and multi-granularity feature spectrum output in a series stage; defining a classification loss function and a weighted kappa loss function, and constructing a loss layer according to the classification loss function and the weighted kappa loss function; optimizing the diabetic retinopathy grading network by using the image training set; and testing the diabetic retinopathy grading network by using a test sample set. The application can realize more accurate diabetic retinopathy grading, and simultaneously obtain category supervision information and ordered supervision information.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of deep learning fine-grained visual classification, and particularly relates to a diabetic retinopathy grading method and system based on CNN and Transformer. BACKGROUND

[0002] According to the data released by the International Diabetes Federation, the number of adults with diabetes reached 537 million in 2021, about one tenth of the number of adults in the world, and is expected to reach 784 million in 2045. Diabetic retinopathy is one of the common complications of diabetes. According to incomplete statistics, one in every three diabetic patients will suffer from diabetic retinopathy. The main symptoms of diabetic retinopathy include microaneurysms, exudates and hemorrhages. The type and number of lesions in the fundus image determine the diabetic retinopathy grade, so how to detect these tiny lesions to achieve diabetic retinopathy grading is a challenging task.

[0003] The CNN-based diabetic retinopathy classification method has a small receptive field, which is beneficial to local feature extraction, but the range of available information is limited. The diabetic retinopathy classification method based on Transformer has a large receptive field, which is beneficial to modeling long-distance relationships, but it is easy to lose local details. In addition, the severity of DR follows a natural order from mild to severe, but the large logarithmic DR grading method ignores the order information between diabetic retinopathies of different severity, and classifies diabetic retinopathy as a multi-class classification problem. SUMMARY

[0004] The purpose of the present application is to provide a diabetic retinopathy grading method and system based on CNN and Transformer, which obtains a more suitable receptive field through feature fusion, discovers more discriminative regions in the fundus image, and realizes more accurate diabetic retinopathy grading. Diabetic retinopathy grading is considered as a multi-class classification problem of joint ordered regression, and class supervision information and ordered supervision information are obtained at the same time, so that the final classification performance of the model reaches the current advanced level.

[0005] To solve the above technical problems, the technical scheme of the present application is as follows: a diabetic retinopathy grading method based on CNN and Transformer, comprising the following steps:

[0006] S1: importing an original image data set with multiple groups of original images, and dividing the original image data set into an image training set and an image test set;

[0007] S2: dividing each original image in the image training set into several image blocks, and generating shuffled images of different granularities;

[0008] S3: Establishing a diabetic retinopathy grading network through a residual network ResNet50 and a Swin-Transformer, and training the diabetic retinopathy grading network by using a progressive strategy; inputting a shuffled image, outputting three intermediate stage feature spectrums with different granularity sizes through the last three layers of the residual network ResNet50; inputting an original image, outputting three intermediate stage feature spectrums with different sizes through the last three layers of the Swin-Transformer; fusing the feature spectrums output by the convolutional neural network at different stages and the feature spectrums output by the Swin-Transformer through a feature fusion module; concatenating the feature spectrums output by the last three layers of the residual network ResNet50 and the Swin-Transformer, and obtaining a multi-scale and multi-granularity feature spectrum output by a concatenated stage;

[0009] S4: Defining a classification loss function and a weighted kappa loss function, and constructing a loss layer according to the classification loss function and the weighted kappa loss function;

[0010] S5: Optimizing the diabetic retinopathy grading network by using an image training set; and testing the diabetic retinopathy grading network by using a test sample set.

[0011] S2 is specifically:

[0012] S21: Dividing an original image I into KxK image blocks, and obtaining an index matrix P I of jxj according to the index of the image block;

[0013] S22: Shuffling the image blocks, generating a shuffled image through a puzzle generator, and obtaining an index matrix P I of the shuffled image according to the index matrix P S of the image blocks;

[0014] S23: Obtaining a matrix P in one-hot form with a size of K 2 x j 2 according to the index matrix P S of the shuffled image.

[0015] S3 is specifically:

[0016] Establishing a diabetic retinopathy grading network, including a residual network, a Swin-Transformer, a feature fusion module, a convolutional layer and a classification layer;

[0017] The progressive strategy is used to train the diabetic retinopathy grading network; the training process specifically includes the following steps: step 1: the feature spectrum a3 output by the third layer of the residual network and the feature spectrum b2 output by the second layer of the Swin-Transformer are fused through the feature fusion module to obtain the first fused feature t3; step 2: the feature spectrum a4 output by the fourth layer of the residual network and the feature spectrum b3 output by the third layer of the Swin-Transformer are fused through the feature fusion module to obtain the second fused feature t4; step 3: the feature spectrum a5 output by the fifth layer of the residual network and the feature spectrum b4 output by the fourth layer of the Swin-Transformer are fused through the feature fusion module to obtain the third fused feature t5; step 4: the original image is used as the input image of the residual network ResNet50 and the Swin-Transformer, and the feature spectra X3, X4, and X5 fused by the residual network ResNet50 and the Swin-Transformer are concatenated to obtain the multi-scale and multi-granularity feature spectrum X output in the concatenation stage concat =Concat(X3,X4,X5);

[0018] The fused features t3, t4, and t5 of steps 1, 2, and 3 are respectively subjected to convolution layers to obtain corresponding feature spectra X3, X4, and X5;

[0019] The residual network ResNet50 extracts image features a of the scrambled image i The last three layers (a3, a4, and a5) of the residual network ResNet50 are used to process the scrambled image respectively; different intermediate stage feature spectra are output for different granular versions of the image, and K = 2 5-i+1 KxK represents the number of blocks in the input puzzle, where i = {3, 4, 5}, corresponding to the output feature spectra a3, a4, and a5 of the three stages respectively;

[0020] The Swin-Transformer extracts image features b of the image training set l The last three layers (b2, b3, and b4) of the Swin-Transformer are used to process the images of the image training set respectively, and correspond to the output of three-stage feature spectra b2, b3, and b4 respectively;

[0021] The feature fusion module includes a spatial attention module, a channel attention module, and a fine-grained interaction module; in steps 1, 2, 3, and 4, the feature fusion module is used to fuse the feature spectra extracted by the corresponding residual network and Swin-Transformer, which is specifically described as follows:

[0022] The spatial attention module includes a channel pooling layer, a 7x7 convolution layer Conv and a Sigmoid activation function, and the CNN branch features are enhanced by spatial attention, and the method is represented as:

[0023]

[0024] The channel attention module includes an average pooling layer, a full connection layer and a Sigmoid activation function, and the CNN branch features are enhanced by channel attention, and the method is represented as:

[0025]

[0026] The fine-grained interaction module includes two 1x1 convolution Conv1 and Conv2, Hadamard product and a 3x3 convolution Conv3, and the feature interaction is realized by the fine-grained interaction module, and the method is represented as:

[0027] c i =Conv3(Conv1(a i )⊙Conv2(b l ));

[0028] The enhanced features of the two branches and the interaction features are fused together, and a residual block is used to realize the complementary relationship between the output features of different branches, and the method is represented as:

[0029]

[0030] The fused feature spectrum is unified to 1024 dimensions by a convolution layer

[0031]

[0032] The classification layer is composed of two full connection layers with Batchnorm and Elu nonlinearity; diabetic retinopathy is divided into five grades, and the classification layer After processing the 1024-dimensional feature vector by the classifier, a 5-dimensional feature vector is obtained as the prediction probability distribution, which is represented as:

[0033]

[0034] Wherein, the prediction probability distribution of step 1, step 2, step 3 and step 4 is represented as V 3 ,V 4 ,V 5 ,V concat .

[0035] S4 is specifically:

[0036] ​S41: In step 1, step 2 and step 3, step i uses V i represents the predicted probability distribution, and in step 4, V n concat represents the predicted probability distribution; V represents the image true label, and the classification loss function is calculated by cross-entropy loss, represented as:

[0037]

[0038]

[0039] where n represents the image index, m represents the total number of images in a batch, represents the classification loss function in step 1, step 2 and step 3, represents the classification loss function in step 4.

[0040] S42: The classification loss function is calculated by weighted kappa loss function, represented as:

[0041]

[0042]

[0043]

[0044] where N represents the number of samples, N m represents the number of samples of class m, Q represents the total number of classes, w m,n represents the quadratic weighting matrix, where |m-n| represents the distance between the predicted class m and the actual class n, t k represents the true class of the kth sample X k , P(q|X k ) represents the conditional probability that the predicted class of the kth sample X k belongs to q;

[0045] S43: The total loss function is obtained by weighted average of cross-entropy loss function and weighted kappa loss function:

[0046]

[0047]

[0048] where β is a hyperparameter, represents the total loss function in step 1, step 2 and step 3, represents the total loss function in step 4.

[0049] S5 is specifically:

[0050] S51: using the image training set, using the random gradient descent and back propagation algorithm based on automatic differentiation technology, optimizing the diabetic retinopathy grading network according to the total loss function;

[0051] S52: using the image test set to test the diabetic retinopathy grading network based on the image training set weight.

[0052] Also provided is a diabetic retinopathy grading system based on CNN and Transformer, comprising a puzzle generation module, a residual network module, a Swin-Transformer module, a feature fusion module, a loss function module and a training and testing module; wherein,

[0053] The puzzle generation module is used to process the image training data set and generate disordered images of different granularity sizes;

[0054] The residual network module is used to use the last three layers of ResNet50 for progressive training, use images of different granularity sizes as input images in steps 1 to 3, and select different intermediate stage feature spectra outputs to obtain local detail information of the object; in step 4, the original image is used as the input image, and the residual network ResNet50 simultaneously outputs the feature spectra of the last three layers, obtaining multi-scale feature information;

[0055] The Swin-Transformer module is used to use the last three layers of Swin-Transformer for progressive training, use the original image as the input image, select different intermediate stage feature spectrum outputs in steps 1 to 3 to obtain global representation information of the object; in step 4, the Swin-Transformer simultaneously outputs the feature spectra of the last three layers, obtaining multi-scale feature information;

[0056] The feature fusion module is used to fuse the feature spectra of the corresponding stages of the residual network and the Swin-Transformer in steps 1 to 3, and output the fused features; in step 4, the feature spectra fused in steps 1 to 3 are concatenated to obtain multi-scale and multi-granularity feature spectra of the concatenated stage;

[0057] The loss function module is used to define the classification loss function and the weighted kappa loss function, and to construct the loss layer according to the classification loss function and the weighted kappa loss function;

[0058] The training and testing module is used to optimize the diabetic retinopathy grading network using the image training set, and test the diabetic retinopathy grading network using the image test set.

[0059] The workflow of the loss function module is as follows:

[0060] In step 1, step 2, step 3, step i uses V i represents the predicted probability distribution, in step 4, using represents the predicted probability distribution, the i-th stage uses V i represents the predicted probability distribution, using V represents the image true label, the classification loss function is calculated by cross-entropy loss, represented as:

[0061]

[0062]

[0063] where n represents the image index, m represents the total number of images in a batch, can represent the classification loss function in step 1, step 2, step 3, represents the classification loss function of step 4.

[0064] The classification loss function is calculated by weighted kappa loss function, represented as:

[0065]

[0066]

[0067]

[0068] where N represents the number of samples, N m represents the number of samples of class m, Q represents the total number of classes, w m,n represents the quadratic weight matrix, where |m-n| represents the distance between the predicted class m and the actual class n, t k represents the true class of the k-th sample X k , P(q|X k ) represents the conditional probability that the predicted class of the k-th sample X k belongs to q;

[0069] The total loss function is obtained by weighted average of cross-entropy loss function and weighted kappa loss function:

[0070]

[0071]

[0072] where β is a hyperparameter, can represent the total loss function in step 1, step 2, step 3, represents the total loss function of step 4.

[0073] The workflow of the puzzle generation module is as follows:

[0074] Divide the original image I into K×K equal image blocks, and obtain an index matrix P of size j×j based on the index of the image blocks. I ;

[0075] Randomly shuffle image blocks and generate a shuffled image using a jigsaw puzzle generator, based on the image block index matrix P. I Obtain the index matrix P of the shuffled image S ;

[0076] Based on the index matrix P of the shuffled image S The magnitude of the isolated heat form is K. 2 ×j 2 The matrix P.

[0077] A computer device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, performs the steps of the method as described in any of the preceding claims.

[0078] A computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the method as described in any of the preceding claims.

[0079] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0080] 1. This invention provides a CNN and Transformer-based method for grading diabetic retinopathy. By using a feature fusion module to fuse multi-level features from a Residual Network (ResNet50) and a Swin-Transformer, it achieves interaction between global and local information. Feature fusion yields a more suitable receptive field, discovers more discriminative regions in fundus images, and achieves more accurate grading of diabetic retinopathy.

[0081] 2. This invention treats the classification of diabetic retinopathy as a multi-class classification problem with joint ordered regression by simultaneously utilizing classification loss function and weighted Kappa loss function. It obtains both class supervision information and ordered supervision information, enabling the model to achieve the current advanced level of classification performance. Attached Figure Description

[0082] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0083] Figure 2 This is a flowchart illustrating steps 1 to 4 in an embodiment of the present invention;

[0084] Figure 3 This is a schematic diagram of the feature fusion module in an embodiment of the present invention. Detailed Implementation

[0085] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0086] Referring to Figure 1 , the diabetes retinopathy grading method based on CNN and Transformer of the embodiment of the present application comprises the following steps:

[0087] S1: importing an original image dataset and dividing the original image dataset into an image training set and an image test set;

[0088] S2: processing each original image in the image training set using a puzzle generator to generate scrambled images of different granularity versions;

[0089] S3: establishing a diabetes retinopathy grading network through a convolutional neural network (ResNet50) and a Swin-Transformer, training the diabetes retinopathy grading network using a progressive strategy; inputting the scrambled images, outputting three intermediate stage feature spectra of different granularity sizes through the last three layers of the residual network ResNet50; inputting the original images, outputting three intermediate stage feature spectra of different sizes through the last three layers of the Swin-Transformer; fusing the feature spectra output by the convolutional neural network at different stages and the feature spectra output by the Swin-Transformer through a feature fusion module; concatenating the feature spectra output by the last three layers of the fusion of the residual network ResNet50 and the Swin-Transformer, obtaining a multi-scale and multi-granularity feature spectrum output by the concatenation stage;

[0090] S4: defining a classification loss function and a weighted kappa loss function, and constructing a loss layer according to the classification loss function and the weighted kappa loss function;

[0091] S5: optimizing the diabetes retinopathy grading network using a training sample set; testing the diabetes retinopathy grading network using a test sample set;

[0092] Further, in step S2, the specific steps are:

[0093] S21: dividing the original image I into KxK image blocks, obtaining an index matrix P of size jxj according to the index of the image block I ;

[0094] S22: Randomly shuffle image blocks, generate a shuffled image using a jigsaw puzzle generator, and then use the image block index matrix P... I Obtain the index matrix P of the shuffled image S ;

[0095] S23: Based on the index matrix P of the shuffled image S The magnitude of the isolated heat form is K. 2 ×j 2 The matrix P;

[0096] The beneficial effect is that the network is trained using images of different granularities at each step, and a progressive training strategy is adopted to obtain multi-scale and multi-granular local detail object features.

[0097] Furthermore, in step S3, the specific steps are as follows:

[0098] A grading network for diabetic retinopathy was established, including a residual network, a Swing-Transformer, a feature fusion module, convolutional layers, and a classification layer.

[0099] A progressive strategy was used to train the diabetic retinopathy grading network, which consisted of four steps.

[0100] ResNet50 residual network extracts image features from scrambled images. i The last three layers (a3, a4, a5) of the ResNet50 residual network are used to process the scrambled image. Specifically, the processing steps include: Step 1, using ResNet50 to output the feature spectrum a3 only at the third layer; Step 2, using ResNet50 to output the feature spectrum a4 only at the fourth layer; Step 3, using ResNet50 to output the feature spectrum a5 only at the fifth layer; and Step 4, simultaneously outputting the feature spectra a3, a4, and a5 from the third, fourth, and fifth layers of ResNet50. Different intermediate stage feature spectra are output for different granularity versions of the image. Let K = 2. 5-i+1 K×K represents the number of pieces in the input puzzle, where i={3,4,5}, which correspond to the feature spectra a3, a4, a5 of the three output stages respectively;

[0101] Swin-Transformer extracts image features from the original training set. l The last three layers (b2, b3, b4) of the Swin-Transformer are used to process the images of the original training set, and the corresponding outputs are the feature spectra b2, b3, b4 of the three stages.

[0102] like Figure 3As shown, the feature fusion module includes a channel attention, a fine-grained interaction module, and a spatial attention. In each step, the feature fusion module is used to fuse the corresponding residual network and the feature spectrum extracted by the Swin-Transformer, which is described in detail as follows:

[0103] The spatial attention includes a channel pooling layer, a 7x7 convolutional layer Conv, and a Sigmoid activation function. The spatial attention is used to enhance the CNN branch features:

[0104]

[0105] The beneficial effect is to enhance the local details of the CNN branch and suppress irrelevant areas.

[0106] The channel attention includes an average pooling layer, a fully connected layer, and a Sigmoid activation function. The channel attention is used to enhance the CNN branch features:

[0107]

[0108] The beneficial effect is to promote global information from the Swin-Transformer branch;

[0109] The fine-grained interaction module includes two 1x1 convolutional layers Conv1 and Conv2, Hadamard product, and a 3x3 convolutional layer Conv3. The fine-grained interaction module is used to realize feature interaction:

[0110] c i = Conv3(Conv1(a i )⊙Conv2(b l )),

[0111] The beneficial effect is to realize fine-grained interaction modeling between features from the two branches;

[0112] Finally, in order to realize the complementary relationship between the output features of different branches, the enhanced features and the interaction features of the two branches are fused together:

[0113]

[0114] The beneficial effect is to realize the complementary relationship between different output features;

[0115] In step 1, the feature fusion module is used to fuse the output feature spectrum a3 of the 3rd layer of the residual network and the output feature spectrum b2 of the 2nd layer of the Swin-Transformer, to obtain the fusion feature t3;

[0116] In step 2, the residual network 4th layer output feature spectrum a4 and the Swin-Transformer 3rd layer output feature spectrum b3 are fused by the feature fusion module to obtain the fusion feature t4;

[0117] In step 3, the residual network 5th layer output feature spectrum a5 and the Swin-Transformer 4th layer output feature spectrum b4 are fused by the feature fusion module to obtain the fusion feature t5;

[0118] Through the convolution layer The channel dimension of the fused feature spectrum is unified to 1024 dimensions:

[0119]

[0120] The fusion features t3, t4, t5 of steps 1, 2 and 3 are respectively obtained through the convolution layer to obtain the corresponding feature spectra X3, X4, X5;

[0121] In step 4, the original image is used as the input image of the residual network ResNet50 and the Swin-Transformer, and the feature spectra X3, X4, X5 fused by the residual network ResNet50 and the Swin-Transformer are concatenated to obtain the multi-scale and multi-granularity feature spectrum output in the concatenation stage

[0122] X concat = Concat (X3, X4, X5).

[0123] The classification layer is composed of two fully connected layers with Batchnorm and Elu nonlinearity. Diabetic retinopathy is divided into 5 grades in total, and the classification layer After processing the 1024-dimensional feature vector by the classifier, a 5-dimensional feature vector is obtained as the prediction probability distribution:

[0124]

[0125] Step 1, step 2, step 3 and step 4 prediction probability distribution V 3 , V 4 , V 5 , V concat .

[0126] The beneficial effect is to fuse the multi-level features of CNN and Swin-Transformer to realize the interaction of local information and global information, obtain a more suitable receptive field, and find more discriminative regions in each step. In order to explore the complementary relationship between the output features of different stages, the multi-scale feature spectra in the previous three steps are fused to enable the network to better learn the information of the complete object.

[0127] Further, in step S4, the specific steps are:

[0128] S41: In each step, the i-th stage uses V i represents the predicted probability distribution, V represents the true label of the image, and the classification loss function is calculated by cross-entropy loss:

[0129]

[0130]

[0131] Where n represents the image index, and the beneficial effect is to minimize the difference between the predicted probability distribution and the actual probability distribution using the inter-class competition mechanism, which is conducive to the network learning of the discriminative features of the fundus image.

[0132] S42: The weighted kappa loss function is:

[0133]

[0134]

[0135]

[0136] Where N represents the number of samples, N m represents the number of samples of class m, Q represents the total number of classes, w m,n represents the quadratic weighting matrix, where |m-n| represents the distance between the predicted class m and the actual class n, t k represents the true class of the k-th sample X k , and P(q|X k ) represents the conditional probability that the predicted class of the k-th sample X k belongs to q.

[0137] The beneficial effect is to establish a penalty mechanism, and the penalty degree is proportional to the square of the distance between the two ratings. This means that the farther the distance, the heavier the penalty. It is conducive to the network learning of the order information between different diabetic retinopathy severity.

[0138] S43: The cross-entropy loss function and the weighted kappa loss function are weighted and averaged to obtain the total loss function:

[0139]

[0140]

[0141] β is a hyperparameter, and the beneficial effect is to obtain class supervision information and order supervision information at the same time, and obtain more accurate classification results.

[0142] Further, in step S5, the specific steps are:

[0143] S51: using the training sample set, using the random gradient descent and back propagation algorithm based on automatic differentiation technology, optimizing the diabetic retinopathy grading network according to the total loss function;

[0144] S52: using the test sample set to test the diabetic retinopathy grading network based on the weight of the training set. The final prediction is the sum of the prediction probabilities of the above four steps. Through the operation of the four stages, the classification accuracy can be better.

[0145]

[0146] The beneficial effect is to optimize the network parameters by random gradient descent, back propagation algorithm and loss function, and then test the training set weight based on the test set.

[0147] A fine-grained visual classification system based on CNN and Transformer diabetic retinopathy grading method, characterized by: a puzzle generation module, a residual network module, a Swin-Transformer module, a feature fusion module, a loss function module and a training test module;

[0148] The puzzle generation module is used to process the image training data set and generate different granularity size versions of the scrambled image;

[0149] The residual network module uses the last three layers of ResNet50 for progressive training, uses images of different granularity sizes as input images in the first three steps, and selects different intermediate stage feature spectra outputs to obtain local detail information of the object; in step 4, the original image is used as the input image, and the residual network ResNet50 outputs the feature spectrum of the last three layers at the same time, obtaining multi-scale feature information;

[0150] The Swin-Transformer module uses the last three layers of Swin-Transformer for progressive training, uses the original image as the input image, selects different intermediate stage feature spectrum outputs in the first three steps to obtain global representation information of the object; in step 4, Swin-Transformer outputs the feature spectrum of the last three layers at the same time, obtaining multi-scale feature information;

[0151] The feature fusion module fuses the feature spectra of the corresponding stages of the residual network and Swin-Transformer in the first three steps, and outputs the fused features; in step 4, the fused feature spectra in the first three steps are concatenated to obtain the multi-scale and multi-granularity feature spectrum of the concatenated stage;

[0152] The loss function module is configured to define a classification loss function and a weighted kappa loss function, and to construct a loss layer according to the classification loss function and the weighted kappa loss function.

[0153] The training test module is configured to optimize the diabetic retinopathy grading network by using the training sample set, and test the diabetic retinopathy grading network by using the test sample set.

[0154] A memory, a processor and a program stored on the memory, wherein the processor implements the steps of the diabetic retinopathy grading method of CNN and Transformer when executing the program.

[0155] The embodiments of the present application test the proposed algorithm on two public datasets, DeepDRiD and IDRiD. The DeepDRiD dataset is a dataset containing 1600 diabetic retinopathy fundus images in total, which are divided into 5 categories, wherein the training set contains 1200 images, the test set contains 400 images, and the image resolution is 1736*1824. The IDRiD dataset is a dataset containing 516 diabetic retinopathy fundus images in total, which are divided into 5 categories, wherein the training set contains 413 images, the test set contains 103 images, and the image resolution is 4288*2848.

[0156] The present application compares the method with several fine-grained visual classification methods, including MC-Loss, P2P-Net, CAL, Swin-Transformer, PMG and CANet. It is worth noting that the method proposed in the present application uses ResNet50 and Swin-Transformer as feature extractors, uses an NVIDIA RTX A6000 GPU and an open source machine learning library Pytorch to realize the proposed method. For the two datasets, the same parameter settings are used, the input image is adjusted to a fixed size of 448*448, and then randomly cropped to a fixed size of 384*384 for subsequent processing. The training batch size is set to 16, and the stochastic gradient descent (SGD) is used as the optimizer. The momentum and weight decay are set to 0.9 and 0.0001 respectively, the initial learning rate of Swin-Transformer is set to 0.0001, the initial learning rate of ResNet50 is set to 0.0002, the learning rate of the newly added module is 0.002, and the number of iterations is 150. The hyperparameter β is set to 0.3 on the DeepDR dataset and 0.2 on the IDRiD dataset.

[0157] Accuracy (Acc) and quadratic weighted kappa (Kappa) were used as evaluation metrics. 1) Acc: the ratio of correctly predicted (positive samples) divided by the total number of predictions (positive samples plus negative samples). 2) Kappa: quadratic weighted Kappa was used, which generally ranges between 0 and 1. Kappa coefficient results are generally used to measure the consistency between two raters. The larger the Kappa value, the higher the degree of consistency.

[0158] As shown in the following table, the method proposed in the present application shows better performance compared with other state-of-the-art methods. The results show that the proposed CNN and Transformer-based diabetic retinopathy grading method can better find the discriminative regions of objects and achieve diabetic retinopathy grading. The proposed method performs better on the DeepDRiD and IDRiD datasets. As shown in Table 1, the method has superiority compared with other methods.

[0159] Table 1

[0160]

[0161]

[0162] Those skilled in the art will easily understand that the above description is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modification, equivalent replacement and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A CNN and Transformer based diabetic retinopathy grading method, characterized in that, The method comprises the following steps: S1: importing an original image dataset with multiple groups of original images, and dividing the original image dataset into an image training set and an image test set; S2: dividing each original image in the image training set into several image blocks, and generating shuffled images of different granularities; S3: establishing a diabetic retinopathy grading network through a residual network ResNet50 and a Swin-Transformer, and training the diabetic retinopathy grading network by using a progressive strategy; inputting the shuffled images, outputting three intermediate stage feature spectrums of different granularities through the last three layers of the residual network ResNet50; inputting the original images, outputting three intermediate stage feature spectrums of different sizes through the last three layers of the Swin-Transformer; fusing the feature spectrums output by the convolutional neural network at different stages and the feature spectrums output by the Swin-Transformer through a feature fusion module; concatenating the feature spectrums output by the last three layers of the residual network ResNet50 and the Swin-Transformer to obtain a multi-scale and multi-granularity feature spectrum output by the concatenation stage; S3 specifically comprises: establishing a diabetic retinopathy grading network, including a residual network, a Swin-Transformer, a feature fusion module, a convolutional layer and a classification layer; training the diabetic retinopathy grading network by using a progressive strategy; the training process specifically comprises, Step 1: fuse the feature spectrum output by the third layer of the residual network through the feature fusion module with the feature spectrum output by the second layer of the Swin-Transformer , to obtain first fused features ; Step 2: fuse the feature spectrum output by the fourth layer of the residual network through the feature fusion module with the feature spectrum output by the third layer of the Swin-Transformer to obtain second fused features ; Step 3: fusing the feature spectrum output by the 5th layer of the residual network through the feature fusion module with the feature spectrum output by the 4th layer of the Swin-Transformer to obtain third fused features ; Step 4: using the original image as the input image of the residual network ResNet50 and the Swin-Transformer, concatenating the feature spectrum of the residual network ResNet50 and the Swin-Transformer fusion , to obtain the multi-scale and multi-granularity feature spectrum output in the concatenation stage ; Fusion features of step 1, step 2 and step 3 Respectively through the convolution layer to get the corresponding feature spectrum ; Residual network ResNet50 extracts image features of the scrambled image , using the last three layers of the residual network ResNet50 respectively process the scrambled image, for different granularities of the input image, set , denotes the number of blocks in the input puzzle, wherein , correspond to the feature spectrum output by the three stages, respectively ; The Swin-Transformer extracts image features of the image training set , using the last three layers of the Swin-Transformer respectively process images of the image training set, respectively corresponding to output three-stage feature spectra ; The feature fusion module comprises a spatial attention module, a channel attention module and a fine-grained interaction module; in steps 1, 2, 3 and 4, the feature fusion module is used to fuse the feature spectrums extracted by the corresponding residual network and Swin-Transformer, and the specific description is as follows: The spatial attention module comprises a channel pooling layer, convolutional layer and a sigmoid activation function, and the CNN branch features are enhanced by the spatial attention, and the method is represented as: ； The channel attention module comprises an average pooling layer, a full connection layer and a Sigmoid activation function, and the method for enhancing the CNN branch features through channel attention is represented as: ； The fine-grained interaction module includes 2 convolution and Hadamard product and one convolution The feature interaction is realized through the fine-grained interaction module, and the method is represented as: c i =Conv3(Conv1(a i )⊙Conv2(b l )) The enhanced features of the two branches and the interaction features are fused together through a residual block to realize the complementary relationship between the output features of different branches, and the method is represented as: ； through the convolutional layers unify the channel dimension of the fused feature spectrum to 1024 dimensions: ； The classification layer consists of two fully connected layers with Batchnorm and Elu nonlinearity; diabetic retinopathy is classified into 5 grades, the classification layer After processing the 1024 dimensional feature vector by the classifier, a 5 dimensional feature vector is obtained as the predicted probability distribution, denoted as: ； wherein the predicted probability distribution of step 1, step 2, step 3 and step 4 are denoted as ; S4: defining a classification loss function and a weighted kappa loss function, and constructing a loss layer according to the classification loss function and the weighted kappa loss function; S4 specifically comprises: S41: In steps 1, 2 and 3, steps is used in step 4 denotes the predicted probability distribution, in step 4, using denotes the predicted probability distribution; using denotes the image true label, the classification loss function is calculated by cross-entropy loss, denoted as: ； wherein, denotes the image index, denotes the total number of images in a batch, may denote the classification loss function in step 1, step 2 and step 3, denotes the classification loss function in step 4; S42: calculating the classification loss function through the weighted kappa loss function, which is represented as: ， ， wherein, denotes the number of samples, denotes the class of the sample, denotes the total number of classes, denotes a quadratic weighting matrix, wherein denotes the predicted class and the actual class between the predicted class denotes the true class of the th sample denotes the predicted class of the th sample belongs to the class under the condition that the predicted class of the​ S43: obtaining a total loss function by weighted average of the cross-entropy loss function and the weighted kappa loss function: wherein, is a hyperparameter, may represent the total loss function in step 1, step 2 and step 3, represents the total loss function of step 4; S5: training the diabetic retinopathy grading network using the image training set to obtain training set weights; and testing the diabetic retinopathy grading network based on the training set weights using the test sample set.

2. The CNN and Transformer-based diabetic retinopathy grading method of claim 1, wherein, S2 specifically comprises: S21: divide the original image into image blocks of equal size ;​​​ S22: Randomly shuffle the image blocks, generate a shuffled image by a puzzle generator, and generate an index matrix of the image blocks according to the index matrix obtain an index matrix of the shuffled image ; S23: Obtain the one-hot form of the index matrix that disorganizes the image of size .​ 3. The CNN and Transformer-based diabetic retinopathy grading method of claim 1, wherein, S5 specifically comprises: S51: using the image training set, using the random gradient descent and back propagation algorithm based on automatic differentiation technology, and optimizing the diabetic retinopathy grading network according to the total loss function; S52: testing the diabetic retinopathy grading network based on the image training set weights using the image test set.

4. A system using a CNN and Transformer-based diabetic retinopathy grading method according to any one of claims 1-3, characterized in that, The method comprises: The puzzle generation module, the residual network module, the Swin-Transformer module, the feature fusion module, the loss function module, and the training test module; wherein The puzzle generation module is used for processing the image training dataset to generate disordered images of different granularity sizes. The residual network module is used for using the last three layers of ResNet50 for progressive training, using images of different granularity sizes as input images in steps 1 to 3, and selecting different intermediate stage feature spectrums to output, to obtain local detail information of the object; in step 4, the original image is used as the input image, and the residual network ResNet50 simultaneously outputs the feature spectrums of the last three layers, to obtain multi-scale feature information. The Swin-Transformer module is used for using the last three layers of Swin-Transformer for progressive training, using the original image as the input image, and selecting different intermediate stage feature spectrums to output in steps 1 to 3, to obtain global representation information of the object; in step 4, the Swin-Transformer simultaneously outputs the feature spectrums of the last three layers, to obtain multi-scale feature information. The feature fusion module is used for fusing the feature spectrums of the corresponding stages of the residual network and the Swin-Transformer in steps 1 to 3, to output fused features; in step 4, the fused feature spectrums in steps 1 to 3 are concatenated, to obtain a concatenated stage multi-scale multi-granularity feature spectrum. The loss function module is used for defining a classification loss function and a weighted kappa loss function, and constructing a loss layer according to the classification loss function and the weighted kappa loss function. The training test module is used for optimizing the diabetic retinopathy grading network by using the image training set, and testing the diabetic retinopathy grading network by using the image test set.

5. The system of claim 4, wherein, The working process of the loss function module is as follows: In step 1, step 2, step 3, step used in step denotes the predicted probability distribution, in step 4, using denotes the predicted probability distribution, in step used in step denotes the predicted probability distribution, using denotes the image true label, the classification loss function is calculated by cross-entropy loss, denoted as: ； wherein, denotes the image index, denotes the total number of images in a batch, may denote the classification loss function in step 1, step 2, step 3, denotes the classification loss function in step 4; The classification loss function is calculated by the weighted kappa loss function, which is represented as: ， ， wherein denotes the number of samples, denotes the class of the sample, denotes the total number of classes, denotes a quadratic weighting matrix, wherein denotes the distance between the predicted class and the actual class , denotes the true class of the th sample , denotes the conditional probability that the predicted class of the th sample belongs to the class . The total loss function is obtained by weighted average of the cross-entropy loss function and the weighted kappa loss function: wherein, is a hyperparameter, may represent the total loss function in step 1, step 2, step 3, represents the total loss function of step 4.

6. The system of claim 4, wherein, The working process of the puzzle generation module is as follows: The original image is divided into image blocks of equal size, and an index matrix of size is obtained according to the index of the image block. ​​​ randomly shuffling the image tiles, generating a scrambled image by a puzzle generator, generating an index matrix of the image tiles obtaining an index matrix of the scrambled image ; According to the index matrix that scrambles the image a matrix of size in one-hot form is obtained.

7. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the method of any one of claims 1-3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to realize the steps of the method of any one of claims 1-3.

Images