Image grading method based on dual-path network

Abstract

The application provides a kind of image grading method based on double path network, solve the diversity of data set in the training process based on deep learning network is often ignored, thereby influence the generalization of model;Further cause fundus picture grading error problem, a kind of image grading method based on double path network, the accuracy of the image grading method exceeds 83%, Kappa exceeds 0.76;The double path network is: SE-CoTNet-152 and Xception;The image grading method includes: step one: preprocessing;Step two: data resampling;Step three: double path network model establishment;Step four: test data set enhancement;Step five: classification result evaluation;Step six: double path network model training.

Description

TECHNICAL FIELD

[0001] The application relates to the field of fundus retina image processing, and in particular to an image grading method based on a double-path network. BACKGROUND

[0002] Diabetic retinopathy (DR) is an eye disease associated with diabetes and is the leading cause of blindness and visual impairment; according to the characteristics and number of lesions, DR is divided into five grades; the probability of blindness of patients increases with the grading of diabetic retinopathy; therefore, accurate DR grading is very necessary for early diagnosis and treatment of patients.

[0003] DR can be detected at an early stage by regular fundus examination, thereby reducing the risk of blindness. However, an increasing number of patients and cumbersome diagnosis processes increase the difficulty of real-time manual analysis and also increase the error rate of diagnosis; computer-aided diagnosis technology can reduce the burden of ophthalmologists, speed up the diagnosis, and enable patients to learn about their condition in a timely manner, so that appropriate treatment plans can be developed to avoid missing the best treatment time.

[0004] However, due to the complexity of the task, its actual clinical application is still challenging; it is difficult to make detailed grading due to the similarity of fundus retina images in color and texture; the lesion location only occupies a very small area in the entire image and is easily overlooked; the reason is that the current deep learning-based network often ignores the diversity of the data set during the training process, thereby affecting the generalization of the model. SUMMARY

[0005] The application provides an image grading method based on a double-path network, which solves the problem that the deep learning-based network often ignores the diversity of the data set during the training process, thereby affecting the generalization of the model, and further causing errors in fundus image grading.

[0006] An image grading method based on a double-path network, the accuracy of the image grading method is more than 83%, and the Kappa is more than 0.76; the double-path network is SE-CoTNet-152 and Xception; the image grading method comprises the following steps:

[0007] Step 1: preprocessing, selecting pictures in a public database, cropping the target area, and normalizing the color and brightness by Gaussian blur;

[0008] Step 2: resampling the data, resampling the data in the public database by using the following formula,

[0009] w t =r i-1 w0+(1-r i-1 )wf ,

[0010] wherein, W0 is the initial resampling weight based on the proportion of the number of fundus images of each grade, r = 0.95, W f = (1, 2, 1.5, 2, 2) ;

[0011] Step three: the double-path network model is established, the CoT module is used to replace the 3*3 convolution in the SE-ResNet-152 to form the SE-CoTNet-152, the separable convolution is used to replace the convolution in the Inception-V3 to form the Xception, and the result is obtained through the global average pool and the FC layer;

[0012] Step four: the test data set is enhanced, the TAA is used for enhancement, each image is rotated by 90°, 180° and 270°, the original image and the rotated image are input into the double-path network for prediction, and the average value of the output result is taken to obtain the classification result;

[0013] Step five: the classification result is evaluated, and the accuracy, the secondary weighted score and the specificity are used as the indexes to evaluate the classification result;

[0014] Step six: the double-path network model is trained, the image input resolution is 320*320, the enhancement or the dropout method is used to avoid overfitting, the initial learning rate is 5*10-5, and when the performance of the verification set fails to improve after 3 rounds, the learning rate is decayed by 0.2 times.

[0015] Further, the method for cutting the target region in step one is that the pixel intensity of each column and each row is scanned.

[0016] Further, the CoT module uses k*k group convolution on the input two-dimensional matrix X, denoted as K 1 , X is converted into Key Maps, Value Maps and Query Maps, wherein the Key Maps are defined as K = X, the Query Maps are defined as Q = X, and the Value Maps are defined as V = XW v , W v is a convolution matrix; the matrix A is calculated through the following formula:

[0017] A = [K 1 , Q]W θ W δ

[0018] wherein, W θ is a 1*1 convolution with a ReLU activation function, and W δ is a 1*1 convolution without an activation function;

[0019] Dynamic feature map K 2 The formula is obtained as follows:

[0020]

[0021] Further, the Xception comprises: a convolution block 1, a convolution block 2 and a convolution block 3.

[0022] Further, the convolution block 1 comprises, in sequence: a separable convolution layer, a rectified linear unit, a separable convolution layer, a maximum pool layer; the convolution block 2 comprises, in sequence: a rectified linear unit, a separable convolution layer, a rectified linear unit, a separable convolution layer, a maximum pool; and the convolution block 3 comprises: three consecutive combinations of a rectified linear unit and a separable convolution layer.

[0023] Further, the formula for calculating the accuracy and specificity in step five is:

[0024]

[0025] Wherein, Acc is the accuracy, Spe is the specificity, TP is the number of correctly identified images, TN is the number of correctly identified remaining images, FP is the number of incorrectly identified images, and FN is the number of incorrectly identified remaining images.

[0026] Further, the method for enhancing the avoidance of overfitting in step six comprises: affine transformation, rotation, color jittering, horizontal flipping and vertical flipping; and the dropout parameter for avoiding overfitting is 0.5.

[0027] Further, the public database is: DDR, APTOS2019 and EyePACS.

[0028] Further, the accuracy and Kappa values of the DDR, APTOS2019 and EyePACS data sets are: 83.10% and 0.8496; 84.18% and 0.9000; and 84.10% and 0.7684.

[0029] The beneficial effects achieved by the present application are:

[0030] 1) An improved dual-path network (CoT-XNet) is proposed for DR grading, which can accurately grade DR in fundus images and has good generalization ability;

[0031] 2) The proposed CoT-XNet shows better performance than the existing SOTA model, with accuracy and Kappa of 83.10% and 0.8496, 84.18% and 0.9000, and 84.10% and 0.7684 in three data sets;

[0032] 3) In most images, the class activation maps of CoT and Xception networks are different and complementary. By concatenating the different visual representations learned by CoT and Xception networks, CoT-XNet can accurately grade DR in fundus images and show good universality. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of the provided drawings

[0034] Figure 1 is the overall flowchart of training and testing CoT-XNet for DR grading;

[0035] Figure 2 is the image preprocessing process diagram;

[0036] Figure 3 is the comparison of CoT module and self-attention block in visual backbone;

[0037] Figure 4 is a schematic diagram of an "extreme" version of an Inception module;

[0038] Figure 5 is a structural diagram of Xception. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0040] Embodiment 1: An image grading method based on a dual-path network.

[0041] This embodiment uses three public fundus image datasets (DDR, APTOS2019 and EyePACS). The datasets are used to train and evaluate the model proposed in this embodiment.

[0042] The following table shows the data distribution of the three fundus image datasets:

[0043]

[0044]

[0045] The overall flow of the dual-path network (CoT-XNet) is as follows:

[0046] The training and testing are included, the training part includes two steps: preprocessing and network training; the testing part includes two steps: preprocessing and TTA, as shown in Figure 1 .

[0047] The purpose of data preprocessing is to reduce information redundancy, because the fundus images in the dataset come from different regions, people and devices, and the brightness, spatial resolution and other aspects of the images are different.

[0048] Step one, preprocessing,

[0049] S1: crop the target area, there are irrelevant black areas around the target area, and the irrelevant black areas are removed by scanning the pixel intensity of each column and each row;

[0050] S2: color and brightness normalization, use Gaussian blur to unify the color and brightness of the image.

[0051] As shown in the accompanying Figure 2 , wherein (a) is the original image, (b) is the cropped image; (c) is the image after color and brightness normalization.

[0052] Step two, data resampling,

[0053] The purpose of data resampling is to solve the class imbalance problem in the dataset, in the ith epoch, the resampling weight of 0 to 4 levels is given by the following formula:

[0054] w t =r i-1 w0+(1-r i-1 )w f (1)

[0055] Wherein, W0 is the initial resampling weight based on the proportion of the number of fundus images of each level, r is the coefficient, r = 0.95, W f =(1, 2, 1.5, 2, 2).

[0056] Step three, dual-path network (CoT-XNet) establishment

[0057] CoT-XNet consists of two parts: SE-CoTNet-152 and Xception, wherein SE-CoTNetD-152 is an improved network framework based on SE-ResNet-152, which retains each stage 1x1 convolution and replaces the 3x3 convolution with a CoT module; the architectural differences are shown in the following table, and SE-CoTNet-152 and Xception obtain 2048-dimensional features, respectively. The features of the two networks are concatenated to form 4096-dimensional features, and all the features are sent to the global average pooling and FC layer for final prediction.

[0058]

[0059]

[0060] Step 3-1: SE-CoTNet-152

[0061] The CoT module is not used in the self-attention module in the visual backbone network, as shown in the attached Figure 3 , assuming a two-dimensional matrix is converted into Key Maps K, Value Maps V and Query Maps Q through convolution matrix W k , W v , W q :

[0062] [K, V, Q] = X · [W k , W v , W q ] (2)

[0063] wherein the matrix size of K, V and Q is HxWxC k , and the local relationship matrix R is obtained, which is expressed by the following formula:

[0064]

[0065] wherein, is represented as a local matrix multiplication operation, is a local relationship matrix, C h is the number of heads.

[0066] The position information is added to Q, and the result is added to the relationship matrix to obtain:

[0067]

[0068] The softmax operation is performed on each head of along the channel dimension to obtain the attention map A, that is:

[0069]

[0070] Y is the concatenation of all the heads' aggregated feature maps, denoted as:

[0071]

[0072] In the traditional self-attention module mechanism, all query-key relationships are learned by independent query-key pairs, without exploring the rich context between them, which greatly limits the learning of visual representation. The CoT module aggregates the mining of context information and the learning of self-attention.

[0073] In Figure 4 (b) figure, assuming that the same input two-dimensional matrix X is converted into Key Maps, Value Maps and Query Maps, Key Maps is defined as K = X; Value Maps is defined as V = XW v ; Query Maps is defined as Q = X.

[0074] The CoT module uses k x k group convolution on the input X to obtain context-related information, denoted as static feature K 1 ; then concatenates the key K 1 and query- of context.

[0075] The matrix A is realized by first concatenating the results of K 1 and query- of context, and then convolving with a 1 x 1 convolution containing a ReLU activation function and a 1 x 1 convolution without an activation function, denoted as:

[0076] A = [K 1 , Q]W θ W δ (7)

[0077] In each head, the local attention matrix A of each spatial position is realized by combining the query feature and the key feature of the context, rather than based on the individual query-key feature pair.

[0078] The dynamic feature map K 2 is obtained by multiplying A and V:

[0079]

[0080] Step 3-2: Xception

[0081] Improvements are made on the basis of the Inception-V3 network, using the "extreme" version, as shown in the attached Figure 4As shown, first use 1x1 convolution to map the cross-channel correlation, and then use separable 3x3 convolution for each channel of the feature map, and all channel features are concatenated.

[0082] The overall structure diagram of Xception is as shown in Figure 1 The details of each module are as shown in Figure 5 Figure 5 In (a), Block1 includes four layers: a separable Conv layer, a ReLU (Rectified Linear Unit), a separable Conv layer, and a max pool layer, and a residual connection with a 1x1 convolution kernel is added in the network layer; in (b), Block2 is composed of five convolution layers, including a ReLU, a separable Conv layer, a ReLU, a separable Conv layer, and a max pooling layer, and a residual connection with a 1x1 convolution kernel is added in the network layer; in (c), Block3 includes six convolution layers, including three consecutive combinations of ReLU and Separable Conv layers, and a residual connection is adopted in all blocks.

[0083] Step four: test data set enhancement

[0084] TTA is used to enhance the test data set. Specifically, each fundus image is rotated by 90°, 180°, and 270°, and the original image and the flipped image are input into Cot-XNeT for prediction, and the average of the output results is taken to obtain the final classification result of the image.

[0085] Evaluation index:

[0086] This embodiment uses three evaluation indexes to evaluate the performance of CoT-XNet model and comparative model in fundus image classification, including accuracy (Acc), secondary weighted Kappa score, and specificity (Spe).

[0087]

[0088]

[0089] Among them, true positive (TP) is the number of correctly identified images, true negative (TN) is the number of correctly identified other images, false positive (FP) is the number of incorrectly identified images, and false negative (FN) is the number of incorrectly identified other images.

[0090] Step five: training and evaluation

[0091] ​The weights of the CoT-XNet model (pre-trained on ImageNet) were obtained from the PyTorch image model library, the input resolution of all images of the trained model was 320x320, and an enhanced method was used to avoid overfitting, including affine transformation, rotation, color jitter, horizontal flip and vertical flip; In addition, the method of dropout (inactivating some neurons by random probability) was also used, and the dropout parameter was set to 0.5.

[0092] The initial learning rate of the training was 5x10 -5 When the performance of the validation set failed to improve after about 3 rounds, the learning rate would be decayed by 0.2 times.

[0093] 30 epochs were trained using the ACProp optimizer, with an eps of 1e-8, betas of [0.9, 0.999], and a weight decay of 1e-5, a cross-entropy loss function as the loss function, the proposed algorithm was implemented by Python 3.6 and Pytorch1.9.1 deep learning library, the workstation was equipped with AMD Ryzen 9 5950x CPU@3.40GHz, memory 128GB and two NVIDIA RTX 3080Ti GPU, memory 12GB.

[0094] After testing, CoT-XNet can accurately grade DR in fundus images and has good generalization ability, the proposed CoT-XNet shows better performance than existing SOTA models, in three data sets, the accuracy and Kappa are 83.10% and 0.8496, 84.18% and 0.9000, and 84.10% and 0.7684, respectively, in most images, the class activation maps of CoT and Xception networks are different and complementary, by concatenating the CoT and Xception networks to learn different visual representations, CoT-XNet can accurately grade DR in fundus images and show good universality.

Claims

1. An image grading method based on a dual-path network, characterized in that: The image grading method has an accuracy of over 83% and a Kappa of over 0.76; The dual-path network is: SE-CoTNet-152 and Xception; the image grading method includes: Step 1: Preprocessing: Select an image from a public database, crop the target area, and normalize the color and brightness using Gaussian blur; Step 2: Data resampling: Use the following formula to resample the data in the public database: w t =r i-1 w0+(1-r i-1 )w f , Where W0 is the initial resampling weight based on the ratio of fundus images at each level, r = 0.95, W f =(1, 2, 1.5, 2, 2); Step 3: Build a dual-path network model. Use the CoT module to replace the 3×3 convolution in SE-ResNet-152 to form SE-CoTNet-152; use separable convolution to replace the convolution in Inception-V3 to form Xception; and obtain the result through global average pooling and FC layers. Step 4: Enhance the test dataset by performing TAA. Each image is rotated 90°, 180°, and 270°. The original image and the rotated image are input into the dual-path network for prediction. The output results are averaged to obtain the classification result. Step 5: Classification result evaluation, using three indicators: accuracy, quadratic weighted score, and specificity to evaluate the classification results; Step 6: Dual-path network model training, with an image input resolution of 320×320, and overfitting avoided by enhancement or dropout. The initial learning rate is 5×100-5. When the performance of the validation set fails to improve after 3 rounds, the learning rate will be decayed by 0.2 times.

2. The image grading method based on a dual-path network according to claim 1, characterized in that: The method of cropping the target area in step 1 is: by scanning the pixel intensity of each column and each row.

3. The image grading method based on a dual-path network according to claim 1, characterized in that: The CoT module uses k×k group convolution on the input two-dimensional matrix X, denoted by K 1 , X is converted into Key Maps, Value Maps, and Query Maps, where Key Maps is defined as K = X, Query Maps is defined as Q = X, and Value Maps is defined as V = XW v , W v is the convolution matrix; the matrix A is calculated by the following formula: A=[K 1 ,Q]W θ W δ Among them, W θ is a 1×1 convolution with ReLU activation function, W δ It is a 1×1 convolution without activation function; Dynamic feature map K 2 Obtained by the following formula:

4. The image grading method based on a dual-path network according to claim 1, characterized in that: The Xception includes: convolution block 1, convolution block 2 and convolution block 3.

5. The image grading method based on a dual-path network according to claim 4, characterized in that: The convolution block 1 includes, in sequence: a separable convolution layer, a rectified linear unit, a separable convolution layer, and a maximum pooling layer; the convolution block 2 includes, in sequence: a rectified linear unit, a separable convolution layer, a rectified linear unit, a separable convolution layer, and a maximum pooling layer; the convolution block 3 includes: three consecutive combinations of a rectified linear unit and a separable convolution layer.

6. The image grading method based on a dual-path network according to claim 1, characterized in that: The calculation formulas for accuracy and specificity in step 5 are: Where Acc is accuracy, Spe is specificity, TP is the number of correctly identified images, TN is the number of correctly identified remaining images, FP is the number of falsely identified images, and FN is the number of falsely identified remaining images.

7. The image grading method based on a dual-path network according to claim 1, characterized in that: The enhanced methods for avoiding overfitting in step 6 include: affine transformation, rotation, color jitter, horizontal flip and vertical flip; the dropout parameter for avoiding overfitting is 0.

5.

8. The image grading method based on a dual-path network according to claim 1, characterized in that: The public databases are: DDR, APTOS2019 and EyePACS.

9. The image grading method based on a dual-path network according to claim 8, characterized in that: The accuracy and Kappa values ​​of the DDR, APTOS2019, and EyePACS datasets are 83.10% and 0.8496, respectively; 84.18% and 0.9000; 84.10% and 0.7684.

Images