A Diabetic Retinopathy Grading Method Based on Multi-Scale Fusion Attention Mechanism

Abstract

A method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism belongs to the field of computer software. Aiming at the disadvantages that it is currently impossible to extract lesion features of different scales and it is difficult to focus on important lesions, resulting in low accuracy of the grading task. First, filters with different convolutional kernel sizes are used to extract features from fundus images, and different weights are assigned to features of different scales, so that the deep learning network can learn and select features of different scales to improve the network's recognition ability for different types and sizes of lesions. At the same time, features of different scales are fused, and an attention mechanism that fuses spatial and channel information is used to assign weights to the fused features to strengthen the selection of high-value features, thereby suppressing other useless features, so as to improve the grading accuracy. The present invention can effectively improve the grading accuracy of diabetic retinopathy.

Description

Technical Field

[0001] The present invention belongs to the field of computer software, and particularly relates to a method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism. Background Art

[0002] Diabetic retinopathy is a common disease, which includes microaneurysms, hemorrhages, soft and hard exudates. According to the number, size and other characteristics of the lesions, diabetic retinopathy can be divided into five grades, including grade 0 - no lesion, grade 1 - mild lesion, grade 2 - moderate lesion, grade 3 - severe lesion and grade 4 - proliferative lesion. The automatic grading of diabetic retinopathy has always been a hot research topic in the field of deep learning. However, the existing methods have the following deficiencies: First, they cannot extract lesion information at different scales; Second, they cannot pay more attention to important lesion areas. This makes the accuracy and robustness of the current deep learning-based diabetic retinopathy grading methods not high. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to propose a method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism. This method can not only extract lesion information from multiple scales, but also pay attention to important lesions, thereby improving the grading accuracy and robustness.

[0004] To achieve the above invention purpose, through research discussions and repeated practices, this method is finally determined to include the following steps:

[0005] Step 1. Construct a training set and preprocess the images therein;

[0006] Step 2. Construct a convolutional module ISK-Block: Write a convolutional layer module based on a multi-scale fusion attention mechanism using the Pytorch framework, in which convolutional kernels of different sizes are used to extract multi-scale features of the picture and fuse them into a new feature map, and then the attention weight matrices of the fusion spatial and channel attention corresponding to the multi-scale feature maps are extracted through an improved fusion attention mechanism. Finally, the weights of different scale features are calculated through a multi-scale selection framework, and the multi-scale feature maps are fused according to the weights in combination with the attention weight matrices to obtain the final fused feature map.

[0007] Step 3. Construct a network: Connect different numbers of ISK-Block modules in series and splice a fully connected layer to map to 5 categories to form a complete network structure. Through experiments, the optional numbers include 18, 50, 101.

[0008] Step 4. Train the model: Input the data processed in Step 1 into the network constructed in Step 3 for training.

[0009] Step 5. Use the trained model to discriminate the test set to obtain the classification results of diabetic retinopathy. Description of the Drawings

[0010] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below.

[0011] Figure 1 Flowchart of the diabetic retinopathy grading method based on the multi-scale fusion attention mechanism

[0012] Figure 2 Structural diagram of the convolutional module ISK-Block based on the multi-scale fusion attention mechanism Detailed Implementation Manner

[0013] Aiming at the problems in the diabetic retinopathy grading method that multi-scale features cannot be extracted and important lesions cannot be focused on, resulting in low accuracy, the present invention combines multi-scale feature extraction with an improved fusion attention mechanism, and proposes a diabetic retinopathy grading method based on the multi-scale fusion attention mechanism.

[0014] The key point in the present invention is the improved convolutional neural network module ISK-Block. This module first performs convolution operations on the same feature map using convolutional kernels of different sizes. Different convolutional kernel sizes mean different receptive fields, that is, different scales, and the feature maps of different scales extracted are fused. Then, an improved fusion attention mechanism is used to calculate the attention for the fused new feature map. The improved fusion attention mechanism can aggregate the attention in both channels and space at the same time, calculate the channel and spatial positions of important features, give different weights, and use this weight to re-weight the feature maps of different scales. Finally, a Softmax operation is performed on the weighted feature maps of different scales, giving different weights to the feature maps of different scales, and fusing the feature maps of different scales into a new feature map according to this weight.

[0015] ISK-Block can easily form a new network. Specifically, 18, 50, or 101 ISK-Blocks can be used in series, and a fully connected layer is spliced to map the network output to multiple categories, constituting a complete network structure with different depths.

[0016] Figure 1 The several steps of the present invention can be described and are specifically described as follows.

[0017] Step 1: To better train the network and improve its performance and efficiency, it is necessary to preprocess the input fundus images. First, remove the black background in the images to prevent it from affecting the network's learning process. Then, randomly crop all the images to a size of 224 pixels * 224 pixels. Finally, to improve the network's computational efficiency, the images also need to be converted into single-channel grayscale images.

[0018] Step 2: Use the Pytorch framework to construct the ISK-Block. Figure 2 This is the network structure of the ISK-Block.

[0019] (1) Let the tensor representing the feature map with the number of channels C, length H, and width W be be the output of any layer of the network, denoted as First, perform multi-scale feature extraction on using 3*3 and 5*5 convolutional kernels to perform convolution operations on , that is, extract features from two different scales. Denote the two scales as scale a and scale b, and the extracted feature maps as U a , U b . Combine U a , U b by matrix addition to fuse into a new feature map U. At this time, U contains feature information at multiple scales. The calculation method is as follows:

[0020] U = U a + U b

[0021] (2) Then, perform fused feature extraction on U. First, perform fused feature extraction on U in different channel dimensions along the height H direction and width W direction of the feature map. The formula is as follows:

[0022]

[0023]

[0024] where are tensors representing the fused features at the h height and w width of the feature map in channel c respectively. ap is the average pooling operation, and F ap represents the mapping corresponding to the average pooling. represent the vectors composed of all pixel values at the h height and w width of the fused feature map U in channel c respectively. U c (i,j) represents the pixel value at position (i,j) of the fused feature map U in channel c. S H is the tensor representing the fused information in the H direction of all channels, with a size of C*H, and S WThe tensor represents the fused information in the W direction across all channels, and its tensor size is C*W.

[0025] (3) For the convenience of subsequent calculations, S H and S W are concatenated to obtain the fused feature tensor S that combines the fused position information and channel information. The formula is as follows:

[0026] S = [S H , S W , S ∈ R C,N , N = H + W

[0027] where [,] represents the concatenation operation, S is a tensor of C*N, and N is the sum of H and W.

[0028] (4) Next, attention calculation is performed on S. Specifically, first, a fully connected layer fc is used to reduce the dimension of S from C*N to d*N dimensions. The tensor after dimension reduction is denoted as Z, where Z ∈ R d,N , and d is a variable parameter used to control the dimension of Z. d = max(C / 16, 32), and max represents the operation of taking the maximum value. The dimension reduction process can be expressed by the following formula:

[0029]

[0030] where represents the mapping relationship corresponding to the fully connected layer fc, represents the parameter matrix of the fully connected layer fc, represents the batch normalization operation (Batch Normalize), and δ represents the non-linear operation ReLu. Reducing the dimension of S from C*N to d*N dimensions can improve the calculation efficiency without losing too much accuracy.

[0031] (5) The attention that combines the fused position information and channel information is mainly guided by Z. Specifically, for the feature map U a extracted from the a scale, first, a convolutional layer composed of 1*1 convolutional kernels is used to re-expand Z into a tensor of C*N dimensions and activate it through the Sigmod activation function to obtain the attention weight matrix ψ a . The formula is as follows:

[0032]

[0033] where ψ a ∈ R C,N , represents the mapping relationship corresponding to the 1*1 convolutional layer corresponding to U a , W a represents the parameter matrix of the 1*1 convolutional layer, Wa ∈R d,N,C,N , where σ represents the Sigmoid activation function.

[0034] In order to calculate the weights on the H height and W width separately in the following, ψ a needs to be split into two parts and which respectively represent the weight matrices in the H direction and W direction of the feature map U a on different channels, where it describes the importance of the features of the feature map U a in the H direction or W direction on different channels.

[0035] For the feature map at scale b, its attention weight matrix ψ b is calculated as follows:

[0036]

[0037] Similar to the feature map U a , denotes the mapping relationship corresponding to the 1*1 convolutional layer of U b . The weight matrices in the H direction and W direction of the feature map U b on different channels are respectively split from ψ b ∈R C,N .

[0038] (6) Finally, in order to give different weights to the features at different scales, Softmax is used to calculate the proportion of the features at different scales, and the final weight matrix is obtained. The weights corresponding to the h height and w height of scale a and scale b on channel c are respectively and and the calculation method is as follows:

[0039]

[0040]

[0041] where exp represents the exponential function with the natural constant e as the base, denotes the tensor the value of the h-th row and c-th column, denotes the tensor the value of the w-th row and c-th column, denotes the tensor the value of the h-th row and c-th column, denotes the tensor the value of the w-th row and c-th column.

[0042] Use respectively represent the corresponding ones at all heights and widths of scale a on channel c that compose the final weight vector, denoted by respectively represent the corresponding ones at all heights and widths of scale b on channel c that compose the final weight vector, where they describe the importance of the features of the feature maps at different scales in different directions on channel c.

[0043] (7) Finally, use the finally obtained weight matrix to weight and fuse the feature maps of different scales into the output feature map V. Specifically, for the c-channel of V, its calculation method is as follows:

[0044]

[0045] where represents the value of any pixel point on channel c of the feature map U a 、U b , the value of any pixel point on channel c, represents the value of the i-th element of the weight vector , represents the value of the j-th element of the weight vector .

[0046] Step 3: Use different numbers of ISK-Blocks to stitch into a complete network structure. After experiments, the effective numbers are 18, 50, and 101. Finally, use the fully connected layer to map the finally output feature map to the five grades of diabetic retinopathy.

[0047] Step 4: Use the cross-entropy function as the loss function, use Adam with a fast convergence rate as the optimizer, with a learning rate of 1*10-3 and a batch size of 64, and randomly assign values to the parameters in the network. Optimize the network through the early stopping method. When the loss of the network on the validation set tends to be flat, stop training and save the network parameters. Specifically, when the training is in 50 iterations and the loss of the network on the validation set fluctuates up and down by no more than 5%, it is considered to tend to be flat.

[0048] Step 5: Use the trained model to predict the test set images.

[0049] In summary, the diabetic retinopathy grading method proposed by the present invention can improve the extraction ability of lesions at different scales in the retinopathy area and pay attention to important lesions, thereby improving the accuracy of network grading.

[0050] The following table shows the global accuracy of the grading method proposed by the present invention on the public datasets Kaggle Eyepacs and Messidor-1, and its accuracy exceeds that of the methods proposed by predecessors.

[0051]

[0052] Although embodiments of the present invention have been shown and described, it will be understood by those of ordinary skill in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present invention, and the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism, characterized in that: It includes the following steps: S1. Obtain a data set, download the publicly available Kaggle EyePacs data set, which contains fundus images of different degrees of diabetic retinopathy; S2. Construct a training set, and divide the data set into a training set and a validation set according to a ratio of 8:2; S3. Perform preprocessing operations on the images. First, remove the black background in the fundus pictures, then randomly crop all pictures into 224 pixels * 224 pixels in size, and finally convert all pictures into single-channel grayscale images; S4. For the multi-level classification task of diabetic retinopathy, establish an improved multi-scale fusion attention module ISK-Block; The improved multi-scale fusion attention module ISK-Block (hereinafter referred to as the ISK-Block module) first performs multi-scale feature extraction on the feature map with an input channel number of C, height of H, and width of W. Convolution operations are performed on the feature map using 3*3 and 5*5 convolution kernels respectively, and then the feature maps of different scales obtained after convolution are added matrix-wise to obtain a fused feature map; then, the features in the height and width directions of the fused feature map are fused and extracted using average pooling operations, and the extracted feature tensors are concatenated into a fused feature tensor that fuses channel and spatial features; then, two convolutional layers are used to calculate the attention on the fused feature tensor to obtain the attention weight matrices corresponding to different scales, and they are respectively split into the attention weight matrices corresponding to the height H direction and the width W direction of different channels; Finally, the Softmax operation is performed element-wise on the attention weight matrices corresponding to the two different scales to obtain the weight matrices corresponding to the features at different scales, and the weight matrices are used to weight the features at different scales and fuse them into the final output feature map; S5. Use different numbers of ISK-Blocks to construct a complete deep convolutional neural network, and concatenate a fully connected layer to map the feature map output by the network to the five grades of diabetic retinopathy; S6. Use the constructed deep convolutional neural network to train the network model on the training set. During training, adjust the parameters of the deep convolutional neural network according to the set learning rate, so as to obtain a network model for the multi-level classification task of diabetic retinopathy.

2. The diabetic retinopathy grading method based on a multi-scale fusion attention mechanism according to claim 1, wherein The fundus image data set obtained in step S1 is divided into five grades according to the severity of the patient's diabetic retinopathy: grade 0 - no lesion, grade 1 - mild lesion, grade 2 - moderate lesion, grade 3 - severe lesion, and grade 4 - proliferative lesion.

3. A method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism according to claim 1, characterized in that, In step S2, before dividing the data set according to a ratio of 8:2, it is necessary to perform flipping and rotation operations on the images in the data set.

4. A method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism according to claim 1, characterized in that, In the ISK-Block module, the feature maps extracted at different scales are fused. It is characterized in that for the feature map U extracted at scale a a and the feature map U extracted at scale b b , the calculation method of the fused feature map U is as follows: U = U a +U b 。 5. A grading method for diabetic retinopathy based on a multi-scale fusion attention mechanism according to claim 1, characterized in that, In the ISK-Block module, two convolutional layers are used to perform attention calculation on the fused feature vector. The feature is that for the fused feature vector S, a complete convolutional layer including a fully connected layer, a batch normalization operation, and a nonlinear ReLu operation is first used to reduce S from the C*(H+W) dimension to the d*(H+W) dimension Z, where d is a variable parameter used to control the dimension of Z, d=max(C / 16,32), and max represents the maximum number operation; the formula is expressed as: where Z ∈ R d,H+W , represents the mapping relationship corresponding to the fully connected layer, represents the parameter matrix of the fully connected layer, represents the batch normalization operation (Batch Normalize), and δ represents the non-linear operation ReLu; Then, for different scales, a convolution layer composed of 1*1 convolution kernel is used to upgrade Z from d*(H+W) dimension to C*(H+W) dimension, and activated by Sigmoid activation function to obtain the attention weight matrix ψ corresponding to the scale. The calculation formula is as follows: ψ=σ(F(Z))=σ(WZ) where ψ ∈ R C,H+W , F represents the mapping corresponding to the convolutional layer composed of 1*1 convolutional kernels corresponding to this scale, W represents the parameters corresponding to the convolutional layer, and W ∈ R d,H+W,C,H+W , and σ represents the Sigmoid activation function; Finally, ψ is split into the attention weight matrices ψ corresponding to the H direction and the W direction H ∈R C,H , ψ W ∈R C,W 。 6. The diabetic retinopathy grading method based on a multi-scale fusion attention mechanism according to claim 1, characterized in that In the ISK-Block module, a Softmax operation is performed element-wise on the attention weight matrices corresponding to two different scales. It is characterized in that for the attention matrices in the H direction and the W direction corresponding to scale a and the attention matrices in the H direction and the W direction corresponding to scale b respectively, and perform an element-wise Softmax operation, and perform an element-wise Softmax operation to obtain the final weight vectors corresponding to different scales in the H direction and the W direction; the weights corresponding to height h and width w of scale a and scale b on channel c are respectively and The calculation method is as follows: where exp represents the exponential function with the natural constant e as the base, represents a tensor the value of the h-th column in the c-th row, represents a tensor the value of the w-th column in the c-th row, represents a tensor the value of the h-th column in the c-th row, represents a tensor the value of the w-th column in the c-th row; Use to represent the final weight vectors corresponding to all heights and widths of scale a on channel c, respectively, and use to represent the final weight vectors corresponding to all heights and widths of scale b on channel c, respectively, where they describe the importance of the features of the feature maps at different scales in different directions on channel c.

7. A grading method for diabetic retinopathy based on a multi-scale fusion attention mechanism according to claim 1, characterized in that The ISK-Block module uses a weight matrix to weight features of different scales and fuse them into the final output feature map. The characteristic of this module is that for the c channel of the final feature map V, the calculation method is as follows: Among them represents the feature map U a , U b is the value of any pixel point on the c channel, represents the value of the i-th number of the weight vector represents the value of the j-th element of the weight vector ​​ 8. A method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism according to claim 1, characterized in that In step S5, a complete deep convolutional neural network is constructed. The number of available ISK-Blocks increases with the network depth, and the optional numbers are 18, 50, and 101. The deep convolutional neural network uses cross entropy as the loss function and Adam as the optimizer.

9. A method for grading diabetic retinopathy based on a multi-scale fusion attention mechanism according to claim 1, characterized in that When training the deep convolutional neural network in step S6, the parameters in the network are set to random numbers, the learning rate is 1*10-3, the batch size is 64, and the network is optimized by the early stopping method. When the loss of the network on the validation set tends to be flat, the training is stopped and the network parameters are saved; when the training is repeated for more than 50 times and the loss of the network on the validation set fluctuates by no more than 5%, it is considered to be flat.

Images