Method, system, device, and medium for feature classification of diabetic retinopathy

Abstract

The application discloses a diabetic retinopathy feature classification method, system, device and medium, wherein the method comprises: acquiring a diabetic retinal lesion image; inputting the lesion image into a preset channel attention network and a preset spatial attention network respectively to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; fusing the first feature map and the second feature map to obtain a third feature map; inputting the third feature map into a capsule network to obtain an output result of the capsule network; and inputting the output result into a preset classifier to obtain a classification result of the lesion image output by the classifier. The application can improve the accuracy and classification efficiency of diabetic retinal lesion image classification, and better cope with the problems caused by large differences between diabetic retinal lesion images and large amounts of diabetic retinal lesion image data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of lesion image classification, in particular to a diabetic retinopathy feature classification method, system, device and medium. BACKGROUND

[0002] Diabetes mellitus (DM) is a metabolic disorder caused by high blood glucose levels in the body. It has gradually become a major problem in the field of global public health, affecting 463 million people worldwide, and is expected to rise to 700 million by 2045. At least one-third of diabetic patients also suffer from diabetes-related eye diseases, of which diabetic retinopathy (DR) is the most common. Diabetic retinopathy causes varying degrees of damage to the retina as chronic diabetes progresses, and is one of the most common causes of visual impairment and blindness. Traditionally, diabetic retinopathy grading is determined based on a comprehensive assessment of different structural features presented in color fundus images, such as the presence of microaneurysms, exudates, hemorrhages, and neovascularization. Therefore, based on the study of early treatment of diabetic retinopathy, five severity grading standards of no apparent retinopathy, mild non-proliferative, moderate, severe and proliferative retinopathy are proposed.

[0003] At present, deep learning is often used to automatically classify the severity of retinopathy. However, in the case of difficulty in obtaining multi-modal data sets, using a single modal data set and relying only on high-level features extracted by deep neural networks cannot well identify or distinguish the lesion features of diabetic retinopathy, which leads to the problem of low classification accuracy of the current automatic classification scheme. SUMMARY

[0004] The present application aims to at least solve one of the technical problems existing in the prior art. To this end, the present application provides a diabetic retinopathy feature classification method, system, device and medium, which can enhance the performance of diabetic retinopathy feature classification in channel attention and spatial attention, and improve the accuracy and robustness of diabetic retinopathy feature classification.

[0005] In a first aspect, embodiments of the present application provide a diabetic retinopathy feature classification method, comprising:

[0006] obtaining a lesion image of a diabetic retina;

[0007] input the lesion image into a preset channel attention network and a preset spatial attention network respectively to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; wherein the channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, the output of an nth channel attention module is fused with the output of an nth spatial attention module to serve as the input of an (n+1)th channel attention module, and the output of the nth spatial attention module serves as the input of an (n+1)th spatial attention module; n is a positive integer in 1 to N-1;

[0008] fuse the first feature map and the second feature map to obtain a third feature map;

[0009] input the third feature map into a capsule network to obtain an output result of the capsule network;

[0010] input the output result into a preset classifier to obtain a classification result of the lesion image output by the classifier.

[0011] The control method according to the embodiment of the application has at least the following beneficial effects:

[0012] On one hand, the lesion image is input into the channel attention network to obtain the first feature map, the channel attention network can assign different weights to different channels, so that the importance of the channel can be quickly judged during subsequent classification, and the accuracy and timeliness of classification are improved; on the other hand, the lesion image is input into the spatial attention network to obtain the second feature map, so that the spatial positioning of the feature is more accurate, and the positioning speed and effect during subsequent classification are improved; moreover, the channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, the output of an nth channel attention module is fused with the output of an nth spatial attention module to serve as the input of an (n+1)th channel attention module, and the output of the nth spatial attention module serves as the input of an (n+1)th spatial attention module, so that the distribution of the channel attention weight and the distribution of the spatial attention weight are strengthened, and a fusion basis is provided for the fusion of the first feature map and the second feature map; the first feature map and the second feature map are fused to obtain the third feature map, the third feature map saves the attention information of the lesion image in space and the attention information of the channel, effectively associates the attention weight of the channel and the attention weight of the space, and the feature map generated by fusion has stronger anti-interference and lower misjudgment rate during classification; the third feature map is input into the capsule network, the capsule network can retrieve spatial information and more important features without losing any information, is adapted to the third feature map fused with the attention weight of the channel and the attention weight of the space, can improve the accuracy of classification, and also improves the classification efficiency.

[0013] According to some embodiments of the application, after the lesion image of the diabetic retinopathy is acquired, the method further comprises:

[0014] The lesion image is subjected to border removal, image enhancement, data enhancement and data balancing.

[0015] According to some embodiments of the application, the nth channel attention module in the channel attention network obtains the output feature by:

[0016] receiving the first initial feature fused from the output of the n-1th channel attention module and the output of the n-1th spatial attention module;

[0017] performing global average pooling on the first initial feature to obtain a channel feature of each channel; wherein the global average pooling comprises:

[0018] p = gap(F n ), p ∈ R Cn , F n ∈ R Cn×Hn×Wn

[0019] wherein p represents the channel feature, F n represents the feature output by the nth channel attention module, gap() represents the global average pooling, R represents the lesion image, Cn represents the channel serial number of the feature output by the nth channel attention module, H and W represent the length and width of the lesion image, respectively;

[0020] inputting the channel feature into a one-dimensional convolution, and obtaining an activation feature by sigmoid function activation:

[0021]

[0022] q = sigmoid(Conv1d(p)), q ∈ R Cn

[0023] wherein k represents the convolution kernel size of the one-dimensional convolution, β and γ represent preset parameters, q represents the activation feature, Conv1d() represents the convolution function, and sigmoid() represents the activation function;

[0024] obtaining a global channel evaluation coefficient according to the activation feature:

[0025]

[0026] wherein t represents the global channel evaluation coefficient, w0 represents a 0-dimensional weight parameter, w i represents a one-dimensional weight parameter, and V idenotes a two-dimensional weight parameter, x i and x j denotes a value in an activation feature q;

[0027] A channel attention map is obtained by the global channel evaluation coefficient and the activation feature:

[0028] r=qxt

[0029] wherein r denotes a channel attention map;

[0030] The channel attention map is multiplied with the first initial feature to obtain an output feature.

[0031] According to some embodiments of the present application, the nth spatial attention module in the spatial attention network obtains an output feature in the following manner:

[0032] receiving a second initial feature output by the (n-1)th spatial attention module;

[0033] performing average pooling and maximum pooling on the second initial feature to obtain spatial information;

[0034] inputting the spatial information into three convolutions respectively to obtain three tensors;

[0035] According to the three tensors, an output feature is calculated:

[0036] S n ' = SA(I max-mean ) = softmax(F q F k T )F v +F v , S n ' ∈ R Cn×Hn×Wn

[0037] wherein S n ' denotes an output feature of the nth spatial attention module, SA denotes self-attention, I max-mean denotes spatial information, F q denotes a query value in the attention mechanism, F v denotes a value in the attention mechanism, F k denotes a key value in the attention mechanism, and softmax() denotes an activation function.

[0038] According to some embodiments of the present application, the first feature map and the second feature map are fused in the following manner:

[0039]

[0040] wherein, F n denotes the feature output by the nth channel attention module, denotes matrix addition, F m denotes the third feature map.

[0041] According to some embodiments of the present application, the capsule network comprises a convolutional layer, a basic capsule network and a routing capsule network connected in sequence, the output of the convolutional layer is input into the basic capsule network, and the output of the basic capsule network is input into the routing capsule network.

[0042] According to some embodiments of the present application, the formula for calculating the output result of the capsule network comprises:

[0043] F1=Cap(ReLU(BN(Conv1(F m ))))

[0044] wherein, F1 denotes the output result of the capsule network, Cap denotes the capsule network, ReLU denotes an activation function, BN denotes batch normalization processing, and Conv1 denotes convolution operation.

[0045] In a second aspect, embodiments of the present application provide a feature classification system for diabetic retinopathy, comprising:

[0046] an image acquisition module configured to acquire an image of a lesion of a diabetic retina;

[0047] a feature map acquisition module configured to input the lesion image into a preset channel attention network and a preset spatial attention network respectively, to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; wherein the channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, the output of the nth channel attention module is fused with the output of the nth spatial attention module to serve as the input of the (n+1)th channel attention module, and the output of the nth spatial attention module serves as the input of the (n+1)th spatial attention module; n is a positive integer in 1 to N-1;

[0048] a feature map fusion module configured to fuse the first feature map and the second feature map to obtain a third feature map;

[0049] a capsule network module configured to input the third feature map into a capsule network to obtain an output result of the capsule network;

[0050] a classification result output module configured to input the output result into a preset classifier to obtain a classification result of the lesion image output by the classifier.

[0051] In a third aspect, embodiments of the present application provide an electronic device, comprising at least one control processor and a memory connected to the at least one control processor in communication; the memory stores instructions executable by the at least one control processor, and the instructions are executed by the at least one control processor to enable the at least one control processor to perform the feature classification method for diabetic retinopathy as described in the first aspect.

[0052] In a fourth aspect, embodiments of the present application provide a computer storage medium, which stores computer executable instructions for causing a computer to perform the feature classification method for diabetic retinopathy as described in the first aspect.

[0053] It should be noted that the beneficial effects between the second to fourth aspects of the present application and the prior art are the same as those of the feature classification method for diabetic retinopathy of the first aspect, which will not be described here.

[0054] Other features and advantages of the present application will be described in the following description, and will become apparent from the description, or will be learned from the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0055] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the following description, taken in conjunction with the accompanying drawings, in which:

[0056] Figure 1 is a flowchart of a feature classification method for diabetic retinopathy provided by an embodiment of the present application;

[0057] Figure 2 is a flowchart of a method for pre-processing an image provided by an embodiment of the present application;

[0058] Figure 3 is a flowchart of a method for obtaining output features by a channel attention module provided by an embodiment of the present application;

[0059] Figure 4 is a flowchart of a method for obtaining output features by a spatial attention module provided by an embodiment of the present application;

[0060] Figure 5 is a structural diagram of a feature classification system for diabetic retinopathy provided by an embodiment of the present application;

[0061] Figure 6 is a structural diagram of an electronic device provided by an embodiment of the present application;

[0062] Figure 7is a schematic diagram of a channel attention module provided by an embodiment of the present application;

[0063] Figure 8 is a schematic diagram of a spatial attention module provided by an embodiment of the present application;

[0064] Figure 9 is a schematic diagram of a feature classification method for diabetic retinopathy provided by an embodiment of the present application;

[0065] Figure 10 is a schematic diagram of a capsule network provided by an embodiment of the present application.

[0066] Figure 11 is a schematic diagram of obtaining a global channel evaluation coefficient provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numbers represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0068] In the description of the present application, if the first, second, etc. are described, it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying the relative importance of the indicated technical features or implicitly indicating the number of the indicated technical features or the sequence of the indicated technical features.

[0069] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, etc. is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present application and simplifying the description, and is not intended to indicate or imply that the device or element indicated must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0070] In the description of the present application, it should be noted that, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be broadly understood, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0071] Reference Figure 1 In some embodiments of the present application, a feature classification method for diabetic retinopathy is provided, comprising:

[0072] Step S100, acquiring a lesion image of a diabetic retina.

[0073] Step S200, input the lesion image into the preset channel attention network and the preset spatial attention network respectively to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; wherein the channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, the output of the nth channel attention module and the output of the nth spatial attention module are fused to serve as the input of the (n+1)th channel attention module, and the output of the nth spatial attention module serves as the input of the (n+1)th spatial attention module; n is a positive integer in 1 to N-1.

[0074] Step S300, fuse the first feature map and the second feature map to obtain a third feature map.

[0075] Step S400, input the third feature map into the capsule network to obtain an output result of the capsule network.

[0076] Step S500, input the output result into the preset classifier to obtain a classification result of the lesion image output by the classifier.

[0077] In step S200 of the method embodiment, first, the lesion image is input into the channel attention network to obtain the first feature map, which can assign different weights to each different channel to quickly judge the importance of the channel during subsequent classification, thereby improving the accuracy and timeliness of classification; then, the lesion image is input into the spatial attention network to obtain the second feature map, which makes the spatial positioning of the features more accurate, thereby improving the positioning speed and positioning effect during subsequent classification; secondly, the channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, the output of the nth channel attention module and the output of the nth spatial attention module are fused to serve as the input of the (n+1)th channel attention module, and the output of the nth spatial attention module serves as the input of the (n+1)th spatial attention module, which can strengthen the allocation of the channel attention weight and the allocation of the spatial attention weight, and provide a fusion basis for the fusion of the first feature map and the second feature map; in step S300 of the method embodiment, the first feature map and the second feature map are fused to obtain the third feature map, the third feature map saves the attention information of the lesion image in space and the attention information of the channel, effectively associates the attention weight of the channel and the attention weight of the space, and the feature map generated by the fusion has stronger anti-interference and lower misjudgment rate when performing classification; in step S400 of the method embodiment, the third feature map is input into the capsule network, the capsule network can retrieve spatial information and more important features without losing any information, is adapted to the third feature map fused with the attention weight of the channel and the attention weight of the space, can improve the accuracy of classification, and also improves the classification efficiency.

[0078] ReferenceFigure 2 In some embodiments of the application, after acquiring the lesion images of diabetic retinopathy, the method further comprises:

[0079] Step S110, removing the frame, image enhancement, data enhancement and data balancing of the lesion image.

[0080] The following will be specifically introduced:

[0081] Remove black edges: the lesion image is a 224x224 picture with a frame, first remove the frame of the lesion image to reduce the interference of irrelevant background factors.

[0082] Image enhancement: some tertiary information of the lesion image is eliminated to improve the classification quality of the lesion image. The contrast limited adaptive histogram equalization algorithm (CLAHE) is more natural in appearance, which helps to eliminate the noise amplification in the histogram equalization family, so the CLAHE algorithm is selected for image enhancement.

[0083] The first stage of the CLAHE algorithm is to use the binarization of the histogram to transform the image. After that, the clipping boundary is used to limit the contrast to a binary count from 0 to 1. Before processing the image segment, the clipping boundary is added to the image. In order to avoid mapping the background area to gray, a specific bin value of the histogram region is applied to generate the entire image region. In order to achieve better mapping, the clipping boundary is applied with the help of histogram clipping. Finally, the final CLAHE image is generated by calculating the region of the image, and then all image pixels are extracted, mapped and interpolated to achieve the purpose of obtaining the optimal efficiency from the image.

[0084] Data enhancement: Gaussian filtering method is selected to enhance the lesion image data and improve the clarity of the image. Gaussian filtering is a linear smoothing filter, which is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing. The weights of the convolution kernel of Gaussian filtering are not the same, the weights of the central pixels are the highest, and the weights of the pixels far from the center are smaller. Its principle is a 2-dimensional Gaussian function. Compared with the mean filter, the efficiency of Gaussian filter is slower, but it can effectively eliminate Gaussian noise and retain more image details.

[0085] Data balancing: the number of DR images of different categories is very different, so the lesion images of negative samples are balanced, rotated (90 degrees, 180 degrees, 270 degrees), horizontally and vertically flipped, and cropped at the four corners and center of the lesion image, to solve the problem of data imbalance of the lesion image.

[0086] Through data preprocessing, the input image standard is unified, the robustness when facing large amount of diabetic retinopathy lesion images is improved, and the environment with large image difference and large number of images is better adapted.

[0087] Referring to Figure 3 In some embodiments of the present application, the nth channel attention module in the channel attention network obtains the output feature in the following manner:

[0088] Step S211, receiving the first initial feature fused by the output of the n-1th channel attention module and the output of the n-1th spatial attention module;

[0089] Step S212, performing global average pooling on the first initial feature to obtain a channel feature of each channel; wherein the global average pooling comprises:

[0090] p = gap(F n ), p ∈ R Cn , F n ∈ R Cn×Hn×Wn

[0091] Wherein, p represents the channel feature, F n represents the feature output by the nth channel attention module, gap() represents the global average pooling, R represents the lesion image, Cn represents the channel number of the feature output by the nth channel attention module, H and W represent the length and width of the lesion image respectively;

[0092] Step S213, inputting the channel feature into a one-dimensional convolution and obtaining an activation feature by sigmoid function activation:

[0093]

[0094] q = sigmoid(Conv1d(p)), q ∈ R Cn

[0095] Wherein, k represents the convolution kernel size of the one-dimensional convolution, β and γ represent preset parameters, q represents the activation feature, Conv1d() represents the convolution function, and sigmoid() represents the activation function;

[0096] Step S214, obtaining a global channel evaluation coefficient according to the activation feature:

[0097]

[0098] Wherein, t represents the global channel evaluation coefficient, w0 represents a 0-dimensional weight parameter, w i represents a 0-dimensional weight parameter, V i represents a 0-dimensional weight parameter, x i and x j represent the values in the activation feature q;

[0099] Step S215, obtaining a channel attention map by the global channel evaluation coefficient and the activation feature:

[0100] r=qxt

[0101] wherein r represents the channel attention map;

[0102] Step S216, multiplying the channel attention map with the first initial feature to obtain the output feature.

[0103] Through global average pooling, the attention to the channel is more concentrated, and through one-dimensional convolution and sigmoid function activation, the weight of the channel attention can be better allocated, the obtained channel attention map is more accurate, and the weight allocation is more reasonable.

[0104] Referring to Figure 4 In some embodiments of the application, the nth spatial attention module in the spatial attention network obtains the output feature in the following manner:

[0105] Step S221, receiving the second initial feature output by the n-1th spatial attention module;

[0106] Step S222, performing average pooling and maximum pooling on the second initial feature to obtain spatial information;

[0107] Step S223, inputting the spatial information into three convolutions respectively to obtain three tensors;

[0108] Step S224, calculating the output feature according to the three tensors:

[0109] S n '=SA(I max-mean )=softmax(F q F k T )F v +F v , S n '∈R Cn×Hn×Wn

[0110] wherein S n ' represents the feature output by the n th spatial attention module, SA represents self-attention, I max-mean represents spatial information, F q represents the query value in the attention mechanism, F v represents the value in the attention mechanism, F k represents the key value in the attention mechanism, and softmax() represents the activation function.

[0111] The spatial information is obtained through average pooling and maximum pooling, so that the spatial information is better preserved, and the query value, the value value and the key value in the attention mechanism are obtained through three convolutions, and the three tensors can better allocate the spatial attention weight for the features.

[0112] In some embodiments of the application, the first feature map and the second feature map are fused by the following method:

[0113]

[0114] F n represents the feature output by the nth channel attention module, represents matrix addition, F m represents the third feature map.

[0115] The first feature map and the second feature map are fused to obtain the third feature map, which preserves the attention information of the lesion image in space and the attention information of the channel, effectively associates the attention weight of the channel and the attention weight of the space, and has stronger anti-interference and lower misjudgment rate when classifying the feature map generated by fusion.

[0116] In some embodiments of the application, the capsule network comprises a convolution layer, a basic capsule network and a routing capsule network connected in sequence, the output of the convolution layer is taken as the input of the basic capsule network, and the output of the basic capsule network is taken as the input of the routing capsule network.

[0117] The convolution layer, the basic capsule network and the routing capsule network connected in sequence are selected as the basic framework of the capsule network, which can retrieve spatial information and more important features without losing any information, adapt to the third feature map with channel attention weight and spatial attention weight, and improve the efficiency and accuracy of classification.

[0118] In some embodiments of the application, the capsule network calculates the output result by the following formula:

[0119] F1=Cap(ReLU(BN(Conv1(F m ))))

[0120] Wherein, F1 represents the output result of the capsule network, Cap represents the capsule network, ReLU represents the activation function, BN represents the batch normalization processing, and Conv1 represents the convolution operation.

[0121] The output result of the capsule network is taken as the input of the classifier, which reduces the operation complexity of the classifier and reduces the operation burden.

[0122] Referring to Figure 7, in order to facilitate the understanding of those skilled in the art, a specific embodiment of the present application provides a feature classification method for diabetic retinopathy, comprising the following steps:

[0123] Firstly, the diabetic retinopathy image is preprocessed:

[0124] Remove black edges: the lesion image is a 224x224 picture with a frame, first remove the frame of the lesion image to reduce the interference of irrelevant background factors.

[0125] Image enhancement: some tertiary information of the lesion image is eliminated to improve the classification quality of the lesion image. The contrast limited adaptive histogram equalization algorithm (CLAHE) is more natural in appearance, which helps to eliminate the noise amplification in the histogram equalization family, so the CLAHE algorithm is selected for image enhancement.

[0126] The first stage of the CLAHE algorithm is to use the binarization of the histogram to transform the image. After that, the clipping boundary is used to limit the contrast to a binary count from 0 to 1. Before processing the image segment, the clipping boundary is added to the image. In order to avoid mapping the background area to gray, a specific bin value of the histogram region is applied to generate the entire image region. In order to achieve better mapping, the clipping boundary is applied with the help of histogram clipping. Finally, the final CLAHE image is generated by calculating the region of the image, and then all image pixels are extracted, mapped and interpolated to achieve the purpose of obtaining the optimal efficiency from the image.

[0127] Data enhancement: Gaussian filtering method is selected to enhance the lesion image data and improve the clarity of the image. Gaussian filtering is a linear smoothing filter, which is suitable for eliminating Gaussian noise and is widely used in the noise reduction process of image processing. The weights of the convolution kernel of Gaussian filtering are not the same, the weights of the central pixels are the highest, and the weights of the pixels far from the center are smaller. Its principle is a 2-dimensional Gaussian function. Compared with the mean filter, the efficiency of Gaussian filter is slower, but it can effectively eliminate Gaussian noise and retain more image details.

[0128] Data balancing: the number of DR images of different categories is very different, so the lesion images of negative samples are balanced, rotated (90 degrees, 180 degrees, 270 degrees), horizontally and vertically flipped, and cropped at the corners and center of the lesion image, to solve the problem of unbalanced data of the lesion image.

[0129] Secondly, the preprocessed lesion image is input into the preset channel attention network and the preset spatial attention network:

[0130] The channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, an output of an nth channel attention module is fused with an output of an nth spatial attention module to serve as an input of an (n+1)th channel attention module, and an output of the nth spatial attention module serves as an input of an (n+1)th spatial attention module; n is a positive integer in 1 to N-1;

[0131] With reference to Figure 8 and Figure 11 , the nth channel attention module in the channel attention network obtains an output feature in the following manner:

[0132] receiving a first initial feature fused from an output of an (n-1)th channel attention module and an output of an (n-1)th spatial attention module;

[0133] performing global average pooling on the first initial feature to obtain a channel feature of each channel; wherein the global average pooling comprises:

[0134] p = gap(F n ), p ∈ R Cn , F n ∈ R Cn×Hn×Wn

[0135] wherein p represents the channel feature, F n represents the output feature of the nth channel attention module, gap() represents the global average pooling, R represents a lesion image, Cn represents a channel serial number of the output feature of the nth channel attention module, and H and W represent a length and a width of the lesion image, respectively;

[0136] inputting the channel feature into a one-dimensional convolution and obtaining an activation feature through a sigmoid function activation:

[0137]

[0138] q = sigmoid(Conv1d(p)), q ∈ R Cn

[0139] wherein k represents a convolution kernel size of the one-dimensional convolution, β and γ represent preset parameters, q represents the activation feature, Conv1d() represents a convolution function, and sigmoid() represents an activation function;

[0140] obtaining a global channel evaluation coefficient according to the activation feature:

[0141]

[0142] wherein t represents the global channel evaluation coefficient;

[0143] The channel attention map is obtained by evaluating the coefficient and the activated feature through the global channel;

[0144] r = q x t

[0145] wherein, r represents the channel attention map;

[0146] The channel attention map is multiplied by the first initial feature to obtain the output feature (the output feature of the last channel attention module is the first feature map).

[0147] Referring to Figure 9 The nth spatial attention module in the spatial attention network obtains the output feature in the following manner:

[0148] The second initial feature output by the (n-1) th spatial attention module is received;

[0149] The second initial feature is subjected to average pooling and maximum pooling to obtain spatial information;

[0150] The spatial information is input into three convolutions respectively to obtain three tensors;

[0151] According to the three tensors, the output feature is calculated (the output feature of the last spatial attention module is the second feature map) :

[0152] S n ' = SA(I max-mean ) = softmax(F q F k T )F v +F v , S n ' ∈ R Cn×Hn×Wn

[0153] wherein, S n ' represents the output feature of the nth spatial attention module, SA represents self-attention, I max-mean represents the spatial information, F q represents the query value in the attention mechanism, F v represents the value in the attention mechanism, F k represents the key value in the attention mechanism, and softmax() represents an activation function.

[0154] Third step, fuse the first feature map and the second feature map to obtain the third feature map:

[0155]

[0156] wherein, F n ' represents the output feature of the nth channel attention module, denotes matrix addition, F m denotes a third feature map.

[0157] In a fourth step, the third feature map is input into a capsule network to obtain an output result of the capsule network.

[0158] F1=Cap(ReLU(BN(Conv1(F m ))))

[0159] wherein F1 denotes the output result of the capsule network, Cap denotes the capsule network, ReLU denotes an activation function, BN denotes batch normalization processing, and Conv1 denotes convolution operation.

[0160] Referring to Figure 10 , the capsule network comprises a convolution layer, a basic capsule network and a routing capsule network connected in sequence, the output of the convolution layer is taken as the input of the basic capsule network, and the output of the basic capsule network is taken as the input of the routing capsule network.

[0161] The algorithm table of the capsule network is shown in Table 1:

[0162] Table 1

[0163]

[0164] The structure information of the capsule network is shown in Table 2:

[0165] Table 2

[0166]

[0167] In a fifth step, the output result is input into a preset classifier to obtain a classification result of the lesion image output by the classifier.

[0168] Referring to Figure 5 , one embodiment of the present application further provides a feature classification system 1000 for diabetic retinopathy, comprising an image acquisition module 1001, a feature map acquisition module 1002, a feature map fusion module 1003, a capsule network module 1004 and a classification result output module 1005, wherein:

[0169] The image acquisition module 1001 is used for acquiring a lesion image of diabetic retina.

[0170] The feature map acquisition module 1002 is configured to input the lesion image into a preset channel attention network and a preset spatial attention network respectively to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; wherein the channel attention network comprises N channel attention modules, the spatial attention network comprises N spatial attention modules, the output of an nth channel attention module is fused with the output of an nth spatial attention module to serve as the input of an (n+1)th channel attention module, and the output of the nth spatial attention module serves as the input of an (n+1)th spatial attention module; n is a positive integer in 1 to N-1.

[0171] The feature map fusion module 1003 is configured to fuse the first feature map and the second feature map to obtain a third feature map.

[0172] The capsule network module 1004 is configured to input the third feature map into a capsule network to obtain an output result of the capsule network.

[0173] The classification result output module 1005 is configured to input the output result into a preset classifier to obtain a classification result of the lesion image output by the classifier.

[0174] It should be noted that, since the feature classification system for diabetic retinopathy in the embodiment and the feature classification method for diabetic retinopathy described above are based on the same inventive concept, the corresponding content in the method embodiment is also applicable to the device embodiment, which will not be described in detail here.

[0175] Reference Figure 6 Another embodiment of the present application also provides an electronic device 6000, which can be any type of smart terminal, such as a mobile phone, a tablet computer, a personal computer, etc.

[0176] Specifically, the electronic device 6000 comprises one or more control processors 6001 and a memory 6002, Figure 6 For example, the control processor 6001 and the memory 6002 can be connected through a bus or other means, Figure 6 For example, the connection is through a bus.

[0177] The memory 6002 as a non-transient computer readable storage medium can be used to store non-transient software programs, non-transient computer executable programs and modules, such as the program instructions / modules corresponding to the electronic device in the embodiment of the present application;

[0178] The control processor 6001 performs various functional applications and data processing of the method for classifying features of diabetic retinopathy by running non-transitory software programs, instructions and modules stored in the memory 6002, i.e., implements the method for classifying features of diabetic retinopathy according to the above method embodiments.

[0179] The memory 6002 can include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; and the data storage area can store data created by use of the method for classifying features of diabetic retinopathy and the like. In addition, the memory 6002 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory 6002 can optionally include a memory remotely disposed relative to the control processor 6001, and these remote memories can be connected to the electronic device 6000 through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0180] In one or more modules are stored in the memory 6002, when being executed by the one or more control processors 6001, perform the method for classifying features of diabetic retinopathy according to the above method embodiments, such as performing the method steps of the above described Figures 1 to 4 .

[0181] The memory serves as a non-transitory computer-readable storage medium, and can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely disposed relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0182] It should be noted that, since the electronic device in the present embodiment and the method for classifying features of diabetic retinopathy described above are based on the same inventive concept, the corresponding contents in the method embodiments are also applicable to the present device embodiments, which will not be described in detail here.

[0183] An embodiment of the present application further provides a computer-readable storage medium storing computer-executable instructions for performing the method for classifying features of diabetic retinopathy according to the above embodiments.

[0184] It should be noted that since the computer readable storage medium in this embodiment and the above-mentioned feature classification method for diabetic retinopathy are based on the same inventive concept, the corresponding contents in the method embodiment are also applicable to the device embodiment, and will not be described in detail here.

[0185] Those of ordinary skill in the art will appreciate that all or certain steps in the above-disclosed methods, systems can be implemented as software, firmware, hardware, and appropriate combinations thereof. Certain physical components or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on computer readable media, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As is well known to those of ordinary skill in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of data such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by a computer. Furthermore, it should be appreciated by those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media.

[0186] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "exemplary embodiment", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the exemplary description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0187] Although embodiments of the present application have been shown and described, those of ordinary skill in the art will appreciate that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method of classifying features of diabetic retinopathy, characterized by, The method comprises: obtaining a diabetic retinopathy image; The lesion image is input into a preset channel attention network and a preset spatial attention network, respectively, to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; wherein, the channel attention network includes N channel attention modules, and the spatial attention network includes N spatial attention modules, the th... The output of the channel attention module and the first The outputs of the aforementioned spatial attention modules are fused together and used as the first... The input of the channel attention module, the first The output of the spatial attention module is used as the first... The input of the spatial attention module; It is a positive integer from 1 to N-1; the first in the channel attention network The channel attention module obtains the output features in the following manner: receive the output of the first channel attention module and the output of the first spatial attention module fuse the first initial feature ​ performing global average pooling on the first initial feature to obtain a channel feature of each channel; wherein the global average pooling comprises: wherein, represents a channel feature, represents a feature output by a th channel attention module, represents a global average pooling, represents a lesion image, represents a feature output by a th channel attention module, and respectively represent the length and width of the lesion image. The channel features are input to a one-dimensional convolution and passed through a function activation to obtain activation features: wherein, represents a convolution kernel size of one-dimensional convolution, and represents a preset parameter, represents an activation feature, represents a convolution function, represents an activation function; obtaining a global channel evaluation coefficient according to the activation feature: wherein represents a global channel assessment coefficient; obtaining a channel attention map through the global channel evaluation coefficient and the activation feature; wherein, represents a 0-dimensional weight parameter, represents a 1-dimensional weight parameter, represents a 2-dimensional weight parameter, and represents an activation feature value in ; multiplying the channel attention map and the first initial feature to obtain an output feature; The first of the spatial attention networks obtains an output feature by the following manner: the first of the spatial attention modules receive the first initial feature output by the first spatial attention module and the second initial feature output by the second spatial attention module performing average pooling and maximum pooling on the second initial feature to obtain spatial information; inputting the spatial information into three convolutions respectively to obtain three tensors; obtaining an output feature according to the three tensors: wherein, denotes the feature output by the i-th spatial attention module, denotes self-attention, denotes spatial information, denotes the query value in the attention mechanism, denotes the value in the attention mechanism, denotes the key value in the attention mechanism, denotes an activation function, and the capsule network comprises a convolutional layer, a basic capsule network and a routing capsule network connected in sequence, an output of the convolutional layer being taken as an input of the basic capsule network, and an output of the basic capsule network being taken as an input of the routing capsule network.​ fusing the first feature map and the second feature map to obtain a third feature map; inputting the third feature map into a capsule network to obtain an output result of the capsule network; inputting the output result into a preset classifier to obtain a classification result of the classifier for the diabetic retinopathy image.

2. The method of claim 1, wherein the classification of the diabetic retinopathy is characterized by, After the diabetic retinopathy image is obtained, the method further comprises: performing frame removal, image enhancement, data enhancement, and data balancing on the diabetic retinopathy image.

3. The method of claim 1, wherein the classification of the diabetic retinopathy is characterized by, The first feature map and the second feature map are fused in the following manner: wherein, represents the output of the i-th channel attention module, represents the i-th feature map output by the i-th channel attention module, represents matrix addition, represents the third feature map.

4. The method of claim 3, wherein the classification of the diabetic retinopathy is characterized by, The formula for calculating the output result of the capsule network comprises: wherein, represents an output result of the capsule network, represents a capsule network, represents an activation function, represents a batch normalization process, represents a convolution operation.

5. A system for classifying features of diabetic retinopathy, the system comprising: comprises: an image acquisition module configured to obtain a diabetic retinopathy image; The feature map acquisition module is used to input the lesion image into a preset channel attention network and a preset spatial attention network, respectively, to obtain a first feature map output by the channel attention network and a second feature map output by the spatial attention network; wherein, the channel attention network includes N channel attention modules, the spatial attention network includes N spatial attention modules, and the... The output of the channel attention module and the first The outputs of the aforementioned spatial attention modules are fused together and used as the first... The input of the channel attention module, the first The output of the spatial attention module is used as the first... The input of the spatial attention module; It is a positive integer from 1 to N-1; the first in the channel attention network The channel attention module obtains the output features in the following manner: receive an output of a first channel attention module and an output of a second channel attention module fuse the output of the first channel attention module and the output of the second channel attention module to obtain a first initial feature fuse the output of the first channel attention module and the output of the second channel attention module to obtain a first initial feature performing global average pooling on the first initial feature to obtain a channel feature of each channel; wherein the global average pooling comprises: wherein, represents a channel feature, represents a feature output by a th channel attention module, represents a global average pooling, represents a lesion image, represents a feature output by a th channel attention module, and respectively represent the length and width of the lesion image. The channel features are input to a one-dimensional convolution and passed through a function activation to obtain activation features: wherein, represents a convolution kernel size of one-dimensional convolution, and represents a preset parameter, represents an activated feature, represents a convolution function, represents an activation function; obtaining a global channel evaluation coefficient according to the activation feature: wherein represents a global channel assessment coefficient; obtaining a channel attention map through the global channel evaluation coefficient and the activation feature; wherein, represents a 0-dimensional weight parameter, represents a 1-dimensional weight parameter, represents a 2-dimensional weight parameter, and represents an activation feature in the value; multiplying the channel attention map and the first initial feature to obtain an output feature; The first in the spatial attention network The spatial attention module obtains the output features in the following manner: receive the first initial feature output by the first spatial attention module and the second initial feature output by the second spatial attention module performing average pooling and maximum pooling on the second initial feature to obtain spatial information; inputting the spatial information into three convolutions respectively to obtain three tensors; obtaining an output feature according to the three tensors: wherein, represents the feature output by the i-th spatial attention module, represents self-attention, represents spatial information, represents a query value in an attention mechanism, represents a value in an attention mechanism, represents a key value in an attention mechanism, represents an activation function, and the capsule network comprises a convolutional layer, a basic capsule network and a routing capsule network connected in sequence, an output of the convolutional layer is taken as an input of the basic capsule network, and an output of the basic capsule network is taken as an input of the routing capsule network.​ a feature map fusion module configured to fuse the first feature map and the second feature map to obtain a third feature map; a capsule network module configured to input the third feature map into a capsule network to obtain an output result of the capsule network; a classification result output module configured to input the output result into a preset classifier to obtain a classification result of the classifier for the diabetic retinopathy image.

6. An electronic device, comprising: The computer readable storage medium stores computer executable instructions for causing a computer to execute the feature classification method for diabetic retinopathy as claimed in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The computer readable storage medium stores computer executable instructions for causing a computer to execute the feature classification method for diabetic retinopathy as claimed in any one of claims 1 to 4.

Images