A method and device for fine-grained grading of retinal lesions
Through the feature extraction network of multi-layer neural network and channel attention mechanism, combined with the fusion of global and local features, the dependence problem of early diagnosis of diabetic retinopathy is solved, the recognition accuracy of micro-lesion points is improved, and efficient retinopathy is achieved Fine-grained grading.
Patent Information
- Application Number
- CN202010966557.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-09-15
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-09-15
AI Technical Summary
Early diagnosis of diabetic retinopathy depends on doctor's expertise and equipment, and the identification accuracy of micro-lesion points is insufficient, resulting in difficulty in diagnosis.
A multi-layer neural network and channel attention mechanism are used to construct a feature extraction network. Through the fusion of global features and local features, the information sorting function is used to obtain a high-information channel layer, and the objective function is optimized to improve the hierarchical results.
High-precision recognition of fine-grained retinopathy grading is achieved, the ability to identify micro-lesion points is improved, and the dependence on doctors' professional knowledge is reduced.
Smart Images

Figure CN112215239B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of medical image big data processing and analysis in the field of computer vision, and in particular relates to a method and device for fine-grained grading of retinal lesions based on a multi-channel attention selection mechanism. Background Art
[0002] Diabetic retinopathy (DR) is one of the most common complications of diabetes and one of the main causes of blindness in humans. Among diabetic patients in my country, the prevalence of diabetic retinopathy is as high as 23%, and this retinopathy cannot be cured in the late stage, so early diagnosis is particularly important. However, there is no early warning sign for diabetic retinopathy. People can only diagnose it by taking fundus images with electronic cameras and other equipment, and then handing them over to professional doctors for analysis. This is extremely dependent on the doctor's professional knowledge and the configuration of related equipment, and it usually takes several days to make a diagnosis. The current large number of potential patients and a small number of experienced personnel have brought great difficulties to the detection of diabetic retinopathy.
[0003] The biggest challenge in grading diabetic retinopathy is that the classification accuracy of fundus lesion images is more precise than that of other image categories. Too small lesions make the differences between classes very subtle and difficult to distinguish. Although a variety of feature extraction methods based on neural networks have been proposed, there are still some deficiencies in the identification of tiny lesions. Summary of the invention
[0004] The purpose of this application is to provide a fine-grained grading method and device for retinal lesions, which uses a multi-layer neural network and a channel attention mechanism to construct a feature extraction network to extract global features and attention features. In addition, when selecting multi-channel attention, a sorting function is used to obtain a high-information channel layer. Local features are obtained through the high-information channel layer and merged with the global features. An objective function is designed, and the grading result is obtained by optimizing the objective function.
[0005] A fine-grained grading method for retinal lesions, comprising:
[0006] Acquire a retinal image to be graded, and extract global features using a first densely connected neural network;
[0007] Input the global features into the attention extraction network to extract attention features and obtain the attention feature map;
[0008] Use the information extraction function to extract the information of each channel of the attention feature map, sort the information of all channels, and select the channels with the top K information to obtain the local channel feature map;
[0009] Performing random discarding processing on the local channel feature map, performing binarization processing on the local channel feature map after the random discarding processing, and selecting a local area in the retinal image to be classified according to the binarization processing result;
[0010] Extracting local features from the local area using a second densely connected neural network;
[0011] The global features and local features are fused and input into a fully connected layer to obtain a classification result.
[0012] Furthermore, the information extraction function is:
[0013] T(A)=Sigmoid(MLP(Maxpool(A))+MLP(Maxpool(A)))
[0014] Among them, Sigmoid is the activation operation, A is the attention feature map, Maxpool represents the global maximum pooling operation, Avgpool represents the global average pooling operation, and MLP represents the multi-layer perceptron operation.
[0015] Furthermore, the method for fine-grained grading of retinal lesions further includes:
[0016] Training a fine-grained retinal lesion classification network;
[0017] The fine-grained grading network for retinal lesions includes:
[0018] The first densely connected neural network is used to extract global features;
[0019] Attention extraction network, used to extract attention features and obtain attention feature maps;
[0020] A multi-channel attention selection network is used to extract the information content of each channel of the attention feature map using an information content extraction function, sort the information content of all channels, select the channels with the top K information content to obtain a local channel feature map; randomly discard the local channel feature map, binarize the local channel feature map after random discarding, and select a local area in the retinal image to be classified according to the binarization result;
[0021] A second densely connected neural network is used to extract local features from the local area using a second densely connected neural network;
[0022] A feature fusion network, used for fusing the global features and the local features;
[0023] The fully connected layer is used to classify the fused features after feature fusion to obtain the classification results.
[0024] Furthermore, the objective function of the fine-grained grading network for retinal lesions is:
[0025]
[0026] Among them, α and β are hyperparameters, is the sum of the classification losses of local features and global features, is the classification loss after the fusion of local features and global features, is the sorting loss;
[0027]
[0028]
[0029]
[0030] Among them, loss is the cross entropy loss, f c is the confidence function, F i ′ is the i-th local feature, class i F i ′ is the label value, F is the global feature, class is the label value of F, f is the hinge loss function, S′ j , S′ i is the amount of information corresponding to the channel feature in the local channel feature map, i, j∈K, C i is the classification confidence of the i-th local feature, C j is the classification confidence of the jth local feature, and C0 is the classification confidence of the global feature.
[0031] The present application also proposes a device for fine-grained grading of retinal lesions, comprising a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the above-mentioned method for fine-grained grading of retinal lesions.
[0032] The present application proposes a method and device for fine-grained grading of retinal lesions, which is a method for fine-grained grading of retinal lesions with high accuracy. The beneficial effects of the present application include: using a densely connected neural network as the basic network, using maximum pooling and mean pooling to obtain channel feature quantities, and obtaining local features according to the sorting and selection of channel feature quantities. By optimizing the multi-channel attention selection model through sorting loss, more attention is paid to channels and corresponding areas with large amounts of information when generating attention feature maps, so that when the amount of information in the feature map is large, the confidence of the feature map for correct classification is also large, so as to obtain more informative local areas. BRIEF DESCRIPTION OF THE DRAWINGS
[0033] Figure 1 A flow chart of the fine-grained grading method for retinopathy for this application;
[0034] Figure 2 This is a schematic diagram of a fine-grained grading network for retinal pathology according to an embodiment of the present application. DETAILED DESCRIPTION
[0035] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0036] like Figure 1 As shown, the present application provides a fine-grained grading method for retinal lesions, comprising:
[0037] Acquire a retinal image to be graded, and extract global features using a first densely connected neural network;
[0038] Input the global features into the attention extraction network to extract attention features and obtain the attention feature map;
[0039] Use the information extraction function to extract the information of each channel of the attention feature map, sort the information of all channels, and select the channels with the top K information to obtain the local channel feature map;
[0040] Performing random discarding processing on the local channel feature map, performing binarization processing on the local channel feature map after the random discarding processing, and selecting a local area in the retinal image to be classified according to the binarization processing result;
[0041] Extracting local features from the local area using a second densely connected neural network;
[0042] The global features and local features are fused and input into a fully connected layer to obtain a classification result.
[0043] The present application classifies retinal images to be classified based on a trained fine-grained retinal lesion classification network. In one embodiment, Figure 2 As shown, the fine-grained grading network of retinal lesions includes:
[0044] The first densely connected neural network is used to extract global features;
[0045] Attention extraction network, used to extract attention features and obtain attention feature maps;
[0046] A multi-channel attention selection network is used to extract the information content of each channel of the attention feature map using an information content extraction function, sort the information content of all channels, select the channels with the top K information content to obtain a local channel feature map; randomly discard the local channel feature map, binarize the local channel feature map after random discarding, and select a local area in the retinal image to be classified according to the binarization result;
[0047] A second densely connected neural network is used to extract local features from the local area using a second densely connected neural network;
[0048] A feature fusion network, used for fusing the global features and the local features;
[0049] The fully connected layer is used to classify the fused features after feature fusion to obtain the classification results.
[0050] It is easy to understand that the process of using the trained retinal lesion fine-grained grading network to grade the retinal images to be graded corresponds one to one with the process of training the retinal lesion fine-grained grading network using the training images. The following is a detailed explanation of the process of training the retinal lesion fine-grained grading network using the training images.
[0051] In this embodiment, the first densely connected neural network is used to extract features from the input retinal image to obtain the global feature F. For example, a DenseNet network is used.
[0052] In one embodiment, the attention extraction network includes three convolution layers with a convolution kernel size of 1×1, and the output channels of the three convolution layers are 128, 64, and 32 respectively. The third convolution layer is followed by batch normalization and ReLU activation function to obtain the attention feature map A. The obtained attention feature map is normalized and the pixel size is quantized to the interval [0, 1]:
[0053]
[0054] in is the attention feature on the i-th layer channel, M is the number of channels of the attention feature map after passing through three convolutional layers, which is set to 32, and H and W are the length and width of the attention feature map, respectively, which are set to 448. min(A i ) represents the minimum value of the attention feature points of the i-th layer channel. max(A i ) represents the maximum value among the attention feature points of the i-th layer channel.
[0055] In one embodiment, the multi-channel attention selection network uses an information extraction function to extract the information of each channel in the attention feature map. The information extraction function is:
[0056] T(A)=Sigmoid(MLP(Maxpool(A))+MLP(Avgpool(A)))
[0057] =Sigmoid(W1(W0(F max ))+W1(W0(F avg )))
[0058] in is the weight factor, M is the number of channels, r represents the reduction rate, Sigmoid is the activation operation, and A is the normalized attention feature map.
[0059] In the process of information extraction, the global maximum pooling and global average pooling based on length and width are calculated to obtain a channel description F of size 1×M max , F avg After being operated by the multi-layer perceptron, the pixels are added one by one. Finally, the Sigmoid activation function is used for activation to generate the final information score map S = {S1, S2, S3, ..., S M}, and the attention feature map A = {A1, A2, A3, ..., A M}One to one correspondence.
[0060] Then, the channel information is sorted, and the channels with the top K information are selected to obtain the local channel feature map A#={A′1, A′2, ..., A′ K}. Randomly discard the channel of the local channel feature map A# and randomly generate a vector with only 0 and 1 values Multiply it with the feature map in the channel dimension to obtain the local channel feature map A′ after random discarding:
[0061] A′=A # *m
[0062] Where m follows the Bernoulli distribution.
[0063] Then the local channel feature map after random discarding is binarized, that is, the pixels in the local channel feature map A′ that are greater than the threshold θ∈[0,1) are set to 1, and the other elements are set to 0, that is,
[0064]
[0065] Where A′ v (i, j) represents the pixel with coordinates (i, j) on the vth channel in A′. The threshold θ can be adjusted as a hyperparameter according to the specific problem.
[0066] Then calculate the rectangular area occupied by the points with pixel 1, map it to the original image, segment the corresponding area in the original image and upsample it to the image size of the model input, treat it as a local area, and obtain a total of K local areas {R1, R2, R3, ..., R K}.
[0067] The second densely connected neural network of the present application is used to extract local features. For example, a DenseNet network is used to input local regions into the second densely connected neural network to obtain local features F′={F′1, F′2, ..., F K ′}, Where K is the number of local regions, M is the number of channels, and H and W are the length and width of the local features, respectively.
[0068] It should be noted that the first densely connected neural network and the second densely connected neural network can be two neural networks, and can also share network parameters, which will not be elaborated here.
[0069] The feature fusion network of the present application is used to fuse the global features and local features, fuse the global features F and the local features F′, and connect them into a fused feature matrix in the channel dimension.
[0070] The fully connected layer of the fine-grained retinal lesion grading network of the present application, that is, the classification layer, inputs the fused feature matrix after feature fusion into the fully connected layer and maps it into a probability matrix, indicating the probability that the image belongs to each disease level.
[0071] It is easy to understand that the present application uses a trained fine-grained retinal lesion grading network to grade the retinal images to be graded. First, the constructed fine-grained retinal lesion grading network needs to be trained, and the objective function needs to be set during the training process.
[0072] The objective function of this application is constructed in the following steps:
[0073] Construct the joint loss, which is based on the cross entropy loss:
[0074]
[0075] Among them, loss is the cross entropy loss, C is the obtained classification confidence, and class is the label value. Class does not participate in the calculation, but only serves as an index of confidence.
[0076] This application requires obtaining local features and global features. The behavior of obtaining local feature maps is regarded as data enhancement, and the local area images are analyzed as enhanced data. Therefore, the sum of the classification losses of local features and global features is calculated separately:
[0077]
[0078] where f c It is a confidence function that calculates the classification probability of each feature area. The first part of the loss function is the sum of the cross entropy losses of all local areas. is the i-th local feature obtained in step 1.9, class i F i ′’s label value, and the second part is to calculate the cross entropy loss of the global area, where F is the global feature and class is the label value of F.
[0079] In addition, the classification loss after the fusion of local features and global features is:
[0080]
[0081] Constructing the sorting loss requires obtaining the classification confidence of the local area. The global features and local features are input into the fully connected layer respectively, and the classification confidence of each image can be obtained C = {C0, C1, C2, ..., C K}, where C0 is the classification confidence of the global feature, C i (i∈[1,K]) is the classification confidence of the i-th local feature.
[0082] The local area classification confidence C = {C1, C2, ..C k}, the information content of the top K feature maps is S′={S′1, S′2, ..., S′ K}, we hope that when C i The larger the value, the corresponding S′ i The larger the value, the greater the loss of all pairs is defined as follows:
[0083]
[0084] Where f is the hinge loss function, so that when C i <C j When S′ i <S′ j , S′ i =T(R i ). Function T(R i ) is the information extraction function in step 1.4.
[0085] The backward propagation of the loss function can be expressed as:
[0086]
[0087] f′ is the derivative of the hinge loss, R i(i∈[1,K]) is the i-th local region.
[0088] Finally, construct the objective function:
[0089]
[0090] where α and β are hyperparameters, is the sum of the classification losses of local features and global features, is the classification loss after the fusion of local features and global features, is the sorting loss.
[0091] loss is the cross entropy loss, f c is the confidence function, F i ′ is the i-th local feature, class i F i ′ is the label value, F is the global feature, class is the label value of F, f is the hinge loss function, S′ j , S ′ i is the amount of information corresponding to the channel feature in the local channel feature map, i, j∈K, C i is the classification confidence of the i-th local feature, C j is the classification confidence of the jth local feature, and C0 is the classification confidence of the global feature.
[0092] It should be noted that the setting of the information extraction function and the objective function in the fine-grained grading network of retinal lesions in the present application can be replaced by other functions by those skilled in the art. The use of other functions to replace the various functions listed in the embodiments of the present application should all fall within the scope of protection of the claims of the present application, and the present application does not impose any restrictions on the specific function form.
[0093] The training process of the fine-grained grading network for retinal lesions in this application is as follows:
[0094] Training dataset preprocessing:
[0095] Prepare training data sets and test data sets, both of which contain pre-prepared retinal images and labels corresponding to the images;
[0096] Grayscale the data set, set the threshold to 7 to generate a mask, and crop to remove the black area;
[0097] The dataset images are uniformly scaled to a fixed size of 512*512;
[0098] The dataset images are processed by Gaussian filtering. The σ parameter in the Gaussian filter is set to 10, and the Gaussian kernel size is set to (0, 0), and its value is calculated by σ.
[0099]
[0100] The original image and the Gaussian filtered image are mixed and weighted, where the weight μ of the original image is set to 4, the weight v of the Gaussian filtered image is set to -4, and γ is set to 128;
[0101] W(X,μ,Y,v,γ)=X·μ+Y·v+γ
[0102] Random data augmentation is performed on each image with a probability of 50%. The data augmentation operations include: randomly scaling the input image to 1 to 1.3 times the size; randomly rotating each image from 0 to 360 degrees with a probability of 50%; mirror flipping. Finally, all images are normalized.
[0103] During the training process, the network parameters are initialized first, and the network weights pre-trained on the ImageNet dataset are used to initialize the network, and the weights of the hidden layer and the output layer are randomly initialized; the batch data size is set to 8, the initial learning rate is 0.0001, the maximum number of iterations is 30 times, and the learning rate is multiplied by 0.1 for iterations 10, 20, and 26. Adam training data is used, and the network model is saved once for each iteration.
[0104] Input training samples and labels into the initialized network, calculate the objective function value, and repeatedly train through the two steps of forward propagation and back propagation until the maximum number of iterations is reached, so that the objective function value is minimized in this data set.
[0105] It is easy to understand that the network training process is a relatively mature technology and will not be elaborated here.
[0106] The process of using the trained fine-grained retinal lesion classification network to classify the retinal images to be classified corresponds to the process of using the training images to train the fine-grained retinal lesion classification network. The difference is that the training process requires the global features and local features to be input into the fully connected layer respectively, and the classification confidence of each image can be obtained in order to calculate the objective function value. That is, the training process also includes the step of inputting the global features and local features into the fully connected layer respectively to obtain the classification confidence of each image.
[0107] After one training is completed, the final saved network model is used as the pre-training model, and the network is retrained until the objective function value basically does not change, and the final network model is retained as the final result.
[0108] After the network training is completed, the retinal image to be graded is input into the trained network to obtain the grading result.
[0109] In one embodiment, the present application also provides a device for fine-grained grading of retinal lesions, comprising a processor and a memory storing a plurality of computer instructions, wherein the computer instructions, when executed by the processor, implement the steps of the above-mentioned method for fine-grained grading of retinal lesions.
[0110] The specific definition of the fine-grained grading device for retinal lesions can be found in the definition of the fine-grained grading method for retinal lesions mentioned above, which will not be repeated here.
[0111] The memory and the processor are electrically connected directly or indirectly to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory stores a computer program that can be run on the processor, and the processor implements the network topology layout method in the embodiment of the present invention by running the computer program stored in the memory.
[0112] The memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory is used to store a program, and the processor executes the program after receiving an execution instruction.
[0113] The processor may be an integrated circuit chip with data processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0114] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A fine-grained grading method for retinal lesions, characterized in that: The method for fine-grained grading of retinal lesions comprises: Acquire a retinal image to be graded, and extract global features using a first densely connected neural network; Input the global features into the attention extraction network to extract attention features and obtain the attention feature map; Use the information extraction function to extract the information of each channel of the attention feature map, sort the information of all channels, and select the channels with the top K information to obtain the local channel feature map; Performing random discarding processing on the local channel feature map, performing binarization processing on the local channel feature map after the random discarding processing, and selecting a local area in the retinal image to be classified according to the binarization processing result; Extracting local features from the local area using a second densely connected neural network; The global features and local features are fused and input into a fully connected layer to obtain a classification result; The method for fine-grained grading of retinal lesions further comprises: Training a fine-grained retinal lesion classification network; The fine-grained grading network for retinal lesions includes: The first densely connected neural network is used to extract global features; Attention extraction network, used to extract attention features and obtain attention feature maps; A multi-channel attention selection network is used to extract the information content of each channel of the attention feature map using an information content extraction function, sort the information content of all channels, select the channels with the top K information content to obtain a local channel feature map; perform random discarding processing on the local channel feature map, perform binarization processing on the local channel feature map after random discarding processing, and select a local area in the retinal image to be classified according to the binarization processing result; A second densely connected neural network is used to extract local features from the local area using a second densely connected neural network; A feature fusion network, used for fusing the global features and the local features; The fully connected layer is used to classify the fused features after feature fusion to obtain the classification results; Among them, the objective function of the fine-grained grading network for retinal lesions is: Among them, α and β are hyperparameters, is the sum of the classification losses of local features and global features, is the classification loss after the fusion of local features and global features, is the sorting loss; Among them, loss is the cross entropy loss, f c is the confidence function, F i ′ is the i-th local feature, class i F i ′ is the label value, F is the global feature, class is the label value of F, f is the hinge loss function, S′ j , S′ i is the amount of information corresponding to the channel feature in the local channel feature map, i,j∈K, C i is the classification confidence of the i-th local feature, C j is the classification confidence of the jth local feature, and C0 is the classification confidence of the global feature.
2. The fine-grained grading method for retinal lesions according to claim 1, characterized in that: The information extraction function is: T(A)=Sigmoid(MLP(Maxpool(A))+MLP(Maxpool(A))) Among them, Sigmoid is the activation operation, A is the attention feature map, Maxpool represents the global maximum pooling operation, Avgpool represents the global average pooling operation, and MLP represents the multi-layer perceptron operation.
3. A fine-grained grading device for retinal lesions, comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by a processor, the steps of the method described in any one of claims 1 to 2 are implemented.
