A method for processing diabetic retinopathy image data

Abstract

The present invention relates to a method for processing diabetic retinopathy image data, and the method includes: data preprocessing: obtaining a high-definition fundus image X, obtaining a preprocessed high-definition fundus image #imgabs0# through noise reduction and detail enhancement, and then obtaining a preprocessed high-definition fundus image #imgabs1# with a smaller resolution through scaling and data enhancement; feature extraction: inputting the preprocessed high-definition fundus image #imgabs2# with a smaller resolution into a convolutional neural network A to obtain a feature map F; respectively adopting an overall processing branch processing method and a region of interest processing branch processing method to obtain a global processing result y whole and a region of interest processing result y attention ; averaging the scores of the global processing result y whole and the region of interest processing result y attention to obtain a final processing result y pred . Compared with the prior art, the present invention has the advantages of high accuracy, high robustness, and saving computing resources.

Description

Technical Field

[0001] The invention relates to the field of image classification, and in particular to a method for processing diabetic retinopathy image data. Background Art

[0002] Diabetic retinopathy (DR) is a series of typical lesions caused by retinal microvascular damage. It is the most common eye complication of diabetes and one of the most important blinding eye diseases worldwide. Early screening, management, treatment and follow-up of DR patients can effectively reduce the occurrence of visual impairment and blindness, but currently limited by medical conditions and patient compliance, little can be done.

[0003] With the increase in the number of people with diabetes and the shortage of ophthalmologists, the traditional DR screening method cannot effectively cope with the dual challenges of the continued increase in DR prevalence and blindness rate. The advent of the wave of artificial intelligence has made it possible to use artificial intelligence technology to assist doctors in screening and treatment. There have been some studies on the image data processing of diabetic retinopathy combined with artificial intelligence technology. There are many commonly used methods that can be divided into: multi-classification problems, regression problems, and multi-task problems.

[0004] 1) Multi-classification problem: Each level of DR is divided into one category, and then a general classification problem solution is used to process the output of the neural network with softmax, and then a cross entropy loss function is used. In DR grading, the number of images at different levels is different, and the data is ordered. The multi-classification problem is based on the relative independence of categories, assuming that the classes are independent of each other and ignoring the inherent order between labels.

[0005] 2) Regression problem: The discrete level labels of DR are regarded as continuous values, where the distances between adjacent classes are equal, and then the general regression problem solution is used to minimize the mean square error (MSE) or mean absolute error (MAE). However, the regression problem violates the non-stationarity of image-related tasks and can easily lead to overfitting.

[0006] 3) Multi-task problem: DR classification is divided into multiple subtasks, each of which is a binary classification problem. Each task divides a certain level of DR classification. Generally, PDR is selected first, then severe NPDR, then moderate NPDR, and mild NPDR, and finally no DR. There are four subtasks in total, and each task needs to set a threshold. However, multi-task is equivalent to calculating the cumulative probability independently by multiple branches, and it cannot guarantee that the function is monotonically decreasing, resulting in low learning efficiency in the early stage of training, and multiple parameters need to be manually fine-tuned.

[0007] Currently, the commonly used fundus image data processing and screening algorithms usually use a single-model classification model. Due to computing power limitations, this type of model usually scales high-definition images to a smaller resolution and then inputs them into the model for processing. Common methods usually receive fundus images with a resolution of 224*224 to 448*448. However, current fundus devices usually have a resolution of more than 1792*1792. These methods can only scale down images that exceed the resolution before processing, resulting in a loss of many details. Summary of the invention

[0008] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for processing diabetic retinopathy image data with high accuracy, high robustness and saving of computing resources.

[0009] The purpose of the present invention can be achieved by the following technical solutions:

[0010] The present invention provides a method for processing diabetic retinopathy image data, the method comprising:

[0011] Data preprocessing: Get a high-definition fundus image X, and obtain a preprocessed high-definition fundus image after noise reduction and detail enhancement. After scaling and data enhancement, a pre-processed high-definition fundus image with a smaller resolution is obtained.

[0012] Feature extraction: Preprocessing high-definition fundus images with lower resolution Input into the convolutional neural network A to obtain a series of feature maps F;

[0013] Overall processing branch: The feature map F is input into the global classification network for global fundus image processing to obtain the global processing result y representing the global hierarchical case classification whole ;

[0014] Region of interest processing branch: Input the feature map F into the region of interest proposal network to obtain the region of interest, combined with the high-definition pre-processed fundus image Get the fundus image of the region of interest After data enhancement and scaling, it is input into the convolutional neural network B for local fundus image processing, and the processing result y of the focus area is calculated. attention ;

[0015] Get the final classification processing result: based on the global processing result y whole The processing result y of the area of ​​interest attention , calculate the final classification result y pred .

[0016] Preferably, the data enhancement includes performing random rotation, random horizontal flipping, random vertical flipping, random lateral translation, random vertical translation, random scaling and random channel conversion on the image.

[0017] Preferably, the convolutional neural network A is ResNets50 with the terminal fully connected network removed.

[0018] Preferably, the network training process of the overall processing branch is:

[0019] The feature map F obtained by the convolutional neural network A is input into the global classification network, and the global processing result y is obtained through overall processing. whole ; Based on the global processing result y whole and the true label y true , calculate the cross entropy loss Loss of the overall processing branch whole ; Train the network by gradient descent.

[0020] Preferably, the network training process in the region of interest processing branch is:

[0021] By processing the cross entropy loss of the branch as a whole whole and the true label y true The contribution of the overall fundus area to the prediction is obtained, and the contribution is normalized to obtain the pseudo label rpn true ; Output the region of interest score rpn through the region of interest network pred , calculate the pseudo label rpn true With the region of interest fraction rpn pred Cross Entropy Loss cam ; Train the network by gradient descent;

[0022] Based on the k highest scores of the region of interest network output, the score of the region of interest RPN pred The corresponding k regions of interest will preprocess the high-definition fundus image Crop and scale the selected region of interest to obtain k fundus images of interest Fundus image of interest Input into the convolutional neural network B, get k local processing scores, and average the scores to get the processing result y of the focus area attention , based on the processing result y of the focus area attention and the true label y true , calculate the cross entropy loss Loss of the branch of interest attention ; Train the network by gradient descent;

[0023] The cross entropy loss Loss of the obtained overall processing branch whole, pseudo label rpn true With the region of interest fraction rpn pred Cross Entropy Loss cam , cross entropy loss Loss of the branch of interest attention Add together to get the final loss value Loss all , whose expression is:

[0024] Loss all =Loss attentnion +Loss cam +Loss whole ,

[0025] The entire network is trained end-to-end using the gradient descent method, and multiple stages are optimized together to ensure that the solution in the final stage is optimal.

[0026] Preferably, the gradient descent method trains the network based on a classification cross entropy loss function, and the classification cross entropy loss function is expressed as:

[0027]

[0028] Where M is the number of categories, y is the actual value, and p is the predicted value.

[0029] Preferably, the gradient descent method is mini-batch gradient descent MBGD, expressed as:

[0030]

[0031] in is the gradient operator, J is the loss function, α is the learning rate, x and y are samples and labels respectively, i is the i-th sample, and n is the number of samples; the label is the annotation category of the data set.

[0032] Preferably, the contribution is calculated using a gradient-weighted class activation heat map based on the Grad-CAM technology, and its expression is:

[0033]

[0034] in represents the weight of the kth feature map for category c, yc represents the probability of category c, represents the pixel at position (i, j) in the kth feature map, It is the gradient weighted class activation heat map of class c. The purpose of ReLU is to add only the part with positive weight when adding the activation maps of each channel at the end, so as to eliminate the interference in the activation heat map that is not related to the target category.

[0035] Preferably, the global classification network includes a global average pooling layer and an output layer fully connected layer, which is activated by a Softmax function, and its output is a category score representing a probability value; the Softmax function expression is:

[0036]

[0037] Among them, x j is the category score of the jth category, and k is the number of categories.

[0038] Preferably, the convolutional neural network B is a region of interest classification network, including a global average layer and a linear classification layer.

[0039] Compared with the prior art, the present invention has the following advantages:

[0040] 1) The present invention designs a method for processing diabetic retinopathy image data combined with self-supervised attention region proposal. By designing a region of interest proposal network to simulate the process of clinicians examining retinal images, the region of interest proposal network can generate candidates for suspicious regions that need more attention with only image-level supervised training, and accurately predict the disease level based on the entire image and its high-resolution suspicious plaques. Compared with the method that does not use region enhancement, the present invention has higher accuracy and robustness;

[0041] 2) The present invention can save computing resources when using high-definition fundus images as input. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0044] The method of this embodiment is expressed in the form of:

[0045] The high-definition fundus image in picture format is input, processed by the method of the present invention, and the output is the grade L of diabetic retinopathy. Disease Grading (five levels) and the confidence level p(x|x∈grade i , i=0...5).

[0046] Diabetic retinopathy grade L Disease Grading The levels are:

[0047] 0-no DR, 1-mild NPDR, 2-moderate NPDR, 3-severe NPDR, 4-PDR.

[0048] Among them, PDR: proliferative diabetic retinopathy, NPDR: non-proliferative diabetic retinopathy, DR: diabetic retinopathy.

[0049] Input: XX_left.jpg

[0050] Output: {'0': 0.812, '1': 0.104, '2': 0.052, , '3': 0.021, '4': 0.011}.

[0051] Combined with Figure 1 The method of this embodiment is described, including:

[0052] Data preprocessing: Get a high-definition fundus image X, and get a picture by noise reduction and detail enhancement After scaling and data enhancement, we get a smaller resolution

[0053] Feature extraction: Input the convolutional neural network A to obtain a series of feature maps F;

[0054] Perform overall branch processing and region of interest branch processing respectively;

[0055] First, the global classification network used for global fundus image processing is used to obtain the global processing result y whole ; The feature map F is also input into the region of interest network to obtain the region of interest, combined with the high-definition pre-processed fundus image Get fundus images of several regions of interest The fundus image of interest is augmented and scaled to the same size, and then input into the convolutional neural network B for local fundus image processing, and the processing result y of the region of interest is obtained by fractional averaging. attention .

[0056] Finally, by processing the result y whole The processing result y of the area of ​​interest attention Average the scores to get the final processing result y pred .

[0057] The training of the network is divided into the training of the overall branch and the training of the branch of interest.

[0058] For the overall branch, first, neural network A and neural network B obtain y through overall processing whole and the true label y trueCalculate the cross entropy loss Loss whole ;

[0059] For the branch of interest, first use the cross entropy loss Loss of the overall branch whole With y true Get the contribution of the overall fundus area to the prediction, normalize it to [0, 1] to get the pseudo label rpn true , the region of interest network outputs the region of interest score rpn pred , by calculating rpn true With RPN pred The cross entropy of Loss cam ;

[0060] Then, we obtain the k RPNs with the highest scores output by the region of interest network. pred The corresponding k regions of interest are cropped and scaled according to the preprocessed high-definition fundus images to obtain k fundus images of interest. And input the convolutional neural network B, and average the k local processing scores to get the processing result y of the focus area attention ,y attention and the true label y true Calculate the cross entropy loss Loss attention ;

[0061] Finally, add the three loss items to get the final loss value Loss all , the expression is:

[0062] Loss all =Loss attentnion +Loss cam +Loss whole

[0063] The entire network is trained end-to-end using the gradient descent method, and multiple stages are optimized together to ensure that the solution in the final stage is optimal.

[0064] Next, the method of the present invention is described in detail.

[0065] 1. Data preprocessing

[0066] 1.1 Noise reduction and detail enhancement:

[0067] ImageProcess(x,y;σ)=αI(x,y)+βG(x,y;σ)*I(x,y)+γ

[0068] Where I(x, y) is the pixel value at the image coordinate (x, y), * represents convolution, and G(x, y; σ) is a Gaussian filter with a=4, β=-4, σ=512 / 30, and γ=128.

[0069] 1.2 Data Enhancement:

[0070] 1) Random rotation: The data image is randomly rotated in the range of [-180, +180] degrees.

[0071] 2) Random horizontal flip: Randomly flip the data image horizontally.

[0072] 3) Random vertical flip: Randomly flip the data image vertically.

[0073] 4) Random lateral shift: The data image is randomly shifted horizontally, and the points outside the input boundary are filled according to the “aaaa|abcd|dddd” pattern in the range of [0.9, 1.1].

[0074] 5) Random vertical translation: The data image is randomly translated vertically, and the points outside the input boundary are filled according to the “aaaa|abcd|dddd” pattern in the range of [0.9, 1.1].

[0075] 6) Random scaling: Randomly scale the data image in length or width, ranging from [0.8, 1.2].

[0076] 7) Random channel conversion: Randomly offset the color channel values ​​of the data image in the range of [x-10, x+10], where x is the specific channel value of the original pixel.

[0077] 1.3 Scaling: Using bilinear interpolation

[0078] 2. Feature extraction

[0079] This embodiment extracts the features of the overall processing branch and the region of interest processing branch through a common feature extraction network, namely, a convolutional neural network A.

[0080] One option of the convolutional neural network A is ResNet50 without the terminal fully connected network, and its structure is shown in Table 1.

[0081] Table 1

[0082]

[0083] 3. Overall processing branch

[0084] After feature extraction, the features are passed through the global classification network to output the global level case classification y whole ,y whole and the true label y true Calculate the cross entropy loss Loss whole .

[0085] Specifically, the output of the last residual block of the convolutional neural network A is connected to a global average pooling layer, and a new output layer fully connected layer is added. After Softmax activation, its output is a 5-category score, where Softmax outputs a value between 0 and 1, similar to the Sigmoid function, which is usually used when the number of classes is greater than 2, and is defined as follows:

[0086]

[0087] The model is trained using the categorical cross entropy loss function. The cross entropy or logarithmic loss function, whose output (class score) is a probability value between 0 and 1, measures the performance of the classification model. The cross entropy loss increases as the predicted output deviates from the actual label. The categorical cross entropy loss is calculated as follows equation (2):

[0088]

[0089] Where M is the number of categories, y is the actual value, and p is the predicted value.

[0090] The optimization algorithm uses mini-batch gradient descent (MBGD). In fact, stochastic gradient descent (SGD) is generally referred to as MBGD. The learning rate is set to 0.01. It is used to update the parameters θ (weights) corresponding to the neural network, as shown in equation (3):

[0091]

[0092] in is the gradient operator, J is the loss function, α is the learning rate, x and y are samples and labels respectively, i is the i-th sample, and n is the number of samples; the label is the annotation category of the data set.

[0093] 4. Region of Interest Processing Branch

[0094] 4.1 Region of Interest Proposal Network

[0095] The ROI processing branch first inputs the feature F obtained by the neural network A into the ROI proposal network. The ROI proposal network is a type of RPN network. The RPN usually generates several bounding boxes on the original image in advance and finally outputs the anchor that is most likely to contain the object, also known as the region candidate. The network is trained by continuously minimizing the deviation between the real object box and the position of the region candidate. Similar to the RPN network, the ROI proposal network only outputs the k candidate regions with the highest confidence scores. In practice, k can be 4. Unlike the RPN network, the purpose of the ROI proposal network is to fit the distribution of the activation map processed as a whole.

[0096] First, the overall processing result y of the overall processing branch is whole and the true label y true Gradients are calculated to calculate the contribution of each region.

[0097] Specifically, a Grad-CAM (Gradient-weighted Class Activation Mapping) technique is used, namely "gradient-weighted class activation heat map", which is expressed as:

[0098]

[0099] in represents the weight of the kth feature map for category c, y c represents the probability of category c, Represents the pixel at position (i, j) in the kth feature map; It is the gradient weighted class activation heat map of class c. The purpose of ReLU is to add only the part with positive weight when adding the activation maps of each channel at the end, so as to eliminate some interference on the activation map that is not related to the target category.

[0100] The obtained activation map is first normalized to the range of [0, 1] and scaled to be consistent with the input size of the region of interest processing branch. Then, the average activation score in the corresponding candidate region is calculated based on the obtained candidate region as the true candidate confidence score rpn true Then the candidate confidence score rpn corresponding to the candidate region obtained by the region of interest network pred Calculate the cross entropy loss Loss cam The network is trained via gradient descent. The cross entropy loss and gradient descent are consistent with the overall processing branch.

[0101] 4.2 Region of Interest Classification Network

[0102] Based on the k RPNs with the highest scores output from the region of interest network in the previous section pred The corresponding k regions of interest are cropped and scaled according to the preprocessed high-definition fundus images to obtain k fundus images of interest. And input the convolutional neural network B, which is the region of interest classification network. The actual implementation can use the convolutional neural network A and add a new global average layer and linear classification layer after the last residual block. Its structure is shown in Table 2 below.

[0103] Table 2

[0104]

[0105] Then the k local processing scores are averaged to get the processing result y of the focus areaattention ,y attention and the true label y true Calculate the cross entropy loss Loss attention The network is trained via gradient descent. The cross entropy loss and gradient descent are consistent with the overall processing branch.

[0106] The scores are averaged using simple average, that is:

[0107]

[0108] in I am interested in fundus images The classification score obtained by the convolutional neural network B, k is the number of regions of interest.

[0109] 5. Final processing classification

[0110] The final result is obtained by averaging the scores:

[0111] Finally, add the losses of the three branches together. all =Loss attentnion +Loss cam +Loss whole , the whole network is trained end-to-end by gradient descent. The gradient descent is consistent with the overall processing branch.

[0112] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for processing diabetic retinopathy image data, characterized in that: The method includes: Data preprocessing: Obtaining high-definition fundus images , after noise reduction and detail enhancement, the pre-processed high-definition fundus image is obtained , and then through scaling and data enhancement, a pre-processed high-definition fundus image with a smaller resolution is obtained ; Feature extraction: Preprocessing high-definition fundus images with lower resolution Input into the convolutional neural network A to obtain a series of feature maps ; Overall processing branch: feature map The global classification network for global fundus image processing is input to obtain the global processing result representing the global hierarchical case classification. ; Region of interest processing branch: feature map Input to the region of interest proposal network to obtain the region of interest, combined with the high-definition pre-processed fundus image , get the fundus image of the region of interest After data enhancement and scaling, it is input into the convolutional neural network B for local fundus image processing to calculate the processing result of the region of interest ; Get the final classification processing results: based on the global processing results Results of processing with the area of ​​interest , calculate the final classification processing result ; The network training process of the overall processing branch is: The feature map obtained by the convolutional neural network A Input into the global classification network, and obtain the global processing result through overall processing Based on the global processing results With the true label , calculate the cross entropy loss of the overall processing branch ; Train the network by gradient descent; The network training process in the region of interest processing branch is as follows: Cross entropy loss by processing the branches as a whole With the true label Get the contribution of the overall fundus area to the prediction, and normalize the contribution to get the pseudo label ; Output the region of interest score through the region of interest network , calculate the pseudo label With the region of interest score The cross entropy ; Train the network by gradient descent; Based on the network output of the region of interest The highest scoring region of interest score Corresponding Regions of interest will be preprocessed into high-definition fundus images According to the selected area of ​​interest, crop and scale to obtain Fundus images of interest ; The fundus image of interest Input to the convolutional neural network B, and get The local processing scores are averaged to get the processing results of the focus area. , based on the processing results of the area of ​​interest With the true label , calculate the cross entropy loss of the branch of interest ; Train the network by gradient descent; The cross entropy loss of the overall processing branch is obtained , pseudo labels With the region of interest score The cross entropy , the cross entropy loss of the branch of interest Add up to get the final loss value , whose expression is: The entire network is trained end-to-end using the gradient descent method, and multiple stages are optimized together to ensure that the solution in the final stage is optimal.

2. The method for processing diabetic retinopathy image data according to claim 1, characterized in that: The data enhancement includes random rotation, random horizontal flipping, random vertical flipping, random lateral translation, random vertical translation, random scaling and random channel conversion of the image.

3. The method for processing diabetic retinopathy image data according to claim 1, characterized in that: The convolutional neural network A is ResNets50 with the terminal fully connected network removed.

4. The method for processing diabetic retinopathy image data according to claim 1, characterized in that: The gradient descent method trains the network based on the classification cross entropy loss function, and the classification cross entropy loss function expression is: in, is the number of categories, is the actual value, is the predicted value.

5. The method for processing diabetic retinopathy image data according to claim 4, characterized in that: The gradient descent method is mini-batch gradient descent MBGD, which is expressed as: in is the gradient operator, is the loss function, is the learning rate, are samples and labels respectively, For the samples, is the number of samples; the label is the label category of the data set.

6. A method for processing diabetic retinopathy image data according to claim 4, characterized in that: The contribution is calculated using a gradient-weighted class activation heat map based on the Grad-CAM technique, and its expression is: in represents the weight of the kth feature map for category c, represents the probability of category c, Indicated in Feature Map The pixel position, is obtained Gradient-weighted class activation heatmaps for classes, The purpose is to add only the parts with positive weights when adding the activation maps of each channel at the end, so as to eliminate the interference in the activation heat map that is not related to the target category.

7. The method for processing diabetic retinopathy image data according to claim 1, characterized in that: The global classification network includes a global average pooling layer and an output layer fully connected layer, which is activated by the Softmax function, and its output is a category score representing a probability value; the Softmax function expression is: in, For the The category scores of the classes, is the number of categories.

8. The method for processing diabetic retinopathy image data according to claim 1, characterized in that: The convolutional neural network B is a region of interest classification network, including a global average layer and a linear classification layer.

Images