A device for classifying diabetic retinopathy based on slice lesion detection

Abstract

The present invention provides a diabetic retinopathy grade classification device based on slice lesion detection, and relates to the technical field of medical image processing. The device performs slice lesion detection on a color fundus image to obtain lesion features; combines the lesion features obtained by the slice lesion detection with the corresponding lesion grade annotation data, trains a diabetic retinopathy grade classification model, and obtains a mapping relationship between the lesion feature data of the color fundus image and the diabetic retinopathy grade; classifies the diabetic retinopathy grade through the trained diabetic retinopathy grade classification model; the device learns the mapping relationship between the lesion features and the lesion grade, effectively improves the accuracy of diabetic retinopathy grade determination, and has important application value for the determination of diabetic retinopathy grade.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and in particular to a device for classifying diabetic retinopathy grades based on slice lesion detection. Background Art

[0002] Diabetic retinopathy (DR) is an eye disease caused by high blood sugar in diabetic patients. Its main pathological features include microaneurysms (MA), hemorrhages (HE), hard exudates (EX) and soft exudates (SE). In clinical medicine, according to the development stage and severity, it is divided into five levels of lesions: normal, mild non-proliferative lesions (NPDR), moderate non-proliferative lesions (NPDR), severe non-proliferative lesions (NPDR) and proliferative lesions (PDR). When the lesion develops to the late stage, it will cause retinal detachment. In severe cases, it may cause glaucoma and eventually lead to blindness. Therefore, effective screening and treatment should be carried out as soon as possible to reduce the probability of blindness caused by diabetic retinopathy.

[0003] With the development of deep learning technology and computer technology, many researchers have used deep convolutional neural networks to solve the problem of diabetic retinopathy grade classification. Among the numerous studies, the following two main solutions have emerged: The first is a traditional deep learning solution, which first processes the fundus image, such as trimming invalid black edges, scaling, histogram equalization, contrast stretching, and color and brightness normalization. The data is then input into the convolutional neural network for training, and the classification results are finally output. The second is an a priori deep learning solution, which first uses an image segmentation algorithm to extract the clinical pathological features of the diabetic retinopathy grade, and then fuses the extracted features with the original image as the input of the classification model, and finally outputs the classification results.

[0004] In traditional deep learning solutions, since high-quality color fundus images have high resolution, directly inputting the entire image into the convolutional neural network without processing requires high computing power of the computer equipment, especially huge demand for GPU memory, and the difficulty of training will increase exponentially. Therefore, the image is usually reduced in size and input into the convolutional neural network, and finally the probability corresponding to each category is output to achieve the purpose of classification. Since the key information for determining the DR level: microaneurysms, will be lost after the image is reduced in size, the accuracy of the diagnosis cannot be guaranteed.

[0005] For a priori deep learning solutions, in the feature extraction stage, since high-quality pixel-level annotated data is difficult to obtain, a small amount of data is usually used as input, and U-NET is used to segment the entire image to extract the lesion features. Considering the computing power and network training difficulty, the image is also reduced before segmentation, which also leads to feature loss. In the image classification stage, compared with traditional deep learning solutions, the introduction of new features will inevitably increase the number of network parameters, which will also have a certain impact on training efficiency. Therefore, the performance of this type of solution still has the potential to be improved. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a diabetic retinopathy grade classification device based on slice lesion detection in view of the above-mentioned deficiencies in the prior art, so as to achieve the classification of diabetic retinopathy grades.

[0007] In order to solve the above technical problems, the technical solution adopted by the present invention is: a diabetic retinopathy grade classification device based on slice lesion detection performs slice lesion detection on a color fundus image to obtain lesion features; the lesion features obtained by the slice lesion detection are combined with the corresponding lesion grade annotation data, and a diabetic retinopathy grade classification model is trained to obtain a mapping relationship between the lesion feature data of the color fundus image and the diabetic retinopathy grade; the diabetic retinopathy grade is classified by the trained diabetic retinopathy grade classification model; specifically, the following steps are included:

[0008] Step 1: preprocess the color fundus image data set with marked levels to obtain fundus image data and slice data;

[0009] By extracting the main content area of ​​the color fundus image, performing CLAHE operation and slicing operation, fundus image data and corresponding slice data are obtained;

[0010] Step 2, performing slice lesion detection on the preprocessed color fundus image to obtain slice lesion feature data;

[0011] For the entire fundus image, the detection results of the same lesion in each slice are combined to form four two-dimensional feature matrices, and the four feature matrices are superimposed to form a 4-channel slice lesion feature data;

[0012] Step 3, synthesizing the slice lesion feature data and the fundus image data to obtain fundus image synthesis data;

[0013] The data of the three channels of the fundus image are superimposed on the lesion feature data to obtain the fundus image composite data of seven channels;

[0014] Step 4: combining the fundus image synthesis data with the grade annotation data;

[0015] Combining the synthesized fundus image data with the graded annotation data corresponding to the fundus image, so that the lesion feature data of the fundus image corresponds to the graded annotation data;

[0016] Step 5: Divide the data set;

[0017] The data set formed by combining the fundus image synthesis data and the graded annotated data is divided into a training set, a validation set, and a test set according to a certain ratio; at the same time, each graded data is divided into these three data sets according to the ratio to reduce the difference between the data sets;

[0018] Step 6: training the diabetic retinopathy grade classification model constructed by the SE-ResNeXt-152 network;

[0019] The training set is input into the diabetic retinopathy grade classification model constructed by the SE-ResNeXt-152 network for training, so that the diabetic retinopathy grade classification model learns the weight of the feature data of each channel in the fundus image synthetic data affecting the grade classification result;

[0020] Step 7: Use the validation set and the test set to test and verify the trained diabetic retinopathy grade classification model;

[0021] Step 8, after the patient's color fundus image is processed in steps 1-4, it is input into the diabetic retinopathy grade classification model tested and verified in step 7 for grade determination;

[0022] The diabetic retinopathy grade classification model judges the input data according to the mapping relationship between the characteristic data of each channel obtained through training and the diabetic retinopathy grade, and finally obtains the patient's lesion grade.

[0023] The beneficial effect of adopting the above technical scheme is that: the diabetic retinopathy grade classification device based on slice lesion detection provided by the present invention retains the key lesion information for determining the diabetic retinopathy grade through slice lesion detection, and then combines the lesion characteristics with the lesion grade, and learns the mapping relationship between the lesion characteristics and the lesion grade through SE-ResNeXt-152 network training, which effectively improves the accuracy of diabetic retinopathy grade determination and has important application value for the determination of diabetic retinopathy grade. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flow chart of grade classification of diabetic retinopathy by a device for grade classification of diabetic retinopathy based on slice lesion detection provided by an embodiment of the present invention;

[0025] Figure 2 The fundus image slice effect diagram provided by the embodiment of the present invention;

[0026] Figure 3 Provided for the embodiment of the present invention is an invalid slice feature marking effect diagram, wherein (a) is an MA feature matrix, (b) is an HE feature matrix, (c) is an EX feature matrix, and (d) is an SE feature matrix;

[0027] Figure 4 Provided for the embodiment of the present invention are effect diagrams of slice lesion feature matrices, (a) is a MA feature matrix, (b) is a HE feature matrix, (c) is a) EX feature matrix, and (d) is a SE feature matrix;

[0028] Figure 5 Provided for the embodiment of the present invention is a synthetic effect diagram of the characteristics of the lesion in the slice;

[0029] Figure 6 Provided for the embodiment of the present invention is a fundus image synthesis data effect diagram. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0031] This embodiment runs the diabetic retinopathy grade classification device based on slice lesion detection of this application on a PC with Windows 10 operating system installed. The CPU used is an Intel Core i76700 processor with 4 cores and 8 threads, a basic frequency of 3.4 GHz and a dynamic acceleration frequency of 4.0 GHz. The GPU is an NVIDIA GTX 1080Ti, which has 3584 CUDA cores and 11 GB video memory to support deep neural network operations. The PC is also equipped with 16 GB of memory and a 1 TB hard drive.

[0032] In this embodiment, a diabetic retinopathy grade classification device based on slice lesion detection performs slice lesion detection on a color fundus image to obtain lesion features; combines the lesion features obtained by the slice lesion detection with the corresponding lesion grade annotation data, trains a diabetic retinopathy grade classification model, and obtains a mapping relationship between the lesion feature data of the color fundus image and the diabetic retinopathy grade; classifies the diabetic retinopathy grade by using the trained diabetic retinopathy grade classification model; in this embodiment, the device classifies the diabetic retinopathy grade, such as Figure 1 , specifically including the following steps:

[0033] Step 1: preprocess the data set of the annotated color fundus image (i.e., diabetic retinopathy image) to obtain fundus image data and slice data;

[0034] By extracting the main content area of ​​the color fundus image, performing CLAHE operation and slicing operation, fundus image data and corresponding slice data are obtained;

[0035] EyePACS is a dataset of diabetic retinopathy image-level annotations. This embodiment uses this dataset to train a diabetic retinopathy classification model. This dataset annotates color fundus images into five categories, where 0 represents normal, 1 represents mild non-proliferative lesions, 2 represents moderate non-proliferative lesions, 3 represents severe non-proliferative lesions, and 4 represents proliferative lesions. Due to the inconsistency of the camera model, lighting conditions, and the level of the staff taking the fundus images, there are problems such as image resolution and exposure quality in this dataset. Therefore, data preprocessing of the EyePACS dataset is particularly important. In this embodiment, the data preprocessing of the EyePACS dataset is specifically as follows:

[0036] (1) Extraction of main content area

[0037] In color fundus images, there are many black pixel areas around the retinal image, which cannot provide effective information for the DR grade classification. If it is input into the network, it will increase the amount of calculation during training. Therefore, the non-effective area at the edge is cropped off and only the main content of the retinal image is retained.

[0038] (2) Perform CLAHE operation

[0039] The full name of the CLAHE algorithm is the Contrast Limited Adaptive Histogram Equalization algorithm, which helps to unify the image brightness, while enhancing the local contrast and highlighting the details of the image and effectively limiting the noise. The lesion area in the fundus image data obtained by the CLAHE operation is more prominent, which is helpful for feature extraction.

[0040] (3) Perform slicing operations

[0041] The fundus image after the CLAHE operation is cut from the pixel position (0,0) from left to right and from top to bottom into slices of size 256×256 to obtain slice data of the fundus image. Figure 2 shown.

[0042] Step 2, performing slice lesion detection on the preprocessed color fundus image to obtain slice lesion feature data;

[0043] Slice lesion detection is essentially a binary classification operation, that is, detecting whether the input slice data contains microaneurysms (MA), hemorrhages (HE), hard exudates (EX) and soft exudates (SE), so the output result of slice lesion detection is 0 or 1. 0 means that the corresponding pathological feature is not detected in the slice, and 1 means that the slice contains corresponding lesion information. After each slice data undergoes MA, HE, EX, and SE lesion detection, for each pathological feature, there is a corresponding two-dimensional feature matrix used to record the detection results of the lesion for each slice of the color fundus image; for the entire fundus image, the detection results of the same lesions in each slice are combined to form 4 two-dimensional feature matrices, and the four feature matrices are superimposed to form a 4-channel slice lesion feature data;

[0044] In this embodiment, the specific process of performing slice lesion detection and obtaining slice lesion feature data is as follows:

[0045] 1) Before lesion detection, the validity of the slices at the four corners of the fundus image is determined. When the slice is confirmed to be invalid, the values ​​of the corresponding coordinate positions in each two-dimensional feature matrix are recorded as 0 according to the slice coordinates in the original image. The effect of invalid slice feature marking is as follows Figure 3 shown.

[0046] 2) Perform MA, HE, EX, and SE slice lesion detection on each of the remaining slices. When a corresponding lesion exists, 1 will be output as the result, and it will be recorded in the lesion feature matrix at the corresponding position. The slice lesion feature matrix is ​​as follows: Figure 4 shown.

[0047] 3) Drawing on the idea of ​​padding images during convolution operations in convolutional neural networks, this embodiment fills the feature matrix with 0 values ​​around it, so that the size of the feature matrix is ​​unified to 32×32. To prevent the 0 value filling from being confused with the 0 value in the original slice detection result, the present invention modifies the 0 in the original feature matrix to 10 and the 1 to 255 before 0 filling. Four feature matrices of size 32×32 are obtained, which are filled with 0 values ​​around them.

[0048] 4) The four feature matrices are superimposed together to synthesize a 4-channel slice lesion feature data. The synthesis effect of the slice lesion feature is as follows: Figure 5 shown.

[0049] Step 3, synthesizing the slice lesion feature data and the fundus image data to obtain fundus image synthesis data;

[0050] Inspired by the matrix superposition operation, the lesion feature data and the corresponding fundus image data are synthesized. The data of the three channels of the fundus image are superimposed on the lesion feature data to obtain the 7-channel fundus image synthesis data;

[0051] In this embodiment, the size of the preprocessed color fundus image is reduced to 64×64; and then the feature matrix is ​​padded to a size of 64×64 using a padding operation. Since the color fundus image is divided into three channels (red, green, and blue), when synthesizing data in this embodiment, the data of these three channels are superimposed on the four feature matrices, and the data size is 64×64×7. The synthesized fundus image data is as follows: Figure 6 shown.

[0052] Step 4: combining the fundus image synthesis data with the grade annotation data;

[0053] Combining the synthesized fundus image data with the graded annotation data corresponding to the fundus image, so that the lesion feature data of the fundus image corresponds to the graded annotation data;

[0054] Step 5: Divide the data set;

[0055] The data set formed by combining the fundus image synthesis data and the graded annotated data is divided into a training set, a validation set, and a test set according to a certain ratio; at the same time, each graded data is divided into these three data sets according to the ratio to reduce the difference between the data sets;

[0056] In this embodiment, the combined data set is divided into a training set, a validation set, and a test set in a ratio of 6:2:2.

[0057] Step 6: training the diabetic retinopathy grade classification model constructed by the SE-ResNeXt-152 network;

[0058] The training set is input into the diabetic retinopathy grade classification model constructed by the SE-ResNeXt-152 network for training, so that the diabetic retinopathy grade classification model learns the weight of the feature data of each channel in the fundus image synthetic data affecting the grade classification result;

[0059] In this embodiment, a SE-ResNeXt-152 deep neural network model is first constructed.

[0060] The ResNeXt-152 network has excellent performance in classification tasks. SENet is a new network structure that uses a strategy called "feature recalibration". The main idea is to find the mapping relationship from the perspective of feature channels, and can well learn the weight of the feature data of each channel on the classification result, which is highly consistent with the characteristics of the data used to train the DR grade classification model in the present invention. And the SE module is not a complete network structure, but a substructure that can be embedded in other classification or detection models. Therefore, the present invention selects the ResNeXt-152 network embedded with the SE module, namely SE-ResNeXt-152, to build the model.

[0061] Then input the training set obtained in step 5 into the SE-ResNeXt-152 deep neural network model for training.

[0062] During the training process, the model can learn the mapping relationship between the lesion features in the synthetic data of the color fundus image and its corresponding grade. In order to reduce the oscillation during gradient descent and improve the training speed and efficiency, the Adam algorithm is used to optimize the network. For the classification of diabetic retinopathy grades, it is essentially a five-class classification problem, so the cross entropy error is used as the loss metric during network training.

[0063] Step 7: Use the validation set and the test set to test and verify the trained diabetic retinopathy grade classification model;

[0064] In order to verify the effectiveness of the diabetic retinopathy grade classification model, this example uses fundus images, slice lesion features, and slice lesion features superimposed on fundus images as inputs to the SE-ResNeXt-152 network. The experimental results show that when the slice lesion features superimposed on fundus images are used as data to train the model, the accuracy, recall rate, and specificity are better than the model that is not trained using slice lesion detection and fundus images. Therefore, the diabetic retinopathy grade classification model is effective and improves the accuracy of diabetic retinopathy grade classification.

[0065] Step 8, after the patient's color fundus image is processed in steps 1-4, it is input into the diabetic retinopathy grade classification model tested and verified in step 7 for grade determination;

[0066] The diabetic retinopathy grade classification model judges the input data according to the mapping relationship between the characteristic data of each channel obtained through training and the diabetic retinopathy grade, and finally obtains the patient's lesion grade.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A method for classifying diabetic retinopathy grades based on slice lesion detection, characterized in that: include: Step 1, preprocessing the color fundus image data set with annotated levels to obtain color fundus image data and slice data; Step 2, performing slice lesion detection on the preprocessed color fundus image to obtain slice lesion feature data; Step 3, synthesizing the slice lesion feature data and the color fundus image data to obtain fundus image synthesis data; Step 4: combining the synthesized fundus image data with the graded annotation data corresponding to the color fundus image, so that the lesion feature data of the fundus image corresponds to the graded annotation data; Step 5: Divide the data set; The data set formed by combining the fundus image synthesis data and the graded annotated data is divided into a training set, a validation set, and a test set according to a certain ratio; at the same time, each graded data is divided into these three data sets according to the ratio to reduce the difference between the data sets; Step 6: training the diabetic retinopathy grade classification model constructed by the SE-ResNeXt-152 network; Step 7: Use the validation set and the test set to test and verify the trained diabetic retinopathy grade classification model; Step 8: After the patient's color fundus image is processed in steps 1-4, it is input into the diabetic retinopathy grade classification model tested and verified in step 7 for grade determination.

2. The method for classifying diabetic retinopathy grades based on slice lesion detection according to claim 1, characterized in that: The step 1 obtains the color fundus image data and the corresponding slice data by extracting the main content area of ​​the color fundus image, performing CLAHE operation and slicing operation.

3. The method for classifying diabetic retinopathy grades based on slice lesion detection according to claim 1, characterized in that: The step 2 combines the detection results of the same lesion in each slice in the entire color fundus image to form four two-dimensional feature matrices, and then superimposes the four feature matrices to form a 4-channel slice lesion feature data.

4. The method for classifying diabetic retinopathy grades based on slice lesion detection according to claim 3, characterized in that: The step 3 superimposes the data of the three channels of the color fundus image on the lesion feature data to obtain seven channels of fundus image synthesis data.

5. The method for classifying diabetic retinopathy grades based on slice lesion detection according to claim 4, characterized in that: In step 6, the training set is input into the diabetic retinopathy grade classification model constructed by the SE-ResNeXt-152 network for training, so that the diabetic retinopathy grade classification model learns the weight of the influence of the feature data of each channel in the fundus image synthetic data on the grade classification result.

6. The method for classifying diabetic retinopathy grades based on slice lesion detection according to claim 5, characterized in that: The SE-ResNeXt-152 network is a ResNeXt-152 network embedded with a SE module; the SE module is not a complete network structure, but a substructure that can be embedded in other classification or detection models.

7. The method for classifying diabetic retinopathy grades based on slice lesion detection according to claim 6, characterized in that: The diabetic retinopathy grade classification model judges the input data according to the mapping relationship between each channel characteristic data obtained through training and the diabetic retinopathy grade, and finally obtains the patient's lesion grade.

Images