A method for identifying and grading diabetic retinopathy based on wide-angle fundus images

Abstract

The present invention discloses a method for identifying and grading diabetic retinopathy based on wide-angle fundus images, comprising the following steps: S1, collecting wide-angle fundus image data of diabetic retinopathy and preprocessing the wide-angle fundus images; S2, inputting the preprocessed wide-angle fundus images into a pre-established quality control neural network model to determine whether there is refractive media opacity. If so, return a report indicating that the output image quality does not meet the standard; S3, based on the quality control neural network model and combined with a multi-scale feature extraction strategy, capture the global structure and local details in the wide-angle fundus images, identify the typical features of common diseases of the diabetic retina, and determine whether there are common fundus diseases. If so, output a prediction report for the corresponding disease; S4, stage the retinopathy of the wide-angle fundus images through a pre-trained staging neural network model and output the staging category with the highest probability.

Description

Technical Field

[0001] The present invention relates to the field of intelligent medical technology, and particularly relates to a method for identifying and grading diabetic retinopathy based on wide-angle fundus images. Background Art

[0002] With the increasing incidence of diabetes year by year, diabetic retinopathy (DR), as one of the most common microvascular complications of diabetes, has become the main cause of vision loss globally. Currently, the early diagnosis and lesion grading of DR in clinical practice mainly rely on the naked-eye observation of ophthalmologists combined with fundus photography technology. However, this method has several significant drawbacks: one is strong subjectivity, and it is difficult to ensure the evaluation consistency among different doctors; the second is low efficiency, and in the face of a large number of patients to be screened, the workload of doctors is heavy; the third is that it is difficult to accurately identify and grade early micro-lesions, and the best treatment opportunity may be missed.

[0003] In recent years, with the rapid development of computer vision and deep learning technologies, automatic recognition and grading methods based on images have shown great potential in the medical field. Although some fundus artificial intelligence (AI) reading technologies have been applied to the screening of DR, this technology provides DR grading results and cannot, like an ophthalmologist, evaluate the severity of DR based on the presence of lesions, which makes patients convinced, and this has hindered the application of deep learning methods in clinical practice. Summary of the Invention

[0004] To solve the above problems, the present invention provides a method for identifying and grading diabetic retinopathy based on wide-angle fundus images.

[0005] The present invention adopts the following technical solutions:

[0006] A method for identifying and grading diabetic retinopathy based on wide-angle fundus images, comprising the following steps:

[0007] S1. Collect wide-angle fundus image data of diabetic retinopathy and preprocess the wide-angle fundus images;

[0008] The preprocessing method of the wide-angle fundus images in step S1 specifically includes:

[0009] S11. Based on an optimized algorithm of feature point matching and image fusion, automatically identify and accurately align each local image, and at the same time eliminate seams through seamless fusion technology to maintain the continuity of image color and brightness;

[0010] S12. According to the image content illumination adaptive adjustment strategy, identify the dark areas and bright areas in the image, and achieve global illumination equalization through a non-linear mapping algorithm while retaining image details;

[0011] S13. Call the scale-vessel detail enhancement filter to enhance the overall contrast of the image, retain and highlight the fine blood vessel structures, and improve the visualization effect of diabetic retinopathy features.

[0012] S2. Input the preprocessed wide-field fundus image into the pre-established quality control neural network model to determine whether there is refractive media opacity. If so, return a report indicating that the output image quality does not meet the standard. If not, proceed to step S3.

[0013] S3. Based on the quality control neural network model and combined with the multi-scale feature extraction strategy, capture the global structure and local details in the wide-field fundus image, identify the typical features of common diseases in diabetic retinopathy, and determine whether there are common fundus diseases. If so, output a prediction report for the corresponding disease. If not, proceed to the staging of diabetic retinopathy.

[0014] S4. Perform retinopathy staging on the wide-field fundus image through a pre-trained staging neural network model and output the staging category with the highest probability.

[0015] S5. Segment the lesions in the wide-field fundus image through pre-trained lesion segmentation models and optic cup and disc segmentation models, and perform pixel-level annotation on the optic cup and disc of the lesions.

[0016] S6. Generate a corresponding screening analysis report according to the staging category and lesion segmentation results.

[0017] Preferably, the quality control neural network model adopts the ResNet50 neural network structure. The prediction results of the quality control neural network model include refractive media opacity, common fundus diseases, and the staging of diabetic retinopathy. Among them, the common fundus diseases include one or more of laser spots, preretinal hemorrhage, vitreous hemorrhage, fibrous proliferation, and traction retinal detachment.

[0018] Preferably, during the training process of the quality control neural network model, Cross Entropy Loss and Asymmetric Loss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. The training period of the quality control neural network model is 100 epochs, and the learning rate is 3e-5.

[0019] Preferably, the staging neural network model adopts the ResNet50 neural network structure. The prediction classifications of the staging neural network model include stage 0, stage 1, stage 2, stage 3, and stage 4, and the output result of the staging neural network model is the classification with the highest probability.

[0020] Preferably, during the training process of the staging neural network model, Cross Entropy Loss and Cost Sensitive Loss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. Among them, Cost Sensitive Loss can be expressed as Lcs = M * P, where M is a 5*5 weight matrix, and the value of the i-th row and j-th column is Mij = |i - j| / 5, and P is the vector representation of the probability predicted for stages 0 - 4. The training period of the quality control neural network model is 100 epochs, and the learning rate is 1e-4.

[0021] Preferably, both the lesion segmentation model and the optic cup and disc segmentation model adopt the semantic segmentation network model structure of EfficientNet-B0. Among them, the segmentation types of image lesions include background, hemorrhage, hard exudate, cotton wool spot, and neovascularization.

[0022] Preferably, during the training process of the lesion segmentation model, Dice Loss and General Union Loss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. Among them, the weighted weight of Dice Loss is inversely proportional to the area of the lesion, and General Union Loss is used to segment small lesions and improve the sensitivity of small lesions.

[0023] Preferably, the training period of the lesion segmentation model is 300 epochs, the learning rate is 0.1 for 0 - 50 epochs, 0.05 for 50 - 150 epochs, 0.01 for 150 - 250 epochs, and 0.001 for 250 - 300 epochs.

[0024] Preferably, Dice Loss and General Union Loss are adopted during the training process of the optic cup and disc segmentation model, and the two are weighted and summed in a certain proportion as the total loss function. During the training process, edge detection is performed on the input image to generate an edge-enhanced image, which is used as an additional input channel and sent into the network for training together with the original image to increase the weight of the edge part, so that the network model can learn more edge features. At the same time, the optic cup and disc segmentation and annotation process is used to locate four quadrants. Taking the midpoint of the segmented optic disc as the origin of the coordinate axis, the x-axis and y-axis are drawn in the horizontal and vertical directions. Based on this, a description of the segmented lesion can be formed.

[0025] After adopting the above technical solutions, compared with the background technology, the present invention has the following advantages:

[0026] The present invention provides a method for identifying and grading diabetic retinopathy based on wide-angle fundus images. By using wide-angle fundus images, the visible range is 6 times that of conventional 45-degree fundus images, covering the retinal edge area, improving the recognition rate of lesions, and ensuring the comprehensiveness and accuracy of lesion analysis. The whole process from image preprocessing, quality control recognition, lesion grading to lesion monitoring is automated. Multi-label classification is performed on common fundus lesions, patients who need referral treatment are analyzed, and key lesions are segmented to provide an interpretable visual analysis basis for lesion grading, assisting doctors in their work and reducing the burden. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flowchart of the method of the present invention;

[0028] Figure 2 It is a distribution diagram of training and test data of the staging neural network model of the present invention;

[0029] Figure 3 It is a result diagram of lesion segmentation of the present invention;

[0030] Figure 4 It is a screening analysis report diagram of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0031] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0032] Embodiment

[0033] As Figures 1 to 4 shown, the present invention discloses a method for identifying and grading diabetic retinopathy based on wide-angle fundus images, including the following steps:

[0034] S1. Collect wide-angle fundus image data of diabetic retinopathy and preprocess the wide-angle fundus images;

[0035] The preprocessing method of the wide-angle fundus images in step S1 specifically includes:

[0036] S11. Based on an optimized algorithm of feature point matching and image fusion, automatically identify and accurately align each local image, and at the same time eliminate seams through seamless fusion technology to maintain the continuity of image color and brightness;

[0037] S12. According to the image content illumination adaptive adjustment strategy, identify the dark areas and bright areas in the image, and achieve global illumination equalization through a non-linear mapping algorithm while retaining image details;

[0038] S13. Call the scale-vessel detail enhancement filter to enhance the overall contrast of the image, retain and highlight the fine blood vessel structures, and improve the visualization effect of diabetic retinopathy features.

[0039] S2. Input the preprocessed wide-field fundus image into the pre-established quality control neural network model to determine whether there is refractive media opacity. If there is, return a report indicating that the output image quality does not meet the standard. If not, proceed to step S3.

[0040] S3. Based on the quality control neural network model and combined with the multi-scale feature extraction strategy, capture the global structure and local details in the wide-field fundus image, identify the typical features of common diseases of the diabetic retina, and determine whether there are common fundus diseases. If there are, output a prediction report for the corresponding disease. If not, proceed to the staging of diabetic retinopathy.

[0041] The quality control neural network model adopts the ResNet50 neural network structure. The prediction results of the quality control neural network model include refractive media opacity, common fundus diseases, the staging of diabetic retinopathy, and other situations. Among them, the common fundus diseases include one or more of laser spots, preretinal hemorrhage, vitreous hemorrhage, fibrous proliferation, and traction retinal detachment.

[0042] In this embodiment, the ResNet50 neural network structure includes an initial convolutional layer, a max pooling layer, four residual block sets, a global average pooling layer, and a fully connected layer. Among them:

[0043] Initial convolutional layer: Perform convolution operations using a 7x7 convolutional kernel with a stride of 2. The number of output channels of this layer is 64.

[0044] Max pooling layer: Usually followed by a max pooling layer after the initial convolutional layer. The pooling window size is 3x3 and the stride is 2.

[0045] Residual block: ResNet50 uses residual blocks to construct deeper networks. Each residual block consists of several layers, including 1x1, 3x3, and 1x1 convolution operations, which are connected by batch normalization and the ReLU activation function between these convolutional layers.

[0046] Four residual block sets: The residual blocks in the ResNet50 network are organized into four sets, namely conv2_x, conv3_x, conv4_x, and conv5_x. The number of residual blocks in each set increases sequentially, with 3, 4, 6, and 3 residual blocks respectively.

[0047] Global average pooling layer: After the last set of residual blocks, a global average pooling layer is usually connected to average the values of each channel in the feature map, reducing the number of parameters in the fully connected layer.

[0048] Fully connected layer: Finally, there is a fully connected layer that converts the output of the global average pooling layer into the final classification result.

[0049] During the training process of the quality control neural network model, Cross Entropy Loss and AsymmetricLoss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. The training cycle of the quality control neural network model is 100 epochs, and the model is iteratively optimized within each epoch. The learning rate is 3e-5. The learning rate is set to 3e-5, which has been experimentally verified to effectively promote the model to converge to a better solution while ensuring the training stability.

[0050] Cross Entropy Loss is responsible for measuring the overall classification accuracy, while Asymmetric Loss alleviates the problem caused by class imbalance by introducing asymmetric penalties for misclassifications, especially increasing the penalty for misclassifications of the minority class (lesion class).

[0051] The training data volume of the quality control neural network model includes four groups of data from four hospitals, namely 2685 groups from Hospital A, 2362 groups from Hospital B, 751 groups from Hospital C, and 312 groups from Hospital D. These four groups of training data include 3917 cases of samples entering the diabetic retinopathy staging, 463 cases of refractive media opacity samples, 423 cases of laser spot samples, 405 cases of preretinal hemorrhage samples, 592 cases of vitreous hemorrhage samples, 494 cases of fibrous proliferation samples, and 458 cases of other diseases.

[0052] Multiple categories can appear in the prediction results of the quality control neural network model. For example, preretinal hemorrhage and fibrous proliferation appear simultaneously. This setting conforms to the clinical reality, and the corresponding disease prediction report is directly output. When only the category of "entering the diabetic retinopathy staging" appears in the prediction, it enters the staging module; if other diseases or the proliferative stage appear, the corresponding disease prediction report is directly output, and no staging and segmentation are performed. The effect of the model is evaluated by the binary classification of whether it enters the diabetic retinopathy staging, with an accuracy of 90.5%, a sensitivity of 95.0%, and an F1 index of 92.5%.

[0053] S4. Perform retinal lesion staging on the wide-angle fundus image through a pre-trained staging neural network model, and output the staging category with the highest probability;

[0054] The staging neural network model adopts the ResNet50 neural network structure. The prediction classifications of the staging neural network model include stage 0, stage 1, stage 2, stage 3, and stage 4, and the output result of the staging neural network model is the classification with the highest probability. The scoring rules for each stage are as follows:

[0055] Stage 0: No diabetic retinopathy.

[0056] Stage 1: The number of bleeding points is less than 4 quadrants, and less than 20 in each quadrant.

[0057] Stage 2: Hard exudates, soft exudates.

[0058] Stage 3: Meeting one of the following conditions: The number of bleeding points is more than 4 quadrants, and more than 20 in each quadrant; venous beading >= 2 quadrants; IRMA >= 1 quadrant.

[0059] Stage 4: Neovascularization.

[0060] The output of the network is the class with the highest probability, rather than multiple classifications. The data includes 2813 cases in the training set + tuning set and 906 cases in the test set, and their staging distributions are as Figure 2 shown.

[0061] During the training process of the staging neural network model, Cross Entropy Loss and Cost Sensitive Loss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. Among them, Cost Sensitive Loss can be expressed as Lcs = M * P, where M is a 5*5 weight matrix, and the value of the i-th row and j-th column is Mij = |i - j| / 5, and P is the vector representation of the probability predicted as stage 0 - 4. This loss function gives higher penalties to more deviated prediction values, that is, the penalty for predicting stage 0 as stage 4 is higher than the penalty for predicting stage 0 as stage 1. After adding this loss, when the prediction is incorrect, the deviation is basically between the previous and next levels. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 1e-4.

[0062] S5. Segment the lesions in the wide-angle fundus image through the pre-trained lesion segmentation model and optic cup and disc segmentation model, and perform pixel-level annotation on the optic cup and disc of the lesions;

[0063] S6. Generate a corresponding screening analysis report according to the staging category and lesion segmentation result.

[0064] Both the lesion segmentation model and the optic cup and disc segmentation model adopt the semantic segmentation network model structure of EfficientNet-B0. Among them, the segmentation types of image lesions include background, hemorrhage, hard exudate, cotton wool spot, and neovascularization. The labeled data of the lesion segmentation model include 803 hemorrhage samples, 507 hard exudate samples, 424 cotton wool spot samples, and 190 neovascularization samples. The labeled data of the optic cup and disc segmentation model are 277 cases.

[0065] During the training process of the lesion segmentation model, Dice Loss and General Union Loss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. Among them, the weighted weight of Dice Loss is inversely proportional to the area of the lesion. Since there are often multiple lesions on the image and the area distribution between lesions is uneven, for example, there are both large hemorrhage patches and hemorrhage points that only occupy a few pixels in hemorrhage. In order to accurately learn the relevant features of small lesions in the case of the coexistence of large and small lesions, we strengthened the weight of the position of small lesions in the loss to offset the impact brought by the uneven area. General Union Loss pays more attention to the mispredicted positions during training. It was originally proposed to deal with the segmentation problem of fine terminal tubular structures and is used to segment small lesions to improve the sensitivity of small lesions.

[0066] The training period of the lesion segmentation model is 300 epochs. The learning rate is 0.1 from 0 to 50 epochs, 0.05 from 50 to 150 epochs, 0.01 from 150 to 250 epochs, and 0.001 from 250 to 300 epochs.

[0067] As Figure 3 shown, the trained lesion segmentation model can identify hemorrhage points and hemorrhage patches (red), hard exudate (yellow), cotton wool spots (orange), and neovascularization (purple). During the training process of the optic cup and disc segmentation model, Dice Loss and General Union Loss are adopted, and the two are weighted and summed in a certain proportion as the total loss function. During the training process, edge detection is performed on the input image and an edge-enhanced image is generated, which is used as an additional input channel and sent into the network for training together with the original image to increase the weight of the edge part, so that the network model can learn more edge features. At the same time, the optic cup and disc segmentation and annotation process is used to locate four quadrants. Taking the midpoint of the segmented optic disc as the origin of the coordinate axis, the x-axis and y-axis are drawn in the horizontal and vertical directions. Based on this, descriptions of the segmented lesions can be formed, such as "hemorrhage patches can be seen in 3 quadrants", "cotton wool spots involve 2 quadrants", etc.

[0068] As described above, it is only the preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims

1. A method for identifying and grading diabetic retinopathy based on wide-angle fundus images, characterized in that: The following steps are involved: S1. Collect wide-angle fundus image data of diabetic retinopathy, and pre-process the wide-angle fundus image, wherein the pre-processing method of the wide-angle fundus image is specifically as follows: S11, based on the optimization algorithm of feature point matching and image fusion, automatically identifies and accurately aligns each local image, and eliminates seams through seamless fusion technology to maintain the continuity of image color and brightness; S12, according to the image content illumination adaptive adjustment strategy, identify the dark and bright areas in the image, and achieve global illumination balance through a nonlinear mapping algorithm while retaining image details; S13, calling the scale vessel detail enhancement filter to enhance the overall contrast of the image, retain and highlight the fine structure of the blood vessels, and improve the visualization effect of the characteristics of diabetic retinopathy; S2, input the pre-processed wide-angle fundus image into the pre-established quality control neural network model to determine whether there is refractive media turbidity. If so, return a report that the output image quality does not meet the standard. If not, proceed to step S3; S3. Based on the quality control neural network model and combined with the multi-scale feature extraction strategy, the global structure and local details in the wide-angle fundus image are captured, the typical characteristics of common diabetic retinopathy diseases are identified, and it is determined whether there are common fundus diseases. If so, a corresponding disease prediction report is output; if not, the diabetic retinopathy staging is entered; S4, staging retinal lesions on wide-angle fundus images using a pre-trained staging neural network model, and outputting the staging category with the highest probability; S5. Segment the lesions in the wide-angle fundus image using the pre-trained lesion segmentation model and the optic cup and optic disc segmentation model, and annotate the optic cup and optic disc of the lesions at the pixel level; S6. Generate a corresponding screening analysis report based on the staging category and lesion segmentation results; The lesion segmentation model and the optic cup and optic disc segmentation model both adopt the semantic segmentation network model structure of EfficientNet-B0, wherein the segmentation types of image lesions include background, hemorrhage, hard infiltration, cotton wool spots and neovascularization. In the training process of the lesion segmentation model, Dice Loss and General Union Loss are used, and the weighted sum of the two is taken as the total loss function in a certain proportion, wherein the weighted weight of the Dice Loss is inversely proportional to the area of ​​the lesion, and the General Union Loss is used to segment small lesions and improve the sensitivity of small lesions. The training cycle of the lesion segmentation model is 300 epochs, the learning rate of 0-50 epoch is 0.1, the learning rate of 50-150 epoch is 0.05, the learning rate of 150-250 epoch is 0.01, and the learning rate of 250-300 epoch is 0.

001. Dice Loss and General Union Loss are used in the training process of the optic cup and optic disc segmentation model. Loss, and the weighted sum of the two in a certain proportion is used as the total loss function. During the training process, the input image is edge detected and an edge enhancement map is generated, which is sent to the network for training together with the original image as an additional input channel to increase the weight of the edge part, so that the network model can learn more edge features. At the same time, the optic cup and optic disc segmentation and annotation process is used to locate the four quadrants, with the midpoint of the segmented optic disc as the origin of the coordinate axis, and the x-axis and y-axis are derived in the horizontal and vertical directions. Based on this, a description of the segmented lesion can be formed.

2. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: The quality control neural network model adopts the ResNet50 neural network structure. The prediction results of the quality control neural network model include refractive media opacity, common fundus diseases, and diabetic retinopathy staging. Among them, the common fundus diseases include one or more of laser spots, anterior retinal hemorrhage, vitreous hemorrhage, fibrosis and traction retinal detachment.

3. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 2, characterized in that: In the training process of the quality control neural network model, Cross Entropy Loss and Asymmetric Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 3e-5.

4. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: The staging neural network model adopts the ResNet50 neural network structure. The prediction classification of the staging neural network model includes stage 0, stage 1, stage 2, stage 3 and stage 4, and the output result of the staging neural network model is the classification with the highest probability.

5. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: In the training process of the phased neural network model, Cross Entropy Loss and Cost Sensitive Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function, where Cost Sensitive Loss is expressed as Lcs = M * P, where M is a 5*5 weight matrix, the value of the i-th row and j-th column is Mij = |ij| / 5, and P is a vector representation of the probability of predicting 0-4 periods. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 1e-4.

Images