Diabetic retinopathy recognition and grading method based on wide-angle fundus image

Abstract

The invention discloses a diabetic retinopathy recognition and grading method based on a wide-angle fundus image, and the method comprises the following steps: S1, collecting the wide-angle fundus image data of diabetic retinopathy, and carrying out the preprocessing of the wide-angle fundus image; s2, inputting the preprocessed wide-angle eye fundus image into a pre-established quality control neural network model, judging whether refractive interstitial opacification exists or not, and if yes, returning and outputting an image quality substandard report; s3, on the basis of the quality control neural network model and in combination with a multi-scale feature extraction strategy, capturing a global structure and local details in the wide-angle fundus image, identifying typical features of common disease species of diabetic retina, judging whether the common disease species of the diabetic retina exist or not, and outputting a corresponding disease species prediction report if the common disease species of the diabetic retina exist; and S4, performing retinopathy staging on the wide-angle fundus image through a pre-trained staging neural network model, and outputting a staging category with the highest probability.

Description

Technical Field

[0001] The present invention relates to the field of smart medical technology, and in particular to a method for identifying and grading diabetic retinopathy based on wide-angle fundus images. Background Art

[0002] With the increasing incidence of diabetes, diabetic retinopathy (DR), as one of the most common microvascular complications of diabetes, has become the main cause of vision loss worldwide. At present, the early diagnosis and lesion classification of DR in clinical practice mainly rely on ophthalmologists' naked eye observation combined with fundus photography technology. However, this method has several significant disadvantages: first, it is highly subjective, and it is difficult to ensure the consistency of evaluation between different doctors; second, it is inefficient, and doctors are burdened with a large number of patients to be screened; third, it is difficult to accurately identify and grade early microlesions, which may miss the best time for treatment.

[0003] In recent years, with the rapid development of computer vision and deep learning technology, image-based automatic recognition and grading methods have shown great potential in the medical field. Although some fundus artificial intelligence (AI) reading technologies have been applied to DR screening, the DR grading results provided by this technology cannot convince patients by assessing the severity of DR by the presence of lesions like ophthalmologists, which hinders the clinical application of deep learning methods. Summary of the invention

[0004] To solve the above problems, the present invention provides a method for identifying and grading diabetic retinopathy based on wide-angle fundus images.

[0005] The present invention adopts the following technical solutions: A method for identifying and grading diabetic retinopathy based on wide-angle fundus images comprises the following steps: S1, collecting wide-angle fundus image data of diabetic retinopathy, and preprocessing the wide-angle fundus image; The method for preprocessing the wide-angle fundus image in step S1 specifically includes: S11, based on the optimization algorithm of feature point matching and image fusion, automatically identifies and accurately aligns each local image, and eliminates seams through seamless fusion technology to maintain the continuity of image color and brightness; S12, according to the image content illumination adaptive adjustment strategy, identify the dark and bright areas in the image, and achieve global illumination balance through a nonlinear mapping algorithm while retaining image details; S13. Calling the scale vessel detail enhancement filter to enhance the overall contrast of the image, retain and highlight the fine structure of the blood vessels, and improve the visualization effect of the characteristics of diabetic retinopathy.

[0006] S2, input the pre-processed wide-angle fundus image into the pre-established quality control neural network model to determine whether there is refractive media turbidity. If so, return a report that the output image quality does not meet the standard. If not, proceed to step S3; S3. Based on the quality control neural network model and combined with the multi-scale feature extraction strategy, the global structure and local details in the wide-angle fundus image are captured, the typical characteristics of common diabetic retinopathy diseases are identified, and it is determined whether there are common fundus diseases. If so, a corresponding disease prediction report is output; if not, the diabetic retinopathy staging is entered; S4, staging retinal lesions on wide-angle fundus images using a pre-trained staging neural network model, and outputting the staging category with the highest probability; S5. Segment the lesions in the wide-angle fundus image using the pre-trained lesion segmentation model and the optic cup and optic disc segmentation model, and annotate the optic cup and optic disc of the lesions at the pixel level; S6. Generate a corresponding screening analysis report based on the staging category and lesion segmentation results.

[0007] Preferably, the quality control neural network model adopts the ResNet50 neural network structure, and the prediction results of the quality control neural network model include refractive media opacities, common fundus diseases, diabetic retinopathy stages and other conditions, wherein the common fundus diseases include one or more of laser spots, anterior retinal hemorrhage, vitreous hemorrhage, fibrosis and retinal detachment.

[0008] Preferably, in the training process of the quality control neural network model, Cross Entropy Loss and Asymmetric Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 3e-5.

[0009] Preferably, the staging neural network model adopts the ResNet50 neural network structure, the prediction classification of the staging neural network model includes stage 0, stage 1, stage 2, stage 3 and stage 4, and the output result of the staging neural network model is the classification with the highest probability.

[0010] Preferably, in the training process of the phased neural network model, Cross Entropy Loss and Cost Sensitive Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function, wherein Cost Sensitive Loss can be expressed as Lcs = M * P, wherein M is a 5*5 weight matrix, the value of the i-th row and j-th column is Mij= |ij| / 5, and P is a vector representation of the probability of predicting 0-4 periods. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 1e-4.

[0011] Preferably, the lesion segmentation model and the optic cup and optic disc segmentation model both adopt the semantic segmentation network model structure of EfficientNet-B0, wherein the segmentation types of image lesions include background, hemorrhage, hard infiltration, cotton wool spots and neovascularization.

[0012] Preferably, in the training process of the lesion segmentation model, Dice Loss and General Union Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function, wherein the weighted weight of the Dice Loss is inversely proportional to the area of ​​the lesion, and the General Union Loss is used to segment small lesions and improve the sensitivity of small lesions.

[0013] Preferably, the training cycle of the lesion segmentation model is 300 epochs, the learning rate of 0-50 epochs is 0.1, the learning rate of 50-150 epochs is 0.05, the learning rate of 150-250 epochs is 0.01, and the learning rate of 250-300 epochs is 0.001.

[0014] Preferably, Dice Loss and General Union Loss are used in the training process of the optic cup and optic disc segmentation model, and the weighted sum of the two in a certain ratio is used as the total loss function. During the training process, edge detection is performed on the input image and an edge enhancement map is generated, which is sent to the network for training together with the original image as an additional input channel, and the weight of the edge part is increased, so that the network model learns more edge features. At the same time, the optic cup and optic disc segmentation and annotation process is used to locate the four quadrants, with the midpoint of the segmented optic disc as the origin of the coordinate axis, and the x-axis and y-axis are derived in the horizontal and vertical directions, based on which a description of the segmented lesion can be formed.

[0015] After adopting the above technical solution, the present invention has the following advantages compared with the background technology: The present invention provides a method for identifying and grading diabetic retinopathy based on wide-angle fundus images. The method utilizes wide-angle fundus images, whose visible range is 6 times that of conventional 45-degree fundus images, covers the retinal edge area, improves the recognition rate of lesions, and ensures the comprehensiveness and accuracy of lesion analysis. The method automates the entire process of image preprocessing, quality control identification, lesion grading, and lesion monitoring, performs multi-label classification on common fundus lesions, analyzes patients who need referral treatment, and performs lesion segmentation on key lesions, providing an explainable visual analysis basis for lesion grading, assisting doctors in their work and reducing their burden. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the method of the present invention; Figure 2 This is a distribution diagram of training and testing data for the phased neural network model of the present invention; Figure 3 This is the lesion segmentation result diagram of the present invention; Figure 4 This is a screening analysis report diagram of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0018] Example like Figures 1 to 4 As shown, the present invention discloses a method for identifying and grading diabetic retinopathy based on wide-angle fundus images, comprising the following steps: S1, collecting wide-angle fundus image data of diabetic retinopathy, and preprocessing the wide-angle fundus image; The preprocessing method of the wide-angle fundus image in step S1 specifically includes: S11, based on the optimization algorithm of feature point matching and image fusion, automatically identifies and accurately aligns each local image, and eliminates seams through seamless fusion technology to maintain the continuity of image color and brightness; S12, according to the image content illumination adaptive adjustment strategy, identify the dark and bright areas in the image, and achieve global illumination balance through a nonlinear mapping algorithm while retaining image details; S13. Calling the scale vessel detail enhancement filter to enhance the overall contrast of the image, retain and highlight the fine structure of the blood vessels, and improve the visualization effect of the characteristics of diabetic retinopathy.

[0019] S2, input the pre-processed wide-angle fundus image into the pre-established quality control neural network model to determine whether there is refractive media turbidity. If so, return a report that the output image quality does not meet the standard. If not, proceed to step S3; S3. Based on the quality control neural network model and combined with the multi-scale feature extraction strategy, the global structure and local details in the wide-angle fundus image are captured, the typical characteristics of common diabetic retinopathy are identified, and the presence of common fundus diseases is determined. If so, the corresponding disease prediction report is output. If not, the diabetic retinopathy staging is entered; The quality control neural network model adopts the ResNet50 neural network structure. The prediction results of the quality control neural network model include refractive media opacities, common fundus diseases, diabetic retinopathy stages and other conditions. Among them, common fundus diseases include one or more of laser spots, anterior retinal hemorrhage, vitreous hemorrhage, fibrosis and traction retinal detachment.

[0020] In this embodiment, the ResNet50 neural network structure includes an initial convolution layer, a maximum pooling layer, four residual block set layers, a global average pooling layer, and a fully connected layer. Among them: Initial convolution layer: Use a 7x7 convolution kernel for convolution operation with a stride of 2. The number of output channels of this layer is 64.

[0021] Max pooling layer: The initial convolutional layer is usually followed by a max pooling layer with a pooling window size of 3x3 and a stride of 2.

[0022] Residual blocks: ResNet50 uses residual blocks to build deeper networks. Each residual block consists of several layers, including 1x1, 3x3, and 1x1 convolution operations, and these convolution layers are connected through batch normalization and ReLU activation functions.

[0023] Four residual block set layers: The residual blocks in the ResNet50 network are organized into four sets, namely conv2_x, conv3_x, conv4_x, and conv5_x. The number of residual blocks in each set increases successively, with 3, 4, 6, and 3 residual blocks respectively.

[0024] Global average pooling layer: After the last set of residual blocks, a global average pooling layer is usually used to average the values ​​of each channel of the feature map to reduce the number of parameters of the fully connected layer.

[0025] Fully connected layer: The last layer is a fully connected layer that converts the output of the global average pooling layer into the final classification result.

[0026] In the training process of the quality control neural network model, Cross Entropy Loss and Asymmetric Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function. The training cycle of the quality control neural network model is 100 epochs, and the model is iteratively optimized in each epoch. The learning rate is 3e-5. The learning rate is set to 3e-5. This learning rate has been verified by experiments and can effectively promote the model to converge to a better solution while ensuring the stability of training.

[0027] Cross Entropy Loss is responsible for measuring the accuracy of the overall classification, while Asymmetric Loss introduces asymmetric penalties for misclassification, especially increasing the penalty for misclassification of minority classes (lesion classes) to alleviate the problems caused by class imbalance.

[0028] The training data volume of the quality control neural network model includes four groups of data from four hospitals, namely 2685 groups from Hospital A, 2362 groups from Hospital B, 751 groups from Hospital C, and 312 groups from Hospital D. These four groups of training data include 3917 samples entering the diabetic retinopathy staging group, 463 samples of refractive media opacity, 423 samples of laser spots, 405 samples of anterior retinal hemorrhage, 592 samples of vitreous hemorrhage, 494 samples of fibrosis, and 458 samples of other diseases.

[0029] The prediction results of the quality control neural network model can have multiple categories, such as the simultaneous occurrence of anterior retinal hemorrhage and fibrosis. This setting conforms to the actual clinical situation and directly outputs the corresponding disease prediction report. When there is only the category of "entering diabetic retinopathy staging" in the prediction, it enters the staging module; if other diseases or proliferation stages appear, it directly corresponds to the disease prediction report, and no staging and segmentation are performed. The effect of the model is evaluated by the binary classification of whether to enter the diabetic retinopathy staging, with an accuracy rate of 90.5%, a sensitivity of 95.0%, and an F1 index of 92.5%.

[0030] S4, staging retinal lesions on wide-angle fundus images using a pre-trained staging neural network model, and outputting the staging category with the highest probability; The staging neural network model uses the ResNet50 neural network structure. The prediction classification of the staging neural network model includes stage 0, stage 1, stage 2, stage 3 and stage 4, and the output result of the staging neural network model is the classification with the highest probability. The scoring rules for each stage are as follows: Stage 0: No diabetic retinopathy.

[0031] Stage 1: Bleeding points < 4 quadrants, < 20 in each quadrant.

[0032] Stage 2: hard exudate, soft exudate.

[0033] Stage 3: Meets one of the following conditions: bleeding points > 4 quadrants, > 20 in each quadrant; venous beading >= 2 quadrants; IRMA >= 1 quadrant.

[0034] Stage 4: Neovascularization.

[0035] The output of the network is the category with the highest probability, rather than multiple categories. The data includes 2813 cases in the training set + tuning set and 906 cases in the test set. The distribution of the stages is as follows Figure 3 shown.

[0036] In the training process of the phased neural network model, Cross Entropy Loss and Cost Sensitive Loss are used, and the weighted sum of the two is taken as the total loss function in a certain proportion. Among them, Cost Sensitive Loss can be expressed as Lcs = M * P, where M is a 5*5 weight matrix, the value of the i-th row and j-th column is Mij = |ij| / 5, and P is the vector representation of the probability of predicting 0-4 periods. This loss function gives a higher penalty to the prediction value with a larger deviation, that is, the penalty for predicting 0 period as 4 period is higher than the penalty for predicting 0 period as 1 period. After adding this loss, when the prediction is wrong, the deviation is basically between the previous and next levels. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 1e-4.

[0037] S5. Segment the lesions in the wide-angle fundus image using the pre-trained lesion segmentation model and the optic cup and optic disc segmentation model, and annotate the optic cup and optic disc of the lesions at the pixel level; S6. Generate a corresponding screening analysis report based on the staging category and lesion segmentation results.

[0038] The lesion segmentation model and the optic cup and optic disc segmentation model both use the semantic segmentation network model structure of EfficientNet-B0, in which the segmentation types of image lesions include background, hemorrhage, hard infiltration, cotton wool spots and neovascularization. The labeled data of the lesion segmentation model include 803 hemorrhage samples, 507 hard infiltration samples, 424 cotton wool spots samples and 190 neovascularization samples. The optic cup and optic disc segmentation model has 277 labeled data.

[0039] In the training process of the lesion segmentation model, Dice Loss and General Union Loss are used, and the weighted sum of the two is taken as the total loss function in a certain proportion. The weight of Dice Loss is inversely proportional to the area of ​​the lesion. Since there are often multiple lesions on the image, and the area distribution between the lesions is uneven, for example, there are both large hemorrhage spots and hemorrhage spots that only occupy a few pixels in size. In order to accurately learn the relevant features of small lesions when large and small lesions exist at the same time, we strengthen the weight of the position of small lesions in the loss to offset the impact of uneven area. General Union Loss pays more attention to the position of prediction errors during training. It was originally proposed to deal with the problem of segmenting small terminal tubular structures. It is used to segment small lesions and improve the sensitivity of small lesions.

[0040] The training cycle of the lesion segmentation model is 300 epochs, the learning rate is 0.1 for 0-50 epochs, 0.05 for 50-150 epochs, 0.01 for 150-250 epochs, and 0.001 for 250-300 epochs.

[0041] like Figure 3 As shown in the figure, the trained lesion segmentation model can identify bleeding spots and hemorrhagic spots (red), hard exudates (yellow), cotton-wool spots (orange) and neovascularization (purple). Dice Loss and General Union Loss are used in the training process of the optic cup and optic disc segmentation model, and the weighted sum of the two in a certain proportion is used as the total loss function. During the training process, the input image is edge detected and an edge enhancement map is generated, which is sent to the network for training together with the original image as an additional input channel to increase the weight of the edge part, so that the network model can learn more edge features. At the same time, the optic cup and optic disc segmentation and annotation process is used to locate the four quadrants, with the midpoint of the segmented optic disc as the origin of the coordinate axis, and the x-axis and y-axis are derived in the horizontal and vertical directions. Based on this, a description of the segmented lesions can be formed, such as "hemorrhagic spots can be seen in 3 quadrants", "cotton-wool spots involve 2 quadrants", etc.

[0042] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for identifying and grading diabetic retinopathy based on wide-angle fundus images, characterized in that: The following steps are involved: S1. Collect wide-angle fundus image data of diabetic retinopathy, and pre-process the wide-angle fundus image, wherein the pre-processing method of the wide-angle fundus image is specifically as follows: S11, based on the optimization algorithm of feature point matching and image fusion, automatically identifies and accurately aligns each local image, and eliminates seams through seamless fusion technology to maintain the continuity of image color and brightness; S12, according to the image content illumination adaptive adjustment strategy, identify the dark and bright areas in the image, and achieve global illumination balance through a nonlinear mapping algorithm while retaining image details; S13, calling the scale vessel detail enhancement filter to enhance the overall contrast of the image, retain and highlight the fine structure of the blood vessels, and improve the visualization effect of the characteristics of diabetic retinopathy; S2, input the pre-processed wide-angle fundus image into the pre-established quality control neural network model to determine whether there is refractive media turbidity. If so, return a report that the output image quality does not meet the standard. If not, proceed to step S3; S3. Based on the quality control neural network model and combined with the multi-scale feature extraction strategy, the global structure and local details in the wide-angle fundus image are captured, the typical characteristics of common diabetic retinopathy diseases are identified, and it is determined whether there are common fundus diseases. If so, a corresponding disease prediction report is output; if not, the diabetic retinopathy staging is entered; S4, staging retinal lesions on wide-angle fundus images using a pre-trained staging neural network model, and outputting the staging category with the highest probability; S5. Segment the lesions in the wide-angle fundus image using the pre-trained lesion segmentation model and the optic cup and optic disc segmentation model, and annotate the optic cup and optic disc of the lesions at the pixel level; S6. Generate a corresponding screening analysis report based on the staging category and lesion segmentation results.

2. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: The quality control neural network model adopts the ResNet50 neural network structure. The prediction results of the quality control neural network model include refractive media opacity, common fundus diseases, diabetic retinopathy stages and other conditions, wherein the common fundus diseases include one or more of laser spots, anterior retinal hemorrhage, vitreous hemorrhage, fibrosis and traction retinal detachment.

3. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 2, characterized in that: In the training process of the quality control neural network model, Cross Entropy Loss and Asymmetric Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 3e-5.

4. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: The staging neural network model adopts the ResNet50 neural network structure. The prediction classification of the staging neural network model includes stage 0, stage 1, stage 2, stage 3 and stage 4, and the output result of the staging neural network model is the classification with the highest probability.

5. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: In the training process of the phased neural network model, Cross Entropy Loss and Cost Sensitive Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function, where Cost Sensitive Loss can be expressed as Lcs = M * P, where M is a 5*5 weight matrix, the value of the i-th row and j-th column is Mij = |ij| / 5, and P is a vector representation of the probability of predicting 0-4 periods. The training cycle of the quality control neural network model is 100 epochs, and the learning rate is 1e-4.

6. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 1, characterized in that: The lesion segmentation model and the optic cup and optic disc segmentation model both adopt the semantic segmentation network model structure of EfficientNet-B0, wherein the segmentation types of image lesions include background, hemorrhage, hard infiltration, cotton wool spots and neovascularization.

7. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 6, characterized in that: In the training process of the lesion segmentation model, Dice Loss and General Union Loss are used, and the weighted sum of the two in a certain proportion is used as the total loss function, wherein the weighted weight of the Dice Loss is inversely proportional to the area of ​​the lesion, and the General Union Loss is used to segment small lesions and improve the sensitivity of small lesions.

8. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 7, characterized in that: The training cycle of the lesion segmentation model is 300 epochs, the learning rate of 0-50 epoch is 0.1, the learning rate of 50-150 epoch is 0.05, the learning rate of 150-250 epoch is 0.01, and the learning rate of 250-300 epoch is 0.

001.

9. The method for identifying and grading diabetic retinopathy based on wide-angle fundus images according to claim 8, characterized in that: In the training process of the optic cup and optic disc segmentation model, Dice Loss and General Union Loss are used, and the weighted sum of the two in a certain ratio is used as the total loss function. During the training process, edge detection is performed on the input image and an edge enhancement map is generated, which is sent to the network for training together with the original image as an additional input channel, and the weight of the edge part is increased, so that the network model learns more edge features. At the same time, the optic cup and optic disc segmentation and annotation process is used to locate the four quadrants, with the midpoint of the segmented optic disc as the origin of the coordinate axis, and the x-axis and y-axis are derived in the horizontal and vertical directions, based on which a description of the segmented lesion can be formed.

Images