Ophthalmic disease image recognition model and ophthalmic disease intelligent diagnosis method and system

Abstract

The invention discloses an ophthalmic disease image recognition model and an ophthalmic disease intelligent diagnosis method and system.The ophthalmic disease image recognition model is obtained by training a reference model, and training comprises deep learning pre-training and learning of visual features of eyes, so that classification of fundus images is facilitated; and carrying out transfer learning training, and adjusting model parameters so as to identify features related to ophthalmic disease diagnosis. By using the pre-trained model weight, the deep learning network can start training from a relatively good initial state. This initialization is generally more quickly convergent than starting training from random weights, since the pre-training model has learned to identify various universal visual features. The knowledge migration can help the model to identify and classify fundus diseases more accurately, especially when visual similarity exists among manifestations of the diseases. The transfer learning enables the model to better identify the modes, especially under the condition that the number of samples is small. In addition, SHAP values of all pixel points of the fundus disease image are calculated, a model interpretation graph is drawn, and whether the model captures correct features or not is judged. Therefore, the scheme can reduce the misjudgment rate of the fundus diseases, thereby improving the diagnosis accuracy of the fundus diseases.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent disease diagnosis, and specifically relates to an ophthalmic disease image recognition model, an ophthalmic disease intelligent diagnosis method and a system. Background Art

[0002] Intelligent auxiliary diagnosis and treatment of fundus diseases is an interdisciplinary field where digitalization and informatics increasingly interact. As an important sensory organ, the eye plays a vital role in receiving external information, and the retina of the fundus is the core structure responsible for capturing a large amount of external information through the principle of optical imaging. This information is then transmitted to the brain for processing. In recent years, the incidence of ophthalmic diseases has surged, especially among young people, which is attributed to the increasing popularity of electronic devices and the formation of bad eye habits. Fundus diseases caused by retinal fundus lesions are the main cause of visual impairment and blindness among various causes. Fundus diseases mainly include eye lesions with a high blindness rate, such as diabetic retinopathy, age-related macular degeneration, and high myopia. These diseases can cause persistent blurred vision, eye deformities, visual field disorders, and ultimately blindness. Therefore, early detection and timely treatment of fundus diseases are a major challenge because they have the potential to prevent visual impairment, blindness, and other visual diseases. The intelligent auxiliary diagnosis system for ophthalmic diseases can not only significantly improve the efficiency and accuracy of diagnosis and treatment, but also has the potential to balance the distribution of medical resources, thereby alleviating the gap in medical services between different regions. This technological advancement has laid the foundation for establishing a unified and efficient classification and diagnosis system for fundus diseases.

[0003] With the rapid growth of information resources, the improvement of hardware facilities and the development of big data technology, more and more studies have applied deep learning technology to the auxiliary diagnosis of fundus diseases and have made some good progress. In recent years, deep learning algorithms have achieved excellent performance in various tasks. The diagnosis of fundus diseases using deep learning methods has become an effective method and has made gratifying progress. In the existing literature, the accuracy achieved by deep learning methods for fundus disease diagnosis is similar to that of ophthalmologists. These studies mainly focus on the diagnosis of specific fundus diseases, and the main research contents include disease classification, lesion image segmentation, target detection and positioning algorithms using deep learning models, but there are still many shortcomings in the application of deep learning in the field of ophthalmic diseases:

[0004] (1) The lack of useful large amounts of labeled clinical data limits the widespread application of deep learning in the diagnosis of fundus diseases. Although unlabeled data can be obtained from fundus disease datasets, labeling large amounts of ophthalmic data requires expertise, which is difficult and expensive in terms of time and money.

[0005] (2) For some rare ophthalmic diseases, the available training samples are extremely limited. This limits the learning and generalization capabilities of deep learning models on these diseases, as the performance of the models depends heavily on large and diverse training data.

[0006] (3) For image recognition tasks, deep learning models can achieve very high prediction accuracy, but due to the opacity of the model, it is impossible to determine whether the model has recognized the key features of fundus disease images, nor can it explain the model's prediction results.

[0007] In view of this, the present invention is proposed. Summary of the invention

[0008] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide an intelligent diagnosis method and system for fundus diseases.

[0009] To achieve the above object, the technical solution of the present invention is:

[0010] In a first aspect, the present invention provides an ophthalmic disease image recognition model, wherein the ophthalmic disease image recognition model is obtained by training a benchmark model, wherein the training comprises:

[0011] Deep learning pre-training, learning the visual features of the eye so that it can classify fundus images;

[0012] Transfer learning training adjusts the model parameters so that it can recognize features relevant to the diagnosis of ophthalmic diseases.

[0013] Furthermore, the pre-training includes:

[0014] Performing a first pre-training of a deep learning model on a benchmark model on a first dataset, wherein the first dataset includes unlabeled natural images obtained from a public dataset;

[0015] The model weights obtained by the first pre-training are used to initialize the model, and a second pre-training of the deep learning model is performed on a second data set, where the second data set includes labeled ophthalmic disease images.

[0016] Furthermore, after the pre-training is completed, the model weights obtained from the pre-training are transferred to a third data set for deep learning model training, and the third data set is a small-scale, labeled multi-category ophthalmic disease image data set.

[0017] Furthermore, the ophthalmic disease images are obtained from a medical site, and ophthalmologists annotate the samples and classify them into slit lamp images and fundus images;

[0018] The slit lamp images include images corresponding to corneal perforation, corneal opacity, corneal laceration, corneal dermoid, corneal epithelial defect, corneal foreign body, hypopyon, pterygium and cornea in a healthy state;

[0019] The fundus images include images corresponding to age-related macular degeneration, diabetic retinopathy, glaucoma, high myopia, hypertensive retinopathy, retinal vein occlusion and leopard-marked fundus.

[0020] Furthermore, the benchmark model includes at least one of the improved models of Inception, ResNet, MobileNet, NASNet, and EfficientNet.

[0021] Furthermore, after the training, the prediction performance of the model is tested to evaluate the generalization ability of the final model.

[0022] In a second aspect, the present invention provides an intelligent diagnosis method for ophthalmic diseases, comprising:

[0023] Acquire ophthalmic images of patients and preprocess them in parallel;

[0024] The preprocessed images are input into the ophthalmic disease image recognition model to predict the patient's ophthalmic disease condition.

[0025] Furthermore, after predicting the patient's ophthalmic disease condition, the prediction results are explained.

[0026] Furthermore, the SHAP method is used to explain the decision-making process of the model prediction results.

[0027] In a third aspect, the present invention provides an intelligent diagnosis system for ophthalmic diseases, using the ophthalmic disease image recognition model as described above, or the ophthalmic disease intelligent diagnosis method as described above, the ophthalmic disease intelligent diagnosis system comprising:

[0028] Annotation module, used to collect, annotate and preprocess data;

[0029] Pre-training module, used to pre-train the benchmark model;

[0030] Migration module, used to perform transfer learning training on the model;

[0031] The prediction module is used to judge multiple types of fundus diseases based on the fine-tuned classifier and obtain the prediction results;

[0032] The explanation module is used to explain the prediction results.

[0033] The present invention has the following beneficial effects:

[0034] 1. By using pre-trained model weights, deep learning networks can start training from a relatively good initial state. This initialization usually converges faster than training from random weights because the pre-trained model has learned to recognize a variety of common visual features.

[0035] 2. Transferring knowledge from other eye disease images may include the ability to identify specific lesions or abnormalities. This knowledge transfer can help the model more accurately identify and classify fundus diseases, especially when there are visual similarities between the manifestations of these diseases. Transfer learning enables the model to better recognize these patterns, especially when the number of samples is small.

[0036] 3. The results of the multi-classification fundus disease intelligent diagnosis model are further explained. The SHAP method can be used to determine the contribution rate of each pixel to the output of the result when the fundus disease image is judged as a certain disease type. With this explanation method, it can be determined whether the model has captured the main features of the fundus disease image, and it can also be determined which factors in the image interfere with the prediction when the model makes a misjudgment, which is conducive to the debugging and correction of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are part of the present invention and are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but do not constitute an improper limitation of the present invention. Obviously, the drawings described below are only some embodiments. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the accompanying drawings:

[0038] Figure 1 It is a schematic diagram of the flow of the ophthalmic disease image recognition model of the present invention;

[0039] Figure 2 It is a schematic diagram of the structure of the intelligent diagnosis system for ophthalmic diseases of the present invention;

[0040] Figure 3 This is the first example of explaining the SHAP value of fundus disease images of the present invention;

[0041] Figure 4 This is the second example of explaining the SHAP value of fundus disease images of the present invention.

[0042] It should be noted that these drawings and textual descriptions are not intended to limit the conceptual scope of the present invention in any way, but are intended to illustrate the concept of the present invention for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0043] [Example 1]

[0044] This embodiment provides an ophthalmic disease image recognition model, which is obtained by training a benchmark model.

[0045] Specifically, the training of the baseline model includes:

[0046] Deep learning pre-training, learning the visual features of the eye so that it can classify fundus images;

[0047] Transfer learning training adjusts the model parameters so that it can recognize features relevant to the diagnosis of ophthalmic diseases.

[0048] By using various benchmark models to leverage knowledge in different but related source fields, transfer learning can improve the efficiency of target learners in specific fields. The final effect of transfer learning is also closely related to the choice of benchmark models. In this embodiment, the enhanced versions of Inception, ResNet, MobileNet, NASNet, and EfficientNet models, InceptionResNetV2, InceptionV3, ResNet101, MobileNetV2, NASNetLarge, EfficientNetB0, and EfficientNetB5, are selected as benchmark models.

[0049] Furthermore, deep learning pre-training includes:

[0050] Performing a first pre-training of a deep learning model on a benchmark model on a first data set, wherein the first data set includes unlabeled natural images obtained from a public data set, wherein the public data set is an ImageNet data set;

[0051] The model weights obtained by the first pre-training are used to initialize the model, and the second pre-training of the deep learning model is performed on the second data set to learn the visual features of the eye to adapt to the downstream fundus image classification task. The second data set includes annotated ophthalmic disease images.

[0052] Ophthalmological disease images are obtained from medical sites. Ophthalmologists annotate the samples and classify them into slit lamp images and fundus images, including:

[0053] The ophthalmic disease image data obtained from the medical site were saved as PNG or JPG format files;

[0054] The acquired ophthalmic disease images are annotated by ophthalmologists. These images are divided into two categories: slit lamp images and fundus images. Slit lamp images mainly involve the following nine diseases: corneal perforation, corneal opacity, corneal laceration, corneal dermoid, corneal epithelial defect, corneal foreign body, hypopyon, pterygium and healthy cornea. Fundus images include seven diseases such as age-related macular degeneration, diabetic retinopathy, glaucoma, high myopia, hypertensive retinopathy, retinal vein occlusion and leopard-shaped fundus. Such classification and annotation lay a solid foundation for subsequent analysis and research.

[0055] The present invention performs data enhancement on two types of ophthalmic data. Before the data is input into the model, it is cropped into eye images and fundus images. The size of each image is modified to 256×256×3, and random rotation, random cropping enhancement, random brightness enhancement, random chromaticity enhancement, random contrast enhancement and random horizontal flipping operations are performed on the training data. Secondly, the ophthalmic disease images are divided into different image sets according to the annotations, and then the data is divided into training sets, verification sets and test sets. Specifically, it includes dividing the images into several sets according to the annotations, and at the same time, 60% of all types of images are used as training sets, 20% as verification sets, and 20% as test sets to evaluate the model, and the evaluation of the model thereafter is based on the comprehensive consideration of the test set and the training set.

[0056] Furthermore, after the pre-training is completed, the model weights obtained from the pre-training are transferred to the third dataset for deep learning model training and fine-tuning. This process aims to adjust the model parameters so that it can identify features related to the diagnosis of ophthalmic diseases, thereby improving the performance and accuracy of the model in the task of fundus disease classification. Among them, the third dataset is a small-scale, annotated multi-category ophthalmic disease image dataset.

[0057] Furthermore, after the training is completed, the prediction performance of the model is tested and the generalization ability of the final model is evaluated, which includes the following two aspects:

[0058] 1. Model validation, based on the test set, the candidate model weights were evaluated from the aspects of accuracy, weighted precision, weighted recall, weighted F1-score, and AUC value. Table 1 lists all the possibilities of the prediction results of a single type of ophthalmic disease.

[0059] Table 1 Examples of prediction results for a single type of fundus disease

[0060]

[0061] The accuracy is:

[0062] The weighted precision is:

[0063]

[0064] The weighted recall is:

[0065]

[0066] False Positive Rate (FPR) is:

[0067] Weighted-F1-score is:

[0068]

[0069] Traditional accuracy indicators can only serve as a reference in model training. In terms of specific issues, the main content of medical diagnosis is to diagnose patients with diseases as positive, that is, the larger the Recall indicator, the better, and to diagnose patients without diseases as negative, that is, the smaller the FPR indicator, the better. Therefore, recall, precision and accuracy are all important criteria for model evaluation in ophthalmic disease diagnosis.

[0070] The ROC (Receiver Operating Characteristic) curve measures the relative relationship between the false positive rate (FPR) and the true positive rate (TPR) of the classification model. Based on the alternative model, we can get a TPR and FPR point pair based on its performance on the test set data, and then map it to a point on the ROC plane. By adjusting the threshold used by the classifier (from 0 to 1), we can get a ROC curve passing through (0,0) and (1,1), and the AUC value is the area under the curve.

[0071] 2. Model selection, based on the evaluation results, the model with the best prediction performance among the candidate models was selected as the final ophthalmic intelligent diagnosis model. The results are shown in Tables 2 and 3, among which the EfficientNetB0 model had the best prediction performance.

[0072] Table 2 Performance of alternative deep learning models

[0073]

[0074] Table 3 Comparison of transfer learning performance using the EfficientNetB0 model on different source datasets

[0075]

[0076] [Example 2]

[0077] This embodiment provides an intelligent diagnosis method for ophthalmic diseases, including:

[0078] Acquire ophthalmic images of patients and preprocess them in parallel;

[0079] The preprocessed image is input into the ophthalmic disease image recognition model provided in the aforementioned embodiment to predict the ophthalmic disease condition of the patient.

[0080] Furthermore, after predicting the patient's ophthalmic disease condition, the prediction results are explained.

[0081] In a preferred embodiment, the SHAP method is used to explain the decision-making process of the model prediction results, and the contribution of each pixel in the ophthalmic disease image to the prediction results is calculated.

[0082] SHAP defines explanation as: where g is the explanation model, z'∈{0,1} M is the coalition vector, M is the maximum coalition size, φ j ∈R is the feature-attributed Shapley value of feature j.

[0083] At the same time, the alliance vector is set. Input 1 indicates that the corresponding feature exists, while input 0 indicates that it does not exist. For the instance x of interest, the alliance vector x' is a vector of all 1s, that is, all eigenvalues ​​exist. The formula is simplified to:

[0084]

[0085] Calculate the Shapley value through SHAP and establish the SHAP kernel:

[0086]

[0087] Where M is the maximum coalition size and |z'| is the number of features currently in instance z'.

[0088] Then build a weighted linear regression model:

[0089]

[0090] The linear model g is trained by optimizing the following optimization function L:

[0091]

[0092] Where Z is the training data, the sum of squared errors of the optimized linear model, and the estimated coefficients of the model φ jThis is the Shapley value that needs to be solved, which is also the contribution rate of each pixel point of the input image to the final result of the fundus disease prediction model. When the global importance of each pixel point of the input image is required, according to the additive property of the Shapley value, we can take the average value of the Shapley absolute value of each feature in the data:

[0093]

[0094] Figure 3 and Figure 4 This is an example of model explanation, which shows the prediction results of the top three most likely diseases from left to right under the given seven fundus disease images. Figure 3 Take the first row of four pictures as an example. The first one is the input image. The second picture shows that the image is most likely to be age-related macular degeneration (AMD), which is also the final prediction result of the model. The red and blue pixels in the image represent the SHAP value of the pixel to this result. Red indicates that the pixel contributes positively to this result, and blue indicates that the pixel contributes negatively to this result. The darker the color, the greater the contribution rate. The same explanation applies to the two pictures on the right. Specifically, taking the prediction result of the optimal model of AMD disease images as an example, in the most likely prediction result, pixels with SHAP values ​​greater than 0.003 appear magenta. The model believes that these pixels have a positive effect on diagnosing AMD as positive, and the area of ​​the pixel is mostly distributed in the edge part; pixels with SHAP values ​​less than -0.003 appear blue. The model believes that these pixels have an inhibitory effect on diagnosing AMD as negative, and the area of ​​the pixel is mostly distributed in the blood vessel position.

[0095] The probability that the optimal model (EfficientNetB0) predicts a certain type of disease as positive is shown in Table 3:

[0096] Table 3 The probability of the optimal model (EfficientNetB0) predicting a certain type of disease as positive

[0097]

[0098] [Example 3]

[0099] This embodiment provides an intelligent diagnosis system for ophthalmic diseases, which adopts the ophthalmic disease image recognition model provided by the previous embodiment, or the intelligent diagnosis method for ophthalmic diseases provided by the previous embodiment.

[0100] Specifically, the intelligent diagnosis system for ophthalmic diseases includes:

[0101] Annotation module, used to collect, annotate and preprocess data;

[0102] Pre-training module, used to pre-train the benchmark model;

[0103] Migration module, used to perform transfer learning training on the model;

[0104] The prediction module is used to judge multiple types of fundus diseases based on the fine-tuned classifier and obtain the prediction results;

[0105] The explanation module is used to explain the prediction results.

[0106] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0107] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as above in the preferred embodiment, it is not used to limit the present invention. Any technician familiar with this patent can make some changes or modifications to equivalent embodiments of equivalent changes by using the technical content suggested above without departing from the scope of the technical solution of the present invention. The implementation scheme in the above embodiment can also be further combined or replaced. However, any simple modification, equivalent change and modification made to the above embodiment based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still falls within the scope of the solution of the present invention.

Claims

1. An ophthalmic disease image recognition model, characterized in that: The ophthalmic disease image recognition model is obtained by training a benchmark model, and the training includes: Deep learning pre-training to learn the visual features of the eye, making it useful for classifying fundus images; Transfer learning training adjusts the model parameters so that it can recognize features relevant to the diagnosis of ophthalmic diseases.

2. The ophthalmic disease image recognition model according to claim 1, characterized in that: The pre-training includes: Performing a first pre-training of a deep learning model on a benchmark model on a first dataset, wherein the first dataset includes unlabeled natural images obtained from a public dataset; The model weights obtained by the first pre-training are used to initialize the model, and a second pre-training of the deep learning model is performed on a second data set, where the second data set includes labeled ophthalmic disease images.

3. The ophthalmic disease image recognition model according to claim 2, characterized in that: After the pre-training is completed, the model weights obtained from the pre-training are transferred to the third data set for deep learning model training. The third data set is a small-scale, labeled multi-category ophthalmic disease image data set.

4. The ophthalmic disease image recognition model according to claim 2, characterized in that: The ophthalmic disease images are obtained from medical sites, and ophthalmologists annotate the samples and classify them into slit lamp images and fundus images; The slit lamp images include images corresponding to corneal perforation, corneal opacity, corneal laceration, corneal dermoid, corneal epithelial defect, corneal foreign body, hypopyon, pterygium and cornea in a healthy state; The fundus images include images corresponding to age-related macular degeneration, diabetic retinopathy, glaucoma, high myopia, hypertensive retinopathy, retinal vein occlusion and leopard-marked fundus.

5. The ophthalmic disease image recognition model according to claim 1, characterized in that: The benchmark model includes at least one of the improved models of Inception, ResNet, MobileNet, NASNet, and EfficientNet.

6. The ophthalmic disease image recognition model according to claim 5, characterized in that: After training, the prediction performance of the model is tested to evaluate the generalization ability of the final model.

7. An intelligent diagnosis method for ophthalmic diseases, characterized in that: include: Acquire ophthalmic images of patients and preprocess them in parallel; The preprocessed images are input into the ophthalmic disease image recognition model to predict the patient's ophthalmic disease condition.

8. The intelligent diagnosis method for ophthalmic diseases according to claim 7, characterized in that: After predicting the patient's ophthalmic disease, the prediction results are explained.

9. The intelligent diagnosis method for ophthalmic diseases according to claim 8, characterized in that: The SHAP method is used to explain the decision-making process of the model's prediction results.

10. An intelligent diagnosis system for ophthalmic diseases, using the ophthalmic disease image recognition model according to any one of claims 1 to 6, or the ophthalmic disease intelligent diagnosis method according to any one of claims 7 to 9, characterized in that: include: Annotation module, used to collect, annotate and preprocess data; Pre-training module, used to pre-train the benchmark model; Migration module, used to perform transfer learning training on the model; The prediction module is used to judge multiple types of fundus diseases based on the fine-tuned classifier and obtain the prediction results; The explanation module is used to explain the prediction results.

Images