A fundus image classification system based on deep mutual learning

Abstract

The application discloses a fundus photograph classification system based on deep mutual learning. The application fuses two kinds of anomaly detection algorithms of Ganomaly and FastFlow, constructs a detection model which is adaptive and robust to various eye diseases, effectively solves the problem that artificial intelligence cannot be used for intelligent diagnosis of a disease due to the lack of abnormal samples of the disease in practical application, improves the universality and accuracy of the disease detection model, saves the precious time of doctors, and improves the diagnosis efficiency of patients. In addition, the detection method can be applied to a fundus camera, and the device plays a great role in the initial screening stage of eye diseases. With slight cost, the device solves the problem that early eye diseases of patients are difficult to be found due to objective reasons such as insufficient medical resources and lack of experienced doctors, and truly helps community and primary hospitals to realize early discovery, early diagnosis and early treatment, so that the tragedy of blindness caused by delayed treatment due to late discovery is avoided.

Description

Technical Field

[0001] This invention belongs to the field of deep learning, and in particular relates to a fundus image classification system based on deep mutual learning. Background Technology

[0002] Data from 2020 shows that among healthcare professionals in Chinese ophthalmology hospitals, there were 36,115 licensed physicians, 30,273 registered nurses, 2,587 pharmacists, and 2,580 technicians. In 2020, China's total population was 1,411,780,000, meaning approximately one licensed physician could provide care for every 40,000 people. This highlights the crucial importance of effectively utilizing limited medical resources to serve such a large patient population.

[0003] Furthermore, manual diagnosis is inefficient: it takes an ophthalmologist nearly a minute to interpret one image, while AI can process 10 images per second. Compared to the traditional method, AI can reduce the screening cost from 50 yuan to 4 yuan per person. In addition, manual diagnosis is prone to errors due to the doctor's subjective reasons and lack of experience.

[0004] In recent years, disease diagnosis technologies based on artificial intelligence (AI), especially deep neural networks (DNNs), have developed rapidly. In the field of medical research, numerous studies have utilized AI for automated diagnosis, with AI diagnostic models achieving accuracy rates approaching those of expert physicians. However, in real-world clinical applications, AI technology is still not widely used. AI model diagnosis is not entirely reliable; if the model has not collected sufficient disease types and information beforehand, it may lead to missed or incorrect diagnoses.

[0005] Therefore, a combination of intelligent and manual diagnosis is needed to balance diagnostic efficiency and reliability. During each patient's visit, the model will make a preliminary assessment of their fundus image, using relevant algorithms to determine if the image is abnormal. If the fundus image is determined to be abnormal, the doctor will carefully analyze the patient's pathological images for further diagnosis, determining the specific disease type and severity. If the fundus image is determined to be normal, no further diagnosis is required from the doctor, saving valuable time and improving patient access efficiency.

[0006] In the medical field of retinal diseases, commonly used detection methods are mostly targeted at a specific disease, requiring a very large amount of data. Furthermore, the training phase of the models necessitates the use of datasets containing anomalies. Obtaining a large number of labeled samples consumes significant human and material resources, leading to the expensive investment required to develop such AI diagnostic systems.

[0007] With the application of intelligent detection models, ophthalmologists may worry that the disease detection range of artificial intelligence systems is too narrow, only able to distinguish between, for example, diabetic retinopathy and normal retina, but unable to detect relatively less common abnormalities, because these abnormalities were not included in the training of deep learning systems. Summary of the Invention

[0008] The purpose of this invention is to address the shortcomings of existing technologies by proposing a fundus image classification system based on deep mutual learning.

[0009] To achieve the above technical objectives, the technical solution of the present invention is as follows: An embodiment of the present invention proposes a fundus imaging classification system based on deep mutual learning, the system comprising:

[0010] The fundus image detection sample module utilizes collected fundus images, including two types of samples: fundus images of diseased individuals and healthy, disease-free individuals. The samples are divided into a training set and a validation set, with the training set containing only healthy, disease-free samples X. mn ∈R 512* The verification set contains fundus image samples X of diseases. lu ∈R 512* and healthy, disease-free fundus image samples X ln ∈R 512* , where n = 1,..., u = 1,..., N represents the number of samples, and 512 is the original pixel size of the image;

[0011] The image preprocessing module preprocesses the fundus image detection sample dataset, including adjusting the image resolution to unify the images to 256*256 pixels; scaling the images; adjusting the most semantically information-rich parts of the fundus image, such as the optic cup and optic disc, to the center position; and removing the black parts around the original image.

[0012] A fundus photography deep mutual learning classification module is developed, using a deep mutual learning framework built on Ganomaly and FastFlow, and a fundus photography classification network built on the open-source deep learning platform PyTorch. Only normal samples are used to construct the training set, and the training set {X} is used to classify the data. mn Train the fundus image classification network; optimize the parameters of the fundus image classification network and define the loss function; the fundus image classification network outputs the fundus image classification results.

[0013] The beneficial effects of this invention are as follows: This invention proposes a fundus photography classification system based on deep mutual learning. It trains a model capable of detecting multiple diseases using only normal samples, and the model's training does not rely on historical data for any single disease. This has significant practical application value in real-world scenarios, and the model's accuracy meets actual usage requirements. The advantage of mutual learning lies in the fact that Ganomaly only constrains latent variables to a high-dimensional space, while FastFlow imposes stricter constraints on variables at each layer of the latent space by calculating the loss function between the latent variables and gradient vectors at each layer. This allows the model after mutual learning to more accurately distinguish between normal and abnormal samples. The improved model shows improved performance across various evaluation metrics, meeting the practical needs of application scenarios such as community hospitals. It plays a beneficial role in promoting and applying portable fundus cameras, reducing the rate of missed diagnoses of ophthalmic diseases in local hospitals due to objective reasons such as a lack of experienced ophthalmologists and expensive medical equipment. Attached Figure Description

[0014] Figure 1 This is a diagram of a model building framework based on deep mutual learning;

[0015] Figure 2 It is a flowchart of model building and application;

[0016] Figure 3 This is the ROC diagram of the deep mutual learning-based model and the basic model;

[0017] Figure 4 It is a confusion matrix diagram between the deep mutual learning-based model and the basic model. Detailed Implementation

[0018] This invention addresses the challenges of diagnosing early-stage fundus diseases and the insufficient number of doctors and medical resources to serve the large patient population. It embeds a deep learning-based fundus photography analysis and disease detection method into a portable fundus camera. This invention can detect uncommon or even previously unseen eye diseases online, significantly reducing the rate of missed diagnoses of early-stage eye diseases.

[0019] This invention proposes a fundus image classification system based on deep mutual learning, comprising:

[0020] The fundus image detection sample module utilizes a non-public dataset from the Second Affiliated Hospital of Zhejiang University, containing images of glaucoma and normal fundus images. The samples are divided into a normal sample set and an abnormal sample set. Data in both subsets (normal and abnormal sets) are randomly shuffled, and the resulting datasets are rearranged according to a new order. The first 80% of the data in the normal sample set is used as the training set, and the last 20% of the data in the normal sample set, along with the abnormal sample set, is used as the validation set. The training set contains only defect-free samples X. mn ∈R 512* The verification set contains defective sample X lu ∈R 512* and defect-free sample X ln ∈R 512* , where n = 1,..., u = 1,..., N represents the number of samples, and 512 is the original pixel size of the image.

[0021] The image preprocessing module processes the training set (X = {X...}) mn Image preprocessing is performed on the array X, where X = 1, ..., X. Each image is randomly selected from the training set X, and its resolution is adjusted to 256*256. Information such as the visual disk and visual cup is placed in the center of the image, and redundant black areas are cropped. Image preprocessing eliminates irrelevant information in the image, enhances the detectability of relevant information, and simplifies the data to the greatest extent possible.

[0022] The fundus image depth-based classification module uses pre-processed training samples to train Ganomaly and FastFlow networks. The Ganomaly network can be divided into three parts. The first part is the generator network: it uses an autoencoder to encode and decode the input image to generate an image. The generator network consists of the following three steps:

[0023] (a) Generator G reads image x and passes it to the encoding network Ge of generator G.

[0024] (b) Perform convolution, batch normalization, and weak ReLU activation on the data:

[0025] output = max(0, W) T X+B)(1)

[0026] Where output represents the output, max() represents taking the maximum value, and W T Let X represent the transpose of the weight matrix, B represent the input, and G represent a constant. After this series of operations, G will transform x∈R. w*h*c Compress into a vector z∈R d .

[0027] (c) The decoder Gd adopts the structure of the DCGAN model generator, using convolutional transpose layers, ReLU activation, batch normalization, and the tanh activation function:

[0028]

[0029] Where x represents the input, e x This represents the exponentiation operation, ultimately resulting in x′∈R. w** .

[0030] 2) The second part of the entire network is the encoder network. The image x' generated by the generator G is fed into the encoder network, which compresses it to obtain z'. The encoder network has the same architectural details as Ge, and its function is to compress x' and x to obtain vectors z' and z respectively. However, the parameterization methods of the two are different. The former minimizes the latent vector through bottleneck features, while the latter explicitly learns the minimum distance through parameterization. The dimensions of z and z' generated by the two are the same.

[0031] 3) Discriminator Network: The discriminator D, introduced by Deep Convolutional Generative Adversarial Networks (DCGAN), is used to distinguish between true and false inputs x and x'. The discriminator consists of convolutional layers, batch normalization layers, and the LeakyReLU activation function, and comprises the following three steps:

[0032] (a) Perform a convolution operation on the input:

[0033]

[0034] The input size is i, the stride is s, the padding is p, the kernel size is k, and the output size is o. Here, the kernel is set to 5*5, the stride is 1, and the padding is 2.

[0035] Instead of deterministic spatial pooling, convolution is used to allow the network to learn downsampling on its own. Downsampling essentially reduces the size of the image, with the main purpose of making the image fit the size of the display area and generating a thumbnail of the corresponding image.

[0036] (b) Perform batch normalization (BN) on the input data. Batch normalization can accelerate learning and convergence. It transforms the input of each layer to a mean of 0 and a standard deviation of one unit. This is a very important technique in deep learning to accelerate convergence and reduce overfitting, and it can help overcome training difficulties caused by improper initialization.

[0037] (c) Using the LeakyReLU function to adjust the zero gradient problem for negative values.

[0038]

[0039] Here, α is set to 0.01.

[0040] 4) Train Fastflow to map X to Z according to the following rules:

[0041]

[0042] From p z (z) Estimating the log-likelihood of image features:

[0043]

[0044] Where z ~ N(o,I) and It is the Jacobian matrix of the bi-objective reversible flow model, i.e., z = f θ (x) and θ is a parameter of the two-dimensional flow model. During inference, the features of anomalous images should not follow a normal distribution, and therefore have a lower probability than the features of normal images. This probability can be used as anomaly scoring.

[0045] Define the loss function and related parameters, and perform mutual learning.

[0046] (1) Define the loss function in Ganomaly that will interact with FastFlow.

[0047] p1=-(ylog(p(x))+(1-y)log-p(x)))(7)

[0048] Where y is the true label value, p(x) is the predicted probability, and log() is the logarithmic function.

[0049] (2) Define the loss function in FastFlow for interacting with Ganomaly:

[0050] p2=-(ylog(p(x))+(1-y)log-p(x)))(8)

[0051] Where y is the true label value, p(x) is the predicted probability, and log() is the logarithmic function.

[0052] (3) Define the loss function for mutual learning between the two models:

[0053]

[0054]

[0055] Among them, D KL The distance represents the KL divergence, p1 and p2 represent the predicted probabilities of the two models, and x represents the distance between them. i Let m represent the i-th sample, m represent the class, there are M classes in total (M=2 here), N represent the total number of samples, and log() represents the logarithmic function.

[0056] (4) Set the number of training epochs and the batch size of training data. Use stochastic gradient descent (SGD) to optimize the loss function. Set the learning rate, decay weight w and momentum factor m for model training.

[0057] Evaluation Model: The overall effectiveness of the proposed model is evaluated using relevant evaluation metrics AUROC and F1Score.

[0058] AUROC is used to evaluate the effectiveness of this detection method; the area under the receiver operating characteristic curve is an indicator used to measure the performance of the classifier; the performance of the classifier is reflected by the area between the receiver operating characteristic curve and the coordinate axis, which can be understood as the expected ranking of a uniformly drawn random positive sample (positive sample) before a uniformly drawn random negative sample (negative sample); AUROC is a value between 0 and 1. When the AUROC value is close to 1, it indicates that the classifier can classify positive and negative samples well.

[0059] The F1Score is used to measure the accuracy of the model, which takes into account both the precision and recall of the model.

[0060]

[0061]

[0062]

[0063] In this context, TP (True Positive) represents a true positive: the actual value is normal and it is correctly predicted as normal data; TN (True Negative) represents a true negative: the actual value is abnormal and it is correctly predicted as abnormal data; FP (False Positive) represents a false positive: the actual value is abnormal and it is incorrectly predicted as a normal sample; and FN (False Negative) represents a false negative: the actual value is normal and it is incorrectly predicted as an abnormal sample.

[0064] The fundus classification results are output using the completed fundus classification network.

[0065] The effectiveness of the present invention will be verified by the following two specific examples.

[0066] Example 1

[0067] The Ophthalmology Center of the Second Affiliated Hospital of Zhejiang University currently comprises 10 specialized departments, including Cataract, Vitreoretinal Disease, Glaucoma, Refractive Surgery, Pediatric Strabismus and Amblyopia, Oculoplastic Surgery and Ocular Tumors, Ocular Surface and Corneal Diseases, Optometry, Neuro-Ophthalmology, and Uveitis, providing comprehensive ophthalmic diagnosis, treatment, and surgery. The diagnosis of eye diseases utilizes numerous medical devices to obtain various modal data related to the patient's eyeball; different diseases often require different examination equipment. Early diagnosis of glaucoma is particularly challenging, even for highly experienced physicians who struggle to diagnose it using only a single modal of information. This invention, however, utilizes only fundus photography—a single modal of information—to significantly improve the diagnostic accuracy of glaucoma beyond current methods, greatly enhancing diagnostic efficiency for physicians.

[0068] The implementation steps of the system of the present invention will be described below, taking into account the specific process of model training and evaluation using the glaucoma dataset:

[0069] A Ganomaly network consisting of a generator, encoder, and discriminator is constructed, and Ganomaly and FastFlow learn from each other.

[0070] The images from the training set are input into the improved model to optimize and adjust the relevant parameters of the network.

[0071] The network was evaluated using a glaucoma dataset. The model was evaluated using metrics such as F1 Score and AUROC, and its performance was compared with existing anomaly detection models. The results are shown in Table 1. The comparison results demonstrate that the model construction method proposed in this invention can be applied to real-world anomaly-free data scenarios, and its performance outperforms existing anomaly detection models.

[0072] Table 1: Comparison of the effects of different models in intelligent glaucoma diagnosis

[0073] Model / Evaluation Metrics F1 Score AUROC Dfm 0.67 0.26 Draem 0.70 0.61 Fastflow 0.79 0.83 Ganomaly 0.76 0.82 Padim 0.74 0.73 Mutual learning 0.92 0.95

[0074] Example 2

[0075] Furthermore, this invention also uses a dataset containing multiple fundus diseases to evaluate the anomaly detection capability of the constructed model for multiple diseases. The implementation steps of this invention will be described below with reference to specific processes:

[0076] A Ganomaly network consisting of a generator, encoder, and discriminator is constructed, and Ganomaly and FastFlow learn from each other.

[0077] Normal images from the training set are input into the improved model to optimize and adjust the network's parameters.

[0078] The model was evaluated using a dataset containing multiple diseases. Evaluation metrics such as F1 Score and AUROC were used to assess the model, and its performance was compared with existing anomaly detection models. The results are shown in Table 2. The comparison results demonstrate that the model construction method proposed in this invention can be applied to real-world anomaly-free data scenarios, and its performance outperforms existing anomaly detection models.

[0079] Table 2: Multi-disease recognition performance of the mutual learning model

[0080] Indicators / Disease Types Multiple diseases F1Score 0.85 AUROC 0.93

[0081] The above examples are used to illustrate the present invention, not to limit it. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A fundus image classification system based on deep mutual learning, characterized in that, The system includes: The fundus image detection sample module utilizes collected fundus images, including two types of samples: fundus images of diseased eyes and healthy, disease-free eyes. The samples are divided into a training set and a validation set, with the training set containing only healthy, disease-free samples. The verification set contains fundus image samples of diseases. and healthy, disease-free fundus image samples ,in , Represents the number of samples. The original pixel size of the image; The image preprocessing module performs image preprocessing on the fundus image detection sample dataset, including adjusting the image resolution to unify the images to 256*256 pixels; scaling the images; adjusting the most semantically information-rich parts of the optic cup and optic disc in the fundus image to the center position; and removing the black parts around the original image. The fundus photography deep mutual learning classification module is built upon a deep mutual learning framework based on Ganomaly and FastFlow, and a fundus photography classification network is constructed using the open-source deep learning platform PyTorch. Only normal samples are used to construct the training set. Fundus image classification network; optimize fundus image classification network parameters and define loss function; fundus image classification network outputs fundus image classification results; The process of training a fundus image classification network includes: Define the loss function in Ganomaly that will interact with FastFlow. Where y is the actual label value. It estimates the probability; log() is a logarithmic function. Define the loss function in FastFlow for interacting with Ganomaly: Where y is the actual label value. It estimates the probability; log() is a logarithmic function. Define a loss function for mutual learning between two models: Among them, D KL The distance represents the KL divergence, p1 and p2 represent the predicted probabilities of the two models, and x represents the distance between them. i Let m represent the i-th sample, m represent the class, there are M classes in total (M=2 here), N represent the total number of N samples, and log() represents the logarithmic function; Set the number of model training cycles and training batch data size The loss function is optimized using stochastic gradient descent, and the learning rate is set during model training. Decay weight and momentum factor .

2. The fundus imaging classification system based on deep mutual learning according to claim 1, characterized in that, The deep mutual learning framework built on Ganomaly and FastFlow includes: Take the labeled training set X and the label set Y as input; For the network and the Internet Initialize them separately; Calculate the prediction probabilities p1 and p2 of the two networks in the current training batch, and obtain the network... Total loss function ; Stochastic gradient descent is used to update the network. Parameters: Calculate the prediction probabilities p1 and p2 of the two networks in the current training batch, and obtain the network... Total loss function ; Stochastic gradient descent is used to update the network. Parameters: Repeat the above steps until the network converges; in, and The learning rate parameter represents the gradient descent parameter. and This indicates differentiation, enabling mutual learning between Ganomaly and FastFlow; 1) Ganomaly's entire network is divided into three parts. The first part is the generator network: it uses an autoencoder to generate an image by encoding and decoding the input image. The generator network consists of the following three steps: (a) Generator G reads image x and passes it to the encoding network Ge of generator G; (b) Perform convolution, batch normalization, and ReLU activation on the data: Where output represents the output, max() represents taking the maximum value, and W T Let X represent the transpose of the weight matrix, and B represent the input. After this series of operations, Ge will... Compress into a vector ; (c) The decoder Gd adopts the structure of the DCGAN model generator, using convolutional transpose layers, ReLU activation, batch normalization, and the tanh activation function: in, Indicates input, This represents exponentiation, ultimately yielding... ; 2) Encoder Network: The image x' generated by generator G is fed into the encoder network, which compresses it to obtain z'. The encoder network has the same architectural details as Ge, and its function is to compress x' and x to obtain vectors z' and z respectively; however, the parameterization methods of the two are different. The former minimizes the latent vector through bottleneck features, while the latter explicitly learns the minimum distance through parameterization. The dimensions of z and z' generated by the two are the same. 3) Discriminator Network: The discriminator D, introduced by a deep convolutional generative adversarial network, is used to distinguish between true and false inputs x and x'. The discriminator consists of convolutional layers, batch normalization layers, and the LeakyReLU activation function, and comprises the following three steps: (a) Perform a convolution operation on the input: Where the input size is i, the stride is s, the padding is p, the kernel size is k, and the output size is o; here, the kernel is set to 5*5, stride=1, padding=2; Instead of deterministic spatial pooling, convolution is used to allow the network to learn downsampling on its own. Downsampling is essentially reducing the size of the image, with the main purpose of making the image fit the size of the display area and generating a thumbnail of the corresponding image. (b) Perform batch standardization on the input data; transform the input of each layer to a mean of 0 and a standard deviation of one unit; (c) Using the LeakyReLU function to adjust the zero gradient problem for negative values; Here, α is set to 0.01; 4) Train Fastflow to map X to Z according to the following rules: (11) from Estimating the log-likelihood of image features: (12) in and It is the Jacobian matrix of the bi-objective reversible flow model, i.e. and , These are the parameters of a two-dimensional flow model.

3. The fundus imaging classification system based on deep mutual learning according to claim 1, characterized in that, The process of constructing the training set is as follows: Acquire fundus images; High-quality fundus images without any diseases were selected as the training set.

4. The fundus imaging classification system based on deep mutual learning according to claim 1, characterized in that, The system also includes an accuracy assessment of the fundus photography classification network; the accuracy assessment process is as follows: AUROC is used to evaluate the effectiveness of this detection method; the area under the receiver operating characteristic curve is an indicator used to measure the performance of the classifier; the performance of the classifier is reflected by the area between the receiver operating characteristic curve and the coordinate axis, which can be understood as the expected ranking of a uniformly drawn random positive sample before a uniformly drawn random negative sample; AUROC is a value between 0 and 1. When the AUROC value is close to 1, it indicates that the classifier can classify positive and negative samples well. The F1Score is used to measure the accuracy of the model, which takes into account both the precision and recall of the model. In this context, TP represents a true positive: the actual value is normal and it is also correctly predicted as normal data; TN represents a true negative: the actual value is abnormal and it is also correctly predicted as abnormal data; FP represents a false positive: the actual value is abnormal and it is incorrectly predicted as a normal sample; and FN represents a false negative: the actual value is normal and it is incorrectly predicted as an abnormal sample.

Images