Method and system for staging diabetic nephropathy using deep learning

Abstract

Embodiments of the invention disclose a method and system for staging diabetic nephropathy using deep learning techniques. The image acquisition unit acquires a set of ophthalmic images of a user. The ophthalmic image set is pre-processed before being input to the first deep learning module. A first deep learning module extracts pathological data indicative of vascular abnormalities from the pre-processed set of ophthalmic images. The first deep learning module quantifies the extracted pathological data and maps the pathological data to diabetic retinopathy stages and urine protein levels. A second deep learning module receives as inputs the quantified pathology data, the mapped diabetic retinopathy staging and urine protein levels, and clinical and demographic parameters. Based on the input, the second deep learning module predicts a stage of diabetic nephropathy.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of and priority to Indian Provisional Application No. 202311001136, the entire disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of the present invention generally relate to deep learning techniques for predicting disease, and more particularly to deep learning techniques for detecting the stage of diabetic nephropathy. Background Art

[0004] Diabetes mellitus (also known as diabetic hyperhidrosis) is a group of metabolic disorders characterized by a common phenotype of high blood glucose concentrations. The global prevalence of diabetes in adults has continued to rise in recent decades. An estimated 415 million people were affected by diabetes in 2015, and the International Diabetes Federation (IDF) predicts this number will reach 642 million by 2040, with the largest increases occurring in Asia, particularly India and China. The rising prevalence of diabetes, coupled with an aging population, will inevitably lead to an increase in microvascular complications such as diabetic retinopathy (DR). DR, a leading cause of preventable blindness, has been shown to increase the risk of all-cause and cardiovascular mortality. DR shares similar pathogenesis with diabetic kidney disease (DKD), also known as diabetic nephropathy (DN), resulting in similar metabolic consequences. DKD is a major contributor to end-stage renal disease (ESRD), ultimately leading to significant morbidity and mortality. Furthermore, the risk of DKD progression increases with the severity of DR.

[0005] Due to the lack of routine screening for microalbuminuria (a range of urine protein levels), diabetic renal disease (DKD) is often missed in its early stages, especially in resource-poor settings. Current management of end-stage renal disease (ESRD) involves frequent dialysis or renal replacement therapy, which means high costs and reduced quality of life for patients and places a heavy burden on the healthcare system.

[0006] Although there is a lot of evidence that renal lesions can be detected by diabetic fundus images (i.e., retinal images that detect DR), previous studies have not focused on predicting the stage of renal disease. No study has analyzed the impact of the presence of DR and non-DR patterns on the stage of renal disease. In addition, none of the early existing technologies classified fundus images into proliferative or non-proliferative DR based on microaneurysms, dot / plaque hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading changes, intraretinal microvascular abnormalities, and infection. Specifically, for proliferative DR, none of the existing technologies used microaneurysms, dot / plaque hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading changes, intraretinal microvascular abnormalities, neovascularization, and vitreous / preretinal hemorrhages to classify fundus images. Therefore, neovascularization and vitreous / preretinal hemorrhage are the decisive pathologies for classifying proliferative and non-proliferative DR stages. In addition, none of the early existing technologies disclosed the technology for determining the criteria for referral to a nephrologist. Summary of the Invention

[0007] Advantageous embodiments of the present disclosure generally solve or circumvent these and other problems, and generally achieve technical advantages.

[0008] The following is a simplified overview of the subject matter to provide a basic understanding of some aspects of the subject matter embodiments. This overview is not a comprehensive overview of the subject matter. It is not intended to identify key / critical elements of the embodiments or to delineate the scope of the subject matter. Its sole purpose is to present some concepts of the subject matter in a simplified form as a prelude to the more detailed description that follows.

[0009] In a first aspect, a system is provided. The system includes an image acquisition unit configured to acquire a set of ophthalmic images of a person. The system also includes an image processing unit configured to process the set of ophthalmic images obtained from the image acquisition unit. The image processing unit includes a preprocessing module configured to preprocess the set of ophthalmic images. The image processing unit also includes a first deep learning module configured to extract pathological data indicating vascular abnormalities from the preprocessed set of ophthalmic images, quantify the extracted pathological data based on the vascular abnormalities, and map the quantified pathological data to a diabetic retinopathy stage and a urine protein level. The image processing unit also includes a second deep learning module configured to receive clinical and demographic parameters, as well as the quantified pathological data, the diabetic retinopathy stage, and the urine protein level from the first deep learning module. The second deep learning module is configured to predict the stage of diabetic nephropathy based on the quantified pathological data, the diabetic retinopathy stage, the urine protein level, and the clinical and demographic parameters.

[0010] In a second aspect, a method is provided. The method includes acquiring a set of ophthalmic images of a person via an image acquisition unit. The method includes processing the set of ophthalmic images obtained from the image acquisition unit via an image processing unit. The processing includes preprocessing the set of ophthalmic images via a preprocessing module. The processing includes extracting pathological data indicating vascular abnormalities from the preprocessed set of ophthalmic images via a first deep learning module. The processing includes quantifying the extracted pathological data based on the vascular abnormalities via the first deep learning module. The processing includes mapping the quantified pathological data to a diabetic retinopathy stage and a urine protein level via the first deep learning module. The processing includes receiving clinical and demographic parameters, as well as the quantified pathological data, the diabetic retinopathy stage, and the urine protein level from the deep learning module via a second deep learning module. The processing includes predicting the diabetic nephropathy stage via the second deep learning module based on the quantified pathological data, the diabetic retinopathy stage, the urine protein level, and the clinical and demographic parameters.

[0011] In a third aspect, a computer-readable medium is provided, wherein the computer-readable medium includes instructions that, when executed by at least one processor in a computer system, cause the computer system to perform the method of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The following drawings illustrate specific examples of systems and methods for implementing the present disclosure, describe some methods and mechanisms, and are not intended to limit the scope of the present invention. The drawings are not drawn to scale (unless otherwise specified) and are intended to be used in conjunction with the explanations in the following detailed description.

[0013] Figure 1 shows a relationship between ocular and renal vascular abnormalities in individuals with diabetes;

[0014] Figure 2 shows a process flow for detecting diabetic kidney disease (DKD) staging according to an example embodiment of the present disclosure;

[0015] Figure 3 shows a process flow for a two-step multi-label image model and a clinical model according to an example embodiment of the present disclosure;

[0016] Figure 4 shows the workflow of a two-step multi-label image model and a clinical model for detecting the stages of diabetic retinopathy (DR) and diabetic nephropathy (DKD), respectively, according to an example embodiment of the present disclosure;

[0017] Figure 5 The architecture of a two-step multi-label image model according to an example embodiment of the present disclosure is shown;

[0018] Figure 6 illustrates the working of a clinical model according to an example embodiment of the present disclosure;

[0019] Figure 7 shows a process flow for detecting DKD stages through a clinical model using the output of a two-step multi-label image model according to an example embodiment of the present disclosure;

[0020] Figure 8 shows a process flow for performing retinal and renal assessments in an ophthalmology clinic and a nephrology clinic, respectively, according to an example embodiment of the present disclosure;

[0021] Figure 9A and Figure 9B Various parameters and data statistics for a multi-label image model and a clinical model according to an example embodiment of the present disclosure are shown;

[0022] Figure 10A and Figure 10B shows confusion matrices for a two-step multi-label image model and a clinical model according to an example embodiment of the present disclosure;

[0023] Figure 11A and Figure 11B shows graphs depicting the performance of a two-step multi-label image model and a clinical model, respectively, according to an example embodiment of the present disclosure;

[0024] Figure 12A and Figure 12B shows pie charts representing percentage distribution of DR stages associated with early-stage DKD or advanced DKD, respectively, according to an example embodiment of the present disclosure;

[0025] Figure 13 A system for implementing an example embodiment of the present disclosure is shown; and

[0026] FIG14 illustrates a method for detecting DKD staging based on DR staging according to an example embodiment of the present disclosure.

[0027] Those skilled in the art will appreciate that the elements in the figures are illustrated for simplicity and clarity and may represent both hardware and software components of the system. In addition, the dimensions of some elements in the figures may be exaggerated relative to other elements to help enhance understanding of the various example embodiments of the present disclosure. Throughout the drawings, it should be noted that the same reference numerals are used to depict the same or similar elements, features, and structures. DETAILED DESCRIPTION

[0028] Example embodiments will now be described. However, the present disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey its scope to those skilled in the art. The terminology used in the detailed description of the specific example embodiments illustrated in the accompanying drawings is not intended to be limiting. In the accompanying drawings, like reference numerals represent like elements.

[0029] This specification may refer to "one", "an" or "some" embodiments in multiple places. This does not necessarily mean that each such reference points to the same embodiment, nor does it mean that the feature only applies to a single embodiment. The individual features of different embodiments may also be combined to provide other embodiments. As used in the present invention, the singular forms "one", "an" and "the" are also intended to include plural forms, unless otherwise expressly stated. It will also be understood that when the terms "comprise" and / or "comprising" are used in this specification, it is indicated that there are stated features, integers, steps, operations, elements and / or parts, but it is not excluded that there are or add one or more other features, integers, steps, operations, elements, parts and / or their combinations. As used in the present invention, when the phrase "at least one of the following items" is before a list of elements, where the elements are connected by "and" or "or", this means that there is at least any element or at least all elements. As used in the present invention, the term "and / or" includes all combinations and arrangements of one or more of the associated listed items.

[0030] Unless otherwise expressly stated or understood in context, conditional language, such as "may" or "might," is generally intended to convey that certain embodiments may include and other embodiments may not include certain features, elements, and / or steps. Thus, such conditional language is generally not intended to imply that a feature, element, and / or step is in any way necessary for one or more embodiments. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intervening elements. Furthermore, "connected" or "coupled" as used herein may include wireless connections or couplings.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used in the present invention have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. It will also be understood that terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the relevant technical background and should not be interpreted in an idealized or overly formal sense unless expressly defined in this invention.

[0032] The accompanying drawings depict simplified structures showing only some of the elements and functional entities, all of which are logical units whose implementation may differ from that shown. The connections shown are logical; actual physical connections may differ. Furthermore, all logical units depicted and described in the figures include the software and / or hardware components necessary for the unit to function. Furthermore, each unit may include one or more components within itself, which are implicitly understood. These components may be operatively coupled to each other and configured to communicate with each other to perform the unit's functions.

[0033] Figure 1 The relationship between ocular and renal vascular abnormalities in diabetic patients is shown. Diabetic retinopathy (DR) and diabetic nephropathy (DKD) are complications of diabetes, and therefore the two diseases have similar metabolic consequences. Diabetes affects the vascular systems of the eyes and kidneys in similar ways. Figure 1 As shown, in the eye, diabetes can cause blood vessel damage, leading to various types of vascular abnormalities, which can cause blood to leak (manifest as bleeding spots). Similarly, blood vessel damage in the kidneys can cause protein to leak into the urine. The extent of blood vessel damage in the eye is similar to that in the kidneys because the vascular defects caused by diabetes also occur in the kidneys, affecting their protein filtration function.

[0034] Figure 2 A process flow 200 for detecting the stage of diabetic kidney disease (DKD) according to an example embodiment of the present disclosure is shown. At process block 102, a fundus or optical coherence tomography (OCT) image is acquired. The fundus image can be acquired by an image acquisition unit, such as, but not limited to, a fundus camera, while the OCT image can be acquired by an image acquisition unit, such as, but not limited to, an OCT machine. Figure 2 The description will be explained in the context of fundus images as the captured images, however, this should be interpreted as non-limiting as OCT images may also be used as an alternative. The fundus image may be a two-dimensional (2D) or three-dimensional (3D) OCT cube representation of the retinal image. In one embodiment, the fundus image may be an infrared image, and in another embodiment, the fundus image may be an autofluorescence image. The image data may be obtained in any format including, but not limited to, JPG, PNG, DCM (DICOM), BMP, GIF, and TIFF.

[0035] At process block 204, the acquired fundus image is input to a computing device having a display on which a user interface is provided. The computing device may receive the acquired fundus image directly or indirectly from the image acquisition device via a wireless communication device (such as Bluetooth, near field communication, Wi-Fi, etc.). The user interface may have an upload option via which the fundus image can be uploaded. The computing device may be located at a physician or medical technician. The physician or medical technician acquires a retinal fundus image of a diabetic patient who comes for treatment.

[0036] At process block 206, the retinal fundus image is input to an artificial intelligence (AI) retinal model (i.e., a two-step multi-label image model) stored in a computing device. The AI retinal model is used to take the fundus image as input and predict DR stage and urine protein level based on the fundus image. In an example embodiment, the AI retinal model can extract a total of 29 pathologies from the fundus image using deep learning techniques. During the training phase, the AI retinal model is used to take as input a plurality of retinal fundus images labeled with lesions (as part of a training dataset) and apply deep learning techniques to process and extract pathologies from the retinal fundus images.

[0037] At process block 208, the AI retinal model quantifies pathological data extracted from the acquired fundus images. In an exemplary embodiment, the pathological data is quantified as: no abnormality, microaneurysms, punctate or macular hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading, intraretinal microvascular abnormalities, neovascularization, and / or vitreous or preretinal hemorrhages. The quantified pathological data is then mapped to DR stages and urine protein levels using a DR mapper. In an exemplary embodiment, the mapping of quantified pathological data to DR stages is performed according to Table A.

[0038]

[0039]

[0040] Table A

[0041] Depending on the type and amount of pathology, the DR stage of a diabetic patient can be predicted. Table A illustrates the following DR stages in order of increasing health risk: no DR, mild non-proliferative DR, moderate non-proliferative DR, severe non-proliferative DR, and proliferative DR. Therefore, by quantifying the pathological data, the DR stage of a diabetic patient can be determined. Similarly, the quantitative pathological data can also be used to predict urine protein levels by mapping them to the standard reference range of urine protein levels (normal value <30 mg / dL, microalbuminuria 30 mg / dL to 300 mg / dL, and macroalbuminuria >300 mg / dL). The mapping of urine protein levels can be based on the four quadrants of the retinal fundus image (see Figure 4 ) The number and type of vascular lesions identified (by using deep learning techniques).

[0042] In one example embodiment, the computing device storing the AI retinal model may also store a clinical model. In another example embodiment, there may be separate computing devices storing the AI retinal model and the DKD model (i.e., clinical model) respectively. At process block 210, the computing device storing the clinical model receives the output of the AI retinal model, i.e., DR staging and urine protein level. In addition, in some embodiments, the computing device also receives clinical data related to the patient. Clinical data may include data about the patient's history of diabetes, whether the patient has other comorbidities, etc. The computing device storing the DKD model may be provided with a UI page on its display, wherein the UI page includes multiple input fields through which DR staging, urine protein level and clinical data can be manually entered. The fundus image collected at process block 202 can also be uploaded via the upload function of the UI page of the computing device.

[0043] At process block 212, the DKD model takes the DR stage, urine protein level, and clinical data as input to output a predicted DKD stage. The predicted DKD stage can be used to assess the severity of DKD, as shown in Table B below.

[0044] DKD staging Classification 0 No DKD 1, 2, or 3 Early 4 or 5 Progressive stage

[0045] Table B

[0046] The predicted DKD stage and / or its classification may be displayed on a UI page of the computing device storing the DKD model.

[0047] In one embodiment, the DKD model performs binary classification to classify the DKD stage as early or advanced. In another embodiment, the DKD model performs multi-label classification to classify the DKD stage as no DKD, early DKD, or advanced DKD (as shown in Table B).

[0048] Figure 3A simple process flow 300 of an AI retinal model (i.e., a two-step multi-label image model) and a DKD model (i.e., a clinical model) is shown. At process block 302, fundus images of diabetic patients are collected and fed to the AI retinal model. At process block 304, the AI retinal model receives the fundus images to predict the DR stage and urine protein level of the diabetic patient based on the fundus images. More specifically, at process block 304, the AI retinal model extracts pathological data from the collected fundus images, which can be used to predict DR stage and urine protein level. At step 306, the AI retinal model quantizes the extracted pathological data into zero or more pathological features (i.e., classifies the fundus images into zero or more pathologies). Based on the quantified pathological data, the DR mapper is responsible for mapping the quantified pathological data to DR stage. The DR mapper can be a DR stage regular expression rule parser that uses deep learning technology to perform mapping according to Table A. The quantified pathological data can also be used to map to urine protein level.

[0049] At process block 308, the output of the AI retinal model (i.e., DR stage and urine protein level) is manually input as input to the DKD model. In some embodiments, the output of the AI retinal model can be fed directly to the DKD model, i.e., manual input of the data may not be required (process block 308). At process block 310, clinical and demographic data about the diabetic patient can be manually input as input to the DKD model. Examples of clinical data include other comorbidities, duration of diabetes, history of hypertension, etc. Demographic data can include age and gender.

[0050] At process block 312, the DKD model can predict the DKD stage within the diabetic patient based on the inputs received at process blocks 306 / 308 and 310. The DKD stage can be classified as no DKD, early DKD, or advanced DKD.

[0051] Figure 4 The workflow of a two-step multi-label image model and a clinical model for detecting diabetic retinopathy (DR) and diabetic nephropathy (DKD) stages based on four quadrants of a fundus image according to an example embodiment disclosed herein is shown.

[0052] The fundus image can be divided into four quadrants: the superior temporal quadrant, the superior nasal quadrant, the inferior temporal quadrant, and the inferior nasal quadrant. At process block 402, the AI retinal model is trained using unsupervised learning to identify the type and amount of pathology. In some embodiments, the AI retinal model can be trained via supervised learning using a labeled training data set. At process block 404, once the AI retinal model has completed training, it will identify the type and amount of pathology in each quadrant. At process block 406, the amount and type of pathology are mapped to DR stage and urine protein level via AI-based technology. At process block 408, based on the DR stage and urine protein level, the DKD model identifies the DKD stage. At process block 410, based on the DKD stage and estimated glomerular filtration rate (eGFR), the progression of DKD is identified. eGFR can be estimated based on a blood sample from the patient.

[0053] Figure 5 The architecture 500 of a two-step multi-label image model (i.e., an AI retinal model) according to an embodiment of the present disclosure is shown. The multi-label image model can extract pathological data representing vascular abnormality patterns from fundus images. The extracted pathological data is then quantified and mapped to DR stage and urine protein level. The extracted pathological data and DR stage are used to build a clinical model (i.e., a DKD model).

[0054] Two-step training of multi-label image models

[0055] In order for the two-step multi-label image model to perform its above-mentioned functions, the multi-label image model can be trained. In an example embodiment, fundus data with 29 types of pathology can be used to train the multi-label image model. In an example embodiment, the dataset used to model the multi-label image model includes 133,273 samples of the training set and 14,779 samples of the test set. All color fundus images can be labeled by clinicians with zero to multiple pathologies and DR stages. The training set can include retinal fundus images with zero or more pathologies, where in the case of zero pathologies, it can be determined that DR is not present. This can help train the two-step multi-label image model to associate DR stages with quantified pathology data.

[0056] In this embodiment, the multi-label image model uses InceptionRestNetV2 as the base model. Before processing the fundus image, the image can be smoothed using techniques such as Gaussian blur, and then pre-processed using BenGraham. In Graham, both scaling and circular cropping can be added.

[0057] In deep learning, convolutional neural networks (CNNs) are perhaps the most commonly used deep neural network class for capturing spatial information in visual imaging. As mentioned earlier, the multi-label image model uses InceptionResNetV2 as its base model. InceptionRestNetV2 is a pre-trained CNN-based network with 164 layers and trained using images from the ImageNet database. This multi-label image model also includes three custom dense layers. By setting the top parameter to false, the last layer of the model is removed, allowing the custom dense layers to be used for training. This is essentially transfer learning, extracting features from the base model and training on fundus data by adding custom dense layers. Using transfer learning achieves efficient resource utilization, as the resource consumption required to train the model from scratch is avoided. The three custom dense layers contain 256, 128, and 29 neurons, respectively. Finally, softmax is used as the activation function for the last layer, as each image may contain zero to multiple pathologies. Cross-entropy is used as the training loss function. Class weights are assigned based on class frequency, with the null class and highly underrepresented classes given fixed, small weights.

[0058] The output of the multi-label model is a classification of the input fundus image into zero or more pathologies, which is essentially multi-label classification.

[0059] Therefore, according to Figure 5 As shown in the example embodiment of FIG, at process step 502, a fundus image having dimensions of 450×450×3 is received. As mentioned above, the fundus image can be a retinal fundus image, i.e., a fundus image associated with a patient's ophthalmic data. At process step 504, Graham preprocessing is performed on the received fundus image to preprocess the fundus image. In some embodiments, prior to performing Graham preprocessing, Gaussian blur is applied to the fundus image to smooth it.

[0060] At process step 506, the pre-processed image is input to the Inception-ResNet v2 architecture. Inception-ResNet v2 is a convolutional neural network trained on over one million images from the ImageNet database. This neural network is used to classify images into multiple categories using deep learning techniques. The neural network consists of a base network and a fully connected network.

[0061] At process step 508, three custom dense layers are added. In one embodiment, these three layers have 256, 128, and 29 neurons, respectively. Furthermore, at process step 508, SoftMax is used as the activation function for the final layer, as zero to multiple pathologies can be used for each image. Dense layers are used to define the relationships between the data values processed by the model. Furthermore, SoftMax is used for the final classification of the data.

[0062] At process step 510, multi-label classification is performed. This classifies the fundus image into one or more pathology categories (e.g., up to 29 categories). Some multi-label classifications include, but are not limited to, classifications for exudates, cotton wool spots, macular edema, punctate hemorrhages, preretinal hemorrhages, drusen, microaneurysms, and venous beading. In some embodiments, the classification may also include a "no vascular abnormality" category. In one embodiment, the multi-label image model may accept inputs such as urine protein level, urine creatinine level, and protein-creatinine ratio to predict DR stage.

[0063] Figure 6 The following illustrates the operation of a clinical model according to an exemplary embodiment of the present disclosure. In the clinical model, feature selection is performed using statistical tests, machine learning forward feature selection techniques, machine learning experiments, and clinical expert opinion. Features can be classified into different categories, such as continuous / categorical or novel. The following table shows the feature classification:

[0064] feature category History of hypertension Classification Urine protein Classification DR staging Classification Cotton-wool spots Classification exudate Classification age Classification gender Classification Other comorbidities Classification duration of diabetes Continuous

[0065] As shown above, while features such as history of hypertension, proteinuria, DR stage, cotton wool spots, exudates, sex, and age are categorical variables, age and diabetes duration features are continuous features. A binary classifier is a meta-estimator that fits multiple decision tree classifiers on various subsamples of a data set. In one embodiment, the binary classifier uses an averaging method to improve prediction accuracy and control overfitting. Controlling overfitting is important so that prediction accuracy can be improved. The subsample size is controlled by the maximum number of samples, otherwise the entire data set is used. The maximum number of features and tree height are used to control overfitting. In one embodiment, the final ensemble model used can be a random forest. The clinical model can receive inputs such as DR stage, range of proteinuria levels, etc., and perform a two-class classification. This can be a two-class classification algorithm based on Inception ResNet that classifies the DKD stage as early or advanced.

[0066] like Figure 3As shown, the selected features can be manually input into the clinical model. In some embodiments, the clinical model can perform multi-label classification (no DKD, early DKD, advanced DKD or late DKD) rather than a binary classification (i.e., early DKD or advanced DKD). In another embodiment, the DKD model can predict the accumulation and release of proteins in the kidney.

[0067] Figure 7 A process flow 700 for detecting DKD stages using a two-step multi-label image model and a clinical model is shown according to an embodiment of the present disclosure.

[0068] According to process flow 702, the multi-label image model receives a fundus image and performs automatic feature extraction of the fundus image to extract pathological data therefrom. At process flow 704, the multi-label image model performs multi-label classification to quantify the extracted pathological data into one or more pathological categories (e.g., exudates and / or cotton wool spots). At step 704, the quantified pathological data is mapped to DR staging and urine protein level. The quantified pathological data may include at least one of the following items: exudates, cotton wool spots, microaneurysms and / or venous beading changes. The DR staging may be calculated by a DR staging regular expression rule parser that maps the quantified pathological data to the DR staging. At process flow 706, the quantified pathological data is mapped to urine protein level. At process flow 706, various clinical and demographic parameters of the diabetic individual are determined.

[0069] According to process flow 708, the clinical model obtains manual input of ophthalmologically relevant features, such as DR stage, quantitative pathological data (e.g., exudates and cotton wool spots), urine protein level, and clinical and demographic data (e.g., age, sex, other comorbidities, duration of diabetes, history of hypertension). Using this input, at process flow 710, the clinical model predicts the DKD stage. In some embodiments, the DKD model uses a binary classification to output early DKD or advanced DKD. In other embodiments (e.g., Figure 7 The DKD model uses multi-label classification to output no DKD, early DKD, or advanced DKD.

[0070] Figure 8The retinal assessment process flow 802 and the renal assessment process flow 804 for the ophthalmology clinic and the nephrology clinic, respectively, according to an embodiment disclosed in the present invention, are shown. In the ophthalmology clinic, when a diabetic patient visits the outpatient doctor, the doctor performs a retinal assessment on the patient. Through the retinal assessment, the DR stage can be determined. The DR stage can be classified as no DR, mild / moderate DR, and severe non-proliferative DR / proliferative DR. The retinal assessment can be performed using an AI retinal model (i.e., a multi-label image model) or by a doctor. The DR stage is then fed into the DKD model. The DKD model predicts the DKD stage, which can be classified as early or progressive. Based on the DKD stage, the doctor in the ophthalmology clinic can determine whether it constitutes a referral criterion for referring the patient to a nephrologist. The different DKD stages and their corresponding classifications are shown below:

[0071]

[0072]

[0073] In some embodiments, the DKD model can predict a DKD stage of 0, which is classified as "no DKD". If the patient is referred to a nephrologist, the patient goes to the nephrology clinic for treatment. At the nephrology clinic, the doctor takes a retinal fundus image of the patient. Patients who go to the nephrology clinic for treatment can be assumed to have diabetic retinopathy (DR) disease. The renal assessment performed on diabetic patients includes applying the DKD model. The output of the renal assessment includes multiple data, such as DKD stage, glomerular filtration rate, serum creatinine level, chronic kidney disease (CKD) stage and proteinuria level. The DKD algorithm detects the DKD stage as early or advanced. CKD stage can be classified as stable cases, slow progressors or rapid progressors.

[0074] Figure 9A Various details of the data subsets used in embodiments of the present disclosure are shown. The training data subset includes data from 643 patients, of which 448 patients were classified as early-stage DKD patients and 195 patients were classified as advanced DKD patients. The validation data subset includes data from 168 patients, of which 113 patients were classified as early-stage DKD patients and 55 patients were classified as advanced DKD patients. The test data subset includes data from 159 patients, of which 125 patients were classified as early-stage DKD patients and 34 patients were classified as advanced DKD patients.

[0075] Figure 9BVarious parameters of the multi-label image model and clinical model according to an example embodiment disclosed in the present invention are shown. The area under the curve (AUC) represents the accuracy of the multi-label image model and the clinical model, which are 79% and 86%, respectively. The F1 score also measures the accuracy of the model on the dataset. The F1 scores of the multi-label image model and the clinical model are 47% and 63%, respectively. The sensitivity of the model represents its true positive rate (TPR). The sensitivity of the multi-label image model and the clinical model are 58% and 79%, respectively. The specificity of the model represents its true negative rate (TNR). The specificity of the multi-label image model and the clinical model is 74% and 80%.

[0076] Figure 10A The confusion matrix of the multi-label image model according to an exemplary embodiment of the present disclosure is shown. The accuracy of the multi-label image model in identifying true negatives is 74%, and the accuracy of identifying true positives is 58%. Figure 10B The confusion matrix of the clinical model according to an exemplary embodiment of the present disclosure is shown. The clinical model has an accuracy rate of 80% in identifying true negatives and an accuracy rate of 79% in identifying true positives.

[0077] Figure 11A Graphs depicting the performance of multi-label image models according to example embodiments disclosed herein are shown. The curve labeled 'A' represents the micro-averaged ROC curve (area = 0.97). The curve labeled 'B' represents the macro-averaged ROC curve (area = 0.83). The curve labeled 'C' represents the ROC curve for the vitreous or preretinal hemorrhage category (area = 0.98). The curve labeled 'D' represents the ROC curve for the neovascularization category (area = 0.99). The curve labeled 'E' represents the ROC curve for the focal or lattice laser scar category (area = 1.00). The curve labeled 'F' represents the ROC curve for the fibrovascular change category (area = 1.00). The curve labeled 'G' represents the ROC curve for the peripheral scattered laser scar category (area = 1.00).

[0078] Figure 11B A graph depicting the performance of a clinical model in terms of true positive rate versus false positive rate according to an example embodiment of the present disclosure is shown. The AUC is 0.86.

[0079] Figure 12A A pie chart showing the percentage distribution of DR stages associated with early-stage DKD in multiple samples according to an exemplary embodiment of the present disclosure. Among early-stage DKD cases, 5% had mild non-proliferative DR, 20% had moderate non-proliferative DR, 8% had no significant retinopathy, 25% had proliferative DR, and 42% had severe non-proliferative DR.

[0080] Figure 12BA pie chart showing the percentage distribution of DR stages associated with advanced DKD in multiple samples according to an exemplary embodiment of the present disclosure. Among advanced DKD cases, 1% had mild non-proliferative DR, 13% had moderate non-proliferative DR, 7% had no significant retinopathy, 36% had proliferative DR, and 43% had severe non-proliferative DR.

[0081] Figure 13 A system 1300 for implementing an example embodiment of the present disclosure is shown. The system 1300 includes an image acquisition unit 1302, an image processing unit 1304, and a display 1312.

[0082] Image acquisition unit 1302 is responsible for acquiring a set of ophthalmic images of a person (e.g., a diabetic patient). In one embodiment, the ophthalmic images may be fundus images, and in another embodiment, the ophthalmic images may be OCT images. Image acquisition unit 1302 may be a device such as a fundus camera or an OCT machine. The set of ophthalmic images may include images acquired from the diabetic patient's left and right eyes.

[0083] The set of ophthalmic images is transmitted to an image processing unit 1304. The image processing unit 1304 includes a pre-processing module 1306, a first deep learning module 1308, and a second deep learning module 1310. The image processing unit 1306 processes the set of ophthalmic images acquired / collected by the image acquisition unit 1302 to determine whether there is a vascular abnormality pattern in the ophthalmic image set for DKD staging.

[0084] The preprocessing module 1306 is responsible for preprocessing the ophthalmic image set. In one embodiment, preprocessing the acquired images includes applying Gaussian blur to smooth the ophthalmic image set, and then applying BenGraham preprocessing to the smoothed ophthalmic image set. In embodiments where a multi-label image model is being trained, the preprocessing module 1306 may preprocess a set of reference images (e.g., a training dataset).

[0085] The preprocessed ophthalmic image set is then fed to a first deep learning module 1308. Module 1308 generates a first high-level feature set based on the ophthalmic image set. More specifically, generating the high-level feature set includes extracting vascular abnormality patterns (i.e., pathological data) from the ophthalmic image set and quantifying the vascular abnormality patterns. Module 1308 can employ a multi-label image model to extract and quantify the vascular abnormality patterns. Module 1308 can use a rule parser (e.g., a DR stage regular expression rule parser) to map the quantified vascular abnormality patterns to DR stages indicating urine protein levels. The first high-level feature set includes the quantified vascular abnormality patterns, DR stages, and urine protein levels.

[0086] The second deep learning module 1310 generates a second high-level feature set based on the ophthalmic image set, the first high-level feature set, and clinical and demographic data. The second deep learning module 1310 can receive the ophthalmic image set directly from the image acquisition unit 1302 or the pre-processing module 1306. The second deep learning module 1310 also receives as input the first high-level feature set generated by the first deep learning module 1308 and the clinical and demographic data. The second high-level feature set includes the diabetic nephropathy stage.

[0087] The display 1312 includes a user interface 1314 through which an ophthalmic image set can be uploaded and thereby transmitted to the image processing unit 1304. The user interface 1314 can also include a plurality of input fields through which a user can manually enter a first high-level feature set and clinical and demographic data for use by the second deep learning module 1310. The second high-level feature set generated by the second deep learning module 1310 can be displayed on the user interface 1314.

[0088] although Figure 13 Although not shown, system 1300 may include at least one memory and at least one processor for performing functionality associated with image acquisition unit 1302, image processing unit 1304, and display 1312. The at least one memory may be a non-transitory computer-readable storage medium capable of storing computer program instructions or computer code for execution by the at least one processor to implement the performance of method 1400 (described below). The at least one memory may be multiple memories distributed across multiple computing devices. The at least one memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), a hard drive, a solid-state drive, static random access memory (SRAM), etc. The at least one processor may be an electronic component that executes computer programs or computer instructions to implement the functionality of the various components of system 1300. The at least one processor may be a single processor or multiple processors. The at least one processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functionality described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. At least one processor may also be implemented as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0089] Figure 14 shows a method 1400 for detecting DKD staging based on DR staging according to an example embodiment of the present disclosure. At step 1402, a set of ophthalmic images of a person is acquired by the image acquisition unit 1302. The set of ophthalmic images may correspond to retinal fundus images or OCT images. At step 1404, the preprocessing module 1306 processes the set of ophthalmic images. The preprocessing may include applying Gaussian blur and Ben Graham preprocessing to smooth the set of ophthalmic images. In another embodiment, the preprocessing module applies preprocessing techniques to clinical and demographic parameters to eliminate noise therein. At step 1406, the first deep learning module 1308 extracts pathological data from the set of ophthalmic images using a multi-label image model. The pathological data indicates vascular abnormalities in the set of ophthalmic images. At step 1408, the first deep learning module 1308 quantifies the extracted pathological data, wherein the quantification is based on the vascular abnormalities in the set of ophthalmic images. At step 1410, the first deep learning module 1308 maps the quantified pathological data to the DR stage with the help of the DR staging rule parser. The first deep learning module 1308 also maps the quantified pathological data to the urine protein level. At step 1412, the second deep learning module 1310 uses a clinical model to receive clinical (e.g., other comorbidities, diabetes course and diabetes history) and demographic (e.g., age and gender) parameters. In one embodiment, the clinical and demographic parameters can also be preprocessed by the preprocessing module 1306. The preprocessing module 1306 can apply Gaussian blur to smooth the clinical and demographic parameters and eliminate noise therein. The preprocessing module 1306 can also apply Ben Graham preprocessing to the smoothed clinical and demographic parameters for denoising. In addition to the clinical and demographic parameters, the second deep learning module 1310 also receives the output of the first deep learning module 1308, i.e., the quantified pathological data, DR stage and urine protein level. The second deep learning module 1310 predicts the DKD stage based on the clinical and demographic parameters and the output of the first deep learning module 1308. The second deep learning module 1310 can also use the set of pre-processed ophthalmic images as input to predict the DKD stage.

[0090] In some embodiments, the image processing unit 1304 may divide each image in the set of ophthalmic images into four quadrants, wherein the first deep learning module 1308 extracts and quantifies pathological data in each quadrant based on the vascular abnormalities in each quadrant. The vascular abnormalities in each quadrant of the ophthalmic image represent vascular abnormalities that lead to protein leakage in the kidney. By quantifying the pathological data in each quadrant, the first deep learning module 1308 can map it to a urine protein level that indicates the extent of protein leakage in the urine.

[0091] For simplicity, embodiments of the present invention are disclosed with reference to a two-step multi-label image model (employed by the first deep learning module 1308) and a clinical model (employed by the second deep learning module 1310). However, this should not be construed as limiting, as in some embodiments, the first deep learning module 1308 and the second deep learning module 1310 may employ multiple models to perform the respective functions. For example, the first deep learning module 1308 may include a first trained deep learning model for extracting and quantifying pathology data in each quadrant of the ophthalmic image in the set of ophthalmic images. The first deep learning module 1308 may include a second trained deep learning model for mapping (also referred to as "classifying") the quantified pathology data to a DR stage. The first deep learning module 1308 may include a third trained deep learning model for mapping the quantified pathology data to a urine protein level. The second deep learning module 1310 may include a fourth trained deep learning model for receiving the DR stage and urine protein level from the second trained deep learning model and the third trained deep learning model, respectively. The fourth trained deep learning model can also predict DKD stage based on input received from the second trained deep learning model and the third trained deep learning model and clinical and demographic parameters.

[0092] In some embodiments, method 1400 may include other steps not shown and / or may omit certain steps not shown, and thus this should not be construed as limiting the scope of the present disclosure.

[0093] In the specification, exemplary embodiments of the present invention have been disclosed. Although specific terms are employed, they are used in a generic and descriptive sense only and not for the purpose of limiting the scope of the invention.

Claims

1. A system comprising: an image acquisition unit configured to acquire a set of ophthalmic images of a person; as well as an image processing unit configured to process the set of ophthalmic images obtained from the image acquisition unit, Wherein, the image processing unit includes: a preprocessing module configured to preprocess the set of ophthalmic images; The first deep learning module is configured to: extracting pathological data indicating vascular abnormalities from a preprocessed set of ophthalmic images; quantifying the extracted pathological data based on the vascular abnormality; and mapping the quantified pathological data to diabetic retinopathy stage and urine protein level; and The second deep learning module is configured to: clinical and demographic parameters were received; receiving the quantified pathological data, the diabetic retinopathy stage, and the urine protein level from the first deep learning module; and The diabetic nephropathy stage is predicted based on the quantified pathological data, the diabetic retinopathy stage, the urine protein level, and the clinical and demographic parameters.

2. The system according to claim 1, wherein: The vascular abnormality in the set of ophthalmic images represents a vascular abnormality in the kidney that causes protein to leak into urine.

3. The system according to claim 2, wherein: The image processing unit is configured to divide each image in the set of ophthalmic images into four quadrants, wherein based on the vascular abnormalities in each quadrant, the first deep learning module uses at least one deep learning technique to extract and quantify pathological data in each quadrant.

4. The system according to claim 3, wherein: The first deep learning module maps the pathological data quantified in each quadrant to a urine protein level indicating the extent of protein leakage in urine.

5. The system according to claim 1, wherein: The pre-processing module is configured to: Apply a Gaussian blur to do at least one of the following: smoothing the set of ophthalmic images; and smoothing the clinical and demographic parameters and removing noise from the clinical and demographic parameters; and Apply Ben Graham preprocessing to at least one of the following: a smoothed set of ophthalmic images; and Smoothed clinical and demographic parameters were used for denoising.

6. The system according to claim 3, wherein: The first deep learning module includes: a first trained deep learning model for extracting and quantifying pathology data in each quadrant of an ophthalmic image in the set of ophthalmic images; a second trained deep learning model for mapping the quantified pathology data to the diabetic retinopathy stage; and A third trained deep learning model is used to map the quantified pathology data to the urine protein level.

7. The system according to claim 6, wherein: The second deep learning module includes: a fourth trained deep learning model, wherein the fourth trained deep learning model: receiving the diabetic retinopathy stage and the urine protein level from the second trained deep learning model and the third trained deep learning model, respectively; and The diabetic nephropathy stage is predicted based on the diabetic retinopathy stage and the urine protein level.

8. The system according to claim 1, wherein: The clinical and demographic parameters include at least one of the following: age, gender, other comorbidities, duration of diabetes, and history of hypertension.

9. The system according to claim 1, wherein: The vascular abnormality in the set of ophthalmic images represents at least one of the following: no abnormality, microaneurysm, punctate or macular hemorrhage, hard exudates, cotton-wool spots, intraretinal hemorrhage, venous beading, intraretinal microvascular abnormality, neovascularization, and vitreous or preretinal hemorrhage.

10. The system according to claim 9, wherein: If the pattern of vascular abnormalities represents no abnormality, the diabetic retinopathy stage is "no diabetic retinopathy." 11. The system according to claim 9, wherein: If the pattern of vascular abnormalities represents microaneurysms, the diabetic retinopathy stage is mild nonproliferative diabetic retinopathy.

12. The system according to claim 9, wherein: If the pattern of vascular abnormalities represents microaneurysms, punctate or macular hemorrhages, hard exudates, and cotton-wool spots, the diabetic retinopathy stage is moderate nonproliferative diabetic retinopathy.

13. The system according to claim 9, wherein: If the pattern of vascular abnormalities represents microaneurysms, punctate or macular hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading, and intraretinal microvascular abnormalities, the diabetic retinopathy stage is severe non-proliferative diabetic retinopathy.

14. The system according to claim 9, wherein: If the pattern of vascular abnormalities represents microaneurysms, punctate or macular hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading, intraretinal microvascular abnormalities, neovascularization, and vitreous or preretinal hemorrhages, the diabetic retinopathy stage is proliferative diabetic retinopathy.

15. The system of claim 1, wherein: The urine protein level was categorized as: normal, microalbuminuria, or macroalbuminuria.

16. The system of claim 1, wherein: The predicted diabetic nephropathy stage is classified as one of the following: no diabetic nephropathy, early diabetic nephropathy, advanced diabetic nephropathy, or late diabetic nephropathy.

17. The system of claim 1, wherein: The predicted diabetic nephropathy stage indicates the progression of renal failure, wherein the renal failure is classified as one of: stable, rapid, or slow.

18. The system of claim 1, wherein: The set of ophthalmic images and the clinical and demographic parameters are used as independent input information of the second deep learning module, respectively.

19. The system of claim 1, wherein: The set of ophthalmic images includes fundus images of the right eye and the left eye.

20. The system of claim 1, wherein: The predictions of the second deep learning module represent the criteria for referral to a nephrologist.

21. The system of claim 1 , comprising a display having a user interface, wherein: The quantified pathological data, the diabetic retinopathy stage, the urine protein level, and the predicted diabetic nephropathy stage are displayed on the user interface.

22. The system of claim 1, wherein: The image acquisition unit is at least one of: a fundus camera; and an optical coherence tomography (OCT) machine.

23. The system of claim 1, wherein: The set of ophthalmic images are infrared images.

24. A method comprising: collecting a set of ophthalmic images of a person by an image acquisition unit; Processing the set of ophthalmic images obtained from the image acquisition unit by an image processing unit, wherein the processing comprises: preprocessing the set of ophthalmic images by a preprocessing module; extracting pathological data indicating vascular abnormalities from a preprocessed set of ophthalmic images through a first deep learning module; quantifying the extracted pathological data based on the vascular abnormality by the first deep learning module; mapping the quantified pathological data to diabetic retinopathy stage and urine protein level through the first deep learning module; Received through the second deep learning module: clinical and demographic parameters; and the quantified pathological data, the diabetic retinopathy stage, and the urine protein level from the first deep learning module; and The second deep learning module predicts the diabetic nephropathy stage based on the quantified pathological data, the diabetic retinopathy stage, the urine protein level, and the clinical and demographic parameters.

25. The method according to claim 24, wherein The vascular abnormality in the set of ophthalmic images represents a vascular abnormality in the kidney that causes protein to leak into urine.

26. The method according to claim 25, comprising: Each image in the set of ophthalmic images is divided into four quadrants by the image processing unit, wherein based on the vascular abnormalities in each quadrant, the first deep learning module uses at least one deep learning technology to extract and quantify pathological data in each quadrant.

27. The method according to claim 26, wherein The first deep learning module maps the quantified pathology data in each quadrant to a urine protein level indicating an extent of protein leakage in urine.

28. The method according to claim 23, wherein Preprocessing the set of ophthalmic images comprises: Apply a Gaussian blur to do at least one of the following: smoothing the set of ophthalmic images; and smoothing a set of clinical and demographic parameters and removing noise from the set of clinical and demographic parameters; and Apply Ben Graham preprocessing to at least one of the following: a smoothed set of ophthalmic images; and Smoothed clinical and demographic parameters were used for denoising.

29. The method according to claim 26, wherein The first deep learning module includes: a first trained deep learning model for extracting and quantifying pathology data in each quadrant of an ophthalmic image in the set of ophthalmic images; a second trained deep learning model for mapping the quantified pathology data to the diabetic retinopathy stage; and A third trained deep learning model is used to map the quantified pathology data to the urine protein level.

30. The method according to claim 29, wherein The second deep learning module includes: a fourth trained deep learning model, wherein the fourth trained deep learning model: receiving the diabetic retinopathy stage and the urine protein level from the second trained deep learning model and the third trained deep learning model, respectively; and The diabetic nephropathy stage is predicted based on the diabetic retinopathy stage and the urine protein level.

31. The method of claim 24, wherein: The clinical and demographic parameters include at least one of the following: age, gender, other comorbidities, duration of diabetes, and history of hypertension.

32. The method of claim 24, wherein: The vascular abnormality in the set of ophthalmic images represents at least one of the following: no abnormality, microaneurysm, punctate or macular hemorrhage, hard exudates, cotton-wool spots, intraretinal hemorrhage, venous beading, intraretinal microvascular abnormality, neovascularization, and vitreous or preretinal hemorrhage.

33. The method according to claim 32, wherein If there is no abnormality, the diabetic retinopathy stage is no diabetic retinopathy.

34. The method of claim 32, wherein: If the pattern of vascular abnormalities represents microaneurysms, the diabetic retinopathy stage is mild nonproliferative diabetic retinopathy.

35. The method of claim 32, wherein: If the pattern of vascular abnormalities represents microaneurysms, punctate or macular hemorrhages, hard exudates, and cotton-wool spots, the diabetic retinopathy stage is moderate nonproliferative diabetic retinopathy.

36. The method of claim 32, wherein: If the pattern of vascular abnormalities represents microaneurysms, punctate or macular hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading, and intraretinal microvascular abnormalities, the diabetic retinopathy stage is severe non-proliferative diabetic retinopathy.

37. The method of claim 32, wherein: If the pattern of vascular abnormalities represents microaneurysms, punctate or macular hemorrhages, hard exudates, cotton-wool spots, intraretinal hemorrhages, venous beading, intraretinal microvascular abnormalities, neovascularization, and vitreous or preretinal hemorrhages, the diabetic retinopathy stage is proliferative diabetic retinopathy.

38. The method of claim 24, wherein: The urine protein level was categorized as: normal, microalbuminuria, or macroalbuminuria.

39. The method of claim 24, wherein: The predicted diabetic nephropathy stage was classified as one of the following: no diabetic nephropathy, early diabetic nephropathy, or advanced diabetic nephropathy.

40. The method of claim 24, wherein The predicted renal disease stage indicates the progression of renal failure, wherein the renal failure is classified as one of the following: stable, rapid, or slow.

41. The method of claim 24, wherein: The ophthalmic images and the clinical and demographic parameters are used as independent input information for the second deep learning module, respectively.

42. The method of claim 24, wherein: The set of ophthalmic images includes fundus images of the right eye and the left eye.

43. The method of claim 24, wherein: The predictions of the second deep learning module represent the criteria for referral to a nephrologist.

44. The method of claim 24, wherein The quantified pathological data, the diabetic retinopathy stage, the urine protein level and the predicted diabetic nephropathy stage are displayed on a user interface of a display.

45. The method of claim 24, wherein The image acquisition unit is at least one of: a fundus camera; and an optical coherence tomography (OCT) machine.

46. The method of claim 24, wherein The set of ophthalmic images are infrared images.

47. A computer-readable medium comprising instructions which, when executed by at least one processor in a computer system, cause the computer system to perform the method of any one of claims 24 to 46.

Images