Automated disease detection using retinal images

Abstract

Described herein is a patient screening system for providing a recommendation for screening a patient for a potential disease or risk of disease based on the patient's health record and retinal image. The patient screening system can include an optical imaging device operable at a physician's office, and an associated method configured to generate the recommendation. The patient screening system can implement various AI / ML models trained on a training dataset of anonymized patient data. The patient screening system can be based on discovering correlations in the training dataset between features of retinal images and health records, and corresponding disease diagnoses contained in the health records. The patient screening system can also implement classifiers for various diseases based on the data in the training dataset. Any patient screening based on the recommendation can be followed up on, and the results of such screening can be used to improve the performance of the patient screening system.

Description

Related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 601,463, filed November 21, 2023, entitled “AUTOMATED DISEASE DETECTIONUSING RETINAL IMAGES”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to a technique for automatically detecting potential diseases based on retinal images and patient health records and providing further screening recommendations related to the detected diseases. Background Technology

[0003] Vision screenings, such as retinal scans, typically include screening for eye diseases. However, many systemic diseases can show detectable signs on a patient's retinal scan, sometimes even in the early stages or other asymptomatic phases of the disease. Retinal scans are non-invasive and can be easily performed in a primary care physician's office as part of routine health screenings. However, primary care physicians may not be able to see signs of disease in retinal scans. Manual analysis of a patient's retinal scan by a retinal specialist, in addition to increasing the cost and complexity of health screenings, may also fail to identify early signs of disease because the retinal specialist may not be familiar with the patient's overall medical history.

[0004] Therefore, it would be advantageous to be able to automatically screen for a variety of potential diseases in patients. Examples of diseases that can be screened include heart disease, kidney disease, neurodegenerative diseases, anemia, sleep apnea, fibromyalgia, multiple sclerosis, and so on.

[0005] The various examples disclosed herein are intended to overcome one or more of the defects noted above. Summary of the Invention

[0006] In an example of this disclosure, a method includes: receiving an image of the retina of a patient's eye; receiving patient data corresponding to the patient from the patient's electronic medical record (EMR) by a processor; determining features in the image; and determining a confidence level associated with a first disease by inputting the features and at least a portion of the patient data into a machine learning (ML) model. The method further includes: determining a recommendation to screen the patient based on the confidence level being higher than a threshold; and providing an output indicating the recommendation to an output device by the processor.

[0007] In another example of this disclosure, a system includes a memory, a processor, and computer-executable instructions stored in the memory and executed by the processor. When executed, the instructions cause the processor to perform operations including: receiving an image of the retina of a patient's eye; receiving patient data corresponding to the patient from the patient's electronic medical record (EMR); determining features in the image; and determining a confidence level associated with a first disease by inputting at least a portion of the features and the patient data into a machine learning (ML) model; determining a recommendation to screen the patient based on the first disease based on the confidence level being higher than a threshold; and providing an output instructing the recommendation to the patient's EMR.

[0008] In yet another example of this disclosure, a non-transitory computer-readable storage medium stores processor-executable instructions that, when executed, cause one or more processors to: receive an image of the retina of a patient's eye from an optical imaging device; access the patient's EMR data from an electronic medical record (EMR) storage device; determine features in the image; determine a confidence level associated with a first disease by inputting the features and at least a portion of the EMR data into a machine learning (ML) model; and determine a recommendation to screen the patient based on the first disease, based on the confidence level being higher than a threshold. Attached Figure Description

[0009] The features, nature, and various advantages of this disclosure will become more apparent upon consideration of the following detailed description taken in conjunction with the accompanying drawings.

[0010] Figure 1 An example environment is shown for recommending screening for one or more diseases based on a combination of a patient's retinal scans and health records.

[0011] Figure 2 A first block diagram of an example process for training a recommendation system to identify diseases or disease risks, as described herein, is shown.

[0012] Figure 3 A second block diagram is shown as an example process for generating recommendations for screening one or more diseases, as described herein.

[0013] Figure 4 A first flowchart illustrating an example method of this disclosure is provided.

[0014] Figure 5 A second flowchart illustrating an example method of this disclosure is provided.

[0015] Figure 6 At least one example device is shown that is configured to implement and / or perform some or all of the functions discussed herein.

[0016] In the accompanying drawings, the leftmost numeral of the reference numeral indicates the figure in which that reference numeral first appears. The same reference numerals are used in different drawings to denote similar or identical items or features. The drawings are not drawn to scale. Detailed Implementation

[0017] This disclosure relates to a disease identification system and a corresponding method, the system being programmed or otherwise configured to generate recommendations for further screening. Such an example disease identification system may be configured to accept one or more retinal images of a patient and the patient's electronic health record as input, and generate recommendations for further screening as output. Although many retinal findings are not always exclusive to a specific disease, when observed in conjunction with medical information in the patient's health record, these findings may indicate early signs of a specific disease and may enable a vision screening system to recommend further screening for said disease. By allowing for early detection and treatment of said disease, such screening can help prevent the disease from progressing to a severe stage and / or prevent life-threatening conditions.

[0018] In examples, disease identification systems can leverage artificial intelligence (AI)-generated correlations between retinal scan features and information in a patient's medical history, including test results and trends. Such disease identification systems can be included in patient health screenings, where an operator can capture retinal images using a vision screening device, and recommendations can be offered to the operator or a clinician who may differ from the operator. The disease identification system can identify one or more diseases associated with features in the retinal images and / or information in the health records based on analysis of information in the retinal images and health records. Specifically, this disclosure relates to methods for screening systemic and / or ophthalmic diseases that develop over time and would otherwise require complex and / or invasive testing for diagnosis. In some examples, such diseases may require input from multiple specialists and may therefore not be diagnosed at an early stage. The methods of this disclosure can recommend screening for such diseases based on data available during medical appointments at primary care physician clinics.

[0019] Based at least in part on confidence levels associated with the determination of one or more diseases, the system can generate output including at least one of a recommendation or diagnosis associated with a patient. Such output (e.g., a recommendation and / or diagnosis) can indicate a detected disease and / or disease risk, indicate that the patient needs additional screening, and / or indicate that the screening was normal (e.g., no disease was detected where the confidence level exceeds a threshold level). In examples, the system can determine diseases using trained machine learning (ML) models or other AI techniques trained using anonymized data from a large number of patient health records (including disease diagnoses and medical test results over extended time periods) and corresponding retinal images. In some examples, the AI ​​techniques can include data-driven discovery of correlations between patient-associated data and disease diagnoses. In such examples, recommendations can be based on discovered correlations, including correlations between diseases and trends in the data. (See reference...) Figures 1 to 6 Various implementations of this disclosure are described in detail. It should be understood that although these figures illustrate the methods and systems of this disclosure, any examples set forth in this specification are not intended to be limiting, but merely to illustrate some of the many possible implementations.

[0020] Figure 1 An example environment 100 is shown for screening patient 102 to identify diseases and / or disease risks that patient 102 may exhibit. This screening may be based at least in part on an operator 104 imaging the patient 102's eyes using an optical imaging device 106. Patient 102 can be any individual monitored in a clinical setting, such as someone presenting for a medical visit at a physician's office. In the example, patient 102 may be screened at a primary care physician's office during a routine health check, and operator 104 may be a healthcare provider such as a physician, physician assistant (PA), nurse, medical student, nursing student, medical technician, etc.

[0021] In the example, the optical imaging device 106 may be configured to acquire one or more images 108 of the retina and / or fundus (which includes the posterior surface of the eye, comprising the retina, macula, optic disc, fovea, and vessels) of at least one eye of the patient 102. In various implementations, the optical imaging device 106 may include an optical coherence tomography (OCT) camera configured to acquire OCT or OCT angiography (OCTA) images of the eye of the patient 102. In some cases, the optical imaging device 106 may include a slit-lamp imaging device configured to acquire slit-lamp images (or projected images) of the eye of the patient 102. In some examples, the optical imaging device 106 may include at least one fluorescence imaging device configured to acquire one or more fluorescence angiography images of the eye of the patient 102. In some examples, the optical imaging device 106 may be configured to generate one or more color fundus (e.g., retinal) photographs (CFP) of the patient 102's eye, one or more fluorescein angiography (FA) images of the patient 102's eye, one or more indocyanine green (ICG) angiography images of the patient 102's eye, one or more autofluorescence (FAF) images of the patient's fundus, or any combination thereof.

[0022] Environment 100 may also include an electronic medical record (EMR) system 110 configured to store EMR data 112 associated with patient 102. As used herein, the terms “electronic health record,” “electronic medical record,” “EMR,” and their equivalents may broadly refer to stored data in any form of storage (e.g., temporary, transient, permanent, etc.) indicating an individual’s medical history and / or medical condition, which may be accessed (e.g., modified and / or retrieved) by one or more computing devices. An individual’s EMR data may include data indicating the individual’s previous or current medical diagnoses, diagnostic tests, or treatments. Furthermore, the EMR data may indicate an individual's demographic information (e.g., age, sex, ethnicity, etc.), individual parameters (e.g., vital signs, blood pressure, body mass index (BMI), etc.), lifestyle information (e.g., smoking or drug use, physical activity level, diet, alcohol use, etc.), records of one or more medical visits attended by the individual, medications prescribed or administered to the individual, treatments administered to the individual (e.g., surgery, outpatient procedures, etc.), results of diagnostic tests performed on the individual, individual identification information (e.g., name, date of birth, etc.), or combinations thereof. In some examples, the EMR system 110 may be implemented on one or more servers, such as servers located in a data center.

[0023] In some examples, the EMR system 110 may be connected to the clinical device 114 via a network 116. The clinical device 114 may include a computing device, such as a device including at least one processor configured to perform operations. In some cases, the operations are stored in memory in an executable format. Examples of computing devices include personal computers, tablet computers, smart TVs (TVs), mobile devices, mobile phones, or Internet of Things (IoT) devices. In some examples, the clinical device 114 may be operated by an operator 104 and may receive images 108 captured by an optical imaging device 106. The clinical device 114 may provide a user interface to the operator 104, for example, to access or input data related to the patient 102 and / or view images 108 captured by the optical imaging device 106. In examples, the clinical device 114 or the optical imaging device 106 may store images 108 of the patient 102's eyes in association with the patient 102's EMR data 112 in the EMR system 110.

[0024] In the example, network 116 can represent one or more communication networks. Examples of communication networks include at least one wired interface (e.g., Ethernet interface, fiber optic interface, etc.) and / or at least one wireless interface (e.g., BLUETOOTH interface, Wi-Fi interface, Near Field Communication (NFC) interface, Long Term Evolution (LTE) interface, New Radio (NR) interface, etc.). In some examples, data or other signals can be transmitted via a wide area network (WAN) such as the Internet. Figure 1 Data is transmitted between components. In some cases, the data may include one or more data packets (e.g., Internet Protocol (IP) packets), datagrams, or a combination thereof.

[0025] In various examples, clinical device 114 may be connected to remote computing device 118 via network 116, such as a server implemented on a cloud platform. In one example, clinical device 114 may upload image 108 captured by optical imaging device 106 to remote computing device 118. Remote computing device 118 may implement image analysis system 120, which receives image 108 captured by optical imaging device 106 as input and analyzes image 108 to determine various features. In other examples, optical imaging device 106 may communicate directly with remote computing device 118 to upload image 108, and / or image analysis system 120 may be implemented entirely or partially on clinical device 114.

[0026] In some examples, the image analysis system 120 may implement one or more image processing components to identify salient landmarks in an image 108 of the eye. As used herein, the term "landmark" and its equivalents may refer to anatomical structures observed in a healthy or diseased eye. Examples of landmarks include one or more of the following: macula, optic disc (OD), retina, cornea, iris, lens, one or more retinal layers, one or more blood vessels, or fovea. The one or more image processing components may include a first machine learning (ML) model configured to output the location of one or more landmarks from a set of landmarks in an input image of the posterior aspect of the eye. For example, the location may be indicated by identifying pixels in the input image corresponding to the region of the respective landmark.

[0027] In some examples, the image analysis system 120 may implement a feature detector to identify different types of ophthalmic features in the image 108. As used herein, the term "feature" and its equivalents may refer to a structure or visible sign within an image of the eye that may be associated with one or more diseases and / or medical conditions. Examples of ophthalmic features may include one or more of the following: microaneurysms, hemorrhages, drusen, exudates, edema, cup / disc ratio (CDR), focal arteriolar constriction, arteriovenous crossing impressions, cotton wool spots, emboli, red spots, retinal albinism, Hollenhorst spots, Ross spots, microinfarctions, coagulated fibrin, neovascularization in other sites (NVE), vitreous hemorrhage (VH), preretinal hemorrhage (PRH), optic disc neovascularization (NVD), venous beading, intraretinal microvascular anomalies (IRMA), vessel diameter and topology, retinal vessel diameter, mean diameter of retinal arterioles and mean diameter of retinal venules summarized as arteriovenous ratio (AVR), etc.

[0028] In some examples, one or more of the feature detectors may include a second machine learning (ML) model configured to output a binary indication of the presence of a corresponding feature and a confidence score, and / or the location of the feature in the input image. Furthermore, the image analysis system 120 may output various properties of the detected ophthalmic feature, such as the feature's location in the image (e.g., which quadrant of the eye the feature is located in, its distance from and / or orientation to one or more of the landmarks, its proximity to one or more landmarks, etc.), and the feature's size in the image (e.g., relative to the size of the landmark, based on the number of pixels, based on the feature area as a percentage of the total retinal area, etc.). In examples, the image analysis system 120 may also implement image processing techniques to determine one or more measurements associated with the detected ophthalmic feature, such as the diameter of a blood vessel, the number of spots, the density of spots, the texture elements of the feature, etc.

[0029] In some examples, the image analysis system 120 can also determine ophthalmic features that indicate colors associated with landmarks in the eye. For example, the image analysis system 120 can determine average color values ​​(e.g., in RGB, HSI, CIE L) that indicate the optic disc, blood vessels, fovea, or retina. a b CIE L u v The image analysis system 120 can also determine ophthalmic features (in color spaces) that indicate the relative intensity between various landmarks or regions of the fundus.

[0030] Furthermore, in some examples of this disclosure, the image analysis system 120 can generate a standardized retinal image from an image 108 captured by the optical imaging device 106. In these examples, the image analysis system 120 generates the standardized retinal image by scaling (e.g., scaling to a standard size and aspect ratio) and normalizing (e.g., histogram stretching to cover the full range of brightness and contrast levels) the image captured by the optical imaging device 106, such that retinal landmarks (e.g., optic disc, fovea, important blood vessels, etc.) are located at predetermined positions relative to the image boundaries of the standardized retinal image. For example, after such normalization, the landmarks can be aligned in multiple different standardized retinal images (e.g., appearing at the same position relative to their respective image boundaries).

[0031] In the example, the image analysis system 120 can associate a timestamp indicating the date / time of capture of image 108 with each of the ophthalmic features and standardized retinal images determined from image 108.

[0032] In the example, the first and second ML models of the image analysis system 120 can be pre-trained based on training images (e.g., from a training dataset). For example, the first ML model can be trained on a first training dataset including images of the posterior portion of the eye (e.g., retinal images) labeled with the set of markers, while the second ML model can be trained on a second training dataset comprising example images depicting ophthalmic features, and ground truth information associated with each image to identify the ophthalmic features depicted therein. In the example, one or more expert annotators can review the images in the training dataset and use the ground truth information to indicate whether the markers and / or individual images depict one or more ophthalmic features.

[0033] In some examples, the image analysis system 120 may also be configured to assess the image quality of the image 108 captured by the optical imaging device 106. As used herein, the term "image quality" and its equivalents may refer to the degree to which an image accurately represents the objects or other items depicted in the image. Several factors may be associated with image quality, such as blurriness or other distortions in the image. In some examples, if the image quality is determined to be below a threshold, the image analysis system 120 may avoid analyzing the image and / or may generate a notification indicating insufficient image quality. The image analysis system 120 may transmit this notification to the clinical device 114, and based on this notification, the operator 104 may re-capture the image using the optical imaging device 106. In examples where the medical image analysis system 120 assesses that the image has sufficient quality, the image analysis system 120 may further analyze the image to detect ophthalmic features. Techniques for identifying exemplary diseases based on ophthalmic features of image 108 (e.g., determined by image analysis system 120) are described in U.S. Patent Application Serial No. 17 / 709,950, filed March 31, 2022, entitled “Automated disease identification based on ophthalmic images,” which is incorporated herein by reference in its entirety and for all purposes.

[0034] In the example, the remote computing device 118 may also implement an EMR data extractor component 122. The EMR data extractor component 122 can process the EMR data 112 to extract data of interest, which may include demographic data (e.g., age, sex, ethnicity, etc.), health parameters (e.g., vital signs, blood pressure, body mass index (BMI), etc.), lifestyle information (e.g., smoking or drug use, physical activity level, diet, alcohol use, etc.), geographic location of residence, previous or current medical diagnoses, diagnostic tests and results, medical treatments, prescriptions, etc. The EMR data extractor component 122 may also ignore certain data in the EMR data 112 (e.g., name, address, emergency contact, etc.) based on their relevance to the disease state. In the example, the EMR data extractor component 122 can convert the data of interest from the EMR data 112 into a predefined standard representation. As an example, EMR data extractor component 122 can create a vector (e.g., an EMR feature vector) that can be a one-dimensional or two-dimensional array of data fields indicating data of interest from EMR data 112 and associate a timestamp indicating the date / time at which the data of interest was identified or added to EMR data 112 with each data field. In some examples, EMR data extractor component 122 may include in the EMR feature vector a first dataset (e.g., numerical values) from EMR data 112 in their respective original forms (e.g., age, BMI value, blood glucose level, etc.), a second dataset represented as levels corresponding to numerical ranges (e.g., 1: low; 2: medium; 3: high) (e.g., 1 to 3, 1 to 5, 1 to 10, etc.), and a third dataset (e.g., text data) represented as category numbers (e.g., 0: non-smoker, 1: smoker in the "smoking status" data field; 0: sedentary, 1: low activity, 2: moderate activity, 3: active, etc. in the "activity level" data field). In some examples, the EMR data extractor component 122 can also classify the values ​​of the EMR data 112 into range levels and input the corresponding range levels into the EMR feature vector. For example, the patient's age can be classified as: 1: under 2 years old; 2: 2 to 5 years old; 3: 5 to 11 years old; 4: 12 to 18 years old; 5: 18 to 54 years old; and 6: 55 years and older, where the EMR feature vector includes range level indicators (1 to 6) corresponding to the patient's age. As another example, the patient's address (e.g., postal code, city name, etc.) can be mapped to a broader geographic area (e.g., county, state, part of a country, etc.) and included in the EMR feature vector.

[0035] In the example, the remote computing device 118 can also implement an AI (e.g., artificial intelligence-based) recommender system 124 to generate recommendations for patient 102 based on image 108 and EMR data 112. For example, image analysis system 120 can provide detected ophthalmic features and / or normalized retinal images as input to AI recommender system 124, and EMR data extractor 122 can provide a representation of EMR data 112 as input to AI recommender system 124.

[0036] In the example, the AI ​​recommender system 124 can be configured to determine whether patient 102 exhibits one or more diseases or disease risks, and generate recommendations based on that determination. As used herein, the term "disease" and its equivalents can refer to a pathological condition or health risk. Examples of diseases that can be identified by the AI ​​recommender system 124 include at least one of the following: heart disease, kidney disease, anemia, Alzheimer's disease, Parkinson's disease, multiple sclerosis, obstructive sleep apnea, fibromyalgia, Lyme disease, stroke risk, heart disease risk, kidney disease risk, etc. In the example, the AI ​​recommender system 124 can also receive EMR data 112 associated with patient 102 from clinical device 114 and / or directly from EMR system 110. In various examples, the AI ​​recommender system 124 can use an image 108 captured by optical imaging device 106, ophthalmic features and / or normalized images determined by image analysis system 120, EMR data 112 associated with patient 102, or a combination thereof, as input to determine that patient 102 exhibits one or more diseases or disease risks.

[0037] In the examples disclosed herein, the AI ​​recommender system 124 may include machine learning models, expert systems, statistical models, etc., trained on a large anonymized training dataset, which includes patient health records containing disease diagnoses and medical test results over an extended period, along with corresponding retinal images. Such training data can be accumulated from patient data in the EMR system 110 and / or from publicly available health research (e.g., heart disease risk studies, sleep apnea studies, diabetes risk studies, etc.). In some examples, during the training phase, the AI ​​recommender system 124 may perform data mining techniques on the large set of anonymized training data to discover correlations between features in the data and disease diagnoses. For example, the AI ​​recommender system 124 may determine a correlation between a feature set including papilledema, hypertension, and high BMI and a diagnosis of obstructive sleep apnea. Based on this correlation, when patient 102 presents with a matching feature set, the AI ​​recommender system 124 may generate recommendations for sleep apnea screening. The AI ​​recommender system 124 and its training phase will refer to… Figure 2and Figure 3 To provide a more detailed description.

[0038] In the examples, the AI ​​recommender system 124 can generate recommendations based on one or more diseases and / or disease risks identified in patient 102. For example, recommendations may indicate a detected disease or disease risk, indicate that the patient needs additional screening, or indicate that the screening was normal (e.g., no disease was detected). In some examples, recommendations may also include an associated urgency level based on the severity of the disease or disease risk, such as follow-up screening within 6 to 12 months, follow-up screening within 3 months, requiring immediate attention, etc. In some examples, the AI ​​recommender system 124 can transmit recommendations to clinical device 114, where recommendations can be output to operator 104 via a user interface. Operator 104 can then take various actions to provide recommendations to patient 102, schedule follow-up screenings, and / or add recommendations to EMR data associated with patient 102. In some examples, the AI ​​recommender system 124 can transmit recommendations to EMR system 110. In such examples, the recommendations can later be accessed from EMR system 110 by other users caring for patient 102 (e.g., physicians, nurses, etc.).

[0039] In some examples, when a particular disease is suspected, different standards of practice may exist for recommendations based on country or geographic region. For example, recommendations for follow-up screening and / or treatment for a particular disease may differ in the United States from those in the United Kingdom. In some examples, differences in standards of practice may be based on the prevalence of the particular disease in a geographic location and / or guidelines of health authorities in that geographic region. In various implementations, the AI ​​recommender system 124 may determine recommendations based on standards of practice in a geographic location within environment 100. In some examples, the AI ​​recommender system 124 may also consider guidelines from patient 102's insurance policy (e.g., as indicated in EMR data) when determining recommendations.

[0040] As used herein, the terms “machine learning,” “ML,” and their equivalents in relation to the image analysis system 120 and the AI ​​recommender system 124 can refer to a computational model that can be optimized to accurately reproduce certain outputs based on certain inputs. In some examples, the ML model includes deep learning models such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, any combination thereof, or other types of NNs. The term “neural network (NN)” and its equivalents can refer to a model with multiple hidden layers, wherein the model receives an input (e.g., at least one vector, matrix, or tensor) and transforms that input by performing operations via the hidden layers. A single hidden layer can include multiple “neurons,” each of which can be disconnected from other neurons in that layer. A single neuron within a particular layer can be connected to multiple (e.g., all) neurons in the preceding layer based on the model architecture. In some examples, the NN can also include at least one fully connected layer that receives a feature map output from the hidden layers and transforms that feature map into the output of the NN. The output of the NN can be in any form based on the purpose of learning the network. For example, the output can be the name of a detected feature, the location of a detected feature, an indication of the presence of a detected feature, or any combination thereof.

[0041] As used herein, the term "CNN" and its equivalents and variations can refer to a neural network model that performs at least one convolution (or cross-correlation) operation on an input image and can generate an output image based on the convolved (or cross-correlated) input image. A CNN can include multiple layers that transform an input image (e.g., an ophthalmological image) into an output image through a convolution or cross-correlation model defined according to one or more parameters. The parameters of a given layer can correspond to one or more filters, which can be digital image filters that can be represented as images (e.g., 2D images). Filters in a layer can correspond to neurons in that layer. Layers in a CNN can convolve or cross-correlate their respective filters with the input image to generate an output image. In various examples, neurons in a layer of a CNN can be connected to a subset of neurons in the previous layer of the CNN, such that the neuron can receive input from the subset of neurons in the previous layer and can output at least a portion of the output image by performing operations (e.g., dot product, convolution, cross-correlation, etc.) on the input from the subset of neurons in the previous layer. A subset of neurons in the previous layer can be defined based on the neuron's "receptive field," which can also correspond to the neuron's filter size. Other types of neural network (NN) frameworks can also be used. For example, the image analysis system 120 may include one or more transformer-based models. For instance, transformer-based models can serve as the backbone of the NN in the image analysis system 120 and / or the AI ​​recommender system 124. For example, a transformer-based model may include an encoder component that generates embeddings by mapping the input to a high-dimensional embedding space, and these embeddings can be used as features, replacing features detected by the image analysis system 120 or other features besides those detected by the image analysis system 120. In some examples, the embeddings generated by the encoder component can be used as input features for other ML models used by the image analysis system 120 and / or the AI ​​recommender system 124.

[0042] It should be understood that, despite Figure 1 A single optical imaging device 106 is depicted, but in additional examples, environment 100 may include any number of local or remote optical imaging devices configured to operate independently and / or in combination to capture various types of retinal images. Furthermore, although... Figure 1 Optical imaging device 106, EMR system 110, clinical device 114, and remote computing device 118 are shown as separate entities, but in some embodiments, one or more of these entities may correspond to the same computing device.

[0043] As discussed in this article, Figure 1An example environment 100 is depicted, which includes components for capturing images of a patient's retina and, based on the retinal images and additional information from the patient's health records, is provided by a trained AI-based recommender system for screening the patient for diseases and / or disease risks.

[0044] Figure 2 An example system 200 is shown for training an AI recommender system 124 based on a large anonymized training dataset 202. The AI ​​recommender system 124 can be trained to discover correlations between patient data (e.g., retinal images and EMR data) and disease diagnoses or future disease risks, and / or to train a classifier for detecting one or more diseases or disease risks based on patient data. In some examples, the training dataset 202 may be stored on a different data server (e.g., cloud storage) than a remote computing device 118.

[0045] like Figure 2 As shown, the training dataset 202 may include retinal images 204 and EMR data 206 from M different individual patients, with each patient corresponding to a data instance m, where m = 1, ..., M. In the example, as referenced... Figure 1 The image analysis system 120 can process the retinal image 204 to extract eye features 208(1) to 208(M) and / or generate one or more standard images 210(1) to 210(M), each corresponding to a data instance (e.g., data from a specific patient). Figure 2 In the example shown, each eye feature in eye feature 208 includes K features (1, ..., K). In some examples, the K features may include multiple instances of the same ophthalmic feature (e.g., AVR ratio, optic disc edema, etc.) associated with different timestamps (indicated by the image analysis system 120 based on the date and / or time of day of the captured underlying retinal image 204), thereby providing historical data of eye feature 208 for each patient. Examples of eye feature 208 are described in U.S. Patent Application Serial No. 17 / 709,950, filed March 31, 2022, entitled "Automated disease identification based on ophthalmic images," which is incorporated herein by reference as described above.

[0046] Similarly, as referenced Figure 1The EMR data extractor 122 can process the EMR data 206 to extract EMR features 212(1) to 212(M), each corresponding to data instance m in M ​​data instances. For example, the EMR data extractor 122 can issue a query to the EMR system 110 (e.g., to a database storing EMR information) requesting anonymized data, and can respond by receiving the EMR data 206, along with the association with corresponding data of the same patient in the retinal image 204. In another example, the EMR data extractor 122 can be given read-only access to the EMR system 110 to extract the EMR data 206 without accessing the patient's identity information. Figure 2 In the example shown, each EMR feature in the EMR feature 212 comprises P features (1, ..., P). The EMR data extractor 122 may also extract one or more diagnoses 214(1) to 214(M), each corresponding to a data instance. For example, some data instances may contain multiple diagnoses in their corresponding EMR data, while others may not have any diagnoses in their EMR data. In some examples, the EMR data extractor 122 may input a diagnosis of “healthy and normal” when the corresponding EMR data does not contain any disease diagnoses. In some examples, each EMR feature and diagnosis 214 in the P EMR features may be associated with a timestamp indicating the basic EMR data collection date and / or time of day. For example, for the same data instance (e.g., m=1), “diagnosis (1)” of diagnosis 214(1) may indicate “normal” at a first time, while “diagnosis (2)” of diagnosis 214(1) may indicate “stroke” at a second time after the first time.

[0047] In some examples, EMR data extractor 122 can determine the quality level associated with each data instance of EMR data 206. In these examples, EMR data extractor 122 can assign a quality level to an EMR data instance based on factors such as the completeness of the record, the level of detail in the record, the regularity of record updates, and whether the record indicates a consistent physician and / or visit location (e.g., a pattern of regular checkups). For example, a first EMR data instance might be assigned a higher quality level than a second EMR data instance that indicates regular updates (e.g., annually or more frequently), contains detailed diagnoses, contains recurring test results, and / or indicates regular physician visits, while the second EMR data instance indicates fewer, less frequent physician visits, few or no test results, and / or primarily emergency room or emergency care visits.

[0048] Training dataset 202 can be accessed by training component 216 to train one or more AI / ML models 218 of AI recommender system 124. In some examples, training component 216 may be implemented on remote computing device 118 and access training data 202 via a network such as network 116. Alternatively, training component 216 may be implemented on a different computing device than remote computing device 118.

[0049] Training dataset 202 can be used to train a feature detector for features associated with a specific disease, as described in U.S. Patent Application Serial No. 17 / 709,950, filed March 31, 2022, entitled "Automated disease identification based on ophthalmic images," which is incorporated herein by reference as stated above. For example, U.S. Patent Application Serial No. 17 / 709,950 provides Tables 1 through 5, which correlate disease conditions with features associated with the patient's eyes and / or the patient's EMR data. In some examples, the AI ​​recommender system 124 can identify a set of diseases and train a detector to detect known features (e.g., established by medical research) associated with that set of diseases. However, other unknown correlations may exist between features 208, 212 and diagnosis 214, for example, correlations that have not yet been established by medical research. For example, certain diagnoses may be associated with features detected in images of the patient's retina, the patient's geographic region, and / or the patient's medical history. In addition, some correlations can be predictive; for example, subsets of features 208 and 212 can be associated with future disease diagnoses (e.g., one year later, three years later, five years later, etc.).

[0050] In the examples disclosed herein, training component 216 may implement data mining techniques to determine the correlation between diagnosis 214 and one or more features of eye feature 208 and / or EMR feature 212. As a non-limiting example, training component 216 may include frequent pattern mining (e.g., frequent itemset mining) and output candidate association rules, which may be one-dimensional or multi-dimensional, indicating that a set of features in training dataset 202 is associated with a specific diagnosis. Training component 216 may select candidate association rules that satisfy a minimum support threshold, where the minimum support threshold indicates the minimum number of data instances in training dataset 202 that support the association between that set of features and the specific diagnosis. Training component 216 may further analyze these candidate association rules using statistical methods to determine the correlation values ​​(e.g., Pearson correlation coefficient, Spearman coefficient, Cramer coefficient, Kendall coefficient, etc.) between the specific diagnosis and the set of features, and filter out candidate association rules whose respective correlation values ​​are less than the minimum correlation threshold. In the example, AI recommender system 124 may use association rules that satisfy the minimum correlation threshold to generate screening recommendations for the disease or disease risk indicated in the diagnosis. The AI ​​recommender system 124 can also determine a confidence score based on the correlation value of the corresponding association rule. As a simplified example, an association rule that meets the minimum correlation threshold can indicate that a feature set (“optic disc edema,” “decreased AVR,” “high BMI,” “age > 55”) is associated with a diagnosis of “obstructive sleep apnea.” In this example, if the data associated with patient 102 matches this feature set (“optic disc edema,” “decreased AVR,” “high BMI,” “age > 55”), the AI ​​recommender system 124 can suggest screening for “obstructive sleep apnea” for patient 102. Furthermore, if the correlation value of the association rule is 0.7, the AI ​​recommender system 124 can output a screening recommendation with a confidence score of 0.7. In this example, the AI ​​recommender system 124 can output a recommendation only if the confidence score associated with that recommendation is higher than the minimum threshold.

[0051] As another example, training component 216 can train a Bayesian belief network based on training data 202. Such a Bayesian belief network is characterized by a conditional probability table, allowing the calculation of the probability of a diagnosis given a set of features. For example, if the disease output probability calculated by the trained Bayesian belief network is higher than a threshold probability, the AI ​​recommender system 124 can suggest screening for that disease. In yet another example, training component 216 can use techniques such as sequential covering algorithms to determine rules (e.g., if-then) based on training data 202, creating a hierarchical decision tree that the AI ​​recommender system 124 can use to determine whether a disease condition has been reached (as leaf nodes) based on features 208, 212 corresponding to a specific patient, such as patient 102.

[0052] In some examples, training component 216 can perform temporal data mining on training dataset 202 to identify trends in features / images 208, 210, 212 associated with disease risk or future disease diagnosis. For example, training component 216 can represent the same feature (e.g., systolic blood pressure) as a time series using timestamps associated with features / images 208, 210, 212, and use data mining techniques to mine patterns in the time series data, thereby generating a predictive model indicating disease risk. In some examples, training component 216 can determine changes over time by comparing a standard image 210 generated from retinal images captured at different times. In some examples, training component 216 can generate derived features to indicate time series information; for example, "elevated blood pressure" can be added as a feature to EMR feature 212 before determining association rules. As an example, training component 216 can use time-series analysis to establish a correlation between an increasing AVR ratio over time and obstructive sleep apnea, adding "increasing AVR ratio" as an eye feature. This feature, combined with other features, can appear in association rules for detecting obstructive sleep apnea. In such an example, if the trend of the AVR ratio calculated from the patient's retinal images over time shows an increasing pattern, and combined with other features identified in the association rules, the AI ​​recommender system 124 can generate screening recommendations for obstructive sleep apnea.

[0053] In some examples, training component 216 can be used to train a classifier for identifying one or more diseases and / or disease risks based on eye features 208, EMR features 212, and / or standard images 210, as shown in references. Figure 3 The following is a more detailed description. In the example, AI / ML model 218 can store association rules (including temporal association rules) discovered as described above, as well as a classifier that can be implemented as a trained neural network. In some examples, training component 216 can be trained using a first portion of training dataset 202 and validated using a second portion (e.g., the remainder) of training dataset 202.

[0054] Various systemic diseases (e.g., fibromyalgia, Lyme disease, kidney disease, stroke risk, etc.) require multiple diagnostic tests, test results over extended time periods (e.g., months or years), and / or symptoms over extended time periods to arrive at a diagnosis. The AI ​​recommender system 124 can provide a faster pathway to diagnosis by identifying the correlation between features of a patient's retinal images and their health records (such as those obtained during routine health screenings at a physician's office) and specific diseases using a data-driven approach, and then proposing screening recommendations for those diseases. As discussed above, physicians performing routine health screenings (e.g., primary care physicians) may not be able to interpret retinal images and therefore may not be able to incorporate the features of a patient's retinal images into a diagnosis of disease or disease risk. Furthermore, the AI ​​recommender system 124 can identify disease risk based on temporal correlations identified in the training dataset 202, thereby recommending screening.

[0055] Table 1 below summarizes some examples of the association between feature sets and diseases.

[0056] Table 1

[0057]

[0058] Figure 3 An example system 300 for generating recommendations for a patient is shown, where the recommendations are based on the output of a classifier 302. In the example, the classifier 302 may include a trained machine learning (ML) model configured to identify the presence of a disease or disease risk when provided with features of a patient's retinal image and health records as input. As shown, system 300 includes an implementation of an AI / ML model 218 of an AI recommender system 124, which includes the classifier 302. In the examples of this disclosure, the AI ​​recommender system 124 may use reference... Figure 2 AI / ML model 218 described Figure 3 The AI / ML model 218, or a combination thereof, described in the document.

[0059] As referenced above Figure 1 As described, the image analysis system 120 can receive and / or identify one or more retinal images 108 (e.g., from computer memory). The retinal images 108 can depict at least one eye of a patient, such as patient 102. For example, the retinal images 108 can include at least one OCT image and / or slit-lamp image of the patient's eye, as well as the retina and / or fundus of the patient's eye. The image analysis system 120 can determine ophthalmic features (including structural and color features) of the retinal images 108 and standardized retinal images, such as referenced... Figure 1As described above. Exemplary ophthalmic features are described in U.S. Patent Application Serial No. 17 / 709,950, filed March 31, 2022, entitled "Automated disease identification based on ophthalmic images," which is incorporated herein by reference. Furthermore, EMR data 112 can be processed by EMR data extractor 122 to extract the patient's EMR feature vector, also as described in the reference. Figure 1 As described.

[0060] In the example, the AI ​​recommendation system 124 can provide at least one subset of features from ophthalmic features determined by the image analysis system 120 and an EMR feature vector from the EMR data extractor component 122 as input to the classifier 302. In some examples, the subset can be based on, as in reference... Figure 2 The correlations found between features and specific diseases are discussed. Alternatively or additionally, the AI ​​recommendation system 124 may provide a standardized retinal image from the image analysis system 120 as input to the classifier 302.

[0061] In some examples, classifier 302 may include a set of ML models (e.g., each ML model is trained to output a confidence level associated with a single disease). In other examples, classifier 302 may include multi-class ML models trained to output indications of one or more diseases from a set of diseases. As examples, classifier 302 may include CNNs, transformer-based models, RNNs, etc. In some examples, classifier 302 may be based on a transformer architecture, and a portion of the input (e.g., the location of ophthalmic features), after tokenization, may include positional encodings indicating the relative location of the input tokens (e.g., relative to a normalized retinal image). In the examples, the diseases evaluated by classifier 302 may correspond to the diseases listed in Table 1.

[0062] In the example, a set of ML models or multiple ML models in classifier 302 can be referenced by training component 216 during the training phase. Figure 2The training dataset 202 described is used for training. In the example, the training component can divide the M data instances of the training dataset 202 into a first part (e.g., 80% of the M data instances) for training classifier 302, and a second part (e.g., the remaining 20% ​​of the M data instances) for validating classifier 302. For example, the second part of the training dataset 202 can be used as test data to evaluate the performance of the trained classifier 302, thereby preventing overfitting and local minima problems that may sometimes be encountered after training an ML model.

[0063] As discussed above, in some examples, training component 216 may train a set of ML models, each trained to output disease indicators 304(1) to 304(N) (e.g., corresponding to the first disease, the second disease, ..., and the Nth disease), where the disease indicators may include a confidence level associated with the presence of the disease. Training component 216 may use a first subset of instances from the training dataset 202 indicating diagnoses of the nth disease to train each such individual disease classifier 302, for example, a classifier for the nth disease. In some examples, training component 216 may also use a second subset of instances from the training dataset 202 indicating diagnoses other than the nth disease as negative examples.

[0064] In one example, the AI ​​recommender system 124 can input a subset of features and / or normalized retinal images into each of the group of ML models and determine a confidence score or probability for each disease 1, ..., N as the output of each model in the group of ML models. In other examples, the AI ​​recommender system 124 can input a subset of features into a multi-class classifier of classifier 302 and receive a confidence score or probability corresponding to disease 1, ..., N as the output. In any example, the AI ​​recommender system 124 can adjust the confidence score received by classifier 302 based on other factors. For example, the AI ​​recommender system 124 can adjust the confidence score based on the quality level of the EMR data 112; for example, the confidence score can be adjusted to low based on a low quality level. In another example, the AI ​​recommender system 124 can adjust the confidence score based on the frequency or probability of a diagnosis; for example, the confidence score associated with a rare disease can be adjusted to be lower. In the example, the AI ​​recommender system 124 can provide the evaluator 306 with confidence scores corresponding to each disease 1, ..., N.

[0065] In the example, evaluator 306 can compare confidence scores to a minimum threshold and provide recommendation 308 for screening the disease for which confidence scores associated with the disease (e.g., disease indicators 304(1) to 304(N)) are higher than the minimum threshold. In some examples, evaluator 306 may also consider healthcare policies specific to a particular geographic location, such as reference, when determining recommendation 308. Figure 1 As described. Evaluator 306 can also determine the severity associated with a disease based on a confidence score associated with the disease (e.g., disease indicators 304(1) to 304(N)) being higher than a second threshold (which is higher than a minimum threshold). In some examples, evaluator 306 may use separate thresholds and / or ranges for different diseases or disease types and / or different patient categories (e.g., age-based categories). In some examples, evaluator 306 may generate recommendations 308 to include a textual summary of the disease, the severity associated with the disease, and / or a description of features used to determine the presence of the disease, for example, by using generative AI techniques such as large language models (LLM).

[0066] In some examples, system 300 may include receiving the results 310 of follow-up screening based on recommendation 308. In an example, if result 310 indicates the presence of a disease indicated for screening in recommendation 308, training component 216 may add result 310 as ground truth to training dataset 202. Alternatively, if result 310 indicates the absence of a disease in recommendation 308, training component 216 may update training dataset 202 to add the outputs of image analysis system 120 and EMR data extractor 122 as new data instances, indicating the absence of the disease screened in result 310. Training component 216 may be based on the updated training dataset, as referenced... Figure 2 As described, the classifier 302 is periodically retrained and / or association rules are mined.

[0067] As discussed in this article, Figure 2 and Figure 3 A system for training and using an AI recommender to provide recommendations for screening patients for diseases or disease risks based on their retinal images and EMR data is illustrated. Other examples of training and using AI recommenders tailored to detect specific disease conditions and abnormalities are also envisioned. For example, if available medical research shows a correlation between specific features of retinal images and / or EMR data and specific disease outcomes, a classifier such as classifier 302 can be trained to output a positive or negative indication of a specific disease outcome when said specific features are provided as input.

[0068] Figure 4 and Figure 5 A flowchart illustrating an example method described herein for generating recommendations for disease screening is provided. Figure 4 and Figure 5 The methods described are illustrated as a set of boxes in a logic flowchart, representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, these boxes represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by a processor, perform the enumerated operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a specific function or implement a specific abstract data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described boxes can be combined in any order and / or in parallel to implement the desired operation. Figure 4 and Figure 5 The method shown in the figure. In some implementations, it can be completely omitted. Figure 4 and Figure 5 The method shown in the diagram has one or more boxes.

[0069] The following text is about Figure 4 and Figure 5 The operations described in the methods shown herein can be performed by any of the devices / systems 114, 118, 200, and 300 described herein and / or their various components. In particular, regarding... Figure 4 and Figure 5 Any operations described in the methods shown can be performed by the AI ​​recommender system 124 or the training component 216. Unless otherwise stated, and for ease of description, reference will be made below. Figure 1 Environment 100 Description shown Figure 4 The method is shown below, and will be referenced in the following text. Figure 2 and Figure 3 The system descriptions 200 and 300 shown are as follows. Figure 5 The method shown.

[0070] refer to Figure 4 In the example process 400 shown, at operation 402, the AI ​​recommender system 124 can receive at least one ophthalmological image of a patient, such as patient 102. (See above reference.) Figure 1 The ophthalmic images can be captured by the optical imaging device 106 during a patient's medical visit to a doctor's office (e.g., a routine health check). The ophthalmic images may include one or more images 108 of the retina and / or fundus of at least one eye. As a non-limiting example, the ophthalmic images may include OCT or OCTA images, fluorescence angiography images, CFP images, etc.

[0071] At operation 404, the AI ​​recommender system 124 may include receiving the patient's electronic health record. The electronic health record (e.g., EMR data 112) can be accessed from an EMR system storing the patient's medical records. The patient's electronic health record may include data indicating the patient's previous or current medical diagnoses, as well as a history of diagnostic tests, medications, or treatments received by the patient. Furthermore, the electronic health record may indicate the patient's demographic information (e.g., age, sex, race, etc.), vital signs (e.g., blood pressure, blood oxygen levels, heart rate, etc.), body mass index (BMI), lifestyle information (e.g., smoking or drug use, physical activity level, diet, alcohol use, etc.), and similar information measured during current and previous medical visits.

[0072] At operation 406, the AI ​​recommender system 124 may include: determining one or more potential diseases or disease risks of a patient by inputting features of ophthalmic images and electronic health records into one or more trained ML models. In the example, the features of the ophthalmic images and electronic health records may be determined by image analysis system 120 and EMR data extractor 122, as referenced. Figure 1 Described, and includes, standardized retinal images. (See reference...) Figure 2 and Figure 3 As described, the ML models can be trained on a large, anonymized training dataset containing retinal images of patients, EMR data, and corresponding disease diagnoses over extended time periods. The one or more ML models (e.g., CNNs, RNNs, transformer-based models, etc.) can be trained to output confidence scores associated with a disease or disease risk in a set of diseases. In some examples, each of the one or more ML models can be trained to detect a single disease. In some examples, the one or more ML models can include decision trees, expert systems, or Bayesian belief networks trained to compute disease probabilities (corresponding to confidence scores) given features as input. In examples, the one or more ML models can output confidence scores associated with the detection of one or more potential diseases.

[0073] At operation 410, the AI ​​recommender system 124 can compare the confidence score obtained at operation 406 with a minimum threshold. In the example, the minimum threshold can vary based on the disease type, patient characteristics (e.g., age), and / or the health policies of the patient's geographic region. For example, if the confidence score is higher than the minimum threshold (operation 410 – yes), the AI ​​recommender system 124 can generate a recommendation at operation 412 indicating the need for follow-up screening for the disease. In the example, the recommendation can be provided to the patient and / or the healthcare provider caring for the patient and can be added to the patient's electronic health record.

[0074] Alternatively, if the confidence score is not higher than a minimum threshold (operation 410 – No), the AI ​​recommender system 124 may generate an indication of normal health status at operation 414. In some examples, the indication of normal health status may not lead to any action by the AI ​​recommender system 124 and / or the healthcare provider receiving the indication.

[0075] Figure 5 An example process 500 is illustrated for training an AI recommender system, such as AI recommender system 124, to identify one or more diseases or disease risks based on a patient's ophthalmological images and electronic health records. Process 500 can be executed by a computing device including at least one processor and memory. In some examples, process 500 can be executed by a different computing device than the computing device 118 implementing AI recommender system 124, as referenced above. Figure 1 As described.

[0076] At operation 502, training component 216 can receive a large number of patients' electronic health records. As described, electronic health records (e.g., from EMR system 110) may include data indicating a patient's previous or current medical diagnoses, as well as a history of diagnostic tests, medications, or treatments received by the patient. Furthermore, electronic health records may indicate a patient's demographic information, vital signs, lifestyle information, etc., such as measurements taken over extended periods and / or during multiple medical visits. Electronic health records may also include physician records from medical visits, results of diagnostic tests performed on the patient, and treatment outcomes.

[0077] At operation 504, training component 216 can receive corresponding ophthalmic images of the patient. As an example, the images may include one or more of the following: OCT images, slit-lamp images, fundus images, or retinal images captured by one or more medical imaging devices configured to acquire ophthalmic images. In some examples, the ophthalmic images may also be stored in an EMR system in association with the patient's electronic health record and may include images captured over a prolonged period. In examples, the images may include images showing the patient's disease condition as well as images showing a normal (e.g., disease-free) condition. In examples, the individual patient's electronic health record and ophthalmic images may be identified as belonging to the same individual.

[0078] At operation 506, training component 216 includes creating a training dataset for one or more disease outcomes identified in health records. (See reference...) Figure 2As described, the training dataset may include features generated from ophthalmic images and electronic health records, such as those determined by the image analysis system 120 and the EMR data extractor 122. Each data instance in the training dataset may correspond to an individual patient and include features of that individual, including a standardized retinal image, as well as a diagnosis indicating one or more disease outcomes (e.g., obtained from the electronic health record corresponding to that individual). In some examples, the training dataset may also include manually provided entries by experts associated with one or more data instances, indicating features (e.g., in ophthalmic images) or diagnoses.

[0079] At operation 508, training component 216 may include training one or more ML models to identify one or more disease outcomes using the training dataset created at operation 506. For example, each training data instance may include a set of features as input and a disease outcome (e.g., as indicated in a corresponding electronic health record) as the target output. In some examples, a training data instance for a specific disease may include a subset of the set of features as input, wherein the subset is based on, as referenced... Figure 2 The described association rules or correlations are established between the subset and the specific disease. In some examples, the one or more ML models may include one or more NNs (e.g., CNN, RNN, graph neural network, etc.), and the training component 216 may be trained based on optimizing the model's parameters using techniques such as backpropagation. In some examples, the ML model may be a decision tree, expert system, or Bayesian belief network, which can be trained by computing the conditional probabilities of disease outcomes given a feature set.

[0080] As discussed above, the ML model trained by training component 216 can be used by AI recommender system 124 to generate recommendations for patients to screen for diseases. In the example, process 500 can be repeated periodically, for example, in response to receiving additional data and / or the passage of time exceeding a time threshold, to keep the ML model updated based on the current data.

[0081] Figure 6 At least one example device 600 is shown, configured to implement and / or perform some or all of the functions discussed herein. Furthermore, device 600 may be implemented as one or more server computers, network elements on dedicated hardware, software instances running on dedicated hardware, or virtualization functions instantiated on a suitable platform such as cloud infrastructure. In the context of this disclosure, it should be understood that device 600 may be implemented as a single device or as multiple devices in which components and data are distributed among them.

[0082] As shown in the figure, device 600, which may correspond to computing device 118, may include memory 602. Memory 602 may be used to store any number of functional components executable by processor 604. In this example, these functional components include instructions or programs executable by processor 604, which, when executed, specifically configure one or more processors 604 to perform actions associated with providing screening recommendations for one or more diseases. For example, memory 602 may store one or more functional components, such as... Figure 1 The illustrated image analysis system 120, EMR data extractor 122, and AI recommender system 124 are shown. Memory 602 may also include files and databases used by one or more functional components. Memory 602 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of information storage technology, such as computer-readable instructions, data structures, program modules, or other data.

[0083] As described herein, processor 604 may be a single processing unit or multiple processing units, and may include single or multiple processing cores, including a central processing unit (CPU), a graphics processing unit (GPU), or both a CPU and a GPU, or other processing units known in the art. For example, processor 604 may be one or more hardware processors and / or logic circuits of any suitable type, specifically programmed or configured to perform the algorithms and processes described herein. Processor 604 may be configured to fetch and execute computer-readable instructions stored in memory 602, which may program processor 604 to perform the functions described herein.

[0084] The device 600 may also include additional data storage devices (removable and / or non-removable), such as, for example, a hard disk, optical disk, or magnetic tape. Such additional storage devices... Figure 6The diagram shows removable storage device 606 and non-removable storage device 608. Tangible computer-readable media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Memory 602, removable storage device 606, and non-removable storage device 608 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD), content-addressable memory (CAM) or other optical storage devices, magnetic tape, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by device 600. Any such tangible computer-readable medium may be part of device 600.

[0085] Device 600 may also include input devices 610, such as a keypad, cursor control, touch-sensitive display, voice input device, etc., and output devices 612, such as a display, speaker, printer, etc. In some examples, input device 610 includes medical imaging equipment, such as those mentioned above. Figure 1 The optical imaging device 106 is described. In a particular implementation, a user can provide input to the device 600 via a user interface associated with the input device 610 and / or the output device 612.

[0086] like Figure 6 As shown, device 600 may also include one or more wired or wireless transceivers 614. For example, transceiver 614 may include a network interface card (NIC), network adapter, LAN adapter, or a physical, virtual, or logical address for connecting to various base stations or networks envisioned herein (e.g., network 116), or for connecting to various user equipment and servers. To increase throughput when exchanging wireless data, transceiver 614 may utilize multiple-input / multiple-output (MIMO) technology. Transceiver 614 may include any type of wireless transceiver capable of wireless, radio frequency (RF) communication. Transceiver 614 may also include other wireless modems, such as modems for Wi-Fi, WiMAX, Bluetooth, or infrared communication.

[0087] Based at least on the description herein, it should be understood that the AI ​​recommender system, apparatus, and method of this disclosure can be used to assist in the identification of one or more vision disorders or disease risks and to recommend screenings for the identified disorders to patients. The AI ​​recommender system can be trained on a large training dataset of anonymized patient data and provides recommendations to patients based on retinal images and electronic medical records collected during medical visits to a physician's office. The system described herein can also implement data mining techniques to discover associations between feature sets and disease outcomes based on data in the training dataset. This recommendation can allow for screening of patients for potential diseases, enabling early diagnosis and treatment before the disease becomes more severe.

[0088] The above description is merely an illustration of the principles of this disclosure, and those skilled in the art can make various modifications without departing from the scope of this disclosure. The examples described above are presented for illustrative purposes and not for limitation. This disclosure may also take many forms besides those expressly described herein. Therefore, it should be emphasized that this disclosure is not limited to the explicitly disclosed methods, systems, and apparatus, but is intended to include variations and modifications to the disclosed content, all of which fall within the spirit and scope of the appended claims.

[0089] As another example, variations can be made to the apparatus or process constraints (e.g., dimensions, configuration, components, sequence of process steps, etc.) to further optimize the structures, devices, and methods shown and described herein. In any case, the structures and devices described herein, and the associated methods, have numerous applications. Therefore, the disclosed subject matter should not be limited to any single example described herein, but should be interpreted in breadth and scope according to the appended claims.

Claims

1. A system comprising: Memory; processor; as well as Computer-executable instructions, which are stored in the memory and executed by the processor to perform operations including: Receive images of the patient's retina; Receive patient data corresponding to the patient from the patient's electronic medical record (EMR); Determine the features in the image; The confidence level associated with the first disease is determined by feeding the features and at least a portion of the patient data into a machine learning (ML) model. Based on the confidence level being higher than a threshold, a recommendation is made to screen the patient for the first disease. as well as The EMR of the patient is provided with an output indicating the recommendations.

2. The system according to claim 1, wherein, The ML model is trained on a training dataset to identify confidence levels associated with the first disease based on the images and the patient data as input.

3. The system according to claim 1, wherein, The training dataset includes: Anonymized patient data and corresponding retinal images associated with multiple patients, and An indicator of normal health or one or more diseases. The anonymized patient data and the corresponding retinal images are extracted from an electronic medical record (EMR) system.

4. The system according to claim 1, wherein the operation further includes: Receive follow-up information indicating whether the patient has been diagnosed with the first disease; The training dataset is expanded to include data points containing at least a portion of the follow-up information, the features, and the patient data; as well as The ML model is updated by retraining using an expanded training dataset.

5. The system according to claim 1, wherein, The ML model is based, at least in part, on determining the correlation between the first disease and the feature or the patient data in the training data.

6. The system according to claim 1, wherein, The first disease is one of the following: obstructive sleep apnea (OSA), anemia, heart disease, kidney disease, multiple sclerosis (MS), or Alzheimer's disease.

7. The system according to claim 1, wherein: The EMR data includes at least one of the following: the patient's age, the patient's blood pressure measurement, or one or more medical test results associated with the patient, and The features include at least one of the following: the brightness level of the optic disc of the retina, the diameter of the blood vessels in the retina, edema of the optic disc, or arteriovenous ratio (AVR).

8. A method, the method comprising: The processor receives images of the patient's retina; The processor receives patient data corresponding to the patient from the patient's electronic medical record (EMR); The processor determines the features in the image; The processor determines the confidence level associated with the first disease by feeding the features and at least a portion of the patient data into a machine learning (ML) model. The processor determines a recommendation to screen the patient based on the confidence level being higher than a threshold. as well as The processor provides an output indicating the suggestion to the output device.

9. The method according to claim 8, wherein, The feature includes at least one of the following: The brightness level of the optic disc of the retina. The diameter of the blood vessels in the retina, The topological structure of the blood vessels in the retina. The edema of the optic disc, or Arteriovenous ratio (AVR).

10. The method according to claim 8, wherein, The patient data includes at least one of the following: The patient's age The patient's gender, The patient's race, The patient's smoking status. The patient's blood pressure measurement, or One or more medical test results associated with the patient.

11. The method according to claim 8, further comprising: The processor receives follow-up information indicating whether the patient has been diagnosed with the first disease; The processor expands the training dataset to include data points containing at least a portion of the follow-up information, the features, and the patient data; as well as The processor updates the ML model by retraining it using an expanded training dataset.

12. The method according to claim 8, wherein, The first disease includes one of the following: Obstructive sleep apnea (OSA). anemia, heart disease, Kidney disease, Multiple sclerosis (MS), or Alzheimer's disease.

13. The method according to claim 8, wherein, The ML model is trained on a training dataset to identify the confidence level associated with the first disease based on the image and the patient data as input. The training dataset includes: anonymized patient data and corresponding retinal images associated with multiple patients, as well as indicators of normal health or one or more diseases associated with each patient.

14. The method according to claim 8, wherein, The ML model includes an expert system that instructs rules for associating the features and the patient data with the probability of occurrence of the first disease.

15. A non-transitory computer-readable storage medium storing processor-executable instructions, which, when executed, cause one or more processors to perform the method according to any one of claims 8 to 14.

Images