Predicting the rate of hypoglycemia by a machine learning system

Abstract

To provide a method, computer-readable medium, and system for predicting rates of hypoglycemia in patients using a machine learning system.SOLUTION: A computing device-implemented method of generating a trained machine learning model using patient data is provided, the method comprising receiving data representing a medical record of a patient diagnosed with diabetes mellitus, determining an expected rate of hypoglycemic events using a machine learning system trained using data representing the medical records of a plurality of patients and corresponding rates of hypoglycemic events for respective patients, and generating an expected rate of the patients.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] Claims of Priority This application claims the benefit of U.S. Provisional Patent Application No. 62 / 689,005, filed on Jun. 22, 2018, the entire content of which is incorporated herein by reference.

Background Art

[0002] Machine learning is a subset of artificial intelligence in the field of computer science, and often uses statistical techniques to give a computer the ability to "learn" from data (i.e., gradually improve performance on a specific task) without being explicitly programmed.

Summary of the Invention

Means for Solving the Problems

[0003] Generally, one aspect of the invention described herein is embodied as a method that includes an operation of receiving data indicating medical records of patients diagnosed with type 1 diabetes. This method includes an operation of using a machine learning system to determine a prediction rate of hypoglycemic events, where the machine is trained using data indicating medical records of a plurality of patients and corresponding rates of hypoglycemic events for each patient. This method also includes an operation of generating a prediction rate for a patient.

[0004] The above and other embodiments may each optionally include one or more of the following features, either alone or in combination. Each of a plurality of patients may use the same type of basal insulin. The method can include the act of using a second machine learning system to determine a second prediction rate of hypoglycemic events, the second machine being trained using medical records of a second plurality of patients and data indicating the corresponding rate of hypoglycemic events for each of the second patients, each of the second plurality of patients using a second type of basal insulin that is different from the first type of basal insulin, and the method may further include the act of comparing the first prediction rate to the second prediction rate. The method can include the act of recommending basal insulin for a patient based on the comparison. The method can include the acts of determining a plurality of prediction rates of hypoglycemic events for a second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system, identifying one or more covariates in the data correlated with the prediction rates of hypoglycemic events based on the data and the plurality of prediction rates of hypoglycemic events, and generating a report identifying the one or more covariates and the corresponding prediction rates of hypoglycemic events. The method can include the act of determining a plurality of prediction rates of hypoglycemic events for a second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system, each of the second plurality of patients having the same covariates, and the method may further include the act of generating a report identifying the covariates and the corresponding prediction rates of hypoglycemic events.

[0005] The present disclosure also provides a computer-readable storage medium connected to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations by implementing the methods provided herein.

[0006] The present disclosure further provides a system for implementing the methods provided herein. The system includes one or more processors and a computer-readable storage medium connected to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations by implementing the methods provided herein.

[0007] It should be understood that embodiments in accordance with the present disclosure may include any combination of the aspects and functions described herein. That is, embodiments in accordance with the present disclosure are not limited to combinations of aspects and are not limited to the functions specifically described herein, but also include any other suitable combinations of the provided aspects and functions.

[0008] Details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and briefly described in the following detailed description of how to implement the invention. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description of how to implement the invention, the drawings, and the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

[0010] Like reference numerals and designations in the various drawings refer to like elements.

[0011] Type 2 diabetes is the seventh leading cause of death and a major cause of morbidity in the United States (U.S.). Approximately 29.1 million people in the U.S. population are affected by type 2 diabetes, and 1.4 million are newly diagnosed each year. The number of affected patients is expected to increase to over 54.9 million by 2030. In 2012, the total cost in the U.S. for diagnosed diabetes was $245 billion ($176 billion in direct medical costs and $69 billion in lost productivity).

[0012] Given the significant and increasing burden of the disease, diabetes complications are becoming increasingly important for effective prevention and management. Hypoglycemia is a frequent and sometimes life-threatening adverse effect of insulin and oral antidiabetic drugs (OADs) in patients with diabetes. In addition to the immediate risk posed by hypoglycemia, recurrent episodes can lead to anxiety about future episodes and have been shown to constitute both patient-driven and physician-driven barriers to optimal glycemic control. The resulting increase in hemoglobin A1c (HbA1c) levels has been linked to an increased risk of microvascular (and sometimes macrovascular) complications.

[0013] Patients with type 1 diabetes (T1DM) are estimated to experience, on average, two mild hypoglycemic events per week and one severe event per year. However, event rates from randomized clinical trials for type 1 patients, expressed as per patient per year, range from 0.15 for severe events to 88.3 for non-severe events. Event rates for patients with type 2 diabetes (T2DM) vary widely between studies, ranging from 0.05 to 26.6 events per patient per year for severe and non-severe events, respectively. However, studies suggest that these studies significantly underestimate the true event rate of hypoglycemia in the real world, particularly for severe events.

[0014] The average cost for hypoglycemic events is also heterogeneous across studies and difficult to pool due to differences in the definition of hypoglycemia and cost estimation methods. Current estimates of the average cost for hypoglycemic events in inpatients range from $2,205 to $17,564.1 The average outpatient cost per hypoglycemic event ranges from $148 to $501.

[0015] Different basal insulins show different rates of hypoglycemia. For example, numerous studies have shown the superiority of insulin glargine 100 units / mL (Lantus) compared to protamine insulin with respect to severe hypoglycemic events. Some basal insulins, such as insulin glargine 300 units / mL (Toujeo), exhibit a flatter, more prolonged pharmacokinetic and pharmacodynamic profile compared to others due to sustained glycemic control over 24 hours.

[0016] Therefore, the burden of hypoglycemia in the United States is substantial, and significant benefits can be obtained by identifying the appropriate type of basal insulin for each patient. Estimates of the resulting cost savings for payers from reducing the medical costs of these patients can help guide payers' formulary decisions and drug price-setting negotiations.

[0017] FIG. 1 shows an environment 100 in which a machine learning model is trained to predict the probability of hypoglycemic events. The systems described herein use electronic medical record data (EMR) 102 to train a machine learning system (EMR includes, but is not limited to, hospital records and physician records and is obtained from a number of different information sources). In some embodiments, the EMR can include information about demographic and socioeconomic categories, coded diagnoses and procedures, prescribed and administered medications, laboratory results, and clinical management data. In some embodiments, the EMR is processed by a processor 104. For example, the EMR can include both structured and unstructured data. Structured data can include information such as the date of visit, patient name, etc. Unstructured data can include free-form text added by a physician (such as a doctor's notes, visit summary, etc.). The EMR processor can use techniques such as natural language processing to extract facts from the unstructured data of the EMR and convert the unstructured data into structured EMR data 106.

[0018] In some embodiments, the EMR processor 104 can filter a portion of the medical records. For example, the EMR processor 104 can select only the medical records of patients who share values for certain variables (referred to as covariates). Examples of covariates can include, for example, gender, geographic region, rate, age group, insurance company, number of years since diagnosis, HbA1c range, body mass index, blood pressure range, diabetes complications, alcohol and / or drug use, and any other physiological or demographic characteristics. The EMR processor 104 can be, for example, one or more computer systems as described below.

[0019] In some embodiments, the same manually created covariates are used for descriptive analysis, hypoglycemia rate prediction modeling, and cost estimate analysis. The predetermined covariates may be based on an expert clinical review of the literature around hypoglycemic events and cost predictors. The predetermined covariates are defined using ICD-9 and ICD-10 diagnostic codes, laboratory values, and drug names and / or National Drug Code (NDC) codes. An EHR dataset (not insurance claims) is used to identify covariates unless the covariate is cost-related, in which case insurance claims data is used.

[0020] A default lookback period of one year prior to treatment initiation is used, although this period can also vary depending on how long the covariates are assumed to persist. For example, the lookback period for cancer can be five years. The reason is that if a patient was diagnosed with cancer more than five years ago and there has been no subsequent diagnosis, it is unlikely that the patient has active cancer at the index date. For covariates such as gender and race that do not affect the likelihood of a patient being captured in the dataset based on the length of their medical history, the lookback period can be eight years or limited only by the amount of data available. Even for irreversible and chronic conditions, the maximum lookback period can be eight years or limited only by the amount of data available.

[0021] An example of the categories of covariates used in both the hypoglycemia rate model and the cost estimate analysis, and their lookback periods, can be as follows: 1. Demographics a. The lookback period for these covariates is eight years. 2. Socioeconomics b. The lookback period for these covariates is eight years. 3. Comorbidities c. The lookback period for these covariates ranges from one year (for reversible / acute conditions) to eight years (for irreversible / chronic conditions). d. The Charlson Comorbidity Index (CCI) score as a separate covariate within the comorbidity category. The CCI is a measure of the patient's comorbid condition, including diabetes complications, associated with the index period (including the diabetes complication category). 4. Diabetes Complications e. The lookback period for these covariates is 8 years. 5. Diabetes Disease State a. The lookback period for these covariates is 1 year for previous hypoglycemic events and 8 years for known-duration diabetes in the dataset. 6. Medication Use a. The lookback period for these covariates is 1 year.

[0022] Additional covariates are included as part of only the cost estimation analysis (not the hypoglycemia rate prediction) because these covariates are predicted to be drivers of hypoglycemia-related costs (not the hypoglycemia rate): 1. Physician Specialty: f. The lookback period is all available data for the "Physician Specialty" covariate and 2 years for "Physician's Most Commonly Prescribed Insulin". 2. Previous Mean Hypoglycemic Event Cost g. The lookback period for this covariate is 1 year.

[0023] Another set of covariates is determined. This set of covariates is not pre-specified but includes all comorbidities, procedures, and prescriptions that existed 1 year prior to the patient's index date (collectively referred to as "markers" below) and is included in the predictive modeling.

[0024] In some embodiments, the following technique for teacherless covariate creation is used. 1. The distance between markers is defined: a. Each marker is associated with a vector that includes the set of patients for whom the marker is true or false. b. In this case, the distance between markers is the Jaccard distance between the vectors. 2. Next, clusters are generated based on these distances: a. Hierarchical clustering based on the distance matrix was performed to obtain a cluster hierarchy. The average distance between clusters was used to hierarchically link the clusters. 3. A level of the hierarchy for extracting clusters is defined: a. The "dissimilarity" method was used to determine at which level of the hierarchy to extract clusters. "Dissimilarity" refers to the dissimilarity of the average distance in the linked clusters: a large value suggests that the clusters should not be linked. 4. The clusters "indicated" by many patient treatments are selected: a. Out of the 1500 clusters formed, 100 indicated by most patient treatments were selected. b. A patient treatment was said to "indicate" a cluster if, during the one-year period prior to the index date, the patient had any of the diagnoses, procedures, or prescriptions (e.g., markers) that constituted the cluster. 5. Medical rationalization of the clusters: a. Clinical experts then examined the obtained clusters with respect to medical logic to validate the parameters used for teacherless cluster generation.

[0025] In some embodiments, the EMR records are filtered, for example by filter 116, after the structured medical records 106 are generated but before the training data 108 is generated.

[0026] The structured EMR data 106 is used to generate training records 108 (collectively, the training set). These training records may represent the available data in the structured EMR data. For example, in some embodiments, a portion of the structured EMR records is used to train a machine learning system, and the remaining records are used to validate the trained machine learning system. In some embodiments, the training records are created at the level of "patient treatment" where the patient is on basal insulin therapy. Thus, multiple training records are created for each individual patient. The unit of analysis is defined as "patient treatment", which is the period during which the patient is observed for basal insulin therapy in the dataset (the period between the treatment index and the end of treatment observation). Hypoglycemic events are the target endpoints only within this patient treatment period.

[0027] The treatment index date is defined as either the exact first start of any basal insulin prescription; or a change in the prescription from one basal insulin to another. Baseline basal insulins include: Gla-300, Gla-100, IDet, IDeg, and NPH. The index basal insulins under study were: Gla-300, Gla-100, IDet, and IDeg.

[0028] For rate calculations, the "period" is interpreted as the period from the aforementioned patient treatment period minus the period of all inpatient stays during this period.

[0029] In some embodiments, the index date is the date of the first prescription of BI or the date of a change in the prescription from one basal insulin to another. The end of treatment is defined as the end of the follow-up period in the dataset, a change in the prescription from the index basal insulin to another BI, or one year after the treatment index date (whichever is earliest).

[0030] In one embodiment, patient treatment is excluded from the training data if the patient treatment meets any of the following criteria: 1. Treatment using multiple types of basal insulin: That is, patient treatment that starts within one week (before or after) of the start of another treatment for the same patient. 2. Patient treatment with any inactive period longer than 270 days out of the 365 days before the index date (inactivity is defined as the absence of timestamped data in the relevant tables in the dataset). 3. Patient treatment with a treatment period of less than one day.

[0031] Separately, inpatient stays are excluded from the patient treatment period because they are often switched to standard basal insulin according to the hospital provider's formulary at the time of admission. Therefore, hypoglycemic events during inpatient stays are not considered to be due to the index basal insulin. For this reason, hypoglycemic events during inpatient stays are not considered to be due to the index basal insulin.

[0032] Training records may include information about hypoglycemic events. In some embodiments, hypoglycemic events are counted within the patient treatment period. The period for determining the hypoglycemia rate was interpreted as the period obtained by subtracting the period of all inpatient stays during this period from the patient treatment period. Hypoglycemic events can be an expected output of the training set. For example, the training set can train a machine learning system to determine the expected number of hypoglycemic events within a certain fixed period (e.g., one month, six months, one year, five years, etc.). Alternatively, the training set can train a machine learning system to determine the expected number of hypoglycemic events within a certain period (e.g., one month, six months, one year, five years, etc.).

[0033] The hypoglycemic event rate can include both severe and non-severe events. Figure 2 is a flowchart of an example of a process for classifying hypoglycemic events as severe or non-severe. The definition of "severe" hypoglycemia can include, for example, ICD-9 / 10 codes that are inherently severe and the administration of intramuscular glucagon. Additionally, natural language processing of the EMR is used to identify hypoglycemia. With regard to severity, any hypoglycemic event that was not severe was defined as "non-severe".

[0034] In some embodiments, an event is defined as hypoglycemia if any of the following criteria are met: 1. ICD-9 and 10 hypoglycemia diagnostic codes 2. Laboratory plasma glucose level ≤ 70 mg / dL 3. Administration of intramuscular glucagon 4. NLP output

[0035] In some embodiments, an NLP-recognized hypoglycemia event is defined as any mention of hypoglycemia, except for those with an indication of negative sentiment or historical event. For example, a "mention" of hypoglycemia may be any event regarding the ordinary expression "*hypoglycemia*", unless the term is exactly one of "awareness of hypoglycemia", "unawareness of hypoglycemia", or "neonatal hypoglycemia". "Negative sentiment" may be any indication where the mention of hypoglycemia is negative, indicating, for example, that hypoglycemia did not occur. A historical event is a record indicating that the mention is about a past event (e.g., "The patient has a history of hypoglycemia"). In some embodiments, a list of negative sentiment and historical keywords is used to exclude the accompanying mention of hypoglycemia.

[0036] In some embodiments, the maximum number of one hypoglycemia event is counted per calendar day. For example, if one hypoglycemia event is recorded at multiple locations of treatment or by multiple definition criteria, only one event is counted.

[0037] In some embodiments, a hypoglycemia event is defined as severe if any of the following conditions are met: 1. The hypoglycemia ICD-9 or 10 diagnostic code is default severe (ICD-9 249.30; 250.30; 250.31; 251.0; ICD-10 E08.641; E09.641; E10.641; E11.641; E13.641; E15). 2. The ICD code for hypoglycemia is flagged as the admission diagnosis, the main reason for treatment at discharge, or present at admission. 3. The hypoglycemia start date occurs on the same day as the emergency department (ED) visit or inpatient admission of the patient. 4. The plasma glucose measurement is <54 mg / dL. 5. Intramuscular glucagon has been administered. 6. The NLP mention of hypoglycemia is accompanied by severity descriptors, including severity terms (e.g., "severe") and attributes (e.g., "urgent"). 7. The NLP hypoglycemia event occurs on the same day as the (ED) visit or inpatient admission of the patient.

[0038] The machine learning environment 110 may include a machine learning trainer 112. The machine learning trainer 112 can train a machine learning model 114 to predict the prediction rate of hypoglycemia events for different patients. The trained machine learning system is used in a variety of different ways, including identifying the most appropriate basal insulin related to the patient's hypoglycemia outcome, thereby improving the health of that patient.

[0039] Generally, machine learning can encompass a wide variety of different techniques that are used to train a machine to perform a specific task without the machine being specifically programmed to perform those tasks. The machine is trained using different machine learning techniques, such as supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the input of interest and the corresponding output are provided to the machine. The machine adjusts its function to provide the desired output when the input is provided. Supervised learning is generally used to teach a computer to solve problems where the result is deterministic, for example, and the training set 108 is used to train the trained machine learning model 114 to predict the likelihood of hypoglycemic events for a given patient or group of patients. In contrast, in unsupervised learning, an input is provided, but the corresponding desired output is not. Unsupervised learning is generally used for classification problems such as customer segmentation (for example, segmenting patients into different groups based on characteristics related to hypoglycemic events). Reinforcement learning describes an algorithm where the machine makes decisions using a trial-and-error approach. Feedback notifies the machine when a good or bad choice has been made. The machine then adjusts its algorithm accordingly.

[0040] During the training process, different algorithms are used, including, among others, generalized linear regression (GLM). Poisson GLM is an algorithm used to model discrete counts based on individual inputs.

[0041] To develop a trained machine learning system that can accurately predict the rate of hypoglycemic events, the machine learning model 114 can be trained with information that does not lead to misunderstandings. However, the treatment of patients with diabetes (and other medical conditions) can be fluid. For example, a patient may switch from one type of basal insulin to another. Thus, in some embodiments, the EMR 102 of some patients (and thus the corresponding training data 108) is excluded from the training data.

[0042] Figure 2 is a flowchart showing an example of a process for classifying an event as an ED / Outpatient (result 210), Inpatient (secondary) (result 212), or Inpatient (primary) (result 214).

[0043] An event is defined as a primary inpatient (result 214) when it meets all of the following criteria: 1. The event is linked to a visit form and the visit type is not ED (step 202). 2. The hypoglycemic event is not identified solely by natural language processing (step 204). 3. The event is found using a diagnosis form (step 206). 4. The diagnosis is marked as "discharge diagnosis", "admission diagnosis", or "present at admission".

[0044] An event is defined as a secondary inpatient when it meets all of the following criteria: 1. The hypoglycemic event is not identified solely by NLP (step 204). 2. If the event is found using a diagnosis form, the diagnosis is not marked as "discharge diagnosis", "admission diagnosis", or "present at admission" (step 206). 3. The event is linked to a visit form using the PTID, the visit type is inpatient, and the hypoglycemic event date is between the visit start date minus 1 day and the visit end date plus 1 day (this provides a buffer for linking the hypoglycemic event to the visit considering that there are often no exact date matches in the dataset) (step 208).

[0045] In some embodiments, as described above, secondary inpatient events are excluded because the patient is often switched to a different basal insulin and the patient's dose is changed during the patient's inpatient stay. Therefore, any hypoglycemic events during this period are not considered to be due to the patient's normal insulin.

[0046] ​For example, an event is defined as an external patient / ED (Outcome 210) if it meets any of the following: 1. The event is linked to a visit form using the PTID, the visit type is ED, and the hypoglycemia event date is between the visit start date minus one day and the visit end date plus one day (Step 202). 2. The event is identified using only natural language processing (Step 204). 3. When the event is identified using a diagnosis form, it is not identified using only NLP, the diagnosis is not marked as discharge diagnosis, admission diagnosis, or present at admission, and no link to the visit form is made (Step 206).

[0047] Training records are used to train a machine learning system when they are created. Different types of machine learning models are trained.

[0048] For example, a trained learning model is embodied as a generalized linear model. Different types of generalized linear models may be appropriate in various scenarios. A zero-inflated negative binomial distribution GLM (zNBGLM) was used. The reason is that zNBGLM models the discrete count of events occurring in a given period by estimating the rate of hypoglycemia events per patient where the hypoglycemia count in the data was most likely to occur. One disadvantage of zNBGLM is that the model allows for a very large number of degrees of freedom and has a tendency to overfit the data for small segments, thereby reducing the generalization performance. Another type of generalized linear model is the Poisson GLM. Poisson GLM is well-suited for modeling discrete counts but does not allow for "overdispersion" (i.e., suppresses the dispersion to be equal to the mean). In Poisson GLM, the number of hypoglycemia events was used as the target variable (outcome), and the length of the observation was used as the offset variable.

[0049] In another example, the trained learning model is embodied as an artificial neural network. An artificial neural network (ANN) or connectionist system is a computing system inspired by the biological neural networks that make up the brains of animals. An ANN is based on a collection of interconnected units or nodes called artificial neurons. Each connection can transmit signals from one artificial neuron to another, like the synapses in a biological brain. An artificial neuron that receives a signal can process the signal and then send the signal to additional artificial neurons connected to that artificial neuron.

[0050] In a common ANN implementation, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated by some non-linear function of the sum of its inputs. The connections between artificial neurons are called "edges". Artificial neurons and edges can have weights that are adjusted as learning progresses (e.g., each input to an artificial neuron is weighted separately). The weights increase or decrease the strength of the signals at the connections. An artificial neuron can have a threshold such that a signal is sent only if the collective signal crosses that threshold. The transfer function along an edge usually has an S-shape, but can also take the form of other non-linear functions, piecewise linear functions, or step functions. Generally, artificial neurons are grouped into multiple layers. Different layers can perform different kinds of transformations on their inputs. Signals move from the first layer (input layer) to the last layer (output layer), sometimes crossing each layer multiple times.

[0051] In some embodiments, the machine learning system is used to identify the rate of hypoglycemic events and the cost of hypoglycemia. LASSO (Least Absolute Shrinkage and Selection Operator) regression is used to select variables. To validate the model, the model is developed (i.e., trained) on 80% of each treatment-specific cohort (referred to as the "training set"). 10-fold cross-validation is used to inform model selection and model parameter optimization. Next, the model is validated on the remaining 20% of each treatment-specific cohort (internal validation). Bootstrapping is used to evaluate the variability of the model estimates (i.e., to generate confidence intervals).

[0052] Once the machine learning system is trained, the system is used to identify patients who may have fewer hypoglycemic events compared to others when treated with a certain type of basal insulin. For example, when the model is trained for each type of basal insulin and for the number of severe and non-severe hypoglycemic events. Next, each model is applied to the entire population treated with basal insulin to obtain an insulin-specific hypoglycemia rate prediction (i.e., an estimate of the hypoglycemia rate in the total population assuming all patients are using a specific basal insulin).

[0053] Next, the system can compare the hypoglycemia rate of a patient based on other variables.

[0054] Figure 3 shows an example of using a trained machine learning model. Patient EMR 310 is processed to generate input 312. Input 312 is provided to each of the trained machine learning models, in this example, the Gla-300 trained machine learning model, the Gla-100 trained machine learning model 304, the IDet trained machine learning model 306, and the IDeg trained machine learning model 308. Each model can generate one output. For example, the Gla-300 trained machine learning model 302 generates a Gla-300 output 314, the Gla-100 trained machine learning model 304 generates a Gla-100 output 316, the IDet trained machine learning model 306 generates an IDet output 318, and the IDeg trained machine learning model 308 generates an IDeg output 320.

[0055] In some embodiments, as described above, two trained machine learning models are generated for each type of basal insulin. The first trained machine learning model is trained to determine the predicted rate of severe hypoglycemic events. The second trained machine learning model is trained to determine the predicted rate of non-severe hypoglycemic events.

[0056] The results of processing these covariates using the machine learning system are analyzed to identify the correlation between different covariates and different hypoglycemia rates for different types of basal insulin. For example, a linear regression model is used to identify the correlation between different covariates and the results predicted by this model.

[0057] In some embodiments, the covariates of the subject are also determined by the machine learning system. For example, machine learning is trained to cluster individuals based on, for example, the rate and severity of hypoglycemic events across multiple variables. In this way, the machine learning system can identify covariates that might not otherwise be noticed.

[0058] In some embodiments, an EMR and a training dataset are used to construct a cost model for predicting the costs of hypoglycemic events in a T2DM population. In some embodiments, the dataset used for treatment cost modeling included all hypoglycemic events in the EHR for T2DM patients who were at least 18 years old at the time of the event and had linked claims data. Severe hypoglycemic events with a cost of $0 were excluded. The study period and covariates were the same as those described for the hypoglycemia prediction model, except when covariates could not be created due to data limitations.

[0059] Gradient boosting trees (which use the prediction errors from previous decision trees to improve the performance of subsequent trees), which have been previously successful in cost prediction, were utilized for cost estimation; this enabled capturing the non - linear and complex relationships underlying hypoglycemia costs. The cost estimator was applied to subgroups identified as drivers of the hypoglycemia differential rate, and the cost per hypoglycemic event was estimated for each subgroup. If important defining variables were missing from a subgroup due to data limitations, the total model cost estimate for one hypoglycemic event was used for that subgroup. Cost savings at the subgroup level were calculated by applying the cost estimate specific to a subgroup of hypoglycemic events to the delta hypoglycemic event rate between the comparator BI.

[0060] Figure 4 shows an example of determining the hypoglycemia rate for various covariates. Input data 402 provided to a machine - learning system and corresponding output data 404 from the machine - learning system described above are provided to a statistical analysis system 406. The statistical analysis system can identify correlations and relationships between different variables in the output data using various statistical techniques. The correlations and relationships are presented as a report 408.

[0061] Another use of the trained machine - learning model includes identifying the appropriate type of basal insulin for a particular patient. For example, a trained machine - learning system can obtain and input a patient's medical record.

[0062] Medical records are provided to each of the trained machine learning models. As described above, each model was used to predict the number of severe or non-severe hypoglycemic incidents (or the probability that a hypoglycemic incident will occur).

[0063] The system can propose basal insulin based on the results of the model. In some embodiments, the system can propose basal insulin with a low risk of severe hypoglycemic events (or non-severe hypoglycemic events). In some embodiments, the system can determine a cost reduction by using a specific basal insulin regimen compared to another related to hypoglycemic events and propose a solution to reduce hypoglycemia-related costs. In some embodiments, the system can propose basal insulin that reduces the rate of hypoglycemic events, but if the results show that two different basal insulins are within a threshold (e.g., an effect within 1%, 5%, or 10%), the system can propose the basal insulin with the lower cost.

[0064] In another embodiment, the trained machine learning model is used to identify patients with a higher likelihood of hypoglycemic events. For example, the system can access a patient's medical record, identify the type of basal insulin the patient is using, and process the medical record with the corresponding trained machine learning model. The trained machine learning model generates an indication of the likelihood or expected frequency and / or severity of hypoglycemic events. If the indication exceeds a threshold (e.g., more than 1 event per week, or a likelihood of severe hypoglycemic events greater than 20%), the patient and / or the patient's physician is notified.

[0065] FIG. 5 is a flowchart of an example of a process 500 for generating a trained machine learning model using patient data. Process 500 is executed by one or more computer systems as described below.

[0066] Process 500 receives, at 502, data indicating the medical records of patients diagnosed with type 1 diabetes.

[0067] At 504, process 500 determines a prediction rate of hypoglycemic events using a machine learning system, which is trained using data indicating the medical records of a plurality of patients and the corresponding rates of hypoglycemic events for each patient. In some embodiments, the plurality of patients are taught to use the same type of basal insulin.

[0068] At 506, process 500 generates a prediction rate for the patient.

[0069] In some embodiments, process 500 can include using a second machine learning system to determine a second prediction rate of hypoglycemic events, where the second machine is trained using data indicating the medical records of a second plurality of patients and the corresponding rates of hypoglycemic events for each second patient, each of the second plurality of patients uses a second type of basal insulin, and this second type of basal insulin is different from the first type of basal insulin; process 500 can further include comparing the first prediction rate with the second prediction rate.

[0070] In some embodiments, process 500 can include recommending a basal insulin for the patient based on the comparison.

[0071] In some embodiments, process 500 can include determining a plurality of prediction rates of hypoglycemic events for a second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system, identifying one or more covariates in the data correlated with the prediction rates of hypoglycemic events based on the data and the plurality of prediction rates of hypoglycemic events, and generating a report identifying the one or more covariates and the corresponding prediction rates of hypoglycemic events.

[0072] In some embodiments, process 500 includes determining a plurality of prediction rates of hypoglycemic events for a second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system, each of the second plurality of patients having the same covariates, and process 500 may further include generating a report identifying the covariates and the corresponding prediction rates of hypoglycemic events.

[0073] The described embodiments and functional operations of the subject matter hereof are implemented as digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed herein and their structural equivalents), or any combination of one or more of these. The described embodiments of the subject matter hereof are implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0074] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of devices, devices, and machines for processing data, examples of which include programmable processors, computers, or multiple processors or computers. The device may also be or separately include dedicated logic circuitry (e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)). In addition to the hardware, the device may optionally include code that creates an execution environment for the computer program (e.g., processor firmware, protocol stack, database management system, operating system, or code constituting one or more combinations of these).

[0075] A computer program, which may be referred to or described as a program, software, software application, module, software module, script, or code, is written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and the computer program is arranged in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computer environment. The computer program may or may not correspond to a file in a file system. The program is stored in a part of a file that holds other programs or data (e.g., a markup language document, a single file dedicated to the program in question, or one or more scripts stored in multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code)). The computer program is arranged to be executed on one computer or at one location, or is distributed over multiple computers located at multiple locations and interconnected by a data communication network so that the program is executed on those multiple computers.

[0076] The processes and logical flows described herein are executed by one or more programmable computers executing one or more computer programs for performing functions by operating on input data and generating output. The processes and logical flows may also be executed by dedicated logic circuitry (e.g., an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)), and the apparatus may also be implemented as this dedicated logic circuitry.

[0077] A computer suitable for executing a computer program can be based on a general-purpose or special-purpose microprocessor or both, or any other type of central processing unit. Generally, the central processing unit receives instructions and data from a read-only memory or a random access memory or both. Indispensable elements of a computer are a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operatively connected to receive, transfer, or both data with a mass storage device, but such a device may not be necessary for a computer. Further, a computer is embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive)).

[0078] Computer-readable media suitable for storing computer program instructions and data include, by way of example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), any form of non-volatile memory on the media and memory devices, magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, as well as CD-ROM disks and DVD-ROM disks. The processor and memory are complemented by or incorporated into dedicated logic circuitry.

[0079] To provide for interaction with a user, embodiments of the subject matter described herein are implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and a pointing device (e.g., a mouse or trackball) that enable the user to provide input to the computer. Other types of devices are similarly used to provide for interaction with the user; for example, feedback provided to the user can be in any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input received from the user can be in any form including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending a document to, and receiving a document from, one device used by the user, e.g., by sending a web page to a web browser in response to a request received from the web browser on the user's device.

[0080] Embodiments of the subject matter described herein are implemented in one computer system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface or a web browser that enables a user to interact with an implementation of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system are interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0081] A computing system can include a client and a server. The client and the server are generally separated from each other and typically interact via a communication network. The relationship between the client and the server is created by computer programs that run on respective computers and have a client - server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a user device (e.g., to display the data to a user interacting with the user device and to receive user input from that user), and this user device functions as a client. Data generated at the user device (e.g., as a result of user interaction) is received at the server from the user device.

[0082] This specification includes details of many specific embodiments, which should be construed not as limitations on the scope of any invention or what can be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features described herein in the context of separate embodiments may be implemented in combination as a single embodiment. Conversely, various features described in the context of a single embodiment may be implemented separately in multiple embodiments or in any suitable sub - combination. Further, although the features are described above as acting in certain combinations and are thus initially claimed as such, one or more of the features of the claimed combination may in some cases be deleted from the combination, and the claimed combination is directed to a sub - combination or a variant of a sub - combination.

[0083] Similarly, although the operations are depicted in the drawings in a particular order, this should not be construed as requiring that the operations be performed in the particular order or sequence shown, or that all of the illustrated operations be performed, to obtain a desirable result. Multitasking and parallel processing may be advantageous in certain circumstances. Further, separating the various system modules and components in the embodiments described above should not be construed as requiring such separation in all embodiments, and the described program components and systems are generally understood to be integrated as a single software product or packaged into multiple software products.

[0084] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still obtain a desirable result. As one example, the processes depicted in the accompanying figures need not be in the particular order or sequence shown to obtain a desirable result. In some cases, multitasking and parallel processing may be advantageous.

Claims

**Claim 1** A method executed by a computer system, comprising: receiving data indicating medical records of patients diagnosed with type 1 diabetes; using a first machine learning model trained using first training data including medical records of a first plurality of patients, each using a first type of insulin, and data representing the corresponding rate of hypoglycemic events for each patient, to process the medical records of the patient to determine a first predicted rate of hypoglycemic events; wherein a second type of insulin is different from the first type of insulin, and each of a second plurality of patients uses the second type of insulin, and using a second machine learning model trained using second training data including medical records of the second plurality of patients and data representing the corresponding rate of hypoglycemic events for each patient, to process the medical records of the patient to determine a second predicted rate of hypoglycemic events; comparing the first predicted rate of hypoglycemic events with the second predicted rate of hypoglycemic events; recommending insulin treatment for the patient based on the comparison, the method comprising the above steps. **Claim 2** determining a plurality of predicted rates of hypoglycemic events for a third plurality of patients by providing data corresponding to the medical records of each of the third plurality of patients to one of the machine learning models; identifying one or more covariates in the data that correlate with the third predicted rate of hypoglycemic events based on the plurality of predicted rates of hypoglycemic events and the data; creating a report identifying the one or more covariates and the corresponding third predicted rate of hypoglycemic events, the method according to claim 1, further comprising the above steps. **Claim 3** determining a plurality of predicted rates of hypoglycemic events for a third plurality of patients by providing data corresponding to the medical records of each of the third plurality of patients to one of the machine learning models, wherein each of the third plurality of patients has the same covariate; generating a report identifying the covariate and the corresponding third predicted rate of hypoglycemic events, the method according to claim 1, further comprising the above steps. **Claim 4** ​ A non-transitory computer-readable medium encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: processing the medical records of a patient using a first machine learning model trained using first training data that includes the medical records of a first plurality of patients, each of whom uses a first type of insulin, and data representing the corresponding rate of hypoglycemic events for each patient, to determine a first predicted rate of hypoglycemic events; wherein a second type of insulin is different from the first type of insulin, and each of a second plurality of patients uses the second type of insulin, and processing the medical records of the patient using a second machine learning model trained using second training data that includes the medical records of the second plurality of patients and data representing the corresponding rate of hypoglycemic events for each patient, to determine a second predicted rate of hypoglycemic events; comparing the first predicted rate of hypoglycemic events to the second predicted rate of hypoglycemic events; recommending insulin treatment for the patient based on the comparison; wherein the non-transitory computer-readable medium causes the one or more computers to perform operations comprising the above. **Claim 5** The operations further comprise: determining a plurality of predicted rates of hypoglycemic events for a third plurality of patients by providing data corresponding to the medical records of each of the third plurality of patients to one of the machine learning models; identifying one or more covariates in the data that correlate to a third predicted rate of hypoglycemic events based on the plurality of predicted rates of hypoglycemic events and the data; creating a report identifying the one or more covariates and the corresponding third predicted rate of hypoglycemic events; The non-transitory computer-readable medium according to claim 4, further comprising the above. **Claim 6** The operations further comprise: determining a plurality of predicted rates of hypoglycemic events for a third plurality of patients by providing data corresponding to the medical records of each of the third plurality of patients to one of the machine learning models, wherein each of the third plurality of patients has the same covariates; Generating a report that identifies the covariance and the rate of the third corresponding predicted hypoglycemic event The non-transitory computer-readable medium of claim 4, further comprising **Claim 7** A system comprising: One or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, are operable to cause the one or more computers to perform operations, the system comprising: Each of a first plurality of patients uses a first type of insulin and uses a first machine learning model trained using first training data including the medical records of the first plurality of patients and data representing the rate of corresponding hypoglycemic events for each patient to process the medical records of a patient to determine a first predicted rate of hypoglycemic events; A second type of insulin is different from the first type of insulin. Each of a second plurality of patients uses the second type of insulin and uses a second machine learning model trained using second training data including the medical records of the second plurality of patients and data representing the rate of corresponding hypoglycemic events for each patient to process the medical records of the patient to determine a second predicted rate of hypoglycemic events; Comparing the first predicted rate of hypoglycemic events with the second predicted rate of hypoglycemic events; Recommending insulin treatment for a patient based on the comparison The system, which is operable to cause the operations to be performed **Claim 8** The operations are: Determining a plurality of predicted rates of hypoglycemic events for a third plurality of patients by providing data corresponding to the medical records of each of the third plurality of patients to one of the machine learning models; Identifying one or more covariates in the data that correlate with a third predicted rate of hypoglycemic events based on the plurality of predicted rates of hypoglycemic events and the data; Creating a report that identifies the one or more covariates and the corresponding third predicted rate of hypoglycemic events The system of claim 7, further comprising **Claim 9** The operations are: By providing data corresponding to the medical records of each of the third plurality of patients to one of the machine learning models, determining a percentage of a plurality of predicted hypoglycemic events for the third plurality of patients, wherein each of the third plurality of patients has the same covariates; Generating a report identifying the covariates and the percentage of the third corresponding predicted hypoglycemic events; The system of claim 7, further comprising.

Images