Predicting rates of hypoglycemia using machine learning system

Abstract

To provide a system, method, and computer product for predicting the rates of hypoglycemia in patients.SOLUTION: A method disclosed herein comprises receiving data representing medical records of a patient diagnosed with diabetes mellitus, and determining a predicted rate of hypoglycemic events using a machine learning system, the machine being trained using data representing medical records of a plurality of patients and corresponding rate of hypoglycemic events for respective patients. The methods also comprises producing a predicted rate for a patient.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] Priority claims This application claims the benefit of U.S. Provisional Patent Application No. 62 / 689,005, filed June 22, 2018, the entire contents of which are incorporated herein by reference. [Background technology]

[0002] Machine learning is a subset of artificial intelligence in the field of computer science that often uses statistical techniques to give computers the ability to "learn" (i.e., progressively improve their performance on specific tasks) from data without being explicitly programmed. Summary of the Invention [Means for solving the problem]

[0003] In general, one inventive aspect of the subject matter described herein is embodied as a method that includes the acts of receiving data indicative of medical records of patients diagnosed with diabetes mellitus. The method includes the act of determining a predictive rate of hypoglycemic events using a machine learning system, the machine being trained with data indicative of the medical records of a plurality of patients and a corresponding rate of hypoglycemic events for each patient. The method also includes the act of generating a predictive rate for the patients.

[0004] The above and other embodiments may each optionally include one or more of the following features, alone or in combination: Each of the multiple patients may use the same type of basal insulin. The method may include determining a second predicted rate of hypoglycemic events using a second machine learning system, the second machine being trained with data indicative of medical records of a second multiple of patients and corresponding rates of hypoglycemic events for each second patient, each of the second multiple of patients using a second type of basal insulin, the second type of basal insulin being different from the first type of basal insulin, and the method may further include comparing the first predicted rate to the second predicted rate. The method may include recommending a basal insulin for the patient based on the comparison. The method may include acts of determining a plurality of predictive rates of hypoglycemic events for a second plurality of patients by providing data corresponding to medical records of each of the second plurality of patients to a machine learning system, identifying one or more covariates in the data that are correlated to the predictive rates of hypoglycemic events based on the data and the plurality of predictive rates of hypoglycemic events, and generating a report identifying the one or more covariates and the corresponding predictive rates of hypoglycemic events. The method may include acts of determining a plurality of predictive rates of hypoglycemic events for the second plurality of patients by providing data corresponding to medical records of each of the second plurality of patients to a machine learning system, each of the second plurality of patients having the same covariates, and the method may further include acts of generating a report identifying the covariates and the corresponding predictive rates of hypoglycemic events.

[0005] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations by implementing the methods provided herein.

[0006] The present disclosure further provides a system for implementing the methods provided herein, the system including one or more processors and a computer-readable storage medium coupled to the one or more processors and having instructions stored thereon, the instructions, when executed by the one or more processors, causing the one or more processors to performs the operations by implementing the methods provided herein.

[0007] It should be understood that embodiments according to the present disclosure can include any combination of the aspects and features described herein, i.e., embodiments according to the present disclosure are not limited to combinations of the aspects and features specifically described herein, but also include any other suitable combinations of the provided aspects and features.

[0008] The details of one or more embodiments of the subject matter described herein are set forth in the accompanying drawings and the detailed description below. Other features, aspects, and advantages of the subject matter will become apparent from the detailed description, the drawings, and the claims. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an environment in which a machine learning model is trained to predict expected rates of hypoglycemic events. [Figure 2] 1 is a flowchart illustrating an example of a process for classifying an event as ED / outpatient, inpatient (secondary), or inpatient (primary). [Figure 3] FIG. 1 illustrates an exemplary process for determining hypoglycemia rates for various covariates. [Figure 4] FIG. 1 shows an example of determining hypoglycemia rates for various covariates. [Figure 5] 1 is a flowchart of an example process for generating a trained machine learning model using patient data. DETAILED DESCRIPTION OF THE INVENTION

[0010] Like reference numbers and designations in the various drawings indicate like elements.

[0011] Diabetes mellitus is the seventh leading cause of mortality and the leading cause of morbidity in the United States (U.S.). Diabetes mellitus affects an estimated 29.1 million people in the U.S. population, with 1.4 million new cases diagnosed each year. The number of affected patients is expected to increase to over 54.9 million by 2030. In 2012, the total cost of diagnosed diabetes in the U.S. was $245 billion ($176 billion in direct medical costs and $69 billion in lost productivity).

[0012] Given the significant and growing burden of the disease, diabetes-associated complications are becoming increasingly important for effective prevention and management. Hypoglycemia is a frequent and potentially fatal adverse effect of insulin and oral antidiabetic drugs (OADs) in diabetic patients. In addition to the immediate risks posed by hypoglycemia, recurrent episodes can lead to anxiety about future episodes, which have been shown to constitute both patient- and physician-initiated barriers to optimal glycemic control. The resulting elevation of hemoglobin A1c (HbA1c) levels has been linked to an increased risk of microvascular (and sometimes macrovascular) complications.

[0013] Patients with type 1 diabetes mellitus (T1DM) are estimated to experience, on average, two mild hypoglycemic events per week and one severe event per year. However, event rates from randomized clinical trials for type 1 patients, expressed per patient per year, range from 0.15 for severe events to 88.3 for nonsevere events. Event rates for patients with type 2 diabetes mellitus (T2DM) vary considerably across studies, ranging from 0.05 to 26.6 events per patient per year for severe and nonsevere events, respectively. However, studies suggest that these studies significantly underestimate the true real-world event rates of hypoglycemia, especially for severe events.

[0014] The average cost for a hypoglycemic event is also heterogeneous across studies and difficult to pool due to differences in hypoglycemia definitions and cost estimation methods. Current estimates of the average cost for an inpatient hypoglycemic event range from $2,205 to $17,564. The average outpatient cost per hypoglycemic event ranges from $148 to $501.

[0015] Different basal insulins have demonstrated hypoglycemic differential rates. For example, numerous studies have demonstrated the superiority of insulin glargine 100 units / mL (Lantus) compared with protamine insulins with regard to severe hypoglycemic events. Some basal insulins, such as insulin glargine 300 units / mL (Toujeo), exhibit flatter, longer-lasting pharmacokinetic and pharmacodynamic profiles compared with others, with sustained glycemic control over 24 hours.

[0016] Thus, the burden of hypoglycemia in the United States is significant, and identifying the appropriate type of basal insulin for each patient could provide significant benefits. Estimates of the resulting cost savings to payers from reduced healthcare costs for these patients can help guide payer formulary decisions and drug pricing negotiations.

[0017] FIG. 1 illustrates an environment 100 in which a machine learning model is trained to predict expected rates of hypoglycemic events. The system described herein uses electronic medical record data (EMR) 102 to train the machine learning system. (EMRs are derived from a number of different sources, including, but not limited to, hospital and physician records.) In some embodiments, the EMR may include information about demographic and socioeconomic categories, coded diagnoses and procedures, prescribed and administered medications, laboratory results, and clinical management data. In some embodiments, the EMR is processed by a processor 104. For example, the EMR may include both structured and unstructured data. Structured data may include information such as visit dates, patient names, etc. Unstructured data may include free-form text added by a physician (such as physician notes, visit summaries, etc.). The EMR processor can use techniques such as natural language processing to extract facts from the unstructured data in the EMR and turn the unstructured data into structured EMR data 106.

[0018] In some implementations, the EMR processor 104 can filter portions of the medical records. For example, the EMR processor 104 can select only medical records of patients who share values ​​for certain variables (called covariates). Examples of covariates can include, for example, gender, geographic region, rates, age group, insurance company, years since diagnosis, HbA1c range, body mass index, blood pressure range, diabetic complications, alcohol and / or drug use, and any other physiological or demographic characteristic. The EMR processor 104 can be, for example, one or more computer systems as described below.

[0019] In some embodiments, the same manually created covariates are used for descriptive analysis, predictive modeling of hypoglycemia rates, and cost estimation analysis. The predetermined covariates may be based on expert clinical reviews of the literature surrounding hypoglycemic events and cost predictors. The predetermined covariates are defined using ICD-9 and ICD-10 diagnosis codes, laboratory values, and drug names and / or National Drug Code (NDC) codes. EHR datasets (not insurance claims) are used to identify covariates unless the covariate is cost-related, in which case insurance claims data is used.

[0020] A default look-back period of 1 year before treatment initiation is used, but this period can also vary depending on how long the covariates are assumed to persist. For example, the look-back period for cancer is It can be 5 years because if a patient has been diagnosed with cancer more than 5 years ago and has not had a subsequent diagnosis, they are unlikely to have active cancer at the index date. For covariates where the length of medical history does not affect the likelihood of a patient being captured in the dataset, such as sex and race, the look-back period can be 8 years, or limited only by the amount of data available. For irreversible and chronic conditions, the maximum look-back period can also be 8 years, or limited only by the amount of data available.

[0021] An example of categories of covariates used in both the hypoglycemia rate model and the cost estimation analysis and their look-back periods may be: 1. Demographics a. The look-back period for these covariates is 8 years.

[0022] 2. Socioeconomics b. The look-back period for these covariates is 8 years.

[0023] 3. Comorbidities c. Look-back periods for these covariates range from 1 year (for reversible / acute conditions) to 8 years (for irreversible / chronic conditions).

[0024] d. Charlson Comorbidity Index (CCI) score as a separate covariate within comorbidity category. The CCI is a measure of a patient's comorbidity status, including diabetes complications, associated with the pre-index period (including diabetes comorbidity categories).

[0025] 4. Diabetic complications e. The look-back period for these covariates is 8 years.

[0026] 5. Diabetes disease state a. The look-back period for these covariates is 1 year for prior hypoglycemic events and 8 years for diabetes of known duration in the dataset.

[0027] 6.Medicinal Use a. The look-back period for these covariates is 1 year.

[0028] Additional covariates are included as part of the cost estimate analysis only (rather than the hypoglycemia rate prediction) because these covariates are predicted to be drivers of hypoglycemia-related costs (rather than hypoglycemia rate): 1. Physician Specialty: f. The look-back period is all available data for the "physician specialty" covariate and 2 years for "physician's most commonly prescribed insulin."

[0029] 2. Average previous hypoglycemic event cost g. The look-back period for this covariate is 1 year.

[0030] Another set of covariates is determined. This set of covariates is not pre-specified, but encompasses all comorbidities, procedures, and prescriptions that existed one year prior to the patient's index date (collectively referred to below as "markers") that are included in the predictive modeling.

[0031] In some embodiments, the following approach for unsupervised covariate creation is used.

[0032] 1. The distance between markers is defined: Each marker is associated with a vector containing the set of patients for which the marker is either true or false.

[0033] b. In this case, the distance between markers is the Jaccard distance between vectors.

[0034] 2. Then, clusters based on these distances were generated: a. Hierarchical clustering based on a distance matrix was performed to obtain a cluster hierarchy. The average distance between clusters was used to link the clusters hierarchically.

[0035] 3. The levels of hierarchy for extracting clusters are defined: a. The "discrepancy" method was used to determine at what level of the hierarchy we wanted to extract clusters. "Discrepancy" refers to the discrepancy in the average distances among linked clusters: a large value suggests that the clusters should not be linked.

[0036] 4. Clusters that "show" many patient treatments are selected: a. Of the 1500 clusters formed, 100 represented by most patient treatments were selected.

[0037] b. A patient treatment was said to "indicate" a cluster if the patient had any of the diagnoses, procedures, or prescriptions (e.g., markers) that constituted the cluster during the year prior to the index date.

[0038] 5. Medical rationalization of the cluster: a. Clinical experts then inspect the resulting clusters for medical logic to validate the parameters used for unsupervised cluster generation.

[0039] In some implementations, the EMR records are filtered, for example by a filter 116, before generating the training data 108 but after generating the structured medical records 106.

[0040] The structured EMR data 106 is used to generate training records 108 (collectively the training set). The training records may represent the available data in the structured EMR data. For example, in some embodiments, a portion of the structured EMR records is used to train a machine learning system, and the remaining records are used to validate the trained machine learning system. In some embodiments, the training records are created at the level of a "patient treatment," where the patient is on basal insulin treatment. Thus, multiple training records are created for each individual patient. The unit of analysis is the "patient treatment," defined as the period during which the patient is observed on basal insulin treatment in the dataset (the period between the treatment index and the end of treatment observation). Hypoglycemic events are the target endpoint only within this patient treatment period.

[0041] The treatment index date is defined as either the very first start of any basal insulin regimen; or the change of regimen from one basal insulin to another. Baseline basal insulins included: Gla-300, Gla-100, IDet, IDeg, and NPH. The index basal insulins of interest in the study were: Gla-300, Gla-100, IDet, and IDeg.

[0042] For purposes of rate calculation, "duration" is interpreted as the patient treatment period as defined above minus the duration of all inpatient stays during this period.

[0043] In some embodiments, the index date is the date of the first prescription of a BI or one basal index. The end of treatment was defined as the end of the follow-up period in the dataset, a change in prescription from the index basal insulin to another BI, or one year after the treatment index date (whichever occurred first).

[0044] In one embodiment, a patient treatment is excluded from the training data if the patient treatment meets any of the following criteria: 1. Treatment with more than one type of basal insulin: i.e., patient treatment initiated within one week (before or after) the initiation of another treatment in the same patient.

[0045] 2. Patient treatments with any period of inactivity longer than 270 of the 365 days prior to the index date (inactivity is defined as the absence of time-stamped data in the relevant table in the dataset).

[0046] 3. Patient treatment duration is less than 1 day.

[0047] Separately, inpatient stays are excluded from the patient treatment period because patients are often switched to standard basal insulin according to the hospital provider formulary upon admission, and therefore hypoglycemic events during inpatient stays cannot be attributed to the index basal insulin.

[0048] The training record may include information about hypoglycemic events. In some embodiments, hypoglycemic events are counted within a patient treatment period. The period for determining the hypoglycemic rate was interpreted as the patient treatment period minus the duration of all inpatient stays during this period. The hypoglycemic events may be a predicted output of the training set. For example, the training set may train a machine learning system to determine the likelihood of the expected number of hypoglycemic events within a fixed period (e.g., 1 month, 6 months, 1 year, 5 years, etc.). Alternatively, the training set may train a machine learning system to determine the expected number of hypoglycemic events within a certain period (e.g., 1 month, 6 months, 1 year, 5 years, etc.).

[0049] The hypoglycemic event rate may include both severe and non-severe events. Figure 2 is a flowchart of an example process for classifying hypoglycemic events as severe or non-severe. The definition of "severe" hypoglycemia may include, for example, ICD-9 / 10 codes that are severe in nature, administration of intramuscular glucagon. Additionally, natural language processing of the EMR is used to identify hypoglycemia. With respect to severity, any hypoglycemic event that was not severe is defined as "non-severe."

[0050] In some embodiments, an event is defined as hypoglycemia if any of the following criteria are met: 1. ICD-9 and 10 Hypoglycemia Diagnostic Codes 2. Laboratory plasma glucose level ≤ 70 mg / dL 3. Intramuscular glucagon administration 4.NLP Output In some embodiments, an NLP-recognized hypoglycemic event is defined as any mention of hypoglycemia, excluding those with a negative sentiment or historical event indication. For example, a "mention" of hypoglycemia may be any event in relation to the regular expression "*hypoglycemia*," as long as the term is not exactly one of "hypoglycemia awareness," "hypoglycemia unawareness," or "neonatal hypoglycemia." A "negative sentiment" may be any indication that the mention of hypoglycemia was negative, for example, indicating that hypoglycemia did not occur. A historical event is a record indicating that the mention is of a past event (e.g., "patient has a history of hypoglycemia"). In some embodiments, lists of negative sentiment and historical keywords are used to filter out accompanying hypoglycemic mentions.

[0051] In some embodiments, a maximum of one hypoglycemic event is counted per calendar day. For example, if one hypoglycemic event is recorded at multiple sites of treatment or by multiple definition criteria, only one event is counted.

[0052] In some embodiments, a hypoglycemic event is defined as severe if any of the following conditions are met: 1. Hypoglycemia ICD-9 or 10 diagnosis code is severe by default (ICD-9 249.30; 250.30; 250.31; 251.0; ICD-10 E08.641; E09.641; E10.641; E11.641; E13.641; E15).

[0053] 2. An ICD code for hypoglycemia is flagged as the admitting diagnosis, the primary reason for discharge treatment, or present at the time of admission.

[0054] 3. Hypoglycemia onset date occurred on the same day as an emergency department (ED) visit or inpatient admission.

[0055] 4. Plasma glucose measurement <54 mg / dL.

[0056] 5. Intramuscular glucagon was administered.

[0057] 6. NLP mentions of hypoglycemia are accompanied by severity descriptors, including severity terms (e.g., “severe”) and attributes (e.g., “urgent”).

[0058] 7. NLP hypoglycemic event occurring on the same day as an (ED) visit or inpatient admission.

[0059] The machine learning environment 110 can include a machine learning trainer 112. The machine learning trainer 112 can train a machine learning model 114 to predict expected rates of hypoglycemic events for different patients. The trained machine learning system can be used in a variety of different ways, including identifying the most appropriate basal insulin associated with a patient's hypoglycemic outcomes, thereby improving the patient's health.

[0060] In general, machine learning can encompass a wide variety of different techniques used to train machines to perform specific tasks without the machine being specifically programmed to perform those tasks. Machines are trained using different machine learning techniques, including, for example, supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, a machine is provided with inputs of interest and corresponding outputs. The machine adjusts its function to provide a desired output when the input is provided. Supervised learning is generally used to teach a computer to solve problems where the outcome is deterministic, for example, a training set 108 is used to train a trained machine learning model 114 to predict the likelihood of a hypoglycemic event for a given patient or group of patients. In contrast, in unsupervised learning, inputs are provided but no corresponding desired output is provided. Unsupervised learning is generally used for classification problems such as customer segmentation (e.g., segmenting patients into different groups based on characteristics associated with hypoglycemic events). Reinforcement learning describes algorithms in which a machine makes decisions using trial and error. Feedback informs the machine when a good or bad choice is made. The machine then adjusts its algorithms accordingly.

[0061] During the training process, different algorithms are used, including among others Generalized Linear Regression (GLM): Poisson GLM is an algorithm used to model discrete counts based on individual inputs.

[0062] To develop a trained machine learning system that can accurately predict the rate of hypoglycemic events, the machine learning model 114 can be trained with non-misleading information. However, patient treatment for diabetes (and other conditions) can be fluid. For example, patients may switch from one type of basal insulin to another. Therefore, in some embodiments, some patients' EMRs 102 (and therefore the corresponding training data 108) are excluded from the training data.

[0063] FIG. 2 is a flow chart illustrating an example process for classifying an event as ED / outpatient (outcome 210), inpatient (secondary) (outcome 212), or inpatient (primary) (outcome 214).

[0064] An event is defined as a primary hospital admission (outcome 214) if it meets all of the following criteria: 1. If the event is linked to a visit table and the visit type is not ED (step 202). 2. Hypoglycemic events are not only identified by natural language processing (step 204). 3. The event is found using a diagnostic table (step 206). 4. Diagnosis is marked as "discharge diagnosis," "admission diagnosis," or "present on admission."

[0065] An event is defined as a secondary hospitalization if it meets all of the following criteria: 1. Hypoglycemic events are not only identified by NLP (step 204). 2. If the event was found using the diagnosis table, the diagnosis is not marked as a "discharge diagnosis," "admission diagnosis," or "present on admission" (step 206). 3. The event is linked to the visit table using the PTID, the visit type is inpatient, and the hypoglycemic event date is between the visit start date minus 1 day and the visit end date plus 1 day (this provides a buffer for linking the hypoglycemic event to the visit given that there is often no exact date match in the dataset) (step 208).

[0066] In some embodiments, as noted above, secondary inpatient events are excluded because patients are often switched to a different basal insulin and their dose is changed during their inpatient stay, so any hypoglycemic events during this period cannot be attributed to the patient's regular insulin.

[0067] For example, an event is defined as an outpatient / ED (outcome 210) if any of the following is met: 1. The event is linked to the visit table using the PTID, the visit type is ED, and the hypoglycemic event date is between the visit start date minus 1 day and the visit end date plus 1 day (step 202).

[0068] 2. Events are identified using only natural language processing (step 204).

[0069] 3. If an event is identified using the diagnosis table, it is not identified using NLP alone, the diagnosis is not marked as a discharge diagnosis, an admission diagnosis, or present at the time of admission, and no linking to the visit table is made (step 206).

[0070] Once created, training records are used to train machine learning systems. Different types of machine learning models can be trained.

[0071] For example, the trained learning model is embodied as a generalized linear model. Different types of generalized linear models may be appropriate in various scenarios. A zero-inflated negative binomial GLM (zNBGLM) was used because it calculates the discrete counts of events occurring in a given time period based on the most likely occurrence of the hypoglycemic counts seen in the data. This is because the zNBGLM models the hypoglycemic event rate per patient by estimating the hypoglycemic event rate per patient. One disadvantage of the zNBGLM is that the model allows too many degrees of freedom and tends to overfit the data to small segments, thereby reducing generalization performance. Another type of generalized linear model is the Poisson GLM. Poisson GLMs are well suited to modeling discrete counts but do not allow "excessive dispersion" (i.e., they constrain the variance to be equal to the mean). In the Poisson GLM, the number of hypoglycemic events was used as the target variable (outcome) and the observation length as the offset variable.

[0072] In another example, the trained learning model is embodied as an artificial neural network. An artificial neural network (ANN), or connectionist system, is a computing system inspired by the biological neural networks that make up animal brains. An ANN is based on a collection of connected units or nodes called artificial neurons. Each connection can transmit a signal from one artificial neuron to another, like a synapse in a biological brain. The receiving artificial neuron can process the signal and then send the signal to further artificial neurons connected to it.

[0073] In a typical ANN implementation, the signals at the connections between artificial neurons are real, and the output of each artificial neuron is calculated by some nonlinear function of the sum of its inputs. The connections between artificial neurons are called "edges." Artificial neurons and edges may have weights that are adjusted as learning progresses (e.g., each input to an artificial neuron is weighted separately). The weights increase or decrease the strength of the signal at the connection. Artificial neurons may have a threshold so that a signal is sent only if the aggregate signal crosses the threshold. The transfer function along the edge typically has an S-shape but can also take the form of other nonlinear functions, piecewise linear functions, or step functions. Artificial neurons are typically assembled into multiple layers. Different layers can perform different types of transformations on their inputs. A signal travels from the first layer (input layer) to the last layer (output layer), possibly after traversing each layer multiple times.

[0074] In some embodiments, a machine learning system is used to identify hypoglycemic event rates and hypoglycemic costs. Lasso regression (LASSO) regularization is used to select variables. To validate the models, models are developed ("trained") on 80% of each treatment-specific cohort (referred to as the "training set"). 10-fold cross-validation is used to inform model selection and model parameter optimization. The models are then validated on the remaining 20% ​​of each treatment-specific cohort (internal validation). Bootstrapping is used to assess the variability of model estimates (i.e., to generate confidence intervals).

[0075] Once a machine learning system is trained, it is used to identify patients who are likely to have fewer hypoglycemic events when treated with one type of basal insulin compared to another. For example, models may be trained for each type of basal insulin and for the number of severe and non-severe hypoglycemic events. Each model is then applied to the entire basal insulin-treated population to obtain insulin-specific hypoglycemia rate predictions (i.e., estimates of the hypoglycemia rate in the total population if all patients were using a particular basal insulin).

[0076] The system can then compare the patient's hypoglycemia rate based on other variables.

[0077] FIG. 3 shows an example of using a trained machine learning model. A patient EMR 310 is processed. The inputs 312 are provided to each of the trained machine learning models, in this example, a Gla-300 trained machine learning model, a Gla-100 trained machine learning model 304, an IDet trained machine learning model 306, and an IDeg trained machine learning model 308. Each model can generate an output. For example, the Gla-300 trained machine learning model 302 generates a Gla-300 output 314, the Gla-100 trained machine learning model 304 generates a Gla-100 output 316, the IDet trained machine learning model 306 generates an IDet output 318, and the IDeg trained machine learning model 308 generates an IDeg output 320.

[0078] In some embodiments, as described above, two trained machine learning models are generated for each type of basal insulin: a first trained machine learning model is trained to determine the expected rate of severe hypoglycemic events, and a second trained machine learning model is trained to determine the expected rate of non-severe hypoglycemic events.

[0079] The results of processing these covariates using a machine learning system are analyzed to identify correlations between different covariates and different rates of hypoglycemia for different types of basal insulin, for example, a linear regression model is used to identify correlations between different covariates and the outcomes predicted by the model.

[0080] In some embodiments, the subject's covariates are also determined by a machine learning system. For example, the machine learning is trained to cluster individuals based on, for example, the rate and severity of hypoglycemic events across multiple variables. In this way, the machine learning system can identify covariates that may otherwise go unnoticed.

[0081] In some embodiments, the EMR and training dataset are used to build a cost model for predicting the cost of hypoglycemic events in the T2DM population. In some embodiments, the dataset used for cost-of-treatment modeling included all hypoglycemic events in the EHR for T2DM patients who were at least 18 years old at the time of the event and had linked claims data. Severe hypoglycemic events with a cost of $0 were excluded. The study period and covariates were the same as those described for the hypoglycemic prediction model, except where covariates could not be created due to data limitations.

[0082] Gradient-boosted trees, which use prediction errors from previous decision trees to improve the performance of subsequent trees, have previously been successful in cost prediction; gradient-boosted trees allow for capturing the nonlinear, complex relationships underlying hypoglycemic costs. The cost estimator was applied to subgroups identified as drivers of hypoglycemic differential rates, and the cost per hypoglycemic event was estimated for each subgroup. When key defining variables were missing for a subgroup due to data limitations, the total model cost estimate for one hypoglycemic event was used for that subgroup. Subgroup-level cost savings were calculated by applying the hypoglycemic event subgroup-specific cost estimates to the delta hypoglycemic event rate between the comparator and reference BI.

[0083] 4 illustrates an example of determining hypoglycemia rates for various covariates. Input data 402 provided to a machine learning system and corresponding output data 404 from the machine learning system are provided to a statistical analysis system 406. The statistical analysis system can identify correlations and relationships between different variables in the output data using various statistical techniques. The correlations and relationships are presented as a report 408.

[0084] Another application of trained machine learning models involves identifying the appropriate type of basal insulin for a particular patient. For example, a trained machine learning system can analyze a patient's medical records to determine the appropriate type of basal insulin. can be obtained and entered.

[0085] The medical records were provided to each of the trained machine learning models, which were then used to predict the number of severe or non-severe hypoglycemic incidents (or the probability of a hypoglycemic incident occurring), as described above.

[0086] The system can suggest a basal insulin based on the results of the model. In some embodiments, the system can suggest a basal insulin that is associated with a lower risk of severe hypoglycemic events (or non-severe hypoglycemic events). In some embodiments, the system can determine cost savings from using a particular basal insulin regime compared to another associated with hypoglycemic events and suggest solutions to reduce hypoglycemia-related costs. In some embodiments, the system can suggest a basal insulin that reduces hypoglycemic event rates, but if two different basal insulins produce results that are within a threshold (e.g., within 1%, 5%, or 10% of effect), the system can suggest the basal insulin that is less costly.

[0087] In another embodiment, the trained machine learning model is used to identify patients who are more likely to experience a hypoglycemic event. For example, the system can access a patient's medical records, identify the type of basal insulin the patient is using, and process the medical records through the corresponding trained machine learning model. The trained machine learning model generates an indication of the likelihood or expected frequency and / or severity of a hypoglycemic event. If the indication exceeds a threshold (e.g., more than one event per week or more than a 20% likelihood of a severe hypoglycemic event), the patient and / or the patient's physician are notified.

[0088] 5 is a flowchart of an example process 500 for generating a trained machine learning model using patient data. Process 500 is performed by one or more computer systems, such as those described below.

[0089] At 502, the process 500 receives data indicating medical records of patients diagnosed with diabetes mellitus.

[0090] At 504, process 500 determines a predicted rate of hypoglycemic events using a machine learning system trained with data representing the medical records of multiple patients and the corresponding rate of hypoglycemic events for each patient. In some embodiments, multiple patients use the same type of basal insulin.

[0091] At 506, the process 500 generates a predicted rate for the patient.

[0092] In some embodiments, process 500 can include determining a second expected rate of hypoglycemic events using a second machine learning system, the second machine trained with data indicative of medical records of a second plurality of patients and corresponding rates of hypoglycemic events for each second patient, each of the second plurality of patients using a second type of basal insulin, the second type of basal insulin being different from the first type of basal insulin; and process 500 can further include comparing the first expected rate to the second expected rate.

[0093] In some embodiments, the process 500 may include recommending a basal insulin for the patient based on the comparison.

[0094] In some embodiments, the process 500 performs a second plurality of patient medical records. determining a plurality of predicted rates of hypoglycemic events for the second plurality of patients by providing corresponding data to a machine learning system; identifying one or more covariates in the data that are correlated to the predicted rates of hypoglycemic events based on the data and the plurality of predicted rates of hypoglycemic events; and generating a report identifying the one or more covariates and the corresponding predicted rates of hypoglycemic events.

[0095] In some embodiments, process 500 includes determining a plurality of expected rates of hypoglycemic events for a second plurality of patients by providing data corresponding to medical records of each of a second plurality of patients to a machine learning system, each of the second plurality of patients having the same covariates, and process 500 may further include generating a report identifying the covariates and the corresponding expected rates of hypoglycemic events.

[0096] Embodiments and functional operations of the subject matter described herein are implemented as digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed herein and structural equivalents thereof), or as a combination of one or more of these. Embodiments of the subject matter described herein are implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of a data processing apparatus). A computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of these.

[0097] The term "data processing apparatus" refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, or multiple processors or computers. An apparatus can also be or otherwise include special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). In addition to hardware, an apparatus can sometimes include code that creates an execution environment for a computer program (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of these).

[0098] A computer program, which may be called or written as a program, software, software application, module, software module, script, or code, may be written in any programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be arranged in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., a markup language document, a single file dedicated to the program in question, or one or more scripts stored in multiple cooperating files (e.g., files storing one or more modules, subprograms, or portions of code)). A computer program may be arranged to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a data communications network.

[0099] The processes and logic flows described herein operate on input data and generate output. The present invention may be implemented by one or more programmable computers executing one or more computer programs to perform functions by the one or more programmable computers. The processing and logic flows may also be performed by, and apparatus may be implemented as, special purpose logic circuitry (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)).

[0100] A computer suitable for running a computer program can be based on a general-purpose or special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operatively connected to receive and / or transfer data from the mass storage devices, although a computer need not have such devices. Furthermore, a computer may be embedded in another device (e.g., a mobile phone, a digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial bus (USB) flash drive), to name just a few).

[0101] Computer-readable media suitable for storing computer program instructions and data include, by way of example only, all forms of non-volatile memory on media and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal or removable), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0102] To provide for interaction with a user, embodiments of the subject matter described herein are implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user, and a keyboard and pointing device (e.g., a mouse or trackball) that allow the user to provide input to the computer. Other types of devices are similarly used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, a computer can interact with a user by sending documents to and receiving documents from one device used by the user, for example, by sending a web page to a web browser on the user's device in response to a request received from that web browser.

[0103] Embodiments of the subject matter described herein are implemented in a computer system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface or web browser that allows a user to interact with an embodiment of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of this system may be implemented in a digital interconnected by any form or medium of data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0104] A computing system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data (e.g., HTML pages) to a user device (e.g., to display the data to and receive user input from a user interacting with the user device), the user device functioning as a client. Data generated at the user device (e.g., a result of user interaction) is received from the user device at the server.

[0105] While this specification contains details of many specific implementations, these should not be construed as limitations on the scope of any invention or on the scope that may be claimed, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Some features that are described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment may also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, while features may be described above as working in certain combinations, and may even be initially claimed as such, one or more features from a claimed combination may, in some cases, be deleted from the combination, and the claimed combination may be subject to subcombinations or variations of subcombinations.

[0106] Similarly, although operations are depicted in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order or sequence shown, or that all of the illustrated operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems are typically integrated into a single software product or packaged in multiple software products.

[0107] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order to achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order, or sequence, shown to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

Claims

1. 1. A method executed on a computing device, comprising: receiving data indicating medical records of patients diagnosed with diabetes mellitus; determining a predicted rate of hypoglycemic events using a machine learning system, the machine being trained with data indicative of medical records of a plurality of patients and a corresponding rate of hypoglycemic events for each patient; generating a predicted rate for the patient; a method executed on the computing device, comprising:

2. 10. The computing device-implemented method of claim 1, wherein each of the multiple patients uses the same type of basal insulin.

3. determining a second predicted rate of hypoglycemic events using a second machine learning system, the second machine being trained with data indicative of medical records of a second plurality of patients and a corresponding rate of hypoglycemic events for each second patient, each of the second plurality of patients using a second type of basal insulin, the second type of basal insulin being different from the first type of basal insulin; comparing the first predicted rate to the second predicted rate; 3. The method executed on the computing device of claim 1 or 2, further comprising:

4. The computing device-implemented method of claim 3 , further comprising recommending a basal insulin for the patient based on the comparison.

5. determining a plurality of predictive rates of hypoglycemic events for the second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system; identifying one or more covariates in the data that are correlated with a predicted rate of hypoglycemic events based on the data and the plurality of predicted rates of hypoglycemic events; generating a report identifying one or more covariates and corresponding predicted rates of hypoglycemic events; The method executed on the computing device of claim 1 further comprising:

6. determining a plurality of predictive rates of hypoglycemic events for a second plurality of patients by providing data corresponding to medical records of each of the second plurality of patients to a machine learning system, wherein each of the second plurality of patients has the same covariates; Generate a report identifying covariates and corresponding predicted rates of hypoglycemic events; The method executed on the computing device of claim 1 further comprising:

7. A non-transitory computer-readable medium encoded with computer program instructions, the instructions, when executed by one or more computers, causing the one or more computers to: receiving data indicating medical records of patients diagnosed with diabetes mellitus; determining a predicted rate of hypoglycemic events using a machine learning system, the machine being trained with data indicative of medical records of a plurality of patients and a corresponding rate of hypoglycemic events for each patient; generating a predicted rate for the patient; The non-transitory computer-readable medium causes the computer to perform operations including:

8. 8. The non-transitory computer-readable medium of claim 7, wherein each of the multiple patients uses the same type of basal insulin.

9. It works like this: determining a second predicted rate of hypoglycemic events using a second machine learning system, the second machine being trained with data indicative of medical records of a second plurality of patients and a corresponding rate of hypoglycemic events for each second patient, each of the second plurality of patients using a second type of basal insulin, the second type of basal insulin being different from the first type of basal insulin; comparing the first predicted rate to the second predicted rate; 9. The non-transitory computer-readable medium of claim 7 or 8, further comprising:

10. The non-transitory computer-readable medium of any one of claims 7 to 9, wherein the operations further include recommending a basal insulin for the patient based on the comparison.

11. It works like this: determining a plurality of predictive rates of hypoglycemic events for the second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system; identifying one or more covariates in the data that are correlated with a predicted rate of hypoglycemic events based on the data and the plurality of predicted rates of hypoglycemic events; generating a report identifying one or more covariates and corresponding predicted rates of hypoglycemic events; The non-transitory computer-readable medium of claim 7 , further comprising:

12. It works like this: determining a plurality of predictive rates of hypoglycemic events for a second plurality of patients by providing data corresponding to medical records of each of the second plurality of patients to a machine learning system, wherein each of the second plurality of patients has the same covariates; Generate a report identifying covariates and corresponding predicted rates of hypoglycemic events; The non-transitory computer-readable medium of claim 7 , further comprising:

13. 1. A system comprising: a computer-implemented method and system for implementing a method of ... receiving data indicating medical records of patients diagnosed with diabetes mellitus; determining a predicted rate of hypoglycemic events using a machine learning system, the machine being trained with data indicative of medical records of a plurality of patients and a corresponding rate of hypoglycemic events for each patient; generating a predicted rate for the patient; The system is operable to cause the system to perform operations including:

14. 14. The system of claim 13, wherein each of the multiple patients uses the same type of basal insulin.

15. It works like this: determining a plurality of predictive rates of hypoglycemic events for the second plurality of patients by providing data corresponding to the medical records of each of the second plurality of patients to a machine learning system; identifying one or more covariates in the data that are correlated with a predicted rate of hypoglycemic events based on the data and the plurality of predicted rates of hypoglycemic events; generating a report identifying one or more covariates and corresponding predicted rates of hypoglycemic events; The system of claim 13 further comprising:

Images