Predicting hypoglycemia ratio by machine learning system

Abstract

Systems, methods, and computer products may predict a hypoglycemia ratio of a patient. A method includes receiving data representative of a medical record of a patient that has been diagnosed with diabetes. The method includes determining predicted ratios of hypoglycemic events using a machine learning system trained using data representative of medical records of a plurality of patients and respective ratios of hypoglycemic events for respective patients. The method further includes generating a predicted ratio of the patient.

Description

[0001] This application is a divisional application of the Chinese patent application with application number 201980053031.4 (application date: June 21, 2019, invention name: Predicting hypoglycemia rate through machine learning system).

[0002] Priority claim

[0003] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 689,005, filed on June 22, 2018, the entire contents of which are incorporated herein by reference. Background Art

[0004] Machine learning is a subset of artificial intelligence within the field of computer science that typically uses statistical techniques to enable computers to "learn" (i.e., gradually improve their performance at a specific task) from data without being explicitly programmed. Summary of the invention

[0005] In general, one innovative aspect of the subject matter described in this specification can be embodied in a method comprising an act of receiving data representing a medical record of a patient, the patient having been diagnosed with diabetes. The method comprises an act of determining a predicted rate of hypoglycemic events using a machine learning system, the machine being trained using data representing medical records of a plurality of patients and data of corresponding rates of hypoglycemic events for each patient. The method also comprises an act of generating a predicted rate of hypoglycemic events for the patient.

[0006] The foregoing embodiments and other embodiments may each optionally include one or more of the following features, individually or in combination. Each of the plurality of patients may use the same type of basal insulin. The method may include the following actions: using a second machine learning system to determine a second predicted rate of hypoglycemic events, the second machine uses data representing the medical records of a second plurality of patients and the corresponding rate of hypoglycemic events of each second patient for training, wherein each of the second plurality of patients uses a second type of basal insulin, the second type of basal insulin is different from the first type of basal insulin, and compares the first predicted rate and the second predicted rate. The method may include the action of recommending basal insulin to the patient based on the comparison. The method may include the following actions: by providing the machine learning system with data corresponding to the medical records of each of the second plurality of patients, determining multiple predicted rates of hypoglycemic events of the second plurality of patients, based on the data and the multiple predicted rates of hypoglycemic events, identifying one or more covariates in the data related to the predicted rate of hypoglycemic events, and generating a report identifying one or more covariates and the corresponding predicted rate of hypoglycemic events. The method may include the following actions: determining multiple predicted rates of hypoglycemic events for the second plurality of patients by providing data corresponding to medical records of each of a second plurality of patients to a machine learning system, each of the second plurality of patients having the same covariates, and generating a report identifying the covariates and corresponding predicted rates of hypoglycemic events.

[0007] The present disclosure also provides a computer-readable storage medium coupled to one or more processors and having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations through implementation of the methods provided herein.

[0008] The present disclosure further provides a system for implementing the method provided herein. The system includes one or more processors, and a computer-readable storage medium coupled to the one or more processors and having stored thereon, wherein the instructions, when executed by the one or more processors, cause the one or more processors to perform operations by implementing the method provided herein.

[0009] It is understood that the embodiments of the present disclosure may include any combination of aspects and features described herein. That is, the embodiments of the present disclosure are not limited to the combinations of aspects and features specifically described herein, but also include any other suitable combinations of aspects and features provided.

[0010] The details of one or more embodiments of the subject matter described in this specification are outlined in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter described will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 Illustrate an environment in which a machine learning model is trained to predict the expected rate of hypoglycemic events.

[0012] Figure 2 is a flow chart illustrating an example of a process for classifying an event as ED / outpatient, inpatient (secondary), or inpatient (primary).

[0013] Figure 3 An example procedure illustrating various covariates used to determine hypoglycemia rates.

[0014] Figure 4 Examples illustrating various covariates in determining hypoglycemia rates.

[0015] Figure 5 is a flow chart of an example of a process for generating a trained machine learning model using patient data.

[0016] Like reference numerals and designations throughout the various drawings represent like elements. DETAILED DESCRIPTION OF THE INVENTION

[0018] Diabetes is the seventh leading cause of death and the leading cause of morbidity in the United States. It affects an estimated 29.1 million Americans, with 1.4 million newly diagnosed each year. By 2030, the number of affected patients is expected to increase to more than 54.9 million. In 2012, the total cost of diagnosed diabetes in the United States was $245 billion ($176 billion in direct medical costs and $69 billion in lost productivity).

[0019] Due to the growing burden of disease, complications associated with diabetes are becoming increasingly important for effective prevention and treatment. Hypoglycemia is a frequent and potentially fatal adverse effect of insulin and oral glucose-lowering drugs (OADs) in patients with diabetes. In addition to the risk of direct hypoglycemia, recurrent episodes lead to fear of future episodes and have been shown to constitute both patient-led and physician-led barriers to optimal glycemic control. Resulting elevated hemoglobin A1c (HbA1c) levels are associated with an increased risk of microvascular (and potentially macrovascular) complications.

[0020] It is estimated that people with type 1 diabetes (T1DM) experience an average of two mild hypoglycemic events per week and an average of one severe event per year. However, event rates from randomized clinical trials of patients with type 1, expressed per patient-year, are 0.15 for severe events and 88.3 for non-severe events. Event rates for type 2 diabetes (T2DM) vary more between studies, ranging from 0.05 to 26.6 events per patient-year for severe and non-severe events, respectively. However, research suggests that these studies significantly underestimate the true rates of hypoglycemia in the real world, particularly for severe events.

[0021] The average cost of a hypoglycemic event is also heterogeneous between studies and difficult to pool due to differences in hypoglycemia definitions and cost estimation methods. Current estimates of the average cost of an inpatient hypoglycemic event range from $2205 to $17,564. The average outpatient cost per hypoglycemic event ranges from $148 to $501.

[0022] Different basal insulins have shown different hypoglycemia rates. For example, multiple studies have shown that insulin glargine 100 units / mL (Lantus) is superior to protamine insulin with regard to severe hypoglycemic events. Some basal insulins, such as insulin glargine 300 units / mL (Toujeo), show flatter and longer pharmacokinetic and pharmacodynamic profiles than other drugs and can provide sustained blood sugar control for more than 24 hours.

[0023] Therefore, the burden of hypoglycemia in the United States is enormous, and significant benefits could be achieved by determining the appropriate type of basal insulin for each patient. Estimating cost savings to payers by reducing the resulting medical costs for these patients could help guide payer decision making and drug pricing negotiations.

[0024] Figure 1 An environment 100 is illustrated in which a machine learning model can be trained to predict a predicted rate of hypoglycemic events. The system described herein uses electronic medical record data (EMR) 102 (which can be obtained from multiple different sources, including but not limited to hospital records and physician records) to train a machine learning system. In some embodiments, the EMR may include information about demographic and socioeconomic categories, coded diagnoses and procedures, prescribed and administered medications, laboratory results, and clinical management data. In some embodiments, the EMR may be processed by a processor 104. For example, the EMR may include structured and unstructured data. Structured data may contain information such as visit dates, patient names, etc. Unstructured data may include free-form text added by a doctor (e.g., doctor's notes, visit summaries, etc.). The EMR processor may use technology such as natural language processing to extract facts from unstructured data in the EMR and convert it into structured EMR data 106.

[0025] In some embodiments, the EMR processor 104 may filter some medical records. For example, the EMR processor 104 may select medical records only for patients that share values, i.e., specific variables, referred to as covariates. Examples of covariates may include, for example, gender, geographic region, ratio, age range, insurance company, years since diagnosis, HbA1c range, body mass index, blood pressure range, diabetic complications, alcohol and / or drug use, and any other physiological or demographic characteristics. The EMR processor 104 may be, for example, one or more computer systems as described below.

[0026] In some embodiments, the same manually created covariates can be used for descriptive analysis, predictive models for hypoglycemia rates, and cost estimation analysis. The predetermined covariates can be based on a professional clinical review of the literature on hypoglycemic events and cost predictors. The predetermined covariates can be defined using ICD-9 and ICD-10 diagnostic codes, laboratory values, drug names, and / or national drug coding (NDC) codes. Unless the covariates are cost-related, EHR data sets (rather than insurance claims) can be used to identify covariates, in which case insurance claims data can be used.

[0027] The default lookback period can use one year prior to the start of treatment, but can be changed depending on the tolerance of the assumed covariates. For example, the lookback period for cancer can be five years, because if a patient was diagnosed with cancer more than five years ago and has not been diagnosed subsequently, it is unlikely that the patient has active cancer at the index date. For covariates whose historical range does not affect their likelihood of being captured in the dataset (such as sex and race), the lookback period can be eight years or can be limited only by the amount of available data. For irreversible and chronic conditions, the maximum lookback period can also be eight years, or can be limited only by the amount of available data.

[0028] Examples of covariate categories and their look-back periods used for the hypoglycemia rate model and cost estimation analyses are as follows:

[0029] 1. Demographics

[0030] a. The look-back period for these covariates is eight years

[0031] 2. Socioeconomics

[0032] b. The look-back period for these covariates is eight years

[0033] 3. Comorbidities

[0034] c. The look-back period for these covariates ranged from 1 year (for reversible / acute conditions) to 8 years (for irreversible / chronic conditions)

[0035] d. The Charlson Comorbidity Index (CCI) score is a unique covariate in the comorbidity category. The CCI represents a measure of the patient's comorbidity profile (including the diabetic complications category) and is correlated with the pre-existing index (including diabetic complications).

[0036] 4. Complications of diabetes

[0037] e. The look-back period for these covariates is eight years

[0038] 5. Diabetes

[0039] a. The look-back period for these covariates was one year for the previous hypoglycemic event in the dataset and eight years for the known duration of diabetes.

[0040] Drug use

[0041] a. The look-back period for these covariates is one year

[0042] Other covariates may be included only as part of the cost estimation analysis (rather than hypoglycemia rate prediction) as these are expected to be drivers of hypoglycemia-related costs (rather than hypoglycemia rates):

[0043] 1. Physician’s expertise:

[0044] f. The look-back period was all available data for the covariate “Physician specialty” and two years for “Physician’s most commonly prescribed insulin”.

[0045] 2. Average cost of previous hypoglycemic events

[0046] g. The look-back period for this covariate is one year

[0047] Another set of covariates may be determined. This set of covariates may not be pre-specified but covers all comorbidities, procedures, and prescriptions (hereinafter collectively referred to as "markers") that occurred within 1 year prior to the patient's index date and may be included in the prediction model.

[0048] In some embodiments, unsupervised covariate creation can be performed using the following method:

[0049] 1. The distance between markers is defined as:

[0050] a. Each label is associated with a vector containing a set of patients labeled as true or false

[0051] b. Then the distance between the markers is the Jaccard distance between the vectors

[0052] 2. Then generate clusters based on these distances:

[0053] a. Perform hierarchical clustering based on the distance matrix to obtain the cluster hierarchy. The average distance between clusters is used for hierarchical linkage clustering

[0054] 3. The hierarchical level of extracted clusters is defined as:

[0055] a. The "Inconsistency" method is used to determine at which hierarchical level we want to extract clusters. "Inconsistency" refers to the inconsistency of the average distances among linked clusters: large values ​​indicate that clusters should not be linked.

[0056] 4. You can choose to cluster many patient treatment "shows":

[0057] a. Of the 1,500 clusters formed, select the 100 clusters that are represented by the majority of patient treatments

[0058] b. A patient treatment is said to “exhibit” a cluster if the patient treatment had any of the diagnoses, procedures, or prescriptions (e.g., markers) that constitute the cluster within the year prior to the index date

[0059] 5. Clustered medical rationalization:

[0060] a. Clinical experts then examine the medical logic of the derived clusters, thereby validating the parameters used for unsupervised cluster generation.

[0061] In some implementations, the EMR records may be filtered, such as by the filter 116 , before the training data 108 is generated but after the structured medical record 106 is generated.

[0062] The structured EMR data 106 can be used to generate training records 108 (collectively referred to as training sets). The training records can represent the available data in the structured EMR data. For example, in some embodiments, a portion of the structured EMR records can be used to train the machine learning system, while the remaining records are used to verify the trained machine learning system. In some embodiments, training records can be created at the "patient treatment" level during which the patient is on basal insulin therapy. Therefore, multiple training records can be created for a single patient. The unit of analysis is "patient treatment", which is defined as the time period in which the patient is observed on basal insulin therapy in the data set (the time period between the therapeutic index and the end of treatment observation). Hypoglycemic events are the target endpoint only during the patient's treatment period.

[0063] The therapeutic index date can be defined as any of the following dates: the first prescription for any underlying diabetes; or a change in prescription from one basal insulin to another. Basal insulins at baseline include: Gla-300, Gla-100, IDet, IDeg, and NPH. The index basal insulins of interest for the study are: Gla-300, Gla-100, IDet, and IDeg.

[0064] For purposes of ratio calculations, “duration” may be considered the duration of the patient’s treatment period as previously described minus the duration of all inpatient stays during that period.

[0065] In some embodiments, the index date is the date of the first prescription of a BI or a change in prescription from one basal insulin to another basal insulin. The end of treatment can be defined as the end of the follow-up period in the dataset, a change in prescription from an index basal insulin to another BI, or one year after the therapeutic index date (whichever occurs first).

[0066] In one embodiment, a patient treatment may be excluded from the training data if the patient treatment meets any of the following criteria:

[0067] 1. Treatment with more than one type of basal insulin: that is, the same patient started one treatment within a week of (before or after) starting another treatment.

[0068] 2. Any patient treatment inactivity for more than 270 days in the 365 days prior to the index date (inactivity was defined as the absence of any time-stamped data in the relevant tables of the dataset).

[0069] 3. Treatment of patients whose treatment duration is less than one day.

[0070] Separately, because patients were typically switched to standard basal insulin upon admission according to the hospital provider's formulary, the duration of the inpatient hospital stay was excluded from the duration of the patient's treatment. Therefore, hypoglycemic events during the inpatient hospital stay could not be attributed to the index basal insulin.

[0071] The training record may include information about hypoglycemic events. In some embodiments, hypoglycemic events may be counted during the patient's treatment. The duration of determining the hypoglycemic ratio is considered to be the duration during the patient's treatment minus the duration of all inpatient hospitalizations during that period. Hypoglycemic events may be the expected output of the training set. For example, the training set may train a machine learning system to determine the likelihood of the number of hypoglycemic events expected to occur within a fixed period (e.g., one month, six months, one year, five years, etc.). Alternatively, the training set may train a machine learning system to determine the number of hypoglycemic events expected to occur within a time period (e.g., one month, six months, one year, five years, etc.).

[0072] The hypoglycemic event rate may include both severe and non-severe events. Figure 2is a flow chart of an example of a process for classifying a hypoglycemic event as severe or non-severe. The definition of 'severe' hypoglycemia may include, for example, ICD-9 / 10 codes for severe in nature, administration of intramuscular glucagon. In addition, natural language processing of the EMR may be used to identify hypoglycemia. In terms of severity, any hypoglycemic event that is not severe may be defined as 'non-severe'.

[0073] In some embodiments, an event is defined as hypoglycemia if any of the following criteria are met:

[0074] 1.ICD-9 and 10 diagnostic codes for hypoglycemia

[0075] 2. Laboratory plasma glucose level ≤70 mg / dl

[0076] 3. Administration of Intramuscular Glucagon

[0077] 4. NLP Output

[0078] In some embodiments, a NLP recognized hypoglycemic event is defined as any statement of hypoglycemia, except for negative sentiment or historical signs of episodes. For example, a "statement" of hypoglycemia can be any event that appears in the form of the conventional expression ".*hypoglycemia.*", unless the term happens to be one of "hypoglycemia awareness", "hypoglycemic consciousness disorder" or "neonatal hypoglycemia". "Negative sentiment" can be negative for the mentioned hypoglycemia, indicating, for example, that there are no signs of hypoglycemia. Historical events are records indicating that what is mentioned is a past event (e.g., "the patient has a history of hypoglycemia"). In some embodiments, a list of negative sentiment and historical keywords can be used to exclude statements of accompanying hypoglycemia.

[0079] In some embodiments, a maximum of one hypoglycemic event may be counted per calendar day. For example, if a hypoglycemic event is recorded by multiple care locations or multiple definition criteria, only one event is counted.

[0080] In some embodiments, a hypoglycemic event may be defined as severe if any of the following conditions are met:

[0081] 1. By default, the ICD-9 or 10 diagnosis code for hypoglycemia is severe (ICD-9 249.30; 250.30; 250.31; 251.0; ICD-10 E08.641; E09.641; E10.641; E11.641; E13.641; E15)

[0082] 2. Hypoglycemia ICD code is marked as an admissible diagnosis, the main reason for visit at discharge or admission

[0083] 3. The onset of hypoglycemia occurred on the same day as an emergency department (ED) visit or inpatient admission

[0084] 4. Presence of plasma glucose measurement <54 mg / dl

[0085] 5. Administration of Intramuscular Glucagon

[0086] 6. The NLP statement for hypoglycemia is accompanied by severity descriptors, including severity terms (e.g., “severe”) and attributes (e.g., “urgent”)

[0087] 7. NLP hypoglycemia event occurs on the same day as an emergency department visit or inpatient admission

[0088] The machine learning environment 110 may include a machine learning trainer 112. The machine learning trainer 112 may train a machine learning model 114 to predict predicted rates of hypoglycemic events for different patients. The trained machine learning system may be used in a variety of different ways, including identifying the most appropriate basal insulin for a patient associated with the patient's hypoglycemic outcomes, thereby improving patient health.

[0089] Typically, machine learning can include a variety of different techniques that are used to train machines to perform specific tasks without being specifically programmed to perform those tasks. Machines can be trained using different machine learning techniques, such as supervised learning, unsupervised learning, and reinforcement learning. In supervised learning, the machine is provided with an input of interest and a corresponding output. The machine adjusts its function to provide the desired output when the input is provided. Supervised learning is typically used to teach computers to solve problems determined by results, for example, a training set 108 can be used to train a trained machine learning model 114 to predict the likelihood of hypoglycemic events for a given patient or patient group. On the contrary, input is provided without providing a corresponding desired output in an unsupervised situation. Unsupervised learning is typically used for classification problems, such as customer segmentation (e.g., dividing patients into different groups based on features associated with hypoglycemic events). Reinforcement learning describes an algorithm that allows a machine to make decisions by trial and error. When a good choice or a bad choice is made, feedback notifies the machine. The machine then adjusts its algorithm accordingly.

[0090] During the training process, different algorithms can be used, including Generalized Linear Regression (GLM). Poisson GLM is an algorithm for modeling discrete counts based on independent inputs.

[0091] In order to develop a trained machine learning system that can accurately predict the rate of hypoglycemic events, the machine learning model 114 can be trained with information that is not misleading. However, diabetic patient treatment (and other medical conditions) can be unstable. For example, a patient can be switched from one basal insulin to another. Therefore, in some embodiments, some patients' EMRs 102 (and therefore corresponding training data 108) can be excluded from the training data.

[0092] Figure 2 is a flow chart illustrating an example of a process for classifying an event as ED / outpatient (outcome 210), inpatient (secondary) (outcome 212), or inpatient (primary) (outcome 214).

[0093] An event was defined as a major hospitalization (outcome 214) if it met all of the following criteria:

[0094] 1. If the event is linkable to the access table and the access type is not ED (step 202)

[0095] 2. Hypoglycemic events can be identified not only through natural language processing (step 304)

[0096] 3. Events can be found using the diagnostic table (step 206)

[0097] 4. Diagnosis marked as "discharge diagnosis", "admission diagnosis" or "present at admission"

[0098] An event was defined as a secondary hospitalization if it met all of the following criteria:

[0099] 1. Hypoglycemia events can be identified not only through NLP (step 204)

[0100] 2. If the event can be found using the diagnosis table, the diagnosis is not marked as "discharge diagnosis", "admission diagnosis" or "present at admission" (step 206)

[0101] 3. The event can be linked to the visit table using the PTID, the visit type is inpatient, and the hypoglycemia event date is between the visit start date minus 1 day to the visit end date plus 1 day (this allows for a buffer to link the hypoglycemia event to a given visit, where there is often no exact date to match in the data set) (step 208)

[0102] In some embodiments, as described above, secondary hospitalization events can be excluded because patients are often switched to different basal insulins and their doses are changed during an inpatient hospital stay. Therefore, any hypoglycemic events during this period cannot be attributed to the patient's usual insulin.

[0103] For example, an event may be defined as an outpatient / ED visit (result 210) if it meets any of the following criteria:

[0104] 1. The event can be linked to the visit table using PTID, the visit type is ED, and the hypoglycemia event date is between the visit start date minus 1 day and the visit end date plus 1 day (step 202)

[0105] 2. Events are identified using natural language processing only (step 204)

[0106] 3. The event is not identified using only NLP, if it can be identified using the diagnosis table, the diagnosis is not marked as a discharge diagnosis, admission diagnosis or present at the time of admission, and cannot be linked to the access table (step 206)

[0107] Once a training record is created, it can be used to train a machine learning system. Different types of machine learning models can be trained.

[0108] For example, the trained learning model can be implemented as a generalized linear model. In various cases, different types of generalized linear models may be appropriate. The zero-inflated negative binomial GLM (zNBGLM) is used because it simulates the discrete event counts that occur in a given time period by estimating the ratio of hypoglycemic events per patient that is most likely to cause the hypoglycemic counts that appear in the data. One disadvantage of the zNBGLM is that the model allows a very large number of degrees of freedom and tends to overfit the data of smaller segments, thereby reducing generalization performance. Another type of generalized linear model is the Poisson GLM. The Poisson GLM is very suitable for modeling discrete counts, but does not allow 'overdispersion' (i.e., constraining the variance to be equal to the mean). The Poisson GLM uses the number of hypoglycemic events as the target variable (outcome) and the length of the observation as the offset variable.

[0109] In another example, the trained learning model can be implemented as an artificial neural network. An artificial neural network (ANN) or connection system is a computing system inspired by the biological neural networks that make up the animal brain. ANN is based on a collection of connected units or nodes called artificial. Each connection can transmit a signal from one artificial neuron to another artificial neuron, just like a synapse in a biological brain. The artificial neuron that receives the signal can process it and then send a signal to other artificial neurons connected to it.

[0110] In a common ANN implementation, the signals at the connections between artificial neurons are real numbers, and the output of each artificial neuron is calculated by some nonlinear function of the sum of its inputs. The connections between artificial neurons are called 'edges'. The weights of artificial neurons and edges can be adjusted as learning proceeds (for example, each input to an artificial neuron can be weighted separately). The weights increase or decrease the signal strength at the connection. Artificial neurons can have thresholds so that signals are sent only when the aggregate signal exceeds the threshold. The transfer functions along the edges typically have an S-shape, but they can also take the form of other nonlinear functions, piecewise linear functions, or step functions. Typically, artificial neurons are clustered into layers. Different layers can perform different kinds of transformations on their inputs. The signal can propagate from the first layer (input layer) to the last layer (output layer), possibly after traversing these layers multiple times.

[0111] In some embodiments, the machine learning system can be used to identify hypoglycemic event rates and hypoglycemic costs. Least absolute shrinkage and selection operator (LASSO) regularization can be used to select variables. In order to verify the model, the model can be developed (trained) on 80% of each treatment-specific cohort (referred to as "training set"). Ten-fold cross validation can be used to inform model selection and model parameter optimization. The model can then be verified on the remaining 20% ​​of each treatment-specific cohort (internal validation). Bootstrapping can be used to assess the variability of model estimates (i.e., generate confidence intervals).

[0112] Once the machine learning system is trained, the system can be used to identify patients who may experience a smaller number of hypoglycemic events with one basal insulin treatment compared to another basal insulin treatment. For example, once the model is trained for each type of basal insulin and the number of severe and non-severe hypoglycemic events. Each model can then be applied to the entire basal insulin treatment population to obtain insulin-specific hypoglycemia ratio predictions (i.e., if all patients use a specific basal insulin, an estimate of the hypoglycemia ratio in the total population).

[0113] The system can then compare the patients' hypoglycemia rates based on the different variables.

[0114] Figure 3An example of using a trained machine learning model is illustrated. A patient EMR 310 can be processed to generate an input 312. The input 312 can be provided to each trained machine learning model, in this example a Gla-300 trained machine learning model, a Gla-100 trained machine learning model 304, an IDet trained machine learning model 306, and an IDeg trained machine learning model 308. Each model can generate an output. For example, the Gla-300 trained machine learning model 302 generates a Gla-300 output 314, the Gla-100 trained machine learning model 304 generates a Gla-100 output 316, the IDet trained machine learning model 306 generates an IDet output 318, and the IDeg trained machine learning model 308 generates an IDeg output 320.

[0115] In some embodiments, as described above, two trained machine learning models can be generated for each type of basal insulin. The first trained machine learning model can be trained to determine the predicted rate of severe hypoglycemic events. The second trained machine learning model can be trained to determine the predicted rate of non-severe hypoglycemic events.

[0116] The results of processing these covariates using a machine learning system can be analyzed to identify correlations between different covariates and different hypoglycemia rates for different types of basal insulin. For example, a linear regression model can be used to identify correlations between different covariates and the outcomes predicted by the model.

[0117] In some embodiments, covariates of interest may also be determined by a machine learning system. For example, a learning machine may be trained to cluster individuals based on, for example, the rate and severity of hypoglycemic events across multiple variables. In this way, a machine learning system may identify covariates that may not otherwise be noticed.

[0118] In some embodiments, a cost model can be established using EMR and training data sets to predict the cost of hypoglycemic events in a T2DM population. In some embodiments, the data set used for treatment cost modeling includes all hypoglycemic events in the EHR of T2DM patients who are at least 18 years old and have linked to the required data when the event occurs. Severe hypoglycemic events with a cost of $0 are excluded. Except when covariates cannot be created due to data limitations, the study period and covariates are the same as those described for the hypoglycemic prediction model.

[0119] Gradient boosted trees (using the prediction errors of previous decision trees to improve the performance of subsequent trees) can be used for cost estimation, which has been successful in previous cost prediction; gradient boosted trees allow the capture of nonlinear, complex relationships behind hypoglycemia costs. The cost estimator is applied to subgroups identified as drivers of differential hypoglycemia rates to estimate the cost of each hypoglycemic event for each subgroup. When a subgroup lacks key defining variables due to data limitations, the overall model cost estimate for hypoglycemic events is used for the subgroup. Cost savings at the subgroup level are calculated by applying subgroup-specific cost estimates for hypoglycemic events to the delta hypoglycemic event ratio between the comparator and reference BIs.

[0120] Figure 4 An example of determining hypoglycemia rates for various covariates is illustrated. As described above, input data 402 provided to the machine learning system and corresponding output data 404 from the machine learning system may be provided to a statistical analysis system 406. The statistical analysis system may use different statistical techniques to identify correlations and relationships between different variables in the output data. The correlations and relationships may be presented as a report 408.

[0121] Another use of a trained machine learning model includes identifying the type of basal insulin that is appropriate for a particular patient. For example, a trained machine learning system may access a patient's medical records and input.

[0122] Medical records can be provided to each trained machine learning model.As described above, each model has been used to predict the number of severe or non-severe hypoglycemic events (or the probability that a hypoglycemic event will occur).

[0123] The system may propose a basal insulin based on the results from the model. In some embodiments, the system may propose a basal insulin that reduces the risk of severe hypoglycemic events (or non-severe hypoglycemic events). In some embodiments, the system may determine the cost savings of using a specific basal blood glucose regimen compared to other regimens associated with hypoglycemic events, and propose solutions to reduce costs associated with hypoglycemia. In some embodiments, the system may propose a basal insulin that reduces the rate of glycemic events, but if the results produced by two different basal insulins are within a threshold (e.g., the efficacy is within 1%, 5%, or 10%), the system may propose a cheaper basal insulin.

[0124] In another embodiment, the trained learning machine model can be used to identify patients who are more likely to have hypoglycemic events. For example, the system can access the patient's medical records, identify the type of basal insulin the patient is using, and process the medical records through the corresponding trained machine learning model. The trained machine learning model generates an indication of the likelihood or expected frequency and / or severity of a hypoglycemic event. If the indication exceeds a threshold (e.g., more than 1 event per week, or a 20% likelihood of a severe hypoglycemic event), the patient and / or the patient's physician can be notified.

[0125] Figure 5 is a flow chart of an example of a process 500 for generating a trained machine learning model using patient data. As described below, process 500 can be performed by one or more computer systems.

[0126] Process 500 receives data 502 representing a medical record of a patient who has been diagnosed with diabetes.

[0127] Process 500 uses a machine learning system to determine a predicted rate of hypoglycemic events 504, training the machine using data representing the medical records of multiple patients and the corresponding hypoglycemic event rates of each patient. In some embodiments, multiple patients are taught to use the same type of basal insulin.

[0128] Process 500 generates 506 a predicted rate for the patient.

[0129] In some embodiments, process 500 may include determining a second predicted rate of hypoglycemic events using a second machine learning system, the second machine being trained using data representing medical records of a second plurality of patients and corresponding rates of hypoglycemic events for each of the second patients, wherein each of the second plurality of patients uses a second type of basal insulin that is different from the first type of basal insulin; and comparing the first predicted rate to the second predicted rate.

[0130] In some embodiments, process 500 can include recommending basal insulin for the patient based on the comparison.

[0131] In some embodiments, process 500 may include determining a predicted rate of multiple hypoglycemic events for the second plurality of patients by providing data corresponding to a medical record of each patient in a second plurality of patients to a machine learning system, identifying one or more covariates in the data that are associated with the predicted rate of hypoglycemic events based on the data and the predicted rate of multiple hypoglycemic events, and generating a report identifying the one or more covariates and the corresponding predicted rate of hypoglycemic events.

[0132] In some embodiments, process 500 may include determining a predicted rate of multiple hypoglycemic events for a second plurality of patients by providing data corresponding to a medical record of each patient in a second plurality of patients to a machine learning system, each patient in the second plurality of patients having the same covariates, and generating a report identifying the covariates and corresponding predicted rates of hypoglycemic events.

[0133] The subject matter and implementation of the functional operations described in this specification may be implemented in digital electronic circuits, tangibly implemented computer software or firmware, computer hardware (including the structures disclosed in this specification and their structural equivalents), or a combination of one or more of them. The implementation of the subject matter described in this specification may be implemented as one or more computer programs (i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier to execute or control the operation of a data processing device through the operation of a data processing device). A computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access storage device, or a combination of one or more of them.

[0134] The term "data processing apparatus" refers to data processing hardware and includes various devices, equipment and machines for processing data, including, for example, a programmable processor, a computer or multiple processors or computers. The apparatus may also be or further include a dedicated logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit)). In addition to hardware, the apparatus may optionally include code that creates an execution environment for a computer program (e.g., code constituting processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them).

[0135] A computer program, which may also be referred to or described as a program, software, software application, module, software module, script or code, may be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document, a single file dedicated to the program in question, or multiple coordinated files (e.g., files storing one or more modules, subroutines or portions of code)). A computer program may be deployed so that the program is executed on one computer, or on multiple computers located at one site or distributed at multiple sites and interconnected by a data communications network.

[0136] The processes and logic flows described in this specification may be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be performed by, and the apparatus may be implemented as, a special purpose logic circuit (e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit)).

[0137] A computer suitable for executing a computer program may be based on a general purpose microprocessor or a special purpose microprocessor or both a general purpose microprocessor and a special purpose microprocessor, or on any other type of central processing unit. Typically, the central processing unit will receive instructions and data from a read-only memory or a random access memory or both a read-only memory and a random access memory. The basic elements of a computer are a central processing unit for performing or executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data; or the computer is operationally coupled to receive data from one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or to transfer data to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or both, however, the computer does not necessarily have such a device. In addition, the computer may be embedded in another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device) (e.g., a universal serial bus (USB) flash drive), to name a few).

[0138] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory on the media and storage devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0139] To provide interaction with a user, embodiments of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with a user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user may be received in any form, including voice, speech, or tactile input. In addition, a computer may interact with a user by sending documents to and receiving documents from a device used by a user, such as by sending a web page to a web browser on a user's device in response to a request received from the web browser.

[0140] Embodiments of the subject matter described in this specification may be implemented in a computing system that includes a back-end component (e.g., as a data server), or includes a middleware component (e.g., an application server), or includes a front-end component (e.g., a client computer with a graphical user interface or a web browser through which a user can interact with an embodiment of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. System components may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs) (e.g., the Internet).

[0141] The computing system may include a client and a server. The client and the server are usually remote relative to each other and usually interact through a communication network. The relationship between the client and the server is generated by a computer program that runs on each computer and has a client-server relationship with each other. In some embodiments, the server sends data (e.g., an HTML page) to a user device that acts as a client (e.g., for the purpose of displaying data to a user interacting with the user device and receiving user input from a user interacting with the user device). Data generated at the user device (e.g., the result of user interaction) can be received from the user device at the server.

[0142] Although this specification contains many specific implementation details, these details should not be interpreted as limitations on the scope of any invention or the scope that can be claimed, but rather as descriptions of features specific to a particular embodiment of a particular invention. Certain features described in the context of separate embodiments in this specification may also be implemented in combination with a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually in multiple embodiments or in any suitable sub-combination. Moreover, although features may be described above as functioning in certain combinations and even initially claimed as such, one or more features in the combination may be removed from the claimed combination in some cases, and the claimed combination may involve a sub-combination or a variation of a sub-combination.

[0143] Similarly, although operations are depicted in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all of the operations shown be performed to achieve a desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0144] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the actions recited in the claims may be performed in a different order and still achieve the desired results. As an example, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some cases, multitasking and parallel processing may be advantageous.

[0145] In summary, the present invention includes but is not limited to the following items:

[0146] 1. A method implemented by a computing device, comprising:

[0147] receiving data representing a medical record of a patient who has been diagnosed with diabetes;

[0148] determining a predicted rate of hypoglycemic events using a machine learning system trained using data representing medical records of a plurality of patients and corresponding rates of hypoglycemic events for the respective patients; and

[0149] A predicted ratio is generated for the patient.

[0150] 2. A method implemented by a computing device according to claim 1, wherein each of the plurality of patients uses the same type of basal insulin.

[0151] 3. The method implemented by the computing device according to item 1 or 2, wherein the method further comprises:

[0152] determining a second predicted rate of hypoglycemic events using a second machine learning system trained using data representing medical records of a second plurality of patients and corresponding rates of hypoglycemic events for each of the second plurality of patients, wherein each of the second plurality of patients uses a second type of basal insulin that is different from the first type of basal insulin; and

[0153] The first predicted ratio is compared to the second predicted ratio.

[0154] 4. The computing device-implemented method of claim 3, further comprising recommending basal insulin for the patient based on the comparison.

[0155] 5. The computing device-implemented method of clause 1, further comprising determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record of each of the second plurality of patients;

[0156] identifying, based on the data and the plurality of predicted rates of hypoglycemic events, one or more covariates in the data that are associated with a predicted rate of hypoglycemic events;

[0157] A report is generated identifying the one or more covariates and the corresponding predicted rates of hypoglycemic events.

[0158] 6. The computing device-implemented method of claim 1, further comprising determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record for each of the second plurality of patients; each of the second plurality of patients having the same covariates;

[0159] A report is generated identifying the covariates and the corresponding predicted rates of hypoglycemic events.

[0160] 7. A non-transitory computer readable medium encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

[0161] receiving data representing a medical record of a patient who has been diagnosed with diabetes;

[0162] determining a predicted rate of hypoglycemic events using a machine learning system trained using data representing medical records of a plurality of patients and corresponding rates of hypoglycemic events for each patient; and

[0163] A predicted ratio is generated for the patient.

[0164] 8. The non-transitory computer readable medium of claim 7, wherein each of the plurality of patients uses the same type of basal insulin.

[0165] 9. The non-transitory computer-readable medium of clause 7 or 8, wherein the operations further comprise:

[0166] determining a second predicted rate of hypoglycemic events using a second machine learning system trained using data representing medical records of a second plurality of patients and corresponding rates of hypoglycemic events for respective second patients, wherein each of the second plurality of patients uses a second type of basal insulin that is different from the first type of basal insulin; and

[0167] The first predicted ratio is compared to the second predicted ratio.

[0168] 10. The non-transitory computer-readable medium of any one of items 7-9, wherein the operation further comprises recommending basal insulin for the patient based on the comparison.

[0169] 11. The non-transitory computer-readable medium of clause 7, wherein the operations further comprise:

[0170] determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record for each of the second plurality of patients;

[0171] identifying, based on the data and the plurality of predicted rates of hypoglycemic events, one or more covariates in the data that are associated with a predicted rate of hypoglycemic events;

[0172] A report is generated identifying the one or more covariates and the corresponding predicted rates of hypoglycemic events.

[0173] 12. The non-transitory computer-readable medium of clause 7, wherein the operations further comprise:

[0174] determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record for each of the second plurality of patients; each of the second plurality of patients having the same covariates;

[0175] A report is generated identifying the covariates and the corresponding predicted rates of hypoglycemic events.

[0176] 13. A system comprising:

[0177] One or more computers and one or more storage devices storing operable instructions, which, when executed by the one or more computers, cause the one or more computers to perform operations including:

[0178] receiving data representing a medical record of a patient, the patient having been diagnosed with diabetes;

[0179] determining a predicted rate of hypoglycemic events using a machine learning system trained using data representing medical records of a plurality of patients and corresponding rates of hypoglycemic events for each patient; and

[0180] A predicted ratio is generated for the patient.

[0181] 14. A system according to claim 13, wherein each of the multiple patients uses the same type of basal insulin.

[0182] 15. The system of clause 13, wherein the operations further comprise:

[0183] determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record for each of the second plurality of patients;

[0184] identifying, based on the data and the plurality of predicted rates of hypoglycemic events, one or more covariates in the data that are associated with a predicted rate of hypoglycemic events;

[0185] A report is generated identifying the one or more covariates and the corresponding predicted rates of hypoglycemic events.

Claims

1. A method implemented by a computing device, comprising: receiving data representing a medical record of a patient, the patient having been diagnosed with diabetes; determining a predicted rate of hypoglycemic events using a machine learning system trained using data representing medical records of a plurality of patients and corresponding rates of hypoglycemic events for the respective patients; and A predicted rate is generated for the patient.

2. The computing device-implemented method of claim 1, wherein each of the plurality of patients uses the same type of basal insulin.

3. The method implemented by the computing device according to claim 1 or 2, wherein the method further comprises: determining a second predicted rate of hypoglycemic events using a second machine learning system trained using data representing medical records of a second plurality of patients and corresponding rates of hypoglycemic events for each of the second plurality of patients, wherein each of the second plurality of patients uses a second type of basal insulin that is different from the first type of basal insulin; and The first predicted ratio is compared to the second predicted ratio. 4 . The computing device-implemented method of claim 3 , further comprising recommending basal insulin for the patient based on the comparison.

5. The computing device-implemented method of claim 1 , further comprising determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record of each of the second plurality of patients; identifying, based on the data and the plurality of predicted rates of hypoglycemic events, one or more covariates in the data that are associated with a predicted rate of hypoglycemic events; A report is generated identifying the one or more covariates and the corresponding predicted rates of hypoglycemic events.

6. The computing device-implemented method of claim 1 , further comprising determining a plurality of predicted rates of hypoglycemic events for a second plurality of patients by providing the machine learning system with data corresponding to a medical record for each of the second plurality of patients; each of the second plurality of patients having the same covariates; A report is generated identifying the covariates and the corresponding predicted rates of hypoglycemic events.

7. A non-transitory computer readable medium encoded with computer program instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising: receiving data representing a medical record of a patient, the patient having been diagnosed with diabetes; determining a predicted rate of hypoglycemic events using a machine learning system trained using data representing medical records of a plurality of patients and corresponding rates of hypoglycemic events for each patient; and A predicted rate is generated for the patient.

8. The non-transitory computer readable medium of claim 7, wherein each of the plurality of patients uses the same type of basal insulin.

9. The non-transitory computer readable medium of claim 7 or 8, wherein the operations further comprise: determining a second predicted rate of hypoglycemic events using a second machine learning system trained using data representing medical records of a second plurality of patients and corresponding rates of hypoglycemic events for respective second patients, wherein each of the second plurality of patients uses a second type of basal insulin that is different from the first type of basal insulin; and The first predicted ratio is compared to the second predicted ratio.

10. The non-transitory computer readable medium of any one of claims 7-9, wherein the operations further comprise recommending basal insulin for the patient based on the comparison.

Images