A system for selecting the optimal combination of models to predict patient risk

Abstract

An automated system for selecting the optimal combination of risk models for a patient population. The selected combination may be monitored by a clinician and used to determine which patients are at highest risk of adverse events or clinical progression. The system may compare risk model data for hundreds or thousands of models with data collected on a target patient population to identify the best model combination for the target group. An example selection method may minimize a cost function that measures the deviation between the model combination and the desired characteristics for the optimal combination. Example factors in the cost function may include the difference between the predicted risk distribution for the target group by using the risk function of the model and the risk distribution for the dataset used to train the model, and the correlation between the risks predicted by the combined models.

Description

[Technical field]

[0001] One or more embodiments of the present invention relate to the field of healthcare information systems and medical devices. More specifically, but not by way of limitation, one or more embodiments of the present invention enable a system to select an optimal combination of models for predicting patient risk. [Background technology]

[0002] Many risk prediction models have been developed and continue to be developed and refined to predict the risk of a particular adverse event occurring for individual patients in a patient population. For example, a sepsis model can predict a patient's risk of developing sepsis within the next 24 hours based on the patient's vital signs, laboratory and demographic data. Medical professionals can use these models to determine which patients are at highest risk for which events and tailor their care accordingly. The challenge facing medical facilities is that there are so many different models that it is impractical to use or monitor them all at the same time. Similarly, even when multiple models exist to predict future risk of the same acute illness, it is difficult to determine which model is best for a particular population without collecting independent data sets or conducting clinical studies.

[0003] Ideally, each patient care team would monitor a relatively small number of risk prediction models to assess the most significant risks to patients in the patient population that the team serves. Currently, the only available method for selecting a risk prediction model is to iteratively experiment with different models to find one that is effective in each environment. This approach requires an additional step of monitoring patient events over time, which is time-consuming and impractical when new risk prediction models are constantly being developed. There are no known systems that automatically analyze the characteristics of a target patient population and suggest the optimal combination of risk prediction models for that patient population.

[0004] Given at least the limitations discussed above, there is a need for a system that selects the optimal combination of models for predicting patient risk. Summary of the Invention [Problem to be solved by the invention]

[0005] One or more embodiments of the present invention may enable a system to select an optimal combination of models for predicting patient risk. The system may automatically compare many model combinations with characteristics of a target patient population to determine the optimal combination for this population. [Means for solving the problem]

[0006] One or more embodiments of the invention may include a plurality of risk models, data regarding a target patient population, and a processor that analyzes the risk model data and the target patient population data to automatically select an optimal combination of models for the target patient population. Each risk model may be associated with an event, one or more physiological systems associated with the event, a patient type, one or more inputs, a function that maps values ​​of the inputs to an occurrence rate of the event for patients of the model's patient type, and training samples, each of which includes a training sample input value and a training sample output value that is the occurrence of the event for the sample. The target patient population data may include a target patient type, a clinical acuity of the target patient, available inputs for the target patient, and a target patient sample including sample input values ​​for the available inputs for the target patient. The processor may select the optimal combination of models for the target patient population using the following steps: filtering the plurality of models and identifying a set of applicable risk models based on the target patient population data. The processor may accept (e.g., from an operator) a model combination cardinality (an integer equal to or greater than 2), where the combination cardinality is a desired number of models to be combined. The processor may generate all relevant combinations of the desired cardinality of applicable risk models, where each different model in the relevant combination has a different physiological system associated with it. A cost function may then be applied to each relevant combination, the cost function measuring the difference between each combination and a theoretically optimal combination for the target patient population. The processor may then select the combination of models with the lowest associated cost.

[0007] In one or more embodiments, the filter for selecting the applicable risk model may ensure that the model inputs are included in the available inputs for the target patient and that the target patient type includes the risk model patient type.

[0008] In one or more embodiments, the cost function may be a weighted sum of cost factors.

[0009] In one or more embodiments, the cost factor may include a factor of the predicted risk distribution difference for each model of the combination. The factor may be based on a statistic of the predicted risk distribution of the target patient group, or the difference between a statistic applied to the predicted risk distribution of the target patient group and a desired value of that statistic. Specific statistics may include, but are not limited to, for example, the mean, percentile, entropy, entropy rate, or distribution divergence. The predicted risk distribution of the target patient group may be calculated by applying the model function to the target patient sample input values ​​and dividing the result by the mean of the training sample output values ​​associated with the model.

[0010] In one or more embodiments, the target value of the risk distribution statistic may be the statistic applied to the risk distribution of the training set, which is the distribution of the training sample output values ​​divided by the mean of those training sample output values. In one or more embodiments, the mean of the predicted risk distribution may be compared to the mean of the risk distribution of the training set.

[0011] In one or more embodiments, the cost factor may be based on the difference between the 99.9th percentile of the predicted risk distribution of the target patient group and the maximum value of the risk indication range.

[0012] In one or more embodiments, the cost factor may include an output correlation factor, which may be a correlation factor between predicted risk distributions of target patient groups in the combined model.

[0013] In one or more embodiments, the cost factors may include a patient type disparity factor for each model combined, which may be based on the difference between the patient type of the model and the patient type of the target patient population.

[0014] In one or more embodiments, the cost factors may include an event frequency factor for each model combined, which may measure how frequently an event occurs in the target patient population data.

[0015] In one or more embodiments, the cost factor may include an entropy estimate (at the relevant scale or multiple scales) for each model combined, which may, for example, measure how smooth the predicted risk distribution of the model's target patient population is relative to the risk distribution in the model's training set.

[0016] In one or more embodiments, the cost factor may include an input distribution similarity factor, e.g., for each combined model, the distribution of individual features in the target patient sample input values ​​may be compared to the distribution of the same features in the training sample input values ​​of the model.

[0017] The above and other aspects, features and advantages of the present invention will become apparent from the following detailed description, taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0018] [Figure 1] 1 illustrates an architecture diagram of an example embodiment of the present invention, in which a model selection system selects an optimal combination of risk prediction models for a target patient population based on sample data collected from the target population. [Diagram 2] 2 shows an example of a display of patient risk prediction results for a combination of deployment models selected by the system of FIG. 1. [Diagram 3] 1 shows an example of a risk prediction model that uses neural networks to map patient data to risk rates. [Figure 4] 1 shows examples of risk prediction models and the physiological systems associated with each model. [Diagram 5] The types of patient groups thought to be associated with each risk prediction model are indicated. [Figure 6] 1 shows a flowchart of example steps performed by a model selection system to identify an optimal combination of models for a target patient population. [Figure 7]We present a cost function framework that may be applied to compute an optimal model with minimal cost, where the cost function measures the difference between a combination of models and a desired property. [Figure 8] 13 shows an example of a distance metric between patient types in the model and patient types in the target patient population. [Figure 9] 1 shows an example of a distance metric between the model's predicted events and ranked adverse events likely to occur in a target patient population. [Figure 10] 1 shows an example of a distance metric between the predicted risk distribution for the training set of the model and the predicted risk distribution in a target patient population. [Figure 11] 1 shows an example of a distance metric based on cross-correlation between risk distributions of different models. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] A system for selecting an optimal combination of models for predicting a patient's risk is described. In the following description of the examples, many specific details are provided to provide a better understanding of the embodiments of the present invention. However, it will be apparent to one skilled in the art that the present invention can be practiced without incorporating all aspects of the specific details described herein. In other examples, specific features, quantities, or measurements that are well known to those skilled in the art have not been described in detail so as not to obscure the present invention. Although examples of the present invention are provided herein, it should be noted that it is the scope of the claims, and any equivalents, that defines the metes and bounds of the present invention.

[0020] 1 illustrates an architecture diagram of example elements of one or more embodiments of the present invention. In an example scenario, an application administrator at a healthcare facility or similar organization 102 desires to install one or more risk prediction models for a target group of patients 103. This target patient group may be, for example, patients in a particular room in a hospital or similar facility, or may be a similar group of patients treated at various locations. Target groups of patients may be identified and grouped based on some or all of, for example, acuity, common procedures, demographic characteristics, etc.

[0021] A database 101 of risk models is available and the organization 102 wishes to use a combination of models from this database that best meets the needs of the target patient population. The database 101 may contain hundreds or thousands of risk models, making it impractical to evaluate or try each model individually. Instead, an automated model selection system 106 may be used to automatically analyze the models and the target patient population data 104, 105 and recommend a combination of models 108 that is optimal for the target patient population 103. The model selection system 106 may run on a processor 110, such as, but not limited to, a server, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile device, a CPU, a GPU, a co-processor, an ASIC, or a network of any number of these devices. The model selection system 106 may be connected to the database 101 and to data generated or obtained from the target patient population 103. After the model selection system 106 generates the recommended model combination 108, this combination, or a variation thereof, may be installed in the risk assessment system for the target patient population 103 in step 109. The risk assessment system may, for example, continuously or periodically monitor a target group of patients for the risks associated with each model of the selected multidimensional combination.

[0022] Because the model selection system is automated, the steps illustrated in Figure 1 may be repeated periodically, or as needed, as additional data is collected on the target patient population, or as the patient composition or health status of the target patient population changes over time. Additionally, the comparison and cost functions described below may be used to monitor how well the selected model combination fits the target patient population over time.

[0023] FIG. 1 illustrates data associated with an example risk model 110 in a risk model database 101. Each risk model may predict the probability (i.e., risk) of a particular event (or events) occurring within a specified time period. The event 111 associated with the example model 110 is a patient developing sepsis within the next 24 hours. The event may be associated, for example, with the onset, change, or resolution of clinical symptoms, or with the need for treatment, such as intubation, or other changes in the patient's condition, such as transfer to an ICU or discharge from a hospital. The model may predict the probability of any type of event, or type of event, that may affect or reflect the patient's treatment or condition. The model may be associated with any time period for which the probability of an event occurring is predicted. The physiological systems 112 associated with the model 110 are systems associated with the sepsis event 111. Each model may be associated with one or more physiological systems associated with the model's event.

[0024] Some models may be applicable to specific patient populations or disease acuity, for example, associated with patient demographic data, patient condition, patient treatment, or other factors that define a group of patients. For example, some models may only apply to pediatric patients, while other models may only apply to adult patients. In some cases, these patient populations may correspond to the type of hospital room that treats this type of patient. Patient populations for a model may be defined based on a combination of various factors, such as patient age or other demographic data, diagnosis at admission, acuity / severity, and N-dimensional distribution of laboratory measurements, vital signs, and other physiological data.

[0025] The model 110 is associated with a patient type 113 of adult patients requiring intensive care, such as those likely to be treated in an intensive care unit. Thus, the model risk prediction for the event 111 is optimized for patients in this type of unit, or for patients with this type of acuity. The type of patient population associated with the model may have any level of granularity, for example, some models may be associated with adult patients in general, while other models may be associated with a very specific patient population, such as patients in an adult cardiac surgery unit.

[0026] Each model calculates event probabilities from an input data set, such as patient demographics, laboratory results, nursing vital signs, bedside measurements, and calculated features based on any of these. The calculated features may be designed to enhance the representational transformation of variables over time, or may better represent information embedded in the time series of measurements, waveform shapes and morphology, and features of disease progression represented by the combination of variables. The model 110 has associated input data 114, which is a set of inputs used to calculate the probability that a patient will develop sepsis. Different models may use different sets of input data. Each model also has an adjoint function 116 that calculates the probability of the model's event (or events) occurring from the input values ​​114. This function may be derived, for example, by applying machine learning techniques to a training data set 115 with input / output data that may be obtained from a similar patient population. For example, for model 110, the training data set 115 may be obtained by collecting input data 114 and output data that labels each patient based on whether they developed sepsis within 24 hours. The models may use any type of machine learning technique, such as, but not limited to, regression, restricted cubic spline regression, neural networks, decision trees, bagged or boosted decision trees, ensembles of models using these methods, etc.

[0027] The model selection system 106 compares the data from each model with the characteristics of the target patient population 103 to determine the best model combination 108. The characteristics of the target patient population may include information 104 known about the target patient population prior to the model selection process. In the example shown in FIG. 1, the patient type 131 of the target patient population 103 is identified as adult general medicine. In some cases, the type of the target patient population may be very specifically identified, while in other cases the target population may have a mix of patient types, or the specific type of patient may not be identified. The target patient population data 104 may also include information 132 about the inputs available for this patient population, such as clinical or demographic data collected for these patients. In some cases, there may be available data 133 about the most frequent events of interest occurring in the target patient population, which may be determined, for example, by reviewing the medical records of the target patients or by interviewing practitioners.

[0028] In one or more embodiments, sample patient data 105 may be collected from the target patient population 103 over a period of time, such as a month, and this data may be used in the model selection process. Data for example patients 120 from the target patient population may include, for example, demographic and medical history information 121, flow sheet inputs and clinical assessments (such as Glasgow Coma Score), vital signs 122 (such as blood pressure, temperature, etc.) taken over time for the patient, monitoring data 123 from bedside monitoring devices such as heart rate monitors, parameters and waveforms from ventilators, EEG waveforms, and lab results 124. This data corresponds to available inputs 132 associated with the target patient population. This data is by way of example, and one or more embodiments may collect any type of information related to a patient's condition, identity, medical history, or treatment. Data for a sample set of patients from the target patient population 103 may be collected over a desired period of time, and this data set 105 may be input into the model selection system 106.

[0029] The model selection system 106 may select an optimal combination of models 108 based on a set of objectives 107. In one or more embodiments, the weights given to the various objectives may be set by a user of the model selection system. Examples of possible objectives 107 in one or more embodiments may include, but are not limited to, selecting models designed for patient types similar to those in the target population, selecting models that predict risk for events similar to those observed in the target patient population or diagnoses, selecting models that use inputs similar to those available for the target patient population, selecting combination models that target different physiological systems, selecting models that generate predicted event probabilities with distributions similar to those of the model's training dataset, and selecting combination models that generate predicted event probabilities that are uncorrelated.

[0030] Selecting models to be designed for patient types similar to those in the target population may be done using the clinical team's and administration's prior knowledge of admission procedures, or by comparing the distribution of individual features in the sample patient data with the distribution of the same features in the model's training dataset, for example using Kullback-Leibler divergence, Mahalanobis distance or other distribution distance metrics.

[0031] The selected model combination 108 may be used in a patient risk assessment system that is installed to monitor patients in the target patient population 103. Figure 2 shows an example of a display 201 showing the output of such a risk assessment system, which may be used by a medical professional 202 in a hospital room to monitor the patient. The selected model combination 108 includes two different models 108a and 108b, so the risk assessment is two-dimensional. The event probability generated by each model is normalized to a relative risk (as described below), and the relative risk for each patient is plotted on two axes 203 and 204, corresponding to the two models 108a and 108b, respectively, in the model combination 108. In this example plot, each patient is identified by a bed number, and a circle with that bed number indicates the current risk level (on the two axes) for that patient. For example, circle 205 shows the two-dimensional risk for a patient in bed number 88, where the relative risk 215 for cardiovascular model 108b in combination 108 is 1.6 and the relative risk 216 for respiratory model 108a in combination 108 is 0.7.

[0032] The dotted circles indicate the recent history of patient risk levels, allowing practitioner 202 to see the changes in risk for each patient. The size of each circle indicates the overall risk, which may be defined, for example, as the distance of the circle from the origin. For example, the large circle 206 indicates that the risk for the patient in bed number 93 is very high, while the dotted circle 207 indicates that this patient's risk is increasing.

[0033] The display 201 of patient risk levels is exemplary, and in one or more embodiments of the present invention, the risk calculated by the selected combination of models may be displayed in any desired manner. The combination of models may be two-dimensional, as in FIG. 2, or may be of any number of dimensions. Displays of risk levels with more than two dimensions may use various methods to show the risk in each dimension, for example, showing multiple plots, or using attributes such as color and size to show the additional dimensions. Alternatively or additionally, the axes may combine multiple models that, when used in combination, provide desired information or improved predictive accuracy.

[0034] FIG. 3 illustrates an example of a calculation of the relative risk 215 for patient #88 of FIG. 2. The function 116b associated with the model 108b is implemented by a neural network in this example. In one or more embodiments, any type of function or algorithm may be used to calculate the event probability. In this example, the neural network function is trained on a training data set 115b associated with the model 108b. The training process may, for example, set weights associated with links between nodes, e.g., the training process sets the weight of link 311 to value 312. (Other links have similar weights, but are not shown.) The training function 116b is then applied to inputs associated with patient #88 to calculate the risk for this patient. The inputs may include the patient's demographic data 121a, vital signs 122a, monitoring data 123a, and lab results 124a. The output of the neural network function 116b may be the event probability predicted by the model. In this example, the model is associated with a single event, and function 116b calculates the event probability for the patient 301. For ease of interpretation, this event probability 301 may be converted to a relative risk (RR) 215 by dividing it by the average probability 302 for all patients in the training data set 115b. Thus, the average relative risk for the training data set is normalized to 1.0.

[0035] In addition to the event probability function, other information that may be associated with the model may include one or more physiological systems associated with the model's events and characteristics of the patient population on which the model is trained. FIG. 4 shows a partial table 401 that includes physiological systems 403 associated with selected model events 402. Some events, such as sepsis 404, may be associated with multiple physiological systems. FIG. 5 shows an example classification of patient types that may be associated with the model. This hierarchical classification first distinguishes between intensive care patients 502, critical care patients 503, and emergency patients 504. Within these broad categories, specialized room types or patient subgroups may be created, especially in large medical facilities. Some patient population types may specialize in certain age groups, such as types 511, 512, 513. Other patient populations, such as types 514, 515, 516, may specialize in certain medical specialties. Some patient types may be classified based on whether they include medical patients, surgical patients, or both, such as types 517, 518, 519, 520. Finally, the general patient type 521 may include multiple types of patients, and the details of the patient composition may not be known (thus, for a target patient group that is a composition of different patient subpopulations, an ensemble of models fitted to the subpopulations may provide a better overall risk prediction than any of these models individually).

[0036] Turning now to the details of the model selection system 106, FIG. 6 outlines the steps that one or more embodiments may perform to determine the optimal model combination for a target patient population. Some or all of these steps may be performed by the processor 110. A set of risk models 101 are input to the system 106 along with the characteristics 104 of the target patient population and sample target patient data 105 (as described above), and an optimal model combination 108 is output. In a first filtering step 601, the models 101 are compared to the target patient population data to filter out models that are not suitable or that differ significantly from the target patient data. For example, filtering 601 may filter out models for patient types that differ significantly from the target patient type. Models may be filtered out if their age group is different from that of the target patient type, e.g., a model for pediatric patients may be deemed inappropriate for adult patients and vice versa. Similarly, in one or more embodiments, a model may be filtered out if, for example, a model is for intensive care patients, but the target patient population is acute care patients.

[0037] The filtering step 601 may also filter the models 101 based on a comparison of available input data from the target room with the inputs required by the models. If a model's inputs are not available from the target room, the model may be removed from consideration (unless default values ​​can reasonably be defined for the missing inputs). For example, a model that predicts risk of myocardial infarction based on cardiac monitor data may be removed from consideration for target rooms that do not have cardiac monitors. After filtering, the remaining applicable models 605 are considered to calculate the optimal model combination.

[0038] Any number of model (dimension) combinations of models may be considered for a target room. Before calculating the optimal combination of this number of models, a combination cardinality (number of models) may be selected 610. A typical choice of cardinality may be, for example, two models for ease of display on a risk chart (as shown in FIG. 2), but any number of model combinations may be considered. In one or more embodiments, the multiple models may be reduced to the target dimension by calculating a composite value of risk (e.g., a weighted, normalized mean square calculation). The next step 612 is to generate a combination of filtered applicable models 605 of the desired number of dimensions. All possible combinations may be considered, but combinations of multiple models corresponding to the same physiological system may be excluded to expand coverage of multiple body systems. For example, a combination including a model of myocardial infarction and a model of hypotension may be excluded since both models are related to the cardiovascular system. Alternatively, an ensemble model including both such models may be considered. After eliminating model combinations corresponding to overlapping physiological systems, the remaining model combinations 613 may be ranked to find the optimal combination.

[0039] The model combinations may be ranked using a "cost function" that is calculated for each model combination in step 614. Conceptually, the cost function may quantify how far a model combination deviates from some optimal value, for example, the cost function may measure the deviation between the characteristics of the model combination and the characteristics of the target patient population. An example of a cost function is described below with reference to FIG. 7. After calculating the cost function for each combination 613, the combination with the lowest cost is selected in step 615 and selected as the optimal model combination 108 for the target patient population.

[0040] FIG. 6 shows an example of the number of models and combinations that may be considered at each step. The actual number will depend on the specific embodiment and the characteristics of the target patient population. In this example, there are 2000 models in the database 101, but after the filtering step 601, there are only 300 applicable models. If two models per combination is used as the combination cardinality selected in step 610, the 300 applicable filtered models will result in 300*299 / 2=44850 possible combinations, many of which may have overlapping physiological systems. After filtering out model combinations corresponding to the same system, 17000 model combinations (per two models) remain to be ranked using the cost function, and the model with the lowest cost among these 17000 models is the output of the model selection system 106.

[0041] Figure 7 shows an example method for calculating a cost function 614 for each combination of models under consideration. The example cost function may be decomposed into multiple additive factors 702 shown in table 701, which correspond to the objectives 107 described with reference to Figure 1. The table shows an optimal (lowest cost) value 703 for each factor, and a "distance metric" 705 for each factor that measures the deviation of the model combination from the optimal value 703. Each factor may have an associated weight 704, which may be set as required for each embodiment and application of the model selection system.

[0042] The cost factors 711-715 in table 701 are exemplary, and in one or more embodiments, any subset of these factors may be used, or additional factors may be incorporated into the cost function.

[0043] Factor 711 measures the difference between the patient types of the combined model and the patient types of the target patient population. Metric 731 shown in Figure 8 may be applied to each combined model. Factor 712 measures the difference between the events associated with the combined model and the frequent events of the target patient population. Metric 732 shown in Figure 9 may be applied to each combined model.

[0044] Factors 713 and 714 measure the difference between the predicted risk distribution of the target patient population and the desired characteristics of this distribution. These differences may be measured for each model in combination. The predicted risk distribution of the target patient population may be calculated, for example, by applying the model functions to sample patient inputs for samples obtained from the target patient population. (In one or more embodiments, the risk may be normalized to a "relative risk" rather than an absolute probability of an event occurring, as described below.) From the predicted risk distribution of the target patient population, one or more cost factors may be calculated in any desired manner. An example method for calculating a cost factor from this predicted risk distribution is to calculate a statistic from the predicted risk distribution and measure the difference between this statistic and a desired optimal value for the target patient population. The desired value of the statistic may be a constant fixed value or may be based on applying the same statistic to the risk distribution of the model's training data set. Examples of statistics that may be used for the cost factors may include, but are not limited to, the mean, median, quartiles, percentiles, range, variance or standard deviation, entropy, divergence, or other functions of the distribution.

[0045] The factor 713 measures the difference between the mean of the predicted distribution of relative risk for each model and the corresponding mean of the predicted relative risk for that model's training dataset. The metric 733 shown in Figure 10 may be applied to each model in the combination.

[0046] The factor 714 measures the difference between the range of the predicted distribution (e.g., as measured by the 99.9th percentile) and a desired range that effectively separates high-risk and low-risk patients. The cost factor is based on the difference between the 99.9th percentile statistic applied to the predicted risk distribution for the target group and a desired maximum value for the risk indication range. For the example factor 714 and associated example metric 734, this maximum value is 6.0. The metric 734 shown in FIG. 10 may be applied to each combined model.

[0047] The factor 715 measures the difference between the observed cross-correlation of the combined models and the ideal value of no correlation (or anti-correlation). A metric 735 shown in FIG. 11 may be applied to combined pairs of models.

[0048] Equation 721 shows an example calculation of a cost function from factors 702. This example cost function is a weighted sum of the squared distances between each factor value and the optimal value 703, using distance metric 705 and weights 704. The cost function may be calculated for each model combination in set 613. The optimal model combination is then obtained in step 615 by minimizing the cost function over the set of model combinations.

[0049] The factors and distance metrics in table 701 are exemplary, and one or more embodiments may use different factors and calculate factor costs in any desired manner. Additionally, the costs associated with individual factors may be combined into a total cost function in any desired manner, including, but not limited to, using a weighted sum of squared distances, as shown in equation 721. Other factors may include, for example, but not limited to, one or both of an entropy factor and an input distribution similarity factor. The entropy factor may include, for example, an entropy estimate (at the relevant scale or multiple scales) for each model combined, which measures how smooth the risk estimated by the model is relative to the estimated risk in the training set. The input distribution similarity factor may compare, for example, the distribution of an individual feature in the sample patient data to the distribution of the same feature in the model's training dataset, using, for example, Kullback-Leibler divergence, Mahalanobis distance, or other distribution distance metric.

[0050] 8, 9, 10 and 11 illustrate distance metrics 705 for factors 702. FIG. 8 illustrates selected values ​​for example distance metrics 731 that measure similarity of patient types. An example distance between patient type 831 (cardiovascular) and selected other patient types is shown. These values ​​are merely exemplary and in one or more embodiments, any desired value may be used to measure the distance between patient types. In the embodiment illustrated in FIG. 8, the distance metric between patient type 831 and itself is 0, representing an exact match. Patient type 832 is closely related to patient type 831 because both are cardiovascular patients, and the distance is set to 1. Patient type 834 is an even greater distance of 5 from patient type 831 because it involves a different physiological system. Patient type 833 is an even greater distance of 6 from patient type 831 because it includes pediatric patients and patient type 831 includes adult patients. Patient type 835 is a medium distance of 3 from patient type 831 because it includes a mix of all patient types and may include cardiovascular patients.

[0051] FIG. 9 shows an example of an event type distance metric 732 value, which may measure, for example, the distance between an event associated with the model and a target patient event. In this example metric, the "distance" between the model and the target patient population is based on the frequency of the model's event occurring in the target patient population compared to other types of events. For a target patient population where the model's event is the most frequent event, the distance metric will be zero. The less frequent the event is in the target patient population, the greater the distance metric. For example, table 901 shows a ranked list of the most frequent events in the target patient population. (Events of interest here are those that typically occur after hospitalization. For example, a hospital room where all septic patients are treated may have a 50% diagnosis of sepsis, but are evident at the time of admission and are not included in the frequency of events of relevance as described herein. Events of interest represent clinical deterioration after hospitalization, such as respiratory failure requiring intubation, bleeding requiring a blood transfusion, or septic shock requiring vasopressors.)

[0052] In this room, sepsis 902 is the most frequently observed event, so the distance 904 between this target patient group and the sepsis model 911 is 0. For the intubation model 912, the event most associated with intubation is COPD exacerbation 903. This event is ranked 6th in frequency in the target patient group, so the distance 905 between the target patient group and this model is 5 (6th minus 1). In one or more embodiments, the distance metric associated with the events in the model may be based on the absolute frequency of the event, rather than the relative ranking of the event in the target patient group. For example, if the frequency of an event in the target patient group is f, an example distance metric may be d(event, target group)=1 / f-1, where a metric of zero (lowest cost) is assigned to an event that occurs in all patients, and a very high metric is assigned as the frequency of the event approaches zero in the target patient group.

[0053] FIG. 10 shows a distance metric based on the predicted risk distribution of the model 101a in the target patient population. As explained with reference to FIG. 1, sample patient data 105 may be collected from the target patient population. This data may not include the actual occurrence of the model's event in the patients of the target population, but may include input data such as vital signs or bedside monitoring that the model uses to predict the probability of the event occurrence. It may therefore be possible to apply the model's function 116a to the input data 105 of this subject population to calculate the predicted event probability for each sample patient of the target population. No data may be present to compare this predicted probability with the actual event occurrence in the target patient population, but the overall distribution of the predicted probabilities may be evaluated against the desired distribution parameters. FIG. 10 shows a distance metric based on two statistics that may be calculated from the predicted risk distribution of the target population: metric 733 compares the mean of the predicted distribution with the mean of the corresponding distribution of the model's training dataset, and metric 734 compares the range (as expressed in 99.9 percentile) with the desired value.

[0054] As shown in FIG. 10, the predicted distribution of risk for the model 101a is calculated by applying the model's function 116a to the sample patient input data 105 for the sample patients of the target group. To simplify comparison and analysis, these absolute risks (event probabilities) may be converted to relative risks in step 1001 by dividing the probability of each event by the average probability of the event in the model's training dataset. The resulting predicted relative risk distribution 1020 is shown as a box plot in FIG. 10. The same function 116a and normalization 1001 are applied to the model's training dataset 115a to obtain the distribution 1010. The average relative risk 1011 of the distribution 1010 of the training dataset is by definition 1.0. The predicted relative risk 1021 (normalized to the mean of the training dataset) of the target patient group differs from the reference value 1011 by a distance 733, which is the distance metric of the factor 713. This metric is zero (minimum cost) if the predicted distribution for the target patient group matches the distribution of the training dataset on average. In one or more embodiments, the distributions 1010 and 1020 may be compared in any desired manner to form one or more distance metrics that measure how closely the predicted distributions approximate the distributions of the training data set. In addition to, or instead of, comparing means, one or more embodiments may compare any statistical value such as the median, quartiles, standard deviation, or entire distributions, or entire distributions using metrics such as the Bhattacharya distance or the Kullback-Leibler divergence.

[0055] Another desirable feature of the prediction distribution 1020 is that the range of predicted values ​​corresponds to a range where a medical professional can effectively distinguish between low and high risk patients with sufficient resolution. Such a range may be incorporated into the risk prediction display, thus influencing model selection. For example, the inventors have found that in some embodiments displaying a range of relative risks from 0.0 to 6.0 (as in the plot of FIG. 2 ), good resolution between low and high risk patients is achieved, and thus a distribution that falls within this range is preferred. Thus, the metric 734 measures the difference between the 99.9th percentile statistic 1022 of the predicted relative risk and the desired maximum value of the risk display range (e.g., 6.0). (The maximum value of the risk display range may vary between different embodiments and may depend on the user interface for displaying the relative risks and the distribution of relative risks for the target patient population.) Using the 99.9th percentile instead of the absolute maximum ensures that nearly all of the predicted relative risks fall within the desired range, while allowing some extreme outliers to exceed the target upper limit (considered to be the maximum of the risk display range).

[0056] FIG. 11 illustrates a distance metric 735 based on the correlation of predicted risks from various models in a model combination. It illustrates a simplified scenario considering a combination of two models selected from three models 1101. As in FIG. 10, the risk prediction function of each of the models 1101 is applied to the input data 105 from the target patient population, and a joint distribution 1102 of the relative risks for the three events associated with the three models 1101 is calculated. From this joint distribution 1102, a cross-correlation 1103 is calculated for each pair of models. These correlations may be used directly as the correlation distance metric 735 for the two model combination, or the distance metric may be any desired function of the correlation between the combined models. The ideal model combination would use models that are uncorrelated, corresponding to a distance metric of zero. In the example illustrated in FIG. 11, the minimum-cost model combination 1104 (based on the cross-correlation factor) is the combination with the lowest correlation between the two models in the combination.

[0057] While the invention disclosed herein has been described in terms of specific embodiments and applications thereof, numerous modifications and variations may be made by those skilled in the art without departing from the scope of the invention as set forth in the claims.

Claims

1. 1. A system for selecting an optimal combination of models for predicting a patient's risk, comprising: A plurality of risk models, each of the plurality of risk models comprising: Events and one or more physiological systems associated with the event; Patient type and One or more inputs; a function that maps values ​​of the one or more inputs to a probability of occurrence of the event in patients of the patient type; training samples, each of the training samples including training sample input values ​​for one or more inputs; training sample output values ​​of the occurrence of the event; Multiple risk models and Target patient population data, the target patient population data comprising: Target patient type, available input of the target patient; A set of target patient samples, each of which comprises: a target patient sample input value of the available inputs of the target patient; a target patient sample; Target patient population data; a processor coupled to the plurality of risk models and the target patient population data, the processor comprising: filtering the plurality of risk models to identify an applicable risk model based on the target patient population data; Accepting a combination cardinality of the model that includes an integer number greater than or equal to 2; generating all relevant combinations of the applicable risk models, where in each combination of all relevant combinations, the number of models in each combination is equal to a combination cardinality of the models; the different models in each of the combinations are associated with a different physiological system or systems; applying a cost function to each combination of all said relevant combinations to calculate an associated combination cost, said cost function measuring the difference between each combination and an optimal combination for said target patient population data; configured to identify, from among all said relevant combinations, a selected combination having an associated minimum combination cost; a processor; A system for selecting the best combination of models to predict patient risk.

2. For each applicable risk model of the applicable risk models: The available inputs for the target patient include one or more inputs associated with each applicable risk model; the target patient types include the patient types associated with each applicable risk model; A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 1.

3. The cost function comprises a weighted sum of cost factors. A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 1.

4. The cost factor is: A factor of the predicted risk distribution difference for each model of each of the combinations. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

5. The predicted risk distribution difference factor associated with each model is based on the difference between a statistic applied to the predicted risk distribution of the target patient group and a desired value of that statistic. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 4.

6. The statistics include one or more of the mean, percentile, entropy, entropy rate, and distribution divergence. A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 5.

7. a predicted risk distribution for the target patient population comprising a distribution of output values ​​of the function applied to the input values ​​of the target patient sample, divided by the mean of the training sample output values ​​associated with each model; A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 5.

8. the desired value of the statistic comprises a statistic applied to a risk distribution of a training set; the risk distribution of the training set includes a distribution of training sample output values ​​associated with each model, divided by the mean of the training sample output values ​​associated with each model; A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 7.

9. The statistics include an average. A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 8.

10. the statistical value includes a 99.9 percentile; the desired value of the statistical value comprises a maximum value of a risk indication range; A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 7.

11. The cost factor further includes an output correlation factor in the model of each combination. A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 7.

12. The output correlation factor in the model of each combination includes a correlation factor between the predicted risk distribution of the target patient group in the model of each combination. A system for selecting an optimal combination of models for predicting a patient's risk as described in claim 11.

13. the cost factors further include a patient type difference factor for each model of each of the combinations based on a difference between the patient type associated with each model and the target patient type. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

14. The cost factors further include an event frequency factor for each model of each of the combinations measuring how frequently the event associated with each model occurs in the target patient population data. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

15. For each applicable risk model of the applicable risk models: The available inputs for the target patient include one or more inputs associated with each applicable risk model; the target patient population includes the patient type associated with each applicable risk model; The cost factor is: an output correlation factor between the models of each combination, the output correlation factor including a correlation coefficient between the predicted risk distributions of the target patient group in the models of each combination; a patient type difference factor for each model of each of the combinations based on a difference between the patient type associated with each model and the target patient type; the cost factors further include an event frequency factor for each model of each of the combinations measuring how frequently the event associated with each model occurs in the target patient population data. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 9.

16. The cost factors further include an entropy factor for each model of each of the combinations that measures how smooth the predicted risk distribution of the target patient group associated with each model is relative to the risk distribution of the training set of each model. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

17. The cost factor is: a distribution of features in the target patient sample inputs; and an input distribution similarity factor for each model of each combination that measures the difference between the distribution of features in the training sample input values. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

Images