A system for selecting the optimal combination of models to predict patient risk

Abstract

An automated system for selecting the optimal combination of risk models for a patient population. The selected combination may be monitored by a clinician and used to determine which patients are at highest risk of adverse events or clinical progression. The system may compare risk model data for hundreds or thousands of models with data collected on a target patient population to identify the best model combination for the target group. An example selection method may minimize a cost function that measures the deviation between the model combination and the desired characteristics for the optimal combination. Example factors in the cost function may include the difference between the predicted risk distribution for the target group by using the risk function of the model and the risk distribution for the dataset used to train the model, and the correlation between the risks predicted by the combined models.

Description

[Technical Field]

[0001] One or more embodiments of the present invention relate to the field of healthcare information systems and medical devices. More particularly, but not by way of limitation, one or more embodiments of the present invention enable a system to select an optimal combination of models for predicting patient risk. [Background technology]

[0002] Many risk prediction models have been developed and continue to be developed and refined to predict the risk of specific adverse events for individual patients in patient populations. For example, a sepsis model can predict a patient's risk of developing sepsis within the next 24 hours based on the patient's vital signs, laboratory tests, and demographic data. Healthcare professionals can use these models to determine which patients are at highest risk for which events and tailor their care accordingly. A challenge facing healthcare facilities is that so many different models exist that it is impractical to use or monitor all of them simultaneously. Similarly, even when multiple models exist to predict future risk for the same acute illness, it is difficult to determine the best model for a particular population without collecting independent data sets or conducting clinical studies.

[0003] Ideally, each patient care team would monitor a relatively small number of risk prediction models to assess the most significant risks for patients in their patient population. Currently, the only available method for selecting a risk prediction model is to iteratively experiment with different models to find one that is effective in each environment. This approach requires the additional step of monitoring patient events over time, which is time-consuming and impractical when new risk prediction models are constantly being developed. There are no known systems that automatically analyze the characteristics of a target patient population and suggest the optimal combination of risk prediction models for that patient population.

[0004] Given at least the above limitations, there is a need for a system that selects the optimal combination of models for predicting patient risk. Summary of the Invention [Problem to be solved by the invention]

[0005] One or more embodiments of the present invention may enable a system to select the optimal combination of models for predicting patient risk. The system may automatically compare many model combinations with the characteristics of a target patient population to determine the optimal combination for this population. [Means for solving the problem]

[0006] One or more embodiments of the present invention may include a plurality of risk models, data regarding a target patient population, and a processor that analyzes the risk model data and the target patient population data to automatically select an optimal combination of models for the target patient population. Each risk model may be associated with an event, one or more physiological systems associated with the event, a patient type, one or more inputs, a function that maps values ​​of the inputs to an incidence of the event for patients of the model's patient type, and training samples, each of the training samples including training sample input values ​​and training sample output values ​​that are the occurrence of the event for the sample. The target patient population data may include a target patient type, clinical acuity of the target patient, available inputs for the target patient, and target patient samples including sample input values ​​for the available inputs for the target patient. The processor may select an optimal combination of models for the target patient population using the following steps: filtering the plurality of models and identifying a set of applicable risk models based on the target patient population data. The processor may accept (e.g., from an operator) a model combination cardinality (an integer greater than or equal to 2), where the combination cardinality is the desired number of models to be combined. The processor may generate all relevant combinations of applicable risk models of the desired cardinality, where each different model in the relevant combination has a different associated physiological system. A cost function may then be applied to each relevant combination, which measures the difference between each combination and a theoretically optimal combination for the target patient population. The processor may then select the combination of models with the lowest associated cost.

[0007] In one or more embodiments, the filter for selecting the applicable risk model may ensure that the model inputs are included in the available inputs for the target patient and that the target patient type includes the risk model patient type.

[0008] In one or more embodiments, the cost function may be a weighted sum of cost factors.

[0009] In one or more embodiments, the cost factor may include a factor for the predicted risk distribution difference for each model in the combination. This factor may be based on a statistic of the predicted risk distribution for the target patient group, or the difference between a statistic applied to the predicted risk distribution for the target patient group and a desired value for that statistic. Specific statistic values ​​may include, but are not limited to, the mean, percentile, entropy, entropy rate, or distribution divergence. The predicted risk distribution for the target patient group may be calculated by applying the model function to the target patient sample input values ​​and dividing the result by the mean of the training sample output values ​​associated with the model.

[0010] In one or more embodiments, the target value of the statistic of the risk distribution may be the statistic applied to the risk distribution of the training set, which is the distribution of the training sample output values ​​divided by the mean of those training sample output values. In one or more embodiments, the mean of the predicted risk distribution may be compared to the mean of the risk distribution of the training set.

[0011] In one or more embodiments, the cost factor may be based on the difference between the 99.9th percentile of the predicted risk distribution for the target patient group and the maximum value of the risk indication range.

[0012] In one or more embodiments, the cost factor may include an output correlation factor, which may be a correlation factor between predicted risk distributions of target patient groups in the combined model.

[0013] In one or more embodiments, the cost factors may include a patient type difference factor for each model combined, which may be based on the difference between the patient type of the model and the patient type of the target patient population.

[0014] In one or more embodiments, the cost factors may include an event frequency factor for each model combined, which may measure how frequently an event occurs in the target patient population data.

[0015] In one or more embodiments, the cost factor may include an entropy estimate (at the relevant scale or scales) for each combined model, which may measure, for example, how smooth the predicted risk distribution of the model's target patient population is relative to the risk distribution in the model's training set.

[0016] In one or more embodiments, the cost factor may include an input distribution similarity factor, e.g., for each combined model, the distribution of individual features in the target patient sample input values ​​may be compared to the distribution of the same features in the model's training sample input values.

[0017] These and other aspects, features and advantages of the present invention will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0018] [Figure 1] 1 illustrates an architecture diagram of an example embodiment of the present invention, in which a model selection system selects an optimal combination of risk prediction models for a target patient population based on sample data collected from the target population. [Figure 2] 1. FIG. 2 shows an example of a display of patient risk prediction results for a combination of deployment models selected by the system of FIG. [Figure 3] An example of a risk prediction model that uses neural networks to map patient data to risk rates is shown. [Figure 4] 1 shows examples of risk prediction models and the physiological systems associated with each model. [Figure 5] The types of patient groups thought to be associated with each risk prediction model are shown. [Figure 6] 1 shows a flowchart of example steps performed by a model selection system to identify an optimal model combination for a target patient population. [Figure 7]We present a cost function framework that may be applied to calculate the optimal model at the minimum cost, where the cost function measures the difference between the model combination and the desired properties. [Figure 8] 10 shows an example of a distance metric between patient types in the model and patient types in the target patient population. [Figure 9] 1 shows an example of a distance metric between the model's predicted events and ranked adverse events likely to occur in the target patient population. [Figure 10] 1 shows an example of a distance metric between the predicted risk distribution for the model's training set and the predicted risk distribution in the target patient population. [Figure 11] 1 shows an example of a distance metric based on cross-correlation between risk distributions of different models. DETAILED DESCRIPTION OF THE INVENTION

[0019] A system for selecting an optimal combination of models for predicting patient risk is described. In the following description of the examples, numerous specific details are provided to provide a more thorough understanding of embodiments of the present invention. However, it will be apparent to one skilled in the art that the present invention can be practiced without incorporating all of the specific details set forth herein. In other instances, specific features, quantities, or measurements that are well known to those skilled in the art have not been described in detail so as not to obscure the present invention. While examples of the present invention are provided herein, it should be noted that it is the scope of the claims, and the full range of any equivalents, that define the metes and bounds of the present invention.

[0020] 1 illustrates an architectural diagram of example elements of one or more embodiments of the present invention. In an example scenario, an application administrator at a healthcare facility or similar organization 102 desires to install one or more risk prediction models for a target group of patients 103. This target patient group may be, for example, patients in a particular room in a hospital or similar facility, or may be a similar group of patients treated at various locations. Target groups of patients may be identified and grouped based on, for example, some or all of acuity, common procedures, demographic characteristics, etc.

[0021] A database 101 of risk models is available, and the organization 102 desires to use a combination of models from this database that best meets the needs of a target patient population. The database 101 may contain hundreds or thousands of risk models, making it impractical to evaluate or try each model individually. Instead, an automated model selection system 106 may be used to automatically analyze the model and target patient population data 104, 105 and recommend a model combination 108 that is optimal for the target patient population 103. The model selection system 106 may operate on a processor 110, such as, but not limited to, a server, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile device, a CPU, a GPU, a coprocessor, an ASIC, or a network of any number of these devices. The model selection system 106 may be connected to the database 101 and to data generated or obtained from the target patient population 103. After the model selection system 106 generates the recommended model combination 108, this combination, or a variation thereof, may be installed in a risk assessment system for the target patient population 103 in step 109. The risk assessment system may, for example, continuously or periodically monitor a target group of patients for the risk associated with each model of the selected multidimensional combination.

[0022] Because the model selection system is automated, the steps shown in Figure 1 may be repeated periodically or as needed as additional data is collected about the target patient population or as the patient demographics or health status of the target patient population change over time. Additionally, the comparison and cost functions described below may be used to monitor how well the selected model combination fits the target patient population over time.

[0023] FIG. 1 illustrates data associated with an example risk model 110 in a risk model database 101. Each risk model may predict the probability (i.e., risk) of a particular event (or events) occurring within a specified time period. The event 111 associated with the example model 110 is a patient developing sepsis within the next 24 hours. The event may be associated, for example, with the onset, change, or resolution of clinical symptoms, or with the need for treatment such as intubation, or other changes in the patient's condition, such as transfer to an ICU or discharge from the hospital. The model may predict the probability of any type of event, or type of event, that may affect or reflect the patient's treatment or condition. The model may be associated with any time period for which the probability of an event occurring is predicted. The physiological system 112 associated with the model 110 is a system associated with the sepsis event 111. Each model may be associated with one or more physiological systems related to the model's event.

[0024] Some models may be applicable to specific patient populations or disease acuity, for example, related to patient demographics, patient condition, patient treatment, or other factors that define a group of patients. For example, some models may apply only to pediatric patients, while other models may apply only to adult patients. In some cases, these patient populations may correspond to the type of hospital room that treats these types of patients. Patient populations for a model may be defined based on a combination of various factors, such as patient age or other demographic data, admission diagnosis, acuity / severity, and N-dimensional distribution of laboratory measurements, vital signs, and other physiological data.

[0025] The model 110 is associated with a patient type 113 of adult patients requiring intensive care, such as those likely to be treated in an intensive care unit. Thus, the model risk prediction for the event 111 is optimized for patients in this type of unit, or for patients with this type of acuity. The type of patient population associated with the model may have any level of granularity; for example, some models may be associated with adult patients in general, while other models may be associated with a very specific patient population, such as patients in an adult cardiac surgery unit.

[0026] Each model calculates event probabilities from an input dataset, such as patient demographics, laboratory results, nursing vital signs, and bedside measurements, along with computed features based on any of this information. The computed features may be designed to enhance the representation of variables over time or may better represent information embedded in the time series of measurements, waveform shape and morphology, and features of disease progression represented by the combination of variables. Model 110 has associated input data 114, which is a set of inputs used to calculate the probability that a patient will develop sepsis. Different models may use different sets of input data. Each model also has an adjoint function 116 that calculates the probability of the model's event (or events) occurring from the input values ​​114. This function may be derived, for example, by applying machine learning techniques to a training dataset 115 with input / output data expected to be obtained from a similar patient population. For example, for model 110, training dataset 115 may be obtained by collecting input data 114 and output data that labels each patient based on whether they developed sepsis within 24 hours. The model may use any type of machine learning technique, such as, but not limited to, regression, constrained cubic spline regression, neural networks, decision trees, bagged or boosted decision trees, or ensembles of models using these methods.

[0027] The model selection system 106 compares the data from each model with the characteristics of the target patient population 103 to determine the optimal model combination 108. The characteristics of the target patient population may include information 104 known about the target patient population prior to the model selection process. In the example shown in FIG. 1 , the patient type 131 of the target patient population 103 is known to be adult general medicine. In some cases, the type of target patient population may be very specific; in other cases, the target population may include a mix of patient types, or the specific type of patient may not be known. The target patient population data 104 may also include information 132 regarding available inputs for this patient population, such as clinical or demographic data collected for these patients. In some cases, there may be available data 133 regarding the most common events of interest occurring in the target patient population, which may be determined, for example, by reviewing the target patient's medical records or by interviewing practitioners.

[0028] In one or more embodiments, sample patient data 105 may be collected from a target patient population 103 over a period of time, such as one month, and this data may be used in the model selection process. Data for patients 120 from the target patient population may include, for example, demographic and medical history information 121, flow sheet input and clinical assessments (e.g., Glasgow Coma Score), vital signs 122 (e.g., blood pressure, temperature) obtained over time for the patient, monitoring data 123 from bedside monitoring devices such as heart rate monitors, parameters and waveforms from ventilators, EEG waveforms, and lab results 124. This data corresponds to available input 132 associated with the target patient population. This data is by way of example, and one or more embodiments may collect any type of information related to a patient's condition, identity, medical history, or treatment. Data for a sample set of patients from the target patient population 103 may be collected over a desired period of time, and this data set 105 may be input into the model selection system 106.

[0029] The model selection system 106 may select an optimal model combination 108 based on a set of objectives 107. In one or more embodiments, the weighting of various objectives may be set by a user of the model selection system. Examples of possible objectives 107 in one or more embodiments may include, but are not limited to, selecting models designed for patient types similar to those in a target population, selecting models that predict risk for events similar to those observed in the target patient population or diagnoses, selecting models that use inputs similar to those available for the target patient population, selecting combination models that target different physiological systems, selecting models that generate predicted event probabilities with distributions similar to those of the model's training dataset, and selecting combination models that generate predicted event probabilities that are uncorrelated.

[0030] Selecting models to be designed for patient types similar to those in the target population may be done using the clinical team's and administration's prior knowledge of admission procedures, or by comparing the distribution of individual features in the sample patient data with the distribution of the same features in the model's training dataset, using, for example, Kullback-Leibler divergence, Mahalanobis distance, or other distribution distance metrics.

[0031] The selected model combination 108 may be used in a patient risk assessment system installed to monitor patients in the target patient population 103. FIG. 2 shows an example display 201 illustrating the output of such a risk assessment system, which may be used by a medical professional 202 in a hospital room to monitor the patient. Because the selected model combination 108 includes two different models 108a and 108b, the risk assessment is two-dimensional. The event probabilities generated by each model are normalized to relative risks (as described below), and the relative risks for each patient are plotted on two axes 203 and 204, corresponding to the two models 108a and 108b, respectively, in the model combination 108. In this example plot, each patient is identified by their hospital bed number, and a circle with that bed number indicates the current risk level (on the two axes) for that patient. For example, circle 205 shows the two-dimensional risk for the patient in bed number 88, where the relative risk 215 for cardiovascular model 108b in combination 108 is 1.6 and the relative risk 216 for respiratory model 108a in combination 108 is 0.7.

[0032] The dotted circles indicate the recent history of patient risk levels, allowing practitioner 202 to see changes in risk for each patient. The size of each circle indicates the overall risk, which may be defined, for example, as the distance of the circle from the origin. For example, large circle 206 indicates very high risk for the patient in bed 93, while dotted circle 207 indicates that this patient's risk is increasing.

[0033] The patient risk level display 201 is exemplary, and in one or more embodiments of the present invention, the risk calculated by the selected combination of models may be displayed in any desired manner. The combination of models may be two-dimensional, as in FIG. 2, or may have any number of dimensions. Displaying risk levels in more than two dimensions may use various methods to show the risk in each dimension, for example, showing multiple plots or using attributes such as color and size to show the additional dimensions. Alternatively or additionally, the axes may combine multiple models that, when used in combination, provide desired information or improved predictive accuracy.

[0034] FIG. 3 illustrates an example calculation of relative risk 215 for patient #88 of FIG. 2. Function 116b associated with model 108b is implemented by a neural network in this example. In one or more embodiments, any type of function or algorithm may be used to calculate the event probability. In this example, the neural network function is trained with training data set 115b associated with model 108b. This training process may, for example, set weights associated with links between nodes; for example, the training process sets the weight of link 311 to value 312. (Other links have similar weights, but are not shown.) Training function 116b is then applied to inputs associated with patient #88 to calculate the risk for this patient. The inputs may include the patient's demographic data 121a, vital signs 122a, monitoring data 123a, and lab results 124a. The output of neural network function 116b may be the event probability predicted by the model. In this example, the model is associated with a single event, and function 116b calculates the event probability for the patient 301. For ease of interpretation, this event probability 301 may be converted to a relative risk (RR) 215 by dividing it by the average probability 302 for all patients in the training data set 115b. Thus, the average relative risk for the training data set is normalized to 1.0.

[0035] In addition to event probability functions, other information that may be associated with a model may include one or more physiological systems associated with the model's events and characteristics of the patient population on which the model is trained. FIG. 4 shows a partial table 401 containing physiological systems 403 associated with selected model events 402. Some events, such as sepsis 404, may be associated with multiple physiological systems. FIG. 5 shows an example classification of patient types that may be associated with a model. This hierarchical classification first distinguishes between intensive care patients 502, critical care patients 503, and emergency patients 504. Within these broad categories, specialized unit types or patient subgroups may be created, particularly in large medical facilities. Some patient population types may be specialized for specific age groups, such as types 511, 512, and 513. Other patient population types, such as types 514, 515, and 516, may be specialized for specific medical specialties. Some patient types may be classified based on whether they include medical patients, surgical patients, or both, such as types 517, 518, 519, and 520. Finally, general patient types 521 may include multiple types of patients, and the specifics of the patient composition may not be known. (Thus, for a target patient group that is a composition of different patient subpopulations, an ensemble of models fitted to the subpopulations may provide a better overall risk prediction than any of these individual models.)

[0036] Turning to the details of the model selection system 106, FIG. 6 outlines the steps that one or more embodiments may perform to determine the optimal model combination for a target patient population. Some or all of these steps may be performed by the processor 110. A set of risk models 101, along with target patient population characteristics 104 and sample target patient data 105 (as described above), are input into the system 106, and an optimal model combination 108 is output. In an initial filtering step 601, the models 101 are compared to the target patient population data to filter out models that are not suitable or that differ significantly from the target patient data. For example, filtering 601 may filter out models for patient types that differ significantly from the target patient type. Models may be filtered out if their age range differs from that of the target patient type; for example, a model for pediatric patients may be deemed inappropriate for adult patients, and vice versa. Similarly, in one or more embodiments, a model may be filtered out if, for example, a model is for intensive care patients but the target patient population is critical care patients.

[0037] The filtering step 601 may also filter the models 101 based on a comparison of available input data from the target room with the inputs required by the model. If a model's inputs are not available from the target room, the model may be removed from consideration (unless default values ​​can reasonably be defined for the missing inputs). For example, a model that predicts risk of myocardial infarction based on cardiac monitor data may be removed from consideration for target rooms that do not have cardiac monitors. After filtering, the remaining applicable models 605 are considered to calculate the optimal model combination.

[0038] Any number of model combinations (dimensions) may be considered for a target room. Before calculating the optimal combination of this number of models, a combination cardinality (number of models) may be selected 610. A typical choice of cardinality may be two models for ease of display on a risk chart (as shown in FIG. 2), but any number of model combinations may be considered. In one or more embodiments, the multiple models may be reduced to the target dimension by calculating a composite risk value (e.g., a weighted, normalized mean square calculation). The next step 612 is to generate a combination of filtered applicable models 605 with the desired number of dimensions. While all possible combinations may be considered, combinations of multiple models corresponding to the same physiological system may be excluded to expand coverage of multiple body systems. For example, a combination including a model for myocardial infarction and a model for hypotension may be excluded because both models relate to the cardiovascular system. Alternatively, an ensemble model including both such models may be considered. After eliminating model combinations corresponding to overlapping physiological systems, the remaining model combinations 613 may be ranked to find the optimal combination.

[0039] The model combinations may be ranked using a "cost function" that is calculated for each model combination in step 614. Conceptually, the cost function may quantify how far a model combination deviates from some optimum value; for example, the cost function may measure the deviation between the characteristics of the model combination and the characteristics of the target patient population. An example of a cost function is described below with reference to FIG. 7. After calculating the cost function for each combination 613, the combination with the lowest cost is selected in step 615 and selected as the optimal model combination 108 for the target patient population.

[0040] FIG. 6 shows an example of the number of models and combinations that may be considered at each step. The actual number will depend on the specific embodiment and the characteristics of the target patient population. In this example, there are 2,000 models in the database 101, but after filtering step 601, only 300 applicable models remain. If two models per combination is used as the combination cardinality selected in step 610, the 300 applicable filtered models yield 300*299 / 2=44,850 possible combinations, many of which may involve overlapping physiological systems. After filtering out model combinations corresponding to the same system, 17,000 model combinations (per two models) remain to be ranked using the cost function, and the model with the lowest cost among these 17,000 models becomes the output of the model selection system 106.

[0041] Figure 7 shows an example method for calculating a cost function 614 for each combination of models under consideration. The example cost function may be decomposed into multiple additive factors 702 shown in table 701, which correspond to the objectives 107 described with reference to Figure 1. The table shows an optimal (lowest cost) value 703 for each factor and a "distance metric" 705 for each factor that measures the deviation of the model combination from the optimal value 703. Each factor may have an associated weight 704, which may be set as needed for each embodiment and application of the model selection system.

[0042] The cost factors 711-715 in table 701 are exemplary, and in one or more embodiments, any subset of these factors may be used, or additional factors may be incorporated into the cost function.

[0043] Factor 711 measures the difference between the patient types of the combined model and the patient types of the target patient population. Metric 731 shown in Figure 8 may be applied to each combined model. Factor 712 measures the difference between the events associated with the combined model and the frequent events of the target patient population. Metric 732 shown in Figure 9 may be applied to each combined model.

[0044] Factors 713 and 714 measure the difference between the predicted risk distribution of the target patient population and the desired characteristics of this distribution. These differences may be measured for each model in combination. The predicted risk distribution of the target patient population may be calculated, for example, by applying the model function to sample patient inputs for samples obtained from the target patient population. (In one or more embodiments, the risk may be normalized to a "relative risk" rather than an absolute probability of an event occurring, as described below.) One or more cost factors may be calculated from the predicted risk distribution of the target patient population in any desired manner. An example method for calculating a cost factor from this predicted risk distribution is to calculate a statistic from the predicted risk distribution and measure the difference between this statistic and a desired optimal value for the target patient population. The desired value of the statistic may be a constant fixed value or may be based on applying the same statistic to the risk distribution of the model's training dataset. Examples of statistics that may be used for cost factors may include, but are not limited to, the mean, median, quartiles, percentiles, range, variance or standard deviation, entropy, divergence, or other functions of the distribution.

[0045] Factor 713 measures the difference between the mean of the predicted distribution of relative risk for each model and the corresponding mean of the predicted relative risk for that model's training dataset. Metric 733, shown in Figure 10, may be applied to each model in the combination.

[0046] The factor 714 measures the difference between the range of the predicted distribution (e.g., as measured by the 99.9th percentile) and a desired range that effectively separates high-risk and low-risk patients. The cost factor is based on the difference between the 99.9th percentile statistic applied to the predicted risk distribution for the target group and the desired maximum value of the risk representation range. For the example factor 714 and associated example metric 734, this maximum value is 6.0. The metric 734 shown in FIG. 10 may be applied to each combined model.

[0047] The factor 715 measures the difference between the observed cross-correlation of the combined models and the ideal value of no correlation (or anti-correlation). The metric 735 shown in Figure 11 may be applied to combined pairs of models.

[0048] Equation 721 shows an example calculation of a cost function from factors 702. This example cost function is a weighted sum of the squared distances between each factor value and the optimal value 703, using distance metric 705 and weights 704. A cost function may be calculated for each model combination in set 613. The optimal model combination is then obtained in step 615 by minimizing the cost function over the set of model combinations.

[0049] The factors and distance metrics in table 701 are exemplary, and one or more embodiments may use different factors, and factor costs may be calculated in any desired manner. Additionally, the costs associated with individual factors may be combined into a total cost function in any desired manner, including, but not limited to, using a weighted sum of squared distances, as shown in equation 721. Other factors may include, for example, but are not limited to, one or both of an entropy factor and an input distribution similarity factor. The entropy factor may include, for example, an entropy estimate (at relevant scales or multiple scales) for each combined model, which measures how smooth the model-estimated risk is relative to the estimated risk in the training set. The input distribution similarity factor may compare, for example, the distribution of individual features in the sample patient data with the distribution of the same features in the model's training dataset, using, for example, Kullback-Leibler divergence, Mahalanobis distance, or other distribution distance metrics.

[0050] 8, 9, 10, and 11 illustrate distance metrics 705 for factors 702. FIG. 8 illustrates selected values ​​for example distance metrics 731, which measure the similarity of patient types. An example distance is shown between patient type 831 (cardiovascular) and selected other patient types. These values ​​are merely exemplary, and in one or more embodiments, any desired value may be used to measure the distance between patient types. In the embodiment shown in FIG. 8, the distance metric between patient type 831 and itself is 0, representing an exact match. Patient type 832 is closely related to patient type 831 because both are cardiovascular patients, and thus the distance is set to 1. Patient type 834 has an even greater distance of 5 from patient type 831 because it involves a different physiological system. Patient type 833 includes pediatric patients, while patient type 831 includes adult patients, and thus has an even greater distance of 6 from patient type 831. Patient type 835 has a medium distance of 3 from patient type 831 because it is a mix of all patient types and may include cardiovascular patients.

[0051] FIG. 9 shows example values ​​for event type distance metric 732, which may measure, for example, the distance between an event associated with a model and an event in a target patient. In this example metric, the "distance" between a model and a target patient population is based on the frequency with which the model's event occurs in the target patient population compared to other types of events. For a target patient population in which the model's event is the most frequent event, the distance metric is zero. The less frequent the event in the target patient population, the greater the distance metric. For example, table 901 shows a ranked list of the most frequent events in the target patient population. (The events of interest here are those that typically occur after hospitalization. For example, a hospital room where all sepsis patients are treated may have a 50% diagnostic rate for sepsis, but are evident at the time of admission and are not included in the frequency of relevant events as described herein. The events of interest represent clinical deterioration after hospitalization, such as respiratory failure requiring intubation, bleeding requiring a blood transfusion, or septic shock requiring vasopressors.)

[0052] In this room, sepsis 902 is the most frequently observed event, so the distance 904 between this target patient group and the sepsis model 911 is 0. For the intubation model 912, the event most associated with intubation is COPD exacerbation 903. This event is ranked 6th in frequency in the target patient group, so the distance 905 between the target patient group and this model is 5 (6th minus 1). In one or more embodiments, the distance metric associated with the events in the model may be based on the absolute frequency of the event rather than the relative ranking of the event in the target patient group. For example, if the frequency of an event in the target patient group is f, an example distance metric may be d(event, target group) = 1 / f - 1, where a metric of zero (lowest cost) is assigned to an event that occurs in all patients and a significantly higher metric is assigned as the frequency of the event approaches zero in the target patient group.

[0053] FIG. 10 illustrates a distance metric based on the predicted risk distribution of the model 101a in a target patient population. As described with reference to FIG. 1, sample patient data 105 may be collected from the target patient population. This data may not include actual occurrences of the model's events in patients in the target population, but may include input data, such as vital signs or bedside monitoring, that the model uses to predict the probability of an event occurring. Thus, it may be possible to apply the model's function 116a to this subject population's input data 105 to calculate a predicted event probability for each sample patient in the target population. While no data is required to compare this predicted probability with the actual occurrence of the event in the target patient population, the overall distribution of predicted probabilities may be evaluated against desired distribution parameters. FIG. 10 illustrates distance metrics based on two statistics that may be calculated from the predicted risk distribution of the target population: metric 733 compares the mean of the predicted distribution with the mean of the corresponding distribution in the model's training dataset, and metric 734 compares the range (as expressed in terms of the 99.9th percentile) with a desired value.

[0054] As shown in FIG. 10 , the predicted distribution of risk for model 101 a is calculated by applying the model's function 116 a to sample patient input data 105 for sample patients in the target group. To simplify comparison and analysis, these absolute risks (event probabilities) may be converted to relative risks in step 1001 by dividing the probability of each event by the average probability of the event in the model's training dataset. The resulting predicted relative risk distribution 1020 is shown as a box plot in FIG. 10 . Applying the same function 116 a and normalization 1001 to the model's training dataset 115 a yields distribution 1010. The average relative risk 1011 of the training dataset distribution 1010 is, by definition, 1.0. The predicted relative risk 1021 (normalized to the training dataset mean) for the target patient group differs from the reference value 1011 by a distance 733, which is the distance metric for factor 713. This metric is zero (minimum cost) if the predicted distribution for the target patient group matches the training dataset distribution on average. In one or more embodiments, distributions 1010 and 1020 may be compared in any desired manner to form one or more distance metrics that measure how closely the predicted distributions match the distributions in the training data set. In addition to, or instead of, comparing means, one or more embodiments may compare any statistical value, such as the median, quartiles, standard deviation, or entire distributions, or entire distributions using metrics such as the Bhattacharyya distance or the Kullback-Leibler divergence.

[0055] Another desirable feature of the predicted distribution 1020 is that the range of predicted values ​​corresponds to a range within which a medical professional can effectively distinguish between low-risk and high-risk patients with sufficient resolution. Such a range may be incorporated into the risk prediction display, thereby influencing model selection. For example, the inventors have found that in some embodiments displaying a range of relative risks from 0.0 to 6.0 (as in the plot in FIG. 2 ), good resolution between low-risk and high-risk patients is achieved, making distributions that fall within this range preferable. Thus, metric 734 measures the difference between the 99.9th percentile statistic 1022 of the predicted relative risk and the desired maximum value of the risk display range (e.g., 6.0). (The maximum value of the risk display range may vary between different embodiments and may depend on the user interface for displaying relative risk and the distribution of relative risks for the target patient population.) Using the 99.9th percentile instead of the absolute maximum ensures that nearly all predicted relative risks fall within the desired range, while allowing some extreme outliers to exceed the target upper limit (considered to be the maximum value of the risk display range).

[0056] FIG. 11 illustrates a distance metric 735 based on the correlation of predicted risks from various models in a model combination. It shows a simplified scenario considering a combination of two models selected from three models 1101. Similar to FIG. 10, the risk prediction function of each of the models 1101 is applied to input data 105 from a target patient population to calculate a joint distribution 1102 of relative risks for three events associated with the three models 1101. From this joint distribution 1102, a cross-correlation 1103 is calculated for each pair of models. These correlations may be used directly as the correlation distance metric 735 for the two-model combination, or the distance metric may be any desired function of the correlation between the combined models. An ideal model combination would use models that are uncorrelated, corresponding to a distance metric of zero. In the example shown in FIG. 11, the minimum-cost model combination 1104 (based on the cross-correlation factor) is the combination with the lowest correlation between the two models in the combination.

[0057] While the invention disclosed herein has been described in terms of particular embodiments and applications thereof, numerous modifications and variations may be made by those skilled in the art without departing from the scope of the invention as set forth in the claims.

Claims

1. 1. A system for selecting an optimal combination of models for predicting patient risk, comprising: A plurality of risk models, each of the plurality of risk models comprising: Events and one or more physiological systems associated with the event; Patient type and one or more inputs; a function that maps values ​​of the one or more inputs to a probability of occurrence of the event in patients of the patient type; training samples, each training sample including training sample input values ​​for one or more inputs; training sample output values ​​of the occurrence of the event; Multiple risk models and Target patient population data, the target patient population data comprising: Target patient types and the available input of the target patient; target patient samples, each of the target patient samples comprising: a target patient sample input value of the target patient's available inputs; a target patient sample; Target patient population data and a processor coupled to the plurality of risk models and the target patient population data, the processor comprising: filtering the plurality of risk models to identify an applicable risk model based on the target patient population data; filtering out models for which required inputs are not present in the available inputs of the target patient; and filtering out models that are inappropriate or different for the target patient type as measurable by a distance metric. Thereby, identifying the applicable risk model; Accepts a combination cardinality of the model containing an integer greater than or equal to two; generating all relevant combinations of the applicable risk models, wherein for each combination of all relevant combinations, the number of models in each combination is equal to the combination cardinality of the models; the different models in each combination are associated with different physiological systems or systems; applying a cost function to each combination of all said relevant combinations to calculate an associated combination cost; The cost function quantifies the deviation between the properties of each combination and the desired properties of the target patient population, the deviation being: a difference between the patient type and the target patient type; the difference between the events associated with each of the risk models and the events occurring in the target patient population; the difference between one or more inputs associated with each of the risk models and the available inputs of the target patient for the target patient population; a predicted risk distribution for the target patient population generated by applying the cost function of each of the risk models to the target patient sample input values; and the difference between each of the risk models and the training set distribution; and a correlation between the predicted outputs of the model in each of the combinations, measuring the difference between each of the combinations and the optimal combination for the target patient population data; configured to identify, from among all said relevant combinations, a selected combination having an associated minimum combined cost; a processor; A system for selecting the best combination of models to predict patient risk.

2. For each applicable risk model of the applicable risk models: the available inputs for the target patient include one or more inputs associated with each applicable risk model; the target patient types include the patient types associated with each applicable risk model; A system for selecting an optimal combination of models for predicting a patient's risk according to claim 1.

3. The cost function comprises a weighted sum of cost factors. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 1.

4. The cost factor is: A factor for the predicted risk distribution difference for each model of each of the combinations. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

5. the predicted risk distribution difference factor associated with each model is based on the difference between a statistic applied to the predicted risk distribution of the target patient group and a desired value of that statistic; A system for selecting an optimal combination of models for predicting a patient's risk according to claim 4.

6. The statistics include one or more of the mean, percentile, entropy, entropy rate, and distribution divergence. A system for selecting an optimal combination of models for predicting a patient's risk according to claim 5.

7. the predicted risk distribution for the target patient population comprises a distribution of output values ​​of the function applied to the input values ​​of the target patient sample, divided by the mean of the training sample output values ​​associated with each model; A system for selecting an optimal combination of models for predicting a patient's risk according to claim 5.

8. the desired value of the statistic comprises a statistic applied to a risk distribution of a training set; the risk distribution of the training set includes a distribution of training sample output values ​​associated with each model, divided by the mean value of the training sample output values ​​associated with each model; The system for selecting an optimal combination of models for predicting a patient's risk according to claim 7.

9. The statistical value includes an average. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 8.

10. the statistical value includes a 99.9 percentile; the desired value of the statistical value comprises a maximum value of a risk indication range; The system for selecting an optimal combination of models for predicting a patient's risk according to claim 7.

11. The cost factor further includes an output correlation factor in the model of each combination. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 7.

12. The output correlation factor in the model of each combination includes a correlation factor between the predicted risk distribution of the target patient group in the model of each combination. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 11.

13. the cost factors further include a patient type difference factor for each model of each combination based on a difference between the patient type associated with each model and the target patient type. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

14. the cost factors further include an event frequency factor for each model of each combination that measures how frequently the event associated with each model occurs in target patient population data. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

15. For each applicable risk model of the applicable risk models: the available inputs for the target patient include one or more inputs associated with each applicable risk model; the target patient types include the patient types associated with each applicable risk model; The cost factor is: an output correlation factor between the models of each combination, the output correlation factor including a correlation coefficient between the predicted risk distributions of the target patient group in the models of each combination; a patient type difference factor for each model of each combination based on the difference between the patient type associated with each model and the target patient type; the cost factors further include an event frequency factor for each model of each combination measuring how frequently the event associated with each model occurs in the target patient population data. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 9.

16. The cost factors further include an entropy factor for each model in each of the combinations, which measures how smooth the predicted risk distribution of the target patient group associated with each model is relative to the risk distribution of the training set for each model. The system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

17. The cost factor is: a distribution of features in the target patient sample input values; an input distribution similarity factor for each model of each combination that measures the difference between the distribution of features in the training sample input values; The system for selecting an optimal combination of models for predicting a patient's risk according to claim 3.

Images