Time-Dependent Triggers for Streaming Data Environments

Abstract

A method for dynamic risk prediction is provided. The method includes receiving a dataset having a first data field and a second data field. The first data field is populated with measurements. The method also includes assigning a first predicted value to the second data field, generating a first risk score and a first set of associated metrics based on the measurements and the first predicted value, and assigning the second predicted value to the second data field. The method also includes calculating a statistically derived metric and determining whether the statistically derived metric exceeds a predetermined threshold, where a predetermined action is recommended if the statistically derived metric exceeds the predetermined threshold. A system and a non-transitory computer-readable medium storing instructions for causing the system to perform the method are also provided.

Description

Technical Field

[0001] [Cross - Reference to Related Applications] This application claims the priority and benefit of U.S. Provisional Patent Application No. 62 / 959,742, entitled "Time - Sensitive Trigger for a Streaming Data Environment," filed on January 10, 2020, which is hereby incorporated by reference in its entirety as if fully set forth below and for all applicable purposes.

[0002] [Technical Field] The present disclosure generally relates to a time - sensitive trigger engine operating in a streaming data environment. More specifically, the present disclosure relates to devices in the healthcare industry that assist healthcare providers in making rapid time - dependent decisions with a high level of confidence from incomplete data instances.

Background Art

[0003] Prediction models often face the problem of missing data when deployed in real - world environments. Conventional solutions to this problem generally involve using some method to impute the missing data so that the model can generate an output. However, in a time - dependent streaming data environment where different parameters, each having various importance, arrive at different times, the complexity increases. In such situations, simply waiting for all the parameters used by the model to arrive is generally not optimal from the perspective of outputting accurate predictions as quickly as possible. Such applications can occur in emergency medical situations or emergencies for medical treatment, or in other environments such as stock investment decisions and other financial configurations. Similarly, in the above - described situations, it is desirable to obtain an early and accurate prediction of the result based on input data that may be incomplete.

Summary of the Invention

[0004] In some embodiments, a method for performing dynamic risk prediction includes receiving a data set including a first data field and a second data field, where a measurement value is input into the first data field. The method also includes substituting a first predicted value into the second data field, generating a first set of a first risk score and associated metrics based on the measurement value and the first predicted value, and substituting a second predicted value into the second data field. The method also includes generating a second set of a second risk score and associated metrics based on the measurement value and the second predicted value, and calculating a statistically derived metric based on the first risk score, the first set of associated metrics, the second risk score, and the second set of associated metrics. The method also includes determining whether the statistically derived metric exceeds a predetermined threshold, where if the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended.

[0005] In some embodiments, the system includes a memory configured to store instructions and one or more processors communicatively coupled to the memory. The one or more processors are configured to execute the instructions and cause the system to receive a data set including a first data field and a second data field, where the first data field is populated with a measurement value. The one or more processors are also configured to substitute a first predicted value into the second data field, generate a first set of a first risk score and associated metrics based on the measurement value and the first predicted value, substitute a second predicted value into the second data field, and generate a second set of a second risk score and associated metrics based on the measurement value and the second predicted value. The one or more processors are also configured to calculate a statistically derived metric based on the first risk score, the first set of associated metrics, the second risk score, and the second set of associated metrics, and determine whether the statistically derived metric exceeds a predetermined threshold, where if the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended, and generating the first set of associated metrics includes determining variability induced in the first risk score by the first predicted value at a between standard deviation value.

[0006] In some embodiments, the non-transitory computer-readable medium stores instructions that, when executed by a computer, cause the computer to perform a method. The method includes receiving a data set including a first data field and a second data field, where the first data field is populated with a measurement value, substituting a first predicted value into the second data field, and generating a first risk score and a first set of associated metrics based on the measurement value and the first predicted value. The method also includes substituting a second predicted value into the second data field, generating a second risk score and a second set of associated metrics based on the measurement value and the second predicted value, calculating a statistically derived metric based on the first risk score, the first set of associated metrics, the second risk score, and the second set of associated metrics, and determining whether the statistically derived metric exceeds a predetermined threshold, where if the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended. Generating the first set of associated metrics includes determining variability in the first risk score induced by the first predicted value in terms of an inter-subject standard deviation value and a within standard deviation value.

[0007] It should be understood that other configurations of the subject technology will be readily apparent to those skilled in the art from the following detailed description, in which various configurations of the subject technology are illustrated and described by way of example. As will be understood, the subject technology is capable of other different configurations, and some of its details are capable of modification in various other respects without departing from the scope of the subject technology. Accordingly, the drawings and detailed description are to be regarded as illustrative in nature and not as restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings, which are included to provide a further understanding and are incorporated in and constitute a part of this specification, illustrate the disclosed embodiments and, together with the description, serve to explain the principles of the disclosed embodiments.

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10A

Figure 10B

Figure 10C

Figure 10D

Figure 10E

Figure 10F

Figure 11

Figure 12

Figure 13A

Figure 13B

Figure 14A

Figure 14B

Figure 14C

Figure 14D

Figure 14E

Figure 14F

Figure 14G

Figure 14H

Figure 14I

Figure 15A

Figure 15B

Figure 15C

Figure 15D

Figure 15E

Figure 15F

Figure 15G

Figure 15H

Figure 15I

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

[0009] In the drawings, elements and steps shown by the same or similar reference numerals are associated with the same or similar elements and steps, unless otherwise indicated.

DETAILED DESCRIPTION OF THE INVENTION

[0010] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that embodiments of the present disclosure may be practiced without some of these specific details. In other instances, well-known structures and techniques have not been shown in detail so as not to obscure the present disclosure.

[0011] OVERALL SUMMARY Machine learning (ML) models often face the problem of missing data when deployed in real-world environments. Traditional ML, artificial intelligence (AI), and neural network (NN) algorithms are trained using large amounts of data input before analysis. Therefore, for a system using any of the above algorithms, it is desirable that a complete set of input data be available before evaluation using the trained ML / AI / NN algorithm. However, in a streaming data environment or other time-dependent configurations, data typically streams into the system in a streaming-based manner beyond the control of the system itself. Additionally, since a streaming data environment collects information asynchronously, different parameters and values, each having varying importance, can be collected by the modeling tool at different times. Thus, the problem of performing time-dependent predictive analysis in a streaming data environment includes, in addition to optimizing traditional metrics for predicting results, such as accuracy, sensitivity, specificity, area under the curve for receiver operating characteristics (AUCROC), minimizing the time to take corrective or preemptive measures (e.g., display of output to an end user, operation of a robot, purchase of a financial product, etc.). This is a technical problem arising from the field of computer data analysis for determining predictable results and taking preemptive measures accordingly. In various embodiments, the solution to this problem is to substitute missing data of a given streaming data instance into the model, calculate a metric that quantifies the certainty of the corresponding prediction, and supply such a metric to a rule-based logic system that controls whether the system takes an action. In various embodiments, the rule-based logic system can operate in a stateful manner, which means that the system can be triggered based on metrics and predictions derived from both the current and previous data instances.The embodiments disclosed herein include a framework, method, method evaluation metric, and secondary applications of such methods to address the problem of deploying machine learning systems in a time-dependent streaming data environment.

[0012] The embodiments disclosed herein provide a solution to the above problem in the form of a trigger logic engine that can predict results based on complete or incomplete input data. In various embodiments, the trigger logic engine quantifies the certainty of the prediction result based on the amount of available data (complete / incomplete, or substitute data) and other statistical values (e.g., variance, standard deviation, etc.) associated with the prediction result(s). When a metric derived from such statistical values is higher than a preselected threshold, the trigger logic engine provides the prediction output (e.g., to a healthcare provider or user who can take an action based on this prediction output). In some embodiments, the trigger logic engine may further provide one or more recommended (or required) actions based on the prediction output. When the certainty of the prediction result is lower (or equal) than a preselected threshold, the trigger logic engine delays the action or output until more time (e.g., when more data becomes available) and repeats the process.

[0013] According to various embodiments, methods and systems consistent with the present disclosure can be applied in the healthcare industry, where healthcare providers (e.g., physicians, nurses, paramedics, etc.) can benefit from a low-risk assessment of emergency situations where medical actions can be critical. In various embodiments, methods and systems as disclosed herein may also be applied in the financial industry, where large amounts of streaming data (e.g., current and previous stock prices of multiple public corporations) can lead to critical decisions based on accurate prediction of results.

[0014] The proposed solution further saves data storage space and reduces network usage due to the shortened time to determination by the methods and systems disclosed herein, thus improving the functionality of the computer itself.

[0015] Many of the examples provided herein illustrate that patient data is identifiable or that the download history of images is stored, but each user may grant explicit permission for the sharing or storage of such patient information. Explicit permission may be granted using privacy controls integrated into the disclosed system. Each user may be notified that such patient information may or is intended to be shared upon obtaining explicit consent, and each patient may, at any time, terminate the sharing of information and delete any stored user information. The stored patient information may be encrypted to protect patient security.

[0016] Exemplary System Architecture FIG. 1 shows an exemplary architecture 100 for time-dependent triggers in a streaming data environment, according to various embodiments. Architecture 100 includes a server 130 and a client device 110 connected via a network 150. One of the many servers 130 is configured to host a memory containing instructions that, when executed by a processor, cause the server 130 to perform at least some of the steps in the methods disclosed herein. At least one of the servers 130 may include or have access to a database containing clinical data for a plurality of patients.

[0017] Server 130 may include any device having suitable processors, memory, and communication capabilities for hosting image collection and a trigger logic engine. The trigger logic engine may be accessible by various client devices 110 via network 150. The client device 110 can be, for example, a desktop computer, a mobile computer, a tablet computer (including, for example, an e - book reader), a mobile device (such as a smartphone or PDA), or any other device having suitable processors, memory, and communication capabilities for accessing the trigger logic engine on one of the servers 130. According to various embodiments, the client device 110 can be used by medical personnel such as physicians, nurses, or emergency responders to access the trigger logic engine on one of the servers 130 in real - time emergency situations (such as in a hospital, clinic, ambulance, or any other public or residential environment). In some embodiments, one or more users of the client device 110 (such as nurses, emergency responders, physicians, and other medical personnel) can provide clinical data to the trigger logic engine within one or more servers 130 via network 150. In still other embodiments, one or more client devices 110 can automatically provide clinical data to the server 130. For example, in some embodiments, the client device 110 can be a blood test unit within a clinic configured to automatically provide patient results to the server 130 through a network connection. The network 150 can include, for example, any one or more of a local area network (LAN), a wide area network (WAN), the Internet, etc. Further, the network 150 can include any one or more of the following network topologies, including but not limited to a bus network, a star network, a ring network, a mesh network, a star - bus network, a tree or hierarchical network.

[0018] Exemplary Trigger System FIG. 2 is a block diagram 200 showing an exemplary server 130 and client device 110 in the architecture 100 of FIG. 1 according to certain aspects of the present disclosure. The client device 110 and the server 130 are communicatively coupled over a network 150 via respective communication modules 218-1 and 218-2 (hereinafter collectively referred to as “communication module 218”). The communication module 218 is configured to interface with the network 150 to transmit and receive information such as data, requests, responses, and commands to other devices on the network. The communication module 218 can be, for example, a modem or an Ethernet® card. The client device 110 and the server 130 can each include a memory 220-1 and 220-2 (hereinafter collectively referred to as “memory 220”), and processors 212-1 and 212-2 (hereinafter collectively referred to as “processor 212”). The memory 220 can store instructions that, when executed by the processor 212, cause either the client device 110 or the server 130 to perform one or more steps in the methods disclosed herein. Accordingly, the processor 212 can be configured to execute instructions such as instructions physically encoded in the processor 212, instructions received from software in the memory 220, or a combination of both.

[0019] According to various embodiments, server 130 may include or be communicatively coupled to database 252-1 and training database 252-2 (hereinafter collectively referred to as "database 252"). In one or more implementations, database 252 may store clinical data for multiple patients. According to various embodiments, training database 252-2 may be the same as or included within database 252-1. Clinical data within database 252 may include measurement information such as non-identifying patient characteristics, vital signs, complete blood count (CBC), comprehensive metabolic panel (CMP), and blood measurements such as blood gases (e.g., oxygen, CO2, etc.), immunological information, biomarkers, cultures, and the like. Non-identifying patient characteristics may include general medical history such as age, gender, and chronic diseases (e.g., diabetes, allergies, etc.). In various embodiments, clinical data may also include actions taken by medical personnel in response to measurement information such as treatment means, drug administration events, dosage, and the like. In various embodiments, clinical data may also include events and outcomes that occurred in a patient's medical history (e.g., sepsis, stroke, cardiac arrest, shock, etc.). Although database 252 is shown as being separate from server 130, in certain aspects, database 252 and trigger logic engine 240 may be hosted within the same server 130 and may be accessible by any other server or client device within network 150.

[0020] Memory 220-2 within server 130 may include a trigger logic engine 240 for evaluating streaming data inputs and triggering actions based on the prediction results. The trigger logic engine 240 may include a modeling tool 242, a statistical tool 244, and an imputation tool 246. The modeling tool 242 may include instructions and commands for collecting relevant clinical data and evaluating possible outcomes. The modeling tool 242 may include commands and instructions from neural networks (NN) such as deep neural networks (DNN), convolutional neural networks (CNN). According to various embodiments, the modeling tool 242 may include machine learning algorithms, artificial intelligence algorithms, or any combination thereof. The statistical tool 244 evaluates previous data collected by the trigger logic engine 240, stored in the database 252, or provided by the modeling tool 242. The imputation tool 246 may provide the modeling tool 242 with data inputs that would otherwise be missing from the measurement information collected by the trigger logic engine 240.

[0021] Client device 110 may access the trigger logic engine 240 through an application 222 installed on the client device 110 or through a web browser. The processor 212-1 may control the execution of the application 222 on the client device 110. According to various embodiments, the application 222 may include a user interface (e.g., graphical user interface - GUI-) that is displayed to the user on the output device 216 of the client device 110. The user of the client device 110 may use the input device 214 to input input data as measurement information or submit a query to the trigger logic engine 240 via the user interface of the application 222. According to some embodiments, the input data {X i (t x )} may be a 1×n vector, where X ijrepresents a data entry j (0 ≦ j ≦ n) that indicates any one of a plurality of clinical data values (or stock prices), which may or may not be available, for a given patient i, and t x represents the collection time at which the data entry was collected. In some cases, the available clinical data values or stock prices may be measured values that are input into at least some of the data fields of the input data {X i (t x )}. The client device 110 may receive, from the server 130, a prediction result M({X i (t x )}, Y i (t x ), Y i (t x )}) in response to the input data {X i (t x ), Y i (t x )}). According to some embodiments, the prediction result M({X i (t x )}, Y i (t x )}) may be determined based on not only the input data {X i (t x )} but also the substitution data {Y i (t x )}. Therefore, the substitution data {Y

[0022] (t i (t x)} is provided to a trigger logic input generation module, which includes an imputation engine and a statistical tool. The imputation engine provides imputed data {Y i (t x )}. According to various embodiments, the model may include a machine learning model configured to predict result O using a training dataset (hereinafter referred to as X train_idealized ), and an artificial intelligence model, a neural network model, or any combination thereof. According to various embodiments, X train_idealized is an m×n matrix, where m refers to the number of patients and n refers to the number of features in the clinical data that may be related to each patient's result. According to various embodiments, for each row in X train_idealized , some or all of the features (e.g., clinical data values) may be available regardless of the actual time at which they are available (e.g., measured by healthcare providers, patients, etc. or otherwise provided).

[0023] According to various embodiments, M is applied to input {X i (t x )}, where the features are assumed to arrive in a streaming-based manner, and thus, for a given patient i, each feature j arrives at any collection time t x . For each feature, the collection time t x may be at a predetermined schedule, asynchronous, or random. The trigger logic engine provides a decision as to whether the system should take an action based on a metric (defined later) derived from the statistical tool. According to various embodiments, the trigger logic engine may decide not to take an action at time t x , and then the same process may be repeated at time t x+1 when new data X i (t x+1 ) may arrive.

[0024] Figure 4 shows a block diagram of a trigger logic input generator for a trigger system according to various embodiments. The training data set timing matrix T(i,j), where feature j represents the time available to patient i, enables the construction of X in the trigger logic input generator. Thus, for each patient i and for each unique time within T(i,j), z instances can be generated per patient, where z = |{T(i,)}|. Each instance corresponds to a specific time t within {T(i,)}, and is a copy of X(i,) except when T(i,j) is greater than t, in which case X(i,j) is replaced with NA. Based on M(X(t)), the statistical tool in the trigger logic input generator determines one or more metrics from a set of metrics that includes a 'between standard deviation' value (BSD(M(X(t)))), a 'within standard deviation' value (WSD(M(X(t)))), and a 'total standard deviation' value (TSD(M(X(t)))). train_idealized Accordingly, for each patient i and for each unique time within T(i,j), z instances can be generated per patient, where z = |{T(i,)}|. Each instance corresponds to a specific time t within {T(i,)}, and is a copy of X(i,) except when T(i,j) is greater than t, in which case X(i,j) is replaced with NA. Based on M(X(t)), the statistical tool in the trigger logic input generator determines one or more metrics from a set of metrics that includes a 'between standard deviation' value (BSD(M(X(t)))), a 'within standard deviation' value (WSD(M(X(t)))), and a 'total standard deviation' value (TSD(M(X(t)))). train_idealized (i,) except when T(i,j) is greater than t, in which case X train_idealized (i,j) is replaced with NA. M(X i (t x )) Based on, the statistical tool in the trigger logic input generator determines one or more metrics from a set of metrics that includes a 'between standard deviation' value (BSD(M(X i (t x )))) and a 'within standard deviation' value (WSD(M(X i (t x )))) and a 'total standard deviation' value (TSD(M(X i (t x )))) of the set of metrics including.

[0025] According to various embodiments, the trigger logic input generator includes a multiple imputation tool that creates m imputed instances X(t) for a given X(t), where X(t) refers to the m-th imputed instance of X(t). For each instance X(t), the missing feature values are X i (t x ) For, m imputed instances X i_m (t x ) creates a multiple imputation tool, X i_m (t x ) is the m-th imputed instance of X i (t x ) refers to. For each instance X i_m (t x ), the missing feature values are X train_idealizedValues drawn from the distribution defined by are substituted. For example, in various embodiments, the multiple imputation tool may perform multiple imputation by chained equations. For each substitution instance, the modeling tool is used to M(X i_m (t x )) is calculated. Then, the value BSD(X i (t x )) is defined as the standard deviation of the set of values {M(X i_1 (t x )), M(X i_2 (t x )), …, M(X i_m (t x ))))}. Thus, in various embodiments, the metric BSD(M(X i (t x ))) can capture the variability induced in the results (e.g., medical results, financial results, etc.) by the missing data Y i (t x ).

[0026] The value for the metric WSD(M(X i (t x ))) can include the inherent variability in a given prediction due to sampling from X train_idealized and the variance of the response for a given input. Depending on the particular model used (e.g., logistic regression, random forest, SVM), the estimate of WSD(M(X i_m (t x ))) can be estimated using standard methods (e.g., standard error of the prediction interval, jackknife estimator, Bayesian estimator, maximum likelihood estimator, etc.).

[0027] The value for the metric TSD(M(X i (t x ))) includes an estimate of the total variance of M(Xi(t)). According to various embodiments, TSD can be obtained using the following formula:

Number

Number

[0028] Figure 5 shows an exemplary table of a dataset including time sequences of multiple clinical tests according to various embodiments. The table shows whether each of a plurality of features (e.g., clinical data) is available or collected for a patient at a given collection time. As can be seen from the table, multiple features can be collected during any given collection time period. Further, according to various embodiments, the same clinical features (e.g., heart rate, respiratory rate, systolic blood pressure, body temperature, etc.) can be repeatedly collected during different collection time periods.

[0029] Figure 6 is a table showing a plurality of features associated with a patient in a time sequence and trigger results for healthcare actions based on this feature according to various embodiments. This table details the patient and shows the arrival of specific data parameters and the moment when the trigger logic fires.

[0030] The table in FIG. 6 includes columns showing a patient, time, feature(s), model output M, and a decision result (e.g., "take an action", Y / N). Thus, for the first patient (e.g., patient 1), at time "0", only feature 3 is collected ({X1(0)} = {NA, NA, X, NA}), and since the model output M(X1(0)) is uncertain, the system does not take an action (N). At the subsequent time 1, for the same patient, a second feature is collected (e.g., feature 2 = Y, {X1(1)} = {NA, Y, X, NA}), but since the model output M(X1(1)) is still uncertain, the system takes no action (N) and waits for more data to be collected. At the subsequent time 2, for the same patient, a first feature is collected (e.g., feature 1 = Z, {X1(1)} = {Z, Y, X, NA}), and even if a fourth data (e.g., feature 4) has not yet been collected, the model output M(X1(2)) is sufficient for the system to take an action (Y).

[0031] According to various embodiments, the time entries in the table can occur at any given time period, and the intervals between different time entries may or may not be the same and may or may not be similar. In various embodiments, the intervals between different time entries may be pre-selected or may be random. Further, in various embodiments, two or more features may be received at a given time interval. The table in FIG. 6 shows, according to various embodiments, how the trigger logic engine can be prepared to take an action even if one or more features are missing from the input data. Thus, in various embodiments, the modeling engine can substitute values for missing data, and based on the statistical analysis of the model values and the substituted data, the trigger logic can determine to take an action with a given degree of certainty.

[0032] Figure 7 is a partial view of an input table associated with features that can trigger an action for a patient in a time sequence according to various embodiments. The input table includes columns indicating a patient, an entry time, and a feature. For simplicity, the table in Figure 7 shows only three features and one patient, but it should be understood that any number of features can be included for one, two, or any number of patients. The input features are shown as elements within a two-dimensional matrix X ij and the label NA indicates missing data. For example, the element X 11 is the value of feature 1 at times 0, 1, and 2 for patient 1. The element X 12 is the value of feature 2 at time 2, and the element X 13 is the value of feature 3 at times 1 and 2.

[0033] Figure 8 is a partial view of a training data set according to various embodiments. The training data set in Figure 8 includes an imputation column that lists missing data (e.g., data labeled "NA" in Figure 7) to be substituted by a modeling tool. According to various embodiments, the modeling tool can substitute multiple values for a single feature at a given instant.

[0034] For example, at time "0", features 2 and 3 are missing in the original data (see Figure 7), and thus, three imputation rows ("1", "2", and "3") are included for each distinct time value "0", "1", and "2". For time "0", in imputation "1", the modeling tool substitutes the value X 01 12 for feature 2 and the value X 01 13 for feature 3. In imputation "2", the modeling tool substitutes the value X 02 12 for feature 2 and the value X 02 13 for feature 3. In imputation "3", the modeling tool substitutes the value X 03 12 for feature 2 and the value X 03 13Substitute into Feature 3. For time "1", in Imputation "1", the modeling tool substitutes the value X 11 12 into Feature 2, and in Imputation "2", the modeling tool substitutes the value X 12 12 into Feature 2, and in Imputation "3", the modeling tool substitutes the value X 13 12 into Feature 2. At time "1", note that since the modeling tool is not substituting a value into Feature 3 as the "true" (or measured) value X 13 for that time is being collected. At time "2", the modeling tool does not provide substitution values since it is collecting the "true" values X 11 X 12 and X 13 for all three features.

[0035] Figure 9 is a partial view of a training data set with model outputs and standard deviations according to various embodiments. Thus, the table in Figure 9 is an extension of the table in Figure 8, with the model output column M(X(t)) and the within-target SD output column WSD(M(X(t))) added. The input data vector X(t) for the M and WSD columns varies according to the input data and substitution data, and the time t is one of three time periods "0", "1", and "2". For example, at time "0", three model outputs respectively associated with different data sets containing different substitution data for Features 2 and 3, M(X 11 , X 01 12 , X 01 13 (0)) for the input data {X 1_1 (0)}, M(X 11 , X 02 12 , X 02 13 (0)) for the input data {X 1_2 (0)}, and M(X 11 , X 03 12 , X 03 13 (0)) for the input data {X1_3 (0)) exists. Each of the model outputs M can be associated with different WSDs given the data value for each feature and the variance of the data values for each feature, regardless of whether the data value is collected from an instrument or device, manually entered by a healthcare provider, or substituted by a modeling tool. Thus, at time “0”, there are three different WSD values, the WSD(M(X 11 ,X 01 12 ,X 01 13}) for the input data {X 1_1 (0)), the WSD(M(X 11 ,X 02 12 ,X 02 13}) for the input data {X 1_2 (0)), and the WSD(M(X 11 ,X 03 12 ,X 03 13}) for the input data {X 1_3 (0)).

[0036] At time “1”, there are three model outputs, each associated with a different dataset that includes different substituted data for feature 2, M(X 11 ,X 11 12 ,X 13 )(1)) for the input data {X 1_1 (1)), M(X 11 ,X 12 12 ,X 13 (1)) for the input data {X 1_2 (1)), and M(X 11 ,X 13 12 ,X 13 (1)) for the input data {X 1_3 (1)). Each of the model outputs M has a WSD(M(X 11 ,X 11 12 ,X 13 (1))), the WSD(M(X 1_1 (1))) for the input data {X11 ,X 12 12 ,X 13} for WSD(M(X 1_2 (1))), input data {X 11 ,X 13 12 ,X 13} for WSD(M(X 1_3 (1))) can be associated with three different WSD values.

[0037] At time "2", the three model outputs, the input data {X 11 ,X 12 ,X 13} for M(X 1_1 (2)), input data {X 11 ,X 12 ,X 13} for M(X 1_2 (2)), and the input data {X 11 ,X 12 ,X 13} for M(X 1_3 (2)). Each model output M is a function of the input data {X 11 ,X 12 ,X 13} for WSD(M(X 1_1 (2))), input data {X 11 ,X 12 ,X 13} for WSD(M(X 1_2 (2))), and the input data {X 11 ,X 12 ,X 13} for WSD(M(X 1_3 (2))) can be associated with three different WSD values. 11 , X 12 , X 13} is the same for the three model outputs, so the value M(X 1_1 (2)), M(X 1_2 (2)), and M(X 1_3 Note that M(X (2)) may be similar. However, in some embodiments, the previous history of model outputs for different assignments at previous times may be different, and the modeling tool may use1_1 (2)), M(X 1_2 (2)), and M(X 1_3 (2)) can provide different outputs for at least one of them.

[0038] Figures 10A - 10F are graphs of exemplary trigger logic rules according to various embodiments. For example, a stateless trigger logic rule may involve triggering an action based on information available to the system at a given time t x and may involve triggering an action based on information available to the system at a given time t. M(X i (t x ))), BSD(M(X i (t x ))), WSD(M(X i (t x ))), and TSD(M(X i (t x ))) being given, various rules can be used to determine whether the system takes an action. The action taken by the system can be conditioned on M(X i (t x ))), BSD(M(X i (t x ))), WSD(M(X i (t x ))), and TSD(M(X i (t x ))).

[0039] Figure 10A shows an absolute BSD rule based on a static BSD threshold. Thus, when BSD(M(X i (t x ))) is less than or equal to a pre - selected constant c1, the system takes an action ("PASS"). Similarly, when BSD(M(X i (t x ))) is greater than c1, the system defers the decision to time t x+1 . Note that according to some embodiments, the absolute BSD rule may be independent of a specific value of the function M(X i (t x )) (hereinafter also referred to as the "score"). More generally, the "score" is M(X i(t x ) can be a function associated with the value of

[0040] Figure 10B shows a dynamic BSD threshold rule based on the ratio of BSD to score. Thus, when the ratio BSD(M(X i (t x ))) / M(X i (t x )) is less than or equal to a preselected constant c2, the system takes an action ("PASS"). Similarly, when BSD(M(X i (t x ))) is greater than c2, the system defers the decision to time t x+1 ("FAIL").

[0041] Figure 10C shows a logical rule based on the ratio of BSD to WSD. Thus, when the ratio BSD(M(X i (t x ))) / WSD(X i (t x )) is less than or equal to a preselected constant c3, the system takes an action ("PASS"). Similarly, when the ratio is greater than c3, the system defers the decision to time t x+1 ("FAIL").

[0042] Figure 10D shows a logical rule based on the ratio of BSD to TSD. Thus, when the ratio BSD(M(X i (t x ))) / TSD(X i (t x )) is less than or equal to a preselected constant c4, the system takes an action ("PASS"). Similarly, when the ratio is greater than c4, the system defers the decision to time t x+1 ("FAIL").

[0043] Figure 10E shows a logical rule based on crossing a score boundary. According to various embodiments, the score can be discretized into risk categories (e.g., low, medium, high) delimited by preselected boundaries such as b1, b2, etc. A method can be used that takes into account the value of the score, the dispersion of the score (among, within, or total), and the boundaries (e.g., b1, b2) that create the risk categories. For example, the score M(X i (t x )) may be associated with or considered a risk score indicating the level of risk of an undesirable outcome (e.g., a clinical emergency, a stock price drop or bankruptcy, etc.). Thus, when the risk score is higher than b1 or b2 and indicates the possibility of an undesirable outcome, it may be desirable for the system to take an action.

[0044] In various embodiments, when the value of M(X i (t x )) is less than b1 and the risk score and BSD satisfy the following equation, M(X i (t x )))+c5·BSD(M(X i (t x )))<b1 Equation (2) (where c5 is a preselected constant, the system takes an action ("PASS"). Further, when the value of M(X i (t x )) is greater than b1 and the risk score, M, and BSD satisfy the following equation, M(X i (t x )))-c5·BSD(M(X i (t x )))>b1 Equation (3) The system takes an action ("PASS").

[0045] When the value of M(X i (t x )) is greater than b2 and the risk score, M, and BSD satisfy the following equation, M(X i (t x)))-c·BSD(M(X i (t x )))>b2 Equation (4) The system takes an action ("PASS").

[0046] Figure 10F shows the polynomial quantile regression boundary. First, a matrix B is created, where each row of B is the BSD(M(X f for a given patient i at a fixed time t i (t f ))) for some or all patients. In various embodiments, t f is relative to some common event experienced by most or all patients such that t f is standardized. Given the matrix B, in various embodiments, polynomial quantile regression is performed on B for a given quantile q to create a function p q . For a given M(X i (t x ))), the system defers the action ("FAIL") for at least the time t i (t x )) when BSD(M(X q (M(X i (t x ))) is greater than or equal to p x+1 . Similarly, the system takes an action ("PASS") when the BSD is less than p q .

[0047] Figure 11 shows the time sequence of actions triggered by a trigger logic engine using stateless trigger logic rules according to various embodiments. Based on the input data X i (t x ), the trigger logic input generator generates M(X i (t x ), BSD(M(X i (t x )), WSD(M(X i (t x )), TSD(M(X i (t x))) is determined. Further, the trigger logic input generator supplies the input to the trigger logic engine for use in the stateless trigger logic rules R (see FIGS. 10A - 10F). Thus, the trigger logic engine is a function R(M(X that generates an output "0" to defer an action ("FAIL") or an output "1" to trigger an action ("PASS") i (t x ))), BSD(M(X i (t x ))), WSD(M(X i (t x ))), TSD(M(X i (t x ))).

[0048] According to various embodiments, the database coupled to the trigger logic engine stores, for a given stateless trigger logic rule R and a given patient i, the values M(X i (t trigger )) and t trigger in the matrix X R_simulated_stateless . The value t trigger is such that R(M(X i (t x ))), BSD(M(X i (t x ))), WSD(M(X i (t x ))), TSD(M(X i (t x )))=1, and may include a time within {T(i,j)} (e.g., the minimum time or one of the lowest time values in the set). In various embodiments, the database also includes standard diagnostic and prognostic metrics for X R_simulated_stateless . In various embodiments, the database may also store metrics associated with the time distribution of the triggers and the proportion of patients the system triggers (e.g., R = 1).

[0049] As shown in FIG. 11, at different times t xAt =0, 1, and 2, under the stateless trigger logic rules, different actions are taken by the system independently of each other (Action A, Action B, and Action C, respectively).

[0050] 12 illustrates a time sequence of actions triggered by a trigger logic engine using stateful trigger logic, according to various embodiments. y The input data collected at a given time t x may include state-dependent logic rules that are taken into account in the decision at <xである。様々な実施形態では、トリガ論理エンジンは、X R_simulated_stateful value M(Xi(t m_trigger )) and t m_trigger Matrix X containing R_simulated_stateful and a database storing, where t m_trigger refers to the mth time (e.g., the mth time the system is triggered for a given patient) such that R(M(Xi(tx)), BSD(M(Xi(tx))), WSD(M(Xi(tx))), and TSD(M(Xi(tx))) = 1. The value of m is determined based on the state-dependent trigger logic and is based on the patient's current (t x ) and previous states. The database also R_simulated_stateful The database may store standard diagnostic and prognostic metrics for. In various embodiments, the database may also include metrics regarding the time distribution of triggers and the percentage of patients for which the system triggers (e.g., R=1). Such a configuration may be desirable to increase the accuracy of predictions in less restrictive time-constrained environments.

[0051] In applications where the tolerance for time is greater, the trigger logic can be implemented in a state - dependent manner. For example, in a stateless environment, the output of the trigger logic engine can be represented as R(M(Xi(tx))), BSD(M(Xi(tx))), WSD(M(Xi(tx))), TSD(M(Xi(tx))), where R refers to a stateless trigger logic rule that outputs a binary number indicating triggering (1) or not triggering (0). Further, the function A can be defined to specify actions that the system can take (such as administration of a drug, provision of a medical treatment, investment or sale of funds, etc.) to prevent undesirable results or produce desirable results. Thus, A can be represented as A(M(Xi(tx)), BSD(M(Xi(tx))), WSD(M(Xi(tx))), TSD(M(Xi(tx))). In a state - dependent environment, R and A can be functions not only of M(Xi(tx)), BSD(M(Xi(tx))), WSD(M(Xi(tx))), and TSD(M(Xi(tx))), but also of M(Xi(ty)), BSD(M(Xi(ty))), WSD(M(Xi(ty))), TSD(M(Xi(ty))) for any y < x. The conditional logic governing this can be arbitrarily complex.

[0052] Thus, in various embodiments, a trigger logic engine that includes a stateful logic engine produces actions A, AB, and ABC at different times t x = 0, 1, and 2 = 0. Action AB can be a result not only of the values {M(Xi(0)), BSD(M(Xi(0))), WSD(M(Xi(0))), TSD(M(Xi(0))}, but also of the values {M(Xi(1)), BSD(M(Xi(1))), WSD(M(Xi(1))), TSD(M(Xi(1))}. Similarly, action ABC is the value {M(X x = 0 at i (0)), BSD(M(X i (0))), WSD(M(X i (0))), TSD(M(X i (0))} and the value at time t x = 1 of {M(X i(1)), BSD(M(X i (1))), WSD(M(X i (1))), TSD(M(Xi(1))}, and the time t x = 2 can be the result of the values {M(Xi(2)), BSD(M(Xi(2))), WSD(M(Xi(2))), TSD(M(Xi(2))}.

[0053] In various embodiments, the matrix X R_simulated_stateless and X R_simulated_stateful are used to quantify the impact of a given set of features conditioned on previous features available in the trigger logic engine. In various embodiments, the trigger logic engine is X R_simulated_stateless and X R_simulated_stateful configured to select, for each entry in either, the set of features that most influenced the decision for a given action A. For example, in various embodiments, the trigger logic engine is more relevant in the result of function A X i (t trigger ) and the value of t trigger , or X i (t m_trigger ) and the value of t m_trigger can be identified.

[0054] In various embodiments, the trigger logic engine determines the set of features that drive a given action A by, in the matrices X R_simulated_stateless and X R_simulated_stateful , identifying the feature values that arrive before t i (t trigger ) in X trigger , or before t i (t m_trigger ) in X m_trigger . In various embodiments, the trigger logic engine accesses a data structure within the matrix T(i,j) (which can be stored in a database) to make this determination. Thus, the trigger logic engine can provide the matrix D conditional , where each row is t trigger or t m_trigger , and t trigger or t m_triggercorresponds to the name of the corresponding set F of features that caused it. In some embodiments, matrix D conditional is coarser and includes class C, or a set of features that drive a given action. Class C may include important features such as CBC features, CMP features, financial features, seasonal features, etc. Matrix D conditional can be stored in a database for use by a trigger logic engine as needed.

[0055] In various embodiments, the trigger logic engine may also determine the proportion of entries for F or C in matrix D conditional . Thus, the proportion of entries for F and C in D conditional can be used in a modeling tool to evaluate the conditional impact of feature F or feature class C in the trigger logic engine. In various embodiments, feature F k or class C k conditional impact is given in relation to one or more of the features or feature classes: for example, the impact of F given Fx, Fy, …, Fz k or the impact of C given Cx, Cy, …, Cz k . In various embodiments, the features Fx, Fy, …, Fz and the feature classes Cx, Cy, …, Cz may vary from patient to patient.

[0056] In various embodiments, the trigger logic engine may determine the isolated effect of F k or C k when driving a given action A. Thus, the trigger logic engine may generate matrices X R_simulated_stateless and X R_simulated_stateful , where the columns of each row of T are permuted. For example, matrix T permuted is formed by independently shuffling the columns within the timing matrix T(i,j) for all i in T. Using T permute , the trigger logic engine generates X R_simulated_stateless and X R_simulated_stateful , similar to D conditional as D isolatedis also generated. Thus, the trigger logic engine determines an isolated influence from the prevalence of feature F isolated in matrix D k or class C k The trigger logic engine can determine an isolated influence from the prevalence of feature F in matrix D or class C.

[0057] More generally, various embodiments may include a trigger logic engine that determines the conditional effect of any feature Fk given Fx, Fy, …, Fz or class Ck given Cx, Cy, …, Cz, where Fx, Fy, …, Fz and Cx, Cy, …, Cz are the same for most or all patients. This can be achieved by appropriately reordering each T(i, ) for all i in T so that, for example, for most or all patients, Fk arrives after Fx, Fy, …, Fz, such that a particular relationship is maintained.

[0058] In various embodiments, the state-dependent logic within the trigger logic engine can identify when the score triggers again within T minutes from the initial trigger (e.g., R = 1). More specifically, in various embodiments, the time T after the initial trigger (R = 1) can be set to 90 minutes. Action A can be to present to the physician that the patient is in a low-risk category, meaning that the patient is currently unlikely to benefit from rapid administration of an antibiotic, and action B can be to present to the physician that the patient is in a medium-risk category, meaning that the patient is currently likely to moderately benefit from rapid administration of an antibiotic with respect to the relevant clinical outcome.

[0059] The current model value M indicates a medium-risk category (where the action the system should take is B), but if it was previously in a low-risk category (where the action the system should take is A, and A is different from B), the trigger logic engine may trigger the system to execute B (e.g., assuming the occurrence of A, AB = B). In various embodiments, action B itself may depend on A. Similarly, action ABC may indicate that action C is taken assuming that actions A and B have been taken (in that order).

[0060] Charts 1300A and 1300B (hereinafter collectively referred to as "Chart 1300") show the time development of the standard deviation distribution across risk factors measured for a plurality of patients over six different time intervals (enumerated as time in time units). The horizontal axis (X-axis) of Chart 1300 indicates the risk factor. The vertical axis (Y-axis) indicates the BSD / WSD ratio in Chart 1300A (see FIG. 13A) and the BSD value in Chart 1300B. Each facet within the plot refers to a specific time in time units for a fixed point in time.

[0061] Chart 1300 is an exemplary diagram of a trigger logic engine designed in relation to sepsis, a disease defined as life-threatening organ dysfunction caused by a dysregulated host response to infection. Early treatment - particularly the use of empirical antibiotics - leads to improved outcomes. However, ambiguous symptoms make the recognition of sepsis difficult and lead to increased mortality. The initial recognition and treatment of sepsis often occur in the emergency department (ED) environment, but the ED can be chaotic and understaffed, making it difficult for healthcare providers to reliably identify and treat this syndrome. Various embodiments solve this problem using a modeling tool as disclosed herein to assess the likelihood that a patient has sepsis and to assess the severity of the patient's condition.

[0062] In various embodiments, the modeling tools and trigger logic engines disclosed herein utilize characteristics that are routinely measured for patients suspected of having sepsis. Some of these characteristics may be present in the electronic medical record (EMR) for the patient (e.g., vitals, CBC, count-related test results, CMP, etc.), and for inpatients suspected of having sepsis who may not have such information in their EMR, specifically measured parameters (e.g., novel plasma proteins, nucleic acids, etc.) may be utilized. Thus, a trigger logic engine trained for sepsis diagnosis and treatment can operate in a highly time-dependent environment where streaming data arrives quickly and asynchronously from different sources.

[0063] In various embodiments, the modeling tool includes, for example, a function M that indicates a risk score in the range of 0 to 1. The risk score can be classified into one of three ranges: low, medium, or high risk. The trigger logic engine can be an action function A that includes results such as presenting the risk score to a physician, nurse, and / or associated healthcare worker, or deferring a decision to a later time (e.g., only for a selected time period or when new symptoms or medical characteristics appear). The action function A may depend on risk factors and other stateful information.

[0064] Figures 14A - 14I are charts 1400A - I (hereinafter collectively referred to as "chart 1400") showing diagnostic performance using a stateless trigger logic engine according to various embodiments. According to various embodiments, chart 1400 can be obtained using a statistical tool within a trigger logic engine that collaborates with a modeling tool and an imputation tool (see trigger logic engine 240, modeling tool 242, statistical tool 244, and imputation tool 246). Thus, the statistical tool can provide standard deviations (e.g., BSD, WSD, and TSD) and variance values for input data and substitution data using one or more mathematical formulas (see Formula 1) disclosed herein. Chart 1400 is collected in various exemplary case scenarios in a stateless configuration for illustrative purposes only (where the modeling tool takes into account the latest information available for imputation of missing data). Each color in the chart refers to the diagnostic performance of a specific stateless trigger logic rule R. By way of non - limiting example, various embodiments may include an imputation tool with an absolute between - variance value of 0.003, a combination of 0.6 BSD and a polynomial boundary for score, a BSD to OOB SD ratio of 0.125, a BSD to score ratio of 0.2, a boundary cross of 2.5, and a combination of 0.9 BSD and a polynomial quantile boundary for score. "Idealized" refers to a scenario where all data becomes available before an output is provided (this is optimal with respect to accuracy but sub - optimal in terms of providing timely predictions).

[0065] Figure 14A is chart 1400A showing the sensitivity - to - specificity response of a trigger logic engine according to various embodiments.

[0066] Figure 14B is chart 1400B showing the precision - to - recall performance of a trigger logic engine according to various embodiments.

[0067] Figure 14C is a chart 1400C showing the sensitivity versus specificity response of a trigger logic engine according to various embodiments. Chart 1400C is applied to the SOFA (sequential organ failure assessment) positive score.

[0068] Figure 14D is a chart 1400D showing the sensitivity versus specificity response of a trigger logic engine according to various embodiments. Chart 1400D is applied to the systemic inflammatory response syndrome (SIRS) negative analysis.

[0069] Figure 14E is a chart 1400E showing the probability spread of sepsis diagnosis determination in various embodiments using a trigger logic engine consistent with the present disclosure. Three different states are shown: non-septic, septic, and septic shock.

[0070] Figure 14F is a chart 1400F showing the probability spread for sepsis determination categories in various embodiments using a trigger logic engine consistent with the present disclosure. Four different categories are shown: OD_N_infection_N, OD_N_infection_Y, OD_Y_infection_N, and OD_Y_infection_Y.

[0071] Figure 14G is a chart 1400G showing the percentage of patients affected by decisions made based on a trigger logic engine as disclosed herein for the various embodiments listed above. The lowest impact is for the combination of a BSD of 0.06 and a polynomial boundary for the score, just over 92%. The highest impact is for decisions made with an absolute value of inter-subject variance of 0.003, with an impact of nearly 97%.

[0072] Figure 14H is a chart 1400H showing the timing for the decisions made by the trigger logic engine as disclosed herein for the various embodiments disclosed above. The time axis (vertical axis, i.e., the ordinate) indicates the time to decision in arbitrary units. The output of the trigger logic engine in chart 1400H indicates one of three risk categories ("0", "1", and "2") for sepsis diagnosis. Generally, the spread of the risk categories appears to be higher for low-risk data and lower for high-risk data.

[0073] Figure 14I is a chart 1400I showing the timing for the decisions made by the trigger logic engine as disclosed herein for the various embodiments disclosed above. The time axis (vertical axis, i.e., the ordinate) indicates the time to decision in arbitrary units. The decision of the trigger logic engine in chart 1400I is to determine the sepsis diagnosis according to three conditions: "non-sepsis", "sepsis", and "septic shock". Generally, the spread of the risk categories appears to be higher for low-risk data and lower for high-risk data.

[0074] Figures 15A - 15I are charts (1500A - I, hereinafter collectively referred to as "Chart 1500") showing diagnostic performance using a stateful trigger logic engine according to various embodiments. According to various embodiments, Chart 1500 can be obtained using a statistical tool within a trigger logic engine that collaborates with a modeling tool and an imputation tool (see trigger logic engine 240, modeling tool 242, statistical tool 244, and imputation tool 246). Thus, the statistical tool can provide standard deviations (e.g., BSD, WSD, and TSD) and variance values for input data and imputed data using one or more mathematical formulas (see Formula 1) disclosed herein. Chart 1500 is collected in various exemplary case scenarios in a stateful configuration for illustrative purposes only (where the modeling tool takes into account previously collected and / or imputed information in addition to the latest information available for imputation of missing data). Each color in the chart refers to the diagnostic performance of a specific stateless trigger logic rule R wrapped around stateful conditions. The specific stateful conditions used in this case were that the score triggered again within T minutes from the initial trigger, the score was currently in the medium - risk category (where the action the system should take is M), but was previously in the low - risk category (where the action the system should take is L, and L is different from M), and the system was triggered to execute M. Note that M itself may depend on L. By way of non - limitation, various embodiments may include an inter - subject variance absolute value imputation tool of 0.003, a combination of 0.6 BSD and a polynomial boundary for the score, a BSD to OOB SD ratio of 0.125, a BSD to score ratio of 0.2, a boundary crossing of 2.5, and a combination of 0.9 BSD and a polynomial quantile boundary for the score. "Idealized" refers to a scenario where all data becomes available before providing an output (this is optimal in terms of accuracy but sub - optimal in terms of providing timely predictions).

[0075] Figure 15A is a chart 1500A showing the sensitivity-versus-specificity response of a trigger logic engine according to various embodiments.

[0076] Figure 15B is a chart 1500B showing the precision-versus-recall performance of a trigger logic engine according to various embodiments.

[0077] Figure 15C is a chart 1500C showing the sensitivity-versus-specificity response of a trigger logic engine according to various embodiments. Chart 1500C is applied to the SOFA (Sequential Organ Failure Assessment) positive score.

[0078] Figure 15D is a chart 1500D showing the sensitivity-versus-specificity response of a trigger logic engine according to various embodiments. Chart 1500D is applied to the systemic inflammatory response syndrome (SIRS) negative analysis.

[0079] Figure 15E is a chart 1500E showing the probability spread of sepsis diagnosis determination in various embodiments using a trigger logic engine consistent with the present disclosure. Three different states are shown: non-septic, sepsis, and septic shock.

[0080] Figure 15F is a chart 1500F showing the probability spread for sepsis determination categories in various embodiments using a trigger logic engine consistent with the present disclosure. Four different categories are shown: OD_N_infection_N, OD_N_infection_Y, OD_Y_infection_N, and OD_Y_infection_Y.

[0081] Figure 15G is chart 1500G, which shows, for the various embodiments listed above, the percentage of patients affected by decisions made based on the trigger logic engine as disclosed herein. The lowest impact is for the combination of a BSD of 0.06 and a polynomial boundary for the score, just over 92%. The highest impact is for the decisions made for an absolute value of the inter-subject variance of 0.003, at nearly 97% impact.

[0082] Figure 15H is chart 1500H, which shows, for the various embodiments disclosed above, the timing for decisions made by the trigger logic engine as disclosed herein. The time axis (vertical axis, i.e., the ordinate) indicates the time to decision in arbitrary units. The output of the trigger logic engine in chart 1500H indicates one of three risk categories ("0", "1", and "2") for sepsis diagnosis. In general, the spread of the risk categories appears to be higher for low-risk data and lower for high-risk data.

[0083] Figure 15I is chart 1500I, which shows, for the various embodiments disclosed above, the timing for decisions made by the trigger logic engine as disclosed herein. The time axis (vertical axis, i.e., the ordinate) indicates the time to decision in arbitrary units. The decision of the trigger logic engine in chart 1500I is to determine sepsis diagnosis according to three conditions: "non-septic", "sepsis", and "septic shock". In general, the spread of the risk categories appears to be higher for low-risk data and lower for high-risk data.

[0084] As expected, the timing for decisions in charts 1500H and 1500I is slightly higher for the stateful logic configuration in the trigger logic engine compared to the stateless logic configuration (see charts 1400H and 1400I).

[0085] FIG. 16 is a chart 1600 showing the probability of taking action on a patient over time based on multiple medical characteristics according to various embodiments.

[0086] FIG. 17 is a bar graph 1700 of risk factors for two different sets of patients across several medical characteristics according to various embodiments. The bar graph 1700 is a visualization of the results of D conditional and shows a time-dependent trigger using XR simulated_stateless Thus, each row within the XR simulated_stateless data matrix of the bar graph 1700 corresponds to t trigger and the name of the corresponding set or class of clinical data (e.g., vital signs, CBC, CMP, etc.) that caused t trigger The bar graph 1700 shows an exemplary percentage of the conditional impact on a trigger logic engine for specific clinical data entries for two different groups of patients from separate clinical sites.

[0087] FIG. 18 is a flowchart showing steps in a method 1800 of performing a medical act on a patient based on a plurality of medical features received or substituted over a time sequence according to various embodiments. Method 1800 may be at least partially performed by any one of a plurality of client devices coupled to one or more servers via a network (e.g., any one of servers 130 and any one of client devices 110, and network 150). For example, according to various embodiments, a server may host one or more medical devices or portable computer devices carried by medical personnel or healthcare workers. Client device 110 may be operated by a user such as a staff member or other personnel within a medical facility, or an emergency team member transporting a patient to a medical facility or a hospital's emergency room or ambulance, or a person accompanying a patient at an individual's residence or public place away from a medical facility. At least some of the steps in method 1800 may be performed by a computer having a processor (e.g., processor 212 and memory 220) that executes commands stored in the computer's memory. According to various embodiments, a user may launch an application within the client device and access a trigger logic engine (e.g., application 222 and trigger logic engine 240) within the server via the network. The trigger logic engine may include modeling tools, statistical tools, and imputation tools (e.g., modeling tool 242, statistical tool 244, and imputation tool 246) for retrieving, supplying, processing clinical data in real time and providing its action recommendations. Further, the steps disclosed in method 1800 may include, among other things, using a trigger logic engine (e.g., database 252) to retrieve, edit, and / or store files in a database that is part of or communicatively coupled to the computer. Methods consistent with the present disclosure may include at least some, but not all, of the steps shown in method 1800, executed in a different sequence.Furthermore, a method consistent with the present disclosure may include at least two or more steps as in method 1800 that are executed overlapping in time or substantially simultaneously.

[0088] Step 1802 includes receiving input data for a modeling tool, where the input data indicates the status of the system.

[0089] Step 1804 includes substituting missing data to obtain substituted data for the modeling tool. In various embodiments, step 1804 includes applying multiple imputation techniques to generate N copies of the patient's data for a particular instance of the patient's data at a particular time. In various embodiments, step 1804 may include replacing a missing data value with one substituted data value. In some embodiments, step 1804 may include replacing each missing data value with one or more substituted data values to evaluate the variability in the imputation model. For example, step 1804 may include creating "N" substituted data values for each missing data value, where each substituted data value is predicted from a slightly different model within the modeling tool to reflect sampling variability.

[0090] Step 1806 includes using the modeling tool to evaluate a score using the input data and the substituted data, where the score is associated with a result based on the status of the system. For each copy of the data, step 1806 may include providing the input data (including the substituted data) to the modeling tool and generating a prediction of the result.

[0091] Step 1808 includes performing a statistical analysis of the score using a statistical tool. In various embodiments, step 1808 includes generating estimates of BSD, WSD, and TSD.

[0092] Step 1810 includes determining the likelihood of the result based on scores and statistical analysis. In various embodiments, step 1810 may include applying conditional logic to BSD, WSD, TSD, scores, and other outputs when the modeling tool provides a score. For example, in various embodiments, step 1810 may include applying a condition when BSD is less than a preselected value and then triggering a particular output or action. In some embodiments, step 1810 may include deferring a determination or output until a further time when the conditional logic is false or not satisfied.

[0093] FIG. 19 is a flowchart showing steps in a method 1900 of performing a medical act on a patient based on a plurality of medical characteristics received or substituted over a time sequence according to various embodiments. The method 1900 may be at least partially executed by any one of a plurality of client devices coupled to one or more servers via a network (e.g., any one of the servers 130 and any one of the client devices 110, and the network 150). For example, according to various embodiments, the server may host one or more medical devices or portable computer devices carried by medical personnel or healthcare workers. The client device may be operated by a user such as a staff member or other personnel within a medical facility, or an emergency team member transporting a patient to an emergency treatment room or ambulance in a medical facility or hospital, or a person accompanying the patient at the patient's personal residence or public place away from the medical facility. At least some of the steps in the method 1900 may be executed by a computer having a processor (e.g., processor 212 and memory 220) that executes commands stored in the computer's memory. According to various embodiments, the user may launch an application within the client device and access a trigger logic engine (e.g., application 222 and trigger logic engine 240) within the server via the network. The trigger logic engine may include modeling tools, statistical tools, and imputation tools (e.g., modeling tool 242, statistical tool 244, and imputation tool 246) for retrieving, supplying, processing clinical data in real time, or providing its action recommendations. Further, the steps disclosed in the method 1900 may include, among other things, using a trigger logic engine (e.g., database 252) to retrieve, edit, and / or store files in a database that is part of or communicatively coupled to the computer. A method consistent with the present disclosure may include at least some, but not all, of the steps shown in the method 1900, executed in a different sequence.Furthermore, a method consistent with the present disclosure may include at least two or more steps as in method 1900 that are executed over time in overlapping or substantially simultaneous fashion.

[0094] Step 1902 includes receiving a data set including a first data field and a second data field, where a measurement value is input into the first data field. According to various embodiments, step 1902 may include receiving, at a server, a measurement value from a client device via a network.

[0095] Step 1904 includes substituting a first predicted value into the second data field. According to various embodiments, step 1904 may further include determining the first predicted value based on the measurement value and a conditional rule associating the first data field with the second data field. According to various embodiments, step 1904 includes determining the first predicted value using a model within a trigger logic engine.

[0096] Step 1906 includes generating a first set of a first risk score and related metrics based on the measurement value and the first predicted value. According to various embodiments, step 1906 includes determining variability induced in the first risk score by the first predicted value at an inter-object standard deviation value. According to various embodiments, step 1906 includes determining variability induced in the first risk score by sampling variability of an intra-object standard deviation. According to various embodiments, step 1906 includes determining a total standard deviation including an inter-object standard deviation and an intra-object standard deviation.

[0097] Step 1908 includes substituting a second predicted value into the second data field.

[0098] Step 1910 includes generating a second set of a second risk score and related metrics based on the measurement value and the second predicted value.

[0099] Step 1912 includes calculating a statistically derived metric based on a first risk score, a first set of associated metrics, a second risk score, and a second set of associated metrics. According to various embodiments, step 1912 includes determining a ratio between a first standard deviation value and a second standard deviation value, each of the first standard deviation value and the second standard deviation value being selected from a first set of associated metrics or from a second set of associated metrics. According to various embodiments, step 1912 includes calculating a polynomial function of the first risk score or the second risk score and comparing a standard deviation selected from a first set of associated metrics and a second set of associated metrics with the polynomial function.

[0100] Step 1914 includes determining whether a statistically derived metric exceeds a predetermined threshold, where when the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended. According to various embodiments, a first set of related metrics corresponds to a first collection time, a second set of related metrics corresponds to a second collection time, and step 1914 includes using stateful logic after the first collection time and the second collection time. According to various embodiments, a first set of related metrics corresponds to a first collection time, a second set of related metrics corresponds to a second collection time, and step 1914 includes using stateless logic after one of the first collection time or the second collection time. According to various embodiments, the dataset includes clinical data about a patient, the clinical data has one of a complete blood count, a comprehensive metabolic panel, or a blood gas, and step 1914 includes determining a confidence level about the likelihood that the patient has septic shock. According to various embodiments, step 1914 includes selecting a predetermined action based on a previous dataset that includes a first previous value for a first data field and a second previous value for a second data field. According to various embodiments, step 1914 may further include providing a graphic chart for display, the graphic chart showing the statistically derived metric.

[0101] Figure 20 is a flowchart showing the steps in a method 2000 of performing a medical act on a patient based on a plurality of medical features received or assigned over a time sequence, according to various embodiments. The method 2000 may be at least partially performed by any one of a plurality of client devices coupled to one or more servers via a network (e.g., any one of the servers 130 and any one of the client devices 110, and the network 150). For example, according to various embodiments, the server may host one or more medical devices or portable computer devices carried by medical personnel or healthcare workers. The client device may be operated by a user such as a staff member or other person within a medical facility, or an emergency team member transporting a patient to a medical facility or hospital emergency room or ambulance, or a person accompanying a patient at a private residence or public place away from a medical facility. At least some of the steps in the method 2000 may be performed by a computer having a processor (e.g., the processor 212 and the memory 220) that executes commands stored in the computer's memory. According to various embodiments, the user may launch an application within the client device and access a trigger logic engine (e.g., the application 222 and the trigger logic engine 240) within the server via the network. The trigger logic engine may include modeling tools, statistical tools, and imputation tools (e.g., the modeling tool 242, the statistical tool 244, and the imputation tool 246) for retrieving, supplying, processing clinical data in real time, and providing its action recommendations. Further, the steps disclosed in the method 2000 may include, among other things, using the trigger logic engine (e.g., the database 252) to retrieve, edit, and / or store files in a database that is part of or communicatively coupled to the computer. Methods consistent with the present disclosure may include at least some, but not all, of the steps shown in the method 2000, performed in different sequences.Furthermore, a method consistent with the present disclosure may include at least two or more steps as in method 2000 that are performed over time in overlapping or substantially simultaneous fashion.

[0102] Step 2002 includes receiving a data set including a first data field and a second data field, where a measurement value is input into the first data field.

[0103] Step 2004 includes substituting a first predicted value into the second data field.

[0104] Step 2006 includes generating a first set of a first risk score and associated metrics based on the measurement value and the first predicted value.

[0105] Step 2008 includes substituting a second predicted value into the second data field.

[0106] Step 2010 includes generating a second set of a second risk score and associated metrics based on the measurement value and the second predicted value.

[0107] Step 2012 includes calculating a statistically derived metric based on the first risk score, the first set of associated metrics, the second risk score, and the second set of associated metrics.

[0108] Step 2014 includes determining whether the statistically derived metric exceeds a predetermined threshold, where if the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended.

[0109] Hardware Overview FIG. 21 is a block diagram showing an exemplary computer system 2100 that can implement the client device 110 and server 130 of FIGS. 1 and 2 and the methods of FIGS. 18-20. In certain embodiments, the computer system 2100 can be implemented using a combination of hardware or software and hardware, integrated within a dedicated server, or integrated with another entity, or distributed across multiple entities.

[0110] The computer system 2100 (e.g., client device 110 and server 130) includes a bus 2108 or other communication mechanism for communicating information, and a processor 2102 (e.g., processor 212) coupled to the bus 2108 for processing information. By way of example, the computer system 2100 can be implemented using one or more processors 2102. The processor 2102 can be a general-purpose microprocessor, a microcontroller, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gate logic, discrete hardware components, or any other suitable entity capable of performing calculations or other operations on information.

[0111] In addition to hardware, computer system 2100 can include code stored in built-in memory 2104 (e.g., memory 220) that creates an execution environment for the computer program, such as random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable PROM (EPROM), registers, hard disk, removable disk, CD-ROM, DVD, or any other suitable storage device coupled to bus 2108 for storing instructions and information to be executed by processor 2102, including code that constitutes a combination of processor firmware, protocol stack, database management system, operating system, or one or more of them. Processor 2102 and memory 2104 can be complemented or incorporated by dedicated logic circuitry.

[0112] The commands are stored in the memory 2104 and are encoded on a computer-readable medium for execution by one or more computer program products, i.e., one or more modules of computer program instructions, for controlling the operation of the computer system 2100, and can be implemented in accordance with any method well known to those skilled in the art, including, but not limited to, computer languages such as data-oriented languages (e.g., SQL, dBase), system languages (e.g., C, Objective-C, C++, assembly), architecture languages (e.g., Java (registered trademark),.NET), and application languages (e.g., PHP, Ruby, Perl, Python). The commands can also be implemented in computer languages such as array languages, aspect-oriented languages, assembly languages, authoring languages, command-line interface languages, compiler-type languages, concurrent languages, curly bracket languages, data flow languages, data-structured languages, declarative languages, esoteric languages, extended languages, fourth-generation languages, functional languages, interactive mode languages, interpreter-type languages, iterative languages, list-based languages, little languages, logic-based languages, machine languages, macro languages, metaprogramming languages, multi-paradigm languages, numerical analysis, non-English-based languages, object-oriented class-based languages, object-oriented prototype-based languages, offside rule languages, procedural languages, reflection languages, rule-based languages, script languages, stack-based languages, synchronous languages, syntax processing languages, visual languages, wirth languages, and xml-based languages. The memory 2104 can also be used to store temporary variables or other intermediate information during the execution of the instructions to be executed by the processor 2102.

[0113] The computer programs described herein do not necessarily correspond to files in a file system. The programs may be stored in part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program, or in multiple cooperating files (e.g., files that hold one or more modules, subprograms, or portions of code). The computer programs may be deployed to be executed on one computer, or located at one site, or distributed across multiple sites and executed on multiple computers interconnected by a communication network. The processes and logical flows described herein may be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output.

[0114] Computer system 2100 further includes a data storage device 2106, such as a magnetic disk or an optical disk, coupled to bus 2108 for storing information and instructions. The computer system 2100 can be coupled to various devices via an input / output module 2110. The input / output module 2110 can be any input / output module. An exemplary input / output module 2110 includes a data port, such as a USB port. The input / output module 2110 is configured to connect to a communication module 2112. An exemplary communication module 2112 (e.g., communication module 218) includes networking interface cards, such as an Ethernet card and a modem. In some aspects, the input / output module 2110 is configured to connect to multiple devices, such as an input device 2114 (e.g., input device 214) and / or an output device 2116 (e.g., output device 216). Exemplary input devices 2114 include a keyboard and a pointing device, such as a mouse or a trackball, through which a user can provide input to the computer system 2100. Other types of input devices 2114, such as a tactile input device, a visual input device, an audio input device, or a brain-computer interface device, can also be used to provide interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback, and the input received from the user can be in any form, including acoustic, voice, tactile, or electroencephalogram input. Exemplary output devices 2116 include a display device, such as an LCD (liquid crystal display) monitor, for displaying information to the user.

[0115] According to one aspect of the present disclosure, the client device 110 and the server 130 can be implemented using the computer system 2100 in response to the processor 2102 executing one or more sequences of one or more instructions included in the memory 2104. Such instructions can be read into the memory 2104 from another machine-readable medium such as the data storage device 2106. Execution of the sequence of instructions included in the main memory 2104 causes the processor 2102 to perform the process steps described herein. One or more processors in a multiprocessing configuration can also be used to execute the sequence of instructions included in the memory 2104. In an alternative aspect, instead of or in combination with software instructions, hardwired circuitry can be used to implement the various aspects of the present disclosure. Accordingly, the aspects of the present disclosure are not limited to a particular combination of hardware circuitry and software.

[0116] Various aspects of the subject matter described in this specification may be implemented in a computing system that includes, for example, backend components as a data server, or includes middleware components such as an application server, or includes frontend components such as a graphical user interface or a client computer having a web browser with which a user can interact with an implementation of the subject matter described in this specification, or includes any combination of one or more such backend components, middleware components, or frontend components. The components of the system may be interconnected by digital data communication in any form or medium, for example, by a communication network. The communication network (e.g., network 150) can include, for example, any one or more of a LAN, WAN, the Internet, etc. Further, the communication network can include any one or more of network topologies including, but not limited to, for example, a bus network, a star network, a ring network, a mesh network, a star-bus network, a tree or hierarchical network, etc. The communication module can be, for example, a modem or an Ethernet card.

[0117] Computer system 2100 can include clients and servers. Clients and servers are generally remote from each other and typically interact via a communication network. The relationship of client and server arises by virtue of computer programs running on respective computers and having a client-server relationship to each other. Computer system 2100 can be, for example, a desktop computer, a laptop computer, or a tablet computer, but is not limited thereto. Computer system 2100 can be incorporated into another device, for example, but not limited to, a cellular phone, a PDA, a mobile audio player, a global positioning system (GPS) receiver, a video game console, and / or a television set-top box.

[0118] As used herein, the term "machine-readable storage medium" or "computer-readable medium" refers to any one or more media that participate in providing instructions to a processor 2102 for execution. Such media can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Non-volatile media includes, for example, optical or magnetic disks, such as data storage device 2106. Volatile media includes dynamic memory, such as memory 2104. Transmission media includes coaxial cables, copper wire, and fiber optics, including the wires that form bus 2108. Common forms of machine-readable media include, for example, a floppy (registered trademark) disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, DVD, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH (registered trademark) EPROM, any other memory chip or cartridge, or any other medium readable by a computer. A machine-readable storage medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition that results in a machine-readable propagated signal, or a combination of one or more of them.

[0119] As used herein, the phrase "at least one of" preceding a series of items modifies the list as a whole, rather than each member (i.e., each item) of the list, with the terms "and" or "or" that separate any of the items. The phrase "at least one of" does not require the selection of at least one item; rather, this phrase can mean, among other things, at least one of any one of the items, and / or at least one of any combination of the items, and / or at least one of each of the items. By way of example, the phrase "at least one of A, B, and C" or "at least one of A, B, or C" can refer to, respectively, only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0120] To the extent that terms such as “include” and “have” are used in the description or claims, such terms are intended to be as encompassing as the term “comprise” is construed when “comprise” is used as a transitional term in the claims. The word “exemplary” as used herein is used for the purpose of meaning “serving as an example, instance, or illustration”. Any embodiment described herein as “exemplary” should not necessarily be construed as preferred or advantageous over other embodiments.

[0121] References to elements in the singular are not intended to mean “only one” unless explicitly stated otherwise, but rather “one or more”. All structural and functional equivalents to the various elements of the disclosed configurations described throughout this disclosure that are known or later become known to those of ordinary skill in the art are hereby expressly incorporated by reference and are intended to be encompassed by the claimed subject matter. Furthermore, nothing disclosed herein is intended to be dedicated to the public, whether or not such disclosure is explicitly recited in the above description.

[0122] This specification includes many details, but these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of particular implementations of the subject matter. The specific features described herein in connection with separate embodiments can also be implemented in combination in a single embodiment. Conversely, the various features described in connection with a single embodiment can also be implemented separately, or in any suitable sub-combination, in multiple embodiments. Furthermore, features may be described above as functioning in a particular combination and may initially be claimed as such, but one or more features from the claimed combination may in some cases be deleted from that combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination.

[0123] While the subject matter of this specification has been described with respect to specific embodiments, other embodiments can be implemented and are within the scope of the following claims. For example, although operations are shown in the drawings in a particular order, this should not be understood as requiring that such operations be performed in that particular order or in a sequential order, or that all of the illustrated operations be performed, to achieve a desired result. The actions recited in the claims can be performed in a different order and still achieve a desired result. As one example, the processes shown in the accompanying drawings do not necessarily require the particular or sequential order shown to achieve the desired result. Multitasking and parallel processing can be advantageous in certain circumstances. Further, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products. Other variations are within the scope of the following claims.

[0124] Enumeration of Embodiments 1. A method for performing dynamic risk prediction is provided, the method comprising receiving a dataset including a first data field and a second data field, wherein a measurement value is input into the first data field, substituting a first predicted value into the second data field, generating a first set of a first risk score and related metrics based on the measurement value and the first predicted value, substituting a second predicted value into the second data field, generating a second set of a second risk score and related metrics based on the measurement value and the second predicted value, calculating a statistically derived metric based on the first risk score, the first set of related metrics, the second risk score, and the second set of related metrics, and determining whether the statistically derived metric exceeds a predetermined threshold, wherein when the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended.

[0125] 2. The method according to embodiment 1, wherein generating the first set of related metrics includes determining the variability induced in the first risk score by the sampling variability of the within-subject standard deviation value.

[0126] 3. The method according to embodiment 1 or 2, wherein calculating the statistically derived metric includes calculating the standard deviation of the first risk score and the second risk score, referred to as the between-subject standard deviation.

[0127] 4. The method according to any one of embodiments 1 to 3, wherein calculating the statistically derived metric includes calculating a total standard deviation including the between-subject standard deviation and the within-subject standard deviation values derived from a mathematical combination of the first risk score, the second risk score, or both.

[0128] Calculating a statistically derived metric includes selecting a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, according to any one of embodiments 1 to 4.

[0129] 6. Calculating a statistically derived metric includes determining a ratio between any two of a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, according to any one of embodiments 1 to 5.

[0130] 7. Calculating a predetermined threshold includes evaluating a polynomial function of a first risk score or a second risk score and comparing the output of the function with a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, according to any one of embodiments 1 to 6.

[0131] 8. Determining whether a statistically derived metric exceeds a predetermined threshold, where a first set of related metrics corresponds to a first collection time and a second set of related metrics corresponds to a second collection time, includes using stateful logic after the first collection time and the second collection time, according to any one of embodiments 1 to 7.

[0132] 9. Determining whether a statistically derived metric exceeds a predetermined threshold, where a first set of related metrics corresponds to a first collection time and a second set of related metrics corresponds to a second collection time, includes using stateless logic after either the first collection time or the second collection time, according to any one of embodiments 1 to 8.

[0133] 10. Assigning the first predicted value to the second data field includes determining the first predicted value based on the measured value and a conditional rule associating the first data field with the second data field, according to any one of embodiments 1 to 9.

[0134] 11. A system is provided that includes a memory configured to store instructions and one or more processors communicatively coupled to the memory. The one or more processors execute the instructions to cause the system to receive a data set including a first data field and a second data field, where the first data field is populated with a measured value; assign a first predicted value to the second data field; generate a first set of a first risk score and related metrics based on the measured value and the first predicted value; assign a second predicted value to the second data field; generate a second set of a second risk score and related metrics based on the measured value and the second predicted value; calculate a statistically derived metric based on the first risk score, the first set of related metrics, the second risk score, and the second set of related metrics; and determine whether the statistically derived metric exceeds a predetermined threshold. Wherein when the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended. Generating the first set of related metrics includes determining the variability induced in the first risk score by the first predicted value at the between-subject standard deviation value.

[0135] 12. To generate the first set of related metrics, the one or more processors execute instructions to determine the variability induced in the first risk score by sampling variability at the within-subject standard deviation, according to the system of embodiment 11.

[0136] 13. To generate a first set of related metrics, one or more processors execute instructions to determine a total standard deviation including an inter-object standard deviation and an intra-object standard deviation, for the system of Embodiment 11 or 12.

[0137] 14. To calculate a statistically derived metric, one or more processors execute instructions to select a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-object standard deviation, or an intra-object standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, for the system of any one of Embodiments 11 to 13.

[0138] 15. To calculate a statistically derived metric, one or more processors execute instructions to determine a ratio between any two of a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-object standard deviation, or an intra-object standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, for the system of any one of Embodiments 11 to 14.

[0139] 16. A non-transitory computer-readable medium is provided that stores instructions that, when executed by a computer, cause the computer to perform a method. The method includes receiving a data set including a first data field and a second data field, where a measurement value is input into the first data field, substituting a first predicted value into the second data field, generating a first set of a first risk score and related metrics based on the measurement value and the first predicted value, substituting a second predicted value into the second data field, generating a second set of a second risk score and related metrics based on the measurement value and the second predicted value, calculating a statistically derived metric based on the first risk score, the first set of related metrics, the second risk score, and the second set of related metrics, and determining whether the statistically derived metric exceeds a predetermined threshold. Wherein, when the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended. Generating the first set of related metrics includes determining the variability induced in the first risk score by the first predicted value in terms of the between-subject standard deviation value and the within-subject standard deviation value.

[0140] 17. In the method, calculating the statistically derived metric includes evaluating a polynomial function of the first risk score or the second risk score and comparing the output of the function with a total standard deviation, between-subject standard deviation, or within-subject standard deviation value derived from a mathematical combination of the first risk score, the second risk score, or both. The non-transitory computer-readable medium according to embodiment 16.

[0141] 18. The first set of related metrics corresponds to a first collection time, the second set of related metrics corresponds to a second collection time, and determining whether the statistically derived metric exceeds a predetermined threshold includes using stateful logic after the first collection time and the second collection time. The non-transitory computer-readable medium according to embodiment 16 or 17.

[0142] The first set of related metrics corresponds to a first collection time, the second set of related metrics corresponds to a second collection time, and determining whether a statistically derived metric exceeds a predetermined threshold includes using stateless logic after one of the first collection time or the second collection time, the non-transitory computer-readable medium according to any one of embodiments 16 to 18.

[0143] Substituting a first predicted value into a second data field includes determining the first predicted value based on a measured value and a conditional rule associating the first data field with the second data field, the non-transitory computer-readable medium according to any one of embodiments 16 to 19.

Claims

**Claim 1**: A method for performing dynamic risk prediction executed by at least one processor, comprising: Receiving a dataset associated with a patient, the dataset including a first data field and a second data field; Entering a measurement value into the first data field, the measurement value being a value of a plasma protein or nucleic acid measured from the patient suspected of having sepsis; Substituting a first predicted value into the second data field via one or more machine learning algorithms trained for sepsis diagnosis and treatment; Generating a first set of a first risk score and related metrics based on the measurement value and the first predicted value; Substituting a second predicted value into the second data field via the one or more machine learning algorithms; Generating a second set of a second risk score and related metrics based on the measurement value and the second predicted value; Calculating a statistically derived metric based on the first risk score, the first set of related metrics, the second risk score, and the second set of related metrics; Determining whether the statistically derived metric exceeds a predetermined threshold. **Including**: A method. **Claim 2**: The method according to claim 1, wherein generating the first set of related metrics includes determining variability induced in the first risk score by sampling variability of an in-target standard deviation value. **Claim 3**: The method according to claim 1, wherein calculating the statistically derived metric includes calculating a standard deviation of the first risk score and the second risk score, referred to as an inter-target standard deviation. **Claim 4**: The method according to claim 1, wherein calculating the statistically derived metric includes calculating a total standard deviation including an inter-target standard deviation and an in-target standard deviation value derived from a mathematical combination of the first risk score, the second risk score, or both. **Claim 5**: ​ ​ ​ Calculating the statistically derived metric includes selecting a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, the method of claim 1.

6. Calculating the statistically derived metric includes determining a ratio between any two of a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, the method of claim 1.

7. Calculating the statistically derived metric includes evaluating a polynomial function of the first risk score or the second risk score and comparing the output of the function to a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, the method of claim 1.

8. The first set of related metrics corresponds to a first collection time, the second set of related metrics corresponds to a second collection time, and determining whether the statistically derived metric exceeds the predetermined threshold includes using stateful logic after the first collection time and the second collection time, the method of claim 1.

9. The first set of related metrics corresponds to a first collection time, the second set of related metrics corresponds to a second collection time, and determining whether the statistically derived metric exceeds the predetermined threshold includes using stateless logic after one of the first collection time or the second collection time, the method of claim 1.

10. Substituting the first predicted value into the second data field includes determining the first predicted value based on the measured value and a conditional rule associating the first data field with the second data field, the method of claim 1.

11. A system, A memory configured to store instructions, one or more processors communicatively coupled to the memory and the one or more processors execute instructions to cause the system to receive a data set including a first data field and a second data field, wherein a measurement value is input into the first data field substitute a first predicted value into the second data field via one or more machine learning algorithms trained for sepsis diagnosis and treatment, wherein the one or more machine learning algorithms are trained using a training data set including a plurality of patients and a plurality of clinical data values of the plurality of patients generate a first set of a first risk score and associated metrics based on the measurement value and the first predicted value substitute a second predicted value into the second data field via the one or more machine learning algorithms generate a second set of a second risk score and associated metrics based on the measurement value and the second predicted value calculate a statistically derived metric based on the first risk score, the first set of associated metrics, the second risk score, and the second set of associated metrics determine whether the statistically derived metric exceeds a predetermined threshold wherein when the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended, and generating the first set of associated metrics includes determining variability in the first risk score induced by the first predicted value in terms of an inter-subject standard deviation value system

12. The system of claim 11, wherein to generate the first set of associated metrics, the one or more processors execute instructions to determine variability in the first risk score induced by sampling variability in terms of an intra-subject standard deviation

13. The system of claim 11, wherein to generate the first set of associated metrics, the one or more processors execute instructions to determine a total standard deviation including an inter-subject standard deviation and an intra-subject standard deviation

14. To calculate the statistically derived metric, the one or more processors execute instructions to select a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, the system of claim 11.

15. To calculate the statistically derived metric, the one or more processors execute instructions to determine a ratio between any two of a first risk score or a second risk score or a mathematical combination of both, a total standard deviation, an inter-subject standard deviation, or an intra-subject standard deviation value derived from the first risk score, the second risk score, or a mathematical combination of both, the system of claim 11.

16. A non-transitory computer-readable medium storing instructions that, when executed by a computer, cause the computer to perform a method, the method comprising receiving a data set including a first data field and a second data field, wherein measurements are input into the first data field, substituting a first predicted value into the second data field via one or more machine learning algorithms trained for sepsis diagnosis and treatment, the one or more machine learning algorithms being trained using a training data set including a plurality of patients and a plurality of clinical data values of the plurality of patients, generating a first set of a first risk score and associated metrics based on the measurement and the first predicted value, substituting a second predicted value into the second data field via the one or more machine learning algorithms, generating a second set of a second risk score and associated metrics based on the measurement and the second predicted value, calculating a statistically derived metric based on the first risk score, the first set of associated metrics, the second risk score, and the second set of associated metrics, determining whether the statistically derived metric exceeds a predetermined threshold including, where when the statistically derived metric exceeds the predetermined threshold, a predetermined action is recommended, and generating the first set of associated metrics includes determining variability in the first predicted value induced by the first risk score in terms of the between-subject standard deviation value and the within-subject standard deviation value, A non-transitory computer-readable medium. **Claim 17** In the method, calculating the statistically derived metric includes evaluating a polynomial function of the first risk score or the second risk score and comparing the output of the function with a total standard deviation, a between-subject standard deviation, or a within-subject standard deviation value derived from a mathematical combination of the first risk score, the second risk score, or both. The non-transitory computer-readable medium according to claim 16. **Claim 18** The first set of associated metrics corresponds to a first collection time, the second set of associated metrics corresponds to a second collection time, and determining whether the statistically derived metric exceeds the predetermined threshold includes using stateful logic after the first collection time and the second collection time. The non-transitory computer-readable medium according to claim 16. **Claim 19** The first set of associated metrics corresponds to a first collection time, the second set of associated metrics corresponds to a second collection time, and determining whether the statistically derived metric exceeds the predetermined threshold includes using stateless logic after one of the first collection time or the second collection time. The non-transitory computer-readable medium according to claim 16. **Claim 20** Substituting the first predicted value into the second data field includes determining the first predicted value based on the measured value and a conditional rule associating the first data field with the second data field. The non-transitory computer-readable medium according to claim 16. **Claim 21** In response to determining that the statistically derived metric does not exceed the predetermined threshold, measuring a second value of a plasma protein or nucleic acid, generating an updated first risk score and an updated first set of associated metrics based on the second measured value and the first predicted value, Generating an updated second set of risk scores and an updated set of associated metrics based on the second measured value and the second predicted value; Calculating an updated statistically derived metric based on the updated first risk score, the updated first set of associated metrics, the updated second risk score, and the updated second set of associated metrics; Determining that the updated statistically derived metric exceeds the predetermined threshold; Recommending a predetermined action; further comprising; The method according to claim 1.

Images