Methods and systems for predicting medical condition risk and triggering predictions

Abstract

A computer implementation system and method are provided for determining when to trigger an algorithm or other computer workflow or program based on input feature availability, so that the algorithm produces a meaningful output. [Solution] The provided method includes determining whether the input features are ready to generate algorithmic output. For example, if the input features have measured values ​​and acceptable values, then the input features are ready. If one or more input features do not have measured values ​​and acceptable values, such as having no measured values ​​or being outdated, the provided method includes determining whether the values ​​can be supplemented for one or more input features without available values, and whether such supplementable input features are ready for supplemental values. Once all input features are determined to be ready, supplemental values ​​for the input features are determined as necessary, and the algorithm is triggered.

Description

Technical Field

[0001] Cross - reference to related applications

[0001] This application claims the priority and benefit of U.S. Provisional Patent Application No. 63 / 459,521, filed on April 14, 2023, the content of which is incorporated herein by reference in its entirety.

[0002] Field

[0002] This application generally relates to machine learning algorithms for predicting medical condition risks. More specifically, this application provides a computer - implemented method for generating a predicted risk that a patient has or will develop a medical condition, and a method for triggering such a prediction.

Background Art

[0003] Background

[0003] The challenges faced by clinicians are to evaluate the progression of a patient's medical condition so that preventive or therapeutic measures can be initiated early. Another challenge can be to diagnose medical conditions that are difficult to classify. For example, sepsis is a major health problem in intensive care units (ICUs) worldwide and is an important cause of death. For example, septic shock is a major cause of death with a mortality rate reaching up to 50%. Despite ongoing efforts to establish firm diagnostic criteria for early sepsis diagnosis, physicians still consider it difficult to accurately identify sepsis patients in the ward and / or emergency department. Previously, the systemic inflammatory response syndrome (SIRS) diagnostic criteria were used to detect sepsis in various clinical situations. In the previous consensus definition of sepsis (Sepsis - 2), the presence of an infectious disease, along with at least two diagnostic criteria for sepsis, at least two diagnostic criteria for SIRS, and organ dysfunction were required. However, these criteria have been claimed to be neither sensitive nor specific, hindering the accurate detection and discrimination of sepsis.

[0004]

[0004] Sepsis can be conceptually described as decompensation of a patient due to infection. The tool currently used to determine patient decompensation and thus assess sepsis is the Serial Organ Failure Assessment (SOFA) score. The use of this score for sepsis was based on a new definition of sepsis-3 as "life-threatening organ failure caused by a dysregulated and mitigated host response to infection." As part of this definition, sepsis can be diagnosed when a patient shows an increase of 2 points or more in the SOFA score. The SOFA score is based on an assessment of the function of six systems: the respiratory, nervous, cardiovascular, hepatic, renal, and coagulation systems. A quicker score, the Quick SOFA score (qSOFA), has also been developed. The qSOFA score indicates a very high probability of sepsis if a patient has any two of the following: Glasgow Coma Scale <15, systolic blood pressure <100 mmHg, and respiratory rate >22 / min.

[0005]

[0005] Sepsis is an acute condition characterized by sudden onset of symptoms, and if not treated promptly, it often leads to adverse consequences. Despite the difficulty in detection, early treatment of the infection is highly effective. However, as the infection triggers an invasive host response that gradually progresses to sepsis, the effectiveness of available treatment options decreases, highlighting the importance of early detection of patients at risk of developing sepsis. Because SOFA and qSOFA are designed to screen populations and are not optimized for individual patients, accurately predicting the risk of a particular patient having or developing sepsis remains a challenge. Accurately predicting the risk of a particular patient having or developing other medical conditions (e.g., acute medical conditions) faces similar challenges to those related to sepsis.

[0006]

[0006] In addition, for a particular computer algorithm (e.g., a machine learning algorithm) or other computer workflow or program to provide output (e.g., a prediction), all input features of the algorithm must have a single available value at the time of output generation. Furthermore, the more accurate the values ​​of the input features, the more accurate the output will be. In some applications of computer algorithms, the values ​​of input features may be obtained from different sources, at different times, at different frequencies, over different periods of time, etc. Therefore, in such applications, when a request is made to the computer algorithm to generate output, the input feature values ​​may be missing or outdated, or there may be multiple values ​​for a single input feature. For example, a machine learning algorithm in the medical field (e.g., to predict a diagnosis) relies on physiological measurements and other laboratory values ​​(e.g., clinical observations) as input features. For example, an input feature may be missing at the time of the prediction request because laboratory measurements have been ordered and the results are not yet available. Another input feature may have outdated values ​​because the values ​​were obtained several days before the prediction request was made, and therefore may not reflect the patient's current condition. Yet another input feature may have multiple values ​​because the values ​​are captured every few seconds (e.g., heart rate). Because the time from the start of the algorithm to outputting the result can be very long, the usefulness of the algorithm can be greatly reduced if it can only output a result when all input features have been received. Conversely, if the algorithm can output a result with a subset of the input features, allowing for faster output, too many important features may be missing at the time of the result, thus potentially reducing the accuracy of the output. Therefore, determining when to trigger the algorithm to produce a useful output based on the availability of the algorithm's input features, which must be both (1) sufficiently accurate to the use case and (2) timely generated, is a technical challenge. [Overview of the project] [Means for solving the problem]

[0007] overview

[0007] This disclosure provides novel and innovative computer-aided methods and systems for timely generating clinically accurate risk assessments for patients who have or are at risk of developing a medical condition, such as an acute medical condition. Generating clinically accurate risk assessments involves predicting the risk level (e.g., probability) of a patient having or developing a medical condition through the use of predictive algorithms such as machine learning models. In various embodiments, the medical condition may be an acute condition that is severe and has a sudden onset, and therefore requires timely treatment, such as sepsis. The machine learning model generates an output (e.g., a prediction) from a set of input features, the values ​​of which are determined from various patient measurements or clinical observations, or from complementary processes. The method may include authentication steps to help prevent unauthorized individuals from accessing the patient's protected health information. In some examples, the received input features (e.g., patient measurements or clinical observations) may have different parameters from the input features on which the machine learning model was trained. For example, the received input features may be in different units or evaluated using different scales or methods than the input features on which the machine learning model was trained, because different organizations may utilize different units, scales, or methods. In such cases, a preprocessing method can be used to determine how the received input features are mapped to the input features on which the machine learning model was trained. The mapping determines any transformation functions for those received input features, such as unit or scale / method conversions. Users of the provided system (such as administrators) can configure mappings for specific organizations and / or configure the normal range for each input feature of an organization. When the input features are displayed to the end user, if any of the features are outside the configured normal range, those features will be displayed as such.

[0008]

[0008] The provided method includes a method for determining when or whether to trigger a machine learning model to generate predictions. The triggering method is based on the availability of input features so that the machine learning model produces a timely and meaningful output rather than producing an output from incomplete information or waiting a long time to receive one or more input features. In a particular example, the triggering method includes imputing values ​​into input features where measurements are missing, according to predetermined logic designed so that the machine learning model still produces a meaningful output using the imputed values. The triggering method is also designed to strike a good balance between getting the algorithm to output in a timely manner and maintaining a good accuracy of the output. The triggering method can be used with any suitable algorithm (e.g., a sepsis prediction machine learning model) or with any other suitable computer workflow or program.

[0009]

[0009] In various embodiments, the methods provided may include an active ordering method for determining when or whether to indicate to a clinician (e.g., via the display of a computing device) or to the hospital's electronic medical record that additional parameters should be obtained for a patient, in order to maximize the likelihood that the machine learning model will be triggered in a timely and accurate manner. In certain examples, this active ordering method may include reading stored data such as the status of information about each input feature, including but not limited to: the availability of the feature, the last known measurement of the feature, the last known order time of the feature, and the current time. The active ordering method may include actively alerting the clinician or communicating with the hospital's electronic medical record to trigger rapid ordering of necessary laboratory or patient assessments to provide the machine learning model with the necessary data as timely as possible. The active ordering method may include communicating with a trigger method to indicate that an input feature is expected to be available in a timely manner, so that the trigger method can adjust a time window to wait for the missing feature before obtaining results.

[0010]

[0010] In various embodiments, the methods provided may include feature importance methods for showing the end user the contribution of each feature to the final prediction result displayed to the end user. These feature importance methods enable the user to understand, on a patient-by-patient basis, which features were most important in creating the prediction result. These feature importance methods may include showing the end user which features were missing and imputed, which features were not used, and / or which features were used by the machine learning model to compute the prediction result. These feature importance methods may further include showing the end user the amount by which each feature increased, decreased, or had no effect, regardless of whether it was imputed or not, relative to the predicted risk for patients with the medical condition of interest.

[0011]

[0011] The methods provided may include monitoring model drift in a machine learning model. Model drift can be monitored indirectly by monitoring data drift. Various checks are performed to identify the likelihood of data drift occurring in a machine learning model. A data drift report may be generated showing the results of the checks used. If at least one of the checks fails, the data drift report indicates that data drift has been detected. The end user can then use the data drift report to determine whether data drift has actually occurred and, if so, to decide on corrective actions to take.

[0012]

[0012] In one example, a computer implementation method includes receiving a plurality of input features of a machine learning model. The machine learning model is configured to generate predictions of whether a patient has or is at risk of developing a medical condition. The method further includes determining whether each of the plurality of input features is in a first state or a second state. The first and second states indicate whether an input feature among the plurality of input features is available to trigger the machine learning model. Each of the plurality of input features has its own set of requirements for being in a first state. The method further includes using the machine learning model to generate predictions of whether a patient has or is at risk of developing a medical condition after determining that each of the plurality of input features is in a first state.

[0013]

[0013] The term “clinician” is used throughout this disclosure, but it should be noted that the technique is not intended to be limited to clinicians. Rather, any appropriate medical professional may use or be involved in the technique. For example, nurses, physician assistants, residents, medical students, etc., may initiate requests for algorithmic predictions or measurement orders at their own discretion or at the direction of a clinician.

[0014]

[0014] Additional features and advantages of the disclosed methods and apparatus will be described in the embodiments and drawings for carrying out the inventions below and will become apparent therefrom. The features and advantages described herein are not exhaustive, and many additional features and advantages will become apparent to those skilled in the art, particularly in view of the drawings and description. Furthermore, it should be noted that the language used herein has been chosen primarily for readability and teaching purposes and does not limit the scope of the subject matter of the invention.

[0015] Brief explanation of the drawing

[0015] To provide a further understanding, the accompanying drawings, which are included and incorporated herein and constitute part of herein, illustrate the disclosed embodiments and, together with the description, help to illustrate the principles of the disclosed embodiments. [Brief explanation of the drawing]

[0016] [Figure 1]

[0016] A block diagram of a system for triggering an algorithm stored in a computing device according to one aspect of the present disclosure is shown. [Figure 2]

[0017] A flowchart of a method for generating a predicted risk level for a patient to have or develop a medical condition according to one aspect of the present disclosure is shown. [Figure 3]

[0018] An exemplary user interface for displaying a predicted risk level according to one aspect of the present disclosure is shown. [Figure 4]

[0019] A flowchart of a method for triggering an algorithm stored in a computing device according to one aspect of the present disclosure is shown. [Figure 5A]

[0020] A flowchart of a method for determining the state of input features included in the method of FIG. 4 according to one aspect of the present disclosure is shown. [Figure 5B]

[0021] A flowchart of an exemplary active ordering method included in the method of FIG. 5A according to one aspect of the present disclosure is shown. [Figure 6]

[0022] An exemplary state determination chart for seven input features according to one aspect of the present disclosure is shown. [Figure 7]

[0023] A block diagram of a computer system according to one aspect of the present technology is shown.

Mode for Carrying Out the Invention

[0017] Detailed Description

[0024] Generally, this application provides a novel and innovative system and computer implementation method for predicting the risk level (e.g., probability) of a patient having or developing a medical condition using a predictive algorithm (e.g., a machine learning model). In various embodiments, the medical condition may be an acute condition that is severe and has a sudden onset, and therefore requires timely treatment, such as sepsis. The remainder of the description refers to the risk of a patient having or developing sepsis as an example to illustrate the concepts described herein, but it will be understood that the concepts described herein may be applicable to other appropriate medical conditions, including non-exclusive examples such as acute kidney injury, acute respiratory distress syndrome, stroke, asthma attack, burn, heart attack, infection, and trauma. In at least some embodiments, upon receiving a request (e.g., from a clinician's input device) to trigger a machine learning model and start the process of generating predictions, the provided system begins to listen for input feature values ​​from various measuring devices or user input devices, such as by reading data stored in the memory of the provided system. The stored data is continuously updated as new information is received from the measuring devices or user input devices. In other words, in such embodiments, the provided system continuously receives and stores input features, but does not perform any triggering methods that involve reading the stored input features (e.g., by the system's processor) until a request is received.

[0018]

[0025] When the trigger method is executed, the stored input features are read until a machine learning model generates a prediction or it is determined that the machine learning model cannot generate a prediction based on the available input feature values. In other embodiments, the trigger method may include continuously reading the stored input features. In such embodiments, the trigger method can wait to proceed further in the method rather than reading the stored input features until a request is received. Input features collected by a measurement device or user input device are received in the provided system (e.g., a computing device) almost instantaneously when the input feature values are collected (e.g., measured) such that the provided system receives the input features in real time. When the input features are collected, the trigger method determines whether all of the input features to the machine learning model have available values and when, and as a result, can generate a prediction, which may include filling in values for the input features. The trigger logic is described in more detail below.

[0019]

[0026] In various embodiments, the computer-implemented method includes authentication and authorization procedures. The authentication and authorization procedures are managed through user, access policy, and authentication client configurations and are designed to prevent unauthorized individuals from accessing a patient's protected health information based on the Health Insurance Portability and Accountability Act (HIPAA) of 1996. The authentication and authorization procedures may be performed before enabling a user to initiate a request to trigger a machine learning model or may be performed after receiving a request and before proceeding with prediction generation.

[0020]

[0027] In various embodiments, at least some of the received input feature values ​​may be preprocessed to match the set of parameters of the input features from which the machine learning model was trained, before the input features are input to the machine learning model. For example, the received input feature values ​​may have different parameters from the input features from which the machine learning model was trained. For instance, the received input feature values ​​may be in different units or evaluated using different scales or methods from the input features from which the machine learning model was trained, because different organizations (e.g., hospitals or other healthcare facilities) may utilize different units, scales, or methods. In such examples, preprocessing techniques can be used to determine how the received input feature values ​​are mapped to the input features from which the machine learning model was trained. The mapping determines any transformation function for those received input feature values, such as unit or scale / method conversion. For example, in various embodiments, the received input feature values ​​may be mapped to appropriate model input features by Logical Observation Identifier Names and Codes (LOINC) codes. LOINC codes are, as is known to those skilled in the art, a global standard for identifying medical laboratory observations that facilitates the exchange and pooling of results such as laboratory tests or vital signs for clinical care, outcome management, and research. Users of the provided system (such as administrators) can configure mappings for specific organizations and / or configure normal ranges for each input characteristic of an organization.

[0021]

[0028] As described above, the prediction method includes a novel and innovative trigger method (e.g., a trigger logic executable by a processor) for determining when to trigger the prediction algorithm (e.g., a machine learning model) based on the availability of input features so that the prediction machine learning model produces a meaningful (e.g., accurate) output, which may include the output of the prediction machine learning model being "no result" so that the clinician is notified in a timely manner if the clinician cannot rely on the prediction at that moment based on the available input features and must rely on manual diagnosis or a second prediction request. The following description of the trigger method relates to the algorithm or machine learning model, but the concepts described herein may also apply to any appropriate computer workflow or program that requires timely input features to produce an output. The provided trigger logic includes determining whether the input feature is in a first state, indicating that it is ready to produce an algorithm output (e.g., has available values). For example, the input feature is in a first state if it has measured and acceptable values. If one or more input features lack a measurement value or are outdated, or otherwise lack both a measurement value and an acceptable value, the provided trigger logic includes determining whether the value can be interpolated for one or more input features without available values, and whether such interpolatable input features are in a first state so that they can be interpolated with a replacement value. In other words, if an interpolatable input feature is in a first state, it has available values ​​(i.e., values ​​to be interpolated).

[0022]

[0029] Completion is the process of replacing missing data with replacement values. Completion has the advantage of providing values ​​for all input features necessary for the algorithm to produce results. However, there is no guarantee that the replacement values ​​are an accurate representation of the true values ​​(e.g., measured values) of the missing input features, and therefore, true values ​​are preferable even if obtaining them takes longer than obtaining replacement values. The provided trigger logic defines a completion strategy that balances the need for critical input features, the need for less critical input features, and the time available to receive values ​​for input features (e.g., features ordered by a physician). For example, certain input features may be considered too critical for the accuracy of the algorithm's predictions, and therefore, values ​​may not be completed for these critical input features. Consequently, if these critical input features do not have measured values, the algorithm will not be triggered to generate predictions. For other input features that can be completed, the provided trigger logic may include waiting a predetermined amount of time to receive measured values ​​before completing the values. The completion strategies of the provided trigger logic are described in more detail below.

[0023]

[0030] In various embodiments, if it is not possible to impart missing values ​​for one or more input features, the provided trigger logic may refrain from triggering the algorithm. Otherwise, in such embodiments, an impartial value is determined for each missing input feature, and the algorithm is then triggered to generate an output from the available values, including the input features and the imparted input features. A consistent, meaningful (e.g., accurate) output from the algorithm is obtained by refraining from triggering the algorithm until all input features have values ​​available in these embodiments, and only by imparting specific input features in specific cases. In other embodiments, the algorithm may be triggered when one or more input features are missing and cannot be imparted. For example, it may be predetermined, or determined as part of the trigger method, that a particular input feature, or a particular combination of an input feature and an impartial input feature, generates a consistently meaningful (e.g., accurate) prediction without one or more missing input features.

[0024]

[0031] The provided trigger logic can be implemented by any suitable computer algorithm with input features, but one exemplary implementation involves a predictive algorithm in a medical context, such as a machine learning algorithm trained to predict whether a patient has or is at risk of developing sepsis. Measuring patient-specific input features for the algorithm depends on the clinical setting (e.g., hospital ordering policies) and the decisions of the healthcare provider (e.g., physician), resulting in significant variation in the availability, frequency, and timeliness of patient-specific input features. For example, a patient may have no measurements of an input feature because a physician chooses not to order a comprehensive metabolic panel for the patient. In another example, a patient may have multiple measurements of an input feature because a heart rate monitor provides high-frequency measurements. In yet another example, a patient may have measurements of an input feature taken much earlier (e.g., 4 days) than when the request for algorithmic prediction is received, and therefore the measurements may not provide an accurate representation of the patient's current health status. In yet another example, measurements of an input parameter may not be available until much later (e.g., 2 days) than when the request for algorithmic prediction is received because laboratory tests take time to complete. All of these exemplary variations in the availability of input features present challenges in rapidly generating meaningful (e.g., clinically accurate) algorithmic predictions of whether a patient has sepsis or is at risk of developing sepsis. The methods provided address these challenges, as will be detailed below.

[0025]

[0032] The remainder of the description of the trigger logic or trigger method provided herein will be described with reference to exemplary algorithms or machine learning models for predicting the risk of patients having or developing sepsis, in order to help describe the embodiments of the trigger logic or trigger method described herein, however, as stated above, the trigger logic or trigger method provided is applicable to any suitable algorithm, computer workflow, or program.

[0026]

[0033] In various embodiments, the provided method may include active ordering logic, which can be executed by a processor (e.g., processor 122), for determining when or whether to indicate to a clinician (e.g., via a computing device display) or to the hospital's electronic medical record that additional parameters should be obtained for a patient, in order to maximize the likelihood that the machine learning model will be triggered in a timely and accurate manner. In a particular example, this active ordering logic may read stored data such as the status of information about each input feature, including but not limited to: the availability of the feature, the last known measurement of the feature, the last known order time of the feature, and the current time. This active ordering logic may actively alert the clinician or communicate with the hospital's electronic medical record to trigger rapid ordering of necessary laboratory or patient assessments to provide the machine learning model with the necessary data as timely as possible. This active ordering logic may include communicating with a trigger method to indicate that an input feature is expected to be available in a timely manner, so that the trigger method can adjust a time window to wait for the missing feature before obtaining results.

[0027]

[0034] In various embodiments, the provided methods may include feature importance methods for showing the end user the contribution of each feature to the final prediction result displayed to the end user. These feature importance methods enable the user to understand, on a patient-by-patient basis, which features were most important in creating the prediction result. These feature importance methods may include showing the end user which features were missing and imputed, which features were not used, and / or which features were used by the machine learning model to compute the prediction result. These feature importance methods may further include showing the end user the amount by which each feature increased, decreased, or had no effect, regardless of whether it was imputed or not, relative to the predicted risk for patients with the medical condition of interest.

[0028]

[0035] In various embodiments, the methods provided include determining whether a machine learning model is exhibiting model drift. Model drift is a decrease in the predictive performance of a model due to systemic or statistical changes, which may result from data drift and / or distribution shifts, however, not all detected data drift or distribution shifts cause a decrease in the predictive performance of a machine learning model. Data drift relates to unexpected or undocumented changes to data structures, semantics, and infrastructure resulting from modern data architectures. These changes are often caused by problems in the data collection process, regardless of the training dataset. For example, examples of data drift include: (1) an undocumented change from one LOINC code to a new LOINC code for an input parameter at a location where a machine learning model is implemented; (2) a failure in the deployment of an IT system at that location causing the machine learning model to stop receiving heart rate measurements from that location; (3) the use of an uncalibrated body temperature probe that causes a shift in body temperature readings; and (4) values ​​stored for use in the machine learning model being inadvertently truncated. Detecting data drift may include generating a data drift report. A data drift report is a report for detecting potential data drift, showing the results of a statistical examination comparing a baseline dataset with an observed dataset. In various embodiments, model drift may be monitored indirectly by monitoring data drift using the methods provided herein.

[0029]

[0036] Distribution shifts are associated with unexpected changes in inputs, labels, and / or the relationships between input and label features in newly observed data that do not match the training dataset of a machine learning model. Distribution shifts occur when the training data of a machine learning model no longer represents the newly observed data. The events that cause distribution shifts are due to environmental or population changes, not software issues. Examples of distribution drift include (1) a new machine being able to obtain measurements of input features at lower levels than observed in the training dataset, (2) a change in the incidence of septic patients in the new observed dataset, and (3) a new drug reducing procalcitonin (PCT) levels that would be routinely measured in septic patients, and therefore changing the relationships identified in the training dataset.

[0030] Prediction system

[0037] Figure 1 shows an exemplary computer network 10 that may be used to carry out various embodiments of this application. Generally, the computer network 10 includes various devices that communicate and function together in collecting, transmitting, requesting, and / or processing data related to the generation of the output of algorithms stored in computing devices. As shown, a communication network 106 enables communication in the computer network 10. The communication network 106 may include, but is not limited to, one or more wireless networks, such as local area networks (LANs), wireless local area networks (WLANs), personal area networks (PANs), campus area networks (CANs), metropolitan area networks (MANs), wide area networks (WANs), wireless wide area networks (WWANs), global systems for mobile communications (GSM), digital advanced mobile telephone services (D-Amps), Bluetooth, Wi-Fi, fixed wireless data, 2G, 2.5G, 3G, 4G, LTE networks, enhanced data rates (EDGE) of the evolution of GSM, general-purpose packet radio services (GPRS), enhanced GPRS, messaging protocols such as TCP / IP, SMS, MMS, enhanced messaging and presence protocol (XMPP), real-time messaging protocol (RTMP), instant messaging and presence protocol (IMPP), instant messaging, USSD, IRC, or any other wireless data network or messaging protocol. The communication network 106 may also include wired networks.

[0031]

[0038] An exemplary computer network 10 includes a computing device 100 that communicates with one or more measuring devices 102 and one or more user input devices 104. The computing device 100 may be a server or other suitable computer system having a processor 122, such as a computer system 700 (Figure 7), and memory 118. Memory 118 stores an algorithm (e.g., a machine learning model 120). Alternatively, the machine learning model 120 may be stored in the server or other suitable computing device that communicates with the computing device 100. The machine learning model 120 may be a suitable supervised, semi-supervised, or unsupervised machine learning model. For example, the machine learning model 120 may be one of a group of machine learning models, including random forests, logistic regression, gradient boosting machines, neural networks, support vector machines, k-means clustering, and hierarchical clustering models. In one example, a random forest model was trained by maximizing the area under the receiver operating characteristic curve (AUROC) between predictions and training labels. In at least some embodiments, memory 118 further stores patient data 124. Patient data 124 may include arbitrary patient data related to input features of the machine learning model 120. Patient data 124 may be continuously updated as new data is received from various measurement devices 102 and user input devices 104.

[0032]

[0039] In various embodiments, the computing device 100 includes an authentication module 110, a preprocessing module 112, a trigger logic module 114, and a data drift module 116. Each of the authentication module 110, preprocessing module 112, trigger logic module 114, and data drift module 116 may be implemented by software executed by a processor 122.

[0033]

[0040] The measuring device 102 may be any suitable device for collecting and transmitting data. In one example, the measuring device 102 may be a heart rate monitor that collects information about a patient's heart rate and transmits that information to a computing device 100. In another example, the measuring device 102 may be any suitable machine that collects or communicates with a machine that collects information related to determining whether a patient has or is at risk of developing sepsis (for example, values ​​such as measurements or clinical observations of any of the input features of the machine learning model 120 described later). Where used below, the term “measurement” and its plural form include manual measurements, machine-derived measurements, and clinical observations.

[0034]

[0041] The user input device 104 can be any suitable device, such as a desktop computer, that can receive and transmit information via user input. For example, a physician can order a comprehensive metabolic panel for a patient from a first user input device 104 (e.g., a tablet computer), and a laboratory technician can input the panel results into a second user input device 104 (e.g., a desktop computer). In this example, an instruction that an order has been placed is sent from the first user input device 104 to the computing device 100, and the panel results are sent from the second user input device 104 to the computing device 100.

[0035]

[0042] The machine learning model 120 generates an output from multiple input features. For example, the machine learning model 120 may generate a prediction of the risk level of a patient who has or will develop a medical condition such as sepsis from multiple input features. The values ​​of the multiple input features can be received by the computing device 100 from one or more measurement devices 102 and / or one or more user input devices 104 and stored in memory 118 as patient data 124. For example, the input features are patient measurements or input information.

[0036]

[0043] The input features of the machine learning model 120 include at least some (e.g., all) of the following: demographic measurements, patient assessments, vital signs, hematological laboratory values, chemical laboratory values, and sepsis-specific biomarker concentrations. In various embodiments, the input features include, or consist of, some or all of the following group of input features: age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessments (e.g., Glasgow Coma Scale, Full Outline of Unresponsiveness (4), etc.), clinical decision support alerts, clinician or other chart notes, diagnostic codes (e.g., ICD-10), procedures performed on the patient (including CPT codes), the patient's current drug therapy (including NDC codes), interventions, patient care settings (e.g., ICU, ED, hospital floor, outpatient clinic, etc.), imaging (e.g., X-ray, CT scan, MRI, fMRI or ultrasound), electrography (e.g., EEG or EKG), endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, inspired oxygen concentration (FiO2), white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, procalcitonin, C-reactive protein, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers (e.g., monocyte distribution width, Cytovale®). IntelliSep®), clinical severity measurements and composite materials (e.g., Serial Organ Failure Assessment (SOFA) score, SOFA component score, Charlson Comorbidity Index, or Acute Physiology and Chronic Health Assessment (APACHE) I and II), biopsy results, microbiological tests (e.g., culture, PCR test, or antigen test), protein biomarkers (e.g., angiopoietin-1, angiopoietin-2, C-reactive protein, cystatin C, D-dimer, E-selectin, fractalkine, FLT3-ligand, GCSF, GDF15, GMCSF, granzyme-B, IFN-alpha, IFN-gamma, IL1-beta, IL1-RA, IL-2, IL-4, IL-6, IL-7, IL-8, IL-10, IL-15, IP-10, lactate dehydrogenase (LDH),Lipopolysaccharide-binding protein (LBP), leptin, MCP1, MIP1-alpha, MIP1-beta, MIP3-alpha, NGAL, pancreatic stone protein, PDL-1, pentraxin-3, procalcitonin, protein C, S100b, TGF-alpha, tropodulin, tissue factor, TNF-alpha, TRAIL, TREM-1, troponin, VCAM-1, and / or VEG-F), genomic biomarkers (e.g., SNPs in CD14, TLR1, TLR2, TLR4, TLR6, TNF-alpha, IL-6, factor V, factor XII, FER, MAN2A1), gene expression or transcriptome biomarkers (e.g., CEACAM4, LAMP1, PLAC8, PLA2G7, CEACAM1, ZDHH) C19, C9orf95, GNA15, BATF, C3AR1, KIAA1370, TGFBI, MTCH1, RPGRIP1, HLA-DPB1, nuclear factor-kappa B (NF-kB), heme oxygenase-1 (HO-1), FCGR2C, CRP , IL-6, IL-10, IL-12, TNF, LY6E, LAMP3, ISG15, USP18, RSAD2, IFI44L, NCOA7, CMPK2, IFI44, BATF2, GBP4, IFIH1, GMPR, CXCL10, IFI27, I SG15, CCL2, OAS3, P2RY14, GCH1, CD274, ID01, ARL4A, BPGM, TNFAIP3, TNFAIP6, IL1R1, PROK2, RGS1, CXCR6, RTP4, TLR3, FZD5, KCNJ2, IR AK2, RGS16, CXCL8, OLR1, CCRL2, SPP1, ST3GAL4, CST7, IL1R2, IL4R, GALM, CTLA4, SOCS1, BCL2L1, TGM2, SLC39A8, CA2, PNP, GATA1, SLC1 A5, RGS16, CD44, PDGFC, CSF2RA, CXCL1, CSF3R, SOCS3, IL18R1, PIM1, LEPR, IL1B, CSF2RB, ITGB3, STAT1, FAS, EFNA1, IER3, EETS2, SLC2 A3, BCL2A1, PFKFB3, CEBPB, BIRC2, ATF3, TUBB2A, G0S2, MERTK, BIK, QSOX1, PYGL, TGFA, P4HA2, LDHA, IRS2, TSPO, HGF, GADD45A, TGFBR3,CD38, PRF1, FASLG, TIMP3, ANKH, LGALS3, PPP2R5B, GPX1, BNIP3L, BCL2L1, KLF7, FOSL2, ADM, MXI1, SELENBP1, PGF, FOXO3, NEDD4L, SLC2A1, SIAH 2, MMP9, CXCR3, CD8A, IFNG, CD8B, JAK2, CCL4, ICAM1, WARS1, KRT1, GPR65, DYRK3, ACHE, ARG1, LCN2, LTF, OLFM4, HLA-DMB, YKT6, PDE4B, TWISTNB , BTN2A2, ZBTB33, PSMB9, CAMK4, TMEM19, SLC12A7, TP53BP1, PLEKHO1, SLC25A22, FRS2, CD24, STX1A, KCNMB4, CRISP2, HTRA1, PPL, RHBDF2, ZCCHC4, DDX6, SENP5, RAPGEF1, DTX2, RELB, DYRK2, CCNB1IP1, TDRD9, ZAP70, ARL14EP, MDC1, and / or ADGRE3), or composite biomarker scores (e.g., SeptiCyte® LAB and SeptiCyte® RAPID, Inflammatix® TriVerity®, or MeMed BV®).

[0037]

[0044] In various embodiments, one or more input features within the above group may encompass different variations of an input feature (e.g., an input feature derived from another input feature). For example, a heart rate input feature may have variations including the maximum heart rate over the past two hours, the change in heart rate over the past six hours, the variance of heart rate measurements over the past hour, one or more features extracted from waveform analysis of heart rate monitor readings, or other appropriate variations. In some embodiments, the input features of the machine learning model 120 may include two or more variations of an input feature. For example, the input features of the machine learning model 120 may include both the maximum heart rate over the past two hours and the variance of heart rate measurements over the past hour.

[0038]

[0045] For example, the input features of the machine learning model 120 include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group.

[0039]

[0046] The first group includes the following input features: age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment (e.g., Glasgow Coma Scale, Full Outline of Unresponsiveness (4)), clinical decision support alerts, clinician or other chart notes, diagnostic codes (e.g., ICD-10), procedures performed on the patient (including CPT codes), patient's current medications (including NDC codes), interventions, patient care setting (e.g., ICU, ED, hospital floor, outpatient clinic), imaging (e.g., X-ray, CT scan, MRI, fMRI or ultrasound), electrography (e.g., EEG or EKG), endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, inspired oxygen concentration (FiO2).

[0040]

[0047] The second group includes the following input features: white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers (e.g., monocyte distribution width, Cytovale® IntelliSep®), clinical severity measures and composite materials (e.g., Serial Organ Failure Assessment (SOFA) score, SOFA component score, Charlson Comorbidity Index, or Acute Physiology and Chronic Health Assessment (APACHE) I and II), biopsy results, and microbiological tests (e.g., culture, PCR test, or antigen test).

[0041]

[0048] The third group includes the following input features: procalcitonin, C-reactive protein, protein biomarkers (e.g., angiopoietin-1, angiopoietin-2, C-reactive protein, cystatin C, D-dimer, E-selectin, fractalkine, FLT3-ligand, GCSF, GDF15, GMCSF, granzyme-B, IFN-alpha, IFN-gamma, IL1-beta, IL1-RA, IL-2, IL-4, IL-6, IL-7, IL-8, IL-10, IL-15, IP-10, lactate dehydrogenase (LDH), lipopolysaccharide-binding protein (LBP), leptin, MCP1, MIP1-alpha, MIP1-beta, MIP3-alpha, NGAL, pancreatic stone protein, PDL-1, pentraxin-3, procalcitonin, protein C, S100b) TGF-alpha, tropodulin, tissue factor, TNF-alpha, TRAIL, TREM-1, troponin, VCAM-1, and / or VEG-F), genomic biomarkers (e.g., SNPs in CD14, TLR1, TLR2, TLR4, TLR6, TNF-alpha, IL-6, factor V, factor XII, FER, MAN2A1), gene expression or transcriptome biomarkers (e.g., CEACAM4, LAMP1, PLAC8, PLA2G7, CEACAM1, ZDHHC19, C9orf95, GNA15, BATF, C3AR1, KIAA1370, TGFBI, MTCH1, RPGRIP1, HLA-DPB1, nuclear factor-kappa B (NF-κB), heme oxygenase-1 (HO- 1), FCGR2C, CRP, IL-6, IL-10, IL-12, TNF, LY6E, LAMP3, ISG15, USP18, RSAD2, IFI44L, NCOA7, CMP K2, IFI44, BATF2, GBP4, IFIH1, GMPR, CXCL10, IFI27, ISG15, CCL2, OAS3, P2RY14, GCH1, CD274, ID 01, ARL4A, BPGM, TNFAIP3, TNFAIP6, IL1R1, PROK2, RGS1, CXCR6, RTP4, TLR3, FZD5, KCNJ2, IRAK2, RGS16, CXCL8, OLR1, CCRL2, SPP1, ST3GAL4, CST7, IL1R2, IL4R, GALM, CTLA4, SOCS1, BCL2L1, TGM2,SLC39A8, CA2, PNP, GATA1, SLC1A5, RGS16, CD44, PDGFC, CSF2RA, CXCL1, CSF3R, SOCS3, IL18R1, PIM1, LEPR, IL 1B, CSF2RB, ITGB3, STAT1, FAS, EFNA1, IER3, EETS2, SLC2A3, BCL2A1, PFKFB3, CEBPB, BIRC2, ATF3, TUBB2A, G0 S2, MERTK, BIK, QSOX1, PYGL, TGFA, P4HA2, LDHA, IRS2, TSPO, HGF, GADD45A, TGFBR3, CD38, PRF1, FASLG, TIMP3 , ANKH, LGALS3, PPP2R5B, GPX1, BNIP3L, BCL2L1, KLF7, FOSL2, ADM, MXI1, SELENBP1, PGF, FOXO3, NEDD4L, SLC2A 1, SIAH2, MMP9, CXCR3, CD8A, IFNG, CD8B, JAK2, CCL4, ICAM1, WARS1, KRT1, GPR65, DYRK3, ACHE, ARG1, LCN2, LT F, OLFM4, HLA-DMB, YKT6, PDE4B, TWISTNB, BTN2A2, ZBTB33, PSMB9, CAMK4, TMEM19, SLC12A7, TP53BP1, PLEKHO 1. SLC25A22, FRS2, CD24, STX1A, KCNMB4, CRISP2, HTRA1, PPL, RHBDF2, ZCCHC4, DDX6, SENP5, RAPGEF1, DTX2, RELB, DYRK2, CCNB1IP1, TDRD9, ZAP70, ARL14EP, MDC1, and / or ADGRE3), or composite biomarker scores (e.g., SeptiCyte® LAB and SeptiCyte® RAPID, Inflammatix® TriVerity®, or MeMed BV®).

[0042]

[0049] For example, the input features of the machine learning model 120 include age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group; white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate from the second group; and procalcitonin and C-reactive protein from the third group.

[0043]

[0050] In another example, the input features of the machine learning model 120 include age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group; white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate from the second group; and procalcitonin, C-reactive protein, and IL-6 from the third group.

[0044]

[0051] In another example, the input features of machine learning model 120 include age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group; white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate from the second group; and procalcitonin, C-reactive protein, angiopoietin-2, IL-6, IL-8, IP-10, and pentraxin-3 from the third group.

[0045]

[0052] In another example, the input features of the machine learning model 120 include age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group; white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate from the second group; and procalcitonin, C-reactive protein, angiopoietin-2, IL-6, IL-8, IP-10, PDL-1, TREM-1, and pentraxin-3 from the third group.

[0046]

[0053] In another example, the input features of machine learning model 120 include age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group; white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate from the second group; and procalcitonin, C-reactive protein, angiopoietin-1, angiopoietin-2, E-selectin, fractalkine from the third group. Contains FLT3-ligand, GCSF, GMCSF, Granzyme-B, IFN-alpha, IFN-gamma, IL1-beta, IL1-RA, IL-2, IL-4, IL-6, IL-7, IL-8, IL-10, IL-15, IP-10, leptin, MCP1, MIP1-alpha, MIP1-beta, MIP3-alpha, NGAL, PDL-1, pentraxin-3, TGF-alpha, tromodulin, tissue factor, TNF-alpha, TRAIL, TREM-1, VCAM-1, and VEG-F.

[0047]

[0054] In another example, the input features of the machine learning model 120 include age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, sex at birth, and body temperature from the first group; creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, calcium, hematocrit, and hemoglobin concentration from the second group; and angiopoietin-2, IL-8, IP-10, and pentraxin-3 from the third group.

[0048]

[0055] Demographic measurements include age, sex at birth, race, and ethnicity. Age can be calculated by subtracting the patient's date of birth from the current date. Age has traditionally been considered a risk factor because the incidence of sepsis increases with age. This may be associated with an increase in chronic diseases that are important in the progression or presence of sepsis.

[0049]

[0056] Patient assessments include the patient's past medical history, current illness or symptoms, neurological assessments (e.g., Glasgow Coma Scale, Full Outline of Unresponsiveness (4)), clinical decision support alerts, clinician chart notes or other chart notes, patient-related diagnostic codes (e.g., ICD-10), procedures performed on the patient (including CPT codes), the patient's current medications (including NDC codes), interventions, patient care settings (e.g., ICU, ED, hospital floor, outpatient clinic, etc.), imaging (e.g., X-ray, CT scan, MRI, fMRI, or ultrasound), electrographic data (e.g., EEG or EKG), endoscopic data, and clinical severity measurements and composite materials (e.g., Continuing Organ Failure Assessment (SOFA) score, SOFA component score, Charlson Comorbidity Index, or Acute Physiology and Chronic Health Assessment (APACHE) I and II).

[0050]

[0057] As used herein, a clinical decision support alert is a notification or action that informs of a patient's medical condition. The notification may result from a software application or a human-initiated action. An example of an alert from a software application would be a machine learning algorithm (e.g., an early sepsis warning system) that uses data from a patient's medical records to generate alerts for patients at high risk of sepsis. An example of an alert from a human-initiated action would be a healthcare professional contacting another healthcare professional's communication device regarding a patient's medical condition.

[0051]

[0058] As used herein, intervention is a treatment, procedure, or other action that a healthcare professional may take to prevent or treat a medical condition or to improve a patient's health. Interventions include, but are not limited to, drug therapies (e.g., antibiotics, vasopressors, immunosuppressants, topical creams), surgeries (e.g., amputations), vaccinations, education and behavioral training, and environmental changes.

[0052]

[0059] The Glasgow Coma Scale score is a measure that gives a score to a clinical scale used to reliably measure a person's level of consciousness. Altered mental state (e.g., as measured by the Glasgow Coma Scale (GCS)) is a useful category for predicting sepsis, as measured in both the Continuing Organ Failure Assessment (SOFA) score and the Quick SOFA score (qSOFA). For example, both the SOFA score and the qSOFA score utilize the Glasgow Coma Scale score.

[0053]

[0060] Vital signs include systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, and blood oxygen saturation. Systolic blood pressure is the pressure exerted when the heart beats and blood is pumped into the arteries. It can be measured using a machine such as a blood pressure monitor. Systolic blood pressure is used to calculate mean arterial pressure, which is intended to assess the cardiovascular component of the SOFA score.

[0054]

[0061] Diastolic blood pressure is the pressure in the arteries when the heart is at rest between beats. It can be measured using a machine such as a blood pressure monitor. Diastolic blood pressure is used to calculate mean arterial pressure, which is intended to assess the cardiovascular component of the SOFA score.

[0055]

[0062] Body temperature is a specific temperature within the body. Body temperature can be measured in various ways, one of which is with a thermometer. Body temperature can be used to identify whether a patient has a fever, as fever patterns can be used to identify possible causes of infection, as infectious fever (fever fluctuating above 1.1°C and not returning to normal) is often caused by infection. Body temperature is also used in the criteria for systemic inflammatory response syndrome (SIRS), which is part of the older Sepsis-2 definition.

[0056]

[0063] Respiratory rate is defined as the number of heartbeats per minute. Respiratory rate can be measured manually or via a device such as a pulse oximeter. Abnormal respiratory rate may indicate an irregular amount of oxygen available to the patient. Tachypnea (increased respiratory rate) is commonly observed in patients with sepsis and is associated with increased 28-day mortality in sepsis patients. Respiratory rate is also associated with SIRS, which was part of the earlier Sepsis-2 definition.

[0057]

[0064] Heart rate is measured as the number of heartbeats over a specific period of time. Heart rate can be measured manually or through a device such as a heart rate monitor. Irregular heart rate may be associated with poor blood circulation and either low or high blood pressure. Heart rate is also associated with SIRS, which was part of the earlier Sepsis-2 definition.

[0058]

[0065] Blood oxygen saturation is a measure of the amount of oxygen-carrying hemoglobin in the blood relative to oxygen-uncarrying hemoglobin. Blood oxygen saturation can be calculated and measured in many ways, including using a pulse oximeter. Blood oxygen saturation is a non-invasive method for estimating arterial blood oxygen saturation (SAO2), which is used in the calculation of PaO2 / FiO2 for the SOFA score of the respiratory system.

[0059]

[0066] Hematological laboratory values ​​include white blood cell count, platelet count, lymphocyte count, and neutrophil count. White blood cell count is a measure of the number of white blood cells in a patient's blood. White blood cell count can be measured using hematological laboratory tests. Irregularities in the white blood cell count may indicate that a person is fighting an infection. White blood cell count is associated with SIRS, which was part of the earlier Sepsis-2 definition.

[0060]

[0067] Platelet count is a measure of the number of platelet cells in a patient's blood. Platelet count can be measured using hematological laboratory tests. Irregularities in platelet count may indicate whether an individual has difficulty with coagulation. Platelets are involved in both hemostasis and immune responses and play a crucial role in sepsis. Platelets are used to assess the coagulation component of the SOFA score.

[0061]

[0068] Lymphocyte count is a measure of the number of lymphocyte cells in a patient's blood. Lymphocyte count can be measured using hematological laboratory tests. Lymphocytes are a type of white blood cell that is part of the immune system. Irregularities in lymphocyte count may indicate the presence of an infection. The neutrophil-to-lymphocyte ratio (NLR) can be calculated using lymphocytes in combination with neutrophils. NLR is a metric associated with mortality in sepsis patients, as non-survivors show a significantly higher NLR.

[0062]

[0069] Neutrophil count is a measure of the number of neutrophil cells in a patient's blood. Neutrophil count can be measured using hematological laboratory tests. Neutrophils are a type of granulocyte, a type of white blood cell that is part of the immune system. Irregularities in lymphocyte count may indicate the presence of infection. The neutrophil-to-lymphocyte ratio (NLR) can be calculated using lymphocytes in combination with neutrophils. NLR is a metric associated with mortality in sepsis patients, as non-survivors show a significantly higher NLR.

[0063]

[0070] Hematocrit is the percentage of red blood cells by volume in the blood. The percentage of hematocrit can be measured using hematological laboratory tests. Patients with low hematocrit are associated with higher disease severity. Hematocrit is associated with the prognosis of sepsis patients admitted to the ICU.

[0064]

[0071] Hemoglobin concentration is a measure of the hemoglobin protein present in red blood cells. Hemoglobin concentration can be measured using hematological laboratory tests. Hemoglobin gives cells the ability to deliver oxygen. Acute decreases in hemoglobin are associated with patients with sepsis and septic shock.

[0065]

[0072] Chemical laboratory values ​​include creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, glucose, calcium, lactate, laboratory-derived biomarkers (e.g., monocyte distribution width, Cytovale®, IntelliSep®), biopsy results, and microbiological tests (e.g., culture, PCR, or antigen tests).

[0066]

[0073] Creatinine is defined as the concentration of creatinine in the blood. Blood creatinine levels can be measured using a chemical laboratory panel. High blood creatinine levels indicate that the renal system may not be filtering creatinine into the urine for elimination. Creatinine is used to assess the renal system component of the SOFA score.

[0067]

[0074] Blood urea nitrogen (BUN) is the amount of blood urea nitrogen remaining in the blood. BUN levels in the blood can be measured using a chemical laboratory panel. Blood urea nitrogen is a waste compound that is transferred from the blood to the urine by the kidneys. High BUN levels may indicate that the renal system is not functioning effectively. Elevated BUN has been associated with the development or presence of sepsis.

[0068]

[0075] Potassium is measured as the concentration of potassium ions in the blood. Blood potassium levels can be obtained by performing a chemical laboratory test. Potassium plays a role in maintaining intracellular fluid levels in the body. Potassium imbalance can lead to fluid imbalances. Fluid imbalances have been associated with many patients admitted to the ICU, and fluid resuscitation is effective in treating patients with severe sepsis or septic shock.

[0069]

[0076] Chloride is measured as the concentration of chloride ions in the blood. Blood chloride levels can be obtained by performing a chemical laboratory test. Chloride plays a role in maintaining intracellular fluid levels in the body. Chloride imbalances can lead to fluid imbalances. Fluid imbalances have been associated with many patients admitted to the ICU, and fluid resuscitation is effective in treating patients with severe sepsis or septic shock.

[0070]

[0077] Total carbon dioxide is a measure of carbon dioxide found in the blood. Total carbon dioxide can be measured in the blood using chemical laboratory tests. Total carbon dioxide can be used to identify metabolic acidosis, a respiratory system dysfunction. An imbalance in total carbon dioxide may be associated with respiratory system irregularities. Elevated total carbon dioxide levels have been associated with increased mortality in patients with sepsis.

[0071]

[0078] Sodium is measured as the concentration of sodium ions in the blood. Blood sodium levels can be obtained by performing chemical laboratory tests. Sodium plays a role in maintaining intracellular fluid levels in the body. Sodium imbalance can lead to fluid imbalances. Fluid imbalances have been associated with many patients admitted to the ICU, and fluid resuscitation may be effective in treating patients with severe sepsis or septic shock.

[0072]

[0079] Albumin is measured as the concentration of albumin present in the blood. Blood albumin levels can be obtained through chemical laboratory testing. Albumin is a protein synthesized by the liver that minimizes capillary leakage into body tissues. Irregularities in albumin levels may indicate problems with the liver or kidneys. Serum albumin has been identified as a predictor of mortality in patients with sepsis.

[0073]

[0080] Bilirubin is measured as the concentration of bilirubin present in the blood. Blood bilirubin levels can be obtained through chemical laboratory tests. Bilirubin is a protein produced during the breakdown of red blood cells. In healthy patients, bilirubin is a waste product filtered by the liver. Bilirubin is used to assess liver components in SOFA score calculations.

[0074]

[0081] Glucose concentration is the amount of glucose present in the blood. Glucose concentration can be measured by performing chemical laboratory tests. In sepsis, blood flow in the body decreases. If glucose cannot overcome the interstitial space and reach its target, this can contribute to hyperglycemia.

[0075]

[0082] Calcium is measured as the concentration of calcium ions in the blood. Blood calcium levels can be obtained by performing chemical laboratory tests. Calcium plays a role in maintaining intracellular fluid levels in the body. Calcium imbalances can lead to fluid imbalances. Fluid imbalances have been associated with many patients admitted to the ICU, and fluid resuscitation may be effective in treating patients with severe sepsis or septic shock.

[0076]

[0083] Lactate is measured as the concentration of lactate present in the blood. Lactate can be measured by laboratory testing. Lactate (or lactate) is produced by tissues during hypoxic conditions. Accumulation of lactate in the blood may indicate that a patient is experiencing organ failure or shock. Lactate appeared as part of the septic shock label in the Sepsis-3 definition.

[0077]

[0084] As used herein, laboratory-derived biomarkers are mathematically derived measurements that measure a patient's phenotype regarding a biological phenomenon. Laboratory-derived biomarkers are measured by taking a sample, measuring one or more biological properties of interest, and using mathematical methods to generate an index or score that correlates with the biological phenomenon. One example is Cytovale® IntelliSep®, which measures the biophysical properties of a single cell by applying controlled pressure to a leukocyte and collecting high-fidelity images. The measured biophysical properties are aggregated and converted into the Intellisep Index® (ISI), which helps in the diagnosis of sepsis in patients suspected or confirmed to have an infection.

[0078]

[0085] As used herein, sepsis-specific biomarkers are biomarkers that have been shown to have high clinical utility for diagnosing or treating patients who have sepsis or are at high risk of developing it. Sepsis-specific biomarkers may not necessarily only demonstrate clinical utility for sepsis, but may also demonstrate utility for multiple medical conditions in parallel with sepsis. Sepsis-specific biomarker concentrations include procalcitonin, C-reactive protein, protein biomarkers (e.g., angiopoietin-1, angiopoietin-2, cystatin C, D-dimer, E-selectin, fractalkine, FLT3-ligand, GCSF, GDF15, GMCSF, granzyme-B, IFN-alpha, IFN-gamma, IL1-beta, IL1-RA, IL-2, IL-4, IL-6, IL-7, IL-8, IL-10, IL-15, IP-10, lactate dehydrogenase (LDH), lipopolysaccharide-binding protein (LBP), leptin, MCP1, MIP1-alpha, MIP1-beta, MIP3-alpha, NGAL, pancreatic stone protein, PDL-1, pentraxin-3, protein C, S100b) TGF-alpha, tropodulin, tissue factor, TNF-alpha, TRAIL, TREM-1, troponin, VCAM-1, and / or VEG-F), genomic biomarkers (e.g., SNPs in CD14, TLR1, TLR2, TLR4, TLR6, TNF-alpha, IL-6, factor V, factor XII, FER, MAN2A1), gene expression or transcriptome biomarkers (e.g., CEACAM4, LAMP1, PLAC8, PLA2G7, CEACAM1, ZDHHC19, C9orf95, GNA15, BATF, C3AR1, KIAA1370, TGFBI, MTCH1, RPGRIP1, HLA-DPB1, nuclear factor-kappa B (NF-kB), heme oxygenase-1 (HO-1), FCGR2C, CRP, IL-6, IL-10, IL-12, TNF, LY6E, LAMP3, ISG15, USP1 8, RSAD2, IFI44L, NCOA7, CMPK2, IFI44, BATF2, GBP4, IFIH1, GMPR, CXCL10, IFI27, ISG15, CCL2, OAS3, P2RY14, GCH1, CD274, ID01, ARL4A, BPGM, TNFAIP3,TNFAIP6, IL1R1, PROK2, RGS1, CXCR6, RTP4, TLR3, FZD5, KCNJ2, IRAK2, RGS16, CXCL8, OLR1, CCRL2, SPP1, ST3GAL4, CST7, IL1R2, IL 4R, GALM, CTLA4, SOCS1, BCL2L1, TGM2, SLC39A8, CA2, PNP, GATA1, SLC1A5, RGS16, CD44, PDGFC, CSF2RA, CXCL1, CSF3R, SOCS3, IL18R 1, PIM1, LEPR, IL1B, CSF2RB, ITGB3, STAT1, FAS, EFNA1, IER3, EETS2, SLC2A3, BCL2A1, PFKFB3, CEBPB, BIRC2, ATF3, TUBB2A, G0S2, MERTK, BIK, QSOX1, PYGL, TGFA, P4HA2, LDHA, IRS2, TSPO, HGF, GADD45A, TGFBR3, CD38, PRF1, FASLG, TIMP3, ANKH, LGALS3, PPP2R5B, GPX1, BNIP3L, BCL2L1, KLF7, FOSL2, ADM, MXI1, SELENBP1, PGF, FOXO3, NEDD4L, SLC2A1, SIAH2, MMP9, CXCR3, CD8A, IFNG, CD8B, JAK 2, CCL4, ICAM1, WARS1, KRT1, GPR65, DYRK3, ACHE, ARG1, LCN2, LTF, OLFM4, HLA-DMB, YKT6, PDE4B, TWISTNB, BTN2A2, ZBTB33, PSMB9, This may include CAMK4, TMEM19, SLC12A7, TP53BP1, PLEKHO1, SLC25A22, FRS2, CD24, STX1A, KCNMB4, CRISP2, HTRA1, PPL, RHBDF2, ZCCHC4, DDX6, SENP5, RAPGEF1, DTX2, RELB, DYRK2, CCNB1IP1, TDRD9, ZAP70, ARL14EP, MDC1, and / or ADGRE3), or composite biomarker scores (e.g., SeptiCyte® LAB and SeptiCyte® RAPID, or Inflammatix® TriVerity®).

[0079]

[0086] C-reactive protein (CRP) is the concentration of CRP in the blood. Blood CRP levels can be obtained through chemical laboratory testing. CRP is a protein produced by the liver, and its levels increase with inflammation. Studies have shown that CRP, when combined with body temperature, can increase the detection of infection.

[0080]

[0087] Procalcitonin (PCT) is measured as the concentration of PCT present in the blood. PCT can be measured by laboratory tests. PCT is released into the bloodstream by tissues in the presence of inflammation. Bacterial infections have been shown to significantly affect PCT synthesis because bacterial endotoxins and cytokines promote PCT secretion by host cells.

[0081]

[0088] The authentication module 110 manages the authentication and authorization of users of the computing device 100 or the software run by the computing device 100. For example, patient measurements and input characteristics are health information protected under the Health Insurance Portability and Accountability Act of 1996 (HIPAA), and therefore only authorized individuals can view such patient measurements and input characteristics. The authentication module 110 is a means to prevent unauthorized individuals from accessing protected health information.

[0082]

[0089] The preprocessing module 112 transforms the input features received by the computing device 100 to match the parameters (e.g., units, scale, method, etc.) of the input features trained by the machine learning model 120. In some examples, the input features received by the computing device 100 may already match the parameters of the corresponding input features trained by the machine learning model 120, and therefore do not require transformation by the preprocessing module 112. In various embodiments, the preprocessing module 112 maps the input features received by Logical Observation Identifier Names and Codes (LOINC) to the appropriate parameters of the corresponding input features trained by the machine learning model 120. LOINC codes are, as is known to those skilled in the art, a global standard for identifying medical laboratory observations, facilitating the exchange and pooling of results such as laboratory tests or vital signs for clinical care, outcome management, and research. The mapping can determine any transformation function for the received input features, such as unit or method transformations. Preprocessing and mapping configuration rules can be set by the user of the computing device 100. Exemplary preprocessing procedures are described in detail below.

[0083]

[0090] The trigger logic module 114 determines whether and when to trigger the machine learning model 120 to generate predictions. An exemplary method illustrating how the trigger logic module 114 makes such a decision is described in more detail in Figures 4-6 below under the subheading "Trigger Method".

[0084]

[0091] Model drift in the machine learning model 120 can be indirectly monitored by monitoring data drift via the data drift module 116. The data drift module 116 performs checks to identify the likelihood of data drift occurring in the machine learning model 120. In at least some embodiments, the data drift module 116 can generate a data drift report showing the results of the checks used. If at least one of the checks fails, the data drift report indicates that data drift has been detected. The end user can then use the data drift report to determine whether data drift has actually occurred and, if so, to decide on corrective actions to take. Further details regarding the data drift detection process are described below.

[0085]

[0092] In some examples other than those illustrated, the components of the computer network 10 may be combined, rearranged, removed, or intermediate components may be added. For example, the measuring device 102 and the user input device 104 may be combined. In another example, the measuring device 102 may transmit information to a computing device in the patient's room, which in turn transmits that information to the computing device 100. In yet another example, as described above, the machine learning model 120 may be stored on a server communicating with the computing device via the network 106.

[0086] Prediction method

[0093] Figure 2 is a flowchart of an exemplary method 200 for generating a prediction of the risk level (e.g., probability) of a patient having or developing a medical condition (e.g., sepsis). In various embodiments, the risk level may be a measure of risk over a predetermined time period (e.g., 24 hours) from the time a request is made to trigger an algorithm (e.g., machine learning model 120) that generates the prediction. In other words, in such embodiments, the generated prediction can no longer be relied upon after the predetermined time period has elapsed. Method 200 may be performed by processing logic that may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both. For example, a computing device 100 may perform Method 200. In at least some embodiments, Method 200 may include authentication and authorization procedures to help prevent unauthorized individuals from accessing HIPAA-protected patient health information. The authentication and authorization procedures are managed by user, access policy, and authentication client configurations. The authentication procedure may be performed before enabling the user to submit a request to trigger the machine learning model 120, or in response to receiving a request before proceeding in method 200.

[0087]

[0094] In block 202, multiple input features of a machine learning model (e.g., machine learning model 120) constructed to generate predictions of the risk of a patient having or developing a medical condition (e.g., sepsis) are received. For example, if machine learning model 120 has 22 input features, then 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, or 22 input features may be received. As will be further described below, input features not received in block 202 may be received at a later point in time, or their replacement values ​​may be added at a later point in time. In at least some embodiments, the received input features are values ​​measured by one of the measurement devices 102, or values ​​measured via clinical observation and input to one of the user input devices 104.

[0088]

[0095] In at least some embodiments, input features are received in real time. In other words, in these embodiments, input features collected by the measuring device 102 or the user input device 104 are transmitted to the computing device 100 instantaneously or nearly instantaneously as soon as the input features are collected (e.g., measured) so that the computing device 100 receives the input features in real time. The computing device 100 may store the input features received in real time in patient data 124 stored in memory 118.

[0089]

[0096] In at least some embodiments, a request is received (e.g., by the processor 122) to trigger the machine learning model 120 and initiate a process to generate predictions of whether a patient has or is at risk of developing a medical condition. In at least some embodiments, upon receiving a request to trigger the machine learning model 120 (e.g., from a clinician's input device), the processor 122 begins to listen for input features from various measurement devices 102 or user input devices 104, such as by reading patient data 124 stored in memory 118. In these embodiments, the processor 122 does not read the patient data 124 before a request is received. Even if the processor 122 has not read the patient data 124, the patient data 124 can be continuously updated as new information is received from the measurement devices 102 or user input devices 104. In other words, in such embodiments, the computing device 100 continuously receives and stores input features in the patient data 124, but the processor 122 does not initiate the execution of method 200 to read the patient data 124 until a request is received. Once the processor 122 begins executing method 200, the stored input features of the patient data 124 are read until the machine learning model 120 generates an output or it is determined that the machine learning model 120 cannot generate a prediction based on the available input features. In other embodiments, the processor 122 may continuously read the stored input features regardless of requests. In such other embodiments, the processor 122 may wait to execute method 200 beyond reading the stored patient data 124 until a request is received.

[0090]

[0097] In various embodiments, receiving a request involves receiving a data packet that includes the following information: the time the request was received, multiple input features of the machine learning model 120 stored in patient data 124 if available, the time the input feature was ordered for each input feature if available, the time the input feature was collected for each input feature if available, and the time the input feature was obtained as a result for each input feature if available.

[0091]

[0098] In various embodiments, Method 200 may include one or more preprocessing steps for transforming one or more of the received input features so that they match the set of parameters of the input features from which the machine learning model 120 was trained (e.g., via preprocessing module 112). Examples of transformations include, namely, mapping laboratory and / or clinical observations (e.g., LOINC) to input features, unit transformations, logarithmic transformations, normalization of input features, standardization of input features, scaling, transformation of continuous measurements to categorical measurements (and vice versa), and linear or nonlinear mapping functions, and the preprocessing steps may include mapping the received input features to one or more parameters of the input features from which the machine learning model 120 was trained. For example, a transformation function may transform the units, scale, or method of the received input features so that they match the units, scale, or method from which the machine learning model 120 was trained. In one example, C-reactive protein input features are typically recorded using LOINC code 1988-5 and measured using a machine that typically outputs in mg / L units (e.g., Roche Cobas® 6000 Analyzer Series Module c501). In this example, the conversion function needs to convert the measured C-reactive protein input features to pg / ml units and convert the input feature values ​​to C-reactive protein measurements measured by the Human Obesity Luminex Performance Assay Panel, because these are the units and methods on which the machine learning model 120 was trained. In another example, the patient's age may be converted to an age category (e.g., 65 years or older, or under 65 years).

[0092]

[0099] In another example, a conversion function is used for procalcitonin input features to convert input features measured using a machine (e.g., Roche Cobas® 6000 Analyzer Series Module c501) from ng / L to pg / ml, and then convert the input features to the Luminex Performance Human Premixed Multi-Analyte Panel Set 3C method.

[0093]

[0100] A user (e.g., administrator) of the provided system (e.g., computing device 100) can configure a mapping for a specific organization and / or configure a normal range for each input feature of the organization. Based on the configured normal range, outliers outside the minimum and maximum values ​​of the range can be identified and removed. When the input features are displayed to the end user, if any of the input features are outside the configured normal range, those input features will be displayed as such.

[0094]

[0101] In block 204, for each of the multiple input features, the state for triggering the machine learning model is determined. For example, an input feature is in a first state, indicating that it is ready for algorithmic prediction, if it is associated with a measurement taken within an acceptable time frame, or if there is sufficient time elapsed for the input feature to be complemented by a replacement value. Each input feature has its own set of requirements for being in the first state. In another example, an input feature is in a second state, indicating that it is not ready for algorithmic prediction, if it is not associated with a measurement taken within an acceptable time frame and is not complemented, or if there is not enough time elapsed for the input feature to be complemented by a replacement value.

[0095]

[0102] For example, a set of requirements for a first input feature among multiple input features might require that the first input feature is critical to the predictions of the machine learning model 120, and that the complementary replacement value reduces the accuracy of the predictions, thus requiring that the first input feature be associated with an acceptable first measurement resulting from being in a first state. Therefore, in this example, the first input feature is not complementary and will be in a second state unless a measurement is received for the first input feature. In some examples, after determining that the first input feature is in a second state, method 200 may include waiting a predetermined amount of time to receive a measurement for the first input feature. The state of the first input feature may be determined periodically during the predetermined amount of time. If a measurement for the first input feature is received at a later time within the predetermined amount of time, the first input feature transitions to a first state. If no measurement for the first input feature is received after the predetermined amount of time, in various embodiments, the machine learning model 120 is not triggered because the first input feature remains in a second state. In other embodiments,

[0096]

[0103] In another example, the set of requirements for a second input feature among multiple input features differs because the second input feature is complementable and therefore does not need to be associated with an acceptable measurement obtained as a result, since it is in the first state. If no measurement is received for the second input feature (e.g., the second input feature is not received), method 200 may include waiting for a predetermined amount of time to receive the second input feature. The second input feature is in the second state and therefore, since the measurement is preferable, even if not necessary, for the accuracy of the predictions of the machine learning model 120, it cannot be complemented with a replacement value until the predetermined amount of time has elapsed. After the predetermined amount of time has elapsed, the second input feature is determined to be in the first state and can be complemented with a replacement value. Thus, the set of requirements for the second input feature to be in the first state includes that the second input feature is complementable and that the predetermined amount of time has elapsed.

[0097]

[0104] Further details regarding determining the state of the input features are provided in Figures 4-7 and their corresponding descriptions under the subheading "Trigger Method".

[0098]

[0105] In block 206, the machine learning model 120 is used to generate a prediction of the risk (e.g., probability) of a patient having or developing a medical condition (e.g., sepsis). In some embodiments, the prediction is generated after determining that each of a plurality of input features is a first state, and as a result, the prediction is based on all of the plurality of input features of the machine learning model 120. In such embodiments, if one or more input features are a second state, no prediction is generated, and the output of the machine learning model 120 is "no result". A consistent, meaningful (e.g., accurate) output from the algorithm is obtained by refraining from triggering the machine learning model 120 to generate predictions until all input features have values ​​available in these embodiments, and only by imputing the values ​​of specific input features in specific cases. In other embodiments, the machine learning model 120 may be triggered if one or more input features are missing that cannot be imputed. For example, it may be predetermined, or determined as part of method 200, that a particular combination of input features, or a particular combination of input features and imputing input features, generates a consistently meaningful (e.g., accurate) prediction without one or more missing input features. In various examples, the risk prediction is a value between 0 and 100, but other appropriate scales can also be used. The prediction may be displayed on a display device (e.g., the display of user input device 104).

[0099]

[0106] In some embodiments, the generated patient risk predictions can be categorized into risk categories. For example, the predicted risk can be categorized into a first risk category (e.g., low risk), a second risk category (e.g., medium risk), a third risk category (e.g., high risk), or a fourth risk category (e.g., very high risk) based on thresholds defining each category. If the predicted risk is below the first threshold, the predicted risk is categorized into the first risk category. In one example, the first risk category is based on the proportion of previous patients in the first risk category who required vasopressors within 24 hours, compared to previous patients in the second, third, and fourth risk categories. If the predicted risk is greater than the first threshold but less than the second threshold, the predicted risk is categorized into the second risk category. In one example, the second risk category is based on the proportion of previous patients in the second risk category who required mechanical ventilation within 24 hours, compared to previous patients in the first, third, and fourth risk categories. If the predicted risk is greater than a second threshold but less than a third threshold (e.g., the 95th percentile of the predicted risk), the predicted risk is classified into a third risk category. In one example, the third risk category is based on the proportion of previous patients in the third risk category who required transfer to the intensive care unit within 24 hours, compared to previous patients in the first, second, and fourth risk categories. If the predicted risk is greater than a third threshold, the predicted risk is classified into a fourth risk category. In one example, the fourth risk category is based on the proportion of previous patients in the fourth risk category whose condition resulted in in-hospital mortality, compared to previous patients in the first, second, and third categories.

[0100]

[0107] In some embodiments, at least some of the input features of the machine learning model 120 may include local feature importance values, which are numerical values ​​associated with the input features that represent the contribution of the input features to the predictions of the machine learning model 120. For example, input features may include Shapley Additive Explanations (SHAP) values, Local interpretable model-agnostic explanations (LIME) values, or other appropriate local feature importance values. In one example, input features may include local feature importance values ​​for each input feature associated with an interpolated value. The use of local feature importance values ​​helps in the interpretation and credibility of the generated predictions. The local feature importance value when a single prediction is added together is equal to the uncalibrated prediction, i.e., the average of the entire predefined dataset (uncalibrated prediction). The additional impact of each input feature on the uncalibrated prediction can be observed. Local feature importance values ​​are properties of any newly generated prediction, rather than global machine learning model properties, and localize the generally used concept of feature importance for individual predictions.

[0101]

[0108] An exemplary process for generating the contribution of input features to local feature importance values ​​involves using complementary input features, a trained machine learning model 120, and training data objects. The calculation of local feature importance values ​​can identify the contribution that input features make to the prediction of observations using a game-theoretic approach. The local feature importance values ​​identify an estimate of the input features' contribution to the training dataset using the training data objects. Due to the computational complexity of calculating local feature importance values, in various embodiments, the process may use a local feature importance value estimation implementation via the fastshap R package. The fastshap R package may use a Monte Carlo simulation with 10 rounds to estimate the contribution of input features to training data objects with a fixed seed. This estimation can be applied to new observations to generate the overall contribution of input features to predictions (e.g., local feature importance values). The local feature importance values ​​of input features can be visually represented on a display for the benefit of the clinician (see, for example, Figure 3).

[0102]

[0109] In at least some embodiments, method 200 may include calibrating the predictive output by the machine learning model 120. For example, a classifier generated by a random forest model may produce class probabilities of rather poor quality, as measured by the squared error of the predicted probabilities (Bryar score), even though it is accurate and achieves a high AUC. One approach to address this problem is by using a calibration model, such as learning a function that maps the original probability estimates or scores to more accurate probability estimates. For example, in various embodiments, the calibration model may be generated by performing Pratt calibration. Pratt calibration can be created by training a logistic regression model with the uncalibrated predictions of the machine learning model 120 to predict calibrated predictions (e.g., Sepsis-3 training labels). Other suitable methods for generating a calibration model may be used in various embodiments, such as isotonic calibration. The calibration model is a logistic model that uses the uncalibrated predictions of the machine learning model 120 as input to generate calibrated predictions. The objective of the calibration model is to establish a relationship between an uncalibrated prediction and the classification of having or developing sepsis within 24 hours of a request to trigger the machine learning model 120. This relationship can be applied to convert future uncalibrated predictions into calibrated predictions. In such a calibration embodiment, the calibrated predictions can be classified into risk categories.

[0103]

[0110] Figure 3 shows an exemplary user interface 300 that may be displayed to an end user (e.g., a clinician) (e.g., on a user input device 104) to evaluate the output of the machine learning model 120. The user interface 300 includes a predicted risk level 302 that a patient will have or develop sepsis within a predetermined time period (e.g., 24 hours) from the time the prediction request is made. In this example, the risk level (e.g., probability) of the patient developing sepsis is 10% within 24 hours from the time the request is made. The user interface 300 further includes a risk category 304, which in this example is a “low-risk category”. It also includes the order time 306 when the request to trigger the machine learning model 120 to generate a prediction was received, and the result time 308 when the prediction was generated. In various embodiments, the computing device 100 may operate in “clinical use” or “non-clinical use” mode. When operating in “clinical use” mode, all outputs of the machine learning model 120 are labeled “for clinical use” to the end-user healthcare provider. When operating in “non-clinical use,” all outputs of the machine learning model 120 are labeled “for non-clinical use” to end-user healthcare providers, indicating that the results are not intended to be used to inform actual patient care. This labeling of results helps ensure patient safety when the device is used in ways other than normal clinical use, such as during the installation or testing of the computing device 100. In this example, the user interface 300 indicates that the mode 310 of the computing device 100 was “non-clinical use.” A scale 312 may be shown to visually display different risk categories. In this example, the scale 312 is a bar and may include different colors for each risk category. An indicator 314 is placed on the scale 312 to visually display the patient’s risk level compared to the risk category.

[0104]

[0111] The input parameters 316 for the machine learning model 120 are displayed at the bottom of the user interface 300. For each input parameter 316, the measured value 318 (if available) and the collection time 320 (if available) of the measured value 318 are shown on the user interface 300. The input parameters 316 are categorized into parameters 326 that increase the patient's risk of developing sepsis, parameters 328 that decrease the patient's risk of developing sepsis, and parameters 330 that were unavailable at the time the prediction was generated. In this example, parameter 330 is shown as unavailable because it did not have a measured value 318 that could be used to generate the prediction, but rather had a value that was interpolated to the parameter. In other examples, parameters 330 with interpolated values ​​may be shown along with those interpolated values ​​and with an indication that the value was interpolated and not measured. For each input parameter 316 with a measured value 318, a bar is shown indicating the contribution of the input parameter 316 to the patient's risk of developing sepsis (e.g., local feature importance value). For example, each input parameter 316 of parameter 326 is associated with a bar 322 that represents the contribution of the input parameter 316 to the increased risk of a patient developing sepsis (e.g., local feature importance value). A larger bar 322 indicates a greater contribution. In another example, each input parameter 316 of parameter 328 is associated with a bar 324 that represents the contribution of the input parameter 316 to the decreased risk of a patient developing sepsis (e.g., local feature importance value). A larger bar 324 indicates a greater contribution.

[0105]

[0112] In some embodiments, Method 200 includes indirectly detecting model drift by detecting data drift in the machine learning model 120. Detecting data drift may include comparing data input to a processing logic that directs Method 200 (e.g., software) during an “observed” period with data during a “baseline” period. The baseline period is intended to select data that represents future observations. The baseline period may be modified by an authenticated end user as needed. The observed period is specific to a particular data drift report. All input data input to the machine learning model 120 during a specified period is considered part of the observed dataset. The comparison process performs several statistical checks to check for differences in data distribution and data availability. In at least some embodiments, a data drift report may be generated that conveys the results of the statistical checks. The detected differences may indicate that data drift has occurred. In various embodiments, if at least one of the statistical checks fails, the data drift report indicates that data drift has been detected for the observed dataset. If a discrepancy is detected, or if the data drift report indicates that data drift has been detected, the end user can evaluate whether data drift occurred, take steps to protect patient safety as needed, and correct the data drift problem as necessary.

[0106]

[0113] In various embodiments, there are two categories of input data monitored for data drift: measurement results and measurement orders. Measurement results are measurements with a recorded time and value. Measurement results serve as input to a machine learning model. Measurement results are monitored for changes in distribution and availability. Measurement orders are measurements that are not obtained as results. Measurement orders are not used directly as input to the machine learning model 120, but rather help determine when the machine learning model 120 is triggered. Since there are no values ​​associated with measurement orders, changes in distribution cannot be monitored. However, measurement orders can monitor changes in availability. In some embodiments, measurement results and measurement orders are used in data drift reporting only if they fall within a time window surrounding the request that triggers the machine learning model 120. In one example, the time window is from the start of the delay period for the measurement type to the result time of the prediction generated by the machine learning model 120, or until the prediction ceases to exist.

[0107]

[0114] A first exemplary statistical test is used to detect changes in the distribution of measurement results. In this first example, the Kolmogorov-Smirnov test is used to compare the distribution of each measurement between the reference set and the observed set. If the null hypothesis of the two-sample Kolmogorov-Smirnov test is rejected based on a specified p-value (e.g., 0.05), this means that the reference set and the observed set do not arise from the same distribution of parameters, indicating that there may be a change in the distribution of measurements between the reference set and the observed set.

[0108]

[0115] A second exemplary statistical test is used to detect changes in the availability of measurement results. In this second example, for each request and each measurement type that triggers the machine learning model 120, the frequency of available measurement results is calculated by dividing the number of available measurements by the length of the time window associated with each request that triggers the machine learning model 120. The calculation is performed for both the baseline set and the observed set. In this second example, a two-sample Kolmogorov-Smirnov test is used to compare the distribution of measurement availability frequencies between the baseline set and the observed set. If the null hypothesis of the two-sample Kolmogorov-Smirnov test is rejected based on a given p-value (e.g., 0.05), this means that the baseline set and the observed set do not arise from the same distribution of parameters, indicating that there may be a difference in the availability of measurement orders between the baseline set and the observed set.

[0109]

[0116] A third exemplary statistical test is used to detect changes in the availability of measurement ordering. In this third example, for each request and each measurement type that triggers the machine learning model 120, the frequency of measurement orders is calculated by dividing the number of measurement orders by the length of the time window associated with each request that triggers the machine learning model 120. The calculation is performed for both the baseline set and the observed set. In this second example, a two-sample Kolmogorov-Smirnov test is used to compare the distribution of measurement ordering frequencies between the baseline set and the observed set. If the null hypothesis of the two-sample Kolmogorov-Smirnov test is rejected based on a specified p-value (e.g., 0.05), this means that the baseline set and the observed set do not arise from the same distribution of parameters, indicating that there may be differences in measurement ordering between the baseline set and the observed set.

[0110] Trigger method

[0117] Figure 4 shows a flowchart of an exemplary method 400 for determining whether to trigger an algorithm (e.g., a machine learning model 120) stored in a computing device (e.g., computing device 100). Method 400 can be performed by processing logic which may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In ellipse 402, it is determined whether a predetermined amount of time (e.g., 3.5 hours) has elapsed since the time a request to trigger the algorithm was received. This predetermined amount of time in ellipse 402 sets a maximum time limit for waiting for input features until the algorithm cannot be triggered and a new request to trigger the algorithm must be made. If the predetermined amount of time has not elapsed (e.g., "false"), in ellipse 406, it is determined whether each of the multiple input features to the algorithm is in a first state such that the multiple input features are ready (e.g., available) for algorithm prediction. In some embodiments, the multiple input features are all of the input features of the algorithm. In other embodiments, the multiple input features are a subset of all of the input features of the algorithm. For example, it may be predetermined, or determined as part of Method 400, that a particular combination of input features smaller than all of the algorithm's input features (some of which may be complemented) generates accurate predictions without one or more missing input features.

[0111]

[0118] Figure 5 shows a flowchart of an exemplary method 500 for determining whether an input feature of an algorithm stored in a computing device is in a first state, such that the input feature is ready (e.g., available) for algorithmic prediction. Method 500 may be performed on each of the multiple input features of the algorithm to determine in ellipse 406 whether each of the multiple input features satisfies its respective set of requirements for being in the first state. Method 500 may be performed by processing logic which may include hardware (circuitry, dedicated logic, etc.), software, or a combination of both. In ellipse 502, the input feature is obtained as a result and it is determined whether it is acceptable. The input feature is obtained as a result when the input feature is measured and received (e.g., from a measuring device 102). The input feature is acceptable such that it is still likely to be accurate if the input feature was measured (e.g., collected from a patient) within a predetermined amount of time from the time the request to trigger the algorithm was received (e.g., an acceptable time window). For example, measurements from a comprehensive metabolic panel are unlikely to accurately represent the patient's current state if the blood test for the panel results was performed four days ago. Therefore, the input features from this comprehensive 4-day metabolic panel are unacceptable because they could negatively impact the accuracy of algorithmic predictions.

[0112]

[0119] The duration for which a measured input feature remains valid, i.e., a predetermined amount of time from the time the request to trigger the algorithm is received, can vary depending on the input feature. For example, a patient's age may change only once a year, while their blood composition may change throughout the day. In one example, the measured input features, namely systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and, if included as input features, inspired oxygen concentration (FiO2) and the Glasgow Coma Scale, remain valid for 6 hours. In another example, the measured input features, namely white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, procalcitonin, C-reactive protein, lactate, and, if included as input features, glucose, calcium, hematocrit, and hemoglobin, remain valid for 24 hours. In addition, the time of measurement for input features can vary for different input features. For example, the measurement time in the laboratory may correspond to the aspiration time (e.g., the time when blood is collected from the patient), and the measurement time of the patient's vital signs may correspond to the time when those vital signs were recorded.

[0113]

[0120] In some embodiments, two or more instances of an input feature may be measured and be of acceptable value. For example, a body temperature sensor may collect a patient's body temperature every few seconds, thereby yielding many measured body temperature input features within an acceptable time window of the body temperature input feature. In such cases, a single measured input feature within the acceptable time window is selected. The single measured value can be selected from negative values ​​in any appropriate way (e.g., minimum, maximum, average, latest, etc.) and may differ for different input features. For example, the minimum measured instance of an input feature may be selected for the following input features: systolic blood pressure, diastolic blood pressure, and blood oxygen saturation. In another example, the maximum measured instance of an input feature may be selected for the following input features: body temperature, respiratory rate, and heart rate. In another example, measured instances of input features can be selected for the following input features: white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, procalcitonin, C-reactive protein, and lactate.

[0114]

[0121] If the measured input feature is obtained as a result and determined to be acceptable in ellipse 502, then in block 504, the input feature is determined to be ready (e.g., available) for algorithmic prediction (e.g., being in the first state). In some examples, the set of requirements for an input feature being in the first state may include that the input feature is obtained as a result and is acceptable. Alternatively, if the input feature is determined to be either not obtained as a result or not acceptable in ellipse 502, then in ellipse 506, it is determined whether the input feature is complementable. In some embodiments, to determine whether an input feature is complementable, a lookup table enumerating each input feature and whether each is complementable (e.g., yes or no) may be referenced. In other embodiments, determining whether an input feature is complementable may depend on other input features. For example, the complementability of an input feature may depend on whether one or more other input features have obtainable and acceptable values. In one example, a C-reactive protein may be complementable if procalcitonin (PCT), albumin, and platelet count are obtained as acceptable values, but otherwise the C-reactive protein is not complementable. In another example, the complementability of an input feature may depend, additionally or alternatively, on the obtainable values ​​of one or more other input features (e.g., being within the same range, meeting a threshold, or not meeting a threshold). In one example, a C-reactive protein may be complementable if albumin is obtained as an acceptable value and procalcitonin (PCT) is obtained as an acceptable value of less than 2log(10)pg / mL, but otherwise the C-reactive protein is not complementable.

[0115]

[0122] In various embodiments, the following input features may be complementable: age, lymphocyte count, neutrophil count, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, and, if included as input features, inspired oxygen concentration (FiO2), Glasgow coma scale, glucose, calcium, hematocrit, and hemoglobin concentration. In various embodiments, the following input features may not be complementable: systolic blood pressure, diastolic blood pressure, blood oxygen saturation, body temperature, respiratory rate, heart rate, white blood cell count, platelet count, creatinine, blood urea nitrogen, procalcitonin, and C-reactive protein.

[0116]

[0123] If an input feature is not complementable, in block 508 it is determined that the input feature is not ready for algorithmic prediction (e.g., a second state). In this example, the set of requirements for an input feature in the first state includes that the result is obtainable and acceptable. In some embodiments, method 500 may terminate for an input feature after determining that it is not ready. In other embodiments, in block 524 it is determined whether a measurement order for the input feature was made within a predetermined amount of time (e.g., a predetermined order time window) with respect to the time of the incoming request that triggers the algorithm (e.g., block 202). The predetermined order time window may capture the period after and / or before the received trigger request. The fact that a measurement order has been made indicates that the measured input feature is considered beneficial (e.g., by the ordering clinician, hospital policy, etc.) and should therefore be utilized if possible. However, a time limit is set for measurement orders to be made in order to balance the benefits of beneficial measured input features with the need for rapid algorithmic prediction. The order time window is intended to generate timely completion of input feature replacement values ​​when the physician does not intend to order measurement of the input features, but gives the physician an opportunity to order measurement of the input features by not completing them too quickly. In one example, a predetermined order time window is 3 hours before the algorithm trigger request and 30 minutes after the trigger request for each input feature. In some embodiments, the predetermined order time window may differ between different input features. If a measurement order is made, the input features remain in the second state in block 508, and method 500 terminates.

[0117]

[0124] Instead, if block 524 determines that there are no measurement orders within a given order time window, block 526 triggers the active ordering method. Figure 5B shows a flowchart of an exemplary active ordering method 550 that may be performed in block 526. The active ordering method 550 is a feature that can be turned on or off for a particular input feature. Thus, in block 552, it is determined whether active ordering is enabled for the input feature. For example, a lookup table can be referenced that enumerates each input feature and whether active ordering is enabled for each input feature (e.g., yes or no). If active ordering is not enabled for the input feature, the input feature remains in the second state in block 554, and method 500 terminates.

[0118]

[0125] If active ordering is enabled for an input feature, block 556 determines whether automatic feature measurement ordering is enabled for that input feature. For example, a lookup table may be referenced that enumerates each input feature and whether automatic feature measurement ordering is enabled for each input feature (e.g., yes or no). If automatic feature measurement ordering is not enabled, block 558 may generate a notification for the input feature prompting the user to generate a measurement order. In some embodiments of the active ordering method 500, a notification that additional parameters should be obtained for a patient may be communicated to the clinician (e.g., via the display of the clinician's user input device 104) so ​​that the clinician can generate a measurement order as timely as possible, in order to maximize the chances that the machine learning model 120 will be triggered in a timely and accurate manner. In some examples, the notification that additional parameters should be obtained may be added, either additionally or alternatively, to the patient's hospital electronic medical record.

[0119]

[0126] Alternatively, if automatic feature measurement ordering is enabled for an input feature, in block 560, the measurement order for the input feature may be automatically generated, rather than relying on a clinician to generate the measurement order, and sent (e.g., by a computing device 100 performing an active ordering method) to a laboratory technician's user input device 104 or measurement device 102, etc. Automatically generating a measurement order may include converting the input feature from which the measurement order is generated into a measurement order. The automatically generated measurement order may instruct a machine (e.g., measurement device 102) or a medical professional (e.g., a laboratory technician) to perform a laboratory test or life assessment. Thus, this active ordering method can actively alert a clinician or communicate with the hospital's electronic medical records, user input device 104, and / or measurement device 102 to trigger rapid orders for necessary laboratory or patient assessments to provide the machine learning model 120 with the necessary data as timely as possible. The laboratory test or life assessment may be configured by the administrator of system 10.

[0120]

[0127] In block 562, the active ordering method 500 may include the processing logic executing an active ordering method 500 that communicates with the processing logic executing a trigger method to indicate that an input feature is expected to be available in a timely manner. For example, the active ordering logic may communicate the time when a measured input feature is expected to be available, based on a measurement order made by either a clinician (or other medical professional) or a computing device 100. In some embodiments, communication from the active ordering logic may cause the trigger logic to adjust the order time window and / or result time limit to wait for a missing input feature before it is available as a result. For example, if a given order time window is expected to expire in 15 minutes, but the active ordering logic communicates that a missing measured input feature is expected to be available in 25 minutes, the trigger logic may extend the order time window (e.g., to expire in 30 minutes) so that the measured input feature can be used in a predictive decision instead of a completed input feature, or to completely refrain from making a prediction that would start the predictive process from the beginning. A similar time extension can be performed for a given result time limit. In some embodiments, at least one of the order time window and result time limit may be extended two or more times. For example, continuing the above example, if after 29 minutes the active ordering logic communicates to the trigger logic that it expects the measurement to be 3 minutes later (i.e., after the extended order time window), the trigger logic may extend the order time window again, for example, by another 5 or 10 minutes. The trigger logic may extend at least one of the order time window and result time limit by any appropriate amount of time (e.g., 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, or 120 minutes) with respect to how much time has elapsed since the prediction request, if the result time limit has been extended, complemented by the remaining time each of the resulting input features remains acceptable, the impact of specific input features on the prediction rather than having an acceptable value, and other appropriate considerations.The trigger logic may also refrain from extending at least one of the order time window and / or result time limit, based on the same considerations described above for extending at least one of the order time window and / or result time limit. Once the order time window and / or result time limit have been updated as necessary, method 500 terminates.

[0121]

[0128] Returning to ellipse 510 in Figure 5A, instead, if the input features are complementary, it is determined whether the measurement order was made within a predetermined time period (e.g., a predetermined order time window) with respect to the time of the incoming request that triggers the algorithm (i.e., block 202). Determining whether a measurement order has been made can be done in the same way as described above for block 524, and has the same advantages.

[0122]

[0129] If no measurement order has been placed within a predetermined order time window, ellipse 512 determines whether the current time is within the predetermined order time window. In other words, it determines whether the time window for placing a measurement order has ended. If it is determined that the current time is outside the predetermined order time window, block 514 determines that the input feature is ready (e.g., in the first state). Conversely, if it is determined that the current time is still within the predetermined order time window, block 516 determines that the input feature is not ready (e.g., in the second state). In other words, there is still time remaining for a measurement order to be placed, so the input feature is not ready and therefore does not need to be filled with values, leaving the possibility of receiving an acceptable input feature as a missing result. In this case, the set of input feature requirements for the first state is either (1) associated with an acceptable value as a result, or (2) it is fillable and the predetermined order time window has elapsed.

[0123]

[0130] In some embodiments, method 500 terminates after determining in block 516 that the input feature is not ready. In other embodiments, method 500 returns to block 526 to determine (e.g., in the same way as block 526 described above) whether a measurement order for the input feature was made within a predetermined order time window with respect to the time of the incoming request that triggers the algorithm (e.g., block 202). If a measurement order has been made, the input feature remains in the second state, and method 500 terminates for this input feature. If block 524 determines that there is no measurement order within the predetermined order time window, block 526 may generate a measurement order, or a notification to generate a measurement order, for the input feature as part of the active ordering logic described above.

[0124]

[0131] Returning to ellipse 510, if there is a measurement order made within a predetermined order time window, ellipse 518 determines whether the current time is within a predetermined result time limit from the time the measurement order was made. The result time limit allows for the collection of ordered measurements and the acquisition of results, while still enabling timely algorithmic predictions even if there is a significant delay in the collection and measurement result process and it is necessary to instead interpolate the values ​​of input features. In this case, the set of requirements for an input feature to be in the first state includes (1) being associated with an acceptable value that can be obtained as a result, or (2) being complementary and the predetermined result time window has elapsed. Examples of input features that may require time for measurements after the measurement order include laboratory inspections, teleobservation phenomena that must be transmitted over long distances, observations made by devices that are not immediately accessible, and many others. In at least one example, the predetermined result time limit is 2 hours from the measurement order time for each input feature. In some embodiments, the predetermined result time limit may differ between different measurement orders associated with different input features. If the current time is determined to be outside the predetermined result time limit, block 520 determines that the input features are ready (e.g., state 1). Conversely, if the current time is determined to still be within the predetermined result time limit, block 522 determines that the input features are not ready (e.g., state 2). In other words, because there is still time remaining before the measurement results are obtained, the input features are not ready, and therefore there is no need to interpolate values ​​into the input features, leaving open the possibility that the input features can be associated with the resulting acceptable values.

[0125]

[0132] Returning to ellipse 406 in Figure 4, if an instance of method 500 determines that one or more of the input features are not ready for algorithmic prediction (e.g., "false"), method 400 returns to ellipse 402 to determine whether a predetermined amount of time has elapsed since the request was received. In some embodiments, method 400 may include generating a notification indicating that the result of the algorithmic prediction is still pending and displaying that notification on the display of the device (e.g., user input device 104). If a predetermined amount of time has elapsed in ellipse 402, in block 404, method 400 includes refraining from triggering the algorithm. In other words, if a predetermined amount of time has elapsed since the time the request was received with each of the input features not ready for algorithmic prediction, method 400 terminates without triggering the algorithm. In some embodiments, method 400 may include generating a notification indicating that the algorithm cannot be triggered and displaying that notification on the display of the device (e.g., user input device 104).

[0126]

[0133] Alternatively, if the predetermined amount of time has not yet elapsed in ellipse 402, method 400 may proceed again to ellipse 406 to determine whether all input features are ready for algorithmic prediction. For example, measured input features resulting from the order of measurement of input features may have resulted from a time prior to the determination of whether each input feature is ready. This loop from ellipse 406 to ellipse 402 can be repeated as many times as necessary. If each of the multiple input features is ready for algorithmic prediction (e.g., true), then in block 408, substitution values ​​are added to the input features that are not obtained and are unacceptable (e.g., block 502).

[0127]

[0134] Determining appropriate substitution values ​​to complement input features can be carried out in a variety of suitable ways. In some embodiments, if an input feature requires a substitution value, the substitution value may be predetermined for the input feature so that the substitution value complements the input feature. For example, in the context of an exemplary algorithm that predicts whether a patient will have or is at risk of having sepsis, if a substitution is needed, the inhaled oxygen concentration (FiO2) input feature can be complemented with a value of 21%, since 21% is the FiO2 value for a person breathing at sea level, and this value is a reasonable assumption if there is no evidence that the patient is receiving oxygen support.

[0128]

[0135] In other embodiments, substitution values ​​to complement specific input features can be computed using a statistical or machine learning model (e.g., stored in memory 118 or on a remote computing device communicating with computing device 100). In such embodiments, a supervised machine learning model (e.g., a random forest model) is trained with complementary input features as labels (i.e., outcomes), and the remainder of the algorithm's input features are features in the model (i.e., covariates, predictors). For example, if an algorithm has multiple input features with four input features and substitution values ​​must be computed for the first input feature, the other three input features are inputs to the machine learning model for generating the substitution values. The machine learning model can be trained using any suitable software. For example, bag completion in the R caret package may be used, which uses a bootstrap aggregation (i.e., bagging) regression tree as the supervised learning model. In some embodiments, a given substitution value can complement some input features, while a value determined based on the model can complement other input features.

[0129]

[0136] If each of the multiple input features is either an acceptable result or is imputed with a replacement value, then in block 210, the algorithm is triggered to generate a prediction based on the input features and the imputed input features.

[0130]

[0137] Figure 6 shows an exemplary state determination chart for seven input features based on methods 400 and 500 detailed above. As shown, feature 1 has an observation (e.g., a measurement) that was obtained as a result of being performed (e.g., measured) within an acceptable time interval, and is therefore determined to be ready (e.g., in the first state). Feature 2 has an observation request (e.g., a measurement order) that does not have a resulting value but is complementary and was performed after the start of the acceptable time interval but before the observed request limit (e.g., within a given order time window). Since the current time is still before the result time limit, feature 2 is determined to be not ready (e.g., in the second state). Feature 3 has no resulting value but is complementary. Feature 3 does not have a measurement order within a given order time window because the measurement order was performed after a given order time window and the current time is after a given order time window. Therefore, feature 3 is determined to be in the first state. Feature 4 has a resulting value, but it was measured outside an acceptable time interval and is therefore unacceptable. Feature 4 is complementary, does not have a measurement order that is not obtained as a result, and the current time is after a predetermined order time window. Therefore, feature 4 is determined to be in the first state.

[0131]

[0138] Feature 5 has a resulting value, but it was measured outside an acceptable time interval and is therefore unacceptable. Feature 5 is not compensable and is therefore determined to be in the second state. Feature 6 has a resulting value that was measured within an acceptable time interval and is therefore determined to be in the first state. Feature 7 has a resulting value that was measured within an acceptable time interval, but the resulting value was measured outside an acceptable time interval (e.g., time observation was performed) and is therefore unacceptable. Feature 7 is not compensable and is therefore determined to be in the second state.

[0132] Computer execution system

[0139] Figure 7 is a block diagram showing a computer system 700 in which embodiments of this teaching may be implemented. In various embodiments of this teaching, the computer system 700 may include a bus 702 or other communication mechanism for communicating information and a processor 704 coupled to the bus 702 for processing information. In various embodiments, the computer system 700 may also include memory, which may be random access memory (RAM) 706 or other dynamic storage device, coupled to the bus 702 for determining instructions to be executed by the processor 704. The memory may also be used to store temporary numerical variables or other intermediate information during the execution of instructions executed by the processor 704. In various embodiments, the computer system 700 may further include read-only memory (ROM) 708 or other static storage device coupled to the bus 702 for storing static information and instructions for the processor 704. A storage device 710, such as a magnetic disk or optical disk, may be provided and coupled to the bus 702 for storing information and instructions.

[0133]

[0140] In various embodiments, the computer system 700 may be coupled via a bus 702 to a display 712, such as a cathode ray tube (CRT) or liquid crystal display (LCD), for displaying information to the computer user. An input device 714, including alphanumeric and other keys, may be coupled to the bus 702 to communicate information and command selections to the processor 704. Another type of user input device is a cursor control 716, such as a mouse, trackball, or cursor directional keys, for communicating directional information and command selections to the processor 704 and for controlling cursor movement on the display 712. This input device 714 typically has two degrees of freedom on two axes, namely a first axis (i.e., x) and a second axis (i.e., y), which allow the device to specify a position in a plane. However, it should be understood that an input device 714 enabling three-dimensional (x, y, and z) cursor movement is also contemplated herein.

[0134]

[0141] In accordance with a particular implementation of this teaching, the results can be provided by a computer system 700 in response to a processor 704 executing one or more sequences of one or more instructions contained in memory 706. Such instructions can be read into memory 706 from another computer-readable medium or computer-readable storage medium, such as a storage device 710. By executing the sequence of instructions contained in memory 706, the processor 704 can be made to execute the process described herein. Alternatively, this teaching can be implemented using hardwired circuits instead of, or in combination with, software instructions. Therefore, the implementation of this teaching is not limited to any particular combination of hardware circuits and software.

[0135]

[0142] The terms “computer-readable medium” (e.g., data store, data storage, etc.) or “computer-readable storage medium” as used herein refer to any medium involved in providing instructions to processor 704 for execution. Such mediums can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. Examples of non-volatile media include, but are not limited to, optical disks, solid-state disks, and magnetic disks, such as storage device 710. Examples of volatile media include, but are not limited to, dynamic memory, such as memory 706. Examples of transmission media include, but are not limited to, coaxial cables, copper wires, and optical fibers, such as wires including bus 702.

[0136]

[0143] Common forms of computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, and EPROMs, FLASH-EPROMs, any other memory chips or cartridges, or any other tangible media from which a computer can read.

[0137]

[0144] In addition to computer-readable media, instructions or data may be provided as signals on a transmission medium included in a communication device or system to provide a sequence of one or more instructions to the processor 704 of the computer system 700 for execution. For example, a communication device may include a transceiver having signals indicating instructions and data. The instructions and data are configured to cause one or more processors to implement the functions outlined in this disclosure. Typical examples of data communication transmission connections include, but are not limited to, telephone modem connections, wide area networks (WANs), local area networks (LANs), infrared data connections, and NFC connections.

[0138]

[0145] It should be understood that the methodologies, flowcharts, figures, and accompanying disclosures described herein can be implemented using the computer system 700 as a standalone device or on a distributed network of shared computing resources, such as a cloud computing network.

[0139]

[0146] The methodologies described herein can be implemented by various means depending on the application. For example, these methodologies can be implemented in hardware, firmware, software, or any combination thereof. In hardware implementations, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described herein, or a combination thereof.

[0140]

[0147] In various embodiments, the methods described herein can be implemented as firmware and / or software programs and applications written in conventional programming languages ​​such as C, C++, Rust, and Python. When implemented as firmware and / or software, the embodiments described herein can be implemented on a non-temporary computer-readable medium in which a program for causing a computer to perform the methods described above is stored. The various engines described herein can be provided on a computer system such as computer system 700, so that the processor 704 receives instructions provided by one or a combination thereof from memory components 706 / 708 / 710 and user input provided via input device 714, and performs the analysis and determination provided by these engines.

[0141] Digital processing devices

[0148] In various embodiments, the systems and methods described herein may include a digital processing device or the use thereof. In various embodiments, the digital processing device may include one or more hardware central processing units (CPUs) or general-purpose graphics processing units (GPGPUs) that perform the functions of the device. In various embodiments, the digital processing device may further include an operating system configured to execute executable instructions. In various embodiments, the digital processing device may optionally be connected to a computer network. In various embodiments, the digital processing device may optionally be connected to the Internet to access the World Wide Web. In various embodiments, the digital processing device may optionally be connected to a cloud computing infrastructure. In various embodiments, the digital processing device may optionally be connected to an intranet. In various embodiments, the digital processing device may optionally be connected to a data storage device.

[0142]

[0149] Depending on the various embodiments, suitable digital processing devices may include, in non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, subnotebook computers, netbook computers, netpad computers, handheld computers, internet appliances, mobile smartphones, tablet computers, and personal digital assistants. Those skilled in the art will recognize that many smartphones are suitable for use in the systems described herein. Those skilled in the art will also recognize that selected televisions, video players, and digital music players with optional computer network connectivity are also suitable for use in the systems described herein. Suitable tablet computers include those with booklet, slate, and convertible configurations, as known to those skilled in the art.

[0143]

[0150] In various embodiments, a digital processing device includes an operating system configured to execute executable instructions. The operating system may be software, for example, containing programs and data that manage the device's hardware and provide services for running applications. Those skilled in the art will recognize that suitable server operating systems include, in non-limiting examples, FreeBSD, OpenBSD, Net-BSD, Linux, Apple® Mac OS X Server®, Oracle® Solaris®, Windows Server®, and Novell® NetWare®. Those skilled in the art will also recognize that suitable personal computer operating systems include, in non-limiting examples, Microsoft® Windows®, Apple® Mac OS X®, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In various embodiments, the operating system is provided by cloud computing. Those skilled in the art will recognize that suitable mobile smartphone operating systems include, but are not limited to, Nokia® Symbian® OS, Apple® iOS®, Research In Motion® BlackBerry OS®, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile® OS, Linux®, and Palm® WebOS®.

[0144]

[0151] In various embodiments, the device includes a storage and / or memory device. A storage and / or memory device is one or more physical devices used to temporarily or permanently store data or programs. In various embodiments, the device is volatile memory and requires power to maintain the stored information. In various embodiments, the device is non-volatile memory and retains the stored information when the digital processing device is not powered. In various embodiments, non-volatile memory includes flash memory. In some embodiments, non-volatile memory includes dynamic random access memory (DRAM). In various embodiments, non-volatile memory includes ferroelectric random access memory (FRAM). In various embodiments, non-volatile memory includes phase-change random access memory (PRAM). In various embodiments, the device is a storage device that, in non-limiting examples, includes CD-ROMs, DVDs, flash memory devices, magnetic disk drives, magnetic tape drives, optical disk drives, and cloud computing-based storage. In various embodiments, the storage and / or memory device is a combination of devices such as those disclosed herein.

[0145]

[0152] In various embodiments, the digital processing device includes a display for transmitting visual information to the user. In various embodiments, the display is a cathode ray tube (CRT). In various embodiments, the display is a liquid crystal display (LCD). In various embodiments, the display is a thin-film transistor liquid crystal display (TFT-LCD). In various embodiments, the display is an organic light-emitting diode (OLED) display. In various embodiments, the OLED display is a passive-matrix OLED (PMOLED) or an active-matrix OLED (AMOLED) display. In various embodiments, the display is a plasma display. In various embodiments, the display is a video projector. In various embodiments, the display is a combination of devices such as those disclosed herein.

[0146]

[0153] In various embodiments, the digital processing device includes an input device for receiving information from a user. In various embodiments, the input device is a keyboard. In various embodiments, the input device is a pointing device, including, in non-limiting examples, a mouse, trackball, trackpad, joystick, game controller, or stylus. In various embodiments, the input device is a touchscreen or multitouchscreen. In various embodiments, the input device is a microphone for capturing audio or other acoustic input. In various embodiments, the input device is a video camera or other sensor for capturing motion or visual input. In various embodiments, the input device is Kinect, Leap Motion, etc. In various embodiments, the input device is a combination of devices, such as those disclosed herein.

[0147] Non-temporary computer-readable storage medium

[0154] In various embodiments, as described above, the systems and methods disclosed herein may include one or more non-temporary computer-readable storage media encoded with a program containing instructions executable by the operating system of a networked digital processing device, on which the methods herein may be performed. In various embodiments, the computer-readable storage medium is a tangible component of the digital processing device. In various embodiments, the computer-readable storage medium is optionally removable from the digital processing device. In various embodiments, the computer-readable storage medium may include, in non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid-state memory, magnetic disk drives, magnetic tape drives, optical disk drives, cloud computing systems and services, and the like. In various embodiments, the programs and instructions are encoded on the medium permanently, substantially permanently, semi-permanently, or non-temporarily.

[0148] Computer program

[0155] In various embodiments, the systems and methods disclosed herein may include or use at least one computer program. A computer program includes a sequence of instructions written to perform a specified task, executable on the CPU of a digital processing device. Computer-readable instructions may be implemented as program modules such as functions, objects, application programming interfaces (APi), and data structures, which perform a particular task or implement a particular abstract data type. Those skilled in the art will recognize that computer programs may be written in various versions of various languages.

[0149]

[0156] The functionality of computer-readable instructions can be combined or distributed as needed in various environments. In various embodiments, a computer program includes one instruction sequence. In various embodiments, a computer program includes multiple instruction sequences. In various embodiments, a computer program is provided from one location. In various embodiments, a computer program is provided from multiple locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plugins, extensions, add-ins or add-ons, or a combination thereof.

[0150] Web application

[0157] In various embodiments, a computer program includes a web application. Those skilled in the art will recognize that, in various embodiments, a web application utilizes one or more software frameworks and one or more database systems. In various embodiments, a web application is built on a software framework such as Microsoft® .NET or Ruby on Rails (RoR). In various embodiments, a web application utilizes one or more database systems, including, in non-limiting examples, relational, non-relational, object-oriented, association, and XML database systems. In various embodiments, suitable relational database systems, in non-limiting examples, include Microsoft® SQL Server, MySQL®, and Oracle®. Those skilled in the art will also recognize that, in various embodiments, a web application is written in one or more versions of one or more languages. A web application can be written in one or more markup languages, presentation-definition languages, client-side scripting languages, server-side coding languages, database query languages, or a combination thereof. In various embodiments, web applications are written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or Extensible Markup Language (XML). In various embodiments, web applications are written to some extent in a presentation-defining language such as Cascading Style Sheets (CSS). In various embodiments, web applications are written to some extent in a client-side scripting language such as Asynchronous JavaScript and XML (AJAX), Flash® ActionScript, JavaScript, or Silverlight®.In various embodiments, the web application is written to some extent in a server-side coding language such as Active Server Pages (ASP), ColdFusion®, Perl, Java®, JavaServer Pages (JSP), Hypertext Preprocessor (PHP), Python®, Ruby, Tel, Smalltalk, WebDNA®, or Groovy. In various embodiments, the web application is written to some extent in a database query language such as Structured Query Language (SQL). In various embodiments, the web application integrates enterprise server products such as IBM® Lotus Domino®. In various embodiments, the web application includes a media player element. In various embodiments, the media player element utilizes one or more of many suitable multimedia technologies, including, but not limited to, Adobe® Flash®, HTML5, Apple® QuickTime®, Microsoft® Silverlight®, Java®, and Unity®.

[0151] Mobile application

[0158] In various embodiments, the computer program includes a mobile application provided to the mobile digital processing device. In various embodiments, the mobile application is provided to the mobile digital processing device during its manufacture. In various embodiments, the mobile application is provided to the mobile digital processing device via a computer network as described herein.

[0152]

[0159] Mobile applications can be created using hardware, languages, and development environments known to those skilled in the art, and by techniques known to those skilled in the art. Those skilled in the art will recognize that mobile applications can be written in several languages. Suitable programming languages, in non-exclusive examples, include C, C++, C#, Objective-C, Java®, JavaScript, Pascal, Object Pascal, Rust, Python®, Ruby, VB.NET, WML, and XHTML / HTML with or without CSS, or a combination thereof.

[0153]

[0160] Suitable mobile application development environments are available from several sources. Commercial development environments include, but are not limited to, Airplay SDK, alcheMo, Appcelerator®, Celsius, Bedrock, Flash Lite, .NET Compact Framework, Rhomobile, and WorkLight Mobile Platform. Other development environments, but are not limited to, Lazarus, MobiFlex, MoSync, and Phonegap, are available free of charge. Mobile device manufacturers also distribute software developer kits, but are not limited to, iPhone and iPad (iOS) SDKs, Android® SDK, BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian SDK, webOS SDK, and Windows® Mobile SDK.

[0154]

[0161] Those skilled in the art will recognize that several commercial forums are available for the distribution of mobile applications, including, but not limited to, the Apple® App Store, Google® Play, Chrome WebStore, BlackBerry® App World, App Store for Palm devices, App Catalog for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo DSi Shop.

[0155] Standalone application

[0162] In various embodiments, a computer program is a program that runs as an independent computer process and includes standalone applications that are not add-ons to existing processes, such as plug-ins. Those skilled in the art will recognize that standalone applications are often compiled. A compiler is a computer program that translates source code written in a programming language into binary object code, such as assembly language or machine code. Suitable compiled programming languages, in non-limiting examples, include C, C++, Objective-C, COBOL, Delphi, Eiffel, Java®, Lisp, Python®, Visual Basic, and VB.NET, or combinations thereof. Compilation is often performed to create an executable program, at least in part. In various embodiments, a computer program includes one or more executable compiled applications.

[0156] Web browser plugin

[0163] In various embodiments, a computer program includes web browser plugins (e.g., extensions). In computing, a plugin is one or more software components that add specific functionality to a large software application. Software application makers support plugins, and third-party developers can create functionality to extend an application, easily add new functionality, or reduce the size of an application. Where supported, plugins can customize the functionality of a software application. For example, plugins are commonly used in web browsers and are used for video playback, interaction generation, virus scanning, and displaying specific file types. Those skilled in the art will be familiar with several web browser plugins, including Adobe® Flash® Player, Microsoft® Silverlight®, and Apple® QuickTime®. In various embodiments, a toolbar includes one or more web browser extensions, add-ins, or add-ons. In various embodiments, a toolbar includes one or more explorer bars, toolbands, or deskbands.

[0157]

[0164] Those skilled in the art will recognize that several plugin frameworks are available that enable the development of plugins in various programming languages, including, but not limited to, C++, Delphi, Java®, PHP, Python®, and VB.NET, or combinations thereof.

[0158]

[0165] A web browser (also called an internet browser) is a software application designed for use with a network-connected digital processing device to search, present, and view information resources on the World Wide Web. Suitable web browsers, in non-exclusive examples, include Microsoft® Internet Explorer®, Mozilla® Firefox®, Google® Chrome, Apple® Safari, Opera Software® Opera®, and KDE Konqueror. In various embodiments, a web browser is a mobile web browser. Mobile web browsers (also called microbrowsers, minibrowsers, and wireless browsers) are designed for use with mobile digital processing devices, including, in non-exclusive examples, handheld computers, tablet computers, netbooks, subnotebooks, smartphones, and personal digital assistants (PDAs). Suitable mobile web browsers, in no particular order, include Google® Android® browser, RIM BlackBerry® browser, Apple® Safari®, Palm® Blazer, Palm® WebOS® browser, Mozilla® Firefox® for mobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web, Nokia® browser, Opera® Software® Opera Mobile, and Sony PSP® browser.

[0159] Software Module

[0166] In various embodiments, the systems and methods disclosed herein include software, server and / or database modules, or incorporate their use in methods according to various embodiments disclosed herein. Software modules can be created by techniques known to those skilled in the art, using machines, software and languages ​​known to those skilled in the art. Software modules disclosed herein are implemented in many ways. In various embodiments, a software module includes files, sections of code, programming objects, programming structures, or a combination thereof. In even more various embodiments, a software module includes multiple files, multiple sections of code, multiple programming objects, multiple programming structures, or a combination thereof. In various embodiments, one or more software modules include, as non-limiting examples, web applications, mobile applications, and standalone applications. In various embodiments, a software module resides within one computer program or application. In various embodiments, a software module resides within two or more computer programs or applications. In various embodiments, a software module is hosted on one machine. In various embodiments, a software module is hosted on two or more machines. In various embodiments, a software module is hosted on a cloud computing platform. In various embodiments, the software module is hosted on one or more machines located in one location. In various embodiments, the software module is hosted on one or more machines located in two or more locations.

[0160] database

[0167] In various embodiments, the systems and methods disclosed herein include one or more databases, or incorporate the use of them in the methods according to the various embodiments disclosed herein. Those skilled in the art will recognize that many databases are suitable for storing and retrieving user, query, token, and result information. In various embodiments, suitable databases include, but are not limited to, relational databases, non-relational databases, object-oriented databases, object databases, entity-relational model databases, association databases, and XML databases. Further non-limited examples include SQL, PostgreSQL, MySQL, Oracle, DB2, and Sybase. In various embodiments, the database is internet-based. Further on the web. Suitable web browsers include, but are not limited to, Microsoft® Internet Explorer®, Mozilla® Firefox®, Google® Chrome, Apple® Safari®, Opera Software® Opera®, and KDE Konqueror. In various embodiments, the web browser is a mobile web browser. Mobile web browsers (also known as microbrowsers, minibrowsers, and wireless browsers) are designed for use on mobile digital processing devices, including, but not limited to, handheld computers, tablet computers, netbooks, subnotebooks, smartphones, and personal digital assistants (PDAs).Suitable mobile web browsers, in no particular order, include Google® Android® browser, RIM BlackBerry® browser, Apple® Safari®, Palm® Blazer, Palm® WebOS® browser, Mozilla® Firefox® for mobile, Microsoft® Internet Explorer® Mobile, Amazon® Kindle® Basic Web, Nokia® browser, Opera® Software® Opera Mobile, and Sony PSP® browser.

[0161]

[0168] In various embodiments, the database is web-based. In various embodiments, the database is cloud computing-based. In other embodiments, the database is based on one or more local computer storage devices.

[0162] Data Security

[0169] In various embodiments, the systems and methods disclosed herein include one or more features for preventing unauthorized access. Security measures can, for example, protect user data. In various embodiments, data is encrypted. In various embodiments, access to the system requires multi-factor authentication and access control layers. In various embodiments, access to the system requires two-factor authentication (e.g., a web-based interface). In various embodiments, two-factor authentication requires the user to enter an access code sent to the user's email or mobile phone in addition to a username and password. In some examples, if the user fails to enter the correct username and password, the user is locked out of the account. In various embodiments, the systems and methods disclosed herein may also include mechanisms for protecting the anonymity of user searches across the user's genome and any genome.

[0163] Additional considerations

[0170] The headers and / or subheadings between sections and subsections in this book are included solely for the purpose of improving readability and do not imply that the functions cannot be combined between sections and subsections. Therefore, sections and subsections do not describe separate embodiments.

[0164]

[0171] While this teaching is described in conjunction with various embodiments, it is not intended to be limited to such embodiments. On the contrary, this teaching encompasses various alternative forms, modifications, and equivalents, as will be understood by those skilled in the art. This description provides preferred exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the disclosure. Rather, this description of preferred exemplary embodiments provides a possible explanation for carrying out various embodiments.

[0165]

[0172] It is understood that various modifications may be made to the function and arrangement of the elements without departing from the spirit and scope set forth in the attached claims. Therefore, such modifications and variations are considered to be within the scope specified in the attached claims. Furthermore, the terms and expressions used are descriptive rather than restrictive, and in using such terms and expressions, there is no intention to exclude equivalents of the illustrated and described features or parts thereof, and it is recognized that various modifications are possible within the scope of the embodiments claimed herein.

[0166]

[0173] In describing various embodiments, this specification may present methods and / or processes as a specific sequence of steps. However, unless the method or process depends on the specific order of steps described herein, it will be readily apparent to those skilled in the art that the method or process should not be limited to the specific order of steps described herein, and the sequence may be changed, still remaining within the spirit and scope of various embodiments.

[0167]

[0174] In this specification, “or” is inclusive and non-exclusive unless explicitly indicated otherwise or by context. Therefore, in this specification, “A or B” means “A, B, or both” unless explicitly indicated otherwise or by context. Furthermore, “and” is both combined and plural unless explicitly indicated otherwise or by context. Therefore, in this specification, “A and B” means “A and B, together or individually” unless explicitly indicated otherwise or by context. The phrase “and / or” means “and” or “or.” For example, A, B, and / or C include A only, B only, C only, a combination of A and B, a combination of A and C, a combination of B and C, or a combination of A, B and C. In other words, “and / or” functions as an inclusive OR.

[0168]

[0175] Some embodiments of this disclosure include a system comprising one or more data processors. In some embodiments, the system includes a non-temporary computer-readable storage medium containing instructions that, when executed on one or more data processors, cause one or more data processors to perform some or all of the methods and / or some or all of the processes disclosed herein. Some embodiments of this disclosure include a computer program product tangibly embodied in a non-temporary machine-readable storage medium, which contains instructions configured to cause one or more data processors to perform some or all of the methods and / or processes disclosed herein.

[0169]

[0176] This description provides specific details to help understand the embodiments. However, it should be understood that embodiments may be carried out without these specific details. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form to avoid obscuring the embodiments with unnecessary details. In other examples, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0170] Embodiment 1. A computer-based method for predicting whether a patient has a medical condition or the risk of developing one, The processor receives multiple input features of a machine learning model configured to generate predictions of whether a patient has a medical condition or the risk of developing one. The processor determines whether each of the multiple input features is in a first state or a second state, where the first and second states indicate whether the input feature among the multiple input features is available to trigger a machine learning model, and each of the multiple input features has its own set of requirements for being in the first state. When all input features in multiple input features are in the first state, the processor uses a machine learning model to generate a prediction of the patient's risk of having or developing a medical condition. A computer implementation method, including

[0171] 2. The computer implementation method according to Embodiment 1, wherein the first input feature among the multiple input features needs to be associated with a first measurement value in order that the first input feature is in a first state.

[0172] 3. The computer implementation method according to Embodiment 2, further comprising the processor refraining from triggering a machine learning model in response to the processor determining that a first input feature is in a second state.

[0173] 4. The computer implementation method according to Embodiment 3, wherein at least some of a plurality of input features are received at a first time point, and the method further comprises a processor receiving the first input features at a second time point following the first time point.

[0174] 5. The processor determines that the first input feature among multiple input features is in the second state, The processor determines that the first input feature is in the second state, and then determines whether a measurement order for the first input feature has been generated. In response to the processor determining that no measurement order for the first input feature has been generated, the processor may (1) issue a notification alerting that a measurement order for the first input feature is required, or (2) generate at least one of the measurement orders for the first input feature. A computer implementation method according to Embodiment 1, further comprising the following:

[0175] 6. A computer implementation according to any one of Embodiments 1 to 5, further comprising a processor interpolating a substitution value into a second input feature among multiple input features before generating predictions using a machine learning model.

[0176] 7. The computer implementation method according to Embodiment 6, wherein the substitution value associated with the second input feature is determined by the processor using the second machine learning model, and the second plurality of input features to the second machine learning model used to compute the substitution value are the plurality of input features excluding the second input feature.

[0177] 8. The computer implementation method according to Embodiment 6, wherein the prediction is automatically generated using a machine learning model by interpolating the replacement values ​​into a second input feature.

[0178] 9. A computer implementation according to any one of embodiments 1 to 8, further comprising transforming a third input feature by a processor so that the transformed third input feature matches the set of parameters on which the machine learning model was trained.

[0179] 10. A computer implementation method according to any one of embodiments 1 to 9, further comprising detecting data drift in a machine learning model using a processor.

[0180] 11. The processor further includes receiving a request to trigger a machine learning model at a first time point, receiving a fourth input feature at a second time point, and determining whether each of the multiple input features is in a first state. At a third point in time, the processor determines the state of a fourth input feature, wherein when the second point in time is within a first predetermined time window, the state of the fourth input feature is determined to be the first state, and when the second point in time is outside the first predetermined time window, the state of the first input feature is determined to be the second state. A computer implementation method according to any one of Embodiments 1 to 10.

[0181] 12. The computer implementation method according to Embodiment 11, wherein the first predetermined time window includes a period prior to the first time point.

[0182] 13. The processor further includes receiving a request to trigger a machine learning model at a first time point, the machine learning model generating a prediction based in part on a fifth input feature among multiple input features, and determining whether each of the multiple input features is in a first state. The processor determines the state of the fifth input feature at the second time point, and determining the state of the fifth input feature is The processor determines that the fifth input feature can be complemented, The processor determines whether the order of the measurement of the fifth input feature began within a second predetermined time window, If the order has not been started and the second time point follows the second predetermined time window, the fifth input feature is the first state. If the order has not been started and the second time point is within the second predetermined time window, the fifth input feature is the second state. To make a judgment, In response to the processor determining that an order has been initiated, it is determined whether a second point in time falls within a third predetermined time window. If the second time point falls within the third predetermined time window, the fifth input feature is the second state. If the second time point follows the third predetermined time window, the fifth input feature is the first state. To make a judgment, The determination that includes, In response to the processor determining that the fifth input feature is in the first state, the replacement measurement value of the fifth input feature is determined, The processor complements the substitution measurement with a fifth input feature, A computer implementation method according to any one of embodiments 1 to 10, including the above.

[0183] 14. The processor determines that the fifth input feature is the second state, In response to the determination that the fifth input feature is in the second state, it is determined that a predetermined amount of time has not yet elapsed since the first time point, At the third point in time, the state of the fifth input feature is determined, A non-temporary computer-readable medium according to Embodiment 13, further comprising the above.

[0184] 15. The computer implementation method according to Embodiment 13, wherein the third predetermined time window begins at the second time point.

[0185] 16. The multiple input features include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group. The first group includes input features consisting of age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment, clinical decision support alerts, clinician or other chart notes, diagnostic codes, procedures performed on the patient, patient's current medications, interventions, patient care settings, medical imaging data or assessments, electrography, endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and inspired oxygen concentration. The second group includes input features consisting of white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers, clinical severity measurements and composites, biopsy data, and microbiological test data. The third group includes input features consisting of procalcitonin, C-reactive protein, protein biomarkers, genomic biomarkers, gene expression or transcriptome biomarkers, and composite biomarker scores. A computer implementation method according to any one of Embodiments 1 to 15.

[0186] 17. Multiple input features are, Age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group, From the second group, white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate, Procalcitonin and C-reactive protein from the third group, A computer implementation method according to Embodiment 16, including the method described above.

[0187] 18. A computer implementation method according to any one of embodiments 1 to 17, further comprising the processor receiving a request to trigger a machine learning model to generate a prediction, wherein, upon receiving the request, the processor begins reading patient data stored in memory, the patient data comprising at least some of a plurality of input features, and at least some of the plurality of input features are received in real time when at least some of the plurality of input features are measured.

[0188] 19. A computer implementation method according to any one of Embodiments 1 to 18, wherein the prediction further includes a category of prediction and an importance value of a sixth input feature among a plurality of input features, the importance value indicating the contribution of the sixth input feature to the prediction.

[0189] 20. A computer implementation method according to any one of embodiments 1 to 19, wherein at least some of the multiple input features include multiple seventh input features, and the method further comprises a processor selecting a single seventh input feature from among the multiple seventh input features that generate a prediction.

[0190] 21. The processor receives an instruction that it is expected that a measurement of the first input feature among multiple input features will be received at a first time outside a given time window, The processor adjusts a given time window to generate an adjusted time window, wherein the first time is within the adjusted time window. A computer implementation method according to any one of embodiments 1 to 20, further comprising the above.

[0191] 22. A computer implementation method according to any one of Embodiments 1 to 21, wherein the multiple input features are all of the input features of the machine learning model.

[0192] 23. A system, Memory and A processor that communicates with memory, Receiving multiple input features of a machine learning model configured to generate predictions of whether a patient has a medical condition or the risk of developing one, Determining whether each of multiple input features is in a first state or a second state, wherein the first and second states indicate whether an input feature among the multiple input features is available to trigger a machine learning model, and each of the multiple input features has its own set of requirements for being in the first state. When all input features in multiple input features are in state 1, a machine learning model is used to generate a prediction of whether the patient has a medical condition or the risk of developing one. A processor configured to perform an operation, A system that includes this.

[0193] 24. The system according to Embodiment 23, wherein a first input feature among a plurality of input features needs to be associated with a first measurement value in order that the first input feature is in a first state.

[0194] 25. The system according to Embodiment 24, further comprising refraining from triggering a machine learning model in response to determining that a first input feature is in a second state.

[0195] 26. The operation is, Determining that the first input feature among multiple input features is in the second state, After determining that the first input feature is in the second state, it is determined whether a measurement order for the first input feature has been generated. In response to determining that an order for the first input feature and a measurement order for the first input feature have not been generated, the system will automatically generate (1) a notification alerting that a measurement order for the first input feature is required, or (2) at least one of the measurement orders for the first input feature. The system according to embodiment 25, further comprising the above.

[0196] 27. The system according to Embodiment 25, wherein at least some of a plurality of input features are received at a first time point, and the method further comprises receiving the first input features at a second time point following the first time point.

[0197] 28. The system according to any of embodiments 23 to 27, further comprising the operation of interpolating a replacement value to a second input feature among multiple input features before generating a prediction using a machine learning model.

[0198] 29. The system according to Embodiment 28, wherein the substitution value associated with the second input feature is determined using the second machine learning model, and the second plurality of input features to the second machine learning model used to compute the substitution value are the plurality of input features excluding the second input feature.

[0199] 30. The system according to Embodiment 28, wherein the prediction is automatically generated using a machine learning model by interpolating the replacement values ​​into a second input feature.

[0200] 31. The system according to any one of embodiments 23 to 30, further comprising transforming a third input feature so that the transformed third input feature matches the set of parameters on which the machine learning model was trained.

[0201] 32. The system according to any one of embodiments 23 to 31, further comprising detecting data drift in a machine learning model.

[0202] 33. The operation is, The processor further includes receiving a request to trigger a machine learning model at a first time point, receiving a fourth input feature at a second time point, and determining whether each of the multiple input features is in a first state. The third point in time involves determining the state of the fourth input feature, wherein when the second point in time is within the first predetermined time window, the state of the fourth input feature is determined to be the first state, and when the second point in time is outside the first predetermined time window, the state of the first input feature is determined to be the second state. including, A non-temporary computer-readable medium according to any of embodiments 23 to 32.

[0203] 34. The system according to embodiment 33, wherein the first predetermined time window includes a period prior to the first time point.

[0204] 35. The operation further includes receiving a request to trigger a machine learning model at a first time point, the machine learning model generating a prediction based in part on a fifth input feature among multiple input features, and determining whether each of the multiple input features is in a first state. At the second point in time, the state of the fifth input feature is determined, and determining the state of the fifth input feature is Determining that the fifth input feature is complementary, Regarding the measurement value of the fifth input feature, the determination is made as to whether the order started within a second predetermined time window. If the order has not been started and the second time point follows the second predetermined time window, the fifth input feature is the first state. If the order has not been started and the second time point is within the second predetermined time window, the fifth input feature is the second state. To make a judgment, In response to the determination that an order has been initiated, the process involves determining whether the second point in time falls within a third predetermined time window. If the second time point falls within the third predetermined time window, the fifth input feature is the second state. If the second time point follows the third predetermined time window, the fifth input feature is the first state. To make a judgment, The determination that includes, In response to determining that the fifth input feature is in the first state, the substitution measurement value of the fifth input feature is determined, Complementing the substitution measurement value with the fifth input feature, A system according to any one of embodiments 23 to 34, including the system described above.

[0205] 36. The operation is, The processor determines that the fifth input feature is the second state, In response to the determination that the fifth input feature is in the second state, it is determined that a predetermined amount of time has not yet elapsed since the first time point, At the third point in time, the state of the fifth input feature is determined, The system according to embodiment 35, further comprising the above.

[0206] 37. The system according to embodiment 35, wherein the third predetermined time window begins at the second time point.

[0207] 38. The multiple input features include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group. The first group includes input features consisting of age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment, clinical decision support alerts, clinician or other chart notes, diagnostic codes, procedures performed on the patient, patient's current medications, interventions, patient care settings, medical imaging data or assessments, electrography, endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and inspired oxygen concentration. The second group includes input features consisting of white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers, clinical severity measurements and composites, biopsy data, and microbiological test data. The third group includes input features consisting of procalcitonin, C-reactive protein, protein biomarkers, genomic biomarkers, gene expression or transcriptome biomarkers, and composite biomarker scores. The system according to any one of embodiments 23 to 37.

[0208] 39. Multiple input features are, Age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group, From the second group, white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate, Procalcitonin and C-reactive protein from the third group, The system according to embodiment 38, including the system described above.

[0209] 40. The operation further includes receiving a request to trigger a machine learning model to generate a prediction, the processor, upon receiving the request, begins reading patient data stored in memory, the patient data including at least some of a plurality of input features, the system according to any one of embodiments 23 to 39, where at least some of the plurality of input features are received in real time when at least some of the plurality of input features are measured.

[0210] 41. The system according to any of Embodiments 23 to 40, wherein the prediction further includes a prediction category and an importance value of a sixth input feature among a plurality of input features, the importance value indicating the contribution of the sixth input feature to the prediction.

[0211] 42. The system according to any of embodiments 23 to 41, wherein at least some of the multiple input features include multiple seventh input features, and the operation further includes selecting a single seventh input feature from the multiple seventh input features that generate a prediction.

[0212] 43. The operation is, Receiving an instruction that the measurement value of the first input feature among multiple input features is expected to be received at a first time outside a predetermined time window, Adjusting a predetermined time window to generate an adjusted time window, wherein the first time is within the adjusted time window. A system according to any one of embodiments 23 to 42, further comprising the above.

[0213] 44. The system according to any of Embodiments 23 to 43, wherein the multiple input features are all of the input features of the machine learning model.

[0214] 45. A non-temporary computer-readable medium for storing instructions, wherein when an instruction is executed by a processor, the processor... Receiving multiple input features of a machine learning model configured to generate predictions of whether a patient has a medical condition or the risk of developing one, Determining whether each of multiple input features is in a first state or a second state, wherein the first and second states indicate whether an input feature among the multiple input features is available to trigger a machine learning model, and each of the multiple input features has its own set of requirements for being in the first state. When all input features in multiple input features are in state 1, a machine learning model is used to generate a prediction of whether the patient has a medical condition or the risk of developing one. A non-temporary computer-readable medium that performs an action, including [the specified action].

[0215] 46. ​​A non-temporary computer-readable medium according to Embodiment 45, wherein a first input feature among a plurality of input features needs to be associated with a first measurement value in order that the first input feature is in a first state.

[0216] 47. The non-temporary computer-readable medium according to Embodiment 46, further comprising refraining from triggering a machine learning model in response to determining that a first input feature is in a second state.

[0217] 48. The operation is, Determining that the first input feature among multiple input features is in the second state, After determining that the first input feature is in the second state, it is determined whether a measurement order for the first input feature has been generated. In response to determining that an order for the first input feature and a measurement order for the first input feature have not been generated, the system will automatically generate (1) a notification alerting that a measurement order for the first input feature is required, or (2) at least one of the measurement orders for the first input feature. A non-temporary computer-readable medium according to Embodiment 47, further comprising the above.

[0218] 49. A non-temporary computer-readable medium according to Embodiment 47, wherein at least some of a plurality of input features are received at a first time point, and the method further comprises receiving the first input features at a second time point following the first time point.

[0219] 50. A non-temporary computer-readable medium according to any of embodiments 45 to 49, further comprising the operation of imputing a substitution value in a second input feature of a plurality of input features before generating a prediction using a machine learning model.

[0220] 51. The non-temporary computer-readable medium according to Embodiment 50, wherein the substitution value associated with the second input feature is determined using a second machine learning model, and the second plurality of input features to the second machine learning model used to compute the substitution value are plurality of input features excluding the second input feature.

[0221] 52. The prediction is automatically generated using a machine learning model, where the replacement value is interpolated into a second input feature, in a non-temporal computer-readable medium as described in Embodiment 50.

[0222] 53. A non-temporary computer-readable medium according to any of embodiments 45 to 52, further comprising transforming a third input feature such that the transformed third input feature matches the set of parameters on which a machine learning model was trained.

[0223] 54. A non-temporary computer-readable medium according to any of embodiments 45 to 53, further comprising detecting data drift in a machine learning model.

[0224] 55. The operation is, The processor further includes receiving a request to trigger a machine learning model at a first time point, receiving a fourth input feature at a second time point, and determining whether each of the multiple input features is in a first state. At the third point in time, the state of the fourth input feature is determined, wherein when the second point in time falls within the first predetermined time window, the state of the fourth input feature is determined to be the first state, and when the second point in time falls outside the first predetermined time window, the state of the first input feature is determined to be the second state. Including making a determination, A non-temporary computer-readable medium according to any of embodiments 45 to 54.

[0225] 56. The non-temporary computer-readable medium according to Embodiment 55, wherein the first predetermined time window includes a period prior to the first time point.

[0226] 57. The operation further includes receiving a request to trigger a machine learning model at a first time point, the machine learning model generating a prediction based in part on a fifth input feature among multiple input features, and determining whether each of the multiple input features is in a first state. At the second point in time, the state of the fifth input feature is determined, and determining the state of the fifth input feature is Determining that the fifth input feature is complementary, Regarding the measurement value of the fifth input feature, the determination is made as to whether the order started within a second predetermined time window. If the order has not been started and the second time point follows the second predetermined time window, the fifth input feature is the first state. If the order has not been started and the second time point is within the second predetermined time window, the fifth input feature is the second state. To make a judgment, In response to the determination that an order has been initiated, the process involves determining whether the second point in time falls within a third predetermined time window. If the second time point falls within the third predetermined time window, the fifth input feature is the second state. If the second time point follows the third predetermined time window, the fifth input feature is the first state. To make a judgment, The determination that includes, In response to determining that the fifth input feature is in the first state, the substitution measurement value of the fifth input feature is determined, Complementing the substitution measurement value with the fifth input feature, A non-temporary computer-readable medium according to any of embodiments 45 to 54, including the above.

[0227] 58. The operation is, The processor determines that the fifth input feature is the second state, In response to the determination that the fifth input feature is in the second state, it is determined that a predetermined amount of time has not yet elapsed since the first time point, At the third point in time, the state of the fifth input feature is determined, A non-temporary computer-readable medium according to Embodiment 57, further comprising the above.

[0228] 59. A non-temporary computer-readable medium according to Embodiment 57, wherein the third predetermined time window begins at the second time point.

[0229] 60. The multiple input features include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group. The first group includes input features consisting of age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment, clinical decision support alerts, clinician or other chart notes, diagnostic codes, procedures performed on the patient, patient's current medications, interventions, patient care settings, medical imaging data or assessments, electrography, endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and inspired oxygen concentration. The second group includes input features consisting of white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers, clinical severity measurements and composites, biopsy data, and microbiological test data. The third group includes input features consisting of procalcitonin, C-reactive protein, protein biomarkers, genomic biomarkers, gene expression or transcriptome biomarkers, and composite biomarker scores. A non-temporary computer-readable medium according to any of embodiments 45 to 59.

[0230] 61. Multiple input features are, Age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure, and body temperature from the first group, From the second group, white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate, Procalcitonin and C-reactive protein from the third group, A non-temporary computer-readable medium according to Embodiment 60, further comprising the above.

[0231] 62. The operation further includes receiving a request to trigger a machine learning model to generate a prediction, wherein, upon receiving the request, the processor begins reading patient data stored in memory, the patient data comprising at least some of a plurality of input features, and at least some of the plurality of input features are received in real time when at least some of the plurality of input features are measured, as described in any embodiment 45 to 61 of the non-temporary computer-readable medium.

[0232] 63. A non-temporary computer-readable medium according to any of embodiments 45 to 62, wherein the prediction further includes a category of prediction and an importance value of a sixth input feature among a plurality of input features, the importance value indicating the contribution of the sixth input feature to the prediction.

[0233] 64. A non-temporary computer-readable medium according to any of embodiments 45 to 63, wherein at least some of the multiple input features include multiple seventh input features, and the method further includes a processor selecting a single seventh input feature from among the multiple seventh input features that generate a prediction.

[0234] 65. The operation is, Receiving an instruction that the measurement value of the first input feature among multiple input features is expected to be received at a first time outside a predetermined time window, Adjusting a predetermined time window to generate an adjusted time window, wherein the first time is within the adjusted time window. A system according to any one of embodiments 45 to 64, further comprising the above.

[0235] 66. A non-temporary computer-readable medium according to embodiments 45-65, wherein the multiple input features are all of the input features of the machine learning model.

Claims

1. A computer-based method for predicting whether a patient has a medical condition or the risk of developing one, The processor receives multiple input features of a machine learning model configured to generate predictions of the risk of the patient having or developing the medical condition, The processor determines whether each of the plurality of input features is in a first state or a second state, wherein the first state and the second state indicate whether an input feature among the plurality of input features is available to trigger the machine learning model, and each of the plurality of input features has a set of requirements for being in the first state. If all of the input features in the plurality of input features are in the first state, the processor uses the machine learning model to generate the prediction of the risk of the patient having or developing the medical condition. A computer implementation method, including

2. The computer implementation method according to claim 1, wherein the first input feature among the plurality of input features needs to be associated with a first measurement value in order that the first input feature is in the first state.

3. The computer implementation method according to claim 2, further comprising the processor refraining from triggering the machine learning model in response to the processor determining that the first input feature is in the second state.

4. The computer implementation method according to claim 3, wherein at least some of the plurality of input features are received at a first time point, and the method further comprises the processor receiving the first input features at a second time point following the first time point.

5. The processor determines that the first input feature among the plurality of input features is in the second state, After the processor determines that the first input feature is in the second state, it determines whether a measurement order for the first input feature has been generated. In response to the processor determining that no measurement order has been generated for the first input feature, the processor may: (1) issue a notification alerting that a measurement order for the first input feature is required, or (2) generate at least one of the measurement orders for the first input feature. The computer implementation method according to claim 1, further comprising:

6. The computer implementation method according to any one of claims 1 to 5, further comprising the processor imputing a substitution value in a second input feature among the plurality of input features before generating the prediction using the machine learning model.

7. The computer implementation method according to claim 6, wherein the substitution value related to the second input feature is determined by the processor using a second machine learning model, and the second plurality of input features to the second machine learning model used to compute the substitution value are the plurality of input features excluding the second input feature.

8. The computer implementation method according to claim 6, wherein the prediction is automatically generated using the machine learning model by interpolating the substitution values ​​to the second input features.

9. The computer implementation method according to any one of claims 1 to 8, further comprising the processor transforming the third input feature such that the transformed third input feature matches the set of parameters on which the machine learning model was trained.

10. The computer implementation method according to any one of claims 1 to 9, further comprising using the processor to detect data drift in the machine learning model.

11. The processor further includes receiving a request to trigger the machine learning model at a first time point, receiving a fourth input feature at a second time point, and determining whether each of the plurality of input features is in the first state. At a third point in time, the processor determines the state of the fourth input feature, wherein when the second point in time is within a first predetermined time window, the state of the fourth input feature is determined to be the first state, and when the second point in time is outside the first predetermined time window, the state of the first input feature is determined to be the second state. The computer implementation method according to any one of claims 1 to 10.

12. The computer implementation method according to claim 11, wherein the first predetermined time window includes a period prior to the first time point.

13. The processor further includes receiving a request to trigger the machine learning model at a first time point, the machine learning model generating the prediction based in part on a fifth input feature among the plurality of input features, and determining whether each of the plurality of input features is in the first state. The processor determines the state of the fifth input feature at a second time point, and determining the state of the fifth input feature is The processor determines that the fifth input feature can be complemented, The processor determines whether the order of the measured value of the fifth input feature started within a second predetermined time window, If an order has not been started and the second time point follows the second predetermined time window, the fifth input feature is in the first state. If an order has not been started and the second time point is within the second predetermined time window, the fifth input feature is in the second state. To make a judgment, In response to the processor determining that the order has started, the process involves determining whether the second time point falls within a third predetermined time window. If the second time point falls within the third predetermined time window, the fifth input feature is in the second state. If the second time point follows the third predetermined time window, the fifth input feature is the first state. To make a judgment, This includes making a determination, In response to the processor determining that the fifth input feature is in the first state, the replacement measurement value of the fifth input feature is determined. The processor complements the substitution measurement value with the fifth input feature, A computer implementation method according to any one of claims 1 to 10, including the method described in any one of claims 1 to 10.

14. The processor determines that the fifth input feature is in the second state, In response to the determination that the fifth input feature is in the second state, it is determined that a predetermined amount of time has not yet elapsed from the first time point, At the third point in time, the state of the fifth input feature described above is determined, The computer implementation method according to claim 13, further comprising:

15. The computer implementation method according to claim 13, wherein the third predetermined time window begins at the second time.

16. The plurality of input features include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group. The first group includes the input features of a group consisting of age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment, clinical decision support alerts, clinician or other chart notes, diagnostic codes, procedures performed on the patient, patient's current medications, interventions, patient care settings, medical imaging data or assessments, electrography, endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and inspired oxygen concentration. The second group includes the input features of a group consisting of white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers, clinical severity measurements and composites, biopsy data, and microbiological test data. The third group includes the input features of a group consisting of procalcitonin, C-reactive protein, protein biomarker, genomic biomarker, gene expression or transcriptome biomarker, and composite biomarker score. The computer implementation method according to any one of claims 1 to 15.

17. The aforementioned multiple input features are, Age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure and body temperature from the first group, From the second group mentioned above, white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate, Procalcitonin and C-reactive protein from the third group mentioned above, The computer implementation method according to claim 16, including the method described in claim 16.

18. The computer implementation method according to any one of claims 1 to 17, further comprising the processor receiving a request to trigger the machine learning model to generate the prediction, wherein, upon receiving the request, the processor begins reading patient data stored in memory, the patient data comprising at least some of the plurality of input features, and the at least some of the plurality of input features are received in real time when the at least some of the plurality of input features are measured.

19. The computer implementation method according to any one of claims 1 to 18, wherein the prediction further includes a category of the prediction and an importance value of a sixth input feature among the plurality of input features, the importance value indicating the contribution of the sixth input feature to the prediction.

20. A computer implementation of the method according to any one of claims 1 to 19, wherein at least some of the plurality of input features include a plurality of seventh input features, and the method further comprises the processor selecting a single seventh input feature from the plurality of seventh input features that generate the prediction.

21. The processor receives an instruction that it is expected that the measurement value of the first input feature among the plurality of input features will be received at a first time outside a predetermined time window, The processor adjusts the predetermined time window to generate an adjusted time window, wherein the first time is within the adjusted time window. A computer implementation method according to any one of claims 1 to 20, further comprising:

22. The computer implementation method according to any one of claims 1 to 21, wherein the plurality of input features are all of the input features of the machine learning model.

23. It is a system, Memory and A processor that communicates with the aforementioned memory, Receiving multiple input features of a machine learning model configured to generate predictions of whether a patient has a medical condition or the risk of developing one, Determining whether each of the plurality of input features is in a first state or a second state, wherein the first state and the second state indicate whether an input feature among the plurality of input features is available to trigger the machine learning model, and each of the plurality of input features has a set of requirements for being in the first state. If all of the input features in the plurality of input features are in the first state, the machine learning model is used to generate the prediction of the risk of the patient having or developing the medical condition. A processor configured to perform operations including, A system that includes this.

24. The system according to claim 23, wherein the first input feature among the plurality of input features needs to be associated with a first measurement value in order that the first input feature is in the first state.

25. The system according to claim 24, further comprising refraining from triggering the machine learning model in response to determining that the first input feature is in the second state.

26. The aforementioned operation is, Determining that the first input feature among the plurality of input features is in the second state, After determining that the first input feature is in the second state, it is determined whether a measurement order for the first input feature has been generated. In response to the determination that the order of the first input feature and the measurement order of the first input feature have not been generated, the system automatically generates (1) a notification alerting that the measurement order of the first input feature is required, or (2) at least one of the measurement orders of the first input feature. The system according to claim 25, further comprising:

27. The system according to claim 25, wherein at least some of the plurality of input features are received at a first time point, and the method further comprises receiving the first input feature at a second time point following the first time point.

28. The system according to any one of claims 23 to 27, wherein the operation further comprises imputing a substitution value in a second input feature of the plurality of input features before generating the prediction using the machine learning model.

29. The system according to claim 28, wherein the substitution value associated with the second input feature is determined using a second machine learning model, and the second plurality of input features to the second machine learning model used to compute the substitution value are the plurality of input features excluding the second input feature.

30. The system according to claim 28, wherein the prediction is automatically generated using the machine learning model by interpolating the substitution values ​​to the second input features.

31. The system according to any one of claims 23 to 30, further comprising transforming the third input feature such that the transformed third input feature matches the set of parameters on which the machine learning model was trained.

32. The system according to any one of claims 23 to 31, wherein the operation further comprises detecting data drift in the machine learning model.

33. The aforementioned operation is, The processor further includes receiving a request to trigger the machine learning model at a first time point, receiving a fourth input feature at a second time point, and determining whether each of the plurality of input features is in the first state. At a third point in time, the state of the fourth input feature is determined, wherein when the second point in time is within a first predetermined time window, the state of the fourth input feature is determined to be the first state, and when the second point in time is outside the first predetermined time window, the state of the first input feature is determined to be the second state. The system according to any one of claims 23 to 32, including the system described in any one of claims 23 to 32.

34. The system according to claim 33, wherein the first predetermined time window includes a period prior to the first time point.

35. The operation further includes receiving a request to trigger the machine learning model at a first time point, the machine learning model generating the prediction based in part on a fifth input feature among the plurality of input features, and determining whether each of the plurality of input features is in the first state. At the second point in time, the state of the fifth input feature is determined, and determining the state of the fifth input feature is Determining that the fifth input feature is complementary, With respect to the measurement value of the fifth input feature, the determination is made as to whether the order started within a second predetermined time window. If an order has not been started and the second time point follows the second predetermined time window, the fifth input feature is in the first state. If an order has not been started and the second time point is within the second predetermined time window, the fifth input feature is in the second state. To make a judgment, In response to the determination that the aforementioned order has been initiated, the determination is made whether the second time point falls within a third predetermined time window. If the second time point falls within the third predetermined time window, the fifth input feature is in the second state. If the second time point follows the third predetermined time window, the fifth input feature is the first state. To make a judgment, This includes making a determination, In response to determining that the fifth input feature is in the first state, the replacement measurement value of the fifth input feature is determined, The substitution measurement value is to be used to complement the fifth input feature, The system according to any one of claims 23 to 34, including the system described in any one of claims 23 to 34.

36. The aforementioned operation is, The processor determines that the fifth input feature is in the second state, In response to the determination that the fifth input feature is in the second state, it is determined that a predetermined amount of time has not yet elapsed from the first time point, At the third point in time, the state of the fifth input feature described above is determined, The system according to claim 35, further comprising:

37. The system according to claim 35, wherein the third predetermined time window begins at the second time.

38. The plurality of input features include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group. The first group includes the input features of a group consisting of age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment, clinical decision support alerts, clinician or other chart notes, diagnostic codes, procedures performed on the patient, patient's current medications, interventions, patient care settings, medical imaging data or assessments, electrography, endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and inspired oxygen concentration. The second group includes the input features of a group consisting of white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers, clinical severity measurements and composites, biopsy data, and microbiological test data. The third group includes the input features of a group consisting of procalcitonin, C-reactive protein, protein biomarker, genomic biomarker, gene expression or transcriptome biomarker, and composite biomarker score. The system according to any one of claims 23 to 37.

39. The aforementioned multiple input features are, Age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure and body temperature from the first group, From the second group mentioned above, white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate, Procalcitonin and C-reactive protein from the third group mentioned above, The system according to claim 38, including the system described in claim 38.

40. The system according to any one of claims 23 to 39, wherein the operation further includes receiving a request to trigger the machine learning model to generate the prediction, and upon receiving the request, the processor begins reading patient data stored in memory, the patient data including at least some of the plurality of input features, and at least some of the plurality of input features are received in real time when at least some of the plurality of input features are measured.

41. The system according to any one of claims 23 to 40, wherein the prediction further includes a category of the prediction and an importance value of a sixth input feature among the plurality of input features, the importance value indicating the contribution of the sixth input feature to the prediction.

42. The system according to any one of claims 23 to 41, wherein at least some of the plurality of input features include a plurality of seventh input features, and the operation further includes selecting a single seventh input feature from the plurality of seventh input features that generate the prediction.

43. The aforementioned operation is, Receiving an instruction that the measurement value of the first input feature among the plurality of input features is expected to be received at a first time outside a predetermined time window, Adjusting a predetermined time window to generate an adjusted time window, wherein the first time is within the adjusted time window. The system according to any one of claims 23 to 42, further comprising:

44. The system according to any one of claims 23 to 43, wherein the plurality of input features are all of the input features of the machine learning model.

45. A non-temporary computer-readable medium for storing instructions, wherein, when an instruction is executed by a processor, the processor... Receiving multiple input features of a machine learning model configured to generate predictions of whether a patient has a medical condition or the risk of developing one, Determining whether each of the plurality of input features is in a first state or a second state, wherein the first state and the second state indicate whether an input feature among the plurality of input features is available to trigger the machine learning model, and each of the plurality of input features has a set of requirements for being in the first state. If all of the input features in the plurality of input features are in the first state, the machine learning model is used to generate the prediction of the risk of the patient having or developing the medical condition. A non-temporary computer-readable medium that performs an action, including [the specified action].

46. The non-temporary computer-readable medium according to claim 45, wherein the first input feature among the plurality of input features needs to be associated with a first measurement value in order that the first input feature is in the first state.

47. The non-temporary computer-readable medium according to claim 46, further comprising refraining from triggering the machine learning model in response to determining that the first input feature is in the second state.

48. The aforementioned operation is, Determining that the first input feature among the plurality of input features is in the second state, After determining that the first input feature is in the second state, it is determined whether a measurement order for the first input feature has been generated. In response to the determination that the order of the first input feature and the measurement order of the first input feature have not been generated, the system automatically generates (1) a notification alerting that the measurement order of the first input feature is required, or (2) at least one of the measurement orders of the first input feature. A non-temporary computer-readable medium according to claim 47, further comprising:

49. The non-temporary computer-readable medium according to claim 47, wherein at least some of the plurality of input features are received at a first time point, and the method further comprises receiving the first input feature at a second time point following the first time point.

50. The non-temporary computer-readable medium according to any one of claims 45 to 49, further comprising the operation of interpolating a substitution value to a second input feature among the plurality of input features before generating the prediction using the machine learning model.

51. The non-temporary computer-readable medium according to claim 50, wherein the substitution value related to the second input feature is determined using a second machine learning model, and the second plurality of input features to the second machine learning model used to compute the substitution value are the plurality of input features excluding the second input feature.

52. The non-temporary computer-readable medium according to claim 50, wherein the prediction is automatically generated using the machine learning model by interpolating the substitution values ​​to the second input features.

53. The non-temporary computer-readable medium according to any one of claims 45 to 52, further comprising transforming the third input feature such that the transformed third input feature matches the set of parameters on which the machine learning model was trained.

54. The non-temporary computer-readable medium according to any one of claims 45 to 53, further comprising detecting data drift in the machine learning model.

55. The aforementioned operation is, The processor further includes receiving a request to trigger the machine learning model at a first time point, receiving a fourth input feature at a second time point, and determining whether each of the plurality of input features is in the first state. The third time step involves determining the state of the fourth input feature, wherein when the second time step is within a first predetermined time window, the state of the fourth input feature is determined to be the first state, and when the second time step is outside the first predetermined time window, the state of the first input feature is determined to be the second state. A non-temporary computer-readable medium according to any one of claims 45 to 54.

56. The non-temporary computer-readable medium according to claim 55, wherein the first predetermined time window includes a period prior to the first time point.

57. The operation further includes receiving a request to trigger the machine learning model at a first time point, the machine learning model generating the prediction based in part on a fifth input feature among the plurality of input features, and determining whether each of the plurality of input features is in the first state. At the second point in time, the state of the fifth input feature is determined, and determining the state of the fifth input feature is Determining that the fifth input feature is complementary, With respect to the measurement value of the fifth input feature, the determination is made as to whether the order started within a second predetermined time window. If an order has not been initiated and the second time point follows the second predetermined time window, the fifth input feature is the first state. If an order has not been started and the second time point is within the second predetermined time window, the fifth input feature is in the second state. To make a judgment, In response to the determination that the aforementioned order has been initiated, the determination is made whether the second time point falls within a third predetermined time window. If the second time point falls within the third predetermined time window, the fifth input feature is in the second state. If the second time point follows the third predetermined time window, the fifth input feature is the first state. To make a judgment, The determination that includes, In response to determining that the fifth input feature is in the first state, the replacement measurement value of the fifth input feature is determined, The substitution measurement value is to be used to complement the fifth input feature, A non-temporary computer-readable medium according to any one of claims 45 to 56, including the following:

58. The aforementioned operation is, The processor determines that the fifth input feature is in the second state, In response to the determination that the fifth input feature is in the second state, it is determined that a predetermined amount of time has not yet elapsed from the first time point, At the third point in time, the state of the fifth input feature described above is determined, A non-temporary computer-readable medium according to claim 57, further comprising:

59. The non-temporary computer-readable medium according to claim 57, wherein the third predetermined time window begins at the second time.

60. The plurality of input features include at least one input feature from a first group, at least one input feature from a second group, and at least one input feature from a third group. The first group includes the input features of a group consisting of age, sex at birth, race, ethnicity, patient's past medical history, patient's current illness or symptoms, neurological assessment, clinical decision support alerts, clinician or other chart notes, diagnostic codes, procedures performed on the patient, patient's current medications, interventions, patient care settings, medical imaging data or assessments, electrography, endoscopy, systolic blood pressure, diastolic blood pressure, body temperature, respiratory rate, heart rate, blood oxygen saturation, and inspired oxygen concentration. The second group includes the input features of a group consisting of white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, lactate, glucose, calcium, hematocrit, hemoglobin concentration, laboratory-derived biomarkers, clinical severity measurements and composites, biopsy data, and microbiological test data. The third group includes the input features of a group consisting of procalcitonin, C-reactive protein, protein biomarker, genomic biomarker, gene expression or transcriptome biomarker, and composite biomarker score. A non-temporary computer-readable medium according to any one of claims 45 to 59.

61. The aforementioned multiple input features are, Age, diastolic blood pressure, heart rate, pulse oximetry, respiratory rate, systolic blood pressure and body temperature from the first group, From the second group mentioned above, white blood cell count, lymphocyte count, neutrophil count, platelet count, creatinine, blood urea nitrogen, potassium, chloride, total carbon dioxide, sodium, albumin, bilirubin, and lactate, Procalcitonin and C-reactive protein from the third group mentioned above, A non-temporary computer-readable medium according to claim 60, including the following:

62. The operation further includes receiving a request to trigger the machine learning model to generate the prediction, the processor, upon receiving the request, begins reading patient data stored in memory, the patient data comprising at least some of the plurality of input features, the at least some of the plurality of input features being received in real time when the at least some of the plurality of input features are measured, the non-temporary computer-readable medium according to any one of claims 45 to 61.

63. The prediction further comprises a category of the prediction and an importance value of a sixth input feature among the plurality of input features, wherein the importance value indicates the contribution of the sixth input feature to the prediction, according to any one of claims 45 to 62.

64. A non-temporary computer-readable medium according to any one of claims 45 to 63, wherein at least some of the plurality of input features include a plurality of seventh input features, and the method further comprises the processor selecting a single seventh input feature from the plurality of seventh input features that generate the prediction.

65. The aforementioned operation is, Receiving an instruction that the measurement value of the first input feature among the plurality of input features is expected to be received at a first time outside a predetermined time window, Adjusting a predetermined time window to generate an adjusted time window, wherein the first time is within the adjusted time window. A non-temporary computer-readable medium according to any one of claims 45 to 64, further comprising:

66. The non-temporary computer-readable medium according to any one of claims 45 to 65, wherein the plurality of input features are all of the input features of the machine learning model.

Images