System and method for evaluating prediction model configured to predict occurrence of clinical event

Abstract

A system and method are provided for evaluating a prediction model configured to predict occurrence of a clinical event in multiple patients during corresponding patient stays, where the prediction model outputs risk scores for the patients. The method includes identifying patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold; providing notifications for risk scores that cross the risk threshold; determining a patient-level confusion matrix based on the notifications, where each patient stay contributes one patient data point to the patient-level confusion matrix; determining an event-level confusion matrix based on the notifications, where the event-level confusion matrix includes prediction lead time and notification limit, where each patient stay contributes one event data point to the event-level confusion matrix; calculating classification metrics for false positives and false negatives; and identifying errors in the predication model based on the classification metrics.

Description

Docket No. 2024PF00489- PATENT -SYSTEM AND METHOD FOR EVALUATING PREDICTION MODEL CONFIGURED TO PREDICT OCCURRENCE OF CLINICAL EVENTFIELD OF THE INVENTION

[0001] The invention relates to the field of medical monitoring and diagnosis, and more specifically to evaluating clinical event prediction models for accurately predicting a clinical event.BACKGROUND OF THE INVENTION

[0002] It is common for hospitals and other care facilities to monitor patients throughout their stays using clinical event predication models for predicting occurrences of clinical events of interest, such as the onset of sepsis and / or circulatory shock. Conventional clinical event prediction models, such as the sepsis prediction model, typically run continuously over time across the patient stay. The clinical event prediction models are evaluated on retrospective datasets for accuracy in order to determine whether the clinical event prediction models need to be modified or replaced. However, existing evaluation approaches are potentially inaccurate in that they utilize a single snapshot of the entire patient stay or are biased towards longer patient stays. Also, because the evaluations were conducted on retrospective datasets, patient measurements as well as ground-truth clinical event times were available.

[0003] Generally, a clinical event prediction model aims to predict the onset of the clinical event with a certain lead time. That is, given the patient’s vital signs, laboratory measurements, and other contextual information, the clinical event prediction model computes risk scores of the patient over time. Conventional performance evaluation techniques aim to quantify how well the risk scores output by the clinical event prediction model predict the onset of the clinical event. This is accomplished given (i) risk scores represents the risk score of the n-th patient stay at time stamp t, and (ii) the event onset timewhere Tnrepresents the event onset time of the n-th patient (where Tn= NULL if the n-th patient does not experience any event).Docket No. 2024PF00489- PATENT -

[0004] A first evaluation technique that is commonly used, as described by Mao et aL, “Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU,” BMJ open 8, no. 1 (2018), extracts one risk score from each patient stay at a fixed time gap with respect to the onset time of the clinical event, such as one hour earlier. Therefore, i _Gis used as the risk scores of the n-th patient stay, where G is the gap time between the time of the risk score and the event onset time Tn. The evaluation technique then extracts ground-truth binary class label {L1;W} of each patient stay. The class label is set to one (1) for patient stays experiencing the clinical event and zero (0) otherwise. Using the risk scores ir .Gand the class labels Ln(n = 1, ••• , N), binary classification metrics are calculated, such as the area under the receiver operating characteristic (AUROC) and area under the precision-recall curve (AUPRC) metrics. After selecting a risk threshold, a confusion matrix is determined, as well as sensitivity, specificity, recall and Fl -score. However, this evaluation technique utilizes only a single snapshot during the patient stay, and therefore fails to consider performance of the clinical event prediction model over time.

[0005] A second evaluation technique, as described by Wong et al., “External validation of a widely implemented proprietary sepsis prediction model in hospitalized patients,” JAMA Internal Medicine 181, no. 8 (2021), pp. 1065-1070, treats all risk scores from one patient stay as independent samples, and therefore uses all the risk scores for the evaluation of the clinical event prediction model. As for the class label, second conventional evaluation technique asks whether the clinical event has started within the next T hours. If so, the corresponding class label is set to one (1), and otherwise set to zero (0). Using all the risk scores over time for all the patient stays and the class labels, binary classification metrics are computed. However, although the second evaluation technique makes use of all the risk scores of each patient stay, it treats the risk scores from the same patient stay independently. However, the key drawback is that later high-risk scores may be ignored after the user starts to act upon earlier high-risk scores. Further, patients with longer stays will contribute more samples, thereby biasing the second evaluation technique toward longer patient stays.

[0006] A third evaluation technique, as described by Reyna et al., “Early prediction of sepsis from clinical data: the PhysioNet / Computing in Cardiology Challenge 2019” Critical Care Medicine 48, no. 2 (2020), pp. 210-217, defines a utility function that quantifies the utility of trueDocket No. 2024PF00489- PATENT - positive (TP), true negative (TN), false positive (FP), and false negative (FN) classifications in predicting the onset of the clinical event. A single utility score is computed and normalized to characterize the performance of the clinical event prediction model over time. However, summarizing the evaluation results in a single utility score makes the results difficult to interpret.SUMMARY OF THE INVENTION

[0007] According to a representative embodiment, a method is provided for evaluating performance of a prediction model configured to predict occurrence of a clinical event in multiple patients based on data collected during corresponding multiple patient stays, where the prediction model runs continuously and outputs risk scores for the patients at a predetermined frequency. The method includes identifying patient stays of the multiple patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold, indicating predicted occurrence of the clinical event; for each of the identified patient stays, providing a notification for each of one or more risk scores that cross the predetermined risk threshold; determining a patient-level confusion matrix based on the notifications, where each patient stay of the multiple patient stays contributes one patient data point to the patient-level confusion matrix; determining an event-level confusion matrix based on the notifications, where the event-level confusion matrix includes prediction lead time and notification limit, where each patient stay of the multiple patient stays contributes one event data point to the event-level confusion matrix, where the prediction lead time indicates a length of time before predicted occurrence of the clinical event that a notification of the clinical event is provided, and where the notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications of the predicted occurrence; calculating patient-level classification metrics for false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics for false positives and false negatives of the even -level confusion matrix; identifying errors in the predication model based on the patient-level classification metrics and the eventlevel classification metrics; and displaying at least one of the patient-level confusion matrix, the event-level confusion matrix, patient-level classification metrics, event-level classification metrics, or the identified errors.Docket No. 2024PF00489- PATENT -

[0008] According to another representative embodiment, a system is provided for evaluating performance of a prediction model configured to predict occurrence of a clinical event in multiple patients during a corresponding multiple patient stays, where the prediction model runs continuously and outputs risk scores for the multiple patients at a predetermined frequency. The system includes a processing unit; a results database storing results of the prediction model; a display; and a non-transitory memory storing instructions that, when executed by the processing unit, cause the processing unit to retrieve the results of the prediction model from the results database; based on the retrieved results, identify patient stays of the multiple patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold, indicating predicted occurrence of the clinical event; for each of the identified patient stays, provide a notification for each of one or more risk scores that cross the predetermined risk threshold; determine a patient-level confusion matrix based on the notifications, where each patient stay of the multiple patient stays contributes one patient data point to the patient-level confusion matrix; determine an event-level confusion matrix based on the notifications, where the event-level confusion matrix includes prediction lead time and notification limit, where each patient stay of the multiple patient stays contributes one event data point to the event-level confusion matrix, where the prediction lead time indicates a length of time before predicted occurrence of the clinical event that a notification of the clinical event is provided, and where the notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications of the predicted occurrence; calculate patient-level classification metrics for false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics for false positives and false negatives of the even -level confusion matrix; identify errors in the predication model based on the patient-level classification metrics and the event-level classification metrics; and cause at least one of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, or the identified errors to be displayed on the display.

[0009] According to another representative embodiment, a non-transitory computer readable medium stores instructions for evaluating performance of a prediction model configured to predict occurrence of a clinical event in multiple patients during corresponding multiple patientDocket No. 2024PF00489- PATENT - stays, where the prediction model runs continuously and outputs risk scores for the multiple patients at a predetermined frequency. When executed by a processing unit, the instructions cause the processing unit to retrieve the results of the prediction model from a results database; based on the retrieved results, identify patient stays of the multiple patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold, indicating predicted occurrence of the clinical event; for each of the identified patient stays, provide a notification for each of one or more risk scores that cross the predetermined risk threshold; determine a patient-level confusion matrix based on the notifications, where each patient stay of the multiple patient stays contributes one patient data point to the patient-level confusion matrix; determine an event-level confusion matrix based on the notifications, where the event-level confusion matrix includes prediction lead time and notification limit, where each patient stay of the multiple patient stays contributes one event data point to the event-level confusion matrix, where the prediction lead time indicates a length of time before predicted occurrence of the clinical event that a notification of the clinical event is provided, and where the notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications of the predicted occurrence; calculate patient-level classification metrics for false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics for false positives and false negatives of the even -level confusion matrix, where errors in the predication model are identified based on the patient-level classification metrics and the event-level classification metrics; and cause at least one of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, or the identified errors to be displayed on a display.BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The example embodiments are best understood from the following detailed description when read with the accompanying drawing figures. It is emphasized that the various features are not necessarily drawn to scale. In fact, the dimensions may be arbitrarily increased or decreased for clarity of discussion. Wherever applicable and practical, like reference numerals refer to like elements.Docket No. 2024PF00489- PATENT -

[0011] FIG. l is a simplified block diagram of a system for evaluating performance of a prediction model configured to predict occurrence of a clinical event in patients based on data collected during corresponding stays, according to a representative embodiment.

[0012] FIG. 2 is a flow diagram of a method for evaluating performance of a prediction model configured to predict occurrence of a clinical event in patients based on data collected during corresponding patient stays, according to a representative embodiment.DETAILED DESCRIPTION OF EMBODIMENTS

[0013] Aspects of the disclosure may be supported by various information technology (IT) backends, including either or both local architectures, either as monoliths, networked, or a combination thereof, and hosted architectures, such as a software as a service (SaaS), platform as a service (PaaS), and / or infrastructure as a service (laaS), or the like. In an example, a supporting infrastructure includes multiple interconnected layers respectively hosting, as an abstraction, various IT processes, services, accounts, and other management components.

[0014] Any of the steps described in relation to examples and / or training described below can be performed by a specific-purpose computer system or general-purpose computer system, or a computer-readable medium, or data carrier system configured to carry out any of the steps described previously. The computer system can include a set of software instructions that can be executed to cause the computer system to perform any of the methods or computer-based functions disclosed herein. The computer system may operate as a standalone device or may be connected, for example using a network, to other computer systems or peripheral devices. As an example, a computer system performs logical processing based on digital signals received via an analogue-to-digital converter.

[0015] In the following detailed description, for the purposes of explanation and not limitation, representative embodiments disclosing specific details are set forth in order to provide a thorough understanding of an embodiment according to the present teachings. Descriptions of known systems, devices, materials, methods of operation and methods of manufacture may be omitted so as to avoid obscuring the description of the representative embodiments. Nonetheless, systems, devices, materials and methods that are within the purview of one of ordinary skill in the art are within the scope of the present teachings and may be used in accordance with theDocket No. 2024PF00489- PATENT - representative embodiments. It is to be understood that the terminology used herein is for purposes of describing particular embodiments only and is not intended to be limiting. The defined terms are in addition to the technical and scientific meanings of the defined terms as commonly understood and accepted in the technical field of the present teachings.

[0016] It will be understood that, although the terms first, second, third, etc. may be used herein to describe various elements or components, these elements or components should not be limited by these terms. These terms are only used to distinguish one element or component from another element or component. Thus, a first element or component discussed below could be termed a second element or component without departing from the teachings of the inventive concept.

[0017] The terminology used herein is for purposes of describing particular embodiments only and is not intended to be limiting. As used in the specification and appended claims, the singular forms of terms “a,” “an” and “the” are intended to include both singular and plural forms, unless the context clearly dictates otherwise. Additionally, the terms “comprises,” “comprising,” and / or similar terms specify the presence of stated features, elements, and / or components, but do not preclude the presence or addition of one or more other features, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0018] Unless otherwise noted, when an element or component is said to be “connected to,” “coupled to,” or “adjacent to” another element or component, it will be understood that the element or component can be directly connected or coupled to the other element or component, or intervening elements or components may be present. That is, these and similar terms encompass cases where one or more intermediate elements or components may be employed to connect two elements or components. However, when an element or component is said to be “directly connected” to another element or component, this encompasses only cases where the two elements or components are connected to each other without any intermediate or intervening elements or components.

[0019] The present disclosure, through one or more of its various aspects, embodiments and / or specific features or sub-components, is thus intended to bring out one or more of the advantages as specifically noted below. For purposes of explanation and not limitation, example embodiments disclosing specific details are set forth in order to provide a thoroughDocket No. 2024PF00489- PATENT - understanding of an embodiment according to the present teachings. However, other embodiments consistent with the present disclosure that depart from specific details disclosed herein remain within the scope of the appended claims. Moreover, descriptions of well-known apparatuses and methods may be omitted so as to not obscure the description of the example embodiments. Such methods and apparatuses are within the scope of the present disclosure.

[0020] The various embodiments described herein provide an evaluation metric for evaluating a continuously running prediction model configured to predict occurrence of a particular clinical event (e.g., sepsis) in patients based on patient information, including vital sign data collected during corresponding patient stays. Generally, the evaluation metric quantifies performance of the prediction model, such that the evaluation results are close to how the prediction model will be used in clinical practice. The embodiments are based on various considerations. First, repeated positive predictions by the prediction model are not clinically relevant because repeat high-risk predictions are commonly “muted” after initial high-risk scores are determined, as described by Reyna et al.. “Rethinking algorithm performance metrics for artificial intelligence in diagnostic medicine,” JAMA 328, no. 4 (2022), pp. 329-330, for example. Therefore, instead of treating each risk score as an individual sample, repeated consecutive risk scores that exceed a predetermined risk threshold will generate only a single notification, which is closer to how the prediction model will be used in practice. Second, to correct for bias towards longer patient stays, each patient stay is weighted equally in the evaluation. Third, temporal aspects of prediction accuracy of the prediction model is characterized by taking both the user’s attention (notification limit) and the prediction lead time into account. For example, when the care facility is an intensive care unit (ICU), the user may be an ICU nurse or physician. The user’s attention is an important resource in this situation, particularly with insufficient staffing. Given the risk of attention fatigue, the user’s attention resource should be taken into account, which is characterized by the notification limit.

[0021] FIG. l is a simplified block diagram of a system for evaluating a prediction model configured to predict occurrence of a clinical event in patients during corresponding stays in a care facility, according to a representative embodiment. The prediction model runs continuously and outputs risk scores for the patients at a predetermined frequency (e.g., every hour) for each patient.Docket No. 2024PF00489- PATENT -

[0022] Referring to FIG. 1, system 100 includes a workstation 105 for implementing and / or managing the processes described herein with regard to evaluating the prediction model. The workstation 105 includes one or more processors indicated by processing unit 120, one or more memories indicated by memory 130, a user interface 122 and a display 124. The system 100 further includes a monitor system 140 that includes one or more medical devices for monitoring vital signs of the patients in the care facility, indicated by representative patient 150, during their respective patient stays, such as a telemetry monitor for monitoring heart rate, a pulse oximeter for monitoring oxygenation, a blood pressure monitor for monitoring blood pressure, a capnograph for monitoring respiration rate, etc., as would be apparent to one skilled in the art. The processing unit 120 communicates with the monitor system 140 through an interface (not shown) to collect real time data from the patient 150 throughout the patient stay. The processing unit 120 also communicates with a patient information database, such as hospital information system (HIS) database 112, to retrieved background information regarding the patient 150, such as age and medical history, which may be relevant to occurrence of the clinical event.

[0023] The memory 130 stores instructions executable by the processing unit 120. When executed, the instructions cause the processing unit 120 to implement one or more processes for evaluating the prediction model configured to predict occurrence of the clinical event. For purposes of illustration, the memory 130 is shown to include software modules, each of which includes the instructions, executable by the processing unit 120, corresponding to an associated capability of the system 100, as discussed below.

[0024] The processing unit 120 is representative of one or more processing devices, and may be implemented by a general purpose computer, a central processing unit (CPU), a digital signal processor (DSP), a graphical processing unit, a computer processor, a microprocessor, a state machine, programmable logic device, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or combinations thereof, using any combination of hardware, software, firmware, hard-wired logic circuits, or combinations thereof. Any processor or processing unit herein may include multiple processors, parallel processors, or both. Multiple processors may be included in, or coupled to, a single device or multiple devices. The term “processor” as used herein encompasses an electronic component able to execute a program or machine executable instruction. A processor may also refer to a collection of processors within aDocket No. 2024PF00489- PATENT - single computer system or distributed among multiple computer systems, such as in a cloudbased or other multi-site application. Programs have software instructions performed by one or multiple processors that may be within the same computing device or which may be distributed across multiple computing devices.

[0025] The memory 130 may include main memory and / or static memory, where such memories may communicate with each other and the processing unit 120 via one or more buses. The memory 130 may be implemented by any number, type and combination of random access memory (RAM) and read-only memory (ROM), for example, and may store various types of information, such as software algorithms, Al machine learning models including the prediction model, and computer programs, all of which are executable by the processing unit 120. The various types of ROM and RAM may include any number, type and combination of computer readable storage media, such as a disk drive, flash memory, an electrically programmable readonly memory (EPROM), an electrically erasable and programmable read only memory (EEPROM), registers, a hard disk, a removable disk, tape, compact disk read only memory (CD- ROM), digital versatile disk (DVD), floppy disk, Blu-ray disk, a universal serial bus (USB) drive, or any other form of storage medium. The memory 130 is a tangible storage medium for storing data and executable software instructions, and is non-transitory during the time software instructions are stored therein. As used herein, the term “non-transitory” is to be interpreted not as an eternal characteristic of a state, but as a characteristic of a state that will last for a period. The term “non-transitory” specifically disavows fleeting characteristics such as characteristics of a carrier wave or signal or other forms that exist only transitorily in any place at any time. The memory 130 may store software instructions and / or computer readable code that enable performance of various functions. The memory 130 may be secure and / or encrypted, or unsecure and / or unencrypted.

[0026] The system 100 may also include databases for storing information that may be used by the various software modules of the memory 130. Representative databases may include the HIS database 112 for storing information about the patients’ medical histories, a training database 113 for storing a training dataset for training the prediction model, and a results database 114 for storing results of the prediction model monitoring multiple patient stays in the care facility, as discussed below. Each of the databases may be implemented by any number, type andDocket No. 2024PF00489- PATENT - combination of RAM and ROM, for example. The various types of ROM and RAM may include any number, type and combination of computer readable storage media, such as a disk drive, flash memory, EPROM, EEPROM, registers, a hard disk, a removable disk, tape, CD-ROM, DVD, floppy disk, Blu-ray disk, USB drive, or any other form of storage medium known in the art. The databases comprise tangible storage mediums for storing data and executable software instructions and is non-transitory during the time data and software instructions are stored therein. The databases may be secure and / or encrypted, or unsecure and / or unencrypted. Th databases may be implemented as separate storage media (as shown), or may be combined with one another and / or included in the memory 130, without departing from the scope of the present teachings.

[0027] The processing unit 120 may include or have access to an Al engine, which may be implemented as software that provides artificial intelligence (e.g., deep learning, neutral network models) and applies machine learning, including the prediction model. The Al engine may reside in any of various components in addition to or other than the processing unit 120, such as the memory 130, an external server, and / or the cloud, for example. When the Al engine is implemented in a cloud, such as at a data center, for example, the Al engine may be connected to the processing unit 120 via the internet or other communication network using one or more wired and / or wireless connection(s). In various embodiments, all or part of the processes provided by first and second machine learning models, discussed below, may be implemented by the Al engine, for example. The training and execution of the first, second and image encoding machine learning models cannot practically be performed in the human mind.

[0028] The user interface 122 is configured to provide information and data output by the processing unit 120, the memory 130 and / or the monitoring system 140 to the user and / or for receiving information and data input by the user. That is, the user interface 122 enables the user to enter data and to control or manipulate aspects of the processes described herein, and also enables the processing unit 120 to indicate the effects of the user’s input. All or a portion of the user interface 122 may be implemented by a graphical user interface (GUI), such as GUI 128 viewable on the display 124. The user interface 122 may include one or more interface devices, such as a mouse, a keyboard, a trackball, a joystick, a microphone, a video camera, a touchpad, a touchscreen, voice or gesture recognition captured by a microphone or video camera, forDocket No. 2024PF00489- PATENT - example. The display 124 may be a monitor such as a computer monitor, a television, a liquid crystal display (LCD), an organic light emitting diode (OLED), a flat panel display, a solid-state display, or a cathode ray tube (CRT) display, or an electronic whiteboard, for example.

[0029] Referring again to the memory 130, the various modules store sets of data and instructions executable by the processing unit 120 to evaluate the prediction model, which is configured to predict occurrence of the clinical event in the patients during their respective patient stays. In the depicted embodiment, the memory 130 includes patient information module 131, which is configured to collect information regarding patients during their corresponding patient stays at the care facility, which may be an intensive care unit at a hospital, for example. Each patient stay begins with admission to the care facility and ends with discharge from the care facility, and typically may between two and seven days, for example. The patients in the care facility are assumed to be at higher risk for experiencing the clinical event, as discussed above. For example, in an intensive care unit, the patients are at higher risk of developing sepsis, particularly post-surgery. The patient information module 131 therefore collects information relevant to the particular clinical event, and may include the vital sign data provided by the monitoring system 140 during their respective patient stays and medical history data retrieved from patient data bases, such as the HIS database 112.

[0030] Prediction module 132 is configured to execute an Al prediction model, which predicts the likelihood of onset of the clinical event in the patients during their respective patient stays at the care facility. The prediction model may be an existing or yet to be determined machine learning prediction model or algorithm capable of predicting the likelihood of occurrence of the clinical event in patients within the purview of one of ordinary skill in the art having the benefit of the present disclosure. The clinical event is a condition considered to be harmful or dangerous to the patients that may become symptomatic (experience onset) in the patients during their patient stays. Examples of such clinical events and predictions models predicting the onset of the same include sepsis, circulatory failure, acute respiratory distress syndrome, and acute kidney injury, although prediction models directed to occurrence of other clinical events, may be incorporated. The prediction model is assumed to run continuously throughout each patient stay, and to output risk scores for the patient at a predetermined frequency (e.g., every hour), where the risk scores indicate the likelihood of the patient developing or actively experiencing theDocket No. 2024PF00489- PATENT - clinical event within a certain time frame.

[0031] Generally, the prediction model is a previously trained machine learning model that receives patient information regarding each of the patients via the patient information module 131 as input, and provides risk scores at the predetermined frequency based on the input patient information as output. The prediction model may be implemented as any suitable type of trainable machine learning model (e.g., deep learning model), such as a convolutional neural network (CNN), an artificial neural network (ANN), a recurrent neural network (RNN), a vision transformer, or a U-net model, for example. The prediction model may be previously trained in a supervised fashion using a training data set including labelled patient information and associated onset (if any) of the clinical event, including the time frame in which the patient experienced the onset of the clinical event relative to the patient information (ground truth data). The labels may be added by experts in the field of the clinical event, or may be labeled automatically through techniques such as ground truth automation, as would be apparent to one skilled in the art. The training data set for training the prediction model may be stored in the training database 113, for example. At inference, the prediction model is able to process new input patient information regarding the patients during corresponding patient stays in the care facility, and to output risk scores that predict likelihoods of onset of the clinical event in the patients based on the same, as discussed above. The results of the prediction model may be stored in the results database 114, for example, along with corresponding actual outcomes associated with patient stays, respectively.

[0032] Notification module 133 is configured to receive the risk scores of the patients from the prediction model and / or the results database 114, and to output notifications indicating when various patients are at high risk of onset of the clinical event based on the risk scores. In an embodiment, the notification module 133 compares the risk scores to a predetermined risk threshold that corresponds to a level of risk that indicates a predetermined level of likelihood that the clinical event will occur, e.g., which may be set by the user, for example. In other words, the risk scores that cross the risk threshold indicate positive predictions of onset of the clinical event. In an embodiment, conversion from the risk score provided by the prediction model to a notification may include two steps. First, the risk threshold is applied to the risk score to convert the risk score to a binary flag, which is set to 1 when the score exceeds the risk threshold and 0Docket No. 2024PF00489- PATENT - otherwise. Second, notification logic is applied to the binary flags over time to output the notification. Different notification logic may be used. For example, the notification logic generate a single notification for one set of consecutive 1 -valued binary flags, as discussed below. The notification logic can be treated as post-processing of the prediction model output, and therefore is not part of the prediction model itself.

[0033] The risk threshold may be set at the user’s discretion, or may be calculated by the prediction model, for example, as would be apparent to one skilled in the art. Prediction models use different risk thresholds for different reasons, and can also be adjusted if needed. For example, if the user is interested in capturing more true positives at the expense of tolerating more false positives, it makes sense to use a lower risk threshold. On the other hand, if the user prefers lower false positives at the expense of capturing a smaller number of true positives (because those missed events can be identified later by other guardrails, such as intensive continuous patient monitoring), the risk threshold can be set higher.

[0034] The notifications may include discrete notifications and consolidated notifications. A discrete notification indicates that one risk score has crossed the risk threshold, but the consecutively generated risk scores (immediately before and after the one risk score) did not cross the predetermined risk threshold. A consolidated notification indicates that multiple consecutively generated risk scores have crossed the risk threshold. Accordingly, one consolidated notification represents a set of consecutive risk scores that have crossed the risk threshold.

[0035] With regard to the consolidated notifications, the risk scores output by the prediction algorithm may repeatedly cross the risk threshold, resulting in multiple consecutive positive predictions (multiple consecutive risks scores crossing the risk threshold). However, these multiple consecutive positive predictions may not be clinically relevant because they are commonly “muted” after an initial crossing of the risk threshold, meaning that they are not provided or simply ignored, as discussed above. Therefore, when evaluating performance of the prediction model, instead of treating multiple consecutive positive predictions as independent samples, they are consolidated into a single, consolidated notification.

[0036] For example, assuming that rtrepresents the risk score of the n-th patient at time stamp t, risk scoresgenerate a single notificationstarting from time t andDocket No. 2024PF00489- PATENT - continuing until timeis the risk threshold and K is a total number of timestamps of the patient stay where a block of K consecutive risk scores all exceed the risk threshold. In this way, a variable number of notifications may be generated for each patient stay.

[0037] Patient-level confusion matrix module 134 is configured to determine a patient-level confusion matrix based on the notifications of risk scores exceeding the risk threshold provided by the notification module 133, if any. The notifications include both the discrete notifications and the consolidated notifications. Each patient stay of the multiple of patient stays being evaluated contributes one patient data point to the patient-level confusion matrix, where the patient data point indicates whether the clinical event has been predicted to occur and whether the clinical event actually does occur during the patient stay. That is, to avoid bias towards longer patient stays in the care facility, each patient stay contributes only one patient data point to the patient-level confusion matrix, regardless of the length of the patient stay. Therefore, multiple notifications belonging to the same patient stay are aggregated to determine the entry contributed to the patient-level confusion matrix.

[0038] Therefore, for each patient that experiences no actual occurrence of the clinical event during the patient stay: (i) one false positive (FP) is contributed to the patient-level confusion matrix when one or more notifications received indicating that the predication model has predicted occurrence of the clinical event at least once, and (ii) one true negative (TN) is contributed to the patient-level confusion matrix when no notifications are received. Further, for each patient that experiences actual occurrence of the clinical event during the patient stay: (iii) one true positive (TP) is contributed to the patient-level confusion matrix when one or more notifications are received indicating that the predication model has at least once predicted occurrence of the clinical event prior to the actual occurrence of the clinical event, and (iv) one false negative (FN) is contributed to the patient-level confusion matrix when no notifications of predicted occurrence of the clinical event are received prior to the actual occurrence of the clinical event.

[0039] For purposes of testing the embodiments, a known light gradient boosting machine (LGBM) regressor was used as the prediction algorithm to predict onset of sepsis as the clinical event in patients. A patient-level confusion matrix, shown in Table 1, was generated usingDocket No. 2024PF00489- PATENT - notifications based on risk scores output by the LGBM regressor, according to a representative embodiment.Table 1

[0040] Table 1 may be displayed on the display 124 and / or stored in the memory 140, enabling evaluation of the LGBM regressor, as discussed below.

[0041] Referring again to memory 130 in FIG. 1, event-level confusion matrix module 135 is configured to determine an event-level confusion matrix based on the notifications of risk scores from each patient exceeding the risk threshold provided by the notification module 133, if any. Again, the notifications include both the discrete notifications and the consolidated notifications. Each patient stay of the multiple of patient stays being evaluated contributes one event data point to the event-level confusion matrix based on the notifications. In particular, the event-level confusion matrix is determined based on prediction lead time and notification limit that are set by the user, where each patient stay in the care facility contributes one event data point to the event-level confusion matrix. The prediction lead time indicates a length of time before the predicted occurrence of the clinical event that a notification of the clinical event is provided based on the risk scores output by the prediction model. The notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that the user will act upon before ignoring additional notifications. Multiple event-level confusion matrices may be determined using different lead times and notification limits, as discussed below.

[0042] With regard to the prediction lead time, in the context of the event-level confusion matrix, a notification atindicates that, starting from time / , the predicted occurrence of the clinical event will start within a time window [t, t+T], where Tis the prediction lead time. Accordingly, determining the event-level confusion matrix includes contributing one true positive (TP) to the event-level confusion matrix when actual occurrence of the clinic eventDocket No. 2024PF00489- PATENT - occurs within the time window [t, t+T], and contributing one false positive (FP) to the eventlevel confusion matrix when no actual occurrence of the clinic event occurs within the time window following the notification.

[0043] With regard to the notification limit, for each patient stay in the care facility, it is assumed that the user will attend to the first few notifications based on risk scores output by the prediction model and take further actions to confirm whether the patient has actually developed the clinical event. However, users become more likely to ignore subsequent notifications as the initial notifications turn out to be event-level false positives. As mentioned above, the maximum number of notifications that the user attends to and acts upon before ignoring subsequent notifications is the notification limit. In practice, the notification limit may be set as a fixed value for each staff shift in the care facility, such as each 12-hour shift in the ICU, for example.

[0044] To avoid bias towards longer patient stays in the care facility, each patient stay contributes only one event data point to the event-level confusion matrix, as mentioned above, regardless of the length of the patient stay. Therefore, multiple notifications belonging to the same patient stay are aggregated to determine whether they contribute one false positive or one true positive to the event-level confusion matrix. Accordingly, for notifications, where M is the number of notifications occurring before the predicted onset of the clinical event and L is the notification limit: (i) one true positive (TP) is contributed to the event-level confusion matrix when tCE—< T holds for any notification of the predicted occurrence of the clinical event within the notification limit (Z), and (ii) one false positive (FP) is contributed to the event-level confusion matrix when tCE—> T holds for all notifications of the predicted occurrence of the clinical event within the notification limit, or when there is no actual occurrence of the clinical event. In the above, tCEis the time of predicted occurrence of the clinical event and tmin (Lis the time of occurrence of the lesser of the total number of notifications (M) or the notification limit (Z). That is, with regard to tminwhen the notification limit (Z) is less than the total number of notifications (M) actually generated during the patient stay, the time Z, is used to determine whether a false positive or a true positive is contributed to the event-level confusion matrix, and when the total number of notifications (M) actually generated is less than the notification limit (Z), the time tM is used.

[0045] Of course, patient stays may result in no notifications (M= 0) of the predicted occurrenceDocket No. 2024PF00489- PATENT - of the clinical event within the notification limit for a patient stay. In this case, (iii) one false negative (FN) is contributed to the event-level confusion matrix when no notifications of the predicted occurrence of the clinical event are provided within the notification limit before the onset time of the actual occurrence of the clinical event, and (iv) one true negative (TN) is contributed to the event-level confusion matrix when no notifications of the predicted occurrence of the clinical event occur within the notification limit, and no actual occurrence of the clinic event occurs within the time window.

[0046] Again, for purposes of testing the embodiments, the known LGBM regressor was used as the prediction algorithm to predict onset of sepsis in patients, as discussed above with reference to Table 1. Event-level confusion matrices were generated using notifications based on risk scores output by the LGBM regressor, according to a representative embodiment. In Table 2, the prediction lead time (7) was varied among 3 hours, 6 hours, 12, hours, 24 hours, and 48 hours, while the notification ( / .) remains fixed at 100 notifications.Table 2

[0047] As shown, as the prediction lead time (7) increases from 3 hours to 48 hours, the number of event-level true positives increases from 120 to 428. These results are expected because longer prediction lead times enable earlier notifications to count as event-level true positives instead of event-level false positives. For example, prediction of the onset of sepsis when the prediction lead time is only 3 hours requires the patient to develop sepsis within the next 3 hours at most following the prediction, or else the prediction will be classified as a false positive. In comparison, early prediction of the onset of sepsis when the prediction lead time is 48 hoursDocket No. 2024PF00489- PATENT - enables the patient to develop sepsis within the day or two following the prediction resulting in the prediction being classified as a true positive.

[0048] In Table 3, the notification limit ( / .) is varied among 1 notification, 2 notifications, 5 notifications, 10 notifications, and 100 notifications, while the prediction lead time (7) remains fixed at 12 hours.Table 3

[0049] As shown, as the notification limit (L) increases from 1 notification to 100 notifications, the number of event-level true positives increases from 184 to 252. These results are also expected because higher notification limits allow more notifications to qualify as event-level true positives. For example, a user with a notification limit of 1 will not consider additional notifications of predicted onset of sepsis after just one false positive, and therefore will miss subsequent notifications that ultimately would result in true positives. Meanwhile, a user with a notification limit of 10 may not miss these subsequent true positive notifications. Although not shown, it is understood that for each combination of prediction lead time and notification limit, one event-level confusion matrix can be generated. Notably, with regard to the event-level confusion matrices, the false negative and true negative entries may be excluded as a practical matter since they do not change with respect to the prediction lead time and the notification limit.

[0050] Tables 2 and 3 also may be displayed on the display 124 and / or stored in the memory 140, enabling evaluation of the LGBM regressor, as discussed below.

[0051] Referring again to the memory 130 in FIG. 1, evaluation module 136 is configured to evaluate the patient-level confusion matrix and the event-level confusion matrix(es) in order toDocket No. 2024PF00489- PATENT - enable evaluation of the prediction model, for example, by identify errors in the prediction model as indicated by the patient-level confusion matrix and the event-level confusion matrix(es). In an embodiment, the evaluation module 136 enable error evaluation by calculating patient-level classification metrics based on the patient-level confusion matrix and calculating event-level classification metrics based on the event-level confusion matrix(es). The patient-level and eventlevel classification metrics include one or more of sensitivity, specificity, recall and precision (e.g., positive predictive value (PPV)), for example.

[0052] Given the patient-level confusion matrix, the patient-level classification metrics are calculated based on analysis of the patient-level false positives and false negatives. For example, by summarizing the top-ranking input feature contributing most to the highest risk score for each patient-level false positive during the patient stay, it may be concluded the prediction model will generate a notification when the patient’s body temperature increases above certain temperature threshold. Further mitigation measures may then be taken to reduce these false positives. The causes of a patient-level false negative may also be diagnosed by inspecting the input missingness or inadequate predictive power of existing input features of the prediction model.

[0053] Compared to the patient-level confusion matrix, the event-level confusion matrix can better capture whether the notifications accurately predict the onset of the clinical event within the prediction lead time (i.e., with the next Zhours). Notably, the true negative count in the patient-level confusion matrix is equal to the true negative count in the event-level confusion matrix, and the false negative count in the patient-level confusion matrix is equal to the false negative count of the event-level confusion matrix. However, the false positive count in the event-level matrix is greater than the false positive in the patient-level matrix, which leads to the true positive count in the event-level matrix being less than true positive in the patient-level matrix. Due to the requirement of predicting onset of the clinical event within the prediction lead time, the notifications occurring much earlier than the onset of the clinical event will be counted as false positives in the event-level matrix, but will be counted as true positives in the patientlevel matrix since the patient-level evaluation does not have the stricter time window requirements on the true positives. Therefore, the classification metrics of the patient-level matrix, including sensitivity, specificity, recall and precision, are the upper bounds of the corresponding classification metrics of the event-level matrix.Docket No. 2024PF00489- PATENT -

[0054] The errors in the prediction model may be identified based on the calculated patient-level classification metrics and event-level classification metrics. The errors may be identified by the user or by existing or yet to be determined error evaluation models capable of associating the calculated patient-level classification metrics and event-level classification metrics with specific errors within the purview of one of ordinary skill in the art having the benefit of the present disclosure. In the patient-level confusion matrix (e.g., Table 1), both false positives (FP) and false negatives (FN) are counted as errors. The same is true for the event-level confusion matrix (e.g., Tables 2 and 3). The acceptable number of FP and FN errors for each prediction model may be set by the user and / or industry standards, e.g., depending on user experience and tolerance, and criticality of clinical events being monitored. One or more of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, and the errors identified in the prediction model may be displayed on the display 124, for example, and / or stored in the memory 130.

[0055] When no errors or an acceptable number of errors are identified, then it is determined that performance of the prediction model is acceptable, and can continue to be used. When an unacceptable number of errors is identified, the prediction model may be modified and / or retrained using updated patient data, or the prediction model may be stopped and / or replaced. Performance of the retrained or new prediction model may then be evaluated according to the embodiments herein.

[0056] When an unacceptable number of errors is identified, the source of the errors may be identified through error analysis. For example, the source of FNs may be an insufficient number of measured input features, and the source of FPs can be context dependent. For example, a sepsis prediction model may learn that the length of time patients stay in the ICU is most predictive of the risk of developing sepsis. As a result, the prediction model may provide positive predictions for patients who stay in the ICU long enough, regardless of their physiological states, due to the context of the patient stays, resulting in FPs. The ill-behavior of the prediction model may therefore be identified through error analysis, and the prediction model may be retrained to weigh additional factors to achieve lower FPs.

[0057] As an example, to identify the source of patient-level FNs, the highest risk score may be considered for each patient, resulting in top-ranking features. The top-ranking features may beDocket No. 2024PF00489- PATENT - summarized in terms of respective feature values and contributions to the prediction risk scores, therefore providing explanations on the causes of the patient-level FNs. To identify the source of patient-level FPs, the first positive prediction of each patient may be considered. Of these FP predictions, input features may be ranked by respective risk contributions. The top-ranking input features may be summarized in terms of feature values and contributions to the prediction risk scores, therefore providing explanations on the causes of the patient-level FPs.

[0058] FIG. 2 is a flow diagram of a method of evaluating performance of a prediction model configured to predict occurrence of a clinical event in multiple patients based on data collected during corresponding multiple patient stays, according to a representative embodiment. The method may be applied on retrospective data for post-deployment monitoring of the prediction model. The method may be implemented at least in part using instructions stored in memory 130 and executable by the processing unit 120 in the system 100, for example.

[0059] Referring to FIG. 2, in block S211, risk scores generated by the prediction model for the multiple patient stays are received. As mentioned above, the prediction model runs continuously and generates risk scores for the patients at a predetermined frequency. The risk scores predict likelihood of the patients developing a predetermined clinical event, such as sepsis, for example, during the corresponding patient stays. The prediction model determines the risk scores based on various factors relating to the patients and the patient stays that are input to the prediction model. Such factors may include monitored vital signs, medical history, laboratory results (e.g., white blood cell counts, creatinine level, glucose level), and age of the patient, for example.

[0060] In block S212, patient stays are identified that include at least one high risk score generated by the prediction model. A high risk score may be a risk score that crosses a predetermined risk threshold. The risk score of each patient stay indicates predicted likelihood of occurrence of the clinical event during that patient stay.

[0061] In block S213, for each of the identified patient stays, a notification is provided for each high risk score that is generated by the prediction model during the patient stay. In an embodiment, the notifications may include discrete notifications and consolidated notifications, as discussed above. A discrete notification indicates a single risk score that has crossed the risk threshold, and a consolidated notification indicates multiple consecutively generated risk scores that have crossed the predetermined risk threshold, such that one consolidated notificationDocket No. 2024PF00489- PATENT - represents a set of consecutive risk scores that have crossed the risk threshold.

[0062] In block S214, a patient-level confusion matrix is determined based on the notifications of high risk scores, e.g., exceeding the risk threshold, provided in block S213. Each patient stay of the multiple of patient stays being evaluated contributes one patient data point to the patientlevel confusion matrix. The notifications of high risk scores are combined so that one data point from each patient stay is contributed to the patient-level confusion matrix.

[0063] In block S215, an event-level confusion matrix is also determined based on the notifications of high risk scores, e.g., exceeding the risk threshold, provided in block S213. Again, each patient stay of the multiple patient stays being evaluated contributes one event data point to the event-level confusion matrix. The event-level confusion matrix includes consideration of prediction lead time and notification limit. As discussed above, the prediction lead time indicates a length of time prior to the predicted occurrence of the clinical event that a notification of the clinical event is provided, and the notification limit indicates a maximum number of notifications by the prediction model of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications by the prediction model.

[0064] In block S216, classification metrics are calculated in order to evaluate the performance of the prediction model. In particular, patient-level classification metrics are calculated based on false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics are calculated based on false positives and false negatives of the eventlevel confusion matrix.

[0065] In block S217, errors are identified in the predication model based on the patient-level classification metrics and the event-level classification metrics, as discussed above. If no errors, or an acceptable number of errors, are identified, then the performance of the prediction model is deemed to be acceptable, and no further action is taken. When an unacceptable number of errors are identified, the prediction model may be retrained using updated patient data, or if this is not possible, the prediction model should be stopped and / or replaced.

[0066] In block S218, one or more of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, and any identified errors are displayed on the display 124.

[0067] In accordance with various embodiments of the present disclosure, the methods describedDocket No. 2024PF00489- PATENT - herein may be implemented using a hardware computer system that executes software programs stored on non-transitory storage mediums. Further, in an exemplary, non-limited embodiment, implementations can include distributed processing, component / object distributed processing, and parallel processing. Virtual computer system processing may implement one or more of the methods or functionalities as described herein, and a processor described herein may be used to support a virtual processing environment.

[0068] Although determining user-preferred positions of calipers in ultrasound images has been described with reference to exemplary embodiments, it is understood that the words that have been used are words of description and illustration, rather than words of limitation. Changes may be made within the purview of the appended claims, as presently stated and as amended, without departing from the scope and spirit of the embodiments. Also, although determining userpreferred positions of calipers in ultrasound images has been described with reference to particular means, materials and embodiments, it is not intended to be limited to the particulars disclosed; rather evaluating quality of an ultrasound imaging system extends to all functionally equivalent structures, methods, and uses such as are within the scope of the appended claims.

[0069] The illustrations of the embodiments described herein are intended to provide a general understanding of the structure of the various embodiments. The illustrations are not intended to serve as a complete description of all of the elements and features of the disclosure described herein. Many other embodiments may be apparent to those of skill in the art upon reviewing the disclosure. Other embodiments may be utilized and derived from the disclosure, such that structural and logical substitutions and changes may be made without departing from the scope of the disclosure. Additionally, the illustrations are merely representational and may not be drawn to scale. Certain proportions within the illustrations may be exaggerated, while other proportions may be minimized. Accordingly, the disclosure and the figures are to be regarded as illustrative rather than restrictive.

[0070] One or more embodiments of the disclosure may be referred to herein, individually and / or collectively, by the term “invention” merely for convenience and without intending to voluntarily limit the scope of this application to any particular invention or inventive concept. Moreover, although specific embodiments have been illustrated and described herein, it should be appreciated that any subsequent arrangement designed to achieve the same or similar purposeDocket No. 2024PF00489- PATENT - may be substituted for the specific embodiments shown. This disclosure is intended to cover any and all subsequent adaptations or variations of various embodiments. Combinations of the above embodiments, and other embodiments not specifically described herein, will be apparent to those of skill in the art upon reviewing the description.

[0071] The Abstract of the Disclosure is provided to comply with 37 C.F.R. § 1.72(b) and is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the foregoing Detailed Description, various features may be grouped together or described in a single embodiment for the purpose of streamlining the disclosure. This disclosure is not to be interpreted as reflecting an intention that the claimed embodiments require more features than are expressly recited in each claim. Rather, as the following claims reflect, inventive subject matter may be directed to less than all of the features of any of the disclosed embodiments. Thus, the following claims are incorporated into the Detailed Description, with each claim standing on its own as defining separately claimed subject matter.

[0072] The preceding description of the disclosed embodiments is provided to enable any person skilled in the art to practice the concepts described in the present disclosure. As such, the above disclosed subject matter is to be considered illustrative, and not restrictive, and the appended claims are intended to cover all such modifications, enhancements, and other embodiments which fall within the true spirit and scope of the present disclosure. Thus, to the maximum extent allowed by law, the scope of the present disclosure is to be determined by the broadest permissible interpretation of the following claims and their equivalents and shall not be restricted or limited by the foregoing detailed description.

Claims

Docket No. 2024PF00489- PATENT -CLAIMS:

1. A method of evaluating performance of a prediction model configured to predict occurrence of a clinical event in a plurality of patients during a corresponding plurality of patient stays, wherein the prediction model runs continuously and outputs risk scores for the plurality of patients at a predetermined frequency, the method comprising: identifying patient stays of the plurality of patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold, indicating predicted occurrence of the clinical event; for each of the identified patient stays, providing a notification for each of one or more risk scores that cross the predetermined risk threshold; determining a patient-level confusion matrix based on the notifications, wherein each patient stay of the plurality of patient stays contributes one patient data point to the patient-level confusion matrix; determining an event-level confusion matrix based on the notifications, wherein the event-level confusion matrix includes prediction lead time and notification limit, wherein each patient stay of the plurality of patient stays contributes one event data point to the event-level confusion matrix, wherein the prediction lead time indicates a length of time before predicted occurrence of the clinical event that a notification of the clinical event is provided, and wherein the notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications of the predicted occurrence; calculating patient-level classification metrics for false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics for false positives and false negatives of the even -level confusion matrix; identifying errors in the predication model based on the patient-level classification metrics and the event-level classification metrics; and displaying at least one of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, or the identified errors.Docket No. 2024PF00489- PATENT -2. The method of claim 1, wherein the notification for the one or more risk scores that cross the predetermined risk threshold comprises a discrete notification corresponding to one risk score that cross the predetermined risk threshold or a consolidated notification corresponding to a set of consecutive risk scores that cross the predetermined risk threshold.

3. The method of claim 1, wherein determining the patient-level confusion matrix comprises: for each patient experiencing no actual occurrence of the clinical event during the patient stay, contributing one false positive to the patient-level confusion matrix for one or more notifications of predicted occurrence of the clinical event by the predication model; for each patient experiencing no actual occurrence of the clinical event during the patient stay, contributing one true negative to the patient-level confusion matrix for no notifications of predicted occurrence of the clinical event by the predication model; for each patient experiencing actual occurrence of the clinical event during the patient stay, contributing one true positive to the patient-level confusion matrix for one or more notifications of predicted occurrence of the clinical event by the predication model prior to the actual occurrence of the clinical event; and for each patient experiencing actual occurrence of the clinical event during the patient stay, contributing one false negative to the patient-level confusion matrix for no notifications of predicted occurrence of the clinical event by the predication model prior to the actual occurrence of the clinical event.

4. The method of claim 1, wherein the patient-level classification metrics and the eventlevel classification metrics comprise at least one of sensitivity, specificity, recall or precision for each patient stay.

5. The method of claim 1, wherein a notificationindicates that, starting from time / , the predicted occurrence of the clinical event will start within a time window [t, t+T], wherein T is the prediction lead time, and wherein determining the event-level confusion matrix comprises:Docket No. 2024PF00489- PATENT - contributing one true positive to the event-level confusion matrix when actual occurrence of the clinic event occurs within the time window; and contributing one false positive to the event-level confusion matrix when no actual occurrence of the clinic event occurs within the time window.

6. The method of claim 5, wherein for M notificationsoccurring before the predicted onset of the clinical event for the notification limit (Z), indicating the maximum number of notifications by the prediction model of the predicted occurrence of the clinical event that the user will act upon before ignoring additional notifications by the prediction model: the M notifications contribute the one true positive to the event-level confusion matrix when< T holds for any notification of the predicted occurrence of the clinical event within the notification limit, wherein tCEis time of predicted occurrence of the clinical event, L is the notification limit, and tLis onset time of the actual occurrence of the clinical event of an L-th notification, and the M notifications contribute one false positive to the event-level confusion matrix when tCE~fmin(L,M) T holds for all notifications of the predicted occurrence of the clinical event within the notification limit, or when there is no actual occurrence of the clinical event.

7. The method of claim 6, wherein determining the event-level confusion matrix further comprises: contributing one false negative to the event-level confusion matrix when no notifications of the predicted occurrence of the clinical event within the notification limit before the onset time of the actual occurrence of the clinical event; and contributing one true negative to the event-level confusion matrix when no notifications of the predicted occurrence of the clinical event occur within the notification limit, and no actual occurrence of the clinic event occurs within the time window.

8. The method of claim 1, wherein the event-level classification metrics are upper bounded by the patient-level classification metrics.Docket No. 2024PF00489- PATENT -9. The method of claim 1, wherein a longer prediction lead time is associated with better performance of the prediction model.

10. The method of claim 1, wherein the clinical event is onset of sepsis.

11. The method of claim 1, wherein the prediction model comprises a light gradient boosting machine (LGBM) regressor model.

12. A system for evaluating performance of a prediction model configured to predict occurrence of a clinical event in a plurality of patients during a corresponding plurality of patient stays, wherein the prediction model runs continuously and outputs risk scores for the plurality of patients at a predetermined frequency, the system comprising: a processing unit; a results database storing results of the prediction model; a display; and a non-transitory memory storing instructions that, when executed by the processing unit, cause the processing unit to: retrieve the results of the prediction model from the results database; based on the retrieved results, identify patient stays of the plurality of patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold, indicating predicted occurrence of the clinical event; for each of the identified patient stays, provide a notification for each of one or more risk scores that cross the predetermined risk threshold; determine a patient-level confusion matrix based on the notifications, wherein each patient stay of the plurality of patient stays contributes one patient data point to the patient-level confusion matrix; determine an event-level confusion matrix based on the notifications, wherein the eventlevel confusion matrix includes prediction lead time and notification limit, wherein each patient stay of the plurality of patient stays contributes one event data point to the event-level confusion matrix, wherein the prediction lead time indicates a length of time before predicted occurrence of29Docket No. 2024PF00489- PATENT - the clinical event that a notification of the clinical event is provided, and wherein the notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications of the predicted occurrence; calculate patient-level classification metrics for false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics for false positives and false negatives of the even -level confusion matrix; identify errors in the predication model based on the patient-level classification metrics and the event-level classification metrics; and cause at least one of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, or the identified errors to be displayed on the display.

13. The system of claim 12, wherein the notification for the one or more risk scores that cross the predetermined risk threshold comprises a discrete notification corresponding to one risk score that cross the predetermined risk threshold or a consolidated notification corresponding to a set of consecutive risk scores that cross the predetermined risk threshold.

14. The system of claim 12, wherein the instructions cause the processing unit to determine the patient-level confusion matrix by: for each patient experiencing no actual occurrence of the clinical event during the patient stay, contributing one false positive to the patient-level confusion matrix for one or more notifications of predicted occurrence of the clinical event by the predication model; for each patient experiencing no actual occurrence of the clinical event during the patient stay, contributing one true negative to the patient-level confusion matrix for no notifications of predicted occurrence of the clinical event by the predication model; for each patient experiencing actual occurrence of the clinical event during the patient stay, contributing one true positive to the patient-level confusion matrix for one or more notifications of predicted occurrence of the clinical event by the predication model prior to the actual occurrence of the clinical event; andDocket No. 2024PF00489- PATENT - for each patient experiencing actual occurrence of the clinical event during the patient stay, contributing one false negative to the patient-level confusion matrix for no notifications of predicted occurrence of the clinical event by the predication model prior to the actual occurrence of the clinical event.

15. The system of claim 12, wherein the patient-level classification metrics and the eventlevel classification metrics comprise at least one of sensitivity, specificity, recall or precision for each patient stay.

16. The system of claim 12, wherein a notification atindicates that, starting from time / , the predicted occurrence of the clinical event will start within a time window [t, t+T], wherein Zis the prediction lead time, and wherein the instructions cause the processing unit to determine the event-level confusion matrix by: contributing one true positive to the event-level confusion matrix when actual occurrence of the clinic event occurs within the time window; and contributing one false positive to the event-level confusion matrix when no actual occurrence of the clinic event occurs within the time window.

17. The system of claim 16, wherein for M notificationsoccurring before the predicted onset of the clinical event for the notification limit ( / .), indicating the maximum number of notifications by the prediction model of the predicted occurrence of the clinical event that the user will act upon before ignoring additional notifications by the prediction model: the M notifications contribute the one true positive to the event-level confusion matrix when tCE< T holds for any notification of the predicted occurrence of the clinical event within the notification limit, wherein tCEis time of predicted occurrence of the clinical event, L is the notification limit, and tLis onset time of the actual occurrence of the clinical event of an L-th notification, and the M notifications contribute one false positive to the event-level confusion matrix when tCE~fmin(L,M) T holds for all notifications of the predicted occurrence of the clinical event within the notification limit, or when there is no actual occurrence of the clinical event.Docket No. 2024PF00489- PATENT -18. The system of claim 17, wherein the instructions further cause the processing unit to determine the event-level confusion matrix by: contributing one false negative to the event-level confusion matrix when no notifications of the predicted occurrence of the clinical event within the notification limit before the onset time of the actual occurrence of the clinical event; and contributing one true negative to the event-level confusion matrix when no notifications of the predicted occurrence of the clinical event occur within the notification limit, and no actual occurrence of the clinic event occurs within the time window.

19. The system of claim 12, wherein the event-level classification metrics are upper bounded by the patient-level classification metrics.

20. A non-transitory computer readable medium storing instructions for evaluating performance of a prediction model configured to predict occurrence of a clinical event in a plurality of patients during a corresponding plurality of patient stays, wherein the prediction model runs continuously and outputs risk scores for the plurality of patients at a predetermined frequency, wherein when executed by a processing unit, the instructions cause the processing unit to: retrieve the results of the prediction model from a results database; based on the retrieved results, identify patient stays of the plurality of patient stays that include at least one risk score generated by the prediction model that crosses a predetermined risk threshold, indicating predicted occurrence of the clinical event; for each of the identified patient stays, provide a notification for each of one or more risk scores that cross the predetermined risk threshold; determine a patient-level confusion matrix based on the notifications, wherein each patient stay of the plurality of patient stays contributes one patient data point to the patient-level confusion matrix; determine an event-level confusion matrix based on the notifications, wherein the eventlevel confusion matrix includes prediction lead time and notification limit, wherein each patientDocket No. 2024PF00489- PATENT - stay of the plurality of patient stays contributes one event data point to the event-level confusion matrix, wherein the prediction lead time indicates a length of time before predicted occurrence of the clinical event that a notification of the clinical event is provided, and wherein the notification limit indicates a maximum number of notifications of the predicted occurrence of the clinical event that a user will act upon before ignoring additional notifications of the predicted occurrence; calculate patient-level classification metrics for false positives and false negatives of the patient-level confusion matrix, and event-level classification metrics for false positives and false negatives of the even -level confusion matrix, wherein errors in the predication model are identified based on the patient-level classification metrics and the event-level classification metrics; and cause at least one of the patient-level confusion matrix, the event-level confusion matrix, the patient-level classification metrics, the event-level classification metrics, or the identified errors to be displayed on a display.

Images