Systems and methods for monitoring artificial intelligence models used to generate or analyze healthcare data

Abstract

A graphical user interface (GUI) for managing healthcare algorithms, wherein the GUI is configured to: simultaneously display (710), to a user, information associated with a number of models (410), wherein each of the number of models is configured to analyze a health metric of a patient; identify (720), to the user, an abnormal model from the number of models (440), wherein the abnormal model is operating outside a safety threshold (516); receive (730), from the user, input associated with the abnormal model (512); and cause (740) an action to be performed in response to receiving the input from the user.

Description

SYSTEMS AND METHODS FOR MONITORING ARTIFICIAL INTELLIGENCE MODELS USED TO GENERATE OR ANALYZE HEALTHCARE DATACROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Patent Application No. 63 / 602,781, filed on November 27, 2023, the entire contents of which are incorporated herein by reference.BACKGROUND

[0002] The present disclosure relates generally to the field of artificial intelligence (Al), and more specifically systems and methods for monitoring Al models used to generate or analyze healthcare data.SUMMARY

[0003] Systems and methods of the present disclosure relate to Algorithmovigilance (i.e., scientific methods and activities relating to the evaluation, monitoring, understanding, and prevention of adverse effects of algorithms in health care). In many contexts, algorithms such as Al models deployed in a healthcare setting are opaque / difficult to understand for end users, and even when understood as to their functions, often perform unexpectedly and may harm people / populations in which they are deployed, thereby undermining trust and raising concerns about the use of algorithms / Al models in healthcare.

[0004] Systems and methods of the present disclosure facilitate a sociotechnical approach to oversight / govemance of Al models deployed in a healthcare context to ensure that such Al models are safe, effective, and equitable. The systems and methods described herein may enable a platform for governance that facilitates integrating isolated algorithms / Al models into a holistic / integrated system to monitor the operation / functioning of algorithms / AI models, enable end users to inform the development of algorithms / AI models, and promote effectiveness, safety, trustworthiness, equity and fairness in the use of algorithms / AI models in a manner that may be more transparent, more cost-effective, and less duplicative than conventional systems and methods. For example, systems and methods of the present disclosure may facilitate an Al governance architecture that is preventative (e.g., incorporatesstability-focused design), preemptive (e.g., facilitates technical oversight), responsive (e.g., facilitates data-driven oversight), and / or reactive (e.g., facilitates end-user reporting).

[0005] Systems and methods described herein may facilitate (i) Al monitoring for organization governance and oversight, (ii) capturing / reporting adverse events, (iii) team-based monitoring, and / or (iv) responding to issues. For example, the systems and methods may monitor algorithm / AI model performance using performance safety bounds, generate metrics (e.g., accuracy / precision metrics, drift metrics, responsiveness metrics, and / or faimess / equity metrics), facilitate a user to take various actions (e.g., investigating a cause of abnormal performance, correcting / updating a model, notifying a clinical team, and / or pausing algorithm / AI model use), facilitate just-in-time feedback between clinicians and technical personnel. Systems and methods of the present disclosure may support a broad range of users (e.g., users with different goals / expertise), use cases (e.g., diverse algorithms / AI models and implementation use cases), meta-data (e.g., enable access to model facts, criticality and organizational contacts), analytics (e.g., support analytic functions including performance metric monitoring and drift detection), flexibility (e.g., provide flexible / customizable temporal displays of algorithm performance / impact), feedback (e.g., receive and display feedback from algorithm end-users), notes (e.g., log notes and action items for each algorithm / AI model), reporting (e.g., provide reporting tools geared for multiple audiences), and / or communication (e.g., support communication about algorithmic management between technical, operational, and clinical teams).

[0006] In some aspects, the techniques described herein relate to a system for monitoring trained artificial intelligence (Al) models used to generate or analyze healthcare data, the system including: a processing circuit including a processor and memory, the memory having instructions stored thereon that, when executed by the processor, cause the processing circuit to: receive model input data corresponding to an input of a trained Al model configured to analyze patient health data or hospital operational data to generate a health metric of a patient; receive timeseries model data describing an output from, or operating parameter derived from, the trained Al model; generate, based on the model input data and the timeseries model data, visualization data including at least one of (i) performance information describing and / or measuring a functioning of the model, (ii) process information describing and / or measuring an interaction between the model and an entity and / or a leading indicator of an ultimate outcome measure, and (iii) outcome information describing and / or measuring a clinical, operational, orhealthcare delivery outcome associated with the patient and / or care delivery; and cause a graphical user interface (GUI) to be displayed to a user.

[0007] In some aspects, the techniques described herein relate to a system, wherein the instructions further cause the processing circuit to: compare the performance information to a threshold; and perform an action in response to comparing the performance information to the threshold. In some aspects, the techniques described herein relate to a system, wherein performing the action includes causing an alert to be displayed to the user, to prompt further investigation as to a possible problem with the algorithm. In some aspects, the techniques described herein relate to a system, wherein performing the action includes disabling or prompting an actor(s) to disable the model. In some aspects, the techniques described herein relate to a system, wherein performing the action includes updating or prompting an actor(s) to update the model. In some aspects, the techniques described herein relate to a system, wherein the performance information includes at least one of: a measure of precision associated with the model; a measure of accuracy associated with the model; an O-to-E ratio; a measure of drift associated with the model; an area under the curve (AUC); a trigger rate stability; a bias; a positive and negative prediction value (PPV); a measure of responsiveness associated with the model; a Brier Score; or a measure of recall associated with the model. In some aspects, the techniques described herein relate to a system, wherein the process information includes at least one of: an acceptance rate associated with the interaction; a fire rate; a number of views associated the model; or a number of times the model has been used.

[0008] In some aspects, the techniques described herein relate to a graphical user interface (GUI) for managing healthcare algorithms, wherein the GUI is configured to: simultaneously display, to a user, information associated with a number of models, wherein each of the number of models is configured to analyze a health metric of a patient; wherein the information includes at least two of (i) performance information describing a functioning of a model of the number of models, (ii) process information describing an interaction between the model of the number of models and an entity, and (iii) outcome information describing a clinical outcome associated with the health metric of the patient the model of the number of models is configured to analyze; identify, to the user, an abnormal model from the number of models, wherein the abnormal model is operating outside a safety threshold; receive, from the user, input associated with the abnormal model; and cause an action to be performed in response to receiving the input from the user.

[0009] In some aspects, the techniques described herein relate to a GUI, wherein the performance information includes at least one of: a measure of precision associated with the model; a measure of accuracy associated with the model; an O-to-E ratio; a measure of drift associated with the model; an area under the curve (AUC); a trigger rate stability; a bias; a positive and negative prediction value (PPV); a measure of responsiveness associated with the model; a Brier Score; or a measure of recall associated with the model. In some aspects, the techniques described herein relate to a GUI, wherein the process information includes at least one of: an acceptance rate associated with the interaction; a fire rate; a number of views associated the model; or a number of times the model has been used.

[0010] In some aspects, the techniques described herein relate to a non-transitory computer- readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to: receive model input data corresponding to an input of a trained Al model configured to analyze patient health data or hospital operational data to generate a health metric of a patient; receive timeseries model data describing an output from, or operating parameter derived from, the trained Al model; generate, based on the model input data and the timeseries model data, visualization data including at least two of (i) performance information describing and / or measuring a functioning of the model, (ii) process information describing and / or measuring an interaction between the model and an entity and / or a leading indicator of an ultimate outcome measure, and (iii) outcome information describing and / or measuring a clinical, operational, or healthcare delivery outcome associated with the patient and / or care delivery; and cause a graphical user interface (GUI) to be displayed to a user.

[0011] In some aspects, the techniques described herein relate to a non-transitory computer- readable storage medium, wherein the instructions further cause the processor to: compare the performance information to a threshold; and perform an action in response to comparing the performance information to the threshold. In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium, wherein performing the action includes causing an alert to be displayed to the user. In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium, wherein performing the action includes disabling the model. In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium, wherein performing the action includes updating the model. In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium, wherein the performance information includes at least one of: a measure of precision associated with the model; a measure of accuracy associated withthe model; an O-to-E ratio; a measure of drift associated with the model; an area under the curve (AUC); a trigger rate stability; a bias; a positive and negative prediction value (PPV); a measure of responsiveness associated with the model; a Brier Score; or a measure of recall associated with the model. In some aspects, the techniques described herein relate to a non- transitory computer-readable storage medium, wherein the process information includes at least one of: an acceptance rate associated with the interaction; a fire rate; a number of views associated the model; or a number of times the model has been used. In some aspects, the techniques described herein relate to a non-transitory computer-readable storage medium, wherein the GUI is used for Al governance.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The above and other aspects and features of the present disclosure will become more apparent to those skilled in the art from the following detailed description of the example embodiments with reference to the accompanying drawings.

[0013] FIG. 1 is a block diagram of a computer system for monitoring Al models, according to an exemplary embodiment.

[0014] FIG. 2 is a flow diagram illustrating a method of monitoring Al models, according to an exemplary embodiment.

[0015] FIGS. 3-6 are graphical user interfaces (GUIs) that may be used to monitor an Al model, according to an exemplary embodiment.

[0016] FIG. 7 is a flow diagram illustrating another method of monitoring Al models, according to an exemplary embodiment.DETAILED DESCRIPTION

[0017] Referring generally to the FIGURES, described herein are systems and methods for monitoring Al models used to generate or analyze healthcare data and / or other data associated with a healthcare context. In many contexts it may be necessary and / or desirable to monitor Al models deployed in a healthcare context. For example, an Al model may be used to predict blood clots in a patient population and it may be desirable to monitor the Al model to determine whether it is operating correctly (e.g., accurately predicting blood clots, not generating false positives and / or false negatives, etc.). As another example, an Al model may be used to monitora patient for sepsis risk and it may be desirable to monitor the Al model to detect any drift in the Al model and correct drift if it occurs (e.g., by retraining the Al model, etc.).

[0018] Systems and methods of the present disclosure facilitate monitoring Al models deployed in a healthcare context. Conventional systems and methods may be impractical for monitoring Al models deployed in a healthcare context. For example, conventional systems and methods may not enable real time monitoring of the use or outcomes associated with an algorithm / Al model. Systems and methods of the present disclosure may overcome limitations associated with conventional systems and methods by (i) continuously monitoring performance information (e.g., metrics, etc.), process information, and / or outcome information associated with Al models deployed in a healthcare context, (ii) identifying Al models that are performing abnormally, (iii) generating and / or displaying a number of metrics associated with the Al models, and / or (iv) performing actions based on continuously monitoring the Al models, thereby facilitating / enabling Al governance in a healthcare context. In various embodiments, systems and methods of the present disclosure combine an inventory of algorithms / Al models, metadata relevant to the algorithms / AI models, an analytics system for monitoring the algorithms / AI models, and / or a visualization system for presenting information to a user into a novel platform for managing the algorithms / AI models in a manner that enables a sociotechnical approach to algorithmic / Al governance.

[0019] Referring now to FIG. 1, computer system 100 for monitoring Al models in a healthcare context is shown, according to an exemplary embodiment. In various embodiments, computer system 100 monitors one or more algorithms (shown as algorithm 180). Algorithm 180 may be an Al model deployed in a healthcare context. For example, algorithm 180 may receive input (e.g., healthcare data) from a data source (e.g., healthcare data source 170) and may generate, via a neural network, one or more outputs based on the input. In various embodiments, algorithm 180 is and / or includes an Al model. For example, algorithm 180 may include a clinical decision support system (e.g., to identify at-risk patients, medication interactions, preventative care reminders, perform early detection of sepsis, perform population health management, etc.), a predictive analytics platform (e.g., to monitor patient vitals / trends and predict clinical deterioration, to generate predictive scoring for intensive care intervention or transfer, to generate risk predictions for chronic conditions, to perform drug efficacy analysis, etc.), and / or a hospital operations and workflow optimization system (e.g., to predict patient discharge, perform bed management, allocate staff, etc.). It should be understood that while FIG. 1 is described in relation to a single algorithm 180, the present disclosure is not limited toa single algorithm and may include any number of algorithms that are monitored simultaneously.

[0020] Healthcare data source 170 may be and / or include an electronic health record. Healthcare data source 170 may store healthcare data associated with one or more patients. For example, healthcare data source 170 may store patient demographics (e.g., name, age, gender, date of birth, contact information, unique identifiers, etc.), socioeconomic data (e.g., address, occupation, language preference, insurance details, etc.), clinical documentation (e.g., medical history, progress notes, nursing notes, etc.), medication and allergies (e.g., prescriptions, dosages, schedules, documented adherence and patient-reported use, previously prescribed medications, known drug allergies / intolerances, etc.), immunization status, laboratory test results, radiology images, vitals (e.g., blood pressure, heart rate, respiratory rate, temperature, body mass index, trends over time, growth charts, etc.), personal statistics (e.g., age, weight, etc.), order and referrals (e.g., medication orders, imaging studies, consultations with specialists, etc.), communication and alerts (e.g., secure communications between providers and patients, drug interactions, abnormal lab values, care reminders, etc.), and / or billing information. Additionally or alternatively, healthcare data source 170 may store other healthcare related information such as operational information associated with the operation of a healthcare facility (e.g., staffing schedules, bed allocations, audit logs, public health data such as vaccination records, etc.). In various embodiments, healthcare data source 170 stores / includes timeseries data.

[0021] In various embodiments, algorithm 180 receives input from healthcare data source 170 and analyzes the input to generate alerts, actions, and / or the like for user device 190. For example, algorithm 180 may analyze a patient’s vitals information stored in healthcare data source 170, generate an early sepsis warning based on the analysis, and transmit an alert to user device 190. As another example, algorithm 180 may analyze patient health data and / or hospital operational data to generate a health metric / alert (e.g., a stroke risk, a sepsis risk, a medication interaction alert, etc.) for a patient and / or trigger further action associated with the patient.

[0022] Computer system 100 may be configured to monitor algorithm 180. For example, computer system 100 may analyze the operation of algorithm 180 to determine whether algorithm 180 is operating correctly (e.g., within preset safety bounds, etc.). As another example, computer system 100 may generate one or more metrics associated with the operation of algorithm 180 and may display the one or more metrics to a user via user device 190 tofacilitate Al governance of algorithm 180. In various embodiments, computer system 100 performs actions in response to monitoring algorithm 180. For example, computer system 100 may generate alerts / send messages to a user, launch applications / trigger other algorithms, retrain algorithm 180, determine whether algorithm 180 is performing properly / whether algorithm 180 requires retraining, pause the operation of algorithm 180, investigate a cause of a failure of algorithm 180, and / or the like. In various embodiments, computer system 100 receives one or more inputs associated with algorithm 180. For example, computer system 100 may receive inputs fed from healthcare data source 170 to algorithm 180 (e.g., log file data, metrics associated with Al models such as fire rates, a measure of how often Al models trigger downstream actions, data related to readmission events, and / or the like).

[0023] In various embodiments, computer system 100 overcomes limitations associated with conventional systems and methods. For example, computer system 100 may facilitate monitoring and optimization of Al models (e.g., via iterative refinement) deployed in a healthcare context. As another example, computer system 100 may facilitate accessing Al model facts, criticality (e.g., dependencies, etc.), organizational contacts, generating analytics (e.g., performance metrics, drift detection) associated with Al models, receiving and displaying feedback from Al model end-users, capturing log notes / action items associated with Al models, and / or supporting communication regarding Al model management between technical, operational, and / or clinical teams. In various embodiments, computer system 100 facilitates Al governance across a healthcare system. For example, a first computer system 100 may monitor Al models deployed at a first location, a second computer system 100 may monitor Al models deployed at a second location, and a third computer system 100 may monitor the Al models at the first location and the second location. In various embodiments, computer system 100 facilitates comparing model performance across locations / units.

[0024] In various embodiments, computer system 100 receives model input data associated with algorithm 180. For example, computer system 100 may receive one or more inputs that are passed to algorithm 180. Model input data may vary based on algorithm 180. For example, a first algorithm 180 may receive a first set of inputs and a second algorithm 180 may receive a second set of inputs and computer system 100 monitoring the first algorithm 180 and the second algorithm 180 may receive both the first set of inputs and the second set of inputs. Model input data may include any information used by algorithm 180 (e.g., data stored in healthcare data source 170, data stored in other databases, etc.). Additionally or alternatively, computer system 100 may receive timeseries model data describing an output from (and / oroperating parameter derived from) algorithm 180. Computer system 100 may generate, based on model input data and / or the timeseries model data, visualization data. The visualization data may include performance information, process information, and / or outcome information. In various embodiments, performance information describes a functioning of algorithm 180. For example, performance information may include a precision metric, a recall metric, a Brier score, an accuracy metric, and / or the like. In various embodiments, process information describes an interaction between algorithm 180 and an entity (e.g., another algorithm, a clinician, a patient, a user, etc.). For example, process information may include a fire rate, a number of views, a measure of how many times an algorithm appears in lists, a measure of the use of an algorithm, an acceptance rate associated with an algorithm, and / or the like. Additionally or alternatively, process information may include and / or describe an intermediate outcome such as a leading indicator of an ultimate outcome measure. For example, process information may include a lab finding. As another example, an algorithm may attempt to reduce an administrative overhead for a healthcare administrator / clinician and the process information may include a measure of how long the healthcare administrator / clinician has spent doing administrative tasks (e.g., how long they have spent interacting with an EHR system, etc.). In various embodiments, outcome information describes a clinical, operational, and / or healthcare delivery outcome associated with a patient. For example, outcome information may include a readmission metric, a clinical deterioration metric, a uterine atony metric, a hospital venous thromboembolism metric, the development of end organ failure, and / or the like. As another example, an algorithm may attempt to reduce an administrative overhead for a healthcare administrator / clinician and the outcome information may include a measure of satisfaction / burnout associated with the healthcare administrator / clinician. In some embodiments, the outcome information is and / or includes a healthcare delivery outcome. For example, the outcome information may include a payment recovery status associated with a healthcare bill. In various embodiments, computer system 100 causes a graphic user interface (GUI) to be displayed to a user. For example, computer system 100 may cause user device 190 to display a GUI including the performance information, the process information, and / or the outcome information. In various embodiments, the GUI facilitates Al governance by allowing users to monitor the operation algorithm 180 and / or perform actions based on the operation of algorithm 180 (e.g., pausing algorithm 180, retraining algorithm 180, investigating a malfunctioning of algorithm 180, etc.).

[0025] Computer system 100 may include processing circuit 110, communication interface 140, storage 150, and / or I / O interface 160. Processing circuit 110 may include processor 120 and / or memory 130. Processor 120 may be a general purpose or specific purpose processor, an application specific integrated circuit (ASIC), one or more field programmable gate arrays (FPGAs), a group of processing components, or other suitable processing components. Processor 120 is configured to execute computer code or instructions stored in memory 130 or received from other computer readable media (e.g., CDROM, network storage, a remote server, etc.). Memory 130 may include one or more devices (e.g., memory units, memory devices, storage devices, and / or other computer-readable media) for storing data and / or computer code for completing and / or facilitating the various processes described in the present disclosure. Memory 130 may include random access memory (RAM), read-only memory (ROM), hard drive storage, temporary storage, non-volatile memory, flash memory, optical memory, or any other suitable memory for storing software objects and / or computer instructions. Memory 130 may include database components, object code components, script components, or any other type of information structure for supporting the various activities and information structures described in the present disclosure. Memory 130 may be communicably connected to processor 120 via processing circuit 110 and may include computer code for executing (e.g., by processor 120) one or more of the processes described herein. For example, memory 130 may have instructions stored thereon that, when executed by processor 120, cause processing circuit 110 to (i) simultaneously display information associated with a number of models, (ii) identify an abnormal model from the number of models, (iii) receive user input associated with the abnormal model, and / or (iv) cause an action to be performed in response to receiving the user input. As another example, memory 130 may have instructions stored there that, when executed by processor 120, cause processing circuit to (i) receive model input data, (ii) receive model data, (iii) generate, based on the model input data and / or the model data, performance information, process information, and / or outcome information, and / or (iv) cause a GUI to be displayed to a user.

[0026] Memory 130 may include model analyzer 132, action module 134, and dashboard module 136. Model analyzer 132 may analyze / monitor an operation / functioning of algorithm 180. For example, model analyzer 132 may compare an output of algorithm 180 to a threshold / bound to determine whether algorithm 180 is operating as expected. In various embodiments, model analyzer 132 generates one or more metrics. For example, model analyzer 132 may generate an accuracy metric, a precision metric, a positive predictive value (PPV)metric, a sensitivity metric, a recall metric, a specificity metric, a negative predicted value, a Brier score, an area under the precision recall curve (AUPRC), an area under the receiver operating characteristic curve (AUC), an observed-to-expected (O-to-E) ratio, a Cox intercept, a Cox slope, an estimated calibration index (ECI), an integrated calibration index (ICI), dynamic calibration curves, a fire rate, an acceptance metric, a predictive equality metric, an equal opportunity metric, a metric parity metric, an equalized odds metric, and / or a treatment equality metric. Additionally or alternatively, model analyzer 132 may perform drift detection and / or generate a bias metric. In some embodiments, model analyzer 132 receives one or more metrics from an external source (e.g., algorithm 180, healthcare data source 170) and use the one or more metrics to analyze / monitor algorithm 180.

[0027] An accuracy metric may describe a proportion of classification predictions that were correct. The accuracy metric may be calculated as:True positives + True negatives Number of observations

[0028] A precision / PPV metric may describe a proportion of positive classification predictions that were correct. The precision / PPV metric may be calculated as:True positivesTrue positives + False positives

[0029] A sensitivity / recall metric may describe a true positive rate. For example, the sensitivity / recall metric may describe a proportion of positive classification predictions among observations with a positive outcome. The sensitivity / recall metric may be calculated as:True positivesTrue positives + False negatives

[0030] A specificity metric may describe a true negative rate. For example, the specificity metric may describe a proportion of negative classification predictions among observations with a negative outcome. The specificity metric may be calculated as:True negatives True negatives + False positives

[0031] A negative predicted value may describe a proportion of negative classification predictions that were correct. The negative predicted value may be calculated as:True negativesTrue negatives + False negatives

[0032] A Brier score is a mean squared difference between an observed outcome and a predicted probability. An AUPRC is a curve based on recall (e.g., sensitivity) vs. precision (PPV). In various embodiments, values greater than the outcome rate indicate an informative model. An AUC is a curve based on false positive rate vs. true positive rate (e.g., sensitivity). The AUC may describe an assessment of the probability that an observation with the outcome is assigned a higher risk estimate than an observation without the outcome. An O-to-E ratio is a ratio of outcome rate to mean predicted probability. A Cox intercept (<z) is the intercept of the logistic calibration curve model (i.e., loglt(y) = a + ft * Ip; where ? = 1). A Cox slope ( ?) is the slope of the logistic calibration curve model (i.e., loglt(y) = a + / 3 * Ip). An ECI is a mean squared difference between predicted probabilities and observed probabilities estimated from flexible calibration curves. An ICI is a mean absolute difference between predicted probabilities and observed probabilities estimated from flexible calibration curves. A dynamic calibration curve is a graphical representation of model performance across a range of predicted probability. The graphical representations may plot predicted probability vs. mean / fitted observed value. In various embodiments, model analyzer 132 dynamically updates curves as data accrues (e.g., such that the dynamic calibration curves evolve to reflect recent performance in the case of performance drift). A fire rate may describe number of predictions generated per time period of interest. An acceptance metric may describe a proportion of model guided CDS recommendations accepted by user. A predictive equality metric may describe a false positive rate differential between subpopulations of interest. An equal opportunity metric may describe an equality of true positive rate between subpopulations of interest. A metric parity metric may describe a differential in other metrics of interest between subpopulations of interest. An equalized odds metric may describe an equality of true positive rate and false negative rate between subpopulations of interest. In various embodiments, the equalized odds metric is measured as the larger of the two differences between true positive rate or false negative rate across groups. A ratio may be measured as the more extreme (e.g., farther from 1) of the ratio of a true positive rate or a false negative rate across groups. A treatment equality metric may describe an equality of the ratio of false negatives to false positives between subpopulations of interest.

[0033] In various embodiments, model analyzer 132 detects performance drift and / or changes in input features distributions using sequential analyses. For example, model analyzer 132 may diagnoses and / or responds to model drift, thereby facilitating continuous improvement and patient safety. As another example, model analyzer 132 may monitor performance, detectchanges, suggest updates in response to performance deterioration, recommend a window of recent data for training updates, and / or select from candidate updating methods to restore model performance. In some embodiments, model analyzer 132 uses fixed, sliding, and / or dynamic lookback windows. For example, model analyzer 132 may use x-charts, drift detection method (DDM), early drift detection method (EDDM), cumulative sum (CUSUM), exponentially weighted moving average (EWMA) charts, and / or adaptive windowing (Adwin). As another example, model analyzer 132 may support a dynamic lookback window for any bounded metric and may generate a calibration error metric using dynamic calibration curves describing an absolute value of a difference between a predicted probability and a fitted probability based on an up-to-date calibration curve (or any other bounded metric such as sensitivity, specificity, AUC, etc.). To continue the example, model analyzer 132 may maintain a timeseries of these errors as data accrue and may continue to grow the window as long as the errors appear to be from a stable process. As the data accrues, model analyzer 132 analyzes sliding pairs of recent and older windows of data, with the sliding window progressively shrinking the set of recent data, to determine if the error metric is increasing among the recent data. If a statistically significant change in error is found, model analyzer 132 may trigger an alert and return the window of recent data that appear to be from a new data context.

[0034] In various embodiments, action module 134 is configured to perform / facilitate one or more actions based on analysis / monitoring of algorithm 180. For example, model analyzer 132 may monitor algorithm 180, identify an out-of-bound error and may trigger action module 134 to perform an action based on the out-of-bound error. The actions may include investigating a cause of an error, correcting and / or updating an Al model, notifying a user, pausing an Al model, launching other applications, generating alerts / sending messages, and / or the like. In some embodiments, action module 134 performs an action in response to receiving user input. For example, action module 134 may receive input from a user via a GUI and may cause algorithm 180 to pause operation (e.g., by making an application programming interface (API) call to algorithm 180, etc.). In some embodiments, the actions are performed by a user. For example, action module 134 may present contact information to a user and the user may use the contact information to pause an algorithm (e.g., by contacting an individual responsible for the algorithm, etc.). In some embodiments, action module 134 performs actions dynamically. For example, action module 134 may automatically (e.g., with little to no user intervention) pause an algorithm in response to determining that the algorithm is functioning outside a safety threshold.

[0035] In various embodiments, dashboard module 136 is configured to generate visualization data and / or cause user device 190 to display a GUI. For example, dashboard module 136 may receive a number of metrics generated by model analyzer 132 and may construct a GUI and cause the GUI to be displayed on user device 190. In various embodiments, dashboard module 136 causes user device 190 to display a GUI including performance information, process information, and / or outcome information. In various embodiments, dashboard module 136 simultaneously displays information associated with a number of Al models, thereby facilitating Al governance of Al models in a healthcare context. In various embodiments, dashboard module 136 identifies / flags Al models that are operating abnormally to facilitate user intervention. For example, model analyzer 132 may determine that algorithm 180 is operating outside a safety threshold / bound and in response dashboard module 136 may identify algorithm 180 for a user (e.g., via user device 190) so that the user can take an action (e.g., pause algorithm 180, retrain algorithm 180, etc.). GUIs are described in greater detail below with reference to FIGS. 3-6.

[0036] Communication interface 140 may facilitate communication with one or more systems / devices. For example, computer system 100 may communicate with an electronic health records (EHR) system to receive vitals information associated with a patient via communication interface 140. In various embodiments, communication interface 140 facilitates communication with algorithm 180. For example, communication interface 140 may receive one or more inputs and / or outputs of algorithm 180. Communication interface 140 may be or include wired or wireless communications interfaces (e.g., jacks, antennas, transmitters, receivers, transceivers, wire terminals, etc.) for conducting data communications with external systems or devices. In various embodiments, communications via communication interface 140 is direct (e.g., local wired or wireless communications). Additionally or alternatively, communications via communication interface 140 may utilize a network (e.g., a WAN, the Internet, a cellular network, etc.).

[0037] Storage 150 may store data / information associated with monitoring / analyzing algorithm 180. For example, storage 150 may store timeseries inputs to algorithm 180 and / or outputs of algorithm 180. As another example, storage 150 may partially or wholly duplicate healthcare data source 170. In various embodiments, storage 150 stores timeseries data. Additionally or alternatively, storage 150 may store algorithm 180. For example, algorithm 180 may operate from within computer system 100. Storage 150 may be and / or include one or morememory devices (e.g., hard drive storage, temporary storage, non-volatile memory, flash memory, optical memory, and / or any other suitable memory device).

[0038] I / O interface 160 may facilitate input / output operations. For example, I / O interface 160 may include a display capable of presenting information from a user and an interface capable of receiving input from the user. In some embodiments, VO interface 160 includes a display device configured to present a GUI to a user. VO interface 160 may include hardware and / or software components. For example, VO interface 160 may include a physical input device (e.g., a mouse, a keyboard, a touchscreen device, etc.) and software to enable the physical input device to communicate with computer system 100 (e.g., firmware, drivers, etc.). In some embodiments, VO interface 160 includes an API to facilitate interaction with external systems (e.g., healthcare data source 170, algorithm 180, etc.).

[0039] Referring now to FIG. 2, method 200 of monitoring Al models is shown, according to an exemplary embodiment. Method 200 may facilitate Al governance in a healthcare context by generating metrics associated with the functioning of a number of Al models, displaying the metrics to a user, and allowing the user to take actions based on the metrics. In various embodiments, method 200 is performed by computer system 100. At step 210, computer system 100 receives model input data and / or model data. In various embodiments, the model input data corresponds to an input to a trained Al model (e.g., algorithm 180, etc.) configured to analyze patient health data and / or hospital operational data to generate a health metric associated with a patient. For example, the model input data may include a patient’s vitals information that an Al model may use to predict the patient’s stroke risk. In various embodiments, the model data describes an output from and / or operating parameter derived from a trained Al model. For example, the model data may include a stroke risk metric generated by a trained Al model. As another example, the model data may include a fire rate metric associated with a trained Al model derived from outputs of the trained Al model.

[0040] At step 220, computer system 100 generates, based on the model input data and / or the model data, visualization data. The visualization data may include performance information, process information, and / or outcome information. In various embodiments, the visualization data includes and / or corresponds to a GUI. GUIs are discussed in greater detail below with reference to FIGS. 3-6. At step 230, computer system 100 causes a GUI to be displayed. In various embodiments, the GUI includes the performance information, the process information, and / or the outcome information. In various embodiments, the GUI facilitatesmonitoring / governance of Al models. For example, the GUI may display a number of metrics associated with the functioning of a number of Al models deployed in a healthcare context and may enable a user to take various actions with respect to the Al models (e.g., pausing the operation of an Al model, retraining an Al model, etc.).

[0041] Referring now to FIGS. 3-6, various GUIs that may facilitate Al monitoring and / or governance are shown, according to several exemplary embodiments. In various embodiments, computer system 100 generates the GUIs of FIGS. 3-6 (shown as GUIs 300-600). For example, computer system 100 may generate visualization data and may transmit the visualization data to user device 190 to cause user device to display GUI 300 to a user. In various embodiments, GUIs 300-600 enable a sociotechnical approach to Al governance by facilitating real-time monitoring of information associated with algorithms / AI models deployed in a healthcare context (e.g., metrics describing how the algorithms / AI models are currently functioning, metrics describing intermediate outcomes associated with entities monitored by the algorithms / AI models, outcomes associated with entities monitored by the algorithms / AI models, etc.).

[0042] Referring now to FIG. 3, GUI 300 is shown, according to an exemplary embodiment. GUI 300 may display one or more Al models (e.g., such as algorithm 180, etc.) (shown as model 310). Model 310 may correspond to a trained Al model configured to analyze patient health data and / or hospital operational data to generate a health metric for a patient. For example, model 310 may be a blood clot prediction model configured to generate a blood clot risk associated with a patient based on information about the patient. GUI 300 may display one or more metrics associated with model 310 (shown as metric 320). In various embodiments, GUI 300 facilitates selecting additional metrics to generate and / or display (shown as additional metrics 330). In various embodiments, GUI 300 identifies an abnormal model (shown as abnormal model 340). For example, GUI 300 may identify a metric corresponding to a model that falls outside a threshold / bound. In some embodiments, identifying an abnormal model includes emphasizing the abnormal model (e.g., using a different color, using an alert, etc.). In some embodiments, identifying an abnormal model includes identifying a metric associated with the abnormal model (e.g., a metric that violates a condition, etc.). In various embodiments, GUI 300 facilitates monitoring / governance of Al models deployed in a healthcare context by simultaneously displaying information associated with a number of Al models in such a way as to allow a user to determine whether the Al models are operating correctly and / or to take corrective action in the event that an Al model deviates from expected behavior. In variousembodiments, GUI 300 facilitates a user to take further action based on the information presented. For example, a user may show alerts associated with an Al model and / or submit feedback to an individual associated with an Al model (e.g., an engineer in charge of deploying the Al model, etc.).

[0043] Referring now to FIG. 4, GUI 400 is shown, according to an exemplary embodiment. GUI 400 may display one or more Al models (e.g., such as algorithm 180, etc.) (shown as model 410). GUI 400 may simultaneously display one or more metrics associated with model 410 (shown as metrics 420). In various embodiments, metrics 420 are grouped by category (shown as category 430). For example, a first number of metrics may be in a performance category and may relate to a functioning of the Al model, a second number of metrics may be in a process category and may relate to an interaction between the Al model and an entity (e.g., another Al model, a user, etc.), and a third number of metrics may be in an outcome category and may relate to a clinical outcome associated with a health metric the Al model is configured to analyzed. In various embodiments, GUI 400 identifies an abnormal value / metric associated with an Al model that may be operating abnormally (shown as abnormal value 440). For example, GUI 400 may identify a value / metric (e.g., how often the Al model appears in lists, a clinical deterioration metric of a patient monitored by the Al model, etc.) that falls outside a threshold / bound, and may emphasize the value / metric for a user (e.g., by highlighting the value / metric, by changing a color of the value / metric, etc.). In some embodiments, GUI 400 surfaces the Al model for review based on identifying the abnormal value / metric. In various embodiments, GUI 400 facilitates a user to take further action based on the information presented. For example, a user may show alerts associated with an Al model, generate reports based on the functioning of an Al model, and / or adjust the setting of an Al model.

[0044] Referring now to FIG. 5, GUI 500 is shown, according to an exemplary embodiment. GUI 400 may facilitate displaying in-depth information associated with a specific Al model. For example, a user may click on an Al model within GUI 400 and cause an accordion dropdown to be displayed as in FIG. 5 (shown as details pane 510). Details pane 510 may display a list of where the Al model is deployed. Details pane 510 is shown to include actions 512 and detailed metrics 514. Actions 512 may enable a user to take various actions associated with the Al model. For example, a user may identify contacts (e.g., names, email addresses, phone numbers, etc.) associated with the Al model, may view additional details associated with the Al model, may generate reports associated with the Al model, may view data that Al model is using, and / or may view user feedback associated with the Al model. Detailed metrics 514may display a metric associated with the Al model over a period of time. For example, detailed metrics 514 may include a graph displaying a readmission rate associated with patients monitored by an Al model over a 9-month window. In various embodiments, a user may select the period of time. In various embodiments, detailed metrics 514 display updates to the Al model (shown as “Update (2.53)” in FIG. 5). In various embodiments, detailed metrics 514 include a threshold associated with the metric (shown as threshold 516). Al models having a metric value that does not satisfy threshold 516 may be classified as abnormal and may trigger review by a user. In various embodiments, GUI 500 identifies an abnormal value / metric associated with the Al model (shown as abnormal value 540). For example, GUI 500 may determine that a readmission metric associated with an Al model does not satisfy a threshold, may emphasize the Al model, and may display detailed metric information associated with the Al model to allow a user to determine why the Al model is operating abnormally (e.g., as shown in FIG. 5, update 2.53 could have caused the “Cornelius” model to behave abnormally and therefore a user should revert Cornelius to a prior version, etc.).

[0045] Referring now to FIG. 6, GUI 600 is shown, according to an exemplary embodiment. GUI 600 may include details pane 610. Details pane 610 may display detailed information associated with the operation / functioning of an Al model. Details pane 610 is shown to include detailed metrics 620, deployment locations 630, performance information 640, process information 650, and outcome information 660. Detailed metrics 620 may display a metric associated with the Al model over a period of time. For example, detailed metrics 620 may include a graph displaying an accuracy metric, a model distribution metric, a readmission metric, and / or an average height metric associated with an Al model. In various embodiments, a user may select the period of time. Deployment locations 630 may display a list of locations / units where the Al model is deployed. In various embodiments, deployment locations 630 facilitate filtering the information displayed in details pane 610. For example, a user may deselect deployment locations and detailed metrics 620 may update to remove information associated with the deselected deployment locations. Performance information 640 may display one or more metrics associated with a functioning of the Al model. For example, performance information 640 may include a Brier score, a PPV, a NPV, an AUC, and / or an O- to-E ratio. Process information 650 may display one or more metrics describing an interaction between the Al model and an entity. For example, process information 650 may include a total patient count (e.g., a number of patients’ data that the Al model analyzed, etc.), a total model runs (e.g., how many times the Al model was executed, etc.), an alerts fired count (e.g., howmany times the Al model generated an alert, etc.), and / or an acceptance rate (e.g., a ratio of how often an output of the Al model is verified by a user, etc.). Outcome information 660 may display one or more metrics describing a clinical outcome associated with patients monitored by the Al model. For example, outcome information 660 may include a DVT rate and / or a PE rate.

[0046] Referring now to FIG. 7, method 800 of monitoring Al models is shown, according to an exemplary embodiment. In various embodiments, method 800 is performed by computer system 100. Method 800 may facilitate monitoring / governance of Al models deployed in a healthcare context by allowing users to simultaneously view a number of metrics associated with the functioning of the Al models and take actions based on the metrics (e.g., pausing Al models that are performing abnormally, etc.). At step 710, computer system 100 may simultaneously display information associated with a number of models (e.g., algorithm 180, etc.). In various embodiments, each of the number of models is configured to analyze a health metric of a patient. In various embodiments, the information includes performance information, process information, and / or outcome information.

[0047] At step 720, computer system 100 may identify an abnormal model from the number of models. For example, computer system 100 may compare a metric corresponding to the performance information, the process information, and / or the outcome information associated with an Al model to a threshold and may identify the Al model as abnormal based on determining that the metric does not satisfy the threshold. In various embodiments, the threshold is a safety threshold. For example, an Al model configured to predict blood clots may be identified as an abnormal model in response to determining that a false positive rate associated with blood clot predictions generated by the Al model exceeds a safety threshold (e.g., 40%, etc.). In various embodiments, step 720 includes emphasizing the abnormal model in a GUI. For example, computer system 100 may update a GUI to change a color of a metric associated with the abnormal model.

[0048] At step 730, computer system 100 may receive an input associated with the abnormal model. In various embodiments, computer system 100 receives the input from a user (e.g., via user device 190, etc.). For example, computer system 100 may receive a request from a user to view details of the abnormal model, to generate a report associated with the abnormal model, to view data used by the abnormal model, to pause a use of the abnormal model, to investigate a cause of the malfunctioning of the abnormal model, and / or to update the abnormal model.

[0049] At step 740, computer system 100 may cause an action to be performed in response to receiving the input. For example, computer system 100 may cause details of the abnormal model to be displayed (e.g., via GUIs 300-600, etc.), may generate a report associated with the abnormal model, may display data used by the abnormal model, may pause a use of the abnormal model, may investigate a cause of the malfunctioning of the abnormal model, and / or to may update the abnormal model. In some embodiments, the action is performed by a user. For example, a user may determine that an algorithm is operating unexpectedly and may retrieve contact information for an individual associated with the algorithm (e.g., via GUIs 300- 600) and contact the individual to have the individual pause the algorithm. As another example, a user may determine that an algorithm is drifting and may retrieve contact information for an individual associated with the algorithm (e.g., via GUIs 300-600) and contact the individual to update / retrain the model using additional data suggested by computer system 100.

[0050] As another example, a user may determine that an outcome associated with an algorithm has deteriorated and may investigate a cause of the deterioration using GUIs 300- 600.

[0051] As utilized herein with respect to numerical ranges, the terms “approximately,” “about,” “substantially,” and similar terms generally mean+ / -10% of the disclosed values, unless specified otherwise. As utilized herein with respect to structural features (e.g., to describe shape, size, orientation, direction, relative position, etc.), the terms “approximately,” “about,” “substantially,” and similar terms are meant to cover minor variations in structure that may result from, for example, the manufacturing or assembly process and are intended to have a broad meaning in harmony with the common and accepted usage by those of ordinary skill in the art to which the subject matter of this disclosure pertains. Accordingly, these terms should be interpreted as indicating that insubstantial or inconsequential modifications or alterations of the subject matter described and claimed are considered to be within the scope of the disclosure as recited in the appended claims.

[0052] It should be noted that the term “exemplary” and variations thereof, as used herein to describe various embodiments, are intended to indicate that such embodiments are possible examples, representations, or illustrations of possible embodiments (and such terms are not intended to connote that such embodiments are necessarily extraordinary or superlative examples).

[0053] The term “coupled” and variations thereof, as used herein, means the joining of two members directly or indirectly to one another. Such joining may be stationary (e.g., permanent or fixed) or moveable (e.g., removable or releasable). Such joining may be achieved with the two members coupled directly to each other, with the two members coupled to each other using a separate intervening member and any additional intermediate members coupled with one another, or with the two members coupled to each other using an intervening member that is integrally formed as a single unitary body with one of the two members. If “coupled” or variations thereof are modified by an additional term (e.g., directly coupled), the generic definition of “coupled” provided above is modified by the plain language meaning of the additional term (e.g., “directly coupled” means the joining of two members without any separate intervening member), resulting in a narrower definition than the generic definition of “coupled” provided above. Such coupling may be mechanical, electrical, or fluidic.

[0054] References herein to the positions of elements (e.g., “top,” “bottom,” “above,” “below”) are merely used to describe the orientation of various elements in the figures. It should be noted that the orientation of various elements may differ according to other exemplary embodiments, and that such variations are intended to be encompassed by the present disclosure.

[0055] The present disclosure contemplates methods, systems, and program products on any machine-readable media for accomplishing various operations. The embodiments of the present disclosure may be implemented using existing computer processors, or by a special purpose computer processor for an appropriate system, incorporated for this or another purpose, or by a hardwired system. Embodiments within the scope of the present disclosure include program products comprising machine-readable media for carrying or having machineexecutable instructions or data structures stored thereon. Such machine-readable media can be any available media that can be accessed by a general purpose or special purpose computer or other machine with a processor. By way of example, such machine-readable media can comprise RAM, ROM, EPROM, EEPROM, or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to carry or store desired program code in the form of machine-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer or other machine with a processor. Combinations of the above are also included within the scope of machine-readable media. Machine-executable instructions include, for example, instructions and data which cause a general-purpose computer, special purpose computer, or special purpose processing machines to perform a certain function or group of functions.

[0056] Although the figures and description may illustrate a specific order of method steps, the order of such steps may differ from what is depicted and described, unless specified differently above. Also, two or more steps may be performed concurrently or with partial concurrence, unless specified differently above. Such variation may depend, for example, on the software and hardware systems chosen and on designer choice. All such variations are within the scope of the disclosure. Likewise, software implementations of the described methods could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various connection steps, processing steps, comparison steps, and decision steps.

[0057] The term “client or “server” include all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, a system on a chip, or multiple ones, or combinations, of the foregoing. The apparatus may include special purpose logic circuitry, e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC). The apparatus may also include, in addition to hardware, code that creates an execution environment for the computer program in question (e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, a cross -platform runtime environment, a virtual machine, or a combination of one or more of them). The apparatus and execution environment may realize various different computing model infrastructures, such as web services, distributed computing and grid computing infrastructures.

[0058] The systems and methods of the present disclosure may be completed by any computer program. A computer program (also known as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, sub programs, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0059] The processes and logic flows described in this specification may be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows may also be performed by, and apparatus may also be implemented as, special purpose logic circuitry (e.g., an FPGA or an ASIC).

[0060] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random-access memory or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data (e.g., magnetic, magneto-optical disks, or optical disks). However, a computer need not have such devices. Moreover, a computer may be embedded in another device (e.g., a vehicle, a Global Positioning System (GPS) receiver, etc.). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks). The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0061] To provide for interaction with a user, implementations of the subject matter described in this specification may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display), OLED (organic light emitting diode), TFT (thin-film transistor), or other flexible configuration, or any other monitor for displaying information to the user. Other kinds of devices may be used to provide for interaction with a user as well; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback).

[0062] Implementations of the subject matter described in this disclosure may be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer) having a graphical user interface or a web browser throughwhich a user may interact with an implementation of the subject matter described in this disclosure, or any combination of one or more such back end, middleware, or front end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a LAN and a WAN, an inter-network (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks).

Claims

WHAT IS CLAIMED IS:

1. A system for monitoring trained artificial intelligence (Al) models used to generate or analyze healthcare data, the system comprising: a processing circuit comprising a processor and memory, the memory having instructions stored thereon that, when executed by the processor, cause the processing circuit to: receive model input data corresponding to an input of a trained Al model configured to analyze patient health data or hospital operational data to generate a health metric of a patient; receive timeseries model data describing an output from, or operating parameter derived from, the trained Al model; generate, based on the model input data and the timeseries model data, visualization data comprising at least one of (i) performance information describing a functioning of the model, (ii) process information describing an interaction between the model and an entity, and (iii) outcome information describing a clinical outcome associated with the patient or a healthcare delivery outcome; and cause a graphical user interface (GUI) to be displayed to a user.

2. The system of claim 1, wherein the instructions further cause the processing circuit to: compare the performance information to a threshold; and perform an action in response to comparing the performance information to the threshold.

3. The system of claim 2, wherein performing the action comprises causing an alert to be displayed to the user.

4. The system of claim 2, wherein the GUI includes contact information for an individual associated with the trained Al model.

5. The system of claim 2, wherein the visualization data comprises at least two of the performance information, the process information, and the outcome information.

6. The system of claim 2, wherein the visualization data comprises timeseries data spanning a period of time.

7. The system of claim 1, wherein the performance information comprises at least one of:a measure of precision associated with the model; a measure of accuracy associated with the model; an O-to-E ratio; a measure of drift associated with the model; an area under the curve (AUC); a trigger rate stability; a bias; a positive and negative prediction value (PPV); a measure of responsiveness associated with the model; a Brier Score; or a measure of recall associated with the model.

8. The system of claim 1, wherein the process information comprises at least one of: an acceptance rate associated with the interaction; a fire rate; a number of views associated the model; or a number of times the model has been used.

9. A graphical user interface (GUI) for managing healthcare algorithms, wherein the GUI is configured to: simultaneously display, to a user, information associated with a plurality of models, wherein each of the plurality of models is configured to analyze a health metric of a patient; wherein the information comprises at least two of (i) performance information describing a functioning of a model of the plurality of models, (ii) process information describing an interaction between the model of the plurality of models and an entity, and (iii) outcome information describing a clinical outcome associated with the health metric of the patient the model of the plurality of models is configured to analyze or a healthcare delivery outcome; identify, to the user, an abnormal model from the plurality of models, wherein the abnormal model is operating outside a safety threshold; receive, from the user, input associated with the abnormal model; and cause an action to be performed in response to receiving the input from the user.

10. The GUI of claim 9, wherein the performance information comprises at least one of: a measure of precision associated with the model; a measure of accuracy associated with the model;an O-to-E ratio; a measure of drift associated with the model; an area under the curve (AUC); a trigger rate stability; a bias; a positive and negative prediction value (PPV); a measure of responsiveness associated with the model; a Brier Score; or a measure of recall associated with the model.

11. The GUI of claim 9, wherein the process information comprises at least one of: an acceptance rate associated with the interaction; a fire rate; a number of views associated the model; or a number of times the model has been used.

12. A non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processor, cause the processor to: receive model input data corresponding to an input of a trained Al model configured to analyze patient health data or hospital operational data to generate a health metric of a patient; receive timeseries model data describing an output from, or operating parameter derived from, the trained Al model; generate, based on the model input data and the timeseries model data, visualization data comprising at least one of (i) performance information describing a functioning of the model, (ii) process information describing an interaction between the model and an entity, and (iii) outcome information describing a clinical outcome associated with the patient or a healthcare delivery outcome; and cause a graphical user interface (GUI) to be displayed to a user.

13. The non-transitory computer-readable storage medium of claim 12, wherein the instructions further cause the processor to: compare the performance information to a threshold; and perform an action in response to comparing the performance information to the threshold.

114. The non-transitory computer-readable storage medium of claim 13, wherein performing the action comprises causing an alert to be displayed to the user.

15. The non-transitory computer-readable storage medium of claim 13, wherein the GUI includes contact information for an individual associated with the trained Al model.

16. The non-transitory computer-readable storage medium of claim 13, wherein the visualization data comprises at least two of the performance information, the process information, and the outcome information.

17. The non-transitory computer-readable storage medium of claim 13, wherein the visualization data comprises timeseries data spanning a period of time and wherein the GUI comprises a graphical representation of the timeseries data spanning the period of time.

18. The non-transitory computer-readable storage medium of claim 12, wherein the performance information comprises at least one of: a measure of precision associated with the model; a measure of accuracy associated with the model; an O-to-E ratio; a measure of drift associated with the model; an area under the curve (AUC); a trigger rate stability; a bias; a positive and negative prediction value (PPV); a measure of responsiveness associated with the model; a Brier Score; or a measure of recall associated with the model.

19. The non-transitory computer-readable storage medium of claim 12, wherein the process information comprises at least one of: an acceptance rate associated with the interaction; a fire rate; a number of views associated the model; or a number of times the model has been used.

20. The non-transitory computer-readable storage medium of claim 12, wherein the GUI is used for Al governance.

Images