Systems, methods and processes for model performance aggregation - Patents.com

Abstract

Disclosed herein are methods, systems, and computer-readable media for a concurrent comparison tool for aggregating performance reports generated by computations of machine learning data model pipelines for technical performance and / or predictive accuracy. The tool can obtain multiple performance measure reports that quantitatively evaluate the performance of the model pipelines based on a configuration file that facilitates validation of the technical performance and predictive accuracy of the models. Furthermore, the tool can broadcast notifications to predefined systems when conditions defined by the configuration file are met for the performance measure reports.

Description

[Technical field]

[0001] FIELD OF THEINVENTION This application relates to machine learning data models, and more particularly, to the comparative evaluation of predictive accuracy and performance measures of distinct versions of machine learning data models. Summary of the Invention

[0002] Summary of the Invention Embodiments of the present disclosure relate to, among other things, methods, systems, and computer-readable media for aggregating performance reports generated by computations of a machine learning data model (hereinafter "model") pipeline for technical performance and / or predictive accuracy. As described below, aspects of the invention discussed below monitor and compare technical performance and / or predictive accuracy by monitoring multiple sub-models of a model pipeline.

[0003] A first embodiment may include a computerized method for aggregating calculated performance measure alerts within an executable reporting job. The method may include automatically initiating an executable reporting job in response to the occurrence of an event programmatically coded into a script of one or more applications maintaining the reporting job to calculate one or more performance measures for the predictive model. In some embodiments, the reporting job includes an operation of configuring delivery of calculated performance measure alerts output by the reporting job based on the predictive model. The computerized method also includes storing the one or more calculated performance measure alerts for the model in a database in response to execution of the operations of the reporting job without delivering a notification of the one or more calculated performance measure alerts. Additionally, the computerized method may include automatically initiating an executable reporting job to calculate another one or more performance measures for the predictive model, and in response to execution of the operations of the reporting job, storing another one or more calculated performance measure alerts in a database and delivering a notification of both the one or more calculated performance measure alerts and the another one or more calculated performance measure alerts.

[0004] Another embodiment may include a system for aggregating calculated performance measures of a predictive model into an alert notification. The system may include at least one processor and a computer storage medium storing computer readable instructions. The computer readable instructions cause the processor to initiate a reporting job in response to an occurrence of an event programmatically coded into the script of one or more applications maintaining the reporting job to calculate one or more performance measures for the predictive model. In some embodiments, the reporting job includes an operation of configuring delivery of a calculated performance measure alert output by the reporting job based on the predictive model. The instructions may also cause the processor to store one or more calculated performance measure alerts for the model in a database in response to execution of the operations of the reporting job without delivering a notification of the one or more calculated performance measure alerts. The instructions may also cause the processor to automatically initiate an executable reporting job to calculate another one or more performance measures for the predictive model and to store another one or more calculated performance measure alerts in a database in response to execution of the operations of the reporting job, as well as deliver a notification of both the one or more calculated performance measure alerts and the another one or more calculated performance measure alerts.

[0005] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The present invention is defined by the claims, as supported by this specification including the Detailed Description.

[0006] The present system and method for aggregated alerts of performance measures of computer models is described in detail below with reference to the figures of the accompanying drawings. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a process diagram illustrating an example process for aggregated alerting of model pipeline performance measures according to some embodiments of the disclosure. [Diagram 2] FIG. 1 illustrates an example system environment that can facilitate aggregated alerting of model pipeline performance measures in accordance with some embodiments of the present disclosure. [Diagram 3] FIG. 1 illustrates an example code schema for a configuration file to facilitate aggregated alerting of model pipeline performance measures, according to some embodiments of the present disclosure. [Figure 3A] FIG. 2 illustrates an example code snippet of a configuration file according to some embodiments of the present disclosure. [Figure 4] FIG. 13 is a diagram of an example aggregated alert notification including model pipeline performance measures, according to some embodiments of the present disclosure. [Diagram 5] FIG. 1 illustrates an example method for condition-based delivery of aggregated alert notifications according to some embodiments of the present disclosure. [Figure 6] FIG. 1 is a block diagram of an example environment suitable for implementing aspects of an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Detailed Description of the Invention Computer-based models are often used to predict future events using currently available data. The performance of these models may be iteratively evaluated during development and after deployment to facilitate continuous improvement and to ensure predictive accuracy and precision. Broadly speaking, predictive modeling of these dynamic or chaotic systems is subject to initial data conditions. For example, a computer model may predict weather patterns for the next day, week, or month, or a computer model may predict hospitalization rates for a patient care facility.

[0009] In such situations, the dynamic or chaotic fluctuations of the data consumed by the model may cause the predictions to become unstable. This instability may be a temporary artifact or may reveal computational, training, data encoding, data decoding, or other technical deficiencies of the model. In contrast to conventional approaches to model pipeline performance detection, aspects described herein provide for aggregating performance reports generated by computations of machine learning data model (hereinafter "model") pipelines for technical performance and / or prediction accuracy.

[0010] definition As used herein, the terms "observed data," "ground truth," "actual values," and "target values" are used interchangeably to refer to empirical data and / or observed real-world information coded as data. For example, observed data includes measured, captured, or recorded values ​​that represent and / or quantify variables for an event or outcome that occurred. In one example, observed data includes values ​​for a particular medical institution's total patient volume that occurred over a defined six-month period as recorded in the institution's historical reporting data.

[0011] The term "prediction data" as used herein refers to any and all data input into and output from a version of a data model. For example, prediction data can include inputs such as training data sets that are ingested to generate and trigger outputs. Additionally or alternatively, prediction data can include outputs generated or created from a data model version, e.g., predictions made by that version of the data model using the inputs. Prediction data can also include metadata associated with the data model, metadata associated with a data version of the data model, metadata associated with sub-model inputs, and / or metadata associated with sub-model outputs. Prediction data can reference other outputs of the data model version.

[0012] As used herein, the terms "model" and "data model" are used interchangeably to refer to a machine learning / artificial intelligence type data model defined by algorithmic decision logic. A data model (and any version thereof) can include features such as decision logic, computational layers, neural networks, Markov chains, weighting algorithms (specific or non-specific to variables, values, layers, sub-models), and / or random forests. Although referred to in the singular, it will be understood that a data model (and any version thereof) can include multiple specific sub-models operating together in a specific sequence or in parallel that contribute to an output, such as a prediction.

[0013] As used herein, "version," "sub-model version," and "data model version" are used interchangeably to refer to a particular iteration of a data model that has a defined configuration for inputs, operations (e.g., of decision logic), and / or outputs that are specific or unique to that particular iteration.

[0014] As used herein, the terms "script" and "computer programming script" are used interchangeably to refer to computer readable and executable instructions / programming code that are expressions of instructions that cause, manage and facilitate the execution of a sequence of operational steps by a computer in an automated or semi-automated manner.

[0015] "Performance Measure" as used herein refers to a measurement obtained that represents and quantifies an aspect of the technical performance and predictive accuracy (or inaccuracy) of a model version and / or other behavior. Performance measures may include, for example, metrics, predictive accuracy, bias, data drift, noise, variance, etc. Examples of metrics include Measured Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), and / or Root Mean Squared Error (RMSE).

[0016] Embodiment Referring to FIG. 1, FIG. 1 is an example data flow diagram illustrating an example process 100 for rule-based aggregation of computed performance measure alerts for a model pipeline. Broadly, the process 100 includes one or more reporting jobs that compute performance measures for a model pipeline. The reporting jobs can include one or more program modules that use, among other things, alert parameters from a configuration file associated with the model pipeline. The alert parameters can set conditions for the reporting jobs to broadcast performance measure reports. For example, an alert module of the reporting job can use an alert frequency parameter to check a database for performance measure reports associated with the model pipeline and that have not been previously broadcast. In at least one embodiment, the alert module aggregates multiple performance measure reports into an aggregated alert message that is broadcast. Unlike conventional immediate alert procedures, aggregated alerts can validate model baselines with one or more previously generated performance measure reports. Such report broadcasting can help, among other things, to reduce unnecessary and / or unhelpful burdens of immediate alert procedures.

[0017] The process 100 may include configuring a reporting job based on one or more configuration files. The configuration files 104 may be used to configure a reporting job (e.g., 102 A , 102 B , 102 C , 102 D ) may be an object file that includes, among other things, metadata information, processing scripts, and output storage locations that facilitate performing operations that may be required to obtain / transform / merge / separate data identified in the output of the model pipeline 106. For example, the configuration file 104 may include query parameters (e.g., query_params of code 308 in FIG. 3A ) that point to the location of other objects that facilitate extracting data from the data output by the model pipeline 106. As another example, the configuration file may include data parameters (e.g., data_params of code 314 in FIG. 3A ) that point to objects that facilitate converting the format of the extracted data into a report format.

[0018] In some embodiments, the configuration file 104 includes an aggregated alert key 108 that points to one or more objects that hold parameters used by the alert module. The object can hold parameters in a data schema that facilitate configuration of the alert module for a reporting job. For example, the data schema can include parameters for, among other things, a frequency of change aggregation (e.g., a value, range, or algorithm that defines how many instances of a reporting job are executed before or during an alert aggregation evaluation), a location of one or more databases (e.g., database 204 of FIG. 2) that store performance measure reports associated with the model pipeline 106, and / or an identification of one or more tables in the database associated with the performance measure reports. With brief reference to FIG. 3A, example code 3200 illustrates a portion of code according to some embodiments of an alert schema. For example, code 3200 includes an alert_frequency parameter 316. As shown, the parameter 316 is defined to require three instances of execution of the reporting job before triggering an aggregated alert calculation operation. The parameter 316 can be checked by the reporting job in any number of ways. For example, the reporting job 102A can include a counter that incrementally increments between each execution instance. In another example, reporting job 102A can read the database location identified by databaseName parameter 318 and the table identified by alertTableName parameter 320 to calculate the number of performance measure reports (e.g., 134, 136, 138) stored in the database and / or database table.

[0019] Additionally, the data schema may include parameters for one or more performance measures. For example, the data schema may include parameters for each of one or more metrics 116, drift 118, violations 120, and / or bias 122 performance measures calculated by the reporting job 102. For each of the performance measures, the data schema may include one or more conditions that define rules for triggering alert aggregation. Returning briefly to FIG. 3A, example code 3200 includes example performance measure parameters for metric parameter 322 and condition parameters 324 and 326. As shown, metric parameter 322 defines the alert aggregation applied to the MAE and MAPE of the performance measure report generated by the reporting job. Additionally, metric parameter 322 also defines that at least three alert events must be present in the performance report to trigger delivery of an aggregated alert. The values ​​and parameters shown in code 3200 are merely examples and are not limiting. Other values, parameters, rules, and schemas may also be included.

[0020] As shown, the process 100 generates one or more reporting jobs (e.g., 102) that output performance measure reports (e.g., PM reports 134). A ) The PM report 134 may include the computational results of the reporting job. To facilitate the generation of the PM report 134, the reporting job loads the configuration file 104 and computes the model pipeline 106 and / or the corresponding sub-model(s) (e.g., sub-model A 132), the configuration file may include query parameters, data parameters, scripts, and performance measures that facilitate extracting data from the one or more outputs, calculating performance measures, and converting the format of the extracted data into a reporting format.

[0021] Bias measure 122 is generally a performance measure that computationally quantifies the weights and / or representations of data elements based on training data. Because biased data can lead to distorted prediction results, detecting bias at the sub-model level can facilitate identification and / or localization of some types of program and / or data errors in the model pipeline.

[0022] When calculating bias for a model or sub-model, predictions from the real-time data insight feature are analyzed with a bias baseline created on the training data. The baseline is generated using the training data that includes features and their predicted values. This helps generate results. The results can be a comprehensive report that describes the progression of feature-level bias of the data against the baseline over time.

[0023] In some embodiments, pre-training bias (evaluating features with actual labels) and post-training bias (evaluating features with actual and predicted label values) are supported. For example, when model pipeline data (actual values) are evaluated, the model monitoring system loads the pre-processed and baseline files into the system and runs the pre-processing algorithm to obtain model insight features and actual values. The bias and configured metrics can then be used to analyze the data in the baseline file to calculate any pre-training bias. In another example, when model pipeline data (actual values ​​and predicted values) are evaluated, the model monitoring system loads the pre-processed and baseline files into the system and runs the pre-processing algorithm to obtain model insight features and actual values. The bias and configured metrics can then be used to analyze the data in the baseline file to calculate any post-training bias.

[0024] Drift measure 118 is generally a performance metric that computationally quantifies the occurrence of unexpected or undocumented changes to the structure or semantics of data used and / or produced by a sub-model. Because drift events can corrupt predictive models or disrupt computational operations, detecting drift at the sub-model level can facilitate the identification and / or localization of some types of programmatic defects in the model pipeline.

[0025] In an example drift calculation, features of a model are analyzed using a baseline created from training data. In an exemplary drift calculation, statistics of the model data against the baseline data are obtained. The drift calculation can then identify any drift that exists for features in the model. Additionally, the drift calculation can operate to analyze multiple models with respective baselines for each model. In the case of multiple models, a baseline file can exist for each model, and features of each model can be mapped to the respective model baselines for each model.

[0026] Violations are measures of when thresholds and / or baselines are violated, such as the model version and features of the particular performance measure in which the violation occurred, the expected and / or target values ​​of the baseline and / or threshold that were violated, the value of the performance at which the violation was determined, etc. Additionally, features in the model or sub-model are analyzed with a baseline created from the training data. During analysis of the model, the occurrence of any new or missing features in the model data is detected. For any valid features in the model data, the data is analyzed with the baseline to detect any data type mismatches, positive, negative, and non-zero variations or violations.

[0027] For example, performance measures can include area under the curve (AUC), receiver operating characteristic (ROC), accuracy, variance, covariance, and / or other similar quantitative measures. Similarly, performance measures can include mean, median, r 2 , standard deviations, percentiles, and / or other similar quantitative measures may be included.

[0028] In general, the metric measures 116 include analytical comparisons of the predictive reliability of the sub-models. For example, the configuration file may include metrics of Measured Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), or Root Mean Squared Error (RMSE), or combinations thereof, for one or more sub-models. MAPE may generally be expressed in the following example, although other expressions of MAPE are intended to be within the scope of the aspects discussed herein.

[0029]

number

[0030] In general, the MAE may be expressed in the following examples, although other expressions of the MAE are intended to be within the scope of the embodiments discussed herein.

[0031]

number

[0032] RMSE is generally the standard deviation of the prediction errors. Thus, RMSE may be expressed in the following examples, although other expressions of RMSE are intended to be within the scope of the aspects discussed herein.

[0033]

number

[0034] In the above example representation, A refers to observed data and F refers to a prediction based on the input data. The configuration file 104 also configures the model pipeline 106 and / or each applicable sub-model (e.g., A 132). Generally, the one or more baseline values ​​define an expected value predicted for a variable or event by a model version based on specified and known inputs. Thus, the baseline "expected" value can be used to determine whether a model version would have produced the same or similar values ​​in its prediction for that variable. Thus, the baseline value can be used to determine whether a model version would have produced the same or similar values ​​in its prediction for the variable, compared to such expected values ​​to determine whether the model pipeline 106, a version of the model pipeline 106, one or more sub-models (e.g., sub-models A 132), and / or the predictive accuracy of one or more sub-model versions. The baseline threshold 112 can be a target value that is a customized and / or leading minimum or maximum value for measuring and evaluating performance measures, such as, for example, metrics (e.g., MAPE, MAE, and / or RMSE). The baseline value may further include a range, for example, to determine whether the model pipeline or sub-model generated predicted values ​​that are within a predefined buffer range of the baseline expectation value of the corresponding variable or not. Additionally or alternatively, the baseline threshold 112 can define values ​​for evaluating other performance measures, such as, for example, data drift, model bias, and / or noise. For each model, a comparison and determination of each performance measure evaluated in light of the baseline threshold is included in the performance reports 134, 136, and 138.

[0035] Process 100 also includes a reporting job (e.g., 102) that executes an alert module.A The alerts module can be configured by report job using the alerts key 108 as discussed above. For example, report job 102 A The reporting job 102 may use one or more frequency parameters (e.g., parameters 316 in FIG. 3) to check whether the conditions of the aggregated alerts are met. A As shown for A may generate a performance measure report 134, as discussed above. D If the frequency parameters are met (as in the case of ), the module triggers a reporting job to pull previous performance measure reports within the aggregate alert time frequency. In some embodiments, the reporting job validates the performance measures included in the summarized report's previous performance measure reports against the baseline 112 to generate the aggregate alert report 130. The aggregate alert report 130 can be broadcast over a network to one or more predefined locations or user accounts. For example, the example aggregate alert report 400 of FIG. 4 can be broadcast as an email, notification, or any other electronic message to a user account associated with a person tasked with maintaining the model pipeline 106.

[0036] Referring to Figure 2, Figure 2 illustrates an example system environment 200 according to an embodiment described herein. The block components of the system environment 200 include computing processes and / or components that may include any combination of hardware, firmware, and / or software. For example, the system environment 200 may include one or more of the devices described in connection with the computing environment 600 of Figure 6.

[0037] As shown in FIG. 2, the system environment 200 includes one or more databases of data clusters 202. The data clusters 202 can hold structured and unstructured data in one or more relational databases (e.g., database 204A or database 204B). The data can relate to any chaotic and / or dynamic system. For example, the data can include values, comments, records, etc. corresponding to multiple people. As people interact with one or more entities that maintain the data clusters 202, additional data can be changed in the database (e.g., added to the database, edited in the database, deleted, or manipulated in any other way). For example, the data can be changed continuously, intermittently, and / or periodically as patients associated with the data are treated, seen, discharged, or otherwise interact with the institution, care network, or region that maintains the data clusters 202. The data may be generated by sensors monitoring the patient (e.g., heart rate and / or weight), data about the patient (e.g., admission date and / or demographics), and / or observations about the patient (e.g., diagnosis of a disease or condition). A model pipeline (e.g., model pipeline 106) may use the data at an initial time point to predict outcomes at future time points. The data may then change as time progresses from the initial time point to future time points. Data held in data cluster 202 may be used at future time points to determine observed outcomes that correspond to the earlier predicted outcomes.

[0038] The system environment 200 also includes a modeling job 206 that includes the model pipeline 106 and one or more sub-models (e.g., sub-model A 132). The modeling job 206 can include programmatic representations that execute the model pipeline 106 continuously, intermittently, and / or periodically. The modeling job 206 can also include programmatic representations that generate one or more outputs corresponding to the model pipeline 106. For example, the modeling job 206 can output one or more insights 208 in a database. In some embodiments, the insights 208 are one or more object files (e.g., JavaScript Object Notation files) that capture metadata associated with the model pipeline 106, the one or more sub-models, and / or the observational data.

[0039] The system environment 200 also includes a reporting job 210. The reporting job 210 can include programmatic representations that calculate performance measure reports continuously, intermittently, and / or periodically. For example, the reporting job 210 can include, among others, a configuration module 212, an alert module 214, and a configuration file database 216. Some embodiments of the reporting job 210 can also include a calculation library 220 that maintains calculation rules and algorithms that can be invoked by one or more configuration files 104. These modules can facilitate aggregated alert notification of performance measures corresponding to the model pipeline. For example, the reporting job 210 can be programmed to launch a configuration model 212 in response to an output from the model pipeline 106. Once launched, the configuration module 212 can access the insights 208 and identify the configuration files 104 that are maintained in the configuration database 216. Although the database 216 is illustrated as part of the reporting job 210, in some embodiments, it can be part of the data cluster 202.

[0040] The configuration file 104 may be one or more object files (e.g., JavaScript Object Notation files). The configuration file 104 is used to configure the model pipeline 106, one or more sub-models (e.g., sub-models 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191 A ~102 D ), and / or object-oriented representations related to one or more performance measures 110. For example, the configuration file 104 may be a data schema 300 as shown in FIG. 3. The data schema 300 may include, or among other things, metadata information (e.g., code 302), performance measurements (e.g., code 304), and output storage locations (e.g., code 306) that facilitate the reporting job 210 to perform operations that may be required to obtain / transform / merge / separate data identified in the insights 208. As shown in FIG. 3A, the data schema 300 may include or call programmatic representations. For example, a metric measure may include code, such as code example 7012 and / or code example 708, including an MAE metric measure. Similarly, the data schema 300 may include one or more drift measures, which may include code, such as code example 7012 and / or code example 708. The data schema 700 may similarly include one or more violation measures and / or one or more bias measures. Additionally, the data schema 300 may include one or more alert keys. For example, data schema 300 may include a programmatic representation such as code example 3200. Alert keys may be configured as standalone representations in schema 300 or may be embedded in one or more performance measures, which may facilitate the incorporation of performance measure-specific aggregate alerting rules and / or job-specific aggregate alerting rules.

[0041] The system environment 200 also includes a network 218. The network 218 can include wireless and / or physical (e.g., hardwired) connections. Exemplary networks include a service provider or carrier telecommunications network, a wide area network (WAN), a local area network (LAN), a wireless local area network (WLAN), a cellular telecommunications network, a Wi-Fi network, a short range wireless network, a wireless metropolitan area network (WMAN), a Bluetooth®-enabled network, an optical fiber network, or combinations thereof. The network 218 generally provides components and devices with access to the Internet and web-based applications. Although not shown, the system environment can also include a reporting dashboard that facilitates interactive presentation of calculated performance measure reports.

[0042] Referring to FIG. 5, FIG. 5 illustrates a method 500 for computing a performance measure of a sub-model according to an embodiment described herein. Each block of the method 500 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in a memory. The method 500 can also be embodied as computer usable instructions stored on a computer storage medium. The method 500 can be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, the method 500 is described with respect to the process 100 of FIG. 1 and / or the system 200 of FIG. 2, by way of example. However, the method can additionally or alternatively be performed by any one system or combination of systems, including but not limited to those described herein.

[0043] The method 500 includes, at block 502, automatically initiating an executable reporting job to compute one or more performance measures for the predictive model. The reporting job may be initiated in response to the occurrence of an event programmatically coded into the script of one or more applications that maintain the reporting job. In some embodiments, the reporting job includes an operation to configure delivery of a computed performance measure alert output by the reporting job based on the predictive model. For example, reporting job 102 A may be initiated in response to detection of one or more outputs from the model pipeline 106.

[0044] The method 500 includes, at block 504, storing the one or more calculated performance measure alerts for the model in a database in response to execution of the reporting job operations without delivering a notification of the one or more calculated performance measure alerts. For example, reporting job 102 A can be configured to use a configuration file 104 to populate and calculate performance measures 110 based on the output from the model pipeline 106. A An alert module of the reporting job 102 (e.g., alert module 214 of FIG. 2) can execute the action defined by the alert key 108. If the conditions of the aggregated alert are not met, the reporting job 102 A completes the calculation and storage of the PM report 134. However, the reporting job 102 A does not deliver aggregated alert reports.

[0045] Method 500 includes, at block 506, automatically initiating an executable reporting job to calculate another one or more performance measures for the predictive model. Similar to block 502, the reporting job may be initiated in response to the occurrence of an event programmatically coded into the script of one or more applications that maintain the reporting job. In some embodiments, the reporting job includes an operation to configure delivery of a calculated performance measure alert output by the reporting job based on the predictive model. For example, reporting job 102 may be initiated in response to an event programmatically coded into the script of one or more applications that maintain the reporting job. D may be initiated in response to detection of one or more outputs from the model pipeline 106. The reporting job may be the same reporting job initiated in block 502, a different reporting job, or another instance of the same reporting job initiated in block 502. In other words, the reporting job 102 D 1) Report job 102 to be executed at a later time A 2) a separate reporting job that calculates other performance measures for the model pipeline 106; or 3) a reporting job 102 that runs in series or in parallel. A It may be another instance of

[0046] The method 500 includes, at block 508, storing the one or more other calculated performance measure alerts in a database and delivering a notification of both the one or more calculated performance measure alerts and the one or more other calculated performance measure alerts in response to execution of an operation of a reporting job. For example, reporting job 102 D can be configured to use the configuration file 104 to populate and calculate performance measures 110 based on the output from the model pipeline 106. D An alert module (e.g., alert module 214 of FIG. 2) of the report job 102 may execute the action defined by the alert key 108. When the conditions of an aggregated alert are met, the report job 102 Dcompletes the computation of the performance measure report of the model pipeline. In addition, the report job 102 D delivers aggregated alert reports 130 based on rules defined in other programs in the Alert Key 108.

[0047] In some embodiments of the method 500, the method may repeat blocks 502 and 504 until the conditions of the alert key for the reporting job are satisfied. The method 500 may then proceed to block 508.

[0048] Referring to FIG. 6, an exemplary computing environment according to an embodiment of the present invention is illustrated. Those skilled in the art will appreciate that the exemplary computing environment 600 is merely one example of a suitable computing environment and is not intended to limit the scope of use or functionality of the present invention. Likewise, the computing environment 600 should not be interpreted as imposing any dependency and / or any requirement with respect to each component and combination of components illustrated in FIG. 6. Those skilled in the art will appreciate that the connections illustrated in FIG. 6 are also exemplary, as other methods, hardware, software, and devices for establishing communication links between components, devices, systems, and entities as illustrated in FIG. 6 may be utilized in embodiments of the present invention. Although the connections are illustrated using one or more solid lines, those skilled in the art will appreciate that the exemplary connections in FIG. 6 may be hardwired or wireless and may use intermediate components that are omitted or not included in FIG. 6 for simplicity. Thus, the absence of components from FIG. 6 should not be interpreted as limiting the present invention to exclude additional components and combinations of components. Additionally, although devices and components are depicted in FIG. 6 as singular devices and components, it will be understood that some embodiments may include multiple devices and components, such that FIG. 6 should not be considered as limiting the number of devices or components.

[0049] Continuing, the computing environment 600 of FIG. 6 is illustrated as being a distributed environment in which components and devices may be remote from one another and may perform separate tasks. The components and devices may communicate with one another and may be linked to one another using a network 602. The network 602 may include wireless and / or physical (e.g., hardwired) connections. Exemplary networks include a service provider or carrier telecommunications network, a wide area network (WAN), a local area network (LAN), a wireless local area network (WLAN), a cellular telecommunications network, a Wi-Fi network, a short-range wireless network, a wireless metropolitan area network (WMAN), a Bluetooth®-enabled network, an optical fiber network, or a combination thereof. The network 602 generally provides the components and devices with access to the Internet and web-based applications.

[0050] The computing environment 600 includes a computing device in the form of a server 604. Although shown as one component in FIG. 6, the present invention may utilize multiple local and / or remote servers in the computing environment 600. The server 604 may include components such as a processing unit, an internal system memory, and a suitable system bus for coupling to various components including a database or database cluster. In some embodiments, the data cluster takes the form of a cloud-based data store, which in some embodiments is accessible by a cloud-based computing platform. The system bus may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus, also known as a mezzanine bus.

[0051] The server 604 may include or have access to computer readable media. The computer readable media may be any available media that can be accessed by the server 604, including volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer readable media may include computer storage media and communication media. Computer storage media may include, but is not limited to, volatile and non-volatile media, as well as removable and non-removable media implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules, or other data. In this regard, computer storage media may include, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage, or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the server 604. Computer storage media does not inherently include non-transitory signals.

[0052] Communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and may include any information delivery media. As used herein, the term "modulated data signal" refers to a signal that has one or more of its attributes set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency (RF), infrared and other wireless media. Combinations of any of the above may also be included within the scope of computer-readable media.

[0053] In an embodiment, the server 604 communicates with one or more remote computers 606 in the computing environment 600 using logical connections. In an embodiment in which the network 602 includes a wireless network, the server 604 may employ a modem to establish communication with the Internet, the server 604 may connect to the Internet using a Wi-Fi or wireless access point, or the server may use a wireless network adapter to access the Internet. The server 604 uses the network 602 to perform bidirectional communication with any or all of the components and devices depicted in FIG. 6. Thus, the server 604 may transmit data to and receive data from the remote computer 606 via the network 602.

[0054] Although shown as a single device, the remote computer 606 may include multiple computing devices. In embodiments having a distributed network, the remote computer 606 may be located in one or more different geographic locations. In embodiments in which the remote computer 606 is multiple computing devices, each of the multiple computing devices may be located across various locations, such as buildings in a campus, medical and research facilities in a medical complex, offices or "branch offices" of a bank / credit company, or may be mobile devices, for example, wearable or carried by personnel or attached to trackable items in a vehicle or warehouse.

[0055] In some embodiments, the remote computer 606 is physically located in a medical environment, such as, for example, a laboratory, an inpatient room, an outpatient room, a hospital, a medical vehicle, a veterinary environment, an outpatient environment, a medical billing office, a financial or administrative office, a hospital administration environment, a home medical environment, and / or a medical professional's office. By way of example, the medical personnel may include medical professionals such as physicians, surgeons, radiologists, cardiologists, and oncologists, paramedics, physician assistants, clinical nurse practitioners, nurses, nursing assistants, pharmacists, nutritionists, microbiologists, laboratory professionals, genetic counselors, researchers, veterinarians, students, etc. In other embodiments, the remote computer 606 may be physically located in a non-medical environment, such as a packing and shipping facility, or deployed within a fleet of delivery or courier vehicles.

[0056] Continuing, the computing environment 600 includes a data store 608. Although the data store 608 is shown as a single component, it may be implemented using multiple data stores communicatively coupled to one another regardless of the geographic or physical location of the memory devices. The exemplary data store may store data in the form of artifacts, server lists, properties associated with servers, environments, properties associated with environments, computer instructions coded in multiple different computer programming languages, deployment scripts, applications, properties associated with applications, release packages, version information for release packages, build levels associated with applications, identifiers of applications, identifiers of release packages, users, roles associated with users, permissions associated with roles, workflows and steps within workflows, clients, servers associated with clients, attributes associated with properties, audit information, and / or audit trails of workflows. The exemplary data store may also store data in the form of electronic records, such as, for example, electronic medical records of patients, transaction records, billing records, task and workflow records, time series event records, and the like.

[0057] Generally, the data store 608 includes physical memory configured to store information encoded with data. For example, the data store 608 can provide storage for computer-readable instructions, computer-executable instructions, data structures, data arrays, computer programs, applications, and other data that support functions and operations undertaken using the exemplary computing environment 600 and components shown in FIG.

[0058] In a computing environment having distributed components communicatively coupled via network 602, program modules may be located in local and / or remote computer storage media, including, by way of example only, memory storage devices. An embodiment of the invention may be described in the context of computer-executable instructions, such as program modules, being executed by a computing device. Program modules may include, but are not limited to, routines, programs, objects, components, and data structures that perform particular tasks or implement particular abstract data types. In an embodiment, server 604 may access, retrieve, communicate, receive, and update information stored in data store 608, including program modules. Thus, server 604 may use a processor to execute computer instructions stored in data store 608 to perform the embodiments described herein.

[0059] Although the internal components of the devices of Figure 6, such as server 604, are not shown, one skilled in the art would understand that the internal components and their interconnections are present in the devices of Figure 6. Accordingly, further details regarding the internal structural devices will not be disclosed further herein.

[0060] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, etc. The present disclosure may also be implemented in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0061] As used herein, the term "and / or" with respect to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, "element A, element B, and / or element C" can include element A only, element B only, element C only, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0062] The subject matter of the present disclosure is specifically described herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors contemplate that the claimed subject matter may be embodied in other ways, including different steps or combinations of steps similar to those described herein, together with other current or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to refer to different elements of the method employed, this term should not be interpreted as implying any particular order between the various steps disclosed herein, unless the order of the individual steps is explicitly described.

Claims

1. 1. A computer-implemented method for aggregating calculated performance measure alerts within an executable reporting job, the method comprising: automatically initiating the executable reporting job in response to the occurrence of an event programmatically coded in a script of one or more applications maintaining the reporting job to calculate one or more performance measures for a predictive model, the reporting job including an operation of configuring delivery of the calculated performance measure alerts output by the reporting job based on the predictive model; the method further comprising: in response to execution of the operations of the reporting job, storing the one or more calculated performance measure alerts for the predictive model in a database without delivering notifications of the one or more calculated performance measure alerts; automatically initiating the executable reporting job to calculate another one or more performance measures for the predictive model; and in response to execution of the operations of the reporting job, storing the another one or more calculated performance measure alerts in the database and delivering notifications of both the one or more calculated performance measure alerts and the another one or more calculated performance measure alerts.

2. The method of claim 1 , wherein the reporting job calculates a plurality of performance measures including one or more performances for the predictive model.

3. The method of claim 1 , wherein the one or more calculated performance measures include at least one measure of a model metric.

4. The method of claim 1 , wherein the one or more calculated performance measures include at least one measure of model drift.

5. The method of claim 1 , wherein the one or more calculated performance measures include at least one measure of model bias.

6. The method of claim 1 , wherein the one or more calculated performance measures include at least one measure of model violation.

7. The method of claim 1 , wherein the performance measure alert comprises one or more generated performance measure reports.

8. 1. A system for aggregating calculated performance measures of a predictive model into an alert notification, the system comprising: at least one processor; and a computer storage medium storing computer-readable instructions that, when executed by the at least one processor, cause the processor to initiate a reporting job, in response to an occurrence of an event programmatically coded in a script of one or more applications maintaining the reporting job, to calculate one or more performance measures for the predictive model, the reporting job including an operation of configuring delivery of the calculated performance measure alerts output by the reporting job based on the predictive model; store the one or more calculated performance measure alerts for the predictive model in a database without delivering a notification of the one or more calculated performance measure alerts in response to execution of the operations of the reporting job; automatically initiate the executable reporting job to calculate another one or more performance measures for the predictive model; and store the another one or more calculated performance measure alerts in the database and deliver a notification of both the one or more calculated performance measure alerts and the another one or more calculated performance measure alerts in response to execution of the operations of the reporting job.

9. The system of claim 8 , wherein the reporting job calculates a plurality of performance measures including one or more performances for the predictive model.

10. The system of claim 8 , wherein the performance measure alert comprises one or more generated performance measure reports.

11. The system of claim 8 , wherein the one or more calculated performance measures include at least one measure of a model metric.

12. The system of claim 8 , wherein the one or more calculated performance measures include at least one measure of model drift.

13. The system of claim 8 , wherein the one or more calculated performance measures include at least one measure of model bias.

14. The system of claim 8 , wherein the one or more calculated performance measures include at least one measure of model violation.

15. A program for causing one or more processors to execute the method according to any one of claims 1 to 7.

Images