Model validation based on sub-model performance

Abstract

Disclosed herein are methods, systems, and computer-readable media for a concurrent comparison tool for evaluating sub-models of a data model pipeline in a deployed or pre-deployment environment. The tool can calculate multiple performance measures that quantitatively evaluate the performance of each sub-model in a data model pipeline based on a configuration file that facilitates validation of the technical performance and predictive accuracy of the sub-model. Additionally, multiple versions of a sub-model deployed in a similar data model pipeline or pre-deployment environment can be comparatively evaluated. A primary version of the sub-model can be identified and deployed.

Description

[Technical field]

[0001] FIELD OF THEINVENTION This application relates to machine learning data models, and more particularly to the comparative evaluation of predictive accuracy and performance measures of different versions of machine learning data models. Summary of the Invention [Means for solving the problem]

[0002] Summary of the Invention Embodiments of the present disclosure relate, inter alia, to methods, systems, and computer-readable media for computationally evaluating sub-models of a machine learning data model (hereinafter "model") pipeline for technical performance and / or predictive accuracy. Some embodiments include deploying a version of the sub-model that exhibits improved technical performance and / or predictive accuracy compared to a currently deployed version of the sub-model. As described below, aspects of the invention monitor and comparatively evaluate technical performance and / or predictive accuracy by monitoring multiple sub-models of a model pipeline.

[0003] A first embodiment may include a computerized method for evaluating and improving sub-model performance and accuracy. The computerized method may include detecting an output from a model pipeline including at least two sub-models, the output of the model pipeline including one or more predictions calculated based on the at least two sub-models. The computerized method may also include obtaining one or more files including a formatted dataset corresponding to the model pipeline, the formatted data including data used to generate the predictions and the one or more predictions, and receiving a configuration file. Additionally, the computerized method may include calculating a plurality of sub-model performance reports from the plurality of datasets based on the configuration file, each of the at least two sub-model performance reports including a plurality of performance metrics determined for a corresponding version of the model.

[0004] Another embodiment may include a system for monitoring multiple sub-models of a model pipeline. The system may include a data model performance monitoring system via one or more processors executing a script. The script may include operations that facilitate receiving at least one of a data set corresponding to multiple sub-models of the model pipeline, the at least one data set including metadata corresponding to predictions and prediction data for each of the multiple sub-models of the model pipeline, performing operations defined by a configuration file corresponding to the model pipeline including instructions for computing multiple sub-model performance measures, and outputting the multiple sub-model performance measures as a report for each of the sub-models based on the configuration file.

[0005] Another embodiment may include a method for monitoring a plurality of models deployed in a model pipeline. The method may generate one or more predictions and one or more model functions using each model in the plurality of models, and identify a performance measurement threshold for each model in the plurality of models. Further, the method may generate a model report for each model in the model pipeline using the performance measurement threshold for each model in the plurality of models, and generate a health score for each model in the plurality of models based on the generated model report. Further, the method may generate a health score for the model pipeline based on the generated health scores.

[0006] This Summary is provided to introduce a selection of concepts in a simplified form that are more fully described in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The present invention is defined by the claims, which are supported by this specification including the Detailed Description.

[0007] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention systems and methods for learning feature vectors for single and cross-sensor object tracking are described in detail below with reference to the accompanying drawings. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a process diagram illustrating an example of calculating sub-model performance measures based on a model pipeline, according to some embodiments of the present disclosure. [Diagram 2] FIG. 13 is another process diagram illustrating an example of calculating sub-model performance measures based on a model pipeline, according to some embodiments of the present disclosure. [Diagram 3] FIG. 13 illustrates an example of an interface for displaying sub-model performance measures in accordance with some embodiments of the present disclosure. [Figure 4]FIG. 1 illustrates an example system environment that can facilitate calculation of sub-model performance measures in accordance with some embodiments of the present disclosure. [Diagram 5] 1 is a block diagram of an example method for calculating one or more sub-model performance measures according to some embodiments of the present disclosure. [Figure 6] FIG. 2 illustrates an example code snippet for sub-model insight, in accordance with some embodiments of the present disclosure. [Figure 7] FIG. 1 illustrates an example code schema for a configuration file to facilitate calculation of sub-model performance measures in accordance with some embodiments of the present disclosure. [Figure 7A] FIG. 13 illustrates an example code snippet of a configuration file for facilitating calculation of sub-model performance measures in accordance with some embodiments of the present disclosure. [Figure 8] FIG. 1 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Detailed Description of the Invention Broadly speaking, computer models may be trained to predict outcomes (e.g., calculate future events based on current data) and / or perform tasks (e.g., trigger some action by a downstream system or device based on data calculations). Often, an overall model includes multiple sub-models that perform predictions or tasks that contribute to an overall prediction or task output. Traditionally, once the sub-models are trained on training data, a model pipeline is constructed and deployed. Once deployed, the model pipeline may be cumulatively evaluated to verify that it is performing within desired performance measures. However, in contrast to traditional approaches of model pipeline performance detection, aspects described herein provide sub-model level quantitative and / or qualitative performance reports. Performance measures may be communicated for supervised evaluation and / or may be performed automatically by the described system.

[0010] Accordingly, the embodiments described herein facilitate the computation of individual sub-model performance measures (e.g., metrics, drift, bias, and / or violations) in a model pipeline that includes multiple sub-models. To accomplish this, the system uses a dataset (e.g., insights) to track the operation of the sub-models. In general, a data store captures the sub-model outputs (e.g., predictions) and the sub-model inputs (e.g., data used by the sub-models that correspond to the functions of the sub-models). The sub-model inputs may be captured in a pre-encoded format (e.g., before conversion to sub-model-readable data) and / or a post-encoded format (e.g., after conversion to sub-model-readable data). For example, the sub-model inputs may be captured before and / or after natural language processing, dimensionality reduction, vectorization, or other suitable data transformation that facilitates ingestion of the data by the sub-models. Similarly, the sub-model outputs may be captured in an encoded or decoded format. For example, the first sub-model (e.g., the sub-models of FIG. 1) may be captured in a decoded format. A104) outputs one or more second models (e.g., the sub-models in FIG. 1 B 106), the data store can capture data that is output by the second sub-model and / or configured for input. In at least one embodiment, the sub-model data is captured in an object-oriented file (e.g., an insight file) in the form of metadata within a programmatically defined schema. The schema can facilitate identification.

[0011] The system can be configured to continuously, periodically, intermittently, and / or on-demand calculate one or more performance measures for the sub-models of the model pipeline. In some embodiments, the system can monitor where the output of the model pipeline is stored and / or monitor systems downstream of the model pipeline that act on the output. For example, in response to storage of the output from the model pipeline (e.g., model pipeline prediction 110 of FIG. 1), the system can extract data associated with one or more sub-models that are held in a data store. An automatically executed pre-processing script can load one or more configuration files and one or more computational dictionaries into a reporting component of the system. The reporting component transforms the data stored in the data store based on the one or more configuration files. Additionally, the script can take all input data and separate the data based on the sub-models.

[0012] definition As used herein, the terms "observed data," "truth," "actual values," and "goals" are used interchangeably to refer to empirical data and / or observed real-world information encoded as data. For example, observed data includes measured, captured, or recorded values ​​that represent and / or quantify an event or variable that has occurred. In one example, observed data includes values ​​for the total number of patients for a particular medical entity that occurred during a defined six-month period as recorded in the medical entity's historical report data.

[0013] The term "predictive data" as used herein refers to all data input into and output from a version of a data model. For example, predictive data may include inputs, such as training data sets, that are ingested to generate and trigger outputs. Additionally or alternatively, predictive data may include outputs generated or derived from a data model version, such as predictions made by a version of a data model using inputs. Predictive data may also include metadata associated with the data model, metadata associated with a data version of the data model, metadata associated with sub-model inputs, and / or metadata associated with sub-model outputs. Predictive data may reference other outputs of a data model version.

[0014] As used herein, the terms "model" and "data model" are used interchangeably to refer to a machine learning / artificial intelligence type data model defined by an algorithmic decision logic. A data model (and any version thereof) may include features such as decision logic, computational layers, neural networks, Markov chains, weighting algorithms (specific or non-specific to variables, values, layers, sub-models), and / or random forests. Although referred to in the singular, it will be understood that a data model (and any version thereof) may include multiple specific sub-models operating together or in parallel in a specific order that contribute to an output, such as a prediction.

[0015] As used herein, “version,” “sub-model version,” and “data model version” are used interchangeably to refer to a particular iteration of a data model, with configuration defined for inputs, operations (e.g., decision logic), and / or outputs specific or unique to that particular iteration.

[0016] As used herein, the terms "script" and "computer programming script" are used interchangeably to refer to computer readable and executable instructions / programming code that is a formula of instructions that automatically or semi-automatically executes, manages and facilitates the execution of a series of computational procedures by a computer.

[0017] As used herein, a "performance measure" refers to a captured measurement that represents and quantifies an aspect of the technical performance and predictive accuracy (or inaccuracy) of a model version and / or other operation. Performance measures may include, for example, metrics, predictive accuracy, bias, data drift, noise, variance, etc. Examples of metrics include Measurement Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), and / or Root Mean Square Error (RMSE).

[0018] Embodiment Referring to FIG. 1, FIG. 1 is an exemplary data flow diagram illustrating an exemplary process 100 for computing performance measures of one or more sub-models of a model pipeline. In general, the process 100 may include a model pipeline 102 that generates model pipeline predictions 110. The model pipeline 102 may receive one or more inputs from one or more data sources (e.g., data cluster 402 of FIG. 4) to compute the model pipeline predictions 110. The data cluster 402 may be a database 152 or multiple databases 152 and 404 that store data facilitating chaotic and / or dynamic systems. For example, medical, meteorological, etc. In other words, the data may change continuously, intermittently, and / or periodically as a patient associated with the data is treated, examined, discharged, or interacts with the institution, care network, or region that maintains the database. The data may be generated by sensors monitoring the patient (e.g., heart rate and / or weight), data about the patient (e.g., admission date and / or demographics), and / or observations about the patient (e.g., diagnosis of a disease or condition).

[0019] The process 100 may include an output or capture of insights 112 that includes data records of all the data used to generate predictions for the sub-models (e.g., 114, 118, and 122) and the predictions themselves (e.g., 116, 120, and 124). The insights 112 may format the data used to generate predictions into a data schema. For example, the model pipeline 102 may include outputs (e.g., A One or more machine learning models (e.g., sub-models) of the model pipeline 102 that generate predictions 116 A 104) may include sub-models (e.g., A 104, sub-model B 106, sub-model N 108) may use different inputs from different data sources. For example, submodels A 104 is a sub-model A Data corresponding to the fields of the function 114 may be used. B 106 is a sub-model B The data corresponding to the fields of the function 118 may be used to N 108 is a sub-model N Data corresponding to the fields of the features 122 may be used. The insight data schema may include sub-model execution metadata (e.g., a timestamp indicating when one or more sub-models are executed, the version of one or more sub-models, and data indicating the ID of the model pipeline associated with one or more sub-models).

[0020] The process 100 may also include a report job 134 that receives the configuration file 126 and the insights 112. Generally, the configuration file includes metadata information required for the monitoring job and processing scripts for obtaining data and storing the data in a manner that can be easily queried for calculation of performance measures. The configuration file 126 may include one or more scripts that facilitate the execution of operations that may be required to obtain / transform / combine / separate data identified in the insights 112. For example, the configuration file 126 may include query parameters (e.g., query_params of code 708 of FIG. 7A) that point to the location of other objects that facilitate extracting data from the insights 112. Similarly, the configuration file 126 may include data parameters (e.g., data_params of code 712 of FIG. 7A) that point to objects that facilitate converting the format of the extracted data into a report format. The configuration file 126 may also include one or more scripts that facilitate the calculation of performance measures for one or more sub-models. The performance measures may include metric measures, bias measures, drift measures, and / or violation measures, etc.

[0021] Bias measures are generally performance measures that computationally quantify the weights and / or representations of data elements based on training data. Because biased data can lead to skewed prediction outcomes, detecting bias at the sub-model level can facilitate the identification and / or localization of certain types of program and / or data errors within a model pipeline.

[0022] In calculating the bias of a model version, predictions from real-time data for insight features are analyzed using one or more baselines. The baselines may be generated from training data that includes features and predicted values. This helps generate results. The results may be a comprehensive report that describes the progression of feature-level bias of the data against the baselines over time.

[0023] Pre- (evaluate features using labels for actual values) and post- (evaluate features using labels for actual values ​​and predicted values) training biases are supported. For example, when model pipeline data (actual values) are evaluated, the model monitoring system loads the pre-processed file and baseline file into the system and runs the pre-processing algorithm to obtain model insight features and actual values. Then, the data is analyzed in the baseline file using bias and configured metrics to calculate pre-training biases. In another example, when model pipeline data (actual values ​​and predicted) are evaluated, the model monitoring system loads the pre-processed file and baseline file into the system and runs the pre-processing algorithm to obtain model insight features and actual values. Then, the data is analyzed in the baseline file using bias and configured metrics to calculate post-training biases.

[0024] A drift measurement is generally a performance metric that computationally quantifies the occurrence of unexpected or undocumented changes to the structure or semantics of data used and / or produced by a submodel. Because drift events can corrupt predictive models or interrupt computational operations, detecting drift at the submodel level can facilitate the identification and / or localization of certain types of programmatic defects within a model pipeline.

[0025] In an example of drift calculation, model functions are analyzed using a baseline created from training data. In an exemplary drift calculation, statistics of model data against baseline data are obtained. Drift calculation can then identify drift present in functions within the model. Furthermore, drift calculation can operate to analyze multiple models using respective baselines for each model. In the case of multiple models, a baseline file can exist for each model, and each model's functions can be mapped to the respective model baselines for each model.

[0026] Violations are measured when thresholds and / or baselines are violated, e.g., features and specific performance measures of the model version where the violation occurred, expected and / or target values ​​of the baseline and / or threshold that were violated, and the value of the performance at which the violation was determined. Additionally, features in the model or sub-model are analyzed using a baseline that is created from the training data. During the analysis of the model, the emergence of new features and missing features in the model data are captured. For valid features in the model data, the data is analyzed using the baseline to capture data type mismatches, positive, negative, and non-zero variations and violations.

[0027] Generally, the metric measurements include analytical comparisons of the predictive reliability of the sub-models. For example, as shown in FIG. 7A, the configuration file may include metrics of Measurement Absolute Percentage Error (MAPE), Mean Absolute Error (MAE), or Root Mean Square Error (RMSE), or combinations thereof, for one or more sub-models. MAPE is generally expressed in the following example, although other expressions of MAPE are contemplated to be within the scope of the aspects described herein.

[0028]

number

[0029] Generally, the MAE is expressed as in the following example, although other formulas for the MAE are contemplated to be within the scope of the aspects described herein.

[0030]

number

[0031] The RMSE is generally the standard deviation of the prediction errors. Thus, the RMSE can be expressed as in the following example, although other expressions for RMSE are contemplated within the scope of the aspects described herein:

[0032]

number

[0033] In the above example formula, A refers to observed data and F refers to prediction based on input data. The configuration file 126 may also include baseline thresholds associated with performance measures for each applicable sub-model. For example, as shown in FIG. 7A, the configuration file may include a maximum threshold for the MAE value for the inpatient sub-model.

[0034] The report job 134 generates one or more sub-model performance measurement reports (e.g., sub-model performance reports) based on the insights 112 and the calculation libraries and data formatting instructions contained in the configuration file 126. A Performance measurement reports 136) can be calculated. For example, by applying calculation libraries and data formatting instructions, A Based on the predictions 116 and the corresponding observational data from the data source 152, the sub-model A The MAPE, MAE, RMSE, or other metric measure applicable to 104 may be calculated.

[0035] The process 100 includes one or more sub-model performance measurement reports (e.g., A Performance Measure Report 136) to submodel scores (e.g., submodel AThe sub-model score may also include converting the calculated performance measures to a corresponding baseline threshold. The sub-model score may be calculated at two levels. The first level may include comparing the calculated performance measures to corresponding baseline thresholds. In some embodiments, a sub-model may "pass" the first level of score if the values ​​calculated for the performance metrics are below a maximum threshold and / or above a minimum threshold. Similarly, a sub-model may "fail" the first level of score if the values ​​calculated for the performance metrics are above a maximum threshold or below a minimum threshold. Additionally, a sub-model may be classified as "at risk" for a first level of score if the values ​​calculated for the performance measures are within a predetermined value of the baseline threshold. For example, as shown in FIG. 7A, a configuration file associated with the inpatient sub-model may include exemplary code 7010 for setting a maximum threshold value for the MAE value for the inpatient sub-model. The inpatient sub-model may pass the first level of score for MAE if the value calculated for MAE by the report job 134 is below the MAE maximum baseline threshold. The configuration file may also include example code 7012 and 7014 for setting drift measurement parameters and processing.

[0036] A second level of sub-model scores may include a quantitatively calculated representation of the sub-models in the model pipeline (e.g., multi-model health score 150). For example, the second level score may include a total number of passing performance measures compared to the total number of performance measures calculated for the sub-model. If the percentage of passing performance measures is above a predetermined value, the sub-model may pass the second level score. In some embodiments, if the total number of passing performance measures is below a predetermined value but above a secondary value, the sub-model may be identified as "at risk". Similarly, if the total number of passing performance measures is below a secondary value, the sub-model may be identified as "failing". In some embodiments, a second level of sub-model scores for one or more performance measure classes (e.g., metric measures, bias measures, drift measures, violation measures) may be calculated.

[0037] In some embodiments of the process 100, the sub-model scores may be viewable through an interface of the reporting dashboard module 142 (e.g., interface 300 of FIG. 3). The interface may present an ordered visual representation of the first level and / or second level sub-model health scores. For example, A Health score 144, submodel B Health Score 146, and / or submodels N Health scores 148 may be summarized for each class of performance measure (e.g., metric, drift, bias, violation) in the first portion of the interface 300. As another example, individualized sub-models A Health score 144, submodel B Health Score 146, and / or submodels N The health score 148 may be accessible through interaction with elements of the interface 300 .

[0038] Additionally or alternatively, some embodiments of the process 100 include automatically replacing sub-models with different versions of the sub-models based on the relative health scores of each sub-model. The monitoring dashboard module 142 can determine a primary version of one of the sub-models based on performance measurement data from the performance reports and / or health scores. For example, different versions of the same sub-model (e.g., sub-model 142) incorporated into a desperate model pipeline can be automatically replaced with a different version of the sub-model based on the relative health scores of each sub-model. A The first version and submodels of AThe scores of the first and second versions of the sub-model (e.g., the first version of the sub-model, the second version of the sub-model, and the second version of the sub-model) can be compared via the monitoring dashboard module 142. The major version of the sub-model can be identified and selected autonomously, such as when the major version shows improved performance measures and / or superior predictive accuracy compared to at least one other version of the sub-model given a system-driven comparison of various performance measures in the performance scores of the sub-model. Alternatively, the major version can be selected semi-autonomously in response to user input, such as a user manual selection of the major version from a list and / or a user confirmation of the system's recommended major version. The system can then perform and / or trigger deployment of the major sub-model version.

[0039] Referring to FIG. 2, FIG. 2 illustrates an example of a predictive model process 200 including an admissions prediction model pipeline 202 according to embodiments described herein. The admissions prediction model pipeline 202 can use data from the data cluster 402 of FIG. 4 to calculate a prediction of the total number of patients admitted to a hospital for the next 12 hours, 24 hours, 36 hours, and 72 hours. To facilitate the overall prediction, the admissions prediction model pipeline 202 can include one or more emergency prediction sub-models 204 and one or more inpatient prediction sub-models 206, etc. The one or more emergency prediction sub-models 204 can use multiple fields in one or more databases of the data cluster 402 of FIG. 4 to predict emergency admissions for the next 12 hours, 24 hours, 36 hours, and 72 hours. Similarly, the one or more inpatient prediction sub-models 206 can use multiple fields in one or more databases of the data cluster 402 of FIG. 4 to predict non-emergency admissions for the next 12 hours, 24 hours, 36 hours, and 72 hours.

[0040] As shown, the process 200 includes generating insights 208 as an output of the model pipeline 202. The insights 208 may be object files (e.g., JavaScript object notation files) that capture metadata about the model pipeline 202 and the emergency sub-model 204 and the hospitalization prediction sub-model 206. For example, the insights 208 may capture factors 210 and values ​​212. The factors 210 may be programmatic expressions in an object file that identify metadata representing the ID of the model pipeline 202, the ID of the sub-model, the time the model is started, and the output of the sub-model, etc. The values ​​212 may be metadata representing features (e.g., other object files, database fields) used by the sub-model, properties of the features, etc. In some embodiments, the insights 208 or another insight may capture metadata representing data similar to that described above for the observational data corresponding to the sub-model output.

[0041] The process 200 includes a report job 222 that generates performance measure reports (e.g., emergency sub-model performance report 224 and non-emergency sub-model performance report 226) based on the calculation dictionary and performance measures in the configuration file 214. The configuration file 214 may be an object file that includes metadata information, processing scripts, output storage locations, etc. that facilitate the report job 222 to perform operations that may be required to obtain / transform / combine / separate data identified in the insights 208. For example, the configuration file may include query parameters (e.g., query_params in code 708 in FIG. 7A ) that point to other object locations that facilitate extracting data from the insights 208. As another example, the configuration file may include data parameters (e.g., data_params in code 714 in FIG. 7A ) that point to objects that facilitate converting the format of the extracted data into a report format. The configuration file 214 may also include one or more scripts that facilitate the calculation of performance measures for one or more sub-models.

[0042] As shown, generating a performance measure report may also include a report job that ingests observed data 220. The observed data 220 may be data stored in the data clusters 402 that represent observations that correspond to predictions. Thus, the observed data 220 may be observed emergency and non-emergency admissions for the same time period that correspond to the emergency sub-model 204 and the inpatient prediction sub-model 206.

[0043] Referring to Figure 4, Figure 4 illustrates an exemplary system environment 400 in accordance with embodiments described herein. The block components of the system environment 400 include computing processes and / or components that may include any combination of hardware, firmware, and / or software. For example, the system environment 400 may include one or more of the devices described in connection with the computing environment 800 of Figure 8.

[0044] As shown in FIG. 4, the system environment 400 includes one or more databases of data clusters 402. The data clusters 402 can maintain structured and unstructured data in one or more relational databases (e.g., database 152 and database 404). The data can relate to any chaotic and / or dynamic system. For example, the data can include values, comments, records, etc. corresponding to multiple people. As people interact with one or more entities maintaining the data clusters 402, additional data in databases 152 and / or database 404 can be changed (e.g., added, edited, deleted, or otherwise manipulated). For example, the data can be changed continuously, intermittently, and / or periodically as a patient associated with the data is treated, seen, discharged, or otherwise interacts with the institution, care network, or region maintaining the data clusters 402. The data can be generated by sensors monitoring the patient (e.g., heart rate and / or weight), data about the patient (e.g., admission date and / or demographics), and / or observations about the patient (e.g., diagnosis of a disease or condition). The data for the first time point may be used by a model pipeline (e.g., model pipeline 102) to predict outcomes for future time points. The data may then be modified as time progresses from the first time point to future time points. The data held in data cluster 402 may be used at future time points to determine observed outcomes that correspond to previously predicted outcomes.

[0045] The system environment 400 also includes a modeling job 406 that includes the model pipeline 102 and one or more sub-models. The modeling job 406 may include program expressions that execute the model pipeline 102 continuously, intermittently, and / or periodically. The modeling job 406 may also include program expressions that generate one or more outputs corresponding to the model pipeline 102. For example, the modeling job 406 may output one or more insights 112 to a database. In some embodiments, the insights 112 are one or more object files (e.g., JavaScript object notation files) that capture metadata associated with the model pipeline 102, one or more sub-models, and / or observational data. For example, the insights 112 may include expressions similar to the observed insight expressions 602, the insight factor expressions 604, and / or the insight value expressions 606, as shown in FIG. 6. Notably, expressions 602, 604, and 606 are merely examples of insight expressions and are not limiting.

[0046] The system environment 400 also includes a reporting job 134 that includes a configuration module 408 and an execution module 410. The reporting job 408 may include program expressions that continuously, intermittently, and / or periodically calculate the sub-model reports. For example, the execution module 410 may be programmed to activate the configuration model 408 in response to an output from the model pipeline 102. Upon activation, the configuration model 408 may access the insight 112 and identify the configuration file 126 that is maintained in the database. In some embodiments, the configuration file 126 is one or more object files (e.g., JavaScript object notation files) that include object-oriented expressions related to the model pipeline 102, one or more sub-models, and / or one or more performance measures. For example, the configuration file 126 may be in a data schema 700 as shown in FIG. 7. The data schema 700 may include metadata information (e.g., code 702), performance measurements (e.g., code 704), and output storage locations (e.g., code 706), etc., to facilitate the report job 134 to perform operations that may be necessary to obtain / transform / combine / separate data identified in the insights 112. As shown in FIG. 7A, the data schema 700 may include pre-configured performance measurements. For example, the data schema 700 may include one or more metric measurements (e.g., code 710), which may include code, such as exemplary code 7010, which includes an MAE metric measurement. Similarly, the data schema 700 may include one or more drift measurements, which may include code, such as exemplary code 7012 and / or exemplary code 708. The data schema 700 may similarly include one or more violation measurements and / or one or more bias measurements. The report job 134 may also include a calculation library 412 that maintains calculation rules and algorithms that may be invoked by one or more configuration files 126.

[0047] The system environment 400 also includes a reporting dashboard 142 that facilitates the calculation of the sub-model health scores. The reporting dashboard 142 may include a user interface 414 that may facilitate an interactive presentation of the calculated sub-model health scores (e.g., the user interface 300). The reporting dashboard 142 may also include a health score calculation module 416. The health score calculation module 416 may include rules that facilitate qualitative and / or quantitative evaluation of the sub-models. For example, a sub-model may "pass" a first level of score if the value calculated for the performance metric is below a maximum threshold and / or above a minimum threshold. Similarly, a sub-model may "fail" a first level of score if the value calculated for the performance metric is above a maximum threshold or below a minimum threshold. Additionally, a sub-model may be classified as "at risk" for a first level of score if the value calculated for the performance metric is within a predetermined value of a baseline threshold.

[0048] The system environment 400 also includes a network 418. The network 418 may include wireless and / or physical connections (e.g., wired connections). Exemplary networks include a service provider's or carrier's communications network, a wide area network (WAN), a local area network (LAN), a wireless local area network (WLAN), a cellular communications network, a Wi-Fi network, a short-range wireless network, a wireless metropolitan area network (WMAN), a Bluetooth®-enabled network, a fiber optic network, or a combination thereof. The network 418 generally provides components and devices with access to the Internet and web-based applications.

[0049] Referring to FIG. 5, FIG. 5 illustrates a method 500 for calculating a performance measure of a sub-model according to an embodiment described herein. Each block of the method 500 described herein includes a computing process that can be performed using any combination of hardware, firmware, and / or software. For example, a processor may execute instructions stored in a memory to perform various functions. The method 500 may also be implemented as computer usable instructions stored in a computer storage medium. The method 500 may be provided by a standalone application, a service or a host service (standalone or in combination with another host service), a plug-in to another product, or the like. Furthermore, the method 500 is described with respect to the process 100 of FIG. 1 or the process 200 of FIG. 2 as an example. However, the method may additionally or alternatively be performed by any one system or combination of systems, including but not limited to those described herein.

[0050] Block 502 of the method 500 includes detecting an output of a model pipeline including one or more sub-models. For example, the model pipeline 102 may output model pipeline predictions 110. Additionally, the model pipeline 102 may output model insights 112. A reporting job 134 may detect the output of the model pipeline predictions 110 or the model insights 112. As another example, the admissions model pipeline 202 may output insights 208. A reporting job 222 may detect the addition of the insights 208 to a database or other data storage medium.

[0051] Block 504 of the method 500 includes obtaining a dataset corresponding to the model pipeline. For example, in response to detecting an output from the model pipeline 102, the reporting job 134 may obtain the insight 112. As another example, the reporting job 222 may obtain the insight 208.

[0052] At block 506, the method 500 includes configuring a reporting module based on a configuration file corresponding to one or more sub-models. For example, the reporting job 134 can read the configuration file 126 to facilitate configuration of calculation libraries and scripting of data processing procedures of the insight 112. As another example, the reporting job 222 can read the configuration file 214 to facilitate configuration of one or more emergency forecasting sub-model performance measure calculation libraries and scripts 216 and one or more hospitalized patient forecasting sub-model performance measure calculation libraries and scripts 218.

[0053] Block 508 of the method 500 includes calculating performance measures for one or more sub-models based on the configured reporting modules. For example, the reporting job 134 may include: A Performance measurements 128, submodels B Performance measurements 130 and / or submodels N One or more scripts in the configuration files 126 may be executed to calculate the performance measures 132. As another example, the reporting job 222 may use scripts in the configuration files 214 to generate the emergency sub-model performance report 224 and / or the non-emergency sub-model performance report 226.

[0054] Some embodiments of the method 500 may also include generating a sub-model health score based on the calculated performance measures. For example, the reporting dashboard 142 may generate a sub-model health score based on the calculated performance measures. A Using data from Performance Measurement Report 136, the submodel A A health score 144 may be calculated. The report dashboard 142 may B Using data from Performance Measurement Report 138, sub-model B A health score 146 can be calculated. The report dashboard 142 can N Using the data from the performance measurement report 140, the submodel NA health score 148 can be calculated. Further, for example, the report dashboard 142 can A Health score 144, submodel B Health Score 146, and / or submodels N The health score 148 data may be used to calculate a multi-model health score 150.

[0055] Some embodiments of the method 500 may also include a comparative analysis of sub-model scores across multiple versions of the sub-model. A Health score 144, submodel B Health Score 146, and / or submodels N Health score 148, sub-model A , submodel B , and / or submodels N , and compare the health scores associated with other versions of the sub-model. If the primary version of the sub-model is not included in the model pipeline to be deployed, some embodiments of method 500 may replace the deployed sub-model with the primary version of the sub-model. The primary model may be deployed because it shows improved technical performance measures and / or improved predictive accuracy compared to another model version (e.g., the data model version currently in use), and therefore the primary sub-model shows improved technical behavior and performance. Thus, the newly deployed primary sub-data model version replaces another sub-model version that is not performing as well. Additionally or alternatively, the primary version may be used as an input to retrain the corresponding data model to generate additional updated versions of the data model.

[0056] Beginning with FIG. 8, an exemplary computing environment is shown in accordance with an embodiment of the present invention. Those skilled in the art will appreciate that the exemplary computing environment 800 is merely one example of a suitable computing environment and is not intended to limit the scope of use or functionality of the present invention. Likewise, the computing environment 800 should not be interpreted as imposing any dependencies and / or any requirements with respect to each component and combination of components illustrated in FIG. 8. The connections shown in FIG. 8 are also exemplary, and those skilled in the art will appreciate that other methods, hardware, software, and apparatus for establishing communication links between components, devices, systems, and entities as illustrated in FIG. 8 may also be utilized to implement the present invention. Although the connections are illustrated using one or more solid lines, those skilled in the art will appreciate that the exemplary connections in FIG. 8 may be wired or wireless and may employ intermediate components that are omitted or not included in FIG. 8 for simplicity. Thus, the absence of a component in FIG. 8 should not be interpreted as limiting the present invention to exclude additional components and combinations of components. Additionally, although devices and components are depicted in FIG. 8 as single devices and components, it will be understood that some embodiments may include multiple devices and components, and thus FIG. 8 is not intended to limit the number of devices or components.

[0057] 8 is illustrated as a distributed environment in which components and devices are remote from each other and can perform separate tasks. The components and devices communicate with each other and may be linked to each other using a network 802. The network 802 may include wireless and / or physical connections (e.g., wired connections). Exemplary networks include a service provider or carrier communication network, a wide area network (WAN), a local area network (LAN), a wireless local area network (WLAN), a cellular communication network, a Wi-Fi network, a short-range wireless network, a wireless metropolitan area network (WMAN), a Bluetooth®-enabled network, a fiber optic network, or a combination thereof. The network 802 generally provides the components and devices with access to the Internet and web-based applications.

[0058] The computing environment 800 includes a computing device in the form of a server 804. Although illustrated as one component in FIG. 8, the present invention may utilize multiple local and / or remote servers within the computing environment 800. The server 804 may include components such as a processing unit, an internal system memory, and an appropriate system bus for coupling to various components including a database or database cluster. In some embodiments, the data cluster takes the form of a cloud-based data store, which in some embodiments is accessible by a cloud-based computing platform. The system bus may be any of several types of bus structures, such as a memory bus or memory controller, a peripheral bus, a local bus, etc., using various bus architectures. By way of example and without limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA®) local bus, and a Peripheral Component Interconnect (PCI) bus, also known as a mezzanine bus.

[0059] The server 804 may include or have access to computer readable media. The computer readable media may be any available media accessible by the server 804, including volatile and non-volatile media, and removable and non-removable media. By way of example and not limitation, the computer readable media may include computer storage media and communication media. The computer storage media may include, but is not limited to, volatile and non-volatile media, as well as removable and non-removable media, implemented in any method or technology for storing information, such as computer readable instructions, data structures, program modules, or other data. In this regard, the computer storage media may include, but is not limited to, random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that may be used to store the required information and that may be accessed by the server 804. The computer storage media itself does not inherently include transitory signals.

[0060] Communication media typically embodied computer-readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and may include any information delivery media. As used herein, the term "modulated data signal" refers to a signal that has one or more attributes set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared and other wireless media. Combinations of any of the above may also be included within the scope of computer-readable media.

[0061] In an embodiment, the server 804 uses logical connections to communicate with one or more remote computers 806 in the computing environment 800. In an embodiment in which the network 802 includes a wireless network, the server 804 may establish communication with the Internet using a modem, the server 804 may connect to the Internet using a Wi-Fi or wireless access point, or the server may access the Internet using a wireless network adapter. The server 804 uses the network 802 to communicate bidirectionally with some or all of the components and devices depicted in FIG. 8. Thus, the server 804 may transmit data to and receive data from the remote computer 806 over the network 802.

[0062] Although remote computer 806 is shown as a single device, multiple computing devices may be included. In embodiments with a distributed network, remote computer 806 may be located in one or more different geographic locations. In embodiments where remote computer 806 is multiple computing devices, each of the multiple computing devices may be located in various locations, such as buildings within a campus, medical and research facilities within a medical facility, offices or "branch offices" of a bank / credit entity, and may also be mobile devices worn or carried by a person, or attached to a trackable item in a vehicle or warehouse, for example.

[0063] In some embodiments, the remote computer 806 is physically located in a medical environment, such as, for example, a laboratory, an inpatient room, an outpatient room, a hospital, a medical vehicle, a veterinary environment, an outpatient environment, a medical billing office, a financial or administrative office, a hospital administration environment, a home medical environment, and / or a medical professional's office. By way of example, the medical professional may include medical professionals such as physicians, surgeons, radiologists, cardiologists, oncologists, etc., paramedics, physician assistants, clinical nurse practitioners, nurses, nurse assistants, pharmacists, nutritionists, microbiologists, laboratory specialists, genetic counselors, researchers, veterinarians, students, etc. In other embodiments, the remote computer 806 may be physically located in a non-medical environment, such as a packing and shipping facility, or deployed within a fleet of delivery or courier vehicles.

[0064] Continuing, the computing environment 800 includes a data store 808. Although the data store 808 is shown as a single component, it may also be implemented using multiple data stores communicatively coupled to each other regardless of the geographic or physical location of the memory devices. The exemplary data store may store data in the form of deliverables, server lists, properties associated with servers, environments, properties associated with environments, computer instructions encoded in multiple different computer programming languages, deployment scripts, applications, properties associated with applications, release packages, version information for release packages, build levels associated with applications, application identifiers, release package identifiers, users, roles associated with users, permissions associated with roles, workflows and workflow steps, clients, servers associated with clients, attributes associated with properties, audit information, and / or audit trails for workflows. The exemplary data store may also store data in the form of electronic records, such as, for example, electronic patient medical records, transaction records, billing records, task and workflow records, time series event records, and the like.

[0065] Generally, data store 808 includes physical memory configured to store information encoded in data. For example, data store 808 can provide storage of computer-readable instructions, computer-executable instructions, data structures, data arrays, computer programs, applications, and other data that supports functions and actions performed using computing environment 800 and the components illustrated in exemplary FIG.

[0066] In a computing environment with distributed components communicatively coupled via network 802, program modules may be located in local and / or remote computer storage media, including, for example, only memory storage devices. An embodiment of the invention may be described in the context of computer-executable instructions, such as program modules, executed by a computing device. Program modules may include, but are not limited to, routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. In an embodiment, server 804 may access, retrieve, communicate, receive, and update information stored in data store 808, including program modules. Thus, server 804 may use a processor to execute computer instructions stored in data store 808 to perform the embodiments described herein.

[0067] Although the internal components of the apparatus of Figure 8 (e.g., server 804) are not shown, one of ordinary skill in the art would understand that there are internal components and their interconnections in the apparatus of Figure 8. Accordingly, additional details regarding the internal construction apparatus will not be disclosed further herein.

[0068] The disclosure may be described in the general context of computer code or machine usable instructions, including computer executable instructions, such as program modules, executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general purpose computers, and more specialized computing devices. The disclosure may also be implemented in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0069] In this specification, the term "and / or" with respect to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, "element A, element B, and / or element C" may include element A only, element B only, element C only, elements A and B, elements A and C, elements B and C, or elements A, B, and C. Furthermore, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0070] The subject matter of the present disclosure has been specifically described herein to satisfy statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Rather, the inventors have contemplated that the claimed subject matter may be embodied in other ways, such as to include different steps or combinations of steps similar to those described in this document, in combination with other current or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to connote various elements of the method used, these terms should not be construed as implying a particular order between the various steps disclosed herein, unless the order of the individual steps is explicitly described.

Claims

1. 1. A computerized method for evaluating and improving performance and accuracy of a model version, the method comprising: detecting output from a model pipeline including at least two sub-models, the output of the model pipeline including one or more predictions calculated based on the at least two sub-models, the method comprising obtaining one or more files including formatted datasets corresponding to the model pipeline, the formatted data including the data used to generate the predictions and the one or more predictions, the method comprising receiving a configuration file and calculating a plurality of sub-model performance reports from the plurality of datasets based on the configuration file, each of the at least two sub-model performance reports including a plurality of performance metrics determined for the corresponding version of the model.

2. 2. The computerized method of claim 1, wherein generating the multiple sub-model performance reports from the multiple datasets based on the configuration file comprises: identifying, from the configuration file, multiple data subsets to extract from the multiple datasets; and extracting the multiple data subsets from the multiple datasets, each of the multiple data subsets being extracted from one of the multiple datasets for the corresponding sub-model.

3. 3. The computerized method of claim 2, wherein generating the sub-model performance reports from the multiple data sets based on the configuration file includes identifying the multiple performance measures to calculate from the configuration file.

4. 4. The computerized method of claim 3, wherein generating the plurality of sub-model performance reports from the plurality of data sets based on the configuration file comprises: executing, for each of the at least two sub-models, a computer script that calculates the plurality of performance metrics for the corresponding version of the model based on the corresponding data subset and generates the sub-model performance report for the corresponding sub-model of the model pipeline.

5. The computerized method of claim 1 , wherein the configuration file includes object-oriented programmatic expressions that identify computational functions that correspond to the plurality of performance metrics.

6. 6. The computerized method of claim 5, wherein the plurality of performance metrics comprises a measurement absolute percentage error (MAPE), a mean absolute error (MAE), or a root mean square error (RMSE).

7. 6. The computerized method of claim 5, wherein the object-oriented programmatic expressions include quantification operations of weights and / or representations of predicted data based on corresponding measurements of training data.

8. 2. The computerized method of claim 1, wherein the object-oriented programmatic expressions include quantification operations for unexpected or undocumented changes to the structure or semantics of data used and / or generated by the sub-model.

9. 1. A system for monitoring multiple sub-models of a model pipeline, the system comprising: a data model performance monitoring system that receives, via one or more processors executing a script, at least one dataset corresponding to multiple sub-models of the model pipeline, the at least one dataset including metadata corresponding to predictions and prediction data for each of the multiple sub-models of the model pipeline; performs operations defined by a configuration file corresponding to the model pipeline including instructions for calculation of a plurality of sub-model performance measures; and outputs the plurality of sub-model performance measures as a report for each of the sub-models based on the configuration file.

10. 10. The system of claim 9, further comprising a dashboard reporting system that, via one or more processors executing a script, uses the plurality of sub-model performance measure reports, calculates a first level score for each sub-model based on the plurality of sub-model performance measure reports and the configuration file, and calculates a second level score for the model pipeline based on the first level score for each of the sub-models and the configuration file.

11. 11. The system of claim 10, wherein the dashboard reporting system further comprises an interface that presents for displaying the first level scores of each sub-model and the second level scores of the model pipeline via the one or more processors executing the script.

12. 12. The system of claim 11, wherein calculating the first level scores for each sub-model comprises obtaining an observation dataset corresponding to an object identified in the at least one dataset.

13. 13. The system of claim 12, wherein calculating the first level scores for each sub-model comprises obtaining a prediction dataset corresponding to objects identified in the at least one dataset.

14. The system of claim 9 , wherein the configuration file includes object-oriented programmatic expressions that identify quantitative parameters for the calculation of the plurality of sub-model performance measurement reports.

15. 1. A method for monitoring a plurality of models deployed in a model pipeline, the method comprising: generating one or more predictions and one or more model functions using each model in the plurality of models; identifying performance measurement thresholds for each model in the plurality of models; generating a model report for each model in the model pipeline using the performance measurement thresholds for each model in the plurality of models; generating a health score for each model in the plurality of models based on the generated model reports; and generating a health score for a model pipeline based on the generated health scores.

16. The method of claim 15 , wherein the performance measure comprises a metric measure, a bias measure, or a drift measure.

17. 16. The method of claim 15, wherein the performance measure comprises a measurement absolute percentage error (MAPE), a mean absolute error (MAE), or a root mean square error (RMSE).

18. The method of claim 16 , wherein the bias measure comprises a quantification of weights and / or representations of predictive data elements based on corresponding measures of training data.

19. The method of claim 16 , wherein the drift measure comprises a quantification operation for unexpected or undocumented changes to the structure or semantics of data used and / or generated by the sub-model.

20. The method according to any one of claims 15 to 19, wherein the performance measurement thresholds are defined within an object file consisting of program expressions executed by one or more processors.

Images