Model verification based on sub-model performance

Abstract

The invention relates to model verification based on sub-model performance. Methods, systems, and computer readable media are disclosed herein for a concurrent comparison tool for evaluating sub-models of data model pipes in a deployed or pre-deployed environment. The tool may compute a plurality of performance measurements that quantitatively assess the performance of each sub-model in the data model pipeline based on a profile that facilitates confirmation of the technical performance and predictive accuracy of the sub-model. In addition, multiple versions of a sub-model deployed in a similar data model pipeline or in a pre-deployment environment may be comparatively evaluated. A leading version of the sub-model may be identified and deployed.

Description

[0001] This application is a divisional application of the PCT international application filed on December 14, 2022, with national application number 202280086792.1 and invention title "Model Verification Based on Sub-Model Performance" which has entered the Chinese national phase. Technical Field

[0002] This application relates to machine learning data models. More specifically, it relates to a comparative evaluation of the predictive accuracy and performance measures of different versions of machine learning data models. Summary of the Invention

[0003] Embodiments of this disclosure particularly relate to methods, systems, and computer-readable media for computationally evaluating the technical performance and / or predictive accuracy of sub-models in a machine learning data model (hereinafter referred to as the "model") pipeline. Some embodiments also include deploying versions of sub-models that exhibit improved technical performance and / or predictive accuracy relative to currently deployed versions of sub-models. As will be described, aspects of the invention discussed below monitor and compare the technical performance and / or predictive accuracy by monitoring multiple sub-models in the model pipeline.

[0004] A first embodiment may include a computerized method for evaluating and improving the performance and accuracy of sub-models. This computerized method may include detecting the output from a model pipeline comprising at least two sub-models, wherein the output of the model pipeline includes one or more predictions computed based on the at least two sub-models. The computerized method may also include retrieving one or more files containing a formatted dataset corresponding to the model pipeline, wherein the formatted data includes data for generating predictions and one or more predictions, and receiving a configuration file. Furthermore, the computerized method may include computed multiple sub-model performance reports from multiple datasets based on the configuration file, wherein each of the at least two sub-model performance reports includes multiple performance metrics determined for a corresponding version of the model.

[0005] Another embodiment may include a system for monitoring multiple sub-models in a model pipeline. This system may include a data model performance monitoring system that executes a script via one or more processors. The script may include operations to receive at least one dataset corresponding to the multiple sub-models of the model pipeline, wherein the at least one dataset includes metadata corresponding to predictions and prediction data for each of the multiple sub-models in the model pipeline; to execute operations defined by a configuration file corresponding to the model pipeline, the configuration file including computational instructions for performance measurements of the multiple sub-models; and to output the performance measurements of the multiple sub-models as a report for each of the sub-models based on the configuration file.

[0006] Another embodiment may include a method for monitoring multiple models deployed within a model pipeline. This method may use each of the multiple models to generate one or more predictions and one or more model features, and identify a performance measurement threshold for each of the multiple models. Furthermore, the method may use the performance measurement threshold for each model within the multiple models to generate a model report for each model within the model pipeline, and generate a health score for each model within the multiple models based on the generated model reports. Additionally, the method may generate a model pipeline health score based on the generated health scores.

[0007] This summary is provided to introduce a set of concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter. The invention is defined by the claims, supported by the specification (including the detailed description). Attached Figure Description

[0008] The system and method for learning feature vectors for tracking single and cross-sensor objects are described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1 A process diagram illustrating an example of calculating sub-model performance measurement based on a model pipeline, according to some embodiments of the present disclosure, is shown.

[0010] Figure 2 Another process diagram is shown as an example of calculating sub-model performance measurement based on model pipeline computation according to some embodiments of the present disclosure;

[0011] Figure 3 Example interfaces for displaying sub-model performance measurements are shown according to some embodiments of this disclosure;

[0012] Figure 4 An example system environment is shown that facilitates the computation of submodel performance measurements according to some embodiments of this disclosure;

[0013] Figure 5 This is a block diagram of an example method for calculating performance measurements of one or more sub-models according to some embodiments of the present disclosure;

[0014] Figure 6 Example code snippets of sub-model insights according to some embodiments of this disclosure are shown;

[0015] Figure 7 Example code patterns of configuration files according to some embodiments of this disclosure are shown to facilitate the calculation of submodel performance measurements;

[0016] Figure 7AExample code snippets of configuration files according to some embodiments of this disclosure are shown to facilitate the calculation of submodel performance measurements; and

[0017] Figure 8 This is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure. Detailed Implementation

[0018] From a higher perspective, computer models can be trained to predict outcomes (e.g., calculate future events based on current data) and / or perform tasks (e.g., calculate actions to trigger downstream systems or devices based on data). In many cases, the overall model comprises multiple sub-models that perform predictions or tasks, thus contributing to the overall prediction or task output. Traditionally, once the sub-models are trained with training data, a model pipeline is built and deployed. Once deployed, the model pipeline can be cumulatively evaluated to verify that it performs within the desired performance metrics. However, compared to traditional methods of model pipeline performance testing, the aspects described herein provide quantitative and / or qualitative performance reports at the sub-model level. Performance measurements can be delivered for supervised evaluation and / or can be automated by the described system.

[0019] Therefore, the embodiments described herein facilitate the computation of performance measurements (e.g., metrics, drift, bias, and / or violations) of individual sub-models in a model pipeline comprising multiple sub-models. To achieve this, the system uses a dataset (e.g., insights) to track sub-model operations. Generally, data storage captures the outputs (e.g., predictions) of (multiple) sub-models and the inputs (e.g., data consumed by (multiple) sub-models corresponding to features of the sub-models). The inputs of (multiple) sub-models can be captured in pre-encoded (e.g., before being transformed into sub-model readable data) and / or post-encoded (e.g., after being transformed into sub-model readable data) formats. For example, the inputs of (multiple) sub-models can be captured before and / or after natural language processing, dimensionality reduction, vectorization, or any other suitable data transformation that facilitates data ingestion by the sub-models. Similarly, sub-model outputs can be captured in encoded or decoded formats. For example, in the first sub-model (e.g., Figure 1 The output of sub-model A 104) is one or more second models (e.g., Figure 1 In the case of input to sub-model B (106), data storage can capture the output and / or data configured for input in the second(multiple) sub-models. In at least one embodiment, sub-model data is captured in an object-oriented file (e.g., an insight file) in the form of metadata in a programmably defined schema. This schema can help with identification.

[0020] The system can be configured to continuously, periodically, intermittently, and / or on-demand compute one or more performance measurements of sub-models of the model pipeline. In some embodiments, the system can monitor the location where the model pipeline output is stored, or it can monitor systems downstream of the model pipeline acting on that output. For example, in response to the storage of output from the model pipeline (e.g., Figure 1 The system (model pipeline prediction 110) can extract data associated with one or more sub-models stored in the data store. An automatically executed preprocessing script can load one or more configuration files and one or more computational dictionaries into the system's reporting component. The reporting component transforms the data stored in the data store based on one or more configuration files. Furthermore, the script can extract all input data and separate the data based on the sub-models.

[0021] definition

[0022] As used herein, the terms “observational data,” “factual situation,” “actual,” and “target” are used interchangeably to refer to empirical data encoded as data and / or observed real-world information. For example, observational data includes measured, captured, or recorded values ​​of events or variables that represent and / or quantify outcomes that have occurred. In one example, observational data includes values ​​for the total number of patients at a specific healthcare entity over a defined six-month time period, such as those recorded in historical reporting data for that healthcare entity.

[0023] As used herein, the term "predictive data" refers to any and all data input to and output from a version of the data model. For example, predictive data may include input(s), such as training datasets ingested to generate and trigger outputs. Additionally or alternatively, predictive data may include output(s) generated or produced from a version of the data model, such as predictions(s) made by that version of the data model using the input(s). Predictive data may also include metadata related to the data model, metadata related to a version of the data model, metadata related to sub-model inputs, and / or metadata related to sub-model outputs. Predictive data may also refer to other outputs of a version of the data model.

[0024] As used herein, the terms “model” and “data model” are used interchangeably to refer to a data model of a machine learning / artificial intelligence type defined by algorithmic decision logic. A data model (and any version thereof) may include features such as decision logic, computational layers, neural networks, Markov chains, weighted algorithms (specific to or not specific to variables, values, layers, sub-models), and / or random forests. Although expressed in the singular, it should be understood that a data model (and any version thereof) may include multiple specific sub-models that, for example, operate together in a specific order or in parallel, contributing to an output such as prediction.

[0025] As used herein, “version,” “submodel version,” and “data model version” can be used interchangeably to refer to a specific iteration of a data model that has defined configurations of inputs, operations (e.g., decision logic), and / or outputs specific to or unique to that particular iteration.

[0026] As used herein, the terms “script” and “computer programming script” are used interchangeably to refer to computer-readable and executable instructions / programming code, which is an expression of instructions that cause, manage, and facilitate a computer to perform a series of operational steps in an automatic or semi-automatic manner.

[0027] As used herein, “performance measurement” refers to a measure of the technical performance and prediction accuracy (or inaccuracy) of the captured representation and quantification of the model version and / or other behavior. Performance measurements may include, for example, metrics, prediction accuracy, bias, data drift, noise, variance, etc. Examples of metrics include measuring percentage absolute error (MAPE), mean absolute error (MAE), and / or root mean square error (RMSE).

[0028] Example

[0029] refer to Figure 1 , Figure 1 This is an example data flow diagram illustrating an example process 100 for computing performance measurements of one or more sub-models in a model pipeline. Generally, process 100 may include a model pipeline 102 that generates model pipeline predictions 110. Model pipeline 102 can draw from one or more data sources (e.g., Figure 4 The data cluster 402 receives one or more inputs to compute model pipeline predictions 110. The data cluster 402 may be a database 152 or multiple databases 152 and 404 that store data contributing to chaotic and / or dynamic systems. For example, healthcare or meteorology. In other words, the data may be continuously, intermittently, and / or periodically changed as patients associated with the data are treated, viewed, discharged, or otherwise interact with the institution, care network, or region maintaining the database. The data may be generated by sensors monitoring patients (e.g., heart rate and / or weight), data about patients (e.g., admission date and / or demographics), and / or observations about patients (e.g., diagnosis of disease or condition).

[0030] Process 100 may include the output or capture of insight 112, which includes data records of all data used to generate predictions (e.g., 114, 118, and 122) for the sub-models and the predictions themselves (e.g., 116, 120, and 124). Insight 112 may format the data used to generate predictions in a data pattern. For example, model pipeline 102 may include one or more machine learning models (e.g., sub-model A 104) that generate outputs (e.g., sub-model A prediction 116). Sub-models (e.g., sub-model A 104, sub-model B 106, and sub-model N 108) may use different inputs from a data source. For example, sub-model A 104 may consume data corresponding to fields of sub-model A feature 114. Sub-model B 106 may use data corresponding to fields of sub-model B feature 118, while sub-model N 108 may use data corresponding to fields of sub-model N feature 122. The data patterns for insights can include submodel execution metadata (e.g., timestamps indicating the execution of one or more submodels, versions of one or more submodels, and data indicating the identity of the model pipeline associated with one or more submodels).

[0031] Process 100 may also include a reporting job 134 that receives configuration file 126 and insights 112. Generally, the configuration file contains metadata information required for the monitoring job and processing scripts to obtain data and store it in a manner easily queryable for performance measurement calculations. Configuration file 126 may include one or more scripts that facilitate the execution of operations that might be needed to retrieve / transform / merge / separate data identified in insights 112. For example, configuration file 126 may include query parameters pointing to locations of other objects that facilitate the extraction of data from insights 112 (e.g., Figure 7A (query_params in code 708). Similarly, configuration file 126 may include data parameters that point to objects that facilitate the conversion of extracted data into a report format (e.g., Figure 7A (refer to the data_params in code 712). Configuration file 126 may also include one or more scripts to facilitate the calculation of performance measurements for one or more sub-models. Performance measurements may include metric measurements, bias measurements, drift measurements, and / or violation measurements, etc.

[0032] Bias measurement is generally a performance measure that is computationally quantified based on the weights and / or representations of data elements on the training data. Since biased data can lead to skewed predictions, detecting bias at the sub-model level can help identify and / or locate certain types of procedural and / or data errors in the model pipeline.

[0033] When calculating the bias of a model version, one or more baseline analyses are used to examine predictions from real-time data on insight features. Baselines can be generated from training data containing features and predicted values. This helps in generating results. These results can be a comprehensive report describing how the data has progressed at the feature level relative to the baseline over a period of time.

[0034] It supports both pre-training (features evaluated with actual labels) and post-training (features evaluated with actual and predicted label values) bias. For example, once the model pipeline data (actual data) has been evaluated, the model monitoring system loads preprocessed and baseline files into the system and executes preprocessing algorithms to obtain model insight features and actual data. Then, the data is analyzed using the baseline file with bias and configured metrics to calculate any pre-training bias. In another example, once the model pipeline data (actual and predicted) has been evaluated, the model monitoring system loads preprocessed and baseline files into the system and executes preprocessing algorithms to obtain model insight features and actual data. Then, the data is analyzed using the baseline file with bias and configured metrics to calculate any post-training bias.

[0035] Drift measurement is typically a performance metric for calculating the occurrence of unexpected or undocumented changes to the structure or semantics of the data consumed and / or generated by a sub-model. Because drift events can corrupt predictive models or disrupt computational operations, detecting drift at the sub-model level facilitates the identification and / or localization of certain types of programming flaws in the model pipeline.

[0036] In an example drift calculation, a baseline created from the training data is used to analyze model features. The example drift calculation retrieves statistics about the model data relative to the baseline data. The drift calculation can then identify any drifts in features within the model. Furthermore, the drift calculation can be used to analyze multiple models with their own baselines. For multiple models, a baseline file can exist for each model, and the features of each model can be mapped to the corresponding model baseline for each model.

[0037] A violation refers to a breach of a threshold and / or baseline. Examples include features of a model version and the specific performance measure where the violation occurred, the expected value and / or target value of the violated baseline and / or threshold, the value of the performance measure for which a violation was determined, etc. Furthermore, a baseline created from the training data is used to analyze features in the model or sub-model. During model analysis, any new or missing features appearing in the model data will be captured. For any valid elements in the model data, the baseline data will be analyzed to capture any data type mismatches, positive, negative, or non-zero variations or violations.

[0038] Generally, metrics include the analysis and comparison of the predictive reliability of sub-models. For example, such as Figure 7AAs shown, the configuration file may include measures of measurement absolute percentage error (MAPE), mean absolute error (MAE), or root mean square error (RMSE), or combinations thereof, for one or more sub-models. MAPE can generally be expressed in the following example, although other expressions of MAPE are contemplated within the scope of the aspects discussed here:

[0039]

[0040] Generally, MAE can be expressed in the following examples, although other expressions of MAE are contemplated within the scope of the aspects discussed here:

[0041]

[0042] RMSE is generally the standard deviation of the prediction error. Therefore, it can be expressed in the following example, although other expressions of RMSE are contemplated within the scope of the aspects discussed here:

[0043]

[0044] In the example above, A refers to the observed data; F refers to the prediction based on the input data. Configuration file 126 may also include baseline thresholds associated with the performance measurement of each applicable sub-model. For example, as... Figure 7A As shown, the configuration file can include the maximum threshold value of the MAE value for the hospitalized sub-model.

[0045] Reporting job 134 can compute one or more sub-model performance measurement reports (e.g., sub-model A performance measurement report 136) based on insight 112 and the computational library and data formatting instructions included in configuration file 126. For example, the application of computational library and data formatting instructions can compute MAPE, MAE, RMSE, or any other metric applicable to sub-model A 104 based on sub-model A prediction 116 and corresponding observation data from data source 152.

[0046] Process 100 may also include converting one or more submodel performance measurement reports (e.g., submodel A performance measurement report 136) into submodel scores (e.g., submodel A health score 144). Submodel scores can be calculated at two levels. The first level may include comparing the calculated performance measurement to a corresponding baseline threshold. In some embodiments, a submodel “passes” the first-level score if the calculated value for the performance metric is below a maximum threshold and / or above a minimum threshold. Similarly, a submodel may “fail” the first-level score if the calculated value for the performance metric is above the maximum threshold or below the minimum threshold. Additionally, for the first-level score, a submodel may be classified as “at risk” if the calculated value for the performance measurement is within a predetermined value of the baseline threshold. For example, as... Figure 7A As shown, the configuration file associated with the inpatient submodel may include example code 7010, which sets a maximum threshold for the MAE value of the inpatient submodel. When the value calculated for MAE by reporting job 134 is below the maximum baseline threshold for MAE, the inpatient submodel can pass the first-level score of MAE. The configuration file may also include example codes 7012 and 7014, which set drift measurement parameters and processing.

[0047] The second-level submodel score can include a quantitatively calculated representation of the submodels in the model pipeline (e.g., a multi-model health score of 150). For example, the second-level score can include a comparison of the total number of passed performance measurements to the total number of performance measurements calculated for the submodel. A submodel can pass the second-level score if the percentage of passed performance measurements is higher than a predetermined value. In some embodiments, a submodel can be identified as "at risk" if the total number of passed performance measurements is lower than a predetermined value but higher than a second value. Similarly, a submodel can be identified as "failed" if the total number of passed performance measurements is lower than a second value. Some embodiments may calculate second-level submodel scores for one or more categories of performance measurements (e.g., metric measurements, bias measurements, drift measurements, violation measurements).

[0048] In some embodiments of process 100, the sub-model scores may be obtained from the interface of the reporting dashboard module 142 (e.g., Figure 3 Viewable via Interface 300. This interface can present an ordered visual representation of the health scores of the first-level sub-models and / or the second-level sub-models. For example, the health score of sub-model A (144), sub-model B (146), and / or sub-model N (148) can be summarized for each type of performance measurement (e.g., metric, drift, bias, violation) in the first part of Interface 300. Alternatively, personalized health scores of sub-model A (144), sub-model B (146), and / or sub-model N (148) can be accessed by interacting with elements of Interface 300.

[0049] Additionally or alternatively, some embodiments of process 100 include automatically replacing submodels with different versions of the submodel based on the relative health score of each submodel. The monitoring dashboard module 142 can determine a leading version of the submodel based on performance measurement data in performance reports and / or health scores. For example, multiple scores of different versions of the same submodel merged into a separate model pipeline (e.g., a first version of submodel A and a second version of submodel A) can be compared via the monitoring dashboard module 142. The leading version of the submodel can be identified and selected autonomously. For example, given a system-driven comparison of various performance measurements in the submodel performance score, the leading version has improved performance measurements and / or higher predictive accuracy relative to at least one other version of the submodel. Alternatively, the leading version can be semi-autonomously selected in response to user input such as a user manually selecting a leading version from a list and / or a user confirming a system-recommended leading version. The system can then execute and / or trigger the deployment of the leading submodel version.

[0050] Go to Figure 2 , Figure 2 An example prediction model process 200, including a hospitalization prediction model pipeline 202, is illustrated according to an embodiment described herein. The hospitalization prediction model pipeline 202 can consume data from... Figure 4 Data from data cluster 402 is used to calculate predictions for total patient admissions to the hospital over the next 12, 24, 36, and 72 hours. To facilitate overall prediction, the hospitalization prediction model pipeline 202 may include one or more emergency prediction sub-models 204 and one or more hospitalization prediction sub-models 206, etc. One or more emergency prediction sub-models 204 can consume... Figure 4 The data cluster 402 uses multiple fields from one or more databases to predict emergency admissions in the next 12, 24, 36, and 72 hours. Similarly, one or more inpatient prediction sub-models 206 can use... Figure 4 Data cluster 402 uses multiple fields from one or more databases to predict non-emergency hospital admissions in the next 12, 24, 36, and 72 hours.

[0051] As shown in the figure, process 200 includes generating insight 208 as the output of model pipeline 202. Insight 208 may be an object file (e.g., a JavaScript object symbol file) that captures metadata about model pipeline 202 and (multiple) emergency sub-models 204 and (multiple) hospitalization prediction sub-models 206. For example, insight 208 may capture factor 210 and value 212. Factor 210 may be a programmatic expression in the object file whose identifier represents metadata such as the identifier of model pipeline 202, the identifier of (multiple) sub-models, the time of model initiation, and the output of (multiple) sub-models. Value 212 may be metadata representing features consumed by (multiple) sub-models and feature attributes (e.g., other object files, database fields, etc.). In some embodiments, insight 208 or another insight may capture metadata similar to that described above for observation data corresponding to the output of a sub-model.

[0052] Process 200 includes a reporting job 222 that generates performance measurement reports (e.g., urgent submodel performance report 224 and non-urgent submodel performance report 226) based on a computational dictionary and performance measurements using configuration file 214. Configuration file 214 may be an object file containing metadata information, processing scripts, and output storage locations that may be needed for reporting job 222 to perform operations such as retrieving / transforming / merging / separating data identified in insight 208. For example, the configuration file may include query parameters pointing to locations of other objects that facilitate data extraction from insight 208 (e.g., ...). Figure 7A (e.g., query_params in code 708). Alternatively, the configuration file may include data parameters that point to objects that facilitate the conversion of extracted data into report formats (e.g., ...). Figure 7A (The code 714's data_params). Configuration file 214 may also include one or more scripts to facilitate the calculation of performance measurements for one or more sub-models.

[0053] As shown in the figure, the generated performance measurement report may also include report job ingestion observation data 220. Observation data 220 may be data stored in data cluster 402, representing observations corresponding to the predictions. Therefore, observation data 220 may be observations of emergency and non-emergency hospitalizations corresponding to the same time period of the emergency sub-model 204 and the hospitalization prediction sub-model 206.

[0054] Go to Figure 4 , Figure 4 An example system environment 400 according to an embodiment described herein is illustrated. Block components of system environment 400 include computing processes and / or components that may include any combination of hardware, firmware, and / or software. For example, system environment 400 may include information about… Figure 8 The computing environment described in 800 is one or more devices.

[0055] like Figure 4 As shown, system environment 400 includes one or more databases of data cluster 402. Data cluster 402 may maintain structured and unstructured data in one or more relational databases (e.g., database 152 and database 404). This data may be associated with any chaotic and / or dynamic system. For example, the data may include values, comments, records, etc., corresponding to multiple people. Additional data may be modified (e.g., added, edited, removed, or manipulated in any other way) in database 152 and / or database 404 when people interact with one or more entities maintaining data cluster 402. For example, data may be modified continuously, intermittently, and / or periodically when a patient associated with the data is treated, viewed, discharged, or otherwise interacts with the institution, care network, or region maintaining data cluster 402. Data may be generated by sensors that monitor patients (e.g., heart rate and / or weight), data about patients (e.g., admission date and / or demographics), and / or observations about patients (e.g., diagnosis of disease or condition). A model pipeline (e.g., model pipeline 102) can use data from a first time point to predict outcomes at future time points. Then, as time progresses from the first time point to future time points, the data can be modified. At future time points, the data stored in data cluster 402 can be used to determine the observations corresponding to the earlier predictions.

[0056] System environment 400 also includes a modeling job 406, which includes a model pipeline 102 and one or more sub-models. Modeling job 406 may include a programmed expression for executing the model pipeline 102 continuously, intermittently, and / or periodically. Modeling job 406 may also include a programmed expression that generates one or more outputs corresponding to the model pipeline 102. For example, modeling job 406 may output one or more insights 112 from a database. In some embodiments, insights 112 are one or more object files (e.g., JavaScript object symbol files) that capture metadata related to the model pipeline 102, one or more sub-models, and / or observation data. For example, insights 112 may include expressions similar to those for observation insight expression 602, insight factor expression 604, and / or insight value expression 606, such as... Figure 6 As shown. It is important to note that expressions 602, 604, and 606 are merely example insight expressions, not restrictive ones.

[0057] System environment 400 also includes a reporting job 134, which includes a configuration module 408 and an execution module 410. The reporting job 408 may include a programmatic expression for continuously, intermittently, and / or periodically calculating sub-model reports. For example, the execution module 410 may be programmed to activate the configuration model 408 in response to output from model pipeline 102. Upon activation, the configuration model 408 can access insight 112 and identify a configuration file 126 stored in a database. In some embodiments, the configuration file 126 is one or more object files (e.g., JavaScript object symbol files) that include object-oriented expressions related to model pipeline 102, one or more sub-models, and / or one or more performance measurements. For example, the configuration file 126 may be located in a database such as... Figure 7 The data schema 700 is shown. Data schema 700 may include metadata information (e.g., code 702), performance measurements (e.g., code 704), and output storage location (e.g., code 706), which facilitates the reporting job 134 performing operations that may be needed to retrieve / transform / merge / separate the data identified in insight 112. Figure 7A As shown, data pattern 700 may include pre-configured performance measurements. For example, data pattern 700 may include one or more metrics (e.g., code 710), which may include codes such as example code 7010 including the MAE metric. Similarly, data pattern 700 may include one or more drift measurements, which may include codes such as example code 7012 and / or example code 708. Data pattern 700 may similarly include one or more violation measurements and / or one or more deviation measurements. Reporting job 134 may also include a computation library 412, whose maintenance can be invoked by one or more configuration files 126 to compute rules and algorithms.

[0058] System environment 400 also includes a reporting dashboard 142 that facilitates the calculation of submodel health scores. Reporting dashboard 142 may include a user interface 414 that facilitates the interactive presentation of submodel health score calculations (e.g., user interface 300). Reporting dashboard 142 may also include a health score calculation module 416. Health score calculation module 416 may include rules that facilitate qualitative and / or quantitative evaluation of (multiple) submodels. For example, a submodel may “pass” a Level 1 score if the calculated value for a performance metric is below a maximum threshold and / or above a minimum threshold. Similarly, a submodel may “fail” a Level 1 score if the calculated value for a performance metric is above a maximum threshold or below a minimum threshold. Additionally, a submodel may be classified as “at risk” for a Level 1 score if the calculated value for a performance metric is within a predetermined value of a baseline threshold.

[0059] System environment 400 also includes network 418. Network 418 may include wireless and / or physical (e.g., hardwired) connections. Exemplary networks include telecommunications networks of service providers or operators, wide area networks (WANs), local area networks (LANs), wireless local area networks (WLANs), cellular telecommunications networks, Wi-Fi networks, short-range wireless networks, wireless metropolitan area networks (WMANs), networks with Bluetooth® capability, fiber optic networks, or combinations thereof. Network 418 typically provides components and devices with access to the Internet and network-based applications.

[0060] Go to Figure 5 , Figure 5 A method 500 for calculating performance measurements of a sub-model according to an embodiment described herein is illustrated. Each block of the method 500 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method 500 can also be implemented as computer-usable instructions stored on a computer storage medium. The method 500 can be provided by a standalone application, service, or managed service (standalone or in combination with another managed service), or a plug-in to another product, to name just a few. Furthermore, as an example, reference is made to... Figure 1 Process 100 or Figure 2 The process 200 is used to describe method 500. However, the method may be additionally or alternatively performed by any system or any combination of systems, including but not limited to those systems described herein.

[0061] Method 500 includes, in block 502, detecting the output of a model pipeline that includes one or more sub-models. For example, model pipeline 102 may output model pipeline prediction 110. Additionally, model pipeline 102 may output model insight 112. Reporting job 134 may detect the output of model pipeline prediction 110 or model insight 112. As another example, hospitalization model pipeline 202 may output insight 208. Reporting job 222 may detect the addition of insight 208 to a database or other data storage medium.

[0062] In box 504, method 500 includes retrieving the dataset(s) corresponding to the model pipeline. For example, in response to detecting output from model pipeline 102, reporting job 134 may retrieve insight 112. As another example, reporting job 222 may retrieve insight 208.

[0063] In box 506, method 500 includes configuring the reporting module based on configuration files corresponding to one or more sub-models. For example, reporting job 134 may load configuration file 126 to configure the computation library and write scripts for the data processing procedure of insight 112. As another example, reporting job 222 may load configuration file 214 to configure one or more emergency prediction sub-model performance measurement computation libraries and scripts 216 and one or more hospitalization prediction sub-model performance measurement computation libraries and scripts 218.

[0064] In box 508, method 500 includes a configuration-based reporting module that calculates performance measurements for one or more sub-models. For example, reporting job 134 may execute one or more scripts in configuration file 126 that calculate performance measurement 128 for sub-model A, performance measurement 130 for sub-model B, and / or performance measurement 132 for sub-model N. As another example, reporting job 222 may use scripts in configuration file 214 to generate an urgent sub-model performance report 224 and / or a non-urgent sub-model performance report 226.

[0065] Some embodiments of method 500 may also include generating sub-model health scores based on computational performance measurements. For example, report dashboard 142 may use data from sub-model A performance measurement report 136 and calculate sub-model A health score 144. Report dashboard 142 may use data from sub-model B performance measurement report 138 and calculate sub-model B health score 146. Report dashboard 142 may use data from sub-model N performance measurement report 140 and calculate sub-model N health score 148. Furthermore, for example, report dashboard 142 may use data from sub-model A health score 144, sub-model B health score 146, and / or sub-model N health score 148 to calculate a multi-model health score 150.

[0066] Some embodiments of method 500 may further include comparative analysis of submodel scores for multiple versions of the submodel. For example, submodel A health score 144, submodel B health score 146, and / or submodel N health score 148 are associated with health scores for other versions of submodel A, submodel B, and / or submodel N. In cases where a leading version of a submodel is not included in the deployed model pipeline, some embodiments of method 500 may replace a deployed submodel with the leading version. A leading submodel can be deployed because it exhibits improved technical performance measures and / or improved prediction accuracy relative to another model version (e.g., the currently used data model version), demonstrating improved technical behavior and performance. Thus, the newly deployed leading sub-data model version replaces another poorly performing submodel version. Additionally or alternatively, the leading version may be used as input to retrain the corresponding data model and generate additional updated versions(s) of the data model(s).

[0067] from Figure 8 To begin, an exemplary computing environment is illustrated according to embodiments of the present invention. Those skilled in the art will understand that the example computing environment 800 is merely one example of a suitable computing environment and is not intended to limit the scope or functionality of the invention. Similarly, computing environment 800 should not be construed as input regarding... Figure 8 Any dependencies and / or requirements of each component and combination(s) of components shown. Those skilled in the art will understand that... Figure 8 The connection shown is also an example, because it is used in situations such as... Figure 8 Other methods, hardware, software, and devices for establishing communication links between the components, devices, systems, and entities shown can be used to implement this invention. Although one or more solid lines are used to describe the connections, those skilled in the art will understand that... Figure 8 Example connections can be hardwired or wireless, and for simplicity, can be... Figure 8 Intermediate components that have been omitted or not included. Therefore, Figure 8 The absence of any component should not be construed as limiting the invention to exclude additional components and combinations of components(s). Furthermore, although the device and components are... Figure 8 While referred to as a single device and component, it is understood that some embodiments may include multiple devices and components, such that... Figure 8 It should not be viewed as a limitation on the number of devices or components.

[0068] continue, Figure 8 The computing environment 800 is shown as a distributed environment, in which components and devices can be geographically isolated and can perform individual tasks. Components and devices can communicate with each other and can be linked to each other using network 802. Network 802 may include wireless and / or physical (e.g., hardwired) connections. Exemplary networks include telecommunications networks of service providers or operators, wide area networks (WANs), local area networks (LANs), wireless local area networks (WLANs), cellular telecommunications networks, Wi-Fi networks, short-range wireless networks, wireless metropolitan area networks (WMANs), networks with Bluetooth® capability, fiber optic networks, or combinations thereof. Network 802 typically provides components and devices with access to the Internet and network-based applications.

[0069] The computing environment 800 includes computing devices in the form of a server 804. Although in Figure 8The diagram shows a single component, but the invention can utilize multiple local and / or remote servers within the computing environment 800. Server 804 may include components such as processing units, internal system memory, and suitable system buses for coupling to various components, including databases or database clusters. In some embodiments, the data cluster takes the form of cloud-based data storage and is accessible by a cloud-based computing platform in some embodiments. The system bus can be any of several types of bus architectures using any of a variety of bus architectures, including memory buses or memory controllers, peripheral bus, and local bus. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA®) local buses, and Peripheral Component Interconnect (PCI) buses, also known as mezzanine buses.

[0070] Server 804 may include or have access to computer-readable media. Computer-readable media can be any available media accessible to server 804, and includes volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media. Computer storage media can include, but is not limited to, volatile and non-volatile media, and removable and non-removable media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. In this regard, computer storage media can include, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, cassette tape, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to server 804. Computer storage media does not include transient signals themselves.

[0071] Communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transmission mechanisms, and may include any information delivery medium. As used herein, the term "modulated data signal" refers to a signal whose one or more properties are set or altered in a manner that encodes information in the signal. By way of example and not limitation, communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media. Any combination of the foregoing may also be included within the scope of computer-readable media.

[0072] In various embodiments, server 804 uses a logical connection to communicate with one or more remote computers 806 within computing environment 800. In embodiments where network 802 includes a wireless network, server 804 may use a modem to establish communication with the Internet, server 804 may connect to the Internet using Wi-Fi or a wireless access point, or server 804 may use a wireless network adapter to access the Internet. Server 804 uses network 802 to communicate with... Figure 8 The server 804 can communicate bidirectionally with any or all of the components and devices shown in the diagram. Therefore, the server 804 can send data to and receive data from the remote computer 806 via the network 802.

[0073] Although shown as a single device, remote computer 806 may include multiple computing devices. In the aspect of having a distributed network, remote computer 806 may be located in one or more different geographical locations. In one embodiment where remote computer 806 is multiple computing devices, each of the multiple computing devices may be located in a different location, such as a building on a campus, a medical and research facility in a medical complex, an office or branch of a bank / credit entity, or may be a wearable or carried by a person, or a mobile device attached to a vehicle or trackable item in a warehouse.

[0074] In some embodiments, the remote computer 806 is physically located in a medical environment, such as a laboratory, inpatient ward, outpatient clinic, hospital, medical vehicle, veterinary environment, mobile environment, medical billing office, financial or administrative office, hospital administration environment, home healthcare environment, and / or medical professional office. For example, medical professionals may include physicians; medical specialists such as surgeons, radiologists, cardiologists, and oncologists; emergency medical technicians; physician assistants; nursing practitioners; nurses; nursing aides; pharmacists; nutritionists; microbiologists; laboratory specialists; genetic counselors; researchers; veterinarians; students, etc. In other embodiments, the remote computer 806 may be physically located in a non-medical environment such as a packaging and transportation facility, or deployed within a fleet of delivery or courier vehicles.

[0075] Continuing, the computing environment 800 includes a data storage 808. Although shown as a single component, the data storage 808 can be implemented using multiple data storages communicatively coupled to each other, regardless of the geographical or physical location of the storage devices. Exemplary data storage may store data in the following forms: artifacts, server lists, attributes associated with servers, environments, attributes associated with environments, computer instructions encoded in various computer programming languages, deployment scripts, applications, attributes associated with applications, version information of release packages, build levels associated with applications, application identifiers, release package identifiers, users, roles associated with users, permissions associated with roles, workflows and steps in workflows, clients, servers associated with clients, attributes associated with attributes, audit information, and / or audit trails for workflows. Example data storage may also store data in the form of electronic records, such as patients' electronic medical records, transaction records, billing records, task and workflow records, chronological event records, etc.

[0076] Generally, data storage 808 includes physical memory configured to store information encoded in data. For example, data storage 808 can be computer-readable instructions, computer-executable instructions, data structures, data arrays, computer programs, applications, and supporting usage examples. Figure 8 The computing environment 800 shown provides storage for other data on the functions and actions performed by the components.

[0077] In a computing environment with distributed components coupled via network 802, program modules may reside in local and / or remote computer storage media, such as memory storage devices alone. Embodiments of the invention can be described in the context of computer-executable instructions, such as program modules, executed by a computing device. Program modules may include, but are not limited to, routines, programs, objects, components, and data structures that perform a specific task or implement a specific abstract data type. In various embodiments, server 804 can access, retrieve, communicate, receive, and update information, including program modules, stored in data storage 808. Therefore, server 804 can use a processor to execute computer instructions stored in data storage 808 to perform the embodiments described herein.

[0078] Although Figure 8 Internal components of the device, such as server 804, are not shown, but those skilled in the art will understand that... Figure 8 The device contains internal components and their interconnections. Therefore, further details regarding the internal structure of the device will not be disclosed here.

[0079] This disclosure can be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be implemented in various system configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure can also be implemented in distributed computing environments, where tasks are performed by remote processing devices linked via a communication network.

[0080] As used herein, the remark "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" could include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" could include at least one element A, at least one element B, or at least one element A and at least one element B. Additionally, "at least one of element A and element B" could include at least one element A, at least one element B, or at least one element A and at least one element B.

[0081] The subject matter of this disclosure is described in detail herein to satisfy legal requirements. However, this specification itself is not intended to limit the scope of this disclosure. Rather, the inventors have envisioned that the claimed subject matter may also be implemented in other ways to include different steps or combinations of steps similar to those described in this document, in conjunction with other present or future techniques. Furthermore, although the terms “step” and / or “box” may be used herein to denote different elements of the method employed, these terms should not be construed as implying any particular order between or between the steps disclosed herein unless the order of the individual steps is explicitly described.

Claims

1. One or more non-transient media having computer-readable instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform a plurality of operations for evaluating the performance of a model version, the operations including: Medical data is input into a model pipeline that includes sub-models, where the output of one sub-model determines the input of another sub-model. The model pipeline prediction calculated based on the sub-model is detected as the output from the model pipeline in response to the input medical data; Retrieve one or more files containing formatted data corresponding to the model pipeline, wherein the formatted data includes information associated with the predictions of the model pipeline; Calculate one or more sub-model performance metrics based on the formatted data; as well as Based on one or more sub-model performance metrics, a sub-model is accessed to be utilized in conjunction with the model pipeline in relation to the operation of the model pipeline.

2. One or more non-transient media according to claim 1, wherein the sub-model was not included in the model pipeline prior to the access.

3. One or more non-transient media according to claim 1 or 2, wherein the sub-model: (a) is configured to operate together with the model pipeline, and (b) is configured based on the operation of the first sub-model in the sub-model.

4. The one or more non-transient media according to claim 1, wherein the sub-model is configured as a second version of the first sub-model among the sub-models.

5. The one or more non-transient media according to claim 1, wherein the operation further comprises: Determine one or more performance metrics associated with the model pipeline, and update the model pipeline via the sub-model based on the one or more performance metrics.

6. The one or more non-transient media of claim 1, wherein the sub-model at least partially corresponds to a first sub-model among the sub-models, and wherein the sub-model is deployed in the model pipeline to affect one or both of a performance metric associated with the first sub-model and an accuracy metric associated with the first sub-model.

7. One or more non-transient media according to claim 1, 5 or 6, wherein the sub-model is configured to replace the first sub-model in the sub-models.

8. A computer-implemented method for evaluating the performance of a model version, the computer-implemented method being executed by one or more hardware processors, and comprising: Medical data is input into a model pipeline that includes sub-models, where the output of one sub-model determines the input of another sub-model. The model pipeline prediction calculated based on the sub-model is detected as the output from the model pipeline in response to the input medical data; Retrieve one or more files containing formatted data corresponding to the model pipeline, wherein the formatted data includes information associated with the predictions of the model pipeline; Calculate one or more sub-model performance metrics based on the formatted data; as well as Based on one or more sub-model performance metrics, a sub-model is accessed to be utilized in conjunction with the model pipeline in relation to the operation of the model pipeline.

9. The computer-implemented method of claim 8, wherein the sub-model was not included in the model pipeline prior to the access.

10. The computer-implemented method of claim 8 or 9, wherein the sub-model: (a) is configured to operate with the model pipeline, and (b) is configured based on the operation of a first sub-model in the sub-model.

11. The computer-implemented method of claim 8, wherein the sub-model is configured as a second version of the first sub-model among the sub-models.

12. The computer-implemented method according to claim 8, further comprising: Determine one or more performance metrics associated with the model pipeline, and update the model pipeline via the sub-model based on the one or more performance metrics.

13. The computer-implemented method of claim 8 or 12, wherein the sub-model at least partially corresponds to a first sub-model among the sub-models, and wherein the sub-model is deployed into the model pipeline to affect one or both of a performance metric associated with the first sub-model and an accuracy metric associated with the first sub-model.

14. The computer-implemented method according to claim 8, 11 or 12, wherein the sub-model is configured to replace the first sub-model in the sub-models.

15. A system having one or more hardware processors, said one or more hardware processors being configured to perform a plurality of operations for evaluating the performance of a model version, said operations including: Medical data is input into a model pipeline that includes sub-models, where the output of one sub-model determines the input of another sub-model. The model pipeline prediction calculated based on the sub-model is detected as the output from the model pipeline in response to the input medical data; Retrieve one or more files containing formatted data corresponding to the model pipeline, wherein the formatted data includes information associated with the predictions of the model pipeline; Calculate one or more sub-model performance metrics based on the formatted data; as well as Based on one or more sub-model performance metrics, a sub-model is accessed to be utilized in conjunction with the model pipeline in relation to the operation of the model pipeline.

16. The system of claim 15, wherein the sub-model was not included in the model pipeline prior to the access.

17. The system of claim 15 or 16, wherein the sub-model: (a) is configured to operate with the model pipeline, and (b) is configured based on the operation of the first sub-model in the sub-model.

18. The system of claim 15, wherein the sub-model is configured as a second version of the first sub-model among the sub-models.

19. The system of claim 15, wherein the operation further comprises: Determine one or more performance metrics associated with the model pipeline, and update the model pipeline via the sub-model based on the one or more performance metrics.

20. The system of claim 15 or 19, wherein the sub-model at least partially corresponds to a first sub-model among the sub-models, and wherein the sub-model is deployed into the model pipeline to affect one or both of a performance metric associated with the first sub-model and an accuracy metric associated with the first sub-model.

Images