System and method for determining model fitness and stability of model deployment in automatic model generation - Patents.com

Abstract

According to one embodiment, described herein are systems and methods for use with a computing environment to provide model suitability and stability determination for model deployment and automated model generation. The model suitability and stability component can provide one or more functions supporting model selection, use of model deployability scores and flags, and mitigation of model drift risk to determine model suitability and stability for a particular application. For example, embodiments can be used with analytical applications, data analytics, or other types of computing environments to provide directly actionable risk predictions, for example, in financial applications or other types of applications.

Description

[Technical field]

[0001] Claiming priority This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 142,826, filed January 28, 2021, entitled "SYSTEM AND METHOD FOR DETERMINATION OF MODEL FITNESS AND STABILITY FOR MODEL DEPLOYMENT IN AUTOMATED MODEL GENERATION," and U.S. Patent Application No. 17 / 586,639, filed January 27, 2022, entitled "SYSTEM AND METHOD FOR DETERMINATION OF MODEL FITNESS AND STABILITY FOR MODEL DEPLOYMENT IN AUTOMATED MODEL GENERATION," the contents of each of which are incorporated herein by reference.

[0002] Technical Field SUMMARY OF THE DISCLOSURE The embodiments described herein generally relate to systems and methods for providing model fitness and stability determinations for data models, data analytics environments, and model deployment and automated model generation. [Background technology]

[0003] background With respect to a system for supporting data analytics and processes that address a particular customer requirement, e.g., forecasting accounts receivable in a customer's financial application, it may be inferred that different customers may require the generation of different models that approximate the characteristics of the underlying data-generating business processes.

[0004] Such models may differ for similar processes in different parts of a client company. Moreover, over time, data-generating business processes may change, and the characteristic distributions of inputs to those processes may also change. Summary of the Invention [Means for solving the problem]

[0005] summary According to one embodiment, described herein are systems and methods for use with a computing environment to provide model suitability and stability determination for model deployment and automated model generation. The model suitability and stability component may provide one or more functions supporting model selection, use of model deployability scores and deployability flags, and mitigation of model drift risk to determine model suitability and stability for a particular application. For example, embodiments may be used with analytical applications, data analytics, or other types of computing environments to provide directly actionable risk predictions, for example, in financial applications or other types of applications. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 illustrates an exemplary data analytics environment according to one embodiment. [Diagram 2] FIG. 2 further illustrates an exemplary data analytics environment according to one embodiment. [Diagram 3] 1 is a diagram further illustrating an exemplary data analytics environment according to one embodiment. [Figure 4] FIG. 2 further illustrates an exemplary data analytics environment according to one embodiment. [Diagram 5] FIG. 2 further illustrates an exemplary data analytics environment according to one embodiment. [Figure 6]FIG. 1 illustrates a determination of model suitability and stability for use in connection with a data analytics environment according to one embodiment. [Figure 7] FIG. 1 illustrates an example comparison of probability scores for different models, according to one embodiment. [Figure 8] FIG. 1 illustrates a process or method for determining model fit and stability according to one embodiment. [Figure 9] FIG. 1 further illustrates a process or method for determining model fit and stability, according to one embodiment. [Figure 10] FIG. 1 illustrates a sorted list of invoices according to one embodiment. [Figure 11] FIG. 1 illustrates the output of a model that analyzes data, according to one embodiment. [Figure 12] 1 is a flowchart illustrating a method for determining model suitability and stability for model deployment in automatic model generation according to one embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Detailed Description As discussed above, with respect to systems supporting data analytics and processes that address particular customer requirements (e.g., predicting accounts receivable in a customer's financial application), it may be inferred that different customers may require the generation of different models that approximate the characteristics of the underlying data-generating business processes.

[0008] Such models may differ for similar processes in different parts of a client company. Moreover, over time, data-generating business processes may change, and the distribution of characteristics of the inputs to those processes may also change.

[0009] According to one embodiment, described herein are systems and methods for use with a computing environment to provide model suitability and stability determination for model deployment and model auto-generation. The model suitability and stability component can provide one or more functions supporting model selection, use of model deployability scores and deployability flags, and mitigation of model drift risk to determine model suitability and stability for a particular application.

[0010] According to various embodiments, the described approaches can be used to address a variety of considerations, such as, for example:

[0011] Model fitting benefits from automation, as manual methods are prohibitively expensive in time and money. When systems and methods use data samples to create a class of models for an enterprise, the system does not have the opportunity to manually tune the models with expert data scientists using case-by-case customer data, as there are thousands of customers and manually examining the vagaries of each data set and tuning the model based on the data is prohibitively expensive. The described approach can systematically find model fits that represent the maximum discriminatory information content extractable from a customer data set when using a broad set of specific model classes.

[0012] Moreover, the use of the scores entails the automatic generation of new models that take into account changes over time and across sectors, and automatically filters thousands of potential model candidates using appropriate metrics to find the most important actionable insights based on predictions, without the need for human intervention.The approach described addresses this particular problem for the space of binary classification models and can be extended to multi-class classification.

[0013] Model drift risk should be mitigated. Although model accuracy metrics can change significantly due to drift between training and testing distributions, the system and method cannot simply use model accuracy metrics as a criterion for model selection. When the input distribution or the distribution of a population sample taken on a particular day or week changes, new models are expected to show significant drift in their decision boundaries, even to the extent that classifications are reversed in multiple instances, such as classifying an invoice that was classified as likely paid yesterday as likely not paid today. The described approach can be used to examine how much the scoring distributions have shifted from the training distributions and how much the shift between the training distributions has changed over time.

[0014] The model must be stable. If it is detected that the model is unstable, with the decision boundaries drifting significantly from day to day, this indicates multiple problems with the model fit. In such cases, the classification decision will continue to change from day to day, even reversing the previous day's prediction, even though the data for the individual instances remains unchanged. The described approach can be used to detect such instability.

[0015] Data Analytics Environment Data analytics as generally described enables the computer-based exploration or analysis of large amounts of data in order to derive conclusions or other information from the data, while business intelligence (BI) tools provide an organization's business users with information that describes the enterprise data in a form that enables the business users to make strategic business decisions.

[0016] Examples of data analytics environments and business intelligence tools / servers include Oracle Business Intelligence Server (OBIS), Oracle Analytics Cloud (OAC), and Oracle Fusion Analytics Warehouse (FAW), which support functions such as data mining or analytics, and analytical applications.

[0017] FIG. 1 illustrates an exemplary data analytics environment according to one embodiment. The exemplary embodiment shown in Figure 1 is provided for the purpose of illustrating an example of a data analytics environment in which various embodiments described herein may be used. According to other embodiments and examples, the approaches described herein may be used with other types of data analytics, database, or data warehouse environments. The components and processes shown in Figure 1 and further described herein with respect to various other embodiments may be provided as software or program code executable by, for example, a cloud computing system or other suitably programmed computer system.

[0018] As shown in FIG. 1, in one embodiment, a data analytics environment 100 may be provided by or otherwise operate on a computer system having computer hardware (e.g., processor, memory) 101 and including one or more software components that operate as a control plane 102 and a data plane 104 and provide access to a data warehouse, data warehouse instance 160 (database 161, or other type of data source).

[0019] In one embodiment, the control plane operates to provide control of a cloud or other software product offered within the context of a SaaS or cloud environment, such as, for example, an Oracle® Analytics Cloud environment, or other type of cloud environment. For example, in one embodiment, the control plane may include a console interface 110 that allows access by customers (tenants) and / or cloud environments having provisioning components 111.

[0020] In one embodiment, the console interface may enable access by customers (tenants) operating a graphical user interface (GUI) and / or a command-line interface (CLI) or other interface, and / or may include an interface used by a provider of a SaaS or cloud environment and its customers (tenants). For example, in one embodiment, the console interface may provide an interface that allows a customer to provision services for use within the SaaS environment and configure those provisioned services.

[0021] According to one embodiment, a customer (tenant) can request provisioning of a customer schema in a data warehouse. The customer can also provide, via a console interface, a number of attributes related to the data warehouse instance, including required attributes (e.g., login credentials) and optional attributes (e.g., size or speed). The provisioning component can then provision the requested data warehouse instance including the data warehouse customer schema and populate the data warehouse instance with the appropriate information supplied by the customer.

[0022] According to one embodiment, the provisioning component may also be used to update or edit a data warehouse instance and / or the ETL processes operating in the data plane, for example, by changing or updating the execution frequency of a requested ETL process for a particular customer (tenant).

[0023] According to one embodiment, the data plane may include a data pipeline or process layer 120 and a data transformation layer 134, which together process operational or transactional data from an organization's enterprise software applications or data environment, such as, for example, business productivity software applications provisioned in a customer's (tenant's) SaaS environment. The data pipeline or process may include various functions to extract transactional data from business applications and databases provisioned in the SaaS environment and load the transformed data into a data warehouse.

[0024] According to one embodiment, the data transformation layer may include data models, such as knowledge models (KMs) or other types of data models, that the system uses to transform transactional data received from business applications and corresponding transactional databases provisioned in the SaaS environment into a model format understood by the data analytics environment. The model format may be provided in any data format suitable for storage in a data warehouse. According to one embodiment, the data plane may also include a data configuration user interface, as well as a mapping configuration database.

[0025] According to one embodiment, the data plane is responsible for performing extract, transform, and load (ETL) operations including extracting transactional data from an organization's enterprise software applications or data environment, such as business productivity software applications and corresponding transactional databases offered in a SaaS environment, transforming the extracted data into a model format, and loading the transformed data into customer schemas in a data warehouse.

[0026] For example, according to one embodiment, each customer (tenant) of the environment can be associated with its own customer tenancy within the data warehouse (which is associated with its own customer schema) and can further be provided read-only access to a data analytics schema, which can be updated periodically or on other basis by a data pipeline or process, e.g., an ETL process.

[0027] According to one embodiment, a data pipeline or process can be scheduled to run at intervals (e.g., hourly / daily / weekly) to extract transactional data from enterprise software applications or data environments, such as, for example, business productivity software applications and corresponding transactional databases 106 provisioned in the SaaS environment.

[0028] In one embodiment, an extraction process 108 can extract transactional data, and upon extraction, a data pipeline or process can insert the extracted data into a data staging area, which can serve as a temporary staging area for the extracted data. Data quality and data protection components can be used to ensure the integrity of the extracted data. For example, in one embodiment, a data quality component can perform validation on the extracted data while the data is temporarily held in the data staging area.

[0029] According to one embodiment, once the extraction process completes the extraction, a data transformation layer can be used to initiate a transformation process that converts the extracted data into a model format that is loaded into a customer schema in the data warehouse.

[0030] According to one embodiment, the data pipeline or process can work with a data transformation layer to transform the data into a model format. A mapping configuration database can store metadata and data mappings that define the data model used by the data transformation. A data configuration user interface (UI) can facilitate accessing and modifying the mapping configuration database.

[0031] According to one embodiment, the data transformation layer can transform the extracted data into a format suitable for loading into a data warehouse customer schema, for example according to a data model. During transformation, the data transformation can perform dimension generation, fact generation, and aggregation generation, as appropriate. Dimension generation can include generating dimensions or fields for loading into the data warehouse instance.

[0032] According to one embodiment, after transforming the extracted data, the data pipeline or process may execute a warehouse load step 150 to load the transformed data into a customer schema of a data warehouse instance. After loading the transformed data into the customer schema, the transformed data may be analyzed and used in a variety of additional business intelligence processes.

[0033] Different customers of a data analytics environment may have different requirements regarding how data is categorized, aggregated, or transformed for purposes of providing data analytics or business intelligence data or for purposes of developing software analytics applications. According to one embodiment, to support such different requirements, semantic layer 180 may include data that defines a semantic model of the customer's data (which is useful for supporting users in understanding and accessing that data using commonly understood business terms) and may provide custom content to presentation layer 190.

[0034] According to one embodiment, the semantic model can be defined, for example in an Oracle environment, as a BI repository (RPD) file, with metadata that defines logical schemas, physical schemas, physical to logical mappings, aggregation table navigation, and / or other constructs that implement various physical, business model and mapping, and presentation layer aspects of the semantic model.

[0035] According to one embodiment, customers can perform modifications to the data source model to support their specific requirements, for example by adding custom facts or dimensions related to the data stored in the data warehouse instance, and the system can extend the semantic model accordingly.

[0036] According to one embodiment, the presentation layer can enable access to the data content using, for example, software analytical applications, user interfaces, dashboards, key performance indicators (KPIs), or other types of reports or interfaces, such as may be provided by products such as Oracle Analytics Cloud, or Oracle Analytics for Applications.

[0037] According to one embodiment, query engine 18 (e.g., OBIS) acts like a federated query engine to provide analytical queries, for example within the Oracle Analytics Cloud environment, via SQL, pushing down operations to supported databases and translating business user queries into the appropriate database-specific query language (e.g., Oracle SQL, SQL Server SQL, DB2 SQL, or Essbase MDX). The query engine (e.g., OBIS) also supports internal execution of SQL operators that cannot be pushed down to the database.

[0038] According to one embodiment, a user / developer can interact with a client computing device 10 that includes computing hardware 11 (e.g., processor, storage, memory), a user interface 12, and an application 14. A query engine or business intelligence server such as OBIS generally operates to process inbound, e.g., SQL, requests against a database model, construct and execute one or more physical database queries, process the data appropriately, and return the data in response to the request.

[0039] To achieve this, in one embodiment, a query engine or business intelligence server may include various components or functions, such as a logical or business model or metadata that describes the data available as the subject area of ​​a query, a request generator that receives incoming queries and converts them into physical queries for use with connected data sources, and a navigator that receives incoming queries, navigates the logical model, and generates physical queries that best return the data needed for a particular query.

[0040] For example, according to one embodiment, a query engine or business intelligence server can employ a logical model that is mapped to the data in the data warehouse by creating a simplified star-schema business model over the various data sources, allowing users to query the data as if it originated from a single source. The information can then be returned to the presentation layer as subject matter domains according to the business model layer mapping rules.

[0041] According to one embodiment, a query engine (e.g., OBIS) can process queries against a database according to a query execution plan 56, which in various embodiments can include various child (leaf) nodes, generally referred to as RqLists, and generate one or more diagnostic log entries. Within the query execution plan, each execution plan component (RqList) represents a block of a query within the query execution plan and is generally translated into a SELECT statement. An RqList can have nested child RqLists, just as a SELECT statement can select from nested SELECT statements.

[0042] According to one embodiment, during operation, the query engine or business intelligence server can create a query execution plan, which can then be further optimized, for example, to perform aggregations of the data required to respond to the request, for example, joining the data and applying further calculations before the results are returned to the calling application via the ODBC interface.

[0043] According to one embodiment, complex multi-pass requests involving multiple data sources may require the query engine or business intelligence server to decompose the query, determine which sources, multi-pass calculations, and aggregations can be used, and generate a logical query execution plan that spans multiple databases and physical SQL statements, with the results then being returned by the query engine or business intelligence server for further combination or aggregation.

[0044] FIG. 2 further illustrates an exemplary data analytics environment according to one embodiment. 2, according to one embodiment, the provisioning component may also include a provisioning application programming interface (API) 112, a number of workers 115, a metering manager 116, and a data plane API 118, as further described below. The console interface may communicate with the provisioning API by making API calls when commands, instructions, or other inputs are received at the console interface, for example, to provision services within the SaaS environment or to make configuration changes to provisioned services.

[0045] In one embodiment, the data plane API can communicate with the data plane. For example, in one embodiment, provisioning and configuration changes directed to services provided by the data plane can be communicated to the data plane via the data plane API.

[0046] In one embodiment, the metering manager may include various functions for metering services and usage of services provisioned via the control plane. For example, in one embodiment, the metering manager may record usage over time of processors provisioned via the control plane for a particular customer (tenant) for billing purposes. Similarly, the metering manager may record the amount of data warehouse storage space partitioned for use by a customer of the SaaS environment for billing purposes.

[0047] According to one embodiment, the data pipeline or process provided by the data plane may include a monitoring component 122, a data staging component 124, a data quality component 126, and a data projection component 128, as further described below.

[0048] According to one embodiment, the data transformation layer may include a dimension generation component 136, a fact generation component 138, and an aggregation generation component 140, as further described below. The data plane may also include a data configuration user interface 130 and a mapping configuration database 132.

[0049] In one embodiment, the data warehouse may include a default data analytics schema (in some embodiments, referred to herein as an analytics warehouse schema) 162 and, for each customer (tenant) of the system, a customer schema 164.

[0050] According to one embodiment, to support multiple tenants, the system may enable the use of multiple data warehouses or data warehouse instances. For example, according to one embodiment, a first warehouse customer tenancy for a first tenant may include a first database instance, a first staging area, and a first data warehouse instance of the multiple data warehouses or data warehouse instances, and a second customer tenancy for a second tenant may include a second database instance, a second staging area, and a second data warehouse instance of the multiple data warehouses or data warehouse instances.

[0051] According to one embodiment, based on the data model defined in the mapping configuration database, the monitoring component can determine dependencies of the multiple different data sets to be converted. Based on the determined dependencies, the monitoring component can determine which of the multiple different data sets should be converted to the model format first.

[0052] For example, in one embodiment, if a first model dataset does not include dependencies on other model datasets and a second model dataset includes a dependency on the first model dataset, the monitoring component may determine to transform the first dataset before the second dataset to accommodate the dependency of the second dataset on the first dataset.

[0053] For example, according to one embodiment, dimensions may include categories of data, such as, for example, "Name," "Address," or "Age." Fact generation involves generating possible values ​​for the data, i.e., generating "measures." Facts can be associated with appropriate dimensions within the data warehouse instance. Aggregation generation involves creating data mappings that compute aggregations of transformed data against existing data within customer schemas in the data warehouse instance.

[0054] According to one embodiment, once any transformations (as defined by the data model) are performed, a data pipeline or process can read the source data, apply the transformations, and push the data to a data warehouse instance.

[0055] According to one embodiment, data transformations can be expressed in rules, and once transformations have occurred, values ​​can be intermediately persisted in a staging area, and data quality and data projection components can validate and check the integrity of the transformed data before it is uploaded to customer schemas in the data warehouse instance. Monitoring can be provided as the extract, transform, load process is executed, for example, on multiple compute instances or virtual machines. Dependencies can also be maintained during the extract, transform, load process, and the data pipeline or process can handle such ordering decisions.

[0056] According to one embodiment, after transforming the extracted data, the data pipeline or process may execute a warehouse load procedure to load the transformed data into a customer schema of a data warehouse instance. After loading the transformed data into the customer schema, the transformed data may be analyzed and used in a variety of additional business intelligence processes.

[0057] FIG. 3 further illustrates an exemplary data analytics environment according to one embodiment. As shown in FIG. 3, according to one embodiment, data can be sourced, for example, from a customer's (tenant's) enterprise software application or data environment (106) or as custom data 109 sourced from one or more customer-specific applications 107 using a data pipeline process and loaded into a data warehouse instance, which in some examples includes the use of object storage 105 for storing the data.

[0058] For example, in an embodiment of an analytics environment such as Oracle Analytics Cloud (OAC), a user can create a dataset that uses tables from different connections and schemas, and the system uses the relationships defined between these tables to create relationships or joins in the dataset.

[0059] According to one embodiment, for each customer (tenant), the system pre-populates that customer's data warehouse instance based on an analysis of data in that customer's enterprise application environment and in the customer tenancy 117 using data analytics schemas maintained and updated by the system in the system / cloud tenancy 114. In this manner, the data analytics schemas maintained by the system enable a data pipeline or process to retrieve data from the customer's environment and load it into the customer's data warehouse instance.

[0060] According to one embodiment, the system also provides, for each customer of the environment, a customer schema that is easily modifiable by the customer and allows the customer to supplement and utilize data within their own data warehouse instance. For each customer, the resulting data warehouse instance acts as a database whose contents are partly controlled by the customer and partly controlled by the environment (system).

[0061] For example, according to one embodiment, a data warehouse (e.g., Oracle Autonomous Data Warehouse (ADW)) may include data analytics schemas and customer schemas for each customer / tenant sourced from its enterprise software applications or data environments. Data provisioned in a data warehouse tenancy (e.g., ADW cloud tenancy) is accessible only to that tenant while simultaneously enabling access to various, e.g., ETL-related or other, functions of the shared environment.

[0062] According to one embodiment, to support multiple customers / tenants, the system enables the use of multiple data warehouse instances; for example, a first customer tenancy may include a first database instance, a first staging area, and a first data warehouse instance, and a second customer tenancy may include a second database instance, a second staging area, and a second data warehouse instance.

[0063] According to one embodiment, for a particular customer / tenant, upon extraction of its data, a data pipeline or process may insert the extracted data into the tenant's data staging area, which may act as a temporary staging area for the extracted data. Data quality and data protection components may be used to ensure the integrity of the extracted data, for example, by performing validation of the extracted data while it is temporarily held in the data staging area. Once the extraction process completes the extraction, a data transformation layer may be used to initiate a transformation process to convert the extracted data into a model format that is loaded into the customer schema of the data warehouse.

[0064] FIG. 4 further illustrates an exemplary data analytics environment according to one embodiment. As shown in FIG. 4 , according to one embodiment, for example, the process of extracting data from a customer (tenant) enterprise software application or data environment, or as customer data sourced from one or more customer-specific applications, and loading the data into a data warehouse instance or refreshing the data in the data warehouse using the data pipeline process described above generally involves three broad stages, which are performed by ETP services 160 or processes including one or more extraction services 163, transformation services 165, and load / publish services 167, and by one or more compute instances 170.

[0065] For example, according to one embodiment, a list of view objects for extraction can be submitted, for example, via a REST call to an Oracle BI Cloud Connector (BICC) component. The extracted files can be uploaded to an object storage component, for example, an Oracle Storage Service (OSS) component, for storing the data. A transformation process takes the data files from the object storage component (e.g., OSS) and loads them into a target data warehouse (e.g., an ADW database) that is internal to the data pipeline or process and not exposed to the customer (tenant), while applying business logic. A load / publish service or process takes the data from, for example, an ADW database or warehouse, and publishes it to a data warehouse instance that is accessible to the customer (tenant).

[0066] FIG. 5 further illustrates an exemplary data analytics environment according to one embodiment. As shown in FIG. 5, which illustrates the operation of a system having multiple tenants (customers) according to one embodiment, data may be sourced from each of the multiple customer (tenant) enterprise software applications or data environments and loaded into the data warehouse instance using, for example, the data pipeline process described above.

[0067] According to one embodiment, a data pipeline or process maintains, for each of multiple customers (tenants), e.g., Customer A 180, Customer B 182, a data analytics schema that is periodically updated by the system according to best practices for a particular analytics use case.

[0068] According to one embodiment, for each of multiple customers (e.g., Customer A, Customer B), the system pre-populates the customer's data warehouse instance based on analysis of data within that customer's enterprise application environment 106A, 106B and within each customer's tenancy (e.g., Customer A tenancy 181, Customer B tenancy 183) using data analytics schemas 162A, 162B maintained and updated by the system, whereby data is retrieved from the customer's environment and loaded into the customer's data warehouse instance 160A, 160B by a data pipeline or process.

[0069] According to one embodiment, the data analytics environment also provides, for each of the environment's multiple customers, customer schemas (e.g., customer A schema 164A, customer B schema 164B) that are easily modifiable by the customer and enable the customer to supplement and utilize data within their own data warehouse instance.

[0070] As described above, according to one embodiment, for each of multiple customers of the data analytics environment, the resulting data warehouse instance operates as a database whose contents are partially controlled by the customer and partially controlled by the data analytics environment (system), including that the database will be pre-populated with appropriate data obtained from the enterprise application environment to support various analytics use cases. Once the extraction process 108A, 108B for a particular customer has completed the extraction, a transformation process can be initiated using a data transformation layer to convert the extracted data into a model format that is loaded into the customer schema of the data warehouse.

[0071] According to one embodiment, launch plans 186 can be used to control the operation of data pipelines or process services for a customer (tenant) for a particular functional area to address the specific needs of the customer.

[0072] For example, according to one embodiment, a launch plan may define a number of extract, transform, and load (publish) services or steps to be executed in a particular order, at a particular time, and within a particular time window.

[0073] According to one embodiment, each customer may be associated with its own launch plan(s). For example, a launch plan for a first customer A may determine which tables are retrieved from that customer's enterprise software application environment (e.g., an Oracle Fusion Applications environment) or determine how services and their processes are sequenced, while a launch plan for a second customer B may similarly determine which tables are retrieved from that customer's enterprise software application environment or determine how services and their processes are sequenced.

[0074] Assessment of model fit and stability According to one embodiment, the system may include means for determining model suitability and stability for model deployment and automatic model generation.

[0075] FIG. 6 illustrates a determination of model fit and stability for use in connection with a data analytics environment, according to one embodiment.

[0076] 6, according to one embodiment, the system may include one or more data models 230. Packaged (out of the box, initial) models 232 may be used to provide packaged content 234 to load data from a customer's enterprise software application or data environment into the data warehouse instance based on the use of ETL or other data pipelines or processes as described above, and the packaged models may then be used to provide the packaged content to a presentation layer 240. Custom models 236 may be used to extend the packaged models or provide custom content 238 to the presentation layer.

[0077] According to one embodiment, the presentation layer can enable access to the data content using software analytical applications, user interfaces, dashboards, key performance indicators (KPIs) 242, or other types of reports or interfaces, such as may be provided by products such as Oracle Analytics Cloud or Oracle Analytics for Applications.

[0078] As further shown in FIG. 6 , in one embodiment, the system may include a model suitability and stability component 250 that may provide one or more functions to support model selection 252, use of model deployability scores and deployability flags 254, and mitigation of model drift risk 256 to determine model suitability and stability for a particular application, as described below.

[0079] In one embodiment, for a client's business need that requires the automatic generation of new models that account for changes over time and across departments, the system can automatically filter through thousands of potential model candidates using appropriate metrics to find the most important actionable insights based on predictions, without the need for human intervention.

[0080] Model Scoring and Selection As described above, in one embodiment, the system includes a model suitability and stability component that can provide one or more features to support model selection in order to determine model suitability and stability for a particular application.

[0081] For example, model selection decisions are important in binary classification problems such as whether a customer pays their accounts receivable on time. In such environments, different classes of metrics can be used to determine the suitability of a model.

[0082] According to one embodiment, a first class of metrics addresses the problem of skewed probability bins, which are generated by different algorithms without calibration and tend to weight the upper (e.g., p=[0.8,0.9]) and lower (e.g., p=[0.1,0.2]) parts of the distribution, or are unevenly distributed such that the highest probability bin (e.g., p=[0.9,1]) has a lower proportion of cases than successively lower probability bins that have a higher proportion of cases, or may have an uneven sawtooth pattern.

[0083] Model success criteria When a well-calibrated model is deployed, the expectation is that instance membership in the probability bins will drop steadily and rapidly (e.g., exponentially) from higher to lower bins, indicating that the model is classifying most cases with high confidence and only a few cases with low confidence.

[0084] According to one embodiment, to remove and deploy only such models, the system employs a metric to find models that meet the above criteria and to remove models that exhibit a sawtooth frequency of instances within the probability bins.

[0085] Probability bin-based scores FIG. 7 illustrates an example comparison of probability scores for different models, according to one embodiment.

[0086] As shown in FIG. 7, according to one embodiment, the score can be based on probability bins, with correct classification decreasing rapidly from higher probability bins to lower bins.

[0087] According to one embodiment, as shown in FIG. 7, scores are generated for two different models, namely models 710 and 720. Each of models 710 and 720 is an example of a model that can be used to determine whether an invoice is paid or not. As shown, the models are divided into 10 probability bins. The scoring models are shown with the number of correct classifications as well as the number of misclassifications, and the weighting of each associated scoring mechanism is also provided. As shown, model 710 has an approximately linear falloff between each probability bin, while model 720 has an exponential falloff from high probability (0.9-1) to low probability.

[0088] According to one embodiment, a result score 711 and 721 for each model can be determined, which indicates that models with an exponential drop in probability are scored higher, indicating a good model that predicts the correct outcome with a high probability.

[0089] According to one embodiment, the exemplary scoring function shown below represents a class of functions with a modified staircase shape, with a descending penalty for non-decreasing number of correctly classified instances from high to low probability bins, and a penalty over all bins for misclassification, normalized by the total number of instances classified:

[0090]

number

[0091] According to one embodiment, a system programmed according to (Equation 1) above takes into account:

[0092]

number

[0093]

number

[0094] According to one embodiment, another example of a scoring function can be shown as follows:

[0095]

number

[0096] FIG. 8 illustrates a process or method for determining model fit and stability according to one embodiment.

[0097] As shown in FIG. 8, according to one embodiment, a score can be determined for a given model of a data set. Once the model generates a probability (e.g., the probability that an invoice will be paid or not), the model output can be collected into probability "bins", i.e., groupings of ranges of probabilities. For example, if the model output is grouped into 10 probability bins, the bin ranges are 0-0.1, 0.1-0.2, 0.2-0.3, 0.3-0.4, 0.4-0.5, 0.5-0.6, 0.6-0.7, 0.7-0.8, 0.8-0.9, and 0.9-1.0. The model can be examined by comparing the model output with the actual outcome (e.g., whether the invoice was actually paid or not) to determine the number of correct classifications and misclassifications for each probability bin.

[0098] It should be noted that, according to one embodiment, the examples discussed and illustrated herein utilize 10 probability bins to demonstrate the scoring process described herein, although more or fewer probability bins may be utilized (e.g., 100 probability bins, with each probability bin covering a probability range of 0.01).

[0099] According to one embodiment, in step 810, the scoring process determines the successive differences of correct classifications and can apply weights to each successively lower probability bin for the low probability bins. The weights applied to each probability bin can be generated automatically, for example, if the model predicts an outcome with a high probability, the high probability bins can be weighted more heavily so that more importance is placed on the model being correct.

[0100] According to one embodiment, the scoring process may then, in step 820, apply a penalty for each classification failure.

[0101] According to one embodiment, in step 830, the scoring process can apply weights to the penalties evaluated in step 820 for each probability bin. The weights can be higher, and even exponentially higher, for high probability bins, as in step 810. Such penalty weights can be generated automatically as well. A high penalty is applied to a failed classification of a high probability bin, since a misclassification of a high probability bin should similarly reduce the score compared to a failed classification in a low probability bin.

[0102] According to one embodiment, in step 840, the scoring process may normalize the generated score by the number of classified samples, i.e., for example, normalization may be dividing the generated score by the number of samples.

[0103] According to one embodiment, in step 850, the scoring process may optionally consider other possibilities, for example by Monte Carlo simulation, to weed out low quality scoring techniques.

[0104] Deployability score / Deployability flag As described herein, in one embodiment, the system includes a model suitability and stability component that can provide one or more functions to support the use of a model's deployability score and deployability flag to determine model suitability and stability for a particular application.

[0105] According to one embodiment, the following approach can be used to determine the deployability score / deployability flag:

[0106]

number

[0107] Now, a system programmed according to (Equation 2) above takes into account:

[0108]

number

[0109] According to one embodiment, the deployability score (ψ) is on a scale of -10 to +10: for perfect classification, ψ is above 10, and for complete misclassification, ψ is below -10.

[0110] According to one embodiment, the model deployability flag may be defined based on the Heaviside step function as follows:

[0111]

number

[0112] Now, a system programmed according to (Equation 3) above takes into account:

[0113]

number

[0114] According to one embodiment, the deployability score can be achieved as follows: The system can use the Matthews Correlation Coefficient (MCC), as shown below in Equation 5, as a proxy for all other measures of correct classification, as precision, recall, accuracy, and F1 score are all accounted for by the MCC.

[0115] The probability bin score is a prerequisite for model hygiene and, once above a base threshold, adds to the overall deployability.

[0116] After going beyond the basic hygiene factor, the deployability score is highly correlated with the MCC and improves along with the probability bin score.

[0117] If the deployability score exceeds the threshold in (Equation 3) above, the system can consider the model deployable.

[0118] For deployment of the initial model, the system may determine that the deployability score > T above.

[0119]

number

[0120] According to one embodiment, to check the deployability of a new model that follows from the original model, the system is deployable as long as the new model continues to have a deployability score >T and is no worse than 1x the original deployability score as judged by the human. This roughly corresponds to a shift of 0.1 or less in the MCC, F1 score, and Area Under the Curve of the Receiver Operator Characteristic (AUC of ROC).

[0121] According to one embodiment, a second class of known metrics can be used to determine how well the classes are differentiated, as determined by the relative counts of true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), such as an F1 score where type I and type II errors are weighted equally.

[0122] According to one embodiment, the described approach allows a customer to choose to weight recall relative to precision, where if the customer wants to prioritize recall over precision, β can be set greater than 1, and if the customer wants to prioritize precision over recall, β can be set less than 1:

[0123]

number

[0124] However, these F-measures and related measures are skewed due to class imbalance, especially when the actual class of interest, such as non-payment cases, is rare relative to paid cases. To address this class imbalance issue, the system can filter the model by the Matthews Correlation Coefficient (MCC).

[0125]

number

[0126] The above judgement is close to 1 for perfect correct classification, close to -1 for misclassification, and close to 0 for random classification. According to one embodiment, models that meet the above scores and have an MCC above 0.5 can be accepted. The Matthews correlation coefficient also applies well to multi-class classification cases.

[0127] Reducing the risk of model drift As described above, in one embodiment, the system includes a model suitability and stability component that can provide one or more functions to support mitigation of model drift risk in order to determine model suitability and stability for a particular application.

[0128] According to one embodiment, model drift risk can be mitigated using model stability detection. Although model accuracy metrics can change significantly due to drifts in training and testing distributions, the system and method do not simply use model accuracy metrics as a criterion for model selection. As the input distribution, or the distribution of a sample of the population taken on a particular day or week, changes, it is expected to see significant drift in the decision boundary in the new model, even to the point of reversing classification in multiple instances, such as classifying an invoice classified as likely to have been paid yesterday as likely not to be paid today.

[0129] According to one embodiment, the system and method should expect changes in the forecast as new data for the same invoice comes in, but should not expect significant changes in the forecast for the same invoice if the independent variables remain substantially the same compared to previous periods.

[0130] However, according to one embodiment, decision boundary shifts can occur when there are significant shifts in the training distributions over time. These shifts can be explicitly detected and addressed to the end user, for example when two distributions diverge substantially from each other such that their measures of central tendency and dispersion are statistically significantly different.

[0131] According to one embodiment, if the system detects that the model is unstable enough that the decision boundary drifts significantly (e.g., from day to day), this indicates multiple problems in model fitting, where classification decisions continue to change from day to day, even reversing the previous day's prediction, even though the data for the individual instances remains unchanged.

[0132] According to one embodiment, the approach described herein can be used to evaluate the stability of the model using a sensitivity metric such that small random perturbations (less than 5% of the standard deviation of the independent variables) are made in a portion of the class instances of interest, and if a significant shift in classification is detected, it can be concluded that a model instability scenario has been reached, or the system and method may be dealing with instances close to the decision boundary. The system can use a normalized distance measure to distinguish between instances close to the decision boundary and cases inside a cluster of instances for a given classification.

[0133] According to one embodiment, we expect to see instability even for instances close to the center of the class cluster, as there are large jumps in the change in classification probability.

[0134] According to one embodiment, the system and method can determine and examine how much the scoring distribution has shifted from the training distribution and how much the shift between the training distributions has occurred over time. To this end, the described approach can use a combination of two scores:

[0135] Model and distribution drift: According to one embodiment, the reduction in F1 score (a measure of accuracy) and Matthews correlation coefficient (MCC) are direct indicators of drift, and whenever the F1 score falls below a threshold (e.g., 0.6) or the MCC falls below a bound (e.g., 0.35), the system can automatically raise an alarm flag to request retraining of the model. Kullback-Leibler Divergence or Bhattacharya distance type measures can be evaluated to determine the shift in the distribution of the input independent variables from the training dataset to the scoring dataset, thereby determining how far the input distribution has deviated from the past training data.

[0136] Model Stability: According to one embodiment, the described approach can be used to provide a scoring mechanism for changes in classification despite negligible changes in the input independent variables.

[0137] FIG. 9 further illustrates a process or method for determining model fit and stability, according to one embodiment.

[0138] As shown in Figure 9, in one embodiment, a process can be utilized to determine if a model is drifting and requires relaxation. This process can also be used to determine risks to the stability of the model.

[0139] For example, the process in Figure 9 can be used to determine if the model is shifting / flipping predictions (e.g., flipping the number of predictions from "paid" to "not paid" from one day to the next - which may be a sign of model instability or degradation).

[0140] According to one embodiment, in step 910, the process can detect one or more signals of model degradation under distribution drift. For example, the process can track MCC and AUC scores to determine if scores are deteriorating. Losses above a threshold can be considered as the model drifting or drifting significantly (e.g., a loss threshold of 0.1 or more). Additionally, the process can evaluate Kullback-Leibler Divergence (also known as relative entropy) or Bhattacharya Distance type measures to determine a shift in the distribution of input independent variables from the training dataset to the scoring dataset.

[0141] According to one embodiment, in step 920, the process may begin a model stability detection and scoring process.

[0142] According to one embodiment, in step 930, the process may determine the distance of each instance (e.g., invoice) from its nearest (e.g., 30 nearest) clusters having the same pre-classification. The distance may be calculated, for example, by determining the Mahalanobis distance of each invoice or instance from its nearest 30 nearest clusters having the same pre-classification.

[0143] According to one embodiment, in step 940, if the process determines that at least one or more of these nearest neighbors reverses the classification in the new version of the model, it may add it to the count of reversed classifications.

[0144] According to one embodiment, in step 950, the process may determine the percentage or proportion of such reverse classifications out of the total number of instances classified.

[0145] According to one embodiment, in step 960, if such reversal classification exceeds a threshold (e.g., 2 percent of the total number of instances without a corresponding increase in MCC), the process may flag the model as being slightly unstable.

[0146] According to one embodiment, in step 970, if such reversal classification exceeds a second threshold (e.g., 10% of the total number of instances without a corresponding increase in MCC), the process flags the model as unstable.

[0147] According to one embodiment, the above-mentioned thresholds may be set, modified, and / or changed based on input received at the system, for example, by a user or administrator.

[0148] According to one embodiment, the described approach uses a Mahalanobis distance-based measure of standard deviation normalized distance between invoices or instances by converting all numerical independent variables (e.g. amount, days overdue, number of follow-ups) into z-scores, converting all categorical independent variables (e.g. customer industry, location, invoice type, invoice line item type) into entropy coded renormalized z-scores, and finding the Euclidean distance (if the covariance matrix is ​​an identity matrix) between the current invoice and a cluster of different invoice types or customer types.

[0149] For example, according to one embodiment, if an invoice's distance from paid invoices is large compared to its unpaid invoices, the process can assign it to a high risk category. As shown in Figure 10, the system can present the user with a list of invoices sorted by risk.

[0150] FIG. 10 is a diagram illustrating a sorted list of invoices according to one embodiment. An exemplary screenshot 1000 may be provided, for example, via the system's user interface, as shown in Figure 10. Based on the model selected resulting from the scoring system described above, various metrics may be provided via the user interface, including, but not limited to, the top 10 invoices by risk and their amounts, the top 10 invoices paid and their amounts, the top 20% invoices and total amount at risk, the top 20% invoices and total amount paid, etc.

[0151] FIG. 11 illustrates the output of a model for analyzing data according to one embodiment. As shown in FIG. 11, an exemplary screenshot 1100 may be provided, for example, via a user interface of the system. Based on the model selected due to the scoring system described above, various metrics related to the probability bins may be provided via the user interface. The system may generate such a chart by creating equal bins of probability intervals and correlating (e.g., Pearson's correlation) all numeric variables with the column of bins. The top number of variables that are correlated with each other (e.g., 5) may then be determined.

[0152] After such a determination, according to one embodiment, the system can determine whether the bin means of these variables differ from the overall population mean by at least a certain percentage (e.g., 50%). If the bin mean differs from the population mean by at least, e.g., 50%, then this variable can be displayed along with a list of explanations.

[0153] FIG. 12 is a flowchart of a method for determining model suitability and stability of model deployment in automatic model generation according to one embodiment.

[0154] According to one embodiment, in step 1210, the method may provide a computer including one or more microprocessors and a data analytics cloud or other computing environment operating on the computer.

[0155] According to one embodiment, in step 1220, the method can provide the multiple models in a data analytics cloud.

[0156] According to one embodiment, in step 1230, the method may score a set of models based on a set of data in the data analytics cloud.

[0157] According to one embodiment, in step 1240, the method may select a model from a set of models based on the scoring.

[0158] According to one embodiment, in step 1220, the method may monitor the model for signs of instability or drift.

[0159] According to various embodiments, the teachings herein may be conveniently implemented using one or more conventional general-purpose or specialized computers, computing devices, machines, or microprocessors, including one or more processors, memory, and / or computer-readable storage media programmed according to the teachings of the present disclosure. Appropriate software coding may be readily produced by skilled programmers based on the teachings of the present disclosure, as will be apparent to those skilled in the software arts.

[0160] In some embodiments, the teachings herein may include a computer program product that is one or more non-transitory computer-readable storage media having instructions stored thereon that can be used to program a computer to perform any of the processes of the present teachings. Examples of such storage media may include, but are not limited to, hard disk drives, hard disks, hard drives, fixed disks or other electromechanical data storage devices, floppy disks, optical disks, DVDs, CD-ROMs, microdrives, and magneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flash memory devices, magnetic or optical cards, nanosystems, or other types of storage media or devices suitable for non-transitory storage of instructions and / or data.

[0161] The foregoing description has been provided for purposes of illustration and description and is not intended to be exhaustive or to limit the scope of protection to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art. For example, although some of the examples provided herein illustrate use in a cloud environment such as Oracle Analytics Cloud, according to various embodiments, the systems and methods described herein can be used in other types of enterprise software applications, cloud environments, cloud services, cloud computing, or other computing environments.

[0162] The embodiments have been chosen and described in order to best explain the principles of the present teachings and their practical application, so that those skilled in the art can appreciate the various embodiments and embodiments with various modifications suited to the particular use contemplated. It is intended that the scope be defined by the appended claims and their equivalents.

Claims

1. 1. A system for determining model suitability and stability for model deployment in automatic model generation, comprising: A computer including one or more microprocessors and a data analytics cloud or other computing environment operating on the computer, The one or more microprocessors include: providing a plurality of models in the data analytics cloud; Scoring the set of models based on a set of data in the data analytics cloud; selecting a model from the set of models based on the scoring; The system operates to monitor the model for signs of instability or drift.

2. Scoring the set of models includes, for each of the set of models: automatically assigning predictions of the model to probability bins of a set of probability bins; determining successive differences of correct classification between successive probability bins; applying a weight to each of the successive differences of correct classification between successive probability bins; The system of claim 1 , wherein the weight applied to each successive difference of a correct classification depends on the probability bin to which it is applied.

3. The system of claim 2 , wherein the weights are greater for more probable bins.

4. Scoring the set of models further comprises, for each of the set of models: applying a penalty for each failure to classify a probability bin; and applying a penalty weight to each of the applied penalties for each failed classification.

5. The system of claim 4 , wherein the penalty weight is greater for more highly probable bins.

6. Scoring the set of models further comprises, for each of the set of models: The system of claim 5 , further comprising normalizing the generated score by the number of samples classified.

7. Monitoring the model for signs of instability or drift includes: Detecting one or more signals of degradation of the model; determining a distance of each instance generated by the model to a cluster of instances having the same prior classification; determining that at least one or more of the nearest neighbors in the new version of the model has a reverse classification; determining a proportion of the reversed classification out of a total number of instances generated by the model; tagging the model as slightly unstable if the determined proportion exceeds a first threshold; and tagging the model as unstable if the determined proportion exceeds a second threshold.

8. 1. A method for determining model suitability and stability for model deployment in automatic model generation, comprising: Providing a computer including one or more microprocessors and a data analytics cloud or other computing environment operating on said computer; providing a plurality of models in the data analytics cloud; The computer scores the set of models based on a set of data in the data analytics cloud; and the computer selecting a model from the set of models based on the scoring; and and the computer monitoring the model for signs of instability or drift.

9. Scoring the set of models includes, for each of the set of models: automatically assigning predictions of the model to probability bins of a set of probability bins; determining successive differences of correct classification between successive probability bins; applying a weight to each of the successive differences of correct classification between successive probability bins; The method of claim 8 , wherein the weight applied to each successive difference of a correct classification depends on the probability bin to which it is applied.

10. The method of claim 9 , wherein the weights are greater for more probable bins.

11. Scoring the set of models further comprises, for each of the set of models: applying a penalty for each failure to classify a probability bin; and applying a penalty weight to each of the applied penalties for each failed classification.

12. The method of claim 11 , wherein the penalty weight is greater for higher probability bins.

13. Scoring the set of models further comprises, for each of the set of models: The method of claim 12 , comprising normalizing the generated score by the number of samples classified.

14. Monitoring the model for signs of instability or drift includes: Detecting one or more signals of degradation of the model; determining a distance of each instance generated by the model to a cluster of instances having the same prior classification; determining that at least one or more of the nearest neighbors in the new version of the model has a reverse classification; determining a proportion of the reversed classification out of a total number of instances generated by the model; tagging the model as slightly unstable if the determined proportion exceeds a first threshold; and tagging the model as unstable if the determined proportion exceeds a second threshold.

15. A program for causing a computer to execute the method according to any one of claims 8 to 14.

Images