Self-Learning Analytics Solution Core

Abstract

Example implementations described herein are directed to a system with multiple sensors associated with multiple devices, the system including: generating multiple meta-pipeline stages to receive a hypothesis canvas and metadata from the multiple sensors, the multiple meta-pipeline stages having the metadata and one or more stages, each of the one or more stages being an analytical application or machine learning application stacked in an execution order based on the metadata and the hypothesis canvas; and running the multiple meta-pipeline stages in the execution order on the metadata to generate an analytical model of the system.

Description

[Technical field]

[0001] The present disclosure is directed to a data analytics platform, and more specifically, to a system and method for facilitating a self-learning analytics solution core. [Background technology]

[0002] In the related art, model creation is a very difficult problem for data analytics, in part due to a resource gap: data scientists and subject matter experts are increasingly rare positions within companies.

[0003] Related techniques have problems with encouraging systems to learn from previously trained models. During a typical data science process, data scientists try several types of models using multiple hyperparameters. This process demands a significant amount of time from data science teams, and in the end, data scientists only retain a few models that are created. Any learning from these models that did not work is discarded, even if they could potentially be useful in building models later. Because no model history is kept, it is impossible to identify what is likely to work in future analyses.

[0004] In the related art, one of the challenges in analysis is to create a model using business requirements. The process of understanding the business problem, translating it into hypothesis testing, and applying the appropriate pipeline is complicated by the infinite possibilities of stages. A stage is one sequential step of the analysis pipeline in a machine learning pipeline.

[0005] Related techniques have problems capturing lessons from previous feature engineering experiments. Data scientists spend a significant amount of time preparing data. All lessons learned from feature engineering and feature selection go to waste because this learning is not stored. Preserving these meta-feature lessons can be important for use in future models. Summary of the Invention [Problem to be solved by the invention]

[0006] To address the problems of the related art, example implementations include a framework that includes a collection of tools and procedures. To facilitate systems that learn from previously trained models and create models using business requirements, example implementations described herein incorporate meta-learning to close those resource gaps and build analytical solutions into products.

[0007] In an exemplary implementation, natural language processing (NLP) is used to create a numerical representation of the hypothesis canvas (text) of the Analytic Solution Core (ASC) and store it in a meta-learning database (DB), which stores the meta-pipeline (used in the previous use case) and is composed of one or more meta-stage components including a collection of analytical applications and machine learning applications. An exemplary implementation includes creating a database that can store all information related to the modeling phase. In the data science cycle, the modeling phase is when data scientists try several different features and models to determine which model is best suited to solving the problem. The database stores all these experiments and uses them in generating future analyses, thereby transforming previous experiments into potential assets for the future.

[0008] The main goal of this solution is to preserve models that may work for future analyses, thereby saving time and resources.

[0009] Furthermore, in response to new demands for solutions, NLP uses probabilistic search to find the likelihood of different sequences of text by using numerical representations of pairs of words (e.g., low likelihood is dissimilar words and high probability is similar words) and distance methods to calculate the similarity of the pairs in a meta-earning DB using the input information.

[0010] The creation of a data science project requires data engineering, data science, and subject matter expertise. Each of these three roles makes a vital contribution to a data science project: data scientists provide business, mathematics, and coding expertise, subject matter experts contribute domain knowledge of the use cases, and data engineers support data scientists and subject matter experts in the data engineering portion of the project. An exemplary implementation enables the creation of a data science project that encapsulates the knowledge of data scientists, subject matter experts, and domain experts in the system. An exemplary implementation uses a hypothesis business canvas, an artifact that serves as input with all the business requirements needed to build an analytical solution.

[0011] The configurability of the analytics solution core solves the problem of lack of reusability of analytics artifacts. Currently, analytics artifacts are not reused in automated procedures. Data scientists need to design the analytics pipeline to reuse stages. Analytics components are connected to build an ASC. This ASC is generated using components and the solution picks the right stage using the hypothesis canvas as the context of the analysis. For example, if the use case is churn analysis and the user wants to use transaction information to perform churn analysis, the dataset for transformation may include recency date, frequency, and amount of purchase from a transaction dataset that encodes category features to create a representation of customer products. This representation becomes the input for some machine learning algorithm to train.

[0012] If a suitable solution core cannot be found, the closest meta-stage is selected from the DB, and the selected components are combined to become a new meta-pipeline ASC using a simulation process that has the ability to learn new tasks and learn which is the best learning method. After learning the best methods, the system applies those methods in future models. All combinations of ASCs are stored in the meta-learning DB.

[0013] In related technology implementations, data science projects discard unused models after data scientists pick a final model. The main goal here is to store all information from all generated models as a reference for future models. Preserving this information may be useful for new analytical experiments.

[0014] The exemplary implementations described herein aim to provide an end-to-end solution for business users: a hypothesis business canvas provides all the requirements necessary for data scientists to build layers that connect to a natural language understanding (NLU) and semantic matching layer that transforms business use cases in an analytics pipeline using analytics solution core properties such as reusability and composability. [Means for solving the problem]

[0015] Aspects of the disclosure include a method for a system including a plurality of sensors associated with a plurality of devices, the method including: generating a plurality of meta-pipeline stages to receive a hypothesis canvas and metadata from the plurality of sensors, the plurality of meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being an analytical application or machine learning application stacked in an execution order based on the metadata and the hypothesis canvas; and executing the plurality of meta-pipeline stages in an execution order on the metadata to generate an analytical model of the system.

[0016] Aspects of the disclosure include a computer program storing instructions for a system including a plurality of sensors associated with a plurality of devices, the instructions including: generating a plurality of meta-pipeline stages to receive a hypothesis canvas and metadata from the plurality of sensors, the plurality of meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being an analytical application or machine learning application stacked in an execution order based on the metadata and the hypothesis canvas; and executing the plurality of meta-pipeline stages in the execution order on the metadata to generate an analytical model of the system. The computer program may be stored in a non-transitory computer-readable medium and executed by one or more processors.

[0017] Aspects of the disclosure include a system including a plurality of sensors associated with a plurality of devices, and means for generating a plurality of meta-pipeline stages to receive a hypothesis canvas and metadata from the plurality of sensors, the plurality of meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being an analytical application or machine learning application stacked in an execution order based on the metadata and the hypothesis canvas, and means for executing the plurality of meta-pipeline stages in an execution order on the metadata to generate an analytical model of the system.

[0018] An aspect of the disclosure includes a management device for a system including a plurality of sensors associated with a plurality of devices, the management device including a processor configured to: generate a plurality of meta-pipeline stages to receive a hypothesis canvas and metadata from the plurality of sensors, the plurality of meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being an analytical application or machine learning application stacked in an execution order based on the metadata and the hypothesis canvas; and run the plurality of meta-pipeline stages in an execution order on the metadata to generate an analytical model of the system. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 illustrates an example of various roles working together as a value chain for an ASC, according to an example implementation.

[0020] [Diagram 2] FIG. 2 is a diagram illustrating an example architecture of a system according to an example implementation.

[0021] [Diagram 3]FIG. 3 illustrates an example of a data scientist curation flow for initiating a process, according to an example implementation.

[0022] [Figure 4] FIG. 4 illustrates an example of an ASC canvas from ASC continuous anomaly monitoring and location identification for an asset in motion, according to an example implementation.

[0023] [Diagram 5] FIG. 5 illustrates an example of a hypothesis canvas from continuous anomaly monitoring and location identification for a moving asset, according to an example implementation.

[0024] [Figure 6] FIG. 6 is a diagram illustrating an example of system awareness according to an example implementation.

[0025] [Figure 7] FIG. 7 illustrates an example of a meta-learning subsystem process according to an example implementation.

[0026] [Figure 8] FIG. 8 is a diagram illustrating an example of a classification tree in accordance with an example implementation.

[0027] [Figure 9] FIG. 9 illustrates an example of a meta-learning process according to an example implementation.

[0028] [Figure 10] FIG. 10 illustrates a reinforcement learning (RL) process according to an example implementation.

[0029] [Figure 11] FIG. 11 illustrates an example meta-pipeline in accordance with one example implementation.

[0030] [Figure 12]FIG. 12 illustrates a system including multiple systems with connected sensors and a management device, according to an example implementation.

[0031] [Figure 13] FIG. 13 illustrates an example computing environment with an example computing device suitable for use in some example implementations. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] The following detailed description provides details of the drawings and exemplary implementations of the present application. For clarity, reference numbers and descriptions of redundant elements between the drawings have been omitted. Terms used throughout the specification are provided by way of example and are not intended to be limiting. For example, the use of the term "automatic" may include fully automatic or semi-automatic implementations with user or administrator control over certain aspects of the implementation, depending on the desired implementation of the person skilled in the art implementing the implementation of the present application. Selections may be made by a user using a user interface or other input means, or may be implemented using a desired algorithm. The exemplary implementations as described herein may be utilized alone or in combination, and the functionality of the exemplary implementations may be implemented by any means according to the desired implementation.

[0033] Analytical code emerges from customer engagement and exhibits useful relevance from a market perspective. To describe the value chain around the Analytical Solution Core (ASC), three main characters are involved: Producers, Curators, and Consumers. Producers include entities that create analytical use cases. Producers create core artifacts and provide those artifacts to Curators. Curators are entities that select, standardize, and improve the quality of the artifacts provided by Producers. Consumers are at the end of the value chain and include entities that consume the artifacts after Curators apply transformation procedures.

[0034] FIG. 1 illustrates an example of various roles working together as a value chain for an ASC, according to one example implementation.

[0035] When an analytical project is required, it is deployed to the system by producers 100, which include a team of data scientists who create a business use case, including a financial and legal profile of the project, such as determining the need for non-disclosure and intellectual property clauses. The data scientists work on building artifacts, including, among other things, a hypothesis canvas and analytical code. At the end of the project, some artifacts are ready to be transformed in the ASC.

[0036] After the project is finished, the Data Science team delivers the customer artifacts to Curator 101, which may include the ASC team, who prepares the artifacts, ensures everything meets the ASC specifications and requirements, controls the quality of the code, and refines the code as needed.

[0037] After the curation process, the artifacts are ready to be deployed for the consumers 102. The producer 100 may have some analytical engagements with the customers. From these engagements, they can test ideas with domain experts, develop relevant analytical code, and convert it into ASC. In converting the code into ASC, the first step for the data scientist is to standardize the code. For this process, the data scientist needs to create artifacts such as python language, hypothesis canvas, original dataset, business documentation, technical documentation, hypothesis canvas template, and ASC canvas template.

[0038] FIG. 2 illustrates an example architecture of the system according to one example implementation. Specifically, FIG. 2 illustrates how subgroups participate in a unified process flow to support an example implementation. As described in FIG. 1, the producer 100 receives analysis projects from customer engagements 200 that are entered into a user interface 201. The curator 101 provides coding refactors, specification definitions, and unit tests 202 to the web service metadata 203. The metadata 203 can be published in the ASC catalog 207 as artifacts 213 via an administrator user interface 204 of the meta-learning system 210.

[0039] The consumer 102 communicates with the meta-learning system 210 via a chatbot 205. The chatbot 205 establishes functionality and interfaces that facilitate natural language processing (NLP) for the hypothesis canvas contextualization 206 to facilitate communication between the consumer 102 and the meta-learning system 210. The meta-learning 210 performs semantic matching 211 on the output of the NLP for the hypothesis canvas contextualization 206 to provide to the interface 212 for querying the data catalog 215. Further description of the elements of the meta-learning 210 is provided below with respect to FIG.

[0040] FIG. 3 illustrates an example of a data scientist curation flow for initiating a process, according to an example implementation.

[0041] The data scientist receives artifacts 213 as "analysis projects from customer engagements." These artifacts 213 may include, but are not limited to, code in the Python language, a hypothesis canvas, an original dataset, documentation with a description of the business part of the use case (processed in the hypothesis canvas template), documentation with a description of the technical part of the use case (processed in the ASC canvas template), etc., depending on the desired implementation.

[0042] With this documentation, the producer 100 submits all packages to a code repository for the curator 101 to start working with it using the UI / UX interface 204. The main purpose of the curator 101 is to transform the code from a series of code pieces into a standardized sequential logic to form pipelines and stages. The curator 101 then validates all the work previously done by the producer 100. The curator 101 performs several main processes:

[0043] The first major process is called "Coding Refactor Specification Definition and Unit Test" group 202, which includes the following procedures:

[0044] 1. Convert analysis codes to ASC standards for publishing to the ASC catalog 207.

[0045] 2. Remove the dependency on the client data source system.

[0046] 3. Include a test dataset and check.

[0047] 4. Includes preparation of training data, and testing and validation of the training data.

[0048] 5. Perform quality assurance measures for your code.

[0049] 6. Test the code for horizontal and vertical applications.

[0050] 7. Create a YML format for processing by YAML.

[0051] The second major process focuses on preparing documentation for the ASC, which includes the following procedures:

[0052] 1. Inspect the ASC model documentation.

[0053] 2. Examine the code comments and standards.

[0054] 3. Rewrite the code from python script format to Jupyter Notebook format.

[0055] 4. Examine the ASC Development Canvas.

[0056] 5. Document the user journey integration.

[0057] 6. Inspect the hypothesis development canvas.

[0058] 7. Create ASC training documentation.

[0059] 8. Inspect and verify training data documentation.

[0060] The third major process is a set of procedures focused on the code enrichment family, which includes methods to enrich the analysis code with additional features.

[0061] 1. Enhancing the original analytical code by adding new features that improve code performance in terms of model processing time and / or accuracy.

[0062] 2. Compliance and software license checks and revisions, if necessary.

[0063] One of the main goals, shown in Figure 3, is to prepare the code and artifacts for future phases, including:

[0064] 1. Creating a YAML file: A data scientist team member of the producer 100 creates a YAML file based on the tool specifications. This YAML file has all the guidelines the application needs to launch and run the ASC.

[0065] 2. Refactoring Python Code: As shown in Figure 1, a data scientist team member of producer 100 converts the Python code into ASC's Python code standard.

[0066] 3. Validation of metadata information from the ASC: Validation of metadata information from the ASC is utilized by the web service metadata 203. This information can be used to perform a contextual search instead of a regular search in a database.

[0067] 4. Creating Unit and Integration Tests: Creating unit tests to test blocks, stages, and pipelines within ASC.

[0068] The above three processes and metadata information are the basic information of the Web service metadata 203. The contents are required for the following artifacts:

[0069] 1. YAML files (pipelines and stages): A set of files in YAML format that have information about pipelines and stages.

[0070] 2. Python file: A Python file with the ASC code.

[0071] 3. Use Case Metadata: Use case information with reference details for the ASC such as the industry, the problem to be solved, and the specifications of the machinery used. The ASC Canvas is discussed below.

[0072] The ASC and the hypothesis canvas are two artifacts that contain different metadata about the technical components of the analysis runtime environment, as well as information about the business use case. Both canvases feed the Natural Language Understanding (NLU) and NLP processes, providing the technical e-business context of the ASC to the system.

[0073] FIG. 4 illustrates an example of an ASC canvas from ASC continuous anomaly monitoring and location identification for an asset in motion, according to an example implementation.

[0074] In an exemplary ASC canvas, there are various sections of content. Such sections may include:

[0075] Model Description 401 provides a brief description of the model and its applicability. Algorithm Library 402 includes a list of libraries and applications required to run the ASC. Software and Hardware Technical Requirements 403 includes a list of resources required to process and store data. License Variables 404 includes a list of licenses for the libraries. Potential Use Cases 405 includes a description of potential use cases that will work with this model. Variables and Data Sources 406 includes a list of variables and data sources required to process the algorithms. Feature Engineering Requirements 407 includes a list of features required to process the raw data and provide more meaningful information to the model. Module Metrics / KPIs 408 includes a list of metrics used to measure the performance of the model. Application Programming Interface (API) (Output) 409 includes a description of the output of the ASC. Operational Comments 410 includes additional comments that are important to the ASC.

[0076] In an example implementation, the datasets used by the ASC to train, test, and infer model features for the following elements: Data schema includes the definition of the dataset, including a description of each column, column names, types of variables, relationships between tables, etc. Data content includes the content of the data.

[0077] 5 illustrates an example of a hypothesis canvas from continuous anomaly monitoring and location identification for an asset in motion, according to an example implementation. Specifically, the hypothesis canvas provides detailed information about the business data of the ASC.

[0078] To create a hypothesis canvas, the interface provides the following elements for input: Hypothesis 501 defines the hypothesis. Key Performance Indicators (KPIs) 502 defines the KPIs by which the progress and success of the hypothesis is measured. Business Value 503 defines the business value of the hypothesis. Stakeholders 504 represent the components that are impacted by the hypothesis. Entities 505 defines the entities for building the analysis. Decisions 506 defines the most important decisions in support of the hypothesis. Predictions 507 defines the predictions that support the decisions.

[0079] Data sources 508 define the data sources to be used. Data sources 508 may also define the score required for the data sources to support the hypothesis (e.g., from 1 being the least valuable to 4 being the most valuable). Variables (magnitude) 509 define the variables that may yield better predictors of performance. Recommendations 510 indicate the recommendations required to support the decision. Barriers 511 include technical, data, and organizational barriers to success. Risk 512 indicates the cost or risk associated with false positives and false negatives. Financial assessment 513 includes a metric of the gain of the hypothesis and the score by which the hypothesis is affected by the barrier (e.g., from 1 being the least valuable to 4 being the most valuable). Barrier assessment 514 includes a variable of the barrier assessment of the hypothesis and the score by which the hypothesis is affected by the barrier (e.g., from 1 being the least valuable to 4 being the most valuable).

[0080] FIG. 6 illustrates an example of system cognition according to an exemplary implementation. The system of FIG. 6 is configured to determine the context of an ASC requested by a user based on a hypothesis canvas. The user provides information to the NLP, NLU, and semantic matching process 206 via a chatbot 205, and the system then translates what the user needs and searches a database to determine whether an ASC exists that meets the user's needs or whether a similar ASC exists. In the exemplary implementation described herein, a query generated from the translation of the hypothesis canvas is used to search or generate a meta-pipeline based on the query. The meta-pipeline consists of metadata including data from the system's underlying devices and one or more meta-stages configured to be executed sequentially on the metadata, each meta-stage including an analytical or machine learning application.

[0081] Some of the components that are generated as part of the system cognition include the following aspects: First, the user exchanges information with the chatbot 205. Then, the natural language processing algorithm 206 removes some "not" words from the user's utterance. Then, the natural language understanding model applies a classification model to recognize the user's intent in the conversation. The chatbot 205 takes this information and answers the user's questions through the use of rule-based algorithms.

[0082] After some interactions, the semantic matching process 211 uses graphing techniques, taxonomies, and ontologies to create fuzzy matches related to identifying semantically related information and compares those representations to a database of hypothesis canvases stored in the web service metadata 203. If the web service finds a semantic match, the ASC is provided to the user from the artifact creation 213 and generic model zoo database 218. If there is no successful match, the process triggers the start of the meta-learning process 210 to create a new ASC using the meta-learning system 216.

[0083] 7 illustrates an example of a meta-learning subsystem process 210 according to an example implementation. The meta-learning process 210 is invoked after the system cognition attempts to select a matching use case without success. The database handles the storage of all predefined pipelines using an ontology / taxonomy.

[0084] 8 illustrates an example of a classification tree 903 according to an example implementation. A classification tree links business perspectives to analysis. For example, ASCs can be predefined to fit the needs of a classification business problem. The classification tree 903 can include various levels 800, 801, 802, 803 that are traversed to link business perspectives to analysis. Examples of levels can include, but are not limited to, ASC 800, analysis 801, problem 802, and business domain 803.

[0085] In addition to the database, it has pre-built procedures of Explainable Artificial Intelligence (AI) 219 to maintain the process and explain the results to the user. This Explainable AI 219 capability also includes a classification tree that contains the best explainable features for the use case. This process is orchestrated by the Model Management process in the following flow for every asset in the asset hierarchy.

[0086] First, the model manager 224 starts the process by using the ASC based on the business user's input and using the assets in the taxonomy and asset model. The model manager looks for the most similar ASC.

[0087] The model manager then uses a predefined classification method to identify the data on the data catalog 215. If necessary, it fills in missing values ​​or prepares synthetic data 214 based on the current sampling. In the case of a sampling method, from the current sample used, the model manager 224 recreates the new data using the distribution of the data to see which distribution represents the sample data set. After this step, it performs a Monte Carlo simulation to generate a random sample of the same distribution. The data catalog 215 and synthetic data process 214 are called for every asset in the asset hierarchy 220.

[0088] If no similar ASC is found, the model manager 224 goes through a process of building an ASC from a meta-learning process described with respect to FIG. 9. Based on an existing ASC or based on a new ASC, the ASC is deployed using a model server 222 that can run the model on a production system 225. The model monitor 221 then monitors the drift of the model and starts a new training phase using new data. If the training phase does not yield sufficient results, the procedure starts a meta-learning process 216 and a data characterization process 217 to find a new model. If the meta-learning process is not successful, the overall solution requires a data scientist to fine-tune it to move forward. This process is repeated for each asset in the asset hierarchy, in parallel for optimal performance of all assets.

[0089] FIG. 9 illustrates an example of a meta-learning process according to an example implementation.

[0090] In the model deployment phase, models are generated from the execution of meta-stages forming the meta-pipeline 905 and model creation 908. The models are deployed by the AutoML library 900 and trained by using reinforcement learning 901 as further described in FIG. 10. The models are evaluated against KPIs by model judgement 902. If the model needs to be recreated, feedback can be provided to the meta-pipeline 905 to modify the meta-stages and create a new model by model creation 908.

[0091] During the model training step, the meta-learning process of Fig. 9 is used to realize alternative ASC methods if they do not exist. The meta-learning phase is defined as follows:

[0092] First, by using the taxonomy 903, the process traverses the metastage repository 904 and selects predefined meta-pipelines 905, metadata 907, and meta-features 906. The concept of meta-learning is a process in which automated learning from previous machine learning models is reused and trained in future models. Metastages are atomic components of an analysis that have been used previously in other use cases and may be useful in future use cases of the same nature. Such metastages may be stored in the metastage repository 904. Meta-pipelines 905 are analysis pipelines composed by metastages that have been used previously in other use cases and that may be useful in future use cases of the same nature.

[0093] 10 illustrates a reinforcement learning (RL) process according to one example implementation. The input of the RL process is several latent models that use the previously defined metadata, meta-features, and meta-pipeline to create several different scenarios.

[0094] The first round of RL is an exploration phase where the best model is selected based on actions, maximizing the reward system, and minimizing regret consisting of a multi-armed bandit approach. Each model is named an agent 1000. The objective is to maximize the instance score and maximize the performance metric. The actions are the selection of different hyperparameters in the model using a hyperparameter policy. The agent 1000 that collects the highest reward is deployed in production by the model management. This is agent exploitation. In the production environment 1001, the model management measures the health state and monitors the behavior of the agent. If the agent's reward is not optimal, the model management removes the current agent from production and selects a second agent to be exploited. This process is repeated until the process exhausts all agents 1000.

[0095] 11 illustrates an example meta-pipeline 1100 according to one example implementation. The meta-pipeline 1100 includes metadata 1102 that is processed by different meta-stages 1101 that can be configured in different ways. Each of these meta-stages 1101 is evaluated and compared to metadata from customers to determine which meta-stage 1101 or combination thereof has the best performance.

[0096] The metastage 1101 may include different analytical applications (e.g., statistical applications, visualization applications, etc.) and / or machine learning applications that are applied and executed sequentially. For example, the metastage 1101 may include units of statistics, analysis, data information, or machine learning models that may be used to create the meta-pipeline 1100 and interconnected in an execution order. Examples of metastages 1101 may include, but are not limited to, metafeatures 1103, metamodels 1104, and metavisualization applications 1105.

[0097] In the following example of a meta-pipeline 1100, metadata 1102 may include, but is not limited to, product hierarchy 1110, transactions 1111, customer marketing attributes 1112, weather data 1113, and crop acreage 1114. Metadata 1102 is linked to a first metastage, which is meta-features 1103. Meta-features 1103 include applications that generate features from metadata 1102. Examples of sub-functions used by meta-features 1103 to generate features from metadata 1102 may include data schema 1115 to format the metadata into a preferred structure, and feature engineering 1116 to derive features from the processed metadata. The meta-features 1103 metastage is then linked to the next metastage, which is meta-model 1104.

[0098] The metamodel 1104 is configured to incorporate the features generated from the metafeatures 1103 to generate a model. In this example, the metamodel 1104 includes a multi-model approach by using a grid search to identify a best-fit model formed from a combination of models. Such models for grid search, random search, or Bayesian optimization may include, but are not limited to, decision trees 1117, random forests 1118, support vector machines 1119, and logistic regression 1120.

[0099] The metamodel 1104 is then linked to the next meta stage which is a meta visualization application 1105. The model generated by the metamodel 1104 is then connected to the meta visualization application 1105 which generates output for visualizing the analysis. Example outputs from the visualization application 1105 may include likelihood p(i) 1121, attributes 1122, and E(Rev)-p(i)*LastYearRevenue 1123.

[0100] Thus, in the exemplary meta-pipeline 1100, metadata 1102 is processed through a meta-stage 1101 in an execution order that includes execution of a meta-features 1103 to process the metadata 1102 and extract features, then execution of a meta-model 1104 to incorporate features from the meta-features 1103 to generate a model, and then execution of a meta-visualization 1105 to generate a visualization output. In another exemplary implementation, another example of a meta-stage 1101 may include a controller application that controls the underlying devices of the system based on the output of the meta-model 1104.

[0101] Using the example implementations described herein, deployment in large-scale systems (e.g., analytical factories, production lines, etc.) can allow models to control the system as they are updated in real-time. In one example, the output can be an analytical system with models that can be deployed in a SaaS service format via a hypothesis canvas and metadata, including analytical code and business artifacts.

[0102] Example implementations described herein can facilitate a data warehouse of models, making it easy for data scientists to check and reuse any model for a given solution. By retaining models that produced "bad" results, data scientists can check and examine areas that did not perform well. For models that performed well, data scientists can keep those models and retrain them on different data or reuse them without retraining.

[0103] 12 illustrates a system including multiple systems with connected sensors and a management device, according to an example implementation. One or more systems 1201-1, 1201-2, 1201-3, and 1201-4 with connected sensors are communicatively connected to a network 1200 that is connected to a management device 1202 that facilitates the functioning of an Internet of Things (IoT) gateway or other manufacturing management system. The management device 1202 manages a database 1203 that includes historical data collected from the sensors of the systems 1201-1, 1201-2, 1201-3, and 1201-4, which may include labeled and unlabeled data received from the systems 1201-1, 1201-2, 1201-3, and 1201-4. In an alternative exemplary implementation, data from the sensors of systems 1201-1, 1201-2, 1201-3, 1201-4 may be stored in a central repository or a central database, such as a dedicated database that captures data such as an enterprise resource planning system, and management device 1202 may access or retrieve the data from the central repository or database. Such systems may include robotic arms with sensors, turbines with sensors, lathes with sensors, etc., depending on the desired implementation.

[0104] FIG. 13 illustrates an example computing environment with an example computing device suitable for use in some example implementations, such as the management device 1202 shown in FIG.

[0105] A computing device 1305 in computing environment 1300 may include one or more processing units, cores, or processors 1310, as well as memory 1315 (e.g., RAM, ROM, and / or the like), internal storage 1320 (e.g., magnetic, optical, solid-state storage, and / or organic), and / or I / O interface 1325, any of which may be connected over a communication mechanism or bus 1330 for communicating information or may be embedded in the computing device 1305. The I / O interface 1325 may also be configured to receive images from a camera or provide images to a projector or display, depending on the desired implementation.

[0106] The computing device 1305 may be communicatively connected to an input / user interface 1335 and an output device / interface 1340. One or both of the input / user interface 1335 and the output device / interface 1340 may be a wired or wireless interface and may be removable. The input / user interface 1335 may include any device, component, sensor, or interface, physical or virtual, that may be used to provide input (e.g., buttons, touch screen interface, keyboard, pointing / cursor control, microphone, camera, Braille, motion sensor, optical reader, and / or the like). The output device / interface 1340 may include a display, television, monitor, printer, speaker, Braille, etc. In some example implementations, the input / user interface 1335 and the output device / interface 1340 may be embedded or physical in the computing device 1305. In other example implementations, other computing devices may function as or provide the functionality of the input / user interface 1335 and the output device / interface 1340 for the computing device 1305 .

[0107] Examples of computing devices 1305 may include, but are not limited to, highly mobile devices (e.g., smart phones, devices in vehicles and other machinery, devices carried by people and animals, etc.), mobile devices (e.g., tablets, notebook computers, laptop computers, personal computers, portable televisions, radios, etc.), and devices not designed for mobility (e.g., desktop computers, other computers, information kiosks, televisions with one or more processors embedded and / or connected thereto, radios, etc.).

[0108] Computing device 1305 may be communicatively connected (e.g., via I / O interface 1325) to external storage 1345 and a network 1350 for communicating with any number of networked components, devices, and systems including one or more computing devices of the same or different configurations. Computing device 1305 or any connected computing device may function as, provide services to, or be referred to as a server, client, thin server, general purpose machine, dedicated machine, or another label.

[0109] I / O interface 1325 may include, but is not limited to, any communication or I / O protocol or standard (e.g., Ethernet, 802.11x, Universal System Bus, WiMax, modem, cellular network protocols, etc.) for communicating information to and / or from at least all connected components, devices, and networks in computing environment 1300. Network 1350 may be any network or combination of networks (e.g., the Internet, a local area network, a wide area network, a telephone network, a cellular network, a satellite network, etc.).

[0110] The computing device 1305 can use and / or communicate using computer usable or computer readable media, including transitory media and non-transitory media. Transitory media include transmission media (e.g., metallic cables, optical fibers), signals, carrier waves, etc. Non-transitory media include magnetic media (e.g., disks and tapes), optical media (e.g., CD-ROMs, digital video disks, Blu-ray® disks), solid-state media (e.g., RAM, ROM, flash memory, solid-state storage), and other non-volatile storage or memory.

[0111] The computing device 1305 may be used to implement techniques, methods, applications, processes, or computer-executable instructions in some exemplary computing environments. The computer-executable instructions may be retrieved from a transitory medium and stored in and retrieved from a non-transitory medium. The executable instructions may originate from one or more of any programming, scripting, and machine language (e.g., C, C++, C#, Java, Visual Basic, Python, Perl, JavaScript, etc.).

[0112] The one or more processors 1310 may run under any operating system (OS) (not shown), in a native or virtual environment. One or more applications may be deployed, including a logic unit 1360, an application programming interface (API) unit 1365, an input unit 1370, an output unit 1375, and an inter-unit communication mechanism 1395 for different units to communicate with each other, with the OS, and with other applications (not shown). The described units and elements may be modified in design, function, configuration, or implementation, and are not limited to the provided description.

[0113] In some example implementations, when information or instructions to execute are received by the API unit 1365, it may be communicated to one or more other units (e.g., logic unit 1360, input unit 1370, output unit 1375). In some cases, logic unit 1360 may be configured to control information flow between units and direct services provided by API unit 1365, input unit 1370, output unit 1375 in some example implementations described above. For example, the flow of one or more processes or implementations may be controlled by logic unit 1360 alone or in conjunction with API unit 1365. Input unit 1370 may be configured to obtain inputs for calculations described in the example implementations, and output unit 1375 may be configured to provide outputs based on the calculations described in the example implementations.

[0114] To receive the hypothesis canvas and metadata as shown in FIG. 5 from the multiple sensors, the processor 1310 may be configured to generate a multiple meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being an analytical application or machine learning application stacked in an execution order based on the metadata and the hypothesis canvas as shown in FIG. 11, and to run the multiple meta-pipeline stages in an execution order on the metadata to generate an analytical model of the system, as shown in FIG. 9.

[0115] The processor 1310 may be configured to perform a reinforcement learning process to select one of the generated meta-pipeline stages to execute on the metadata, as shown in FIG. 10.

[0116] The processor 1310 may be configured to generate multiple meta-pipeline stages by translating the hypothesis canvas to generate a query for a semantic search using natural language processing, performing a semantic search from the query against a meta-learning database to search for one or more pipelines associated with the multiple devices, generating meta-pipeline stages from the one or more pipelines, and managing the meta-pipeline stages in a repository configured to track learning models generated from the meta-pipeline stages, metadata associated with the learning models, and meta-features associated with the learning models, as shown in FIGS.

[0117] The processor 1310 may be configured to evaluate the analytical model against one or more key performance indicators extracted from the hypothesis canvas, as shown in Figures 9 and 10, and modify one or more of the meta-pipeline stages based on the evaluation.

[0118] As shown in Figures 7-11, the processor 1310 can be configured to execute a plurality of meta-pipeline stages in an execution order on the metadata to generate an analytical model of the system, including generating a plurality of candidate analytical models having various hyper-parameters selected from the hyper-parameter policy; evaluating the analytical models against one or more key performance indicators extracted from the hypothesis canvas includes evaluating the plurality of candidate analytical models against sensor data received from a plurality of sensors processed through the plurality of candidate analytical models using a multi-armed bandit approach; and modifying one or more of the plurality of meta-pipeline stages based on the evaluation further includes selecting one of the candidate analytical models that maximizes rewards collected using the multi-armed bandit approach as the analytical model; and modifying one or more of the plurality of pipeline stages according to the selected one of the candidate analytical models.

[0119] As shown in FIG. 5, the hypothesis canvas may include multiple data sources that identify one of multiple sensors to obtain metadata, hypotheses associated with the system, expected predictions from the hypotheses, and risks associated with the hypotheses.

[0120] As shown in FIG. 9, the processor 1310 may be configured to execute a meta-learning process configured to compare data structures from different metadata and meta-features, data structures from the existing dataset and the new dataset, and data structures from different datasets to apply the meta-pipeline to the new dataset.

[0121] The processor 1310 may be configured to control one or more of the plurality of devices by deploying the analytical model on the system. For example, the processor 1310 may be configured to power on / off the device, activate an andon near the device to flag a foreman, adjust device settings in real time, etc. based on the output of the analytical model, depending on the desired implementation.

[0122] Some portions of the detailed descriptions are presented in terms of algorithms and symbolic representations of operations within a computer. These algorithmic descriptions and symbolic representations are the means used by those skilled in the data processing arts to convey the substance of their innovations to others skilled in the art. An algorithm is a series of defined steps leading to a desired end state or result. In exemplary implementations, the steps performed require specific quantities of physical manipulations to achieve specific results.

[0123] Unless otherwise specifically stated, and as will be apparent from the description, throughout the description of the specification, descriptions utilizing terms such as "processing," "calculating," "calculating," "determining," "displaying," and the like, are understood to include the actions and processes of a computer system or other information processing device that manipulates and converts data represented as physical (electronic) quantities in the registers and memory of the computer system to other data similarly represented as physical quantities in the memory or registers of the computer system or other information storage, transmission, or display device.

[0124] Exemplary implementations may also relate to an apparatus for performing the operations of the present specification. This apparatus may be specially constructed for the required purposes, or may include one or more general-purpose computers selectively activated or reconfigured by one or more computer programs. Such computer programs may be stored on a computer-readable medium, such as a computer-readable storage medium or a computer-readable signal medium. Computer-readable storage media may include tangible media, such as, but not limited to, optical disks, magnetic disks, read-only memories, random access memories, solid-state devices and drives, or any other type of tangible or non-transitory medium suitable for storing electronic information. Computer-readable signal media may include media such as carrier waves. The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. A computer program may include a pure software implementation that includes instructions to perform the operations of a desired implementation.

[0125] Various general-purpose systems may be used with the programs and modules according to the examples herein, or it may prove convenient to construct more specialized apparatus to perform the desired method steps. Moreover, the example implementations are not described with reference to a particular programming language. It will be appreciated that a variety of programming languages ​​may be used to implement the techniques of the example implementations described herein. Instructions in one or more programming languages ​​may be executed by one or more processing devices (e.g., a central processing unit (CPU), processor, or controller).

[0126] As known in the art, the above operations may be performed by hardware, software, or some combination of software and hardware. Various aspects of the exemplary implementations may be implemented using circuits and logic devices (hardware), while other aspects may be implemented using instructions stored on a machine-readable medium (software) that, when executed by a processor, cause the processor to perform methods for implementing the implementations of the present application. Furthermore, some exemplary implementations of the present application may be performed exclusively in hardware, while other exemplary implementations may be performed exclusively in software. Furthermore, various functions described may be performed in a single unit or may be distributed among several components of various methods. When performed by software, the methods may be executed by a processor, such as a general-purpose computer, based on instructions stored on a computer-readable medium. If necessary, the instructions may be stored on the medium in compressed and / or encrypted format.

[0127] Moreover, other implementations of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the techniques of the present application. Various aspects and / or components of the described exemplary implementations may be used alone or in any combination. It is intended that the specification and exemplary implementations be considered as examples only, with the true scope and spirit of the present application being indicated by the following claims.

Claims

1. 1. A method for a system including a plurality of sensors associated with a plurality of devices, the method comprising: receiving a hypothesis canvas and metadata from the plurality of sensors; generating a plurality of meta-pipeline stages for generating an analytical model of the system, the plurality of meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being connected according to an execution order based on the metadata and the hypothesis canvas, an analytical application performing processing related to the analysis of the system based on the metadata or a machine learning application training a learning model; executing the plurality of meta-pipeline stages on the metadata in accordance with the execution order to generate an analytical model of the system; Including, The generating of the plurality of meta-pipeline stages comprises: Transforming the hypothesis canvas to generate queries for semantic search using natural language processing; performing a semantic search against a meta-learning database from the query to retrieve one or more pipelines associated with the plurality of devices; generating the meta-pipeline stage from the one or more pipelines; storing the meta-pipeline stages in a repository configured to track metadata associated with the learned models, and the learned models generated from the meta-features associated with the learned models and generating features from the metadata; A method comprising:

2. 2. The method of claim 1, further comprising: executing a reinforcement learning process to select one of the generated meta-pipeline stages for execution on the metadata.

3. evaluating the analytical model against one or more key performance indicators extracted from the hypothesis canvas; modifying one or more of the meta-pipeline stages based on the evaluation; and The method of claim 1 further comprising:

4. executing the plurality of meta-pipeline stages in an execution order on the metadata to generate the analytical model of the system includes generating a plurality of candidate analytical models, the candidate analytical models having different hyper-parameters selected from a hyper-parameter policy for selecting hyper-parameters; evaluating the analytical models against one or more key performance indicators extracted from the hypothesis canvas includes evaluating the candidate analytical models against sensor data received from the sensors that has been processed through the candidate analytical models using a multi-armed bandit approach; The modifying one or more of the plurality of meta-pipeline stages based on the evaluation includes: selecting as the analytical model one of the candidate analytical models that maximizes rewards collected using the multi-armed bandit approach; modifying the one or more of the plurality of pipeline stages according to the selected one of the candidate analysis models; The method of claim 3 , comprising:

5. The method of claim 1 , further comprising controlling one or more of the plurality of devices by deploying the analytical model in the system.

6. 2. The method of claim 1, wherein the hypothesis canvas includes a plurality of data sources that identify one of the plurality of sensors for obtaining the metadata, a hypothesis associated with the system, a prediction expected from the hypothesis, and a risk associated with the hypothesis.

7. 10. The method of claim 1, further comprising: executing a meta-learning process configured to compare data structures from different metadata and meta-features, data structures from the existing dataset and the new dataset, and data structures from the different dataset, to apply the meta-pipeline to a new dataset.

8. 1. A non-transitory computer readable medium storing instructions for a system including a plurality of sensors associated with a plurality of devices, the instructions comprising: generating a plurality of meta-pipeline stages for generating an analytical model of the system, the plurality of meta-pipeline stages including the metadata and one or more stages, each of the one or more stages being connected according to an execution order based on the metadata and the hypothesis canvas, an analytical application performing processing related to the analysis of the system based on the metadata or a machine learning application training a learning model; executing the plurality of meta-pipeline stages on the metadata in accordance with the execution order to generate an analytical model of the system; Including, The generating of the plurality of meta-pipeline stages comprises: Transforming the hypothesis canvas to generate queries for semantic search using natural language processing; performing a semantic search against a meta-learning database from the query to retrieve one or more pipelines associated with the plurality of devices; generating the meta-pipeline stage from the one or more pipelines; storing the meta-pipeline stages in a repository configured to track metadata associated with the learned models, and the learned models generated from the meta-features associated with the learned models and generating features from the metadata; A non-transitory computer readable medium comprising:

9. 10. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise performing a reinforcement learning process that selects one of the generated meta-pipeline stages for execution on the metadata.

10. The instruction: evaluating the analytical model against one or more key performance indicators extracted from the hypothesis canvas; modifying one or more of the meta-pipeline stages based on the evaluation; and 10. The non-transitory computer-readable medium of claim 8, further comprising:

11. executing the plurality of meta-pipeline stages in an execution order on the metadata to generate the analytical model of the system includes generating a plurality of candidate analytical models, the candidate analytical models having different hyper-parameters selected from a hyper-parameter policy for selecting hyper-parameters; evaluating the analytical models against one or more key performance indicators extracted from the hypothesis canvas includes evaluating the candidate analytical models against sensor data received from the sensors that has been processed through the candidate analytical models using a multi-armed bandit approach; The modifying one or more of the plurality of meta-pipeline stages based on the evaluation includes: selecting as the analytical model one of the candidate analytical models that maximizes rewards collected using the multi-armed bandit approach; modifying the one or more of the plurality of pipeline stages according to the selected one of the candidate analysis models; 11. The non-transitory computer-readable medium of claim 10, comprising:

12. 10. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise controlling one or more of the plurality of devices by deploying the analytical model in the system.

13. 10. The non-transitory computer-readable medium of claim 8, wherein the hypothesis canvas includes a plurality of data sources that identify one of the plurality of sensors for obtaining the metadata, a hypothesis associated with the system, a prediction expected from the hypothesis, and a risk associated with the hypothesis.

14. 10. The non-transitory computer-readable medium of claim 8, wherein the instructions further comprise executing a meta-learning process configured to compare data structures from different metadata and meta-features, data structures from the existing dataset and the new dataset, and data structures from the different dataset to apply a meta-pipeline to a new dataset.

Images