Techniques consisting of adaptive and context-aware automation services for machine learning (ML)

Abstract

The server system may receive a plurality of inputs identifying a location of data, describing a prediction for a machine learning application, and one or more constraints for the machine learning application. The server system may access a memory containing one or more ontologies for the data. The server system may extract one or more attributes of the data set to find a first ontology related to the prediction for the machine learning application based on the one or more constraints. The server system may compose a product graph based on the first ontology stored in the memory, the one or more constraints, and one or more previous product graphs, wherein the product graph associates the one or more data objects with a set of nodes and edges, wherein the edges represent links between nodes comprising basic units of the data structure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Non-Provisional Application No. 16 / 892,724, filed June 4, 2020, entitled “TECHNIQUES FOR ADAPTIVE AND CONTEXT-AWARE AUTOMATED SERVICE COMPOSITION FOR MACHINE LEARNING (ML)”, which is incorporated herein by reference in its entirety and for all purposes. This application also claims priority to U.S. Non-Provisional Application No. 16 / 892,935, filed June 4, 2020, entitled “TECHNIQUES FOR ADAPTIVE PIPELINING COMPOSITION FOR MACHINE LEARNING (ML)”, which is incorporated herein by reference in its entirety and for all purposes. This application also claims priority to U.S. Non-Provisional Application No. 17 / 019,254, filed September 12, 2020, entitled “TECHNIQUES FOR SERVICE EXECUTION AND MONITORING FOR RUN-TIME SERVICE COMPOSITION,” which is incorporated herein by reference in its entirety and for all purposes. This application also claims priority to U.S. Provisional Patent Application No. 62 / 900,537, filed September 14, 2019, entitled “AUTOMATED MACHINE LEARNING SYSTEMS AND METHODS,” which is incorporated herein by reference in its entirety and for all purposes. Technical Field

[0003] The present disclosure relates to systems and techniques for machine learning. More particularly, the present disclosure relates to systems and techniques for generating and managing libraries for machine learning applications. Background Art

[0004] Machine learning has a wide range of applications, such as search engines, medical diagnostics, text and handwriting recognition, image processing and recognition, load forecasting, marketing and sales diagnostics, chatbots, autonomous driving, etc. Various types and versions of machine learning models can be generated for similar applications using training data based on different technologies, languages, libraries, etc., and therefore may lack interoperability. In addition, different models may have different performance in different contexts and / or for different types of input data. Data scientists may not have the programming skills to generate the code necessary to build custom machine learning models. In addition, available machine learning tools do not store the various machine learning model components as part of a library to allow efficient reuse of routines in other machine learning models.

[0005] Existing machine learning applications can require a lot of programming knowledge from data scientists to design and construct machine learning applications to solve specific problems. An intuitive interface can help data scientists construct machine learning applications through a series of queries.

[0006] Some organizations may store data from multiple clients or vendors in customizable schemas. These customizable schemas may not match the standardized data storage schemas used by existing machine learning models. Therefore, these other systems will need to perform a reconciliation process before using the stored data. This reconciliation process can be either manual or automated through tedious extract, transform, and load (ELT) processes before using the data to generate machine learning applications.

[0007] Machine learning applications based solely on metrics (e.g., quality of service (QoS) or key performance indicators) may not be sufficient to compose a pipeline for adaptive architectures with minimal human intervention. Pre-existing machine learning tools do not combine non-logical and logical-based semantic services to generate machine learning applications.

[0008] Existing machine learning tools tend to maximize classification accuracy on new data from the same source. In doing so, these existing frameworks attempt to compose an optimal pipeline based on model performance. However, model performance alone does not cover situations where the pipeline must be adapted at runtime due to changes in online data metrics or QoS violations. Summary of the Invention

[0009] Certain aspects and features of the present disclosure relate to a machine learning platform that generates a library of components to generate machine learning models and machine learning applications. The machine learning infrastructure system allows users (i.e., data scientists) to generate machine learning applications without having to understand the cloud-based network infrastructure in detail or how to generate the code used to build the model. The machine learning platform can analyze the identified data and the desired prediction and performance characteristics provided by the user to select one or more library components and associated APIs to generate the machine learning application. The machine learning technology can monitor and evaluate the output of the machine learning model to allow feedback and adjustment to the model. The machine learning application can be trained, tested, and compiled to be exported as a standalone executable code.

[0010] The machine learning platform can generate and store one or more library components that can be used in other machine learning applications. The machine learning platform can allow users to generate profiles that allow the platform to make recommendations based on the user's historical preferences. The model creation engine can detect the number and type of infrastructure resources necessary to achieve the desired results within the desired performance criteria.

[0011] Chatbots provide an intuitive interface that allows data scientists to build machine learning applications without extensive programming experience. Using a conversational interface, chatbots translate natural language into a structured representation of a machine learning solution. Chatbots can be used to indicate the location of data, select the type of machine learning solution, display the optimal solution that best satisfies the constraints, and recommend the best environment for deploying the solution.

[0012] A self-adjusting, company-wide discovery and integration feature can review a client's data repository, review the labels used for various data patterns, and efficiently map the client's data patterns to classifications used by the machine learning model. Various techniques can automatically select features that are predictive for each individual use case (i.e., one client), effectively making the machine learning solution independent of the application developer. A weighted list of common representations for each feature of a particular machine learning solution can be generated and stored. When new data is added to the data repository, the matching service can automatically detect which features should be fed into the machine learning solution based at least in part on the weighted list. The weighted list can be updated as new data is made available to the model.

[0013] Existing data ontologies can be used to generate machine learning solutions for high-precision searches of related services, allowing for pipelines to be composed with minimal human intervention. Data ontologies can be used to create combinations of non-logical and logic-based semantic services, significantly outperforming both options in terms of accuracy. QoS and product KPI constraints can be used as part of architectural selection. For datasets without existing ontologies, one or more ontologies can be generated.

[0014] The proposed system can use machine learning applications to solve problems using the best available model at construction time. The adaptive pipeline composition service can identify one or more new models and incorporate them into the machine learning application. The machine learning application with the new model can be tested offline and the results can be compared with the ground truth data. If the performance of the machine learning application with the new model is better than the previously used model, then the machine learning application can be upgraded and automatically promoted to production. One or more parameters can also be discovered. The new parameters can be incorporated into the existing model in offline mode. The machine learning application with the new parameters can be tested offline and the results can be compared with the previous results with the existing parameters. If the performance of the new parameters is better than the existing parameters compared with the ground truth data, then the machine learning application can be automatically promoted to production.

[0015] According to some embodiments, a method may include receiving a first input, wherein the first input identifies a location of data; receiving a second input, wherein the second input describes a prediction for a machine learning application; receiving a third input, wherein the third input includes one or more constraints for the machine learning application; accessing a memory containing one or more ontologies of data, wherein each ontology describes a hierarchical structure of one or more data sets for one or more data objects, wherein each data object exhibits one or more properties that allow the data objects to be classified into one or more sets of data objects according to a rule; extracting one or more properties of the one or more data sets to find a first ontology related to the prediction for the machine learning application based on the one or more constraints; composing a product graph based on the first ontology, the one or more constraints, and one or more previous product graphs stored in the memory, wherein the product graph associates the one or more data objects with a set of nodes and edges, wherein the edges represent links between the nodes; wherein the nodes comprise basic units of the data structure; and storing the product graph in the memory.

[0016] According to some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium comprises configured instructions, which may include one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to: receive a first input, wherein the first input identifies a location of data; receive a second input, wherein the second input describes a prediction for a machine learning application; receive a third input, wherein the third input comprises one or more constraints for the machine learning application; access a memory containing one or more ontologies of data, wherein each ontology describes a hierarchical structure of one or more data sets for one or more data objects, wherein each data object exhibits one or more properties that allow the data objects to be classified into one or more sets of data objects according to a rule; extract one or more properties of the one or more data sets to find a first ontology related to the prediction for the machine learning application based on the one or more constraints; compose a product graph based on the first ontology, the one or more constraints, and one or more previous product graphs stored in the memory, wherein the product graph associates the one or more data objects with a set of nodes and edges, wherein the edges represent links between the nodes; wherein the nodes comprise basic units of the data structure; and store the product graph in the memory.

[0017] According to some embodiments, a system may include one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured as: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising: receiving a first input, wherein the first input identifies a location of data; receiving a second input, wherein the second input describes a prediction for a machine learning application; receiving a third input, wherein the third input comprises one or more constraints for the machine learning application; accessing a memory containing one or more ontologies of data, wherein each ontology describes a hierarchical structure of one or more data sets for one or more data objects, wherein each data object exhibits one or more properties that allow the data objects to be classified into one or more sets of data objects according to a rule; extracting one or more properties of the one or more data sets to find a first ontology related to the prediction for the machine learning application based on the one or more constraints; composing a product graph based on the first ontology stored in the memory, the one or more constraints, and one or more previous product graphs, wherein the product graph associates the one or more data objects with a set of nodes and edges, wherein the edges represent links between the nodes; wherein the nodes comprise basic units of the data structure; and storing the product graph in the memory.

[0018] According to some embodiments, a method may include identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define higher-level characteristics of a first data set prior to a machine learning process; identifying one or more first parameters used by the first machine learning model; identifying one or more first results for the first machine learning model for the one or more first parameters; identifying one or more metrics for the first machine learning model, wherein the metrics define how to measure the performance of the first machine learning model; comparing the one or more first results to a second data set to calculate a first score based at least in part on a difference between the one or more first results and the second data set, wherein the second data set includes ground truth data; referencing an ontology of the first data set to identify a second machine learning model based at least in part on comparing metadata of the second machine learning model to the metrics of the first machine learning model; testing the second machine learning model on the first data set to produce one or more second results; comparing the one or more second results to the second data set to generate a second score; and replacing the first machine learning model with the second machine learning model for the machine learning application based on the second score being less than the first score.

[0019] According to some embodiments, a computer program product tangibly embodied in a non-transitory machine-readable storage medium includes configured instructions, which may include one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured to: identify a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define higher-level characteristics of a first data set prior to a machine learning process; identify the one or more first parameters used by the first machine learning model; identify one or more first results for the first machine learning model for the one or more first parameters; identify one or more metrics for the first machine learning model, wherein the metrics define how to measure the performance of the first machine learning model; compare the one or more first results with a second data set to calculate a first score based at least in part on a difference between the one or more first results and the second data set, wherein the second data set includes basic fact data; reference an ontology of the first data set to identify the second machine learning model based at least in part on comparing metadata of the second machine learning model with the metrics of the first machine learning model; test the second machine learning model on the first data set to produce one or more second results; compare the one or more second results with the second data set to generate a second score; and replace the first machine learning model with the second machine learning model for the machine learning application based on the second score being less than the first score.

[0020] According to some embodiments, a system may include one or more memories; and one or more processors, communicatively coupled to the one or more memories, configured as: one or more data processors; and a non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations including: identifying a first machine learning model, wherein the first machine learning model utilizes one or more first parameters, wherein the one or more first parameters identify and define higher-level characteristics of a first data set prior to a machine learning process; identifying one or more first parameters used by the first machine learning model; identifying one or more first results for the first machine learning model for the one or more first parameters; identifying one or more metrics for the first machine learning model, wherein the metrics define how to measure the performance of the first machine learning model; comparing the one or more first results with a second data set to calculate a first score based at least in part on a difference between the one or more first results and the second data set, wherein the second data set includes ground truth data; referencing an ontology of the first data set to identify a second machine learning model based at least in part on comparing metadata of the second machine learning model with the metrics of the first machine learning model; testing the second machine learning model on the first data set to produce one or more second results; comparing the one or more second results with the second data set to generate a second score; and replacing the first machine learning model with the second machine learning model for the machine learning application based on the second score being less than the first score.

[0021] According to some embodiments, a method may include receiving two or more quality of service (QoS) dimensions for a multi-objective optimization model. The two or more QoS dimensions include at least a first QoS dimension and a second QoS dimension. The method may include maximizing the multi-objective optimization model along the first QoS dimension. Maximizing includes selecting one or more pipelines for the multi-objective optimization model in the software architecture that meet the QoS expectations specified for the first QoS dimension and the second QoS dimension. The ordering of the pipelines may depend on the QoS dimensions that are optimized and de-optimized and to what extent. The multi-objective optimization model may be partially de-optimized along the second QoS dimension to meet the QoS expectations for the first QoS dimension. Thus, a trade-off exists between the first QoS dimension and the second QoS dimension.

[0022] According to some embodiments, a non-transitory computer-readable medium may store one or more instructions. The one or more instructions, when executed by one or more processors of a server system, may cause the one or more processors to: receive two or more quality of service (QoS) dimensions for a multi-objective optimization model, wherein the two or more QoS dimensions include at least a first QoS dimension and a second QoS dimension; maximize the multi-objective optimization model along the first QoS dimension, wherein the maximization includes selecting one or more pipelines for the multi-objective optimization model in a software architecture that meet QoS expectations specified for the first QoS dimension and the second QoS dimension, wherein the ordering of the pipelines depends on the QoS dimensions that are optimized and deoptimized and to what extent, wherein the multi-objective optimization model partially deoptimizes along the second QoS dimension to meet the QoS expectations for the first QoS dimension and thereby a tradeoff exists between the first QoS dimension and the second QoS dimension.

[0023] According to some embodiments, a method may include retrieving data associated with historical outputs of a machine learning model compared to a set of quality of service metrics and key performance indicator metrics. The method may include receiving one or more inputs from an environment monitoring agent. The environment monitoring agent may receive information regarding at least one of: system resources, a concept of the machine learning model, data corruption, and data availability for the machine learning model. The method may include determining a change in at least one of: system resources, concept of the machine learning model, data corruption, and data availability for the machine learning model. The method may include determining whether a change in at least one of the system resources, concept of the machine learning model, data corruption, and data availability for the machine learning model would cause a predicted output of the machine learning model to change by more than a predetermined amount. When a change in at least one of the system resources, concept of the machine learning model, data corruption, and data availability for the machine learning model would cause a predicted output of the machine learning model to change by more than a predetermined amount, the method may include identifying one or more remedial actions for the machine learning model to correct the change. The method may include displaying an alert to notify a user of the change in at least one of the system resources, concept of the machine learning model, data corruption, and data availability for the machine learning model, and the one or more remedial actions.

[0024] According to some embodiments, a non-transitory computer-readable medium may store one or more instructions that, when executed by one or more processors of a cloud-based server system, may cause the one or more processors to retrieve data associated with historical outputs of a machine learning model compared to a set of quality of service metrics and key performance metrics. The instructions may cause the one or more processors to receive one or more inputs from an environment monitoring agent.

[0025] An environment monitoring agent receives information about at least one of: resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model. Instructions may cause one or more processors to determine a change in at least one of: resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model. Instructions may cause one or more processors to determine whether a change in at least one of the resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model will cause a predicted output of the machine learning model to change by more than a predetermined amount. When a change in at least one of the resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model will cause a predicted output of the machine learning model to change by more than a predetermined amount, the instructions may cause the one or more processors to identify one or more remedial actions for the machine learning model to correct the change. The method may include displaying an alert to notify a user of the change in at least one of the resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model, and the one or more remedial actions.

[0026] These and other embodiments are described in detail below.For example, other embodiments are directed to systems, devices, and computer-readable media associated with the methods described herein.

[0027] A better understanding of the nature and advantages of embodiments of the present disclosure may be obtained with reference to the following detailed description and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The description refers to the following drawings, in which like reference numerals are used in different drawings to illustrate like or similar components.

[0029] Figure 1 is a block diagram illustrating an exemplary machine learning infrastructure system.

[0030] Figure 2 is a flow chart illustrating an exemplary technique for generating a machine learning application.

[0031] Figure 3 is a flow chart illustrating another exemplary technique for generating a machine learning application.

[0032] Figure 4 Depicted is a distributed system implementing a robotic system for communicating with end users using a messaging application, according to various embodiments.

[0033] Figure 5 Depicted is an integrated system including a robotic system and a robotic analysis system for monitoring, analyzing, visualizing, and improving the performance of the robotic system, according to various embodiments.

[0034] Figure 6Depicted is a distributed system implementing a robotic system for communicating with end users using a messaging application, in accordance with certain embodiments.

[0035] Figure 7 A flow diagram illustrating a technique for generating a machine learning application using a chatbot.

[0036] Figure 8 Illustrate a flow chart for protecting a serialized model and storing the private key in a secure location.

[0037] Figure 9A and 9B Illustrated are exemplary user interface screens generated by a software development environment.

[0038] Figure 10 Illustrated is an exemplary distributed computing environment including a code integration request prediction server that can be used to receive and process input data, generate and use machine learning or artificial intelligence models to generate output that predicts code integration requests.

[0039] Figure 11 An exemplary flow diagram for a live code analysis plug-in is illustrated.

[0040] Figure 12 An exemplary flow chart for automated on-demand library checking is illustrated.

[0041] Figure 13A and 13B Illustrated are exemplary user interface screens generated by a software development environment.

[0042] Figure 14 A flow diagram illustrating an exemplary technique for self-tuning enterprise-wide feature discovery and integration is illustrated.

[0043] Figure 15 A flow diagram of a second exemplary technique for self-tuning enterprise-wide feature discovery and integration is illustrated.

[0044] Figure 16 A block diagram illustrating an exemplary system for generating a product graph using an ontology is illustrated.

[0045] Figure 17 An exemplary flow diagram illustrating the composition of adaptive and context-aware automated services for machine learning is shown.

[0046] Figure 18 Illustrated is a block diagram for generating one or more pipelines.

[0047] Figure 19 An exemplary flow chart for adaptively pipelined composition services is illustrated.

[0048] Figure 20 A diagram illustrating a system for service execution and monitoring runtime service composition.

[0049] Figure 21 An exemplary flow diagram for service execution and monitoring of a runtime service composition is illustrated.

[0050] Figure 22 is a simplified diagram illustrating a distributed system for implementing one of the embodiments.

[0051] Figure 23 is a simplified block diagram illustrating one or more components of a system environment.

[0052] Figure 24 An exemplary computer system is illustrated in which various embodiments of the present disclosure may be implemented. DETAILED DESCRIPTION

[0053] Certain embodiments of the present disclosure relate to systems, devices, computer-readable media, and computer-implemented methods for implementing various techniques for machine learning. Machine learning techniques can allow users (i.e., data scientists) to generate machine learning applications without requiring detailed knowledge of cloud-based network infrastructure or how to generate code for building models. A machine learning platform can analyze identified data and user-provided desired prediction and performance characteristics to select one or more library components and associated APIs to generate machine learning applications.

[0054] Machine learning technology can employ chatbots to indicate the location of the data, select the type of machine learning solution, display the optimal solution that best meets the constraints, and recommend the best environment to deploy the solution.

[0055] The techniques described herein may include self-tuning company-wide discovery and integration features that can audit a client's data repositories, audit the labels used for various data patterns, and efficiently map the client's data patterns to classifications used by the machine learning model. Various techniques can automatically select features that are predictive for each individual use case (i.e., a client), effectively making the machine learning solution agnostic to the application developer. A weighted list of common representations of each feature for a particular machine learning solution can be generated and stored.

[0056] This technology can leverage existing data ontologies to generate machine learning solutions for high-precision searches of relevant services, thereby forming a pipeline with minimal human intervention. For datasets without existing ontologies, one or more ontologies are generated.

[0057] The technology can employ adaptive pipeline composition services to identify and incorporate one or more new models into a machine learning application. The machine learning application with the new model can be tested offline, and the results compared to ground truth data. If the machine learning application with the new model outperforms the previously used model, the machine learning application can be upgraded and automatically promoted to production.

[0058] I. Machine Learning Infrastructure Platform

[0059] Figure 1 is a block diagram illustrating an exemplary machine learning platform 100 for generating machine learning models. The machine learning platform 100 has various components that can be distributed across different networks and computing systems. A machine learning infrastructure library can store one or more components for generating machine learning applications 112. All infrastructure required to put machine learning applications 112 into production can be encapsulated and stored in the library.

[0060] The machine learning configuration and interaction with the model composition engine 132 allows for selection of various library components 168 (e.g., pipelines 136 or workflows, microservice routines 140, software modules 144, and infrastructure modules 148) to define the training and inference logic for building the machine learning application 112. Different parameters, variables, scaling, settings, etc. for the library components 168 can be specified or determined by the model composition engine 132. The complexity required to routinely create the machine learning application 112 can be largely automated using the model composition engine 132.

[0061] The library component 168 can be extensible to allow the definition of multiple environments (e.g., different Kubernetes clusters) in which various parts of the application can be deployed to achieve any specified quality of service (QoS) or key performance metric (KPI). A Kubernetes cluster is a collection of node machines used to run containerized applications. Extensibility can hide or abstract the complexity of the machine learning platform 100 from application developers. The monitoring engine 156 can monitor the operation of the machine learning application 112 based on the KPI / QoS metrics 160 to ensure that the machine learning application 112 is performing as required. In addition, the monitoring engine 156 can seamlessly test new or evolving machine learning applications end-to-end at different scales, settings, loads, settings, etc. The monitoring engine 156 can recommend various adjustments to the machine learning application 112 by signaling the required changes to the model composition engine 132.

[0062] In some embodiments, to address scalability issues, the machine learning platform 100 creates an infrastructure based on a microservices architecture, making it robust and scalable. For example, various microservice routines 140 and infrastructure modules 148 can be configured and customized to be embedded in the machine learning application 112. The machine learning platform 100 can allow developers to define the amount of resources (e.g., CPU, memory) required by different library components 168 of the machine learning application 112.

[0063] The machine learning platform 100 can generate highly customizable applications. Library components 168 contain a collection of predefined, ready-made workflows or pipelines 136 that application developers can incorporate into new machine learning applications 112. A workflow specifies various microservice routines 140, software modules 144, and / or infrastructure modules 148 that are configured in a specific way for a type or class of problems. Furthermore, it is possible to define new workflows or pipelines 136 by reusing library components or modifying existing workflows or pipelines 136. Infrastructure modules 148 can also include services such as data collection, process monitoring, and logging.

[0064] The model composition engine 132 can be executed on one or more computing systems (e.g., infrastructure 128). The model composition engine 132 can receive input from the user 116 through the interface 104. The interface 104 can include various graphical user interfaces with various menus and user-selectable elements. The interface 104 can include a chatbot (e.g., a text-based or voice-based interface). The user 116 can interact with the interface 104 to identify one or more of the following: the location of the data, the desired predictions for the machine learning application, and various performance metrics of the machine learning model. The model composition engine 132 can interface with the library component 168 to identify various pipelines 136, microservice routines 140, software modules 144, and infrastructure models 148 that can be used in the creation of the machine learning model 112.

[0065] The model composition engine 132 can output one or more machine learning applications 112. The machine learning applications 112 can be stored locally on a server or in a cloud-based network. The model composition engine 132 can output the machine learning applications 112 as executable code that runs on various infrastructures 128 through the infrastructure interface 124.

[0066] The model execution engine 108 can execute the machine learning application 112 on the infrastructure 128 using one or more infrastructure interfaces 124. The infrastructure 128 can include one or more processors, one or more memories, and one or more network interfaces, one or more buses, and control lines, which can be used to generate, test, compile, and deploy the machine learning application 112. In various embodiments, the infrastructure 128 can reside on a remote system 152 that is remote from the user 116. The infrastructure 128 can interact with the model execution engine 108 through the infrastructure interface 124. The model execution engine 108 can input performance characteristics (e.g., KPI / QoS metric storage 160) and hosted input data 164. The model execution engine 108 generates one or more results from the machine learning application 112.

[0067] The KPI / QoS metric storage 160 can store one or more metrics that can be used to evaluate the machine learning application 112. The metrics can include reasoning query metrics, performance metrics, sentiment metrics, and test metrics. The metrics can be received from the user 116 via the user interface 104.

[0068] The monitoring engine 156 can receive the results of the model execution engine 108 and compare the results to performance characteristics (e.g., KPI / QoS metrics 160). The monitoring engine 156 can test the machine learning application 112 using ground truth data to ensure that the model can be executed as expected. The monitoring engine 156 can provide feedback to the model composition engine 132. The feedback can include adjustments to one or more variables used in the machine learning model 112 or the selected machine learning model.

[0069] The library components 168 may include various pipelines 136, microservice routines 140, software modules 144, and infrastructure modules 148. A software pipeline 136 may be composed of a series of computational processes (e.g., commands, program execution, tasks, threads, processes, etc.).

[0070] Microservice routines 140 can be used in an architectural approach to building applications. As an architectural framework, microservices are distributed and loosely coupled to allow changes to one aspect of an application without disrupting the entire application. The benefit of using microservices is that the development team can quickly build new components of the application to meet changing development requirements. The microservice architecture breaks down the application into its core functions. Each function is called a service and can be built and deployed independently, which means that individual services can operate (and fail) without negatively impacting other services. A microservice can be the core function of an application that runs independently of other services. By storing various microservice routines 140, the machine learning platform 100 can incrementally generate machine learning applications by identifying and selecting various different components from the library component 168.

[0071] A software module 144 can include a block of code that forms part of a program containing one or more routines. One or more independently developed modules make up a program. An enterprise-level software application may contain several different software modules 144, and each module may serve a unique and separate operation. A module interface can express the elements provided and required by a module. The elements defined in the interface can be detectable by other modules. An implementation can include working code corresponding to the elements declared in the interface. Modular programming can be related to structured programming and object-oriented programming, both of which have the same goal of facilitating the construction of large software programs and systems by breaking them down into smaller pieces. Although the historical usage of these terms has been inconsistent, "modular programming" as used herein refers to the high-level decomposition of the code of an entire program into pieces: structured programming to the low-level code use of structured control flow, and object-oriented programming to the data use of objects, a type of data structure. In object-oriented programming, the use of interfaces as an architectural pattern to construct modules is called interface-based programming.

[0072] The infrastructure module 148 may include the technology stack necessary to put machine learning algorithms into production in a stable, scalable, and reliable manner. A technology stack may include the set of software subsystems or components required to create a complete platform, so that no additional software is required to support the application. For example, to develop a web application, an architect may define the stack as the target operating system, web server, database, and programming language. Another version of the software stack may be the operating system, middleware, database, and application. The components of the software stack may be developed independently of each other by different developers. The stack may extend from the data science tools used to select and train machine learning algorithms to the hardware on which these algorithms run and the databases and message queues from which they extract data sets.

[0073] Machine learning platform 100 may include one or more data storage locations 170. A user may identify one or more data storage locations 170. Data storage location 170 may be local (e.g., in a storage device electrically connected to the processing circuitry and interfaces used to generate, test, and execute applications). In various embodiments, data storage location 170 may be remote (e.g., accessible via a network such as a local area network or the Internet). In some embodiments, data storage location 170 may be a cloud-based server.

[0074] The data used for machine learning models 112 often includes personally identifiable information (PII), thus triggering certain protections provided by privacy laws. One way to protect the information contained in data storage 170 can be to encrypt the data using one or more keys. Public key cryptography, or asymmetric cryptography, is a cryptographic system that uses a key pair: a public key that can be widely distributed and a private key that is known only to the owner of the data. The private key can be stored in key storage module 172 to enable decryption of data for use by machine learning platform 100.

[0075] Model execution engine 108 may use hosted input data 164 to execute and test machine learning application 112. Hosted input data 164 may include a portion of data stored at data store 170. In various embodiments, a portion of hosted input data 164 may be identified as test data.

[0076] Figure 2 Illustrated is a process 200 for generating a machine learning application using a machine learning library infrastructure. Alternative embodiments may be combined, separated, or otherwise altered. Figure 2 To change the functionality described in the blocks shown in the figure. Figure 2 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0077] Functionality may include providing users with access to a machine learning platform for developing production code customized for a hardware platform and user-defined requirements. Interfaces accessible to the user may include a graphical user interface on a touch screen display, a keyboard, a keypad, a microphone for receiving voice commands, a chatbot, and / or a pointing device (e.g., a mouse, a trackball for selecting one or more items in the graphical user interface).

[0078] At 202, the functionality may include receiving a first user input identifying a location of data. In various embodiments, the user may use the interface to identify one or more data locations to be used to generate the machine learning model. As described above, the data may be stored locally or remotely. In various embodiments, the user may enter a network location of the data (e.g., an Internet Protocol (IP) address). In various embodiments, the user may select a folder from a plurality of folders on a storage device (e.g., a cloud storage device). In various embodiments, the user may select a file on a portable media device (e.g., a USB thumb drive, an optical disc, or a solid state or magnetic disk drive). In various embodiments, the technology may suggest a location for the data based at least on the user's previous interactions with the machine learning interface.

[0079] At 204, functionality includes receiving a second user input identifying a problem for which the machine learning application can generate a solution. In various embodiments, the second user input can specify the type of problem for which the user wants to implement machine learning. In various embodiments, the problem can be identified by entering text via a user interface. In various embodiments, the problem can be entered as native language speech or text (e.g., by using a chatbot). The technology can decipher the native language to understand the objectives of the machine learning model. Some types of problems that machine learning can solve can include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. These technologies can identify one or more keywords in the native language to recommend or select a specific machine learning algorithm.

[0080] In various embodiments, a user may select a problem type via a graphical user interface. The technology may display several generic models available to the user via the graphical user interface. The user may select one of the models by selecting one of the model descriptions using a pointing device or a touchscreen display. In addition to the generic models, additional custom models may also be selected and modified to solve the problem received as the second input.

[0081] At 206, the functionality includes receiving a third input of one or more performance requirements of the machine learning application. The third input can be entered as native language speech or text (e.g., by using a chatbot) or selected via an interface (e.g., a graphical user interface).

[0082] Performance requirements may include quality of service (QoS) metrics. QoS metrics are objective, system-related characteristics that provide insight into the performance of a delivered service at the network / transport level. QoS metrics are parameters that reflect the quality of service on the sender side, not the application side. Example QoS metrics may include system latency and reliability.

[0083] Performance requirements can include one or more key performance indicators (KPIs). A key performance indicator is a measurable value that demonstrates how effectively a model achieves its objectives. KPIs can be problem / solution specific and can include a measure of the accuracy of the results of a machine learning application compared to some ground truth test data.

[0084] The third user input may include an inference indicator. An inference indicator is a word that typically appears at the beginning of a premise or conclusion. An inference indicator reminds the system which premise or conclusion to follow.

[0085] The third user input may include a sentiment indicator. A sentiment indicator is a graphical or numerical indicator designed to show how a group feels about a certain condition or state (e.g., the market or economy). Sentiment indicators are intended to quantify how current beliefs and positions influence future behavior.

[0086] The third user input may include test metrics. Test metrics may include quantitative measurements that help estimate the progress, quality, and health of software testing efforts. Test metrics quantitatively define the extent to which a system, system component, or process possesses a given attribute. Example test metrics may include schedule variance, test design efficiency, number of defects per test hour, and fixed defect percentage.

[0087] The third user input may include training metrics. Training metrics help evaluate the performance of the model. Example training metrics may include classification accuracy, log loss, area under the curve, F1 score, mean absolute error, and mean squared error. The accuracy metric is the ratio of the number of correct predictions divided by the number of predictions made. The log loss metric works by penalizing misclassifications. The area under the curve (AUC) can be used for binary classification problems. The AUC of a classifier is equal to the probability that the classifier will rank a randomly selected positive example higher than a randomly selected negative example. The F1 score is used to measure the accuracy of the test. The F1 score is the harmonic mean between precision and recall. The range of the F1 score is [0, 1]. The F1 score can tell the user how accurate the classifier is (how many instances it correctly classified) and how robust it is (it does not miss a large number of instances).

[0088] The mean absolute error is the average of the differences between the original values and the predicted values. The mean absolute error provides a measure of how far the predictions differ from the actual output. However, the mean absolute error does not provide an idea of the direction of the error, that is, whether the model is underpredicting or overpredicting the data.

[0089] Mean Squared Error (MSE) is very similar to Mean Absolute Error, the only difference being that MSE averages the squares of the differences between the original and predicted values. The advantage of MSE is that it is easier to compute gradients, whereas Mean Absolute Error requires complex linear programming tools to compute gradients. When the system squares the errors, the effects of larger errors are more pronounced than smaller errors, so the model can now focus more on larger errors.

[0090] The third user input can include maintainability metrics. Maintainability metrics can be defined as the ease with which a system can be maintained. It is an internal property of the system and cannot be directly measured. A solution to this problem is to build models to predict maintenance process metrics from software product metrics such as cohesion, coupling, and complexity.

[0091] The third user input may include a reusability metric. Reusability metrics can be used to measure code reuse and reusability. Reusability metrics can be categorized into six types: reuse cost-benefit model, maturity assessment, amount of reuse, failure modes, reusability, and reuse library metrics.

[0092] At 208, functionality includes determining that one or more library components will be selected for generating a machine learning model to prototype a machine learning application to meet performance requirements. The one or more library components may perform production functions. The library components may include at least one of pipelines 136, workflow and microservice routines 140, software modules 144, and infrastructure modules 148, as described above with respect to Figure 1 described.

[0093] The library components 168 may include metadata identifying the characteristics and functionality of each library component 168. The technology may determine one or more library components 168 to select based at least in part on the identified problem received via the second input to achieve the performance metric of the third input. One or more variables of each library component may be adjusted to customize the machine learning model to achieve a solution to the identified problem.

[0094] At 210, the functions may include identifying one or more application programming interfaces stored in a library, wherein the application programming interfaces link one or more components to form a machine learning application. An application programming interface (API) may be used to connect one or more library components to generate a machine learning model.

[0095] In various embodiments, production functionality may include at least one of load balancing, failover caching, security, testing capabilities, auditing capabilities, scalability, predictive performance, training models, predictive power, maintenance, debugging capabilities, and reusability. Load balancing refers to the process of distributing a collection of tasks across a collection of resources (computing units) with the goal of making their overall processing more efficient. Load balancing techniques can optimize the response time of each task, thereby avoiding uneven overloading of computing nodes while other computing nodes are idle. The primary mechanism for maintaining high system availability is called failover. In this approach, a failed primary system is replaced by a backup system; that is, processing fails over to the backup system. Many high availability configurations also provide mechanisms for disaster recovery, which is to restore system availability when the failover mechanism cannot maintain system availability. Testing capabilities can address the difficulty of testing the expected performance characteristics of machine learning applications. Auditing capabilities can address the ability of machine learning applications to be evaluated against internal controls. Predictive capacity can refer to the percentage of available processing power required to execute machine learning applications.

[0096] After a machine learning model has been generated, it can be used to train the machine learning model to the desired performance parameters using training data. Training can adjust one or more variables for the generated model.

[0097] At 212 , the functionality may include storing the machine learning application in memory.

[0098] At 214, functionality includes testing the machine learning model against performance requirements. After the machine learning model has been generated and trained, the technology can be tested by the model execution engine using unseen test data. The test data can be a selected percentage of the training data retained from the trained model. The test data can also be a collection of separate data. The monitoring engine can determine whether the machine learning model meets key performance indicators / quality of service metrics. Feedback from the monitoring engine can be sent to the model combination engine to provide recommendations for revising the machine learning model.

[0099] At 216, the functionality includes compiling a machine learning application based on the machine learning model according to the results of the testing of the machine learning model. After the machine learning model has been trained and tested, it can be compiled to output a standalone executable code.

[0100] It should be recognized that Figure 2 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present invention may perform the above steps in a different order. Moreover, Figure 2The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0101] Figure 3 Illustrated is a process 300 for generating a machine learning application from a machine learning library infrastructure. Alternative embodiments may be combined, separated, or otherwise modified. Figure 3 To change the functionality described in the blocks shown in the figure. Figure 3 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0102] At 302, functionality includes receiving a first user input identifying a location of data. In various embodiments, a user can use an interface to identify the location of data to be used to generate a machine learning model. In various embodiments, a user can enter a network location of the data (e.g., an Internet Protocol (IP) address). In various embodiments, a user can select a folder from a plurality of folders on a storage device (e.g., a cloud storage device). In various embodiments, a user can select a file on a portable medium (e.g., a USB thumb drive, an optical disc, or a solid-state or magnetic disk drive).

[0103] At 304, the functionality includes receiving a second user input regarding a desired prediction of a machine learning (ML) application. In various embodiments, the second user input may specify the type of problem the user wants to implement machine learning to solve. In various embodiments, the problem may be entered as native language speech or text (e.g., by using a chatbot). The technology may decipher the native language to understand the objectives of the machine learning model. Some types of problems that machine learning can solve may include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. The technology will identify one or more keywords in the native language speech to recommend or select a specific machine learning algorithm.

[0104] At 306, functionality includes collecting data in streams (with chunking) or batches. Chunking is a term that refers to the process of taking individual pieces of information (chunks) and grouping them into larger units. By grouping each piece into a larger whole, you can increase the amount of information you can retain. The model merging engine can access a data store to collect data for generating the machine learning model. The data can be stored locally or in a cloud-based network.

[0105] At 308, functionality includes validating the data to check for data characteristics (e.g., whether the data shape is within expectations). Center is the median and / or mean of the data. Spread is the range of the data. And, shape describes the type of graph. Four ways to describe the shape are whether it is symmetrical, how many peaks it has, whether it is skewed to the left or right, and whether it is uniform. If the shape is not within the expected range, the model merging engine can notify the user and prompt the user to locate other / additional data.

[0106] At 310, functionality includes identifying issues before they manifest downstream in the pipeline. The model merging engine can review the machine learning query and data to ensure that the identified data has the characteristics required for the query or question. If an issue is identified, the model merging engine can alert the user and potentially recommend changes to settings or data to resolve the identified issue.

[0107] At 312, functionality includes constructing characteristics (i.e., attributes / features) of the problem. The model merging engine can interpret user input to identify the type of problem the user wants to solve using machine learning. After the problem is identified, the problem can be deconstructed into the attributes and features required to solve that problem. For example, if the user desires to determine the most productive workers at a facility, the system will need to understand the characteristics or attributes that define a productive worker. In some embodiments, productivity can be determined by the number of units produced per unit of work time. These attributes can then include data on the number of hours worked, the number of units produced, and the number of defective units produced.

[0108] At 314, functionality includes determining an outcome to be predicted. The technology can leverage the information, data, and library components identified by the second user to determine one or more output metrics that the machine learning application can predict. In various embodiments, the machine learning platform can present the output metrics to the user. In various embodiments, the user can select one or more output metrics from a plurality of available output metrics.

[0109] At 316, functionality includes continuously monitoring QoS / KPI values to validate the model. In various embodiments, the monitoring engine can evaluate one or more QoS or KPI metrics to determine whether the model meets performance specifications. In various embodiments, the machine learning platform can notify the user of the monitored values and alert the user if the QoS / KPI metric falls outside of a specified threshold.

[0110] At 318, functionality includes training a machine learning model with predictions against QoS / KPI determinations. In various embodiments, data may be used to train a machine learning model. Variables of the model may be adjusted based on output values based on QoS / KPI metrics.

[0111] At 320, functionality includes providing controls to adjust the model. The controls can be executed via a chatbot, a graphical user interface, or one or more user-selectable menus. The controls allow the user to adjust the results of the model by adjusting the variables used for the selected algorithm. In various embodiments, the controls display the result values as the model is adjusted.

[0112] At 322, functionality includes automatically adjusting the model as needed. In various embodiments, the values of the algorithm can be automatically adjusted to achieve the desired QoS / KPI results. In various embodiments, these values can be adjusted within a defined range of values. Adjustments can include selecting different pipelines 136, microservice routines 140, software modules 144, and infrastructure modules 148.

[0113] At 324, functionality includes compiling the model into a machine learning application. In various embodiments, the model can be compiled into a standalone executable code. In various embodiments, the model can be compiled to form part of another program.

[0114] At 326, functionality includes monitoring the predictions to assess whether the results are within expectations. In various embodiments, the monitoring engine can provide feedback to the model execution engine to inform the user whether the model provides results within the expected range.

[0115] At 328, the functionality optionally provides an interpretation and explanation of the model so that the model drivers can be understood by the user. In various embodiments, the machine learning platform can provide the user with the results of the model. The results can explain in detail the impact of various drivers on the model results. In this way, the user can make an informed decision on how to adjust the model to achieve the desired results.

[0116] At 330 , functionality includes catching anomalies before they manifest as inaccurate predictions. In various embodiments, the monitoring engine may receive information regarding output values that fall outside certain thresholds.

[0117] At 332, functionality includes automatically or semi-autonomously remediating inaccurate data. In various embodiments, the monitoring engine may determine that received data may be inaccurate. In various embodiments, the monitoring engine may notify a user that the data may be inaccurate. The monitoring engine may solicit input from the user to verify or correct the suspect data. In various embodiments, the monitoring engine may automatically correct the suspect data.

[0118] It should be recognized that Figure 3The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present disclosure may perform the above steps in a different order. Moreover, Figure 3 The individual steps shown in the figure may include multiple sub-steps, which may be performed in various orders as appropriate for the individual steps. In addition, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0119] II. Chatbots for defining machine learning solutions

[0120] The machine learning model is trained to generate predictive results for code integration requests. In one aspect, techniques can be used to define a machine learning solution, including receiving a first input (e.g., auditory, textual, or GUI) describing a problem for the machine learning solution. Figure 1 As shown in , the model composition engine 132 can rewrite the first input into one or more text snippets. The model composition engine 132 can determine the user's intention to create a machine learning architecture based at least in part on the one or more text snippets. These techniques can include associating the one or more text snippets with one or more machine learning frameworks of multiple models. These techniques can include presenting (e.g., an interface or audio) one or more machine learning models to the user. The model composition engine 132 can receive a selection of one or more machine learning models (e.g., classification, recommendation, reinforcement learning). The model composition engine 132 can receive several other user inputs, including a second input identifying a data source for the machine learning architecture and a third input identifying one or more constraints for the machine learning architecture (e.g., resources, location, security, or privacy). The model composition engine 132 can generate multiple codes for the machine learning architecture based at least in part on the selected model, the second input identifying the data source, and the third input identifying the one or more constraints. The generated code can be stored in a memory.

[0121] An intelligent assistant (e.g., a chatbot) can be used to enable a user to generate a machine learning system. Technology can be used to automatically generate a machine learning system to assist the user. In some cases, the user may not be a software developer and may have little or no experience in machine learning techniques or software programming. In some embodiments, the user can interact with the intelligent assistant. The interaction can be auditory, textual, or through a graphical user interface. The chatbot can use an ontology to translate natural language input into a structural representation of a machine learning solution. In this way, users can use artificial intelligence without having to become data scientists.

[0122] The techniques can receive multiple inputs from a user. Based on the inputs, these techniques can determine the user's intent to establish a machine learning architecture. In this technique, an intelligent assistant can analyze the inputs and recommend various options to the user based on the analysis. These techniques can also generate code for the machine learning architecture. This code can be stored and reused for one or more different machine learning processes. The disclosed techniques simplify the process of developing intelligent applications.

[0123] Intelligent assistants can use chatbots. Chatbots are software modules that communicate via audio or text, acting as dialogue systems for user interaction. Chatbots can use sophisticated natural language processing systems or scan input for keywords and then extract responses from a database with the best matching keywords or similar wording patterns. Chatbots can be used to build AI systems that can answer questions. In this way, AI can be used to translate information provided to software modules and hardware infrastructure in plain language.

[0124] Users (e.g., application developers) can automatically configure machine learning infrastructure via a conversational interface (e.g., a chatbot). Users can define how machine learning predictions are consumed (e.g., "via REST API" or "save to file"). Users can also define the location of the data. Users can also identify which additional services the machine learning infrastructure may require (e.g., monitoring, logging, and alerting). Constraints (e.g., resources, location, security, privacy) can be identified by the user.

[0125] In the following description, various embodiments are described. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, embodiments may be readily apparent to those skilled in the art without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described embodiments.

[0126] introduce

[0127] Digital assistants are AI-driven interfaces that help users complete various tasks in natural language conversations. For each digital assistant, customers can assemble one or more skills. Skills (also referred to herein as chatbots, robots, or skillbots) are individual robots that specialize in specific types of tasks, such as tracking inventory, submitting time cards, generating machine learning applications, and creating expense reports. When an end user interacts with a digital assistant, the digital assistant evaluates the end user's input and routes the conversation to and from the appropriate chatbot.

[0128] Intents allow a chatbot to understand what the user wants the chatbot to do. Intents consist of a sequence of typical user requests and statements, also known as utterances (e.g., "Generate a classifier application," "Determine the most efficient employee from employment records," etc.). As used herein, an utterance or message can refer to a collection of words exchanged during a conversation with the chatbot (e.g., one or more sentences). Intents can be created by providing a name that describes a user action (e.g., "Generate a classifier") and compiling real-life user statements or utterances typically associated with triggering actions. Because the chatbot's cognition is derived from these intents, each intent can be created from a robust (one to two dozen utterances) and varied dataset so that the chatbot can interpret ambiguous user input. This enriched utterance set enables the chatbot to understand what the user wants when receiving messages such as "Use dataset A" or "Recognize dataset A as data"—messages that have the same meaning but are expressed differently. The intents and utterances belonging to them together constitute the chatbot's training corpus. By training a model with a corpus, the user can essentially turn that model into a reference tool to parse end-user input into a single intent. Users can improve the cognitive acuity of chatbots through multiple rounds of intent testing and intent training.

[0129] However, building a chatbot that can determine end-user intent based on user utterances is a challenging task, in part due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). Therefore, the chatbot may need to be trained, monitored, debugged, and retrained to improve the chatbot's performance and the chatbot's user experience. In conventional systems, a training model, essentially a default training model hard-coded into the design system, is provided for training and retraining a digital assistant or chatbot. For example, a first model can be provided that requires only a small training corpus so that it can use matching rules to develop entities, intents, and a training corpus. When the training corpus matures to the point where testing reveals highly accurate intent parsing, a second model can be used to add a deeper dimension to the chatbot's cognition by training using machine learning based on word vectors and other text-based features. These default training models are often inflexible in the training methods they employ. Therefore, without a more flexible training method, it can be difficult to identify the root cause of a chatbot's sub-optimal performance and determine how to improve the chatbot.

[0130] Analytics systems can be integrated with a bot system to monitor events that occur during a conversation between an end user and the bot system, aggregate and analyze the collected events, and provide information to the user based on the analytics that can be used to improve the performance of the bot system, including the performance of intent classification. However, (i) these systems typically do not identify the root cause of sub-optimal performance (instead, they provide information such as "bot A failed to recognize intent x times within period n"), and it is up to the user to identify the root cause of the problem from the information, and (ii) these types of analytics systems become ineffective when hundreds or thousands of bots are employed within the bot system.

[0131] Therefore, a different approach is needed to address these issues. In various embodiments, an analysis system can be integrated with the robotic system. The analysis system can collect conversation logs and histories and identify information related to individual and / or aggregated end-user conversations with the robotic system as paths, which include different nodes representing different stages or states of the conversation. For example, an end-user conversation with the robotic system can be represented by a path showing transitions from state to state, where each state can be represented by a node on the path. Statistics of user conversations with the robotic system can be generated for each node. The path includes (i) the number of conversations that flowed through the intent-specific path of the conversation flow for a given period of time, (ii) the number of conversations maintained between each state and the different execution paths taken because the conversation branched due to a value being set (or not set) or reached a deadlock due to some other issue (such as a misbehaving custom component), and (iii) the final state that provides insight into whether the conversation ultimately succeeded or failed. The analysis tool can then use the information generated for each path and node to retrain the robotic system or individual robots responsible for the intent / path.

[0132] In some embodiments, an event collector of the analysis system may collect one or more attributes of one or more events associated with a conversation set of the bot system. The event collector may be reconfigurable to selectively collect desired attributes of desired events. The one or more events may include, for example, at least one of a conversation event, a bot state event, an intent resolution event, an entity resolution event, an error event, a timeout event, or a custom event. Subsequently, an analysis engine of the analysis system may use one or more filtering criteria selected by a user to select one or more conversations from the conversation set based on the one or more attributes of the one or more events collected by the event collector. The one or more filtering criteria may include, for example, conversations ending in a specific state, conversations starting in a specific state, completed or incomplete conversations, conversations associated with a specific end-user intent, conversations from a specific channel or location, conversations occurring during a specific time period, and the like. For the selected one or more conversations, the analysis engine may calculate statistics for the conversation set, statistics for conversations associated with a specific end-user intent, statistics for completed conversations, statistics for incomplete conversations, statistics for conversations for which no end-user intent was determined, or any combination thereof. The analysis engine may generate options for improving the bot system based on the calculated statistics.

[0133] In some embodiments, the analysis engine may generate an aggregated path graph for the selected one or more conversations. The aggregated path graph may include a plurality of nodes and a plurality of connections between the plurality of nodes. Each of the plurality of nodes may correspond to a respective state of the robot system. Each of the plurality of connections may represent a transition from one state of the robot system to another state of the robot system. The plurality of nodes may include a start node and an end node. In some embodiments, the aggregated path graph may include a number associated with each respective connection, wherein the number may indicate a total number of conversations that include a transition represented by the respective connection.

[0134] The analysis engine can combine the statistical information with the aggregated path graph to determine additional information, such as how many conversations flowed through the intent-specific path of the dialog flow for a given period of time, the number of conversations maintained between each state, and the different execution paths taken by the conversation due to branching caused by values being set (or not set), or due to some other problem (such as a misbehaving custom component). Optionally, the statistical information and the aggregated path graph can be used to retrain the robot system to improve the performance of the robot system, such as retraining the robot system's intent classification model to more accurately determine user intent.

[0135] In some embodiments, a graphical user interface (GUI) may display information related to individual and / or aggregated end-user conversations with a robotic system as a path that includes different nodes representing different stages or states of the conversation. For example, an end-user conversation with a robotic system may be represented by a path showing transitions from state to state, where each state may be represented by a node on the path. Statistics of user conversations with the robotic system may be generated and displayed graphically by the GUI. The path visualization of the analysis system may allow an administrator or developer to filter or select groups of end-user conversations with the robotic system based on different criteria. The analysis system may also provide the administrator or developer of the robotic system with the option to select and review individual conversations of interest. The visualized path information may allow the administrator or developer of the robotic system to visually monitor and analyze how end-users interact with the robotic system and how the robotic system performs during the conversation to identify underperforming elements of the robotic system and possible improvements.

[0136] As described above, the analysis system can provide information about end-user conversations with the robotic system through the GUI at various levels of summary, including all conversations, conversations that meet specific criteria, conversations associated with specific intents or end states, and individual conversations. Thus, the analysis system can allow administrators or developers of the robotic system to identify specific end-user utterances and intents associated with incomplete or unsuccessful conversations, thereby identifying and improving underperforming elements of the robotic system. By analyzing and improving the performance of the robotic system, the end-user experience with the robotic system can be improved.

[0137] Robotics and analytical systems

[0138] A bot (also known as a skill, chatbot, chatbot, or conversational bot) is a computer program that can perform a conversation with an end user. Bots can generally respond to natural language messages (e.g., questions or comments) through messaging applications that use natural language messages. Enterprises can use one or more bot systems to communicate with end users through messaging applications. Messaging applications, which can be referred to as channels, can be messaging applications preferred by end users that the end user has already installed and is familiar with. Thus, end users can chat with the bot system without downloading and installing new applications. For example, messaging applications can include OTT messaging channels (e.g., Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo, or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions that extend native or hybrid / responsive mobile applications or web applications with chat functionality, or voice-based input (such as devices or applications with interfaces that interact using Siri, Cortana, Google Voice, or other voice input).

[0139] In some examples, a robot system can be associated with a uniform resource identifier (URI). The URI can use a string to identify the robot system. The URI can be used as a webhook for one or more messaging applications. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post-call messages) from the messaging application system. The HTTP post-call message can be directed from the messaging application system to the URI. In some embodiments, the message can be different from the HTTP post-call message. For example, the robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received by the robot system as messages, it should be understood that the message can be an HTTP post-call message, an SMS message, or any other type of communication between two systems.

[0140] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just as they would with other people. In some cases, the interaction can include the end user saying "hello" to the robot, the robot responding with "hi," and asking the end user how it can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring funds from one account to another; an informational interaction with, for example, an HR robot, such as checking a vacation balance; or an interaction with, for example, a retail robot, such as discussing a return or seeking technical support.

[0141] In some embodiments, the robot system can intelligently handle end-user interactions without having to interact with the administrator or developer of the robot system. For example, the end user can send one or more messages to the robot system to achieve a desired goal. The message can include certain content, such as text, emoticons, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., a Representational State Transfer (REST) call to an enterprise service with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user to enter additional input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user rather than passively responding to the end user's utterances. This article describes various techniques for identifying explicit calls to the robot system and determining the input of the called robot system. In some embodiments, the master robot performs explicit call analysis based on detecting the call name in the utterance. In response to the detection of the call name, the utterance can be refined for input to the skill robot associated with the call name.

[0142] A conversation with a robot can follow a specific conversation flow that includes multiple states. The flow can define what happens next based on the input. In some embodiments, a state machine that includes user-defined states (e.g., end-user intent) and actions taken in or from states to states can be used to implement a robot system. The conversation can take different paths based on the end-user's input, which affects the decisions the robot makes for the flow. For example, in each state, based on the end-user's input or utterance, the robot can determine the end-user's intent to determine the appropriate action to take next. As used herein and in the context of an utterance, the term "intent" refers to the intention of the user who provided the utterance. For example, a user can intend for the robot to engage in a conversation to generate a machine learning application, so that the user's intent can be represented by the utterance "build a machine learning application". The user intent can point to a specific task that the user wants the chatbot to perform on the user's behalf. Therefore, the utterance can be expressed as a question, command, request, etc. that reflects the user's intent. The intent can include a goal that the end-user wants to achieve.

[0143] In the context of chatbot configuration, the term "intent" is used herein to refer to configuration information used to map user utterances to specific tasks / actions or categories of tasks / actions that the chatbot can perform. To distinguish the intent of an utterance (i.e., user intent) from the chatbot's intent, the latter is sometimes referred to herein as a "bot intent." A bot intent can include a set of one or more utterances associated with that intent. For example, an intent for generating a machine learning application can include various permutations of utterances expressing a desire to generate a machine learning application to solve a problem. These associated utterances can be used to train the chatbot's intent classifier, enabling the intent classifier to subsequently determine whether an input utterance from the user matches a machine learning application intent. A bot intent can be associated with one or more conversation flows, used to initiate a conversation with the user and establish a specific state. For example, the first message for a machine learning application intent might be the question "What problem are you trying to solve?" In addition to the associated utterances, a bot intent can also include named entities associated with the intent. For example, a machine learning application intent can include variables or parameters used to perform the task of generating a machine learning application, such as the type of problem, the data to be used, performance characteristics, various constraints, and so on. The values of the entities are typically obtained through conversation with the user.

[0144] Figure 4 is a simplified block diagram of a distributed environment 400 for generating chatbots for machine learning applications. The distributed environment 400 includes a Digital Assistant Builder Platform (DABP) 402 that enables businesses to create and deploy digital assistants for their users. For the purposes of this disclosure, a "digital assistant" is an entity that helps users of the digital assistant complete various tasks through natural language conversation. A digital assistant can be implemented using software alone (e.g., a digital assistant is a digital entity implemented using programs, codes, or instructions that can be executed by one or more processors), using hardware, or using a combination of hardware and software. A digital assistant can be implemented or implemented in various physical systems or devices, such as computers, phones, watches, appliances, vehicles, etc. Digital assistants are sometimes also referred to as chatbot systems. The DABP 402 can be used to create one or more digital assistants (or DAs) 406. The DABP 402 can be used by multiple businesses to create digital assistants for their users. For example, Figure 4 As shown in , a user 404 representing a particular enterprise can use DABP 402 to create and deploy a digital assistant 406 for users of the particular enterprise.

[0145] Once digital assistant 406 is deployed, user 408 can use digital assistant 406 to perform various tasks via natural language-based conversations with digital assistant 406. As part of the conversations, user 408 can provide one or more user inputs 410 and obtain responses 412 from digital assistant 406. Through these conversations, the user can request that one or more tasks be performed by digital assistant 406, and in response, digital assistant 406 is configured to perform the user-requested tasks and respond with an appropriate response to the user.

[0146] User input 410 is in natural language and is referred to as an utterance. User utterances can be in text form (e.g., when a user types something as input to digital assistant 406) or in audio input or speech form (e.g., when a user speaks something as input to digital assistant 406). Utterances are typically in the language spoken by user 408. When user input 410 is in speech form, the speech input is converted into a textual utterance in that particular language, which is then processed by digital assistant 406. Various speech-to-text processing techniques can be used to convert speech or audio input into textual utterances, which are then processed by digital assistant 406.

[0147] The text utterance input by user 408 or generated by converting the spoken input into text form can be a text snippet, a sentence, multiple sentences, etc. The digital assistant 406 is configured to apply natural language understanding (NLU) technology to the text utterance to understand the meaning of the user input. As part of the NLU processing of the utterance, the digital assistant 406 is configured to perform processing to understand the meaning of the utterance, which involves identifying one or more intents and one or more entities corresponding to the utterance. After understanding the meaning of the utterance, the digital assistant 406 can perform one or more actions or operations in response to the understood meaning or intent.

[0148] For example, user input 410 may request the generation of a machine learning application, e.g., “I want to build an image classifier”. The digital assistant 406 is configured to understand the meaning of the utterance and take appropriate actions, which may involve requesting user input regarding the type of classifier the user wants to build, the data source, various performance metrics, and various constraints. The response 412 provided by the digital assistant 406 may also be in the form of natural language, which may involve natural language generation (NLG) processing performed by the digital assistant 406. Once the digital assistant 406 has the necessary information from the user, the digital assistant 406 may cause the generation of the machine learning application. The digital assistant 406 may end the conversation with the user by outputting information indicating that the machine learning application has been generated.

[0149] In some embodiments, an utterance received as input by the digital assistant 406 undergoes a series of processing steps or a pipeline of processing steps. These steps may include, for example, parsing the utterance, understanding the meaning of the utterance, refining and reconstructing the utterance to form a more understandable utterance structure, determining an action to be performed in response to the utterance, causing the action to be performed, generating a response to be output to the user in response to the user utterance, outputting the response to the user, etc.

[0150] The NLU processing performed by a digital assistant (such as digital assistant 406) may include various NLP-related processing, such as sentence parsing (e.g., tokenization, lemmatization, identifying part-of-speech tags for sentences, identifying named entities in sentences, generating dependency trees to represent sentence structure, splitting sentences into clauses, analyzing individual clauses, resolving anaphora, performing chunking, etc.). Digital assistant 406 may use an NLP engine and / or a machine learning model (e.g., an intent classifier) to map end-user utterances to specific intents (e.g., specific tasks / actions or categories of tasks / actions that a chatbot can perform). For example, a machine learning-based NLP engine may learn to understand and classify natural language conversations from end users and extract necessary information from the conversations so that precise actions can be taken, such as executing a transaction or looking up data from a back-end recording system. In some embodiments, NLU processing or portions thereof is performed by digital assistant 406 itself. In some other embodiments, digital assistant 406 may use other resources to perform portions of the NLU processing. For example, the grammar and structure of a sentence may be identified by processing the sentence using a parser, a part-of-speech tagger, and / or a named entity recognizer. In one embodiment, for the English language, the parser, part-of-speech tagger, and named entity recognizer provided by the Stanford Natural Language Processing (NLP) group are used to analyze sentence structure and grammar. These can be provided as part of the Stanford CoreNLP toolkit.

[0151] While the various examples provided in this disclosure show utterances in the English language, this is by way of example only. In some embodiments, the digital assistant 406 is also capable of handling utterances in languages other than English. In some embodiments, the digital assistant 406 provides subsystems (e.g., components that implement NLU functionality) that are configured to perform processing for different languages. These subsystems can be implemented as pluggable units that can be called using service calls from the NLU core server. This makes NLU processing flexible and scalable for each language, including allowing different processing orders. Language packs can be provided for various languages, where the language packs can register a list of subsystems that can provide services from the NLU core server and, if desired, can also utilize the provided general subsystems.

[0152] A digital assistant, such as digital assistant 406, can be available to its users through a variety of different channels, such as, but not limited to, via certain applications, via social media platforms, via various messaging services and applications, and other applications or channels. A single digital assistant can have multiple channels configured for it so that it can run simultaneously and be accessed by different services.

[0153] A digital assistant contains or is associated with one or more skills. In some embodiments, these skills are separate chatbots (called skill bots) designed to interact with users and perform specific types of tasks, such as tracking inventory, submitting time cards, creating expense reports, ordering food, checking bank accounts, making appointments, purchasing widgets, etc. For example, for Figure 4 In the embodiment depicted in , digital assistant 406 includes skills 416 - 1 , 416 - 2 , etc. For purposes of this disclosure, the term “skill” is used synonymously with the term “skill bot.”

[0154] Each skill associated with the digital assistant helps the user of the digital assistant complete tasks through a conversation with the user, where the conversation can include a combination of text or audio input provided by the user and responses provided by the skill bot. These responses can be in the form of text or audio messages to the user and / or using simple user interface elements (e.g., a list of choices) presented to the user for selection.

[0155] There are various ways to add skills or skill bots to a digital assistant. In some cases, a business can develop a skill bot and then add it to a digital assistant using DABP 402. In other cases, a skill bot can be developed and created using DABP 402 and then added to a digital assistant created using DABP 402. In still other cases, DABP 402 provides an online digital store (referred to as a "skill store") that offers a variety of skills for a wide range of tasks. The skills offered through the skill store can expose various cloud services. Users 404 of DABP 402 can access the skill store via DABP 402, select desired skills, and add the selected skill bot to a digital assistant created using DABP 402. Scales from the skill store can be added to the digital assistant as is or in a modified form (for example, a user of DABP 402 can select and clone a specific skill bot provided by the skill store, customize or modify the selected skill bot, and then add the modified skill bot to a digital assistant created using DABP 402).

[0156] In some embodiments, the digital assistant created and deployed using DABP 402 is implemented using a master robot / child (or sub) robot paradigm or architecture. According to this paradigm, the digital assistant is implemented as a master robot that interacts with one or more child robots that are skill robots. For example, in Figure 4 In the embodiment depicted in , the digital assistant 406 includes a master robot 414 and skill robots 416 - 1 , 416 - 2 , etc. that are child robots of the master robot 414. In some embodiments, the digital assistant 406 itself acts as the master robot.

[0157] The digital assistant implemented according to the master-sub robot architecture enables the user of the digital assistant to interact with multiple skills through a unified user interface. When the user interacts with the digital assistant 406, the user input is received by the master robot 414, which then processes the user input to identify the user request and, based on the processing, determines whether the user request task can be handled by the master robot 414 itself. Otherwise, the master robot 414 selects the appropriate skill robot 416-1, 416-2, or 416-3 to handle the user request and routes the conversation to the selected skill robot 416-1, 416-2, or 416-3. This enables the user 408 to talk to and use several skill robots configured to perform specific tasks through a common single interface. For example, for a digital assistant 406 developed for an enterprise, a master robot 414 of the digital assistant 406 can interface with skill robots 416 - 1 , 416 - 2 , etc. having specific functions, such as a CRM robot for performing functions related to customer relationship management (CRM), an ERP robot for performing functions related to enterprise resource planning (ERP), a human capital management (HCM) robot for performing functions related to HCM, etc. In this way, the end user or consumer 408 of the digital assistant 406 only needs to know how to access the digital assistant 406 .

[0158] In the master robot / sub robot infrastructure, the master robot is configured to be aware of a list of skill robots. The master robot has access to metadata identifying the various available skill robots, and for each skill robot, the skill robot's capabilities including the tasks that the skill robot can perform. Upon receiving a user request in the form of an utterance, the master robot is configured to identify or predict a specific skill robot from the multiple available skill robots that can best serve or handle the user request. The master robot then routes the utterance (or part of the utterance) to that specific skill robot for further handling. Thus, control flows from the master robot to the skill robots. The master robot can support multiple input and output channels.

[0159] Although Figure 4The embodiment in FIG. 4 shows a digital assistant 406 including a master robot 414 and skill robots 416-1, 416-2, and 416-3, but this is not intended to be limiting. The digital assistant may include various other components (e.g., other systems and subsystems) that provide the functionality of the digital assistant. These systems and subsystems may be implemented solely in software (e.g., code or instructions stored on a computer-readable medium and executable by one or more processors), solely in hardware, or in an embodiment using a combination of software and hardware.

[0160] DABP 402 provides the infrastructure, services, and features that enable users of DABP 402 to create a digital assistant, including one or more skill bots associated with the digital assistant. For example, a skill bot can be created by cloning an existing skill bot, cloning an existing skill bot and then modifying it, or it can be created from scratch using the tools and services provided by DABP 402. In some embodiments, DABP 402 provides a skill store or skill catalog that provides multiple skill bots for performing various tasks. Users of DABP 402 can clone skill bots from the skill store and create new skill bots.

[0161] DABP 402 also enables users (e.g., skill robot designers) to create skill robots from scratch. In some embodiments, at a high level, creating a skill robot involves the following steps:

[0162] (1) Configuring settings for new skill bots

[0163] (2) Configure one or more intents for the skill robot

[0164] (3) Configure entities for one or more intents

[0165] (4) Training skill robots

[0166] (5) Create a dialogue flow for the skill bot

[0167] (6) Adding custom components to skill robots

[0168] (7) Testing and deploying skill robots

[0169] (1) Configuring Settings for a New Skill Bot - The skill bot designer can specify one or more invocation names for the skill bot being created. These invocation names can be used in utterances to clearly identify and invoke the skill bot in the digital assistant. The skill bot designer can also specify example utterances for the skill bot. These example utterances represent what the skill bot would say. When receiving user input, the digital assistant's intent analysis engine compares the user input to these example utterances to determine whether to invoke a specific skill bot.

[0170] (2) Configuring one or more intents for a skill bot - A skill bot designer can configure one or more intents (also called bot intents) for the skill bot being created. These intents identify tasks that the skill bot can perform for the user of the digital assistant. Each intent is given a name. For example, for a skill bot that can be configured to generate a machine learning model, intents can be specified for the skill bot by the skill bot designer. For each intent, the skill bot designer specifies a set of example utterances that represent and illustrate the meaning of the intent and are generally associated with the task performed by that intent. Thus, a permutation of typical user requests and statements can be specified as example utterances for an intent.

[0171] (3) Configuring entities for one or more intents of a skill bot - In some cases, additional context may be required to enable the skill bot to correctly respond to a user request. For example, there may be situations where a user input utterance is parsed as the same intent in the skill bot. One or more entities may be specified for certain intents configured for the skill bot. Thus, entities are used to add context to the intent itself. Entities help to more fully describe the intent and enable the skill bot to complete the user request. In some embodiments, there are two types of entities:

[0172] (a) built-in entities provided by DABP 402, and (2) custom entities that can be specified by the skill bot designer. Built-in entities are general entities that can be used by various bots. Examples of built-in entities include, but are not limited to, entities related to time, date, address, number, email address, duration, recurring time period, currency, phone number, URL, etc. Custom entities are used for more customized applications.

[0173] (4) Training a Skill Bot - A skill bot is configured to receive user input, parse or otherwise process the received input, and identify or select an intent associated with the received user input. To do this, the skill bot must be trained. In some embodiments, the skill bot is trained based on the intents configured for the skill bot and example utterances associated with the intents (collectively referred to as training data) so that the skill bot can parse the user input into one of its configured intents. In some embodiments, the skill bot is represented by a model trained using the training data and allows the skill bot to discern what the user said (or in some cases, attempted to say). DABP 402 provides a variety of different training techniques that can be used by skill bot designers to train skill bots, including various machine learning-based training techniques, rule-based training techniques, and / or combinations thereof, as described herein with respect to a DAG-based framework. In some embodiments, a portion (e.g., 80%) of the training data is used to train the skill bot model, while another portion (e.g., the remaining 20%) is used to test or verify the model. Once trained, the skill bot can be used to process and respond to user utterances. In some cases, the user utterance may be a question that requires only an answer and does not require further conversation. To handle situations like this, you can configure a Q&A intent for your skill bot. This allows the skill bot to output responses to user requests without having to update the dialog definition. Creating a Q&A intent is similar to creating a regular intent. However, the dialog flow for a Q&A intent differs from that for a regular intent.

[0174] (5) Creating a conversation flow for a skill bot - The conversation flow specified for a skill bot describes how the skill bot reacts when resolving different intents of the skill bot in response to received user input. The conversation flow defines the operations or actions that the skill bot will take (e.g., how the skill bot responds to user utterances, how the skill bot prompts for user input, how the skill bot returns data). A conversation flow is like a flowchart that the skill bot follows. The skill bot designer specifies the conversation flow using a language such as Markdown. In some embodiments, a version of YAML called OBotML can be used to specify conversation flows for skill bots. The conversation flow definition for a skill bot acts as a model for the conversation itself, allowing the skill bot designer to orchestrate the interactions between the skill bot and the users of the skill bot service.

[0175] In some embodiments, a dialog flow definition includes three sections:

[0176] (a) Context section

[0177] (b) Default transition section

[0178] (c) Status section

[0179] Context Section - Skill bot designers can define variables used in the dialog flow in the Context Section. Other variables that can be named in the Context Section include, but are not limited to, variables for error handling, variables for built-in or custom entities, and user variables that enable skill bots to recognize and maintain user preferences.

[0180] Default Transition Section - A skill bot's transitions can be defined in either the Dialog Flow State section or the Default Transition section. Transitions defined in the Default Transition section serve as a fallback and are triggered when no applicable transition is defined in the state or when the conditions required to trigger a state transition are not met. The Default Transition section can be used to define routes that allow a skill bot to gracefully handle unexpected user actions.

[0181] State Segment - A dialog flow and its associated operations are defined as a series of temporary states that manage the logic within the dialog flow. Each state node within the dialog flow definition names a component that provides the functionality required at that point in the dialog. Thus, states are structured around components. States contain component-specific properties and define transitions to other states that are triggered after component execution.

[0182] You can use state sections to handle special cases. For example, you might want to give users the option to temporarily leave the first skill they're working in to do something in a second skill on your digital assistant. For example, if a user is in a conversation with a shopping skill (e.g., they've already made some purchase choices), they might want to jump to a banking skill (e.g., to make sure they have enough money for the purchase), and then return to the shopping skill to complete their order. To address this, you can configure an action in the first skill to initiate an interaction with a second, different skill on the same digital assistant, and then return to the original flow.

[0183] (6) Adding Custom Components to a Skill Bot - As described above, a state specified for a skill bot in a dialog flow is named a component that provides the required functionality corresponding to the state. Components enable a skill bot to perform a function. In some embodiments, DABP 402 provides a collection of preconfigured components for performing a wide range of functions. A skill bot designer can select one or more of these preconfigured components and associate them with a state in a skill bot dialog flow. A skill bot designer can also create custom or new components using the tools provided by DABP 402 and associate the custom component with one or more states in a skill bot dialog flow.

[0184] (7) Testing and Deploying Skill Bots - DABP 402 provides several features that enable skill bot designers to test the skill bots they are developing. The skill bots can then be deployed and included in digital assistants.

[0185] While the above description describes how to create a skill robot, similar techniques can also be used to create a digital assistant (or master robot). At the master robot or digital assistant level, built-in system intents can be configured for the digital assistant. These built-in system intents are used to identify general tasks that the digital assistant itself (i.e., the master robot) can handle without invoking the skill robot associated with the digital assistant.

[0186] At the master bot or digital assistant level, when a user enters a phrase or utterance into the digital assistant, the digital assistant is configured to perform processing to determine how to route the conversation. The digital assistant uses a routing model, which can be rule-based, AI-based, or a combination thereof, to determine this. The digital assistant uses the routing model to determine whether the conversation corresponding to the user input will be routed to a specific skill for handling, handled by the digital assistant or master bot itself based on built-in system intents, or handled as a different state in the current conversation flow.

[0187] In some embodiments, as part of this processing, the digital assistant determines whether the user input identifies a skill robot using its invocation name. If an invocation name is present in the user input, it is treated as an explicit invocation of the skill robot corresponding to the invocation name. In this scenario, the digital assistant can route the user input to the explicitly invoked skill robot for further processing. If no specific invocation is present, in some embodiments, the digital assistant evaluates the received user input and calculates confidence scores for the system intent and the skill robot associated with the digital assistant. The scores calculated for the skill robot or system intent indicate how likely it is that the user input represents the task the skill robot is configured to perform or represents a system intent. Any system intent or skill robot whose associated calculated confidence score exceeds a threshold (e.g., a confidence threshold routing parameter) is selected as a candidate for further evaluation. The digital assistant then selects a specific system intent or skill robot from the identified candidates to further process the user input. In some embodiments, after one or more skill robots are identified as candidates, the intents associated with those candidate skills are evaluated (according to the intent model for each skill) and a confidence score is applied to each intent. In general, any intent with a confidence score exceeding a threshold is considered a candidate flow. If a specific skill bot is selected, the user input is routed to that skill bot for further processing. If a system intent is selected, one or more actions are performed based on the selected system intent.

[0188] Figure 5 A distributed system 500 implementing a robot system for communicating with end users using a messaging application is depicted in accordance with certain embodiments. System 500 can include a robot system 520, one or more messaging application systems 515, and one or more end-user devices, such as one or more mobile devices 510. In some examples, the messaging application can be installed on an electronic device (e.g., a desktop computer, a laptop computer, a mobile device 510, etc.).

[0189] The messaging application can be facilitated by a messaging platform, such as messaging application system 515. Mobile device 510 can connect to messaging application system 515 via a first network (e.g., the Internet). Messaging application system 515 can be a messaging platform provided by a third party, such as Facebook, Tencent, Google, Microsoft, etc. Messaging application system 515 can manage content sent and received across multiple mobile devices or other end-user devices via the messaging application.

[0190] The robot system 520 (e.g., implemented on one or more servers) can also be communicatively connected to the messaging application system 515 to send and receive messages. Communication between the messaging application system 515 and the robot system 520 can be through a second network (e.g., the Internet). The first network and the second network can be the same network, or they can be similar or completely different networks. The messaging application system 515 can use the Internet to route content (e.g., a message or information from a message) from the mobile device 510 to the robot system 520. In some embodiments, the destination of the content (e.g., the identification of the robot system 520) can be included in the content as a nominal recipient. In some embodiments, the robot system 520 can also be configured to communicate with two or more messaging application systems 515.

[0191] As discussed above, content exchanged between end users or between end users and a robotic system may include, for example, text, emoticons, audio, media (e.g., pictures, videos, links), or any other method of conveying information.

[0192] The robot system 520 can receive content from the messaging application system 515 using a connector 530, which acts as an interface between the messaging application system 515 and the robot system 520. In some embodiments, the connector 530 can normalize the content from the messaging application system 515 so that the robot system 520 can analyze the content across different messaging application systems. Content normalization can include formatting the content from each type of messaging application into a common format for processing. In some embodiments, the robot system 520 can include one or more connectors for each messaging application (such as Messenger, Instant messaging software, Mobile text and voice messaging communications services, Messenger, Messenger and Skype messenger, short message service (SMS)). In some embodiments, connector 530 can route content to message input queue 540. Message input queue 540 can include a buffer (e.g., a first-in, first-out (FIFO) buffer) that stores content in the order it was received. In some embodiments, each connector 530 can be associated with one or more message input queues.

[0193] When the message processor 550 becomes available, the message input queue 540 can send the content to the message processor 550. In some embodiments, the message processor 550 can pull the content from the message input queue 540. The message processor 550 can parse the message and determine the intent of the parsed message, as described in detail below. In some embodiments, the message processor 550 may include a natural language processor 552 and an intent determination subsystem 554. The natural language processor 552 can parse the message and perform certain semantic analysis, such as identifying the subject, predicate (e.g., action) and / or object. The intent determination subsystem 554 can determine the end-user intent based on the parsed message. As described above, the intent can include the purpose of the message. For example, the purpose of the message can be to generate a machine learning model. In some embodiments, parameters associated with the intent to more specifically define or clarify the action to be taken can also be extracted from the message by the natural language processor 552 and / or the intent determination subsystem 554. These parameters can be referred to as entities.

[0194] After the message processor 550 determines the end-user intent based on the content, the determined intent (and parameters associated with the intent) can be sent to the action engine 560. The action engine 560 can be used to determine the action to be performed based on the intent (and parameters associated with the intent) as described above and the current state (or context) of the state machine. For example, the action engine 260 can send certain outbound content to the message output queue 570 as a response and / or can send commands to some enterprise services (such as enterprise services 525) or retrieve information from them. The message output queue 570 can send the outbound content to the connector 530. The connector 530 can then send the outbound content to the messaging application system indicated by the action engine 560, which can be the same as or different from the messaging application system 515. The messaging application system 515 can then forward the outbound content to the messaging application on the mobile device 510.

[0195] The robot system 520 can communicate with one or more enterprise services (e.g., enterprise service 525), one or more storage systems for storing and / or analyzing messages received by the robot system 520, or a content system for providing content to the robot system 520. The enterprise service 525 can communicate with one or more of the connectors 530, the action engine 560, or any combination thereof. The enterprise service 525 can communicate with the connectors 530 in a manner similar to the messaging application 515. The enterprise service 525 can send content to the connectors 530 for association with one or more end users. The enterprise service 525 can also send content to the connectors 530 to cause the robot system 520 to perform an action associated with the end user. The action engine 560 can communicate with the enterprise service 525 to obtain information from the enterprise service 525 and / or instruct the enterprise service 525 to take an action identified by the action engine 560.

[0196] In some embodiments, the bot system 520 may include one or more timers. The timers may cause the action engine 560 to send content to the end user using the connector 530 and the messaging application 515 after a certain amount of time has passed. In some embodiments, the timer may send content to the bot system 520 similarly to the end user or enterprise service 525. For example, the timer may send a message to the bot system 520 for analysis, as the message from the end user is to be analyzed.

[0197] As described above, building a chatbot system (such as a user intent classification engine that can determine end-user intent based on end-user utterances) is a challenging task, in part due to the subtlety and ambiguity of natural language and the dimensionality of the input space (e.g., possible user utterances) and the size of the output space (the number of intents). As a result, the chatbot may need to be trained, monitored, debugged, and retrained in order to improve the performance of the chatbot and the user experience of the chatbot. In many cases, it may be difficult to more specifically identify the root cause of the lower-than-expected performance of the chatbot system and determine how to improve the chatbot system without the use of analytical or optimization tools.

[0198] In some cases, a bot owner, developer, or administrator may want to monitor the operational status of a bot and understand how the bot is being used and where end users abandon the bot in order to improve the bot. For example, a bot owner, developer, or administrator may want to know which bot conversations were successful and which were not, in order to identify and diagnose underperforming elements of the bot system.

[0199] According to some embodiments, an analysis system can be integrated with the robotic system. The analysis system can monitor events that occur during conversations between end users and the robotic system, aggregate and analyze the collected events, and graphically provide information about the conversations at different levels of summary, such as all conversations, different categories of conversations, and individual conversations, on a graphical user interface. For example, the graphical user interface can display options for filtering or selecting certain types of conversations or individual conversations and graphically display the selected information, for example, by visualizing the path of the conversation. The analysis system can also provide suggestions, options, or other information for improving the robotic system.

[0200] Figure 6 An integrated system 600, according to certain embodiments, is depicted, comprising a robotic system (such as robotic system 220) and a robotic analytics system for monitoring, analyzing, visualizing, and improving the performance of the robotic system. As shown, the robotic system may include a connector 630 and multiple robotic engines, such as a conversation engine 612, an intent modeler 614, an entity resolver 616, and a custom component 618. The robotic system may also include a database 640, a management application programming interface (API) 650, a user interface 654, and a UI server 652. The robotic analytics system may include a collector 655, an enrichment engine 660, a database 670, and a REST server 680. The robotic analytics system may also include a user interface 692 and a UI server 690. The collector 655 of the robotic analytics system may collect events 605 occurring at the robotic system. Feedback 694 from the robotic analytics system may be provided to the robotic system via the user interface 692 and the user interface 654.

[0201] Connector 630 can act as an interface between the robot system and one or more end users through one or more channels such as channels 620 and 622. Each channel can be a messaging application, such as a messaging channel (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack or SMS), a virtual personal assistant (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), a mobile and web application extension of an extended native or hybrid / responsive mobile application or web application with chat capabilities, or a voice-based input (such as a device or application with an interface for interacting using Siri, Cortana, Google Voice or other voice input). In some embodiments, connector 130 can normalize content from different channels so that the robot system can analyze content across different messaging application systems. The content normalization process can include formatting the content from each type of messaging application into a common format for processing. In some embodiments, the robot system can include one or more connectors for each channel.

[0202] The intent modeler 614 can be used to determine the end-user intent associated with the end-user utterance. In some embodiments, the intent modeler 314 for determining the end-user's intent based on one or more messages received by the robot system from the end-user can use a natural language processor to mark the parts of the speech (verbs, nouns, adjectives), find lemmas / stems (runs / running / ran->run) and mark entities (Texas->LOCATION). In some embodiments, the intent modeler 314 can normalize the message. For example, "Mary ran to Texas" can become "PERSON run toLOCATION". The intent modeler can also include logic to detect words with the same meaning within the end-user message.

[0203] After normalization, the probability that the occurrence of a word can represent a particular intent can be determined. In some examples, the probabilities can be combined using basic probability arithmetic as if they were independent. Some probabilities can be based on the presence of the word, or on certain linguistic elements such as negation or the presence of personal pronouns.

[0204] Another level of rule can be a template rule, which is a combination of words. In some examples, each sentence in the training dataset can automatically become a rule once normalized. In such examples, the training dataset may include a very small number of short sentences. A template rule can return a probability of 1. New rules can be generated from rules via an inductive process.

[0205] If a particular word or set of words (such as a verb) is important for an intent, the probability can be manipulated by using more examples of this word (and its synonyms) and fewer examples of this word for other intents.

[0206] You can also provide examples to prevent the model from making incorrect assertions. For example, specific subphrases or words that only appear for a specific intent can lead to incorrect assertions. Similarly, you can prevent the model from synthesizing broad rules using similar sentences belonging to different training intents.

[0207] The entity resolver 616 can identify entities (e.g., objects) associated with the end-user intent. For example, in addition to the end-user intent identified by the intent modeler 614, such as "order a pizza," the entity resolver 616 can resolve entities associated with the intent, such as pizza type, toppings, etc.

[0208] The dialogue engine 612 can be used to handle conversations between end users and the robot system. For example, the dialogue engine 612 can respond to end user utterances based on the end user intent identified by the intent modeler 614 and the entities associated with the end user intent identified by the entity resolver 616. In some embodiments, the dialogue engine 612 can use a state machine that includes user-defined states (e.g., end user intents) and actions taken in or from states to handle conversations with end users.

[0209] Custom components 618 may include custom modules for a particular robot system. For example, a financial robot may include custom components that may be used, for example, to check balances, transfer funds, or pay bills.

[0210] Database 640 can be used to store data for the robot system, such as data used for classification models, conversation logs, etc. Management API 650 can be used by administrators or developers of the robot system to manage the robot system, such as retraining classification models, editing intents, or otherwise modifying the robot system. Administrators or developers can use user interface 654 and UI server 652 to manage the robot system.

[0211] A variety of events can be generated during the operation of a robotic system. Events can be generated based on one or more instructions included in the robotic system. For example, an event can be generated when the robotic system enters a specific state, where the specific state is defined by the robotic system's administrator or developer. As events are generated, they can be collected, stored, and analyzed by the robotic analytics system. When events are captured, additional information associated with the event can also be collected, where the additional information can indicate the current context in which the event was generated.

[0212] For example, a chat event can be generated by the dialog engine 612. A chat event can include a message received by the robot system from an end-user device (referred to as msg_received). msg_received can include one or more of the following parameters or variables: the content of the message, the time the message was received by the robot system, the language of the received message, device characteristics (e.g., version or name), operating system characteristics (e.g., version or name), geographic location characteristics (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, tenant ID, etc.), timestamps (e.g., device created, device sent, collector-derived timestamps), channels, etc.

[0213] A conversation event may also include a message sent by the robot system to the end-user device (referred to as msg_sent). msg_sent may include one or more of the following: the content of the message (e.g., the text or HTML of the message), the time the message was sent by the robot system, the language of the message, the creator of the message (e.g., the robot system or the end-user device), device properties, operating system characteristics, browser characteristics (e.g., version or name), application characteristics (e.g., version or name), geographic location characteristics (e.g., Internet Protocol address, latitude, longitude, etc.), identification information (e.g., user ID, session ID, robot system ID, tenant ID, etc.), channel (e.g., Facebook or webhook), etc.

[0214] The dialogue engine 612 may also generate dialogue state execution events. As described above, the dialogue engine 612 may use a state machine to determine the flow of a conversation with an end user. The state machine may include a set of states and rules for transitioning between states. The dialogue engine 612 may execute the state machine for each end-user conversation and generate a dialogue state execution event for each state that the dialogue engine 612 steps through to process an end-user utterance. Attributes of a dialogue state execution event may include, for example, state name, component name, next action, entity match, intent match, variables, user query statement, response statement, execution time, communication language, device characteristics, operating system characteristics, geolocation characteristics, identification information, timestamp, channel, and so on. The state name may be the name of the currently executed state or an "error state." The component name may be the name of the bot component being executed for the current state. The next action may be the next action to be executed. The entity match may be the entity parsed in the current message. The intent match may be the intent parsed using a score value. The variable may be the variable value of the current state. The query statement may be the message sent by the end user. The response statement may be the message sent to the end user. The execution time may be the timestamp of the completed state execution. The communication language may be the language of the message being converted. Device and / or operating system characteristics may be associated with the end user interacting with the robotic system. Browser and / or application characteristics may be associated with the end user interacting with the robotic system. Geographic location characteristics may be the location of the end user interacting with the robotic system.

[0215] As a result of execution of the intent modeler 614, an intent resolution event may occur. The intent modeler 614 may use a trained or otherwise defined classification model to identify an end-user intent from a set of intents based on an end-user utterance. The results of the intent classification may be captured as intent resolution event attributes, which may include, for example, a final intent classification result (e.g., the identified intent) and a confidence score associated with each corresponding intent in the set of intents.

[0216] Entity resolver 616 can generate entity resolver events. An entity is an object associated with an end-user intent. Entity definition rules can be determined when creating a bot system. For example, in addition to resolving an end-user intent such as "order a pizza," the bot system can also use entity resolver 616 to resolve associated entities such as pizza type, toppings, etc. Entity resolver events can be captured during entity resolution. Examples of characteristics associated with entity resolver events include entity name, applied rules, search terms, resolved state, query statement, entity type, execution time, communication language, device characteristics, operating system characteristics, browser characteristics, application characteristics, geolocation characteristics, identification information, timestamp, channel, etc. The entity name can be the name of the entity currently being resolved. The applied rules can be, for example, precede, follow, or aggregate. The search terms can be from, to, destination, origin, etc. The resolved state can be the conversation state resolved for the entity. The query statement can be a message containing an entity value. The entity type can be system or derived. The execution time can be the timestamp of the entity resolution. The communication language can be the language of the message being converted. Device and / or operating system characteristics may be associated with an end user interacting with a robotic system. Browser and / or application characteristics may be associated with an end user interacting with a robotic system. Geographic location characteristics may be the location of an end user interacting with a robotic system.

[0217] Custom components can also generate events, such as predefined events or custom events. Predefined events can be characteristics captured when executing a custom component. Examples of predefined event attributes include: component name, event name, payload, execution time, communication language, device characteristics, operating system characteristics, browser characteristics, application characteristics, geolocation characteristics, identification information, timestamp, channel, etc. The component name can be the name of the currently executing custom component. The event name can be invoked, invocation_failed, resolved, resolved_failed, etc. The payload can be the cause of the failure, a stack trace, etc. in the event of a failure. The execution time can be a timestamp indicating when the event occurred. The communication language can be the language of the message being converted. Device and / or operating system characteristics can be associated with the end user interacting with the robotic system. Browser and / or application characteristics can be associated with the end user interacting with the robotic system. The geolocation characteristic can be the location of the end user interacting with the robotic system.

[0218] Custom components can also emit custom events during the execution of the custom component. Examples of custom event properties may include component name, event name, custom payload, time of execution, communication language, device characteristics, operating system characteristics, browser characteristics, application characteristics, geolocation characteristics, identification information, timestamp, channel, etc. Component name may be the name of the currently executing custom component. Event name may be a user-defined event name (e.g., Balance_Retrieved). Payload may be, for example, {"amount":"USD 100","account":"checking"}. Time of execution may be a timestamp indicating when the event occurred. Communication language may be the language of the message being converted. Device and / or operating system characteristics may be associated with the end user interacting with the robotic system. Browser and / or application characteristics may be associated with the end user interacting with the robotic system. Geolocation characteristics may be the location of the end user interacting with the robotic system.

[0219] Error events and timeout events can also be generated by the robot system during execution. Error events can be generated when an error occurs. Timeout events can be generated when the end-user chat is inactive for a period of time. This timeout event can be configured at the channel.

[0220] When the robot system converses with the end user and generates corresponding events, the robot analysis system can collect the events and additional information. For example, the collector 655 can collect the events and additional information and send the collected information to a queue. In some embodiments, the collector 655 can be configurable and can be programmed to collect the different events and / or event attributes described above as desired. For example, the collector 655 can be configured to capture conversation state attributes, intent resolution attributes, entity resolution attributes, and error and timeout attributes. In some embodiments, the collector 655 can also be configured to collect information about events 695 generated by systems other than the robot system.

[0221] The enrichment engine 660 can perform validation and enrichment on the collected events and other information and write them to the database 670. For example, based on the collected IP address, the enrichment engine 660 can determine the location of the end user associated with the IP address. As another example, the enrichment engine 660 can extract certain features from the collected information, such as determining the web browser or channel used by the end user. The REST server 680 can analyze the enriched events and other information and generate various reports based on certain aggregate metrics 672. The reports can be displayed to the owner, administrator, or developer of the robotic system on a user interface 692 via the UI server 690. The owner, administrator, or developer of the robotic system can provide feedback 694 to the robotic system to improve the robotic system.

[0222] Figure 7 A process 700 is illustrated for a technique for an intelligent assistant to generate a machine learning solution. Alternative embodiments may be combined, separated, or otherwise modified. Figure 7 To change the function, click the function described in the box shown in the figure. Figure 7 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and 24 and the computing devices, storage devices, network infrastructure, and servers described below.

[0223] A robot (also known as a chatbot, chatbot, or conversational robot) is a computer program that can simulate a conversation with a human user. A robot can generally respond to natural language messages (e.g., questions or comments) through a messaging application that uses natural language messages. An enterprise can use one or more robot systems to communicate with end users. The messaging application, which can be referred to as a channel, can be a user-preferred messaging application that the end user has already installed and is familiar with. Therefore, the end user can chat with the robot system without downloading and installing a new application. For example, the messaging application can include over-the-top (OTT) messaging channels (such as Facebook Messenger, Facebook WhatsApp, WeChat, Line, Kik, Telegram, Talk, Skype, Slack, or SMS), virtual personal assistants (such as Amazon Dot, Echo or Show, Google Home, Apple HomePod, etc.), mobile and web application extensions of native or hybrid / responsive mobile or web applications with chat functionality, or voice-based input (such as a device or application with an interface that interacts using Siri, Cortana, Google Voice, or other voice input).

[0224] In some examples, a robot system can be associated with a uniform resource identifier (URI). The URI can use a string to identify the robot system. The URI can be used as a webhook for one or more messaging applications. The URI can include, for example, a uniform resource locator (URL) or a uniform resource name (URN). The robot system can be designed to receive messages (e.g., hypertext transfer protocol (HTTP) post-call messages) from the messaging application system. The HTTP post-call message can be directed from the messaging application system to the URI. In some embodiments, the message can be different from the HTTP post-call message. For example, the robot system can receive a message from a short message service (SMS). Although the discussion herein may refer to communications received as messages by the robot system, one of ordinary skill in the art will recognize that the message can be an HTTP post-call message, an SMS message, or any other type of communication between two systems.

[0225] End users can interact with the robot system through conversational interactions (sometimes called conversational user interfaces (UIs)), just as they would with other people. In some cases, the interaction can include the end user saying "hello" to the robot, the robot responding with "hi," and asking the end user how it can help. In some cases, the interaction can also be a transactional interaction with, for example, a banking robot, such as transferring funds from one account to another; an informational interaction with, for example, an HR robot, such as checking a vacation balance; or an interaction with, for example, a retail robot, such as discussing a return or seeking technical support.

[0226] In some embodiments, the robot system can handle user interactions without interacting with the administrator of the robot system. For example, an end user can send one or more messages to the robot system in order to achieve a desired goal. The message can include certain content, such as text, emoticons, audio, images, video, or other methods of conveying the message. In some embodiments, the robot system can convert the content into a standardized form (e.g., a REST call to an enterprise service with appropriate parameters) and generate a natural language response. The robot system can also prompt the end user to enter additional input parameters or request other additional information. In some embodiments, the robot system can also initiate communication with the end user.

[0227] A conversation with a robot can go through a specific flow that includes multiple states. The flow can define what happens next based on input. In some embodiments, a state machine that includes user-defined states (e.g., end-user intent) and actions taken in or from states can be used to implement the robot system. The conversation can take different paths based on the end-user's input, which can affect the decisions the robot makes for the flow. For example, in each state, based on the end-user's input or utterance, the robot can determine the end-user's intent to determine the appropriate action to take next.

[0228] An intent can include the goal an end user wants to achieve. Intents map end-user input to actions that the backend system should perform on the end user's behalf. Therefore, based on the phrases spoken by the end user in natural language, the bot maps the user utterance to a specific use case or task, such as ordering a pizza, getting an account balance, transferring money, making a purchase, returning an item, and so on. Human conversations are often nonlinear in nature. End users may often branch into different states during the conversation. For example, if an end user wants to transfer funds from Account A to a recipient, the end user can begin a conversation with the bot system, asking the bot to pay the recipient for dinner. For example, the bot might respond with "From which account?" The end user might choose a checking account but then realize they're unsure of the balance in the account. Therefore, the end user might switch context to ask about the balance and recent transactions, and so on. In other words, the end user can trigger a change in flow and state, for example, from transferring money to checking the balance and then to recent transactions. At some point, the end user may decide to return to the original intent—paying the recipient. Therefore, one of the tasks of the bot system is to determine user intent from natural language utterances.

[0229] Bots can use natural language processing (NLP) engines and / or machine learning models (e.g., intent classifiers) to map user utterances to specific intents. For example, a machine learning-based NLP engine can learn to understand and classify natural language conversations from end users and extract the necessary information from the conversations so that precise actions can be taken, such as executing a transaction or looking up data from a backend recording system.

[0230] Figure 7 A flow diagram of a process 700 for generating a machine learning application using a chatbot is illustrated.

[0231] At 702, the technique may include receiving a description of a method to be used to generate a Figure 1The interface 104 shown in generates a first input of a problem to be solved by a machine learning solution. The first input can be received via auditory (e.g., spoken by a user and received by a microphone of the system), text (e.g., entered by a keyboard), or interaction with a graphical user interface (GUI). For example, a human resources manager may desire to identify employees who have shown improvement over the past few review cycles. Here, the user may be a human resources manager who may have experience using software tools but no experience programming machine learning tools or software development. The problem can be defined as determining employee trends. A machine learning algorithm can be used to analyze employee records, identify information about each employee's performance, and perform comparative analysis of historical reviews to determine each employee's performance trends. The machine learning algorithm can compare each employee's performance trends against the performance trends of other employees.

[0232] At 704, the technology can include rewriting the input into one or more text segments. In embodiments where the first input is received auditorily, the technology can use natural language processing to rewrite a captured audio clip containing the input into one or more text segments. If the first input is received as one or more words of text, the linguistics module can determine whether the one or more words of text are compatible with the automated machine learning system. A translation algorithm can be used to translate the one or more text segments into a compatible language. The one or more text segments can be stored in a memory of the system.

[0233] At 706, the technique may include determining the intent of the first input based at least in part on classifying the one or more text segments to create a machine learning architecture. The technique may compare the one or more text segments to keywords associated with creating a machine learning model. If the model composition engine 132 determines that the one or more keywords are present in the one or more text segments, then Figure 1 The model composition engine 132 shown in can start the process. If one or more keywords are not present in one or more text segments, then the technology can return to the beginning and continue to receive input.

[0234] At 708, the technique may include combining one or more text segments with a database stored in a database such as Figure 1 16. The model composition engine 132 may be configured to be associated with one or more of the plurality of machine learning models in the library component 168 shown in FIG. Each of the one or more machine learning models may have associated metadata. The associated metadata may be compared to the one or more text snippets. In this manner, the model composition engine 132 maps the first input or query of the user 116 to certain phrases to determine the intent of the user 116. If the correlation between the one or more text snippets and the associated metadata exceeds a predetermined percentage, then Figure 1The model composition engine 132 shown in FIG 1 identifies a machine learning model as being relevant to one or more text segments. The model composition engine 132 can recommend relevant machine learning models to the user. The model composition engine 132 can present the relevant models via a user interface, a chatbot, or a display.

[0235] At 710, the technology can present the relevant models to the user. The type of machine learning solution can be selected via an intelligent assistant (e.g., a chatbot). Possible machine learning solutions can include, but are not limited to, classification models, recommender models, or reinforcement learning models. The model composition engine 132 can display the relevant models on the interface 104. The model composition engine 132 can present the relevant models to the user via an intelligent assistant (e.g., a chatbot). In various embodiments, the model composition engine 132 can present multiple models to the user for selection.

[0236] At 712, the technology may include receiving a selection of a machine learning model, wherein the selection is selected from one or more machine learning models. The user 116 may select the model through the interface 104. The interface 116 may be a graphical user interface including, but not limited to, a checkbox, a drop-down menu, or a graphical symbol that can be selected by the user 116. The interface 116 may also receive audio information from the user 116 that indicates the selection of the machine learning model. The interface 116 may receive the selection of the model and transmit the selection information to the model composition engine 132. For example, the user 116 may select between a binary or multi-class model. The user 116 may also select a tree-based model or a vector-based machine type model.

[0237] At 714, the technique may include receiving a second input for the location of the data. The second input may include the location of the data source used to generate the machine learning model. The data may be stored locally in memory. The data may also be stored on one or more remote servers (e.g., cloud servers). The model composition engine 132 may identify potential locations of the data and present them to the user 116 via the interface 104.

[0238] At 716, the technique may include receiving a third input for one or more constraints. Constraints may include, but are not limited to, system resources, location, security requirements, and privacy. The model composition engine 132 may identify one or more application constraints based at least in part on the selected model. The model composition engine 132 may present the potential constraints to the user 116 via the interface 104.

[0239] In various embodiments, the technology may include a fourth input. The fourth input may include additional services required by the machine learning model. Additional services may include monitoring, logging, and alerting services. The model composition engine 132 may identify one or more additional services based at least in part on the selected model. The model composition engine 132 may present potential services to the user 116 via the interface 104.

[0240] At 718, the technique can include generating a plurality of codes for the machine learning architecture. The model composition engine 132 can generate a trained model using the selected model and the selected data.

[0241] The technology disclosed herein uses a hierarchical classification model to create and / or query a hierarchical classification model to identify a classification based on user input data (e.g., to determine user intent). In certain embodiments, a computer-implemented method for querying a hierarchical classification model to associate an input with a class from a plurality of classes is disclosed. The computer-implemented method may include receiving an input from a hierarchical classification model, the hierarchical classification model comprising a plurality of classification models and having a tree structure comprising a plurality of nodes on a plurality of layers, and sequentially classifying the input as being associated with a class from a plurality of classes using a set of classification models associated with a set of nodes in the tree structure. The plurality of nodes may include leaf nodes and non-leaf nodes, wherein the non-leaf node may include a root node on a first layer of the plurality of layers. Each of the leaf nodes and the non-leaf nodes may be associated with a corresponding classification model from a plurality of classification models of the hierarchical classification model. Each leaf node may be associated with at least two but less than a threshold number of classes. Each non-leaf node may have two child nodes, wherein each child node may be associated with a corresponding set of classes from a plurality of classes. The difference in the number of classes in the two sets of classes may be zero or one, and the classification model associated with the non-leaf node may be configured to classify the input into the two sets of classes. The set of nodes may include a node on each level of the tree structure and may form a path from the root node to the leaf nodes. Each node in the set of nodes, except the root node, may be a child node of a node on an immediately upper level of the tree structure.

[0242] In some embodiments, classifying the input as being associated with a class from among the plurality of classes may include classifying the input as belonging to a class associated with a first child node of the root node, using a binary classification model associated with the root node, wherein the first child node may be on a second level of the tree structure. In some embodiments, classifying the input as belonging to a class associated with the first child node of the root node may include: determining, using the binary classification model, a first value indicating a likelihood that the input belongs to the class associated with the first child node of the root node; determining, using the binary classification model, a second value indicating a likelihood that the input belongs to the class associated with the second child node of the root node; and classifying the input as belonging to the class associated with the first child node of the root node based on determining that the first value is greater than the second value or based on determining that the first value is greater than a threshold. In some embodiments, classifying the input as being associated with a class from among the plurality of classes may further include classifying, using a second binary classification model associated with the first child node on the second level, the input as belonging to a class associated with a child node of the first child node, wherein the child node of the first child node may be on a third level of the tree structure. In some embodiments, the second binary classification model may include a support vector machine (SVM) classifier, a logistic regression classifier, a naive Bayes classifier, a decision tree classifier, a nearest neighbor classifier, or a neural network classifier.

[0243] In some embodiments, classifying the input as being associated with a class from the plurality of classes may include classifying the input as being associated with a class using a multi-class classification model associated with a leaf node in the set of nodes, wherein the multi-class classification model may be configured to distinguish between inputs associated with two or more individual classes. In some embodiments, the multi-class classification model may include a multi-class support vector machine (SVM) classifier, a K-nearest neighbor classifier, or a neural network classifier.

[0244] In some embodiments, each path in the tree structure from the root node to the corresponding leaf node may include the same number of nodes. In some embodiments, the threshold number may be 6 or less. The plurality of classes may include 20 or more classes. In some embodiments, the input may correspond to a user utterance to the chatbot, and the plurality of classes may correspond to a user intent associated with the user utterance.

[0245] In certain embodiments, a non-transitory computer-readable medium may store a plurality of instructions executable by one or more processors. When executed by the one or more processors, the plurality of instructions may cause the one or more processors to receive an input through a hierarchical classification model, the hierarchical classification model being configured to associate the input with a class from a plurality of classes and including a plurality of classification models. The hierarchical classification model may have a tree structure comprising a plurality of nodes across multiple layers. The plurality of nodes may include leaf nodes and non-leaf nodes, wherein the non-leaf nodes may include root nodes on the first layer of the plurality of layers. Each of the leaf nodes and the non-leaf nodes may be associated with a corresponding classification model from the plurality of classification models of the hierarchical classification model. Each leaf node may be associated with at least two classes but less than a threshold number. Each non-leaf node may have two child nodes, wherein each child node may be associated with a corresponding set of classes from the plurality of classes. The difference between the number of classes in the two sets of classes may be zero or one, and the classification models associated with the non-leaf nodes may be configured to classify the input into the two sets of classes. The plurality of instructions may also cause the one or more processors to sequentially classify the input as being associated with a class from the plurality of classes through the set of classification models associated with the set of nodes in the tree structure. The set of nodes may include a node on each level of the tree structure and may form a path from the root node to the leaf nodes. Each node in the set of nodes, except the root node, may be a child node of a node on an immediately upper level of the tree structure.

[0246] In certain embodiments, a system may include one or more processors and a memory coupled to the one or more processors and storing instructions. When executed by the one or more processors, the instructions may cause the system to receive input via a hierarchical classification model configured to associate the input with a class from a plurality of classes and including a plurality of classification models. The hierarchical classification model may have a tree structure comprising a plurality of nodes across multiple layers. The plurality of nodes may include leaf nodes and non-leaf nodes, wherein the non-leaf nodes may include root nodes on a first layer of the plurality of layers. Each of the leaf nodes and the non-leaf nodes may be associated with a corresponding classification model from the plurality of classification models of the hierarchical classification model. Each leaf node may be associated with at least two classes but less than a threshold number. Each non-leaf node may have two child nodes, wherein each child node may be associated with a corresponding set of classes from the plurality of classes. The difference between the number of classes in the two sets of classes may be zero or one, and the classification models associated with the non-leaf nodes may be configured to classify the input into the two sets of classes. The instructions may also cause the one or more processors to sequentially classify the input as being associated with a class from the plurality of classes using the set of classification models associated with the set of nodes in the tree structure. The set of nodes may include a node on each level of the tree structure and may form a path from the root node to the leaf nodes. Each node in the set of nodes, except the root node, may be a child node of a node on an immediately upper level of the tree structure.

[0247] At 720, the technology may include saving the plurality of codes to a memory. The plurality of codes may be saved in a memory of a computing system or in a memory of a remote server (e.g., a cloud server). The plurality of codes may be executable code. The plurality of codes may be configured to be incorporated into one or more applications.

[0248] In various embodiments, user 116 can select how to receive the solution. The solution can be received via interface 104. The solution can be received via an intelligent assistant (e.g., a chatbot). The solution can be communicated via a speaker. The intelligent assistant can also suggest multiple predefined environments in which to present the information. Additional environments can also be provided for the chatbot.

[0249] In various embodiments, the intelligent assistant can analyze the one or more selected constraints to generate a second plurality of codes for the machine learning architecture based on optimizing the one or more selected constraints. The model execution engine 108 can display the optimized solution generated by the second plurality of codes to the user. In various embodiments, the user can identify test data to determine the output of the machine learning model. Before deployment, the intelligent assistant can display the optimal solution that satisfies the constraints to the user.

[0250] The model composition engine 132 can be tested during the development of the machine learning framework. For example, if we are creating a model to test worker productivity, the model composition engine 132 can look at employee record data, build a machine learning model to optimize productivity, apply a sample data set, and provide the answer to the user 116. This answer can be compared to the user's opinion of the most productive users to help validate the model. The chatbot can also ask the user 116 if they would like a quarterly report on the most productive workers.

[0251] In some embodiments, the end user can control the development. If a simple interface (e.g., a drop-down menu) is used, the user can interact with the chatbot to change the default view of the drop-down list. The chatbot can ask what is desired in the drop-down list. The chatbot can modify the drop-down list based on user preferences. In some embodiments, the end user can interact with the model composition engine 132. In this way, the end user can customize the interaction with the artificial intelligence system.

[0252] In some embodiments, the chatbot will obtain the user's initial intent to create a new machine learning solution or modify an existing machine learning solution.

[0253] The model composition engine 132 may use many default settings. Even if all settings are not specified by the user 116, the default settings can help manually assist in building the machine learning architecture. For example, the user 116 may not specify where the training model should be stored or the data source used to train the model. In various embodiments, the model composition engine 132 can use these default settings when creating the training model. In various embodiments, the intelligent assistant can detect that the user 116 did not specify settings during the process of generating the model. The intelligent assistant can ask the user 116 for settings or recommend settings to the user 116.

[0254] In various embodiments, the model composition engine 132 tracks user explicit input for assumptions and default settings. The model composition engine 132 can use these explicit inputs to create new machine learning models or processes.

[0255] In various embodiments, a computer-implemented method may include one or more services including monitoring, logging, and alerting for a machine learning architecture.

[0256] In various embodiments, the computer-implemented method may include receiving a fourth input identifying how the machine learning solution is presented.

[0257] In various embodiments, a computer-implemented method may include analyzing one or more constraints to generate a second plurality of codes for a machine learning architecture based at least in part on optimizing the one or more constraints. The method may also include generating an optimized solution. The method may include displaying the optimized solution.

[0258] In various embodiments, a computer-implemented method may include deploying a machine learning architecture via an intelligent assistant interface.

[0259] In various embodiments, the first input may include at least one of an auditory input, a textual input, and a graphical user interface input.

[0260] In various embodiments, the one or more machine learning models may include at least one of a classifier model, a recommender model, and a reinforcement learning model.

[0261] In various embodiments, the one or more constraints may include at least one of resources, location, security, and privacy.

[0262] It should be recognized that Figure 7 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present disclosure may perform the above steps in a different order. Moreover, Figure 7The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0263] III. Automated customization and deployment of a given product to any new client

[0264] Once the core machine learning product that solves the core problem for a particular instantiation is specified, it is matched with a machine learning model. The machine learning model is specified from a library component 168, which includes a pipeline 136 that specifies various microservice routines 140, software modules 144, and / or infrastructure modules 148. Functional areas of the library component 168 are customized on a per-instantiation basis to accommodate the unique client's data, QoS, KPIs, and other requirements, such as automatic adaptation of features for the library component 168, automatic bias removal in the machine learning model, automated model training to achieve QoS and KPIs, and automated microservice routine deployment configuration.

[0265] Various publicly available technologies allow for the automation of large parts of the traditional development process. Automation can be accomplished at scale through frameworks that instantiate instances of machine learning models for specific situations. Even after the machine learning model is compiled into the machine learning application, monitored anomalies can be caught before they manifest as poor predictions. This can be in the form of various QoS or KPI values for the entire machine learning application, or as intermediate results for various library components.

[0266] IV. Secure Serialization of Prediction Pipelines (Including Models)

[0267] Persistent machine learning models can often be implemented in executable code (e.g., Pickle). Such executable code can be vulnerable to security vulnerabilities (such as malicious code injection). A problem exists if the storage is compromised and a bad agent can tamper with the serialized machine learning model by modifying the various library components 168.

[0268] In various embodiments, the library component 168 can be cryptographically signed when stored and / or for each instance of the serialized machine learning model pipeline 136. Each library component 168 can have a unique key for each component instance or be shared across instances of the machine learning model. The private key can be stored in a secure location separate from the library component 168 in the key storage 172. Access to the secure location can be authenticated and authorization checks performed. Encryption and / or one-way hashing functions can be used to detect tampering with the code. Decryption / encryption can be performed using the remotely stored key for each library component 168 and, optionally, the machine learning model itself. In this way, malicious code cannot enter the library component 168 or integrate the library component 168 into the machine learning model or compiled machine learning application. Any type of Python model (etc.) and step in the pipeline or workflow can be signed or encrypted. Once a library component 168 or machine learning model is customized for a given instance, it can be encrypted or signed. Uncustomized library components can also be signed or encrypted so that hackers cannot infect multiple instances.

[0269] Figure 8 Illustrated is a process 800 for securely serializing a prediction pipeline (including a model). Alternative embodiments may be combined, separated, or otherwise modified. Figure 8 To change the function, click the function described in the box shown in the figure. Figure 8 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0270] At 802, the functionality may include receiving a library component.

[0271] At 804 , the functionality may include generating a unique key pair for one or more library components.

[0272] At 806, functionality may include remotely storing keys.The key storage may be separate from one or more library components.

[0273] At 808, functionality may include signing / encryption and repository components.

[0274] At 810, functionality may include authenticating a repository component with a remote key upon request.

[0275] At 812, the functionality may include using the library component in a machine learning model.

[0276] At 814, the functionality may include generating a unique key for the machine learning model.

[0277] At 816 , functionality may include remotely storing the model key.

[0278] At 818, functionality may include authenticating all interactions with the machine learning model with the model key.

[0279] At 820, functionality may include rejecting any component or model that fails authentication.

[0280] It should be recognized that Figure 8 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present disclosure may perform the above steps in a different order. Moreover, Figure 8 The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0281] V. Automated on-demand library checking (JAVA / PYTHON / GO, with all dependencies)

[0282] Various embodiments described herein relate to automation and / or on-demand analysis and prediction for code integration requests. For example, during the development, maintenance or modification of the source code for a software project or component, software developers can integrate various external code libraries into the source code. For example, such external code libraries can include open source software developed and distributed by a third-party open source provider. Some open source libraries and other external code libraries can have associated licenses, and software developers who wish to integrate any code from the external code library into their own proprietary software projects must agree to and comply with these licenses. The software license associated with some libraries (for example, free and open source software (FOSS) licenses) can be consistent and compatible with the corresponding licenses of other software libraries, or inconsistent and / or incompatible. In addition, some libraries and other external code libraries can have known security vulnerabilities and / or cost considerations, which can make these libraries more or less expected to be integrated into the software project. In addition, open source libraries and other external code libraries themselves can include dependencies on other libraries or code libraries, which can have separate licenses, costs, security issues and other dependencies.

[0283] Because of the potential risks, issues, and impacts of integrating external libraries and code bases into software projects, organizations may include a software architecture authorization system to analyze code integration requests and approve or deny such code integration requests based on one or more potential code integration issues, including licensing compliance or compatibility, security vulnerabilities, cost, further software dependencies, recency and priority of the software project, availability of security patches, and the existence of more secure alternative libraries. Such software architecture authorization systems may perform the analysis and approval of requests to integrate software libraries or any external code base using a fully automated process, a fully manual process, or a semi-automated and semi-manual process. Many such authorization systems involve significant time delays in processing requests from customers developing software projects to integrate software libraries or external code, and such systems may not provide any explanation or suggestions for alternative software bases.

[0284] Thus, various techniques for performing on-demand analysis and prediction of automation and / or code integration requests are described herein (e.g., methods, computing devices and systems, non-transitory computer-readable storage media storing programs, codes, or instructions executable by one or more processors, etc.). In certain embodiments, a code integration request prediction system can be implemented and used to analyze source code and predict the results of a request to integrate an external code library (e.g., an open source library) into a software project. In some embodiments, the code integration request prediction system can be implemented as a real-time static source code analysis plug-in within a software development environment. As described in more detail below, such a system can provide predictions of the results of potential code integration requests, including specific results (e.g., approval or rejection of the use and integration of the software library), corresponding confidence levels, reasons for the predictive results, and suggestions for alternative libraries. Additionally or alternatively, the code integration request prediction system can be implemented as a separate, standalone system that can review and analyze source code in real time during code development or maintenance, or via a source code crawling and analysis process. For example, a source code crawler can be used to detect the presence of external code libraries integrated in previously written source code (e.g., via references to known software libraries or code comments), and the software architecture authorization system can perform analysis and predictions on whether the external code libraries will be or should be allowed to be integrated into the software project.

[0285] As described above, certain techniques described herein can be implemented to predict the outcome of a software code integration request. In some embodiments, a model execution engine (e.g., within a code integration request prediction server or within a plug-in within a software development environment) can receive input data corresponding to a request to integrate an external code base into a source code project or component. This input data can identify one or more external code bases (e.g., open source software functions, libraries, etc.) associated with the source code project and component, including the external code base to be integrated and / or additional external code bases that have previously been integrated into the same project or component. In addition, the input data for the code integration request can include one or more characteristics of the source code project or component, such as the source code component's associated product or project, associated developers or organizations, the purpose of integrating the external code base, or functionality to be fully utilized in the external code base. The model execution engine can then access one or more machine learning (e.g., artificial intelligence-based) models trained using historical software code integration requests. Using (one or more) trained machine learning models, the model execution engine can determine a predicted outcome of a request to integrate external code into a source code project / component (e.g., via a software architecture authorization system). After determining a predicted outcome of the code integration request, the predicted outcome can be provided to, for example, a developer within a software development environment via an interface. In some embodiments, if the predicted outcome of the code integration request is that the organization's software architecture authorization system is likely to reject the code integration request, additional information can be generated and provided to the developer / user, such as a recommendation that led to the rejection of the code integration request and / or an alternative code base that can support similar functionality.

[0286] Thus, the various techniques described herein can provide significant technical advantages in the context of real-time software development, software maintenance, and software source code analysis and review. Specific technical advantages can include enabling real-time scanning of source code to detect code integration from external sources, improving code scanning to identify potential security vulnerabilities and licensing issues, either for code under development or for previously deployed legacy code.

[0287] Various aspects of the present disclosure provide various techniques (e.g., methods, systems, devices, computer-readable media storing computer-executable instructions for performing computing functions, etc.) for generating and using machine learning models to predict the results of code integration requests. As discussed in more detail below, machine learning models can be generated and trained based on code integration requests previously submitted to and processed by the software architecture authorization system. Based on the machine learning and artificial intelligence-based techniques used, one or more models can be trained that can be developer-specific, project-specific, and organization-specific, meaning that the trained models can output different result predictions, confidence levels, reasons, and recommendations, depending on the current developer, project, and organization. Machine learning models can also be trained based on specific inputs received related to previous code integration requests (e.g., the software library to be integrated, the target source code module, the reason for the code integration request, and / or the functionality to be used in the library, etc.). Then, after the generation and training of one or more machine learning models, such models can be used to predict the results of potential code integration requests (e.g., approval or rejection of authorization). Such models may also be used to autonomously and independently identify causes associated with predictions (eg, security vulnerabilities, licensing incompatibilities, etc.), and / or instead suggest alternative software libraries that may be integrated to provide the desired functionality.

[0288] Now refer to Figure 9A and 9B , shows two examples of user interface screens generated by a software development environment. Figure 9A , a user interface 900 of a software development environment includes a first window containing a block of source code 910a that a software developer user is currently viewing, writing, or updating. During the development process, a user may desire to integrate an open source software library or other external code library. Unlike proprietary software developed within an organization, an external code library can refer to any software developed and / or distributed by a third-party provider. Thus, an external code library can include software functions or modules within an open source software library, as well as other non-open source software provided by other third-party software distributors.

[0289] In this example, a software developer may select the software library import component 920 to browse and select one or more open source libraries to integrate into the current software project 900. For example, the user interface button 920 may invoke a list of available open source libraries from which the developer may select to incorporate into the current software project. In such an example, the user may select a library, view, and import (e.g., copy, link, or reference, etc.) one or more code blocks or functions within the external software library, thereby integrating the external code library into the current software project. Thus, as Figure 9BAs shown in FIG, the updated source code project 900 includes an open source code base 910b integrated into the software project. Figure 9B 9. The example code shown in FIG. 9 is merely illustrative. In other examples, the integration of an external code library (e.g., an open source library) can take many different forms, including directly copying (e.g., cutting and pasting) an annotated portion of the source code from the open source library, or linking to the open source library and calling one or more functions, etc. Furthermore, while user interface component 920 is shown in this example to allow a developer to browse and select from a list of available open source libraries, it should be understood that component 920 need not be included in other embodiments, and that a developer can use any other available technique to integrate code based on external code into software project 900.

[0290] in addition, Figure 9A and 9B The example shown in the depicts a software source code project developed in real time by a developer. As discussed below, when a developer uses a trained model to predict the results of a code integration request during real-time software development, a specific technical advantage can be obtained. Such technical advantages can include real-time scanning and analysis of potential security vulnerabilities, licensing conflicts, and real-time code optimization. Therefore, in such embodiments, some or all of the components of the code integration prediction server 910 described below can be implemented via plug-in components in the software development environment 920. Alternatively, the components of the code integration prediction server 910 can be implemented as independent services, and the development environment 920 can access the independent services in real time in response to user actions during the software development process. However, in other embodiments, the trained model for predicting the results of a code integration request does not need to be used during real-time software development, but can be used to analyze previously written source code. For example, one or more trained models described herein can be used together with a source code crawler tool or other software analysis tool to identify which open source libraries (or other external code libraries) are used in a software project or component. For example, a source code crawler tool can identify one or more comments in the source code as matching specified comments associated with a known open source library. In other examples, compiled and / or linked software components can be analyzed, and / or the execution of components can be analyzed to detect which open source libraries are used by the code.

[0291] Now refer to Figure 10, shows an example of a distributed computing environment 1000, which includes a code integration request prediction server 1010, which can be used to receive and process input data, generate and use machine learning or artificial intelligence models to generate outputs that predict code integration requests. As shown in this example, the code integration request prediction server (or prediction server) 1010 can communicate with various client devices 1050, software development environment 1020, and other various systems via one or more communication networks 1040 to generate and train machine learning models and use the trained models to predict code integration request results. As shown in this example, the prediction server 1010 can be implemented as one or more independent computer servers, systems, or services that are configured to receive requests from the software development environment 1020. In other embodiments, the prediction server 1010 can be implemented as a plug-in component within the software development environment 1020. Furthermore, while this example illustrates that the client device and system 1050 indirectly interact with the prediction server 1010 via the development environment 1020, in other cases, the client device 1050 can interact directly with the prediction server 1010 to generate and train models, execute models to make predictions about code integration requests, and the like. For example, the client device 1050, the development environment component 1020, or other components within the environment (e.g., historical data repository 1030, software architecture authorization system 1060, open source library 1070, and external data source 1080) can interface with one or more application programming interfaces (APIs) 1012 and / or user interface components 1014 supported by the prediction server 1010 to train and generate machine learning models to predict outcomes for specific code integration requests. The prediction server 1010, discussed in more detail below, can include various hardware and / or software systems and subcomponents, including a trained machine learning model 1015, as well as a model training system 1016 and a model execution system 1018. Using these systems, along with a data repository 1030 storing historical code integration request data and / or other external data sources 1080, the prediction server 1010 can train one or more machine learning or artificial intelligence models. Execution of the one or more machine learning or artificial intelligence models can generate output that can be evaluated to predict the outcome of a request for a specific external code repository integrated within a software project or component.

[0292] As shown in the example computing environment 1000, the client system 1050 may include a client computing device of a software developer user, a software analyst / auditor user, etc. As discussed below, the client system 1050 may initially interact with the software architecture authorization system 1060 during multiple code integration requests, and its request and response data may be stored and analyzed to be used as model training data. After generating a trained model, the client device may initiate access to the prediction server 1010 (e.g., directly or indirectly via the development environment 3020) to execute the trained model and predict the results of the code integration request.

[0293] As discussed in more detail below, prediction server 1010 can generate, train, and execute machine learning model 1015, which is configured to provide specific predictions of outcomes for potential code integration requests. The trained model used to predict outcomes can be based on previous / historical code integration requests and include user-specific, library-specific, and other context-specific data, such as the specific developer, project, and reason for the developer wanting to include an external library, as well as the recency of the project / system, other external code libraries used by the same project / component, etc. Model training system 1016 can retrieve data from data repository 1030 and / or client system 1050 to train model 115 to generate predictive outcomes for code integration requests, calculated for the specific user, software project, and code integration request characteristics. Model execution system 1018 can access trained machine learning model 1015, provide and format input data (e.g., code integration request data) to trained model 1015, and determine predicted outcomes based on execution of the model. The output of the trained model 1015 can be provided to a client device 1050 or other output system via the API 1012 and / or the user interface component 1014. In addition, the output of the trained model 1015 can include not only a prediction of the outcome of the code integration request (e.g., approval or rejection), but also various related data, such as a confidence value associated with the prediction, one or more causal factors on which the prediction is based (e.g., security issues, licensing incompatibilities, other code dependencies, etc.), and one or more recommendations for alternative open source libraries (or other external code libraries) determined to have a higher probability of approval.

[0294] In the embodiments discussed herein, the client computing devices and systems 1050 may include mobile devices (e.g., smartphones, computing tablets, personal digital assistants (PDAs), wearable devices, etc.) running various mobile operating systems and enabled with Internet, email, short message service (SMS), Or other communication protocols. The client computing device and system 1050 may also be a general-purpose computer, such as a personal computer and / or laptop computer running various operating systems. The client system 1050 may also be a workstation computer running any of a variety of operating systems. Alternatively or additionally, the client system 1050 may be any other electronic device, such as a thin client computer, an Internet-enabled game console system and / or a personal messaging device or other device capable of communicating via (one or more) networks 1040. Although the computing environment 1000 shown is shown as having three client systems 120, it should be understood that any number of client computing devices may be supported. Other devices (such as sensor devices, etc.) may also interact with the prediction server 1010.

[0295] The communication network(s) 1040 shown in the computing environment 1000 may be any type of network familiar to those skilled in the art that may support data communications using any of a variety of commercially available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), AppleTalk, and the like. By way of example only, the network(s) 1040 may be a local area network (LAN), such as a LAN based on Ethernet, Token Ring, and the like. The network(s) 1040 may also be or include a wide area network, such as the Internet. The network(s) 1040 may include virtual networks, including but not limited to virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (e.g., in accordance with the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suite, and / or any other wireless protocol); and / or any combination of these and / or other networks.

[0296] The code integration request prediction server 1010 may be a server system including one or more general-purpose computers, dedicated server computers (for example, including PC (personal computer) servers, Servers 1010 may be servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. In various embodiments, server 1010 may be adapted to run one or more services or software applications described herein. For example, server 1010 may include a web server and / or an application server configured to perform processing and execute the software components described herein to implement various embodiments and aspects of the present disclosure.

[0297] The prediction server 1010 can execute an operating system including any available server operating system. The prediction server 1010 can also run any of a variety of additional server applications and / or middle-tier applications, including HTTP (Hypertext Transfer Protocol) servers, FTP (File Transfer Protocol) servers, CGI (Common Gateway Interface) servers, Servers, database servers, etc. Exemplary database servers include, but are not limited to, those commercially available from Oracle, Microsoft, Sybase, IBM (International Business Machines), etc.

[0298] The data repository 1030 may include a database or data storage structure that stores various data related to previous (or historical) code integration requests. Such historical data may include data detailing specific characteristics of each code integration request (e.g., developer identity and characteristics, specific client device and characteristics, network(s) on which the request was made, characteristics of the specific software component / project, open source libraries (or other external code libraries) requested to be included, other open source libraries already used by the software component / project, functionality of the software project, planned deployment environment and security threats expected to be faced by the software project (e.g., untrusted networks or computers), reasons provided by the developer for requesting the software library integration (if any), and any other relevant data about the code integration request. In addition, the historical data may also include the corresponding results of each previous code integration request. The corresponding result data may indicate whether each code integration request was approved by the software architecture authorization system 1060, and the reasons why the software architecture authorization system 1060 approved or rejected the request. The data repository 1030 may reside in a variety of locations. For example, one or more of the data repositories 1030 may reside on a non-transitory storage medium local to (and / or residing in) the server 1010. Alternatively, the data repository 1030 may be stored externally separately from the prediction server 1010 and may communicate with the server 1010 via a network-based or dedicated connection. In some embodiments, the data repository 1030 may reside in a storage area network (SAN). Similarly, any necessary data or files for performing the functions of the model server 1010 may be appropriately stored locally on the prediction server 1010 and / or stored remotely. In some embodiments, the data repository 1030 may include a relational database suitable for storing, updating, and retrieving data in response to formatted queries and commands.

[0299] A machine learning model is trained to generate predictions for code integration requests.

[0300] Now refer to Figure 11, a flowchart illustrating the process of training a machine learning model to generate a prediction result of a code integration request, i.e., a prediction of whether a potential request to integrate an external code base into a software component / project is approved by the software architecture authorization system. As described below, the process of receiving and storing training data and generating and training the model structure can be performed by various devices operating within the computing environment 1000 (including the model training system 1016 of the prediction server 1010). However, it should be understood that reference Figure 10 The described techniques need not be tied to any particular device or server within computing environment 1000, but rather may be implemented by any computing system and device described or supported herein.

[0301] At 1102, the model training system 1016 and / or other systems within the prediction server 1010 may retrieve code integration request data, for example, from a historical code integration request data repository. For example, the code integration request data may include one or more data sets corresponding to requests previously made by a developer to integrate an external code library into a software project / component. For example, the code integration request data retrieved at 1102 may include specific characteristics of each of a plurality of code integration requests and a corresponding response. As described above, such request characteristics may include, for example, the identity and characteristics of the developer user who initiated the request, the specific client device (and its hardware, software, and / or network characteristics) that initiated the request, the communication network(s) over which the request was made, the open source library (or other external code library) requested by the developer, and one or more characteristics of the specific software component or project into which the developer requested the open source library to be integrated. Characteristics of a software component or project may include the functionality of the software component / project, its stage in development, security requirements associated with the software, other open source libraries used by the software, the recency of the software's maintenance (e.g., an indication of whether the project is still maintained and whether dependencies are available or missing), security patches installed on the software, the purpose and organizational priority of the software, and the computing and network environment into which the software will be deployed (e.g., accessible to untrusted users, computers, or networks). Additionally, characteristics of a code integration request may include any description or reason provided by the developer at the time of the request to explain or justify the request to include the particular requested library. Such a description may also include an indication of whether the code integration request is related to a previous code integration request that was requested and then denied by the software architecture authorization system 960.

[0302] At 1104, the model training system 916 and / or other systems within the prediction server 1010 can retrieve details and characteristics of the corresponding results of each previous code integration request. This previous / historical result data corresponds to the responses made by the software architecture authorization system 1060 to the code integration requests received from the developers, which may be necessary to train the machine learning model to predict the results. In some embodiments, the previous / historical result data for the code integration request may only include a binary value indicating whether the previous code integration request was allowed (e.g., approved or rejected). However, in other embodiments, additional relevant data may be included in the previous / historical result data retrieved at 1104, such as one or more reasons or causal factors provided by the software architecture authorization system 1060 when approving or rejecting the previous request. Such reasons or causal factors may indicate that the code integration request was rejected, for example, due to a security vulnerability associated with the requested open source library, or because the organization does not comply with the license terms of the open source library, or because the license terms of the open source library are incompatible with the terms of other licenses used by the software project and / or other open source libraries, etc.

[0303] At 1106, the model training system 1016 (or other component with the prediction server 1010) can generate one or more model data structures, and at 1106, a machine learning algorithm based on a training data set (including any, some, or all of the code integration request / result data received in steps 1102-1104) can be used to train the model. In various embodiments, a variety of different types of training models can be used, including classification systems that perform supervised or semi-supervised learning techniques, such as naive Bayes models, decision tree models, logistic regression models, or deep learning models, or any other prediction system based on machine learning or artificial intelligence that can perform supervised or unsupervised learning techniques. For each machine learning model or model type, an appropriate data structure can be generated in step 1106, and in step 1108, the model can be trained using a corresponding machine learning algorithm based on the training data received in steps 1102 and 1104.

[0304] At 1108, a model can be trained based at least in part on (1) the code integration request data and (2) the corresponding result data.

[0305] At 1110, the model trained using the machine learning or artificial intelligence algorithm in step 1108 may be revised (or adjusted) based on additional relevant data (optionally) received from one or more external data sources 1080. For example, external data sources 1080 may include updated open source software license terms, one or more software license compatibility matrices, updated security issue data (e.g., known security vulnerabilities, available security patches, etc.), software or computing infrastructure cost data, etc. For example, the machine learning model 1015 trained in step 1108 may be supplemented with a specific set of rules that may either override or weight the predictive results typically generated by the machine learning model 1015. For example, based on the applied machine learning algorithm and historical training data 1030, the predicted result from the trained model for a particular code integration request may typically be to approve the integration of the requested open source library. However, if a security issue in the requested open source library has been newly discovered and has not yet been resolved, that information may be retrieved from the external database of security vulnerabilities 1080 and used to temporarily override the trained model 1015 only in the context of requesting that particular open source library. As another example, if an open source library or other external code base has been recently updated, the machine learning model 1015 can be adjusted to include a temporary weighted preference against using the recently updated library based on concerns about stability and backward compatibility. In yet another example, if the license terms associated with a requested open source library have recently changed, the machine learning model 1015 can be adjusted to predict a rejection of a code integration request for that library until the organization can review the updated license terms.

[0306] Finally, in step 1112, the trained machine learning model used to predict the outcome of the code integration request can be stored within the server 1010 or elsewhere within the system. As described above, in some embodiments, the trained model can be stored within and executed from a plug-in component of the software development environment 1020. Additionally or alternatively, the trained model can be stored and / or exported to other systems for use within the source code crawling and analysis process of previously written code blocks.

[0307] Now refer to Figure 12 , shows a flowchart illustrating the process of executing a machine learning model to generate a predictive result for a code integration request. The features and steps described below, including receiving input data, accessing and using a trained model, generating output predictive results, determining relevant cause data and / or recommendations, etc., can be performed by various devices operating within the computing environment 1000 (including the model execution system 1018 of the code integration request prediction server 910). However, it should be understood that reference Figure 12The described techniques need not be tied to any particular device or server within computing environment 1000, but rather may be implemented by any computing system and device described or supported herein.

[0308] At 1201, the prediction server 1010 may receive data corresponding to a code integration request. Thus, the data received in step 1201 may initially be input by the client device 1050 into the development environment 1020, or directly into the API 1012 or user interface 1014 of the prediction server 1010. The input data received in step 1201 may at least identify the open source library (or other external code library) to be integrated, and the source code component / project into which the requested open source library may be integrated. As described above, the prediction server 1010 may include one or more application programming interfaces (APIs) 1012 configured to support communication with various client systems, including authentication and security components for authenticating and authorizing client devices and / or users, so that the appropriate functionality of the prediction server 1010 is only provided to authorized users. Furthermore, in some embodiments, the prediction server 1010 may provide one or more user interfaces 1014 to allow a user to directly request the prediction results of the code integration request in step 1201. Figure 13A and 13B An example screen of a user interface configured to enable a user to request a predictive result of a code integration request is shown in FIG. Figure 13A As shown in , a user has initiated a request (eg, either via user interface 1014 or via software development environment 1020) to integrate an open source library "LIBRARY 1" into a current block of software code.

[0309] In step 1202, prediction server 1010 may retrieve data corresponding to one or more other related external code libraries associated with the source code block. For example, if the request in step 1201 is a developer attempting to integrate the open source library "LIBRARY 1" into the source code block currently being developed, the related external code libraries retrieved in step 1202 may correspond to additional open source libraries already used / referenced by the current source code block. As discussed above, the presence of other libraries / external code libraries in the current source code block may potentially create security vulnerabilities (based on a single external library or a combination of external libraries) or may cause licensing compatibility issues related to whether the code integration request is approved. Therefore, in some embodiments, in step 1202, prediction server 1010 may scan all current source code blocks to detect any references or calls to external libraries, and analyze the code to detect embedded code comments indicating that the code block was copied from an external library. Additionally or alternatively, development environment 1020 and / or prediction server 1010 may maintain a list of all open source libraries (or other external code blocks) that have been integrated into each different source code component / project.

[0310] In step 1203, the prediction server 1010 may receive or retrieve one or more characteristics of the source code block into which the external code is to be integrated. The characteristics retrieved in step 1203 may include any relevant data associated with the source code block that can potentially be used by the trained model to determine whether to approve the code integration request. For example, such characteristics may include the identity and characteristics of the developer initiating the request, the specific client device 1050 from which the request was initiated (and its hardware, software, and / or network characteristics), and the communication network(s) over which the request was made. Additional characteristics of the software component or project may include a description or functionality of the software, its current stage in development, any security requirements associated with the software, the recency of software maintenance (e.g., indicating whether the project is still under maintenance and dependencies are available or missing), security patches installed on the software, the purpose and organizational priority of the software, and the computing and networking environment in which the software will be deployed (e.g., accessible to untrusted users, computers, or networks). Still other examples of software characteristics that may be received in step 1203 include any description or reason provided by the developer at the time of the request to explain or justify the request to include the specific requested library. Thus, in some embodiments, the prediction server 1010 may retrieve source code block characteristics by generating and outputting a user interface to allow a developer to provide information (e.g., component product, functionality and purpose, and / or reason for requesting an external library).

[0311] In some embodiments, the characteristics retrieved in step 1203 may also include an indication of whether the code integration request is related to a previous code integration request that was requested and then rejected by the software architecture authorization system 1060. Figure 12 A. In this example, user interface 1200 includes a dialog box 1210a that asks a developer whether a current code integration request (i.e., request "LIBRARY 1") is related to a previously rejected code integration request (i.e., request "LIBRARY 2"). As discussed above, machine learning model 1015 can be trained based in part on the relationship between different external libraries, including learning such relationships based on a sequence of code integration requests in which a developer requests one external code library (e.g., LIBRARY 2), the request is rejected, and then the developer requests a different external code library (e.g., LIBRARY 1) that can perform similar functionality.

[0312] At 1204, the prediction server 1010 may access one or more of the trained models 1015 and provide the data received in steps 1201-1203 as input to the trained models. Figure 12 As described, the trained machine learning or artificial intelligence model 1015 may have been trained to predict the outcome of a code integration request (e.g., approval or rejection of the code integration) based on various factors, including characteristics of the requested library, source code block, and / or other relevant factors. Thus, the machine learning or artificial intelligence model 1015 may have been trained using a training data set that includes previously determined code integration requests made by developers and the results provided by the authorization system 1060. Additionally, the trained machine learning model 1015 may be trained to learn the outcome determination based on the reasons / justifications provided by the developer for the request and the corresponding reasons / causal factors with the approval or rejection outcome provided by the authorization system 1060.

[0313] In step 1205, one or more outputs may be generated by the trained machine learning model 1015 in response to the input data provided to the model in step 1204. As described above, the model outputs may correspond to predicted outcomes that the authorization system 1060 may provide in response to a particular code integration request. In some embodiments, the trained model 1015 may provide not only a binary predictive outcome (e.g., approval or rejection of the request), but also additional relevant information (such as reasons / causal factors for the predictive outcome and / or alternative library recommendations). For example, the model-based determination of whether to approve or reject a code integration request may be based on issues such as licensing compliance or compatibility, detected security vulnerabilities, software or computing infrastructure costs, additional software dependencies, the recency and priority of the software project, the availability of security patches, the existence of more secure alternative libraries, etc.

[0314] In step 1206, the prediction server 1010 evaluates the predictive results received from the trained model as potential code integration requests. If the trained model determines that the code integration request may be approved (1206: yes), then in step 1207, an indication of approval can be output to the developer. In some embodiments, the prediction server 1010 is implemented as a plug-in component within the development environment 1020, and in such cases, the plug-in can provide a user interface to notify the user of the predicted approval. The user interface can also be configured to allow the user to formally submit a code integration request to the software architecture authorization system 1060, which request has now received a predicted approval. In addition, in some embodiments, the development environment 1020 can be configured to temporarily allow the developer to use / include the requested external code based on the predicted approval of the trained model.

[0315] Conversely, if the trained model determines that the code integration request is likely not to be approved (1206: No), then in step 1208, the prediction server 1010 may determine one or more additional relevant data that may be provided to the user along with the predicted rejection result. In some embodiments, in the event that the code integration request is predicted to be rejected, the trained model 1015 may provide a list of possible reasons / causal factors for the predicted rejection. For example, referring briefly to Figure 12 B. In this example, the model execution system 1018 has indicated that the code indicates that the predicted outcome of the request is that the request will be rejected. Thus, in this example, a dialog box 1210b has been provided to allow the developer several additional options, including the option to proceed with the request to the software architecture authorization system 1060 regardless of the predicted outcome, or to review / analyze the details and causal factors of the predicted rejection, or to receive other suggestions for open source libraries (or other external code libraries) that may be acceptable alternatives to the requested library and that may be more likely to be approved for integration into the software component / project.

[0316] For example, in some embodiments, Figure 1 The model execution system 118 shown in can be configured to call the model multiple times, where specific pieces of input data can be modified and the corresponding prediction results can be analyzed to determine which input data may be key (e.g., outcome-determining) factors. As a brief example illustrating this functionality, if the developer provides a reason for requesting an open source library in step 1202, and the request is rejected in step 1205, the model execution system 118 may automatically re-call the trained model using a different reason (e.g., retrieved from historical code integration request data 1030), with all other input data remaining unchanged. If the re-called trained model predicts approval, the reason provided by the developer for requesting the open source library can be identified as one of the causal factors for predicting rejection in step 1208. A similar process can be performed by re-calling the trained model with minor modifications to other individual input data received in steps 1201-1203 while keeping the other input data unchanged.

[0317] In addition, in some embodiments, the trained model 1015 can be used to determine one or more alternative open source libraries (or other external code libraries) that can perform similar functions to the requested code that received a predicted rejection. For example, if a developer initiates a code integration request for a first open source library and the request is rejected in step 1205, the model execution system 1018 can automatically re-invoke the trained model 1015 one or more times to request different but functionally similar open source libraries (e.g., based on software provider, library type, library / function description, user feedback, etc.). For any different but functionally similar open source libraries for which the trained model 1015 predicts an approval outcome, those open source libraries can be identified as potential alternatives that can be provided to the developer as suggestions.

[0318] Additionally, as discussed above, the trained model 1015 can be trained to learn the relationships between different open source libraries (or other external code libraries) based on a sequence of code integration requests. For example, if a developer requests a first external code library (e.g., LIBRARY 2), the request is rejected, and the developer then requests a different external code library (e.g., LIBRARY 1) and indicates that the request is related to the first request (see 1210a), then the trained model 1015 or other components within the prediction service can learn that LIBRARY 1 and LIBRARY 2 may be acceptable alternatives for performing at least some of their respective functionalities.

[0319] Finally, in step 1209, an indication of the predicted rejection may be output to the developer. In embodiments where the prediction server 1010 may be implemented as a plug-in within the development environment 1020, the plug-in may provide a user interface to notify the user of the predicted rejection, such as Figure 12 B. The determined reasons and / or causal factors, as well as one or more suggestions for alternative open source libraries determined in step 1208, may also be provided to the developer via the development environment 1020 and / or via a separate user interface 1014.

[0320] Many variations and modifications of the disclosed embodiments may also be used. For example, an application software may be used instead of an activity guide to provide an activity guide writer. Some embodiments may dynamically suggest categories based on the administrator's position or title.

[0321] VI. Automatic Self-Tuning Feature Discovery and Integration

[0322] Some organizations store data from multiple clients, vendors, and / or domains in customizable schemas. When developing machine learning solutions that work across these different data schemas, the coordination step is often done either manually or through a tedious extract, transform, and load (ETL) process. For a given machine learning problem (e.g., “I want to predict sales” or “Who is the most productive employee?”), this service will crawl the entire data repository across clients / vendors / domains and automatically detect equivalent entities (e.g., adding a column for “location” or “address” to the data structure). The service will also automatically select features that are predictive for each individual use case (i.e., one client / vendor / domain), effectively making the machine learning solution agnostic to the organization’s application developers.

[0323] Feature discovery isn't limited to analyzing feature names, but also includes feature content. For example, this feature could detect a specific distribution of dates or data that fits a previously known characteristic distribution. Combining more than one of these factors can allow the system to match and discover even more features.

[0324] In this embodiment, a weighted list of common representations of each feature used for this particular machine learning problem is stored, with respect to its impact on the model's predictions. Features may be initially gathered to catch up with past data stores. As new data from new clients / vendors / domains is added to the data repository, the matching service automatically detects which features should be fed to the machine learning solution based on the previously calculated weighted list. This pre-processing of features allows for faster execution of machine learning problems. Based on the features found for the new clients / vendors / domains, the weighted list is updated to improve the machine learning model. This weighted list is periodically updated based on new data intake and used to improve feature selection for existing clients.

[0325] Figure 14 Illustrated is a process 1400 for automatically self-adjusting company-wide feature discovery and integration techniques. Alternative embodiments may be combined, separated, or otherwise modified. Figure 14 To change the functionality described in the blocks shown in the figure. Figure 14 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0326] At 1402, functionality includes receiving instructions to design a new machine learning application. In various embodiments, the instructions may be received via a user interface. In various embodiments, the instructions may be received via a chatbot. The technology may employ natural language processing to determine machine learning models and metrics that may be used to design the new machine learning application.

[0327] At 1404, functionality includes electronic "crawl" across data storage domains. The system may receive access to one or more data storage locations. The data storage locations may be local or remote. In various embodiments, the data storage devices may be in one or more cloud storage facilities. The client may provide the machine learning system with access to the data storage devices, including the data's labels and data architecture.

[0328] At 1406, functionality includes extracting one or more features from a data store. The data store may include one or more tags that characterize the data. The technique may automatically detect equivalent entities for the one or more tags. For example, a feature labeled "location" may also identify a data tag for "address." In various embodiments, the technique may review the one or more tags that characterize the data to determine one or more features from the data. The technique may extract the one or more features, storing the features and associated data locations (e.g., data addresses) in a memory. The techniques may also identify and select features that are predictive for each individual use case (i.e., one client), effectively making the machine learning solution agnostic to the application developer. In some embodiments, features may be extracted via metadata contained in each category of the stored data.

[0329] At 1408, the functionality includes developing a weighted list of common representations for each feature. The technology can determine a ranking for each of the one or more features extracted from the data store based at least in part on an impact of the one or more features on a solution applied using machine learning.

[0330] For example, determining the ranking may include determining a machine learning algorithm from a plurality of algorithms stored in a library, wherein the algorithm combines one or more features to calculate a result. The machine learning algorithm may be modified by removing a first feature of the one or more features. The technique may include calculating a first result of the modified machine learning algorithm. The first result may be compared to ground truth data. The technique may include calculating a ranking of the first feature based at least in part on comparing the first result to the ground truth data. The importance of the first feature may be ranked higher than one or more other results because the difference between the first result and the ground truth data is reduced. The technique may include assigning weights to one or more features in the second list based at least in part on the ranking of the features for the solution of the machine learning application.

[0331] At 1410, functionality includes automatically detecting features from a new data store based on a weighted list. In various embodiments, the technique may identify metadata used to identify features in the data. The technique may use the weighted list to determine which features to incorporate into a machine learning solution. Features that rank higher and produce results closer to the ground truth data may better predict the desired machine learning solution. Therefore, incorporating these features into a machine learning application may be advantageous.

[0332] At 1412 , functionality includes feeding the features to a machine learning solution. Figure 1 The monitoring engine 156 shown in can provide feedback to the model merging engine regarding features to be incorporated into the machine learning solution.

[0333] At 1414, functionality includes updating the weighted list based on new data. When new data is added to the data store, the matching service can automatically detect which features should be input into the machine learning solution based at least in part on the previously calculated weighted list. Based on the features found for the new data, the weighted list can be updated. This list can be periodically updated based on new data and used to improve feature selection for existing models.

[0334] It should be recognized that Figure 14 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present invention may perform the above steps in a different order. Moreover, Figure 14 The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0335] Figure 15Illustrated is a process 1500 for a technique for an intelligent assistant to generate a machine learning solution. Alternative embodiments may be combined, separated, or otherwise altered Figure 15 To change the functionality described in the blocks shown in the figure. Figure 15 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0336] At 1502, functionality may include receiving a first input identifying one or more locations of a data set. In various embodiments, a user may use the interface to identify one or more locations of data to be used to generate a machine learning model. As described above, the data may be stored locally or remotely. In various embodiments, the user may, for example, specify a network location of the data (e.g., an Internet Protocol (IP) address). In various embodiments, the user may select a folder from a plurality of folders on a storage device (e.g., a cloud storage device). In various embodiments, the user may select a file on a portable media device (e.g., a USB thumb drive, an optical disc, or a solid-state or magnetic disk drive). In various embodiments, the technology may suggest a location for the data based at least on the user's previous interactions with the machine learning interface.

[0337] At 1504, functionality may include receiving a second input. The second input may identify a problem to generate a solution using a machine learning application. In various embodiments, the second user input may specify the type of problem for which the user wants to implement machine learning. In various embodiments, the problem may be identified by entering text via a user interface. In various embodiments, the problem may be entered as native language speech or text (e.g., by using a chatbot). The technology may decipher the native language to understand the objectives of the machine learning model. Some types of problems that machine learning may solve may include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. These technologies may identify one or more keywords in the native language to recommend or select a specific machine learning algorithm.

[0338] In various embodiments, a user may select a problem type via a graphical user interface. The technology may display several generic models available to the user via the graphical user interface. The user may select a model by selecting one of the model descriptions using a pointing device or a touchscreen display. In addition to the generic models, additional custom models may also be selected and modified to solve the problem received as a second input.

[0339] At 1506, functionality may include analyzing the data to extract one or more tags for patterns in the dataset. The one or more tags may describe the type of data contained in that portion of the dataset. For example, a data tag such as "address" may include information about the address entry. The one or more tags may be extracted and stored in memory along with the location of the corresponding data. The tags may be part of the stored data. For example, a customer may have provided tags for the dataset. The technology may also include generating tags for features found in the dataset. These tags may be generated by analyzing metadata for the stored dataset and determining a category that best describes that portion of the dataset. In this manner, the technology is not limited to naming conventions used by customers to describe data. For example, the technology may determine feature content (e.g., date) and determine a particular distribution of data that matches a previously known or discovered feature with a very typical distribution.

[0340] At 1508, functionality may include generating a first list of common categories for each of the one or more tags of the pattern. For each extracted term, the techniques may reference a dictionary to obtain equivalents for the tag. For example, for "address," equivalent terms may include residence, home, house, dwelling, location, domicile, residence, mailing address, etc.

[0341] At 1510, functionality may include accessing a library of terms stored in memory, where the terms correspond to categories known to the machine learning model. The machine learning model may include a list of one or more terms (and their equivalents) currently used by the existing model. For example, the machine learning model may refer to the term "location" instead of the term "address."

[0342] At 1512, functionality may include associating one or more tags with a category, at least in part, by identifying a category for each of the one or more tags. The shared category list may be compared to a library of terms stored in memory. The tags may be part of the stored data. For example, a customer may have provided the tags. The technique may include generating tags for features found in a dataset. Data tags may also be generated by analyzing the content of the dataset and / or metadata of the dataset. In this manner, the association is not limited to customer-provided tags.

[0343] At 1514, the functionality may include generating a mapping of one or more labels to categories of the machine learning model. The mapping may identify a location in the dataset for each category of the machine learning model.

[0344] At 1516, the functionality may include storing the mapping in memory. The mapping may be updated as data in the storage locations changes (additions / deletions) or as tags change.

[0345] In various embodiments, the functionality may include analyzing the data to extract one or more features described by the dataset. The method may include generating a second list, wherein the second list identifies one or more features of the dataset. The technique may determine a ranking for each of the one or more features in the second list based at least in part on the impact of the one or more features on a solution applied using machine learning. The technique may assign weights to the one or more features in the second list based at least in part on the ranking of the features on the solution applied using machine learning.

[0346] In various embodiments, determining the ranking may include determining a machine learning algorithm from a plurality of algorithms stored in a library, wherein the algorithm combines one or more features to calculate a result, modifying the machine learning algorithm by removing a first feature from the one or more features, calculating a first result of the modified machine learning algorithm, comparing the first result of the modified machine learning algorithm to ground truth data, and calculating a ranking for the first feature based at least in part on comparing the first result to the ground truth data, wherein the first feature is ranked higher in importance compared to one or more other results due to a reduced difference between the first result and the ground truth data.

[0347] In various embodiments, the method may include identifying a new location for the additional data. For example, the method may include analyzing the additional data to identify one or more new features. The new feature is not identified on a first list of one or more features in memory. The method may include generating a revised list of one or more features in memory, including the one or more new features. The method may include determining a revised ranking of each of the one or more features and the one or more new features in the revised list based at least in part on an impact of the one or more new features on a solution using the machine learning application. The method may include assigning a weight to each ranked feature in the revised list based at least in part on the revised ranking of the new features for the solution generated by the machine learning application.

[0348] In various embodiments, the method may include presenting a mapping of one or more labels to categories of a machine learning model. The technique may include receiving a second input. The second input may associate labels of one or more labels for a pattern of data with categories of one or more categories known to the machine learning model.

[0349] In various embodiments, the method may further include extracting a dataset stored in one or more locations. The method may include storing the extracted dataset in a memory. The method may include renaming one or more tags of the dataset to match the mapping of the one or more tags.

[0350] In various embodiments, the method may include identifying new labels for the one or more labels, wherein the new labels are unrelated to the categories of the machine learning data; and adding the new labels and associated metadata to a library of terms stored in memory.

[0351] It should be recognized that Figure 15 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present disclosure may perform the above steps in a different order. Moreover, Figure 15 The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0352] VII. Efficient and Accurate Assignment of Confidence to Machine Learning Predictions for Enterprise Software Systems

[0353] Various embodiments can increase confidence in machine learning model predictions or at least quantify the level of confidence guaranteed by assigning confidence levels to machine learning predictions. These point predictions can carry uncertainty. This uncertainty is typically not considered or properly calibrated before a software system uses a point prediction from a machine learning model. In various embodiments, a technique can provide users of a software system that leverages machine learning with an assessment of the quality of a point prediction from a machine learning model. The quality can be exposed using multiple QoS or KPI factors, either as a temporary result or as overall performance.

[0354] Point prediction techniques can be implemented by searching a set of training data as a query instance to obtain suitable candidate machine learning models or pipeline 136 instances to calculate confidence estimates for candidate machine learning models for point prediction. This technique can generate uncertainty estimates for a particular machine learning system for point prediction. The search system can use a method to efficiently calculate the estimate while taking into account the machine learning software and underlying system requirements. For example, for data with a poor fit for the point prediction, the confidence measure will be low.

[0355] The confidence score can reflect the likelihood that this type of machine learning model will perform beyond the confidence level that a particular outcome is correct. For example, a machine learning model might predict the most likely winner of a sporting event with 90% accuracy, but might have a 60% confidence level for tomorrow's game because one team has too many new players. The color of the point prediction can be displayed on the screen based on the confidence level in the estimate.

[0356] VIII. Adaptive and Context-Aware Automation Service Composition for Machine Learning

[0357] Ontology is a specification of conceptualization or formal naming, as well as the definition of the types, characteristics and relationships of entities that really or substantially exist in a particular domain of the present disclosure. Except that the data model is specifically related to data, the ontology model provides much of the same information. The data model provides entities that will become tables in a relational database management system (RDBMS), and attributes will become columns with specific data types and constraints, and relationships will be identifying and non-identifying foreign key constraints. What the data model does not provide is a machine-interpretable definition of vocabulary in a particular domain. The data model will not contain a vocabulary that defines the entire domain, but rather a data dictionary will contain information about entities and attributes associated with specific data elements. The ontology consists of classes arranged in a hierarchical manner according to the classification of subclasses and superclasses, slots with descriptions that define value constraints, and the values of these slots. When the ontology is complete, and when each instance of these elements (classes and slots) is defined and any restrictions are added or refined, the knowledge base is defined.

[0358] For example, the class person includes all people, and a specific person is an instance of this class. Most ontologies focus on classes of things, and these classes can be broken down into subclasses of things. For example, person can be broken down into Gender, Ethnicity, Race, or Country of Citizenship. Defining a domain based on a machine-readable ontology allows for the free exchange of information and knowledge. Ontologies enable information sharing between different systems within the same domain. There are many freely available ontologies from various industries. Ontologies are a way to standardize vocabulary across domains. For example, the term "practitioner" may have the same meaning in one healthcare system as in another. Similarly, the profession "practitioner" will have the same attributes and the same meaning on both sites. This standardization allows for greater flexibility and will enable faster application development and information sharing.

[0359] Ontologies can be applied to datasets (e.g., images, medical data) to solve a problem (e.g., product recommendations or outlier detection) or to determine the metrics to apply to the problem. Ontologies can provide a hierarchical organization of the dataset to provide a level of scalability. The output of this process is a product graph, which is a composition of models, pipelines, features, and metrics used to generate a machine learning solution. For example, for certain medical applications (e.g., radiology), it may be important to use higher resolution images to detect anomalies in images. The KPI may focus on the accuracy of detecting these anomalies. However, the latency of the process needs to be considered. If the image processing time is too long to be useful, then the machine learning process may not be effective. Therefore, the process may recommend using images with a slightly lower resolution to process the image efficiently. The product graph will help data scientists shape the model while taking these types of trade-offs between latency and accuracy into account.

[0360] refer to Figure 16 , embodiments of the block diagram use ontologies to generate machine learning product graphs for use in designing machine learning models or applications. In various embodiments, context-aware semantic machine learning (ML) services can enable users to perform high-precision searches for services and automatic composition of machine learning deployment services based on formal ontology-based representations of service semantics that can include QoS and product KPIs. This forms a scalable means for high-precision searches for related services to compose pipelines 136, such as Figure 1 As shown in [1], an adaptive machine learning architecture is defined with minimal human intervention. An appropriate combination of non-logic-based and logic-based semantic service selection significantly outperforms both options in terms of accuracy. QoS and KPI constraints are also used as part of the architecture selection (for example, given that latency cannot exceed X, a service infrastructure is required to process large amounts of input data within a specified timeframe in the pipeline that composes the machine learning model).

[0361] For example, medical applications can be used to detect diabetes caused by a condition called diabetic retinopathy through eye exams. Many people with early-stage diabetic retinopathy experience no symptoms. As diabetic retinopathy progresses, symptoms include blurred vision, blindness, floaters or shadows, and loss of vision. There are two types of diabetic retinopathy. In the early stages, non-proliferative diabetic retinopathy occurs, in which blood vessels in the retina begin to leak. If this leakage leads to a buildup of retinal fluid, blurred vision may occur.

[0362] A comprehensive eye exam is the only way to detect diabetic retinopathy in its early stages, before symptoms appear. An image classifier can be used to detect diabetic retinopathy by performing a higher resolution analysis of images of the eye. An ontology with various categories of eye images can be developed, such as no retinopathy, mild retinopathy, and various degrees of retinopathy. The reasoner 1616 can apply the ontology to eye images and understand that other neural networks have been applied to accurate image classification. An entry layer can be constructed to adapt the image classifier to the eye image ontology to detect different degrees of retinopathy in images of the patient's eyes. The product graph will be to read the image, transform the image, rescale the image to a specific resolution, load the image into a pre-built neural network, analyze the image to output a classification of the image (e.g., a numerical value) that can indicate the degree of diabetic retinopathy.

[0363] A. Ontology Modeling and Construction Mechanism 1608

[0364] The semantic profile for the machine learning platform 100 may include: the functional semantics of each microservice routine 140 (feature selection, outlier detection, interpretability) in terms of semantic annotations of its input and output parameters; logic-based descriptions of preconditions (and effects that should hold before or after the execution of the service); non-functional service parameters related to service provenance (e.g., feature engineering also consists of feature selection services - the provenance of the services that make up the service, as it can affect the runtime performance of the machine learning application 112); quality of service (QoS) requirements, covering latency, response time, computing resources, throughput, reliability, availability, success rate (mean time between failures of the service), usability, privacy policy, etc.; and product key performance indicators (KPIs), such as "increase purchases by 40% in the first month" that can be used for product recommendations.

[0365] B. Reasoner Engine 1604 Processing Ontology 1616

[0366] The ontology 1616 created (a description of how services interact between data and control flows and QoS benchmarks) is used to annotate services with concepts defined in the ontology based on formal logic so that from a machine learning perspective, the intelligent agent and reasoner engine 1604 can determine the formal service semantics and compose them based on the optimal runtime expectations of the product. In various embodiments, the ontology requires a "cold start" because the ontology has not yet been created. The ontology builder 1608 can be used to create an ontology that creates new branches in an existing ontology. The reasoner engine 1604 guides the search for the best combination of components (e.g., models, metrics) to solve the problem. The reasoner engine 1604 determines the software functions and hardware to be combined into the product map 1620.

[0367] C. Search-based Architecture Adaptation Engine 1612

[0368] The architecture adaptation engine 1612 uses meta-learning to connect the various blocks used to generate the product graph 1620. The interpolation process selects the best software features and hardware for a specific machine learning problem query. This technique can transform the product graph comparability problem into a computational search problem, which involves defining a search space (i.e., available services), or a set of possible solutions based on established ontological descriptors of services that have previously become successful. The result is the best possible subset of requirements that matches the user request, given the requirement constraints (such as limited resources and interdependencies between requirements). Before deploying the production graph, the user can be provided with a proposed product graph 1620 with a set of trade-offs between metrics of model performance and QoS and other user requirements and constraints (requirement risk). The search space is typically very large and it is impractical to explore it exhaustively. This embodiment uses meta-heuristics (search techniques can be used to find near-optimal or "good enough" solutions) based on the user's selection.

[0369] For example, a user may generate a semantic query with QoS requirements and a specific question being answered. The best product graph 1620 may be suggested for the desired answer taking into account the QoS requirements. There are various library components 168 that define software functionality and hardware that may meet that request, such as Figure 1 The ontology 1616 stores parameters for software functions and hardware so that selections can be made automatically for the product map 1620. The product map 1620 can have a library component 168, such as Figure 1 As shown in , and the hardware components for building machine learning products to comply with semantic queries. The architecture adaptation engine 1612 can determine the best options for the product graph 1620.

[0370] Product graph 1620 is a visual representation of data representing a mathematical structure used to study pairwise relationships between objects and entities. In computing, a graph database (GDB) is a database that uses a graph structure for semantic queries, with nodes, edges, and properties to represent and store data. The key concept in the system is the graph (or edge, or relationship). A graph associates data items in a repository with a collection of nodes and edges, with edges representing the relationships between nodes. These relationships allow data in the repository to be directly linked together and, in many cases, retrieved with a single operation. Graph databases prioritize preserving relationships between data. Querying relationships within a graph database is fast because they are permanently stored in the database itself. Relationships can be intuitively visualized using graph databases, making them useful for highly interconnected data. Graph databases are a subset of NoSQL databases created to address the limitations of existing relational databases. While graph models explicitly list the dependencies between data nodes, relational models and other NoSQL database models link data through implicit connections. Graph databases are designed to allow for simple and fast retrieval of complex hierarchical structures that are difficult to model in relational systems.

[0371] Adaptive and context-aware automated service composition leads to a holistic approach of combining different KPIs, QoS, types of issues, types of data to build the product graph 1620. This increases the awareness of the model.

[0372] Figure 17 Illustrated is a process 1700 for generating a technique for adaptive and context-aware automated service composition for machine learning. Alternative embodiments may be combined, separated, or otherwise altered Figure 17 To change the functionality described in the blocks shown in the figure. Figure 17 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0373] At 1702, functionality can include receiving a first user input identifying a location of data. In various embodiments, a user can use an interface to identify the location of data to be used to generate a machine learning model. In various embodiments, a user can enter a network location of the data (e.g., an Internet Protocol (IP) address). In various embodiments, a user can select a folder from a plurality of folders on a storage device (e.g., a cloud storage device). In various embodiments, a user can select a file on a portable medium (e.g., a USB thumb drive, an optical disc, or a solid-state or magnetic disk drive).

[0374] At 1704, the functionality may include receiving a second user input identifying a problem for which the machine learning application can generate a solution. In various embodiments, the second user input may specify the type of problem for which the user wants to implement machine learning. In various embodiments, the problem may be identified by entering text via a user interface. In various embodiments, the problem may be entered as native language speech or text (e.g., by using a chatbot). The technology can decipher the native language to understand the objectives of the machine learning model. Some types of problems that machine learning can solve include classification, regression, product recommendations, medical diagnosis, financial analysis, predictive maintenance, image and sound recognition, text recognition, and tabular data analysis. These technologies can identify one or more keywords in the native language to recommend or select a specific machine learning algorithm.

[0375] For example, for the diabetic retinopathy example, the user can specify that the problem is a medical problem, specifically a classifier for images.

[0376] In various embodiments, a user may select a problem type via a graphical user interface. The technology may display several generic models available to the user via the graphical user interface. The user may select one of the models by selecting one of the model descriptions using a pointing device or a touchscreen display. In addition to the generic models, additional custom models may also be selected and modified to solve the problem received as a second input.

[0377] At 1706, the functionality may include receiving a third input of one or more constraints for the machine learning application. The third input may be entered as native speech or text (e.g., using a chatbot) or selected via an interface (e.g., a graphical user interface). One example of a constraint is the latency of the system when executing the machine learning application. Another constraint may be the accuracy of the classification.

[0378] At 1708, functionality may include accessing a memory containing one or more ontologies of data. Each ontology may describe a hierarchical structure of one or more data sets for one or more data objects. Each data object may exhibit one or more properties that allow the data object to be classified into one or more sets of data objects according to a rule.

[0379] At 1710, functionality may include extracting one or more attributes of one or more data sets to find a first ontology relevant to applying predictions to a machine learning application based on one or more constraints. For example, the technology may learn from previous ontologies that certain KPIs and metrics are effective for solving certain solutions. Therefore, the technology may recommend incorporating various KPIs or metrics into the product graph.

[0380] At 1712, the functionality may include composing a product graph based on a first ontology stored in memory, one or more constraints, and one or more previous product graphs. A product graph associates one or more data objects with a set of nodes and edges for the data. Edges represent links between nodes. Nodes comprise the basic unit of a data structure.

[0381] For example, the search-based adaptation engine 1612 can search for metadata about various machine learning models that would be effective for solving a given problem. For example, an image classifier for detecting potential skin cancer may provide the accuracy needed to detect diabetic retinopathy. This technology can select the model, the required transformations, and ultimately, the specific pipeline.

[0382] At 1714 , the functionality may include storing the product image in memory.

[0383] In various embodiments, the method may include generating an ontology based at least in part on analyzing attributes of one or more data objects. The one or more attributes may include one or more functional semantics for each service. The functional semantics may include at least one of feature selection, outlier detection, and an interpretability metric.

[0384] The one or more properties may include one or more preconditions and one or more effects. The preconditions may be conditions that exist before the service is executed and the one or more effects may be conditions that exist after the service is executed.

[0385] The one or more attributes may also include one or more parameters related to the provenance of the service.

[0386] The one or more attributes may include one or more metrics. The one or more metrics may include a quality of service metric. The quality of service metric includes at least one of a latency metric, a response time metric, a throughput metric, a reliability metric, an availability metric, a success rate metric, and a privacy metric.

[0387] The one or more attributes may include one or more indicators. The one or more indicators may include key performance indicators.

[0388] The technique may include storing the ontology in a memory.

[0389] The technique may include generating metadata from the one or more ontologies, wherein the metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on runtime expectations. The technique may include saving the service metadata to a memory.

[0390] The technique may include searching for one or more services based at least in part on one or more descriptors from one or more previous product graphs. In various embodiments, the search uses a metaheuristic approach to determine a solution. The technique may include correlating the one or more descriptors with a prediction of a machine learning application. The technique may include generating a list of available services based at least in part on the correlation. The technique may include storing the list of available services in a memory.

[0391] The technique may include presenting a product graph on a display. The product graph may include one or more metrics of performance of the product graph. The product graph may include determining a set of tradeoffs, wherein the set of tradeoffs includes selecting one or more constraints in light of a prediction applied by a machine learning application. The technique may include presenting the set of tradeoffs on a display.

[0392] It should be recognized that Figure 17 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present disclosure may perform the above steps in a different order. Moreover, Figure 17 The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0393] IX. Adaptive Pipelining Composition Services

[0394] Existing machine learning systems do not incorporate features that continuously improve models through background processes. In this way, a model can be initially selected from available models and a machine learning application can be constructed. When the machine learning application is deployed, a background process can be used to identify other potential models that can improve various aspects of the machine learning application. For example, the model can be made available after the machine learning application is generated. The adaptive pipeline composition system can identify new models, test the models offline, verify the models, and if the model improves the performance of the machine learning application, the technology can replace the existing model in the deployed system with the new model. In this way, the machine learning application continuously improves its results. In addition, the model can be changed when new data is discovered or different data is identified as predictive.

[0395] refer to Figure 18 , shows a block diagram for composing a machine learning service that uses sample data. Given some sample data 1808, an adaptive pipeline composition service 1800 can compose a machine learning pipeline 1836 (composed of machine learning services) that maximizes performance properties on new data from the same source. A performance detector 1812 can evaluate various performance characteristics that can include classification accuracy, various model metrics, general QoS metrics, or other non-machine learning model related KPIs. Other frameworks use automatic learning to attempt to automatically compose the best pipeline based on model performance. However, model performance alone does not cover the case where the pipeline must adapt to some changes in online data metrics or QoS compliance at runtime. The adaptive pipeline composition service 1800 optimizes operation both offline and at runtime.

[0396] In an offline situation, a user can use the adaptive pipeline composition service 1800 to define the library components 168 of the pipeline, such as Figure 1 , to solve a specific problem. Prior learning / patterns of similar use cases are used to determine the pipeline 1836 for the newly specified problem.

[0397] In the online scenario, pipeline engine 1804 continuously explores and evaluates alternative pipelines and / or pipeline sub-blocks at runtime based on previous learning / patterns from similar use cases. Pipeline engine 1804 can run other pipeline candidates in shadow mode to compare performance in real time. If an alternative machine learning pipeline composition outperforms the existing pipeline, adaptive pipeline composition service 1800 composes the new pipeline and is responsible for upgrading it in shadow mode until it meets the conditions for automatic promotion to production. Once it performs better online, the redesigned pipeline replaces the old one.

[0398] Some embodiments have a threshold for improvement specified before replacement occurs. Other machine learning models may have different thresholds before replacement occurs. A new version of pipeline 1836 can be tested with the machine learning model before replacement occurs. For example, the new pipeline can be run in shadow mode for a period of time before replacement. The shadow pipeline can be further tuned to tailor it to a specific instance of the machine learning model.

[0399] Figure 19 Illustrated is a process 1900 for automatically composing a machine learning pipeline for use in a production environment that is executed as a background process of a machine learning application. Alternative embodiments may be combined, separated, or otherwise altered. Figure 19 To change the functionality described in the blocks shown in the figure. Figure 19 The functional components of one or more blocks shown in FIG may include hardware and / or software components of a distributed system, including Figure 22 、 23 and computing devices, storage devices, network infrastructure, and servers shown in and 24 and described below.

[0400] This technique can include both foreground and background processes. The foreground process can involve using the selected machine learning application to generate results. The background process can also be employed to analyze various aspects of the machine learning model and parameters to determine whether the process can be improved to provide better predictions for the machine learning application. Furthermore, the background process can determine whether the selected parameters and model slow down the machine learning application to the point where the output does not meet the performance requirements outlined in the service level agreement (SLA). In such cases, the model can be optimized to meet the requirements of the service level agreement.

[0401] At 1902, the functionality may include identifying a first machine learning model. The first machine learning model utilizes one or more first parameters. The one or more first parameters identify and define higher-level characteristics of a first data set prior to a machine learning process.

[0402] In machine learning, the term "hyperparameters" may be used to distinguish them from standard model parameters. A machine learning model is a mathematical formula with multiple parameters that need to be learned from the data. This is the crux of machine learning: fitting the model to the data. This is accomplished through a process called model training. In other words, by training the model with existing data, we are able to fit the model parameters. However, there is another type of parameter that cannot be directly learned from the regular training process. These parameters express "high-level" characteristics of the model, such as its complexity or how quickly it should learn. These are called hyperparameters. Hyperparameters are usually fixed before the actual training process begins. Hyperparameters are configurations external to the model and their values cannot be estimated from the data. Hyperparameters are often used during the process to help estimate model parameters.

[0403] At 1904, functionality may include identifying one or more first parameters used by a first machine learning model. The one or more first parameters may be specified by a practitioner. The one or more first parameters may often be set using heuristics. The one or more first parameters are often tuned for a given predictive modeling problem. A data scientist may not know the optimal values for the model hyperparameters on a given problem. Instead, the data scientist may use rules of thumb, replicate values used on other problems, or find the optimal values through trial and error. When tuning a machine learning algorithm for a particular problem, such as when using grid search or random search, the technique adjusts the hyperparameters or commands of the model to find the parameters of the model that produce the most skilled predictions. Some examples of model hyperparameters include: the learning rate used to train a neural network, the number of leaves or depth of a tree, the number of latent factors in a matrix factorization, the number of hidden layers in a deep neural network, and the number of clusters in k-means clustering.

[0404] At 1906, the functionality may include identifying one or more first results for a first machine learning model. The one or more first results are solutions to the model using production values for one or more parameters and the production model. As the model and / or parameters are varied in the machine learning application, the one or more first results form a basis for comparing various results.

[0405] At 1908, the functionality may include identifying one or more metrics for the first machine learning model. These metrics define how the performance of the first machine learning model can be measured. The metrics may include a KPI or QoS as described above.

[0406] At 1912, the functionality may include comparing the one or more first results to a second data set. The second data set includes ground truth data. By comparing the first results to the ground truth data, the technique may determine how accurate the model is using the selected one or more parameters and the selected model. Comparing the one or more first results to the second data set may calculate a first score. The first score may be a measure of the difference between the first results output by the model and the ground truth data used for the model. Using the one or more first parameters and the first model, the first score provides a measure of the quality of the model under some given metric.

[0407] At 1914, the functionality may include referencing an ontology of the first dataset to identify a second machine learning model based at least in part on comparing metadata of the second machine learning model to metrics for the first machine learning model. The second machine learning model may be identified for testing with the current machine learning model using a background process.

[0408] At 1916, functionality may include testing a second machine learning model on the first data set to produce one or more second results. The technology may test the second machine learning model in a background process. In this way, the production model is not affected by the testing and evaluation process.

[0409] At 1918, the functionality may include comparing the one or more second results to a second data set to generate a second score. The second data set includes ground truth data representing a desired output of the machine learning application. Using the one or more first parameters and the second machine learning model, the second score provides a measure of the model quality under some given metric, such as accuracy.

[0410] At 1920, the functionality may include replacing the first machine learning model with a second machine learning model for the machine learning application based on the second score being less than the first score. If the technology determines that the second machine learning model provides results that are closer to the ground truth data, then the technology may replace the machine learning model used by the machine learning application. In various embodiments, the first machine learning model may be replaced with the second machine learning model in shadow mode until the second machine learning model meets one or more conditions for automatically promoting the second machine learning model to production.

[0411] In various embodiments, the technique may include analyzing an ontology of a first data set to identify one or more second parameters based at least in part on comparing metadata of the first parameters to metrics for a first machine learning model. The technique may include testing the second machine learning model using the one or more second parameters to produce one or more third results. The technique may include comparing the one or more third results to the second data set to generate a third threshold. The technique may include replacing the one or more first parameters with the one or more second parameters for the machine learning application based on the third threshold being less than the second threshold.

[0412] In various embodiments, the technique may include generating a log including one or more first parameters, a first machine learning model, a second machine learning model, one or more first results, and one or more second results. The technique may include storing the log in a memory.

[0413] In various embodiments, the technique may include analyzing the logs to determine one or more patterns.

[0414] In various embodiments, the technology may include storing supplemental metadata about the second machine learning model based at least in part on the one or more first parameters and the one or more second results. In various embodiments, the metadata includes at least one of the number of levels of the decision tree and the number of parameters of the algorithm of the second machine learning model.

[0415] It should be recognized that Figure 19 The specific steps shown in provide specific techniques for generating machine learning applications according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present invention may perform the above steps in a different order. Moreover, Figure 19 The individual steps shown in the foregoing may include multiple sub-steps that may be performed in various orders as appropriate for the individual steps. Additionally, additional steps may be added or removed based on specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0416] X. Service Execution and Monitoring for Runtime Service Composition

[0417] During the execution of a machine learning service or pipeline, the environment is constantly changing, so user-defined desired states can become invalid. Invalid states can include changes in the environment, data corruption, model performance degradation, and / or the availability of new features. One purpose of the monitoring engine 156 is to provide the model composition engine 132 and the model execution engine 108 with an up-to-date view of the state of the execution environment for the machine learning platform 100 and to adhere to the QoS specifications defined when the machine learning service was composed.

[0418] Machine learning services and their ontologies are defined in deployable service descriptions. The model composition engine 132 uses these descriptions to assemble component services, triggering a search for the optimal architecture model at runtime. The architecture model includes pipelines 136 that specify any microservice routines 140, software modules 144, and infrastructure modules 148, as well as any customizations and interdependencies. Multiple QoS parameters associated with service execution (e.g., response time, latency, throughput, reliability, availability, success rate) are also based on the type of data input to the pipeline (volume, velocity) and the class of the pipeline (classifier, recommender system). Therefore, service composition with a large number of candidate services is a multi-objective optimization problem that can be solved to automate runtime adaptation. During service composition, multiple services can be combined in a specific order based on their input-output dependencies to produce the desired product graph. In addition to providing the required solution for pipeline X with data input Y, it is also necessary to ensure that end-to-end QoS requirements specified by the product team (KPIs) and the environment in which we operate are met. The execution engine schedules and invokes machine learning service instances to be composed and served at runtime.

[0419] Many variations and modifications of the disclosed embodiments may also be used. For example, various functions, blocks, and / or software may be distributed across a network, WAN, and / or cloud package. Machine learning software may also be run in a distributed manner across a network, WAN, and / or cloud infrastructure.

[0420] Figure 20 A simplified diagram of a system for service execution and monitoring runtime service components is illustrated. The system can detect when one or more conditions exist that can degrade the performance of a machine learning model. The system can identify one or more actions that can be taken to prevent, mitigate, or resolve any issues caused by changes to at least one of the system's resources, machine learning model concepts, data corruption, and data availability to the machine learning model.

[0421] The model monitoring agent 2002 can monitor the system's environment and the performance of the machine learning model. The model monitoring agent 2002 can monitor the historical performance of the model and the performance of the model compared to key performance indicators (KPIs) and service quality metrics.

[0422] The model monitoring agent 2002 can monitor concept drift. In the real world, concepts are often not stable but change over time. Typical examples are weather prediction rules and customer preferences. The underlying data distribution can also change frequently. These changes often make the model built on the old data inconsistent with the new data, so it is necessary to update the model regularly. This problem can be called concept drift, which complicates the task of learning models from data and requires special methods that are different from commonly used techniques, which regard the arriving instances as equally important contributors to the final concept. The monitoring engine 2002 can monitor customer data to detect whether concept drift is a potential problem for machine learning applications.

[0423] The model monitoring agent 2002 can monitor data corruption. Data corruption refers to errors that occur in computer data during writing, reading, storage, transmission or processing, which cause unexpected changes to the original data. Computers, transmission and storage systems can use multiple measures to provide end-to-end data integrity or error-free. Generally speaking, when data corruption occurs, the file containing that data will produce unexpected results when accessed by the system or related applications. The results can range from minor data loss to system crashes. For example, if a document file is corrupted, then when people try to open that file with a document editor, they will receive an error message, so the file may not open or may open with some data corrupted (or in some cases, completely corrupted, making the document incomprehensible). The model monitoring agent 2002 can monitor data and detect potential problems with data corruption.

[0424] The model monitoring agent 2002 can monitor new customer data. The new customer data may include different types of data that the model may not have been able to use before. The model monitoring agent 2002 can not only detect the presence of additional data of the same type used by the model, but also detect new types of data that can provide better predictions.

[0425] The model monitoring agent 2002 may notify the user of model degradation or if a KPI is not being met or currently cannot be met.

[0426] The external monitoring agent 2004 can detect one or more environmental changes. The environmental change can include a change in available memory. The environmental change can include a change in the availability of a processing node. The environmental change can include a change in network bandwidth.

[0427] The external monitoring agent 2004 can access historical data 2006. Historical data 2006 can be used to compare the predicted output of the model. If the predicted output is within the acceptable range of the historical data, then remedial action may not be needed. Remedial actions may cause other problems that are potentially worse than any changes to the system. The external monitoring agent can also save the output of the model to a database for future monitoring. In various embodiments, historical data 2006 can store a historical collection of problems and the solutions or remedial actions given to them. This will tilt the application in a different direction, toward adjusting the running of the machine learning model at 2008 to obtain the predicted remedial actions.

[0428] At 2008 , the system can adjust the operational model. The system can make one or more changes to prevent, mitigate, or resolve any issues presented by the system changes detected by the model monitoring agent 2002 and the external monitoring agent 2004 .

[0429] For some environmental changes, the system can replace transformations that are expensive in processing, memory, or bandwidth. Other remedial measures can include replacing or pruning one or more model parameters. Reducing model parameters reduces the requirements of the model. In various embodiments, the system can reduce model complexity by changing one or more of the library components.

[0430] For data corruption, the system can temporarily discard or remove the corrupted features. In various embodiments, the system can adjust the pipeline and / or layer to remove the corrupted features.

[0431] For concept drift issues, the system can force retraining the model using different data. In various embodiments, the different data includes the latest data. In various embodiments, the concept drift issue can be addressed by changing the size of the window selection to avoid corrupted data.

[0432] In cases where new data becomes available, the system can analyze the data to discover one or more new features. The system can evaluate the impact of the new data on the model metrics. In various embodiments, the system can discard any deviations due to sensitive attributes.

[0433] In some embodiments, the system can roll back to a previous model version 2010 to mitigate one or more detected issues. In some embodiments, the system can compose a new model pipeline using one or more model components 2012, replacing the model microservice with a different component. The system can save metadata about the currently running model. The metadata can be monitored by an external monitoring agent 2004.

[0434] Figure 21 An exemplary flow diagram for service execution and monitoring runtime service composition is illustrated.

[0435] Figure 21 is a flow chart of an example process 2100 for techniques for service execution and monitoring runtime service composition. In some embodiments, Figure 21 One or more process blocks of may be performed by a server system (e.g., a cloud-based server system). In some embodiments, Figure 21 One or more process blocks of may be performed by another device or group of devices separate from or including the cloud-based server.

[0436] At 2110, process 2100 may include receiving two or more quality of service (QoS) dimensions for a multi-objective optimization model, wherein the two or more QoS dimensions include at least a first QoS dimension and a second QoS dimension. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402 and / or data feed 2424, etc., as Figure 24 ) can receive two or more quality of service (QoS) dimensions for a multi-objective optimization model, as described above. In some embodiments, the two or more QoS dimensions include at least a first QoS dimension and a second QoS dimension.

[0437] At 2120, process 2100 may include maximizing a multi-objective optimization model along a first QoS dimension. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, etc., as shown in FIG. Figure 24 ) can maximize a multi-objective optimization model along the first QoS dimension, as described above.

[0438] At 2130, maximization can include selecting one or more pipelines that meet the QoS expectations specified for the first QoS dimension and the second QoS dimension for a multi-objective optimization model in the software architecture. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402 and / or data feed 2424, etc., as shown in FIG. Figure 24 ) can include selecting one or more pipelines that meet QoS expectations specified for a first QoS dimension and a second QoS dimension for a multi-objective optimization model in a software architecture.

[0439] At 2140, the ordering of the pipelines depends on which QoS dimensions are optimized and deoptimized and to what extent. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402 and / or data feed 2424, etc., as shown in FIG. Figure 24 ) can include ordering of pipelines depending on which QoS dimensions are optimized and deoptimized and to what extent.

[0440] At 2150, the multi-objective optimization model is partially deoptimized along the second QoS dimension to meet the QoS expectations for the first QoS dimension. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402 and / or data feed 2424, etc., as shown in FIG. Figure 24 and described below) may include, in part, being partially de-optimized along the second QoS dimension in order to comply with QoS expectations for the first QoS dimension.

[0441] At 2160, there is a tradeoff between the first QoS dimension and the second QoS dimension. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402 and / or data feed 2424, etc., as shown in FIG. Figure 24 ) may include a tradeoff between a first QoS dimension and a second QoS dimension.

[0442] In various embodiments, process 2100 may include retrieving data associated with historical outputs of a machine learning model. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, etc.) may be compared to a set of quality of service metrics and key performance indicators. Figure 24 and described below) can retrieve data associated with historical outputs of a machine learning model, as described above.

[0443] In various embodiments, process 2100 may include receiving one or more inputs from an environment monitoring agent, wherein the environment monitoring agent receives information about at least one of: resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, as Figure 24) and described below can receive one or more inputs from an environment monitoring agent, as described above. In some embodiments, the environment monitoring agent receives information about at least one of: resources of the system, concepts of the machine learning model, data corruption, and data availability for the machine learning model.

[0444] In various embodiments, process 2100 may include determining a change in at least one of: a resource of the system, a concept of the machine learning model, data corruption, and data availability to the machine learning model. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, etc.) Figure 24 , and described below) can determine a change in at least one of the following: the system's resources, the machine learning model's concepts, data corruption, and the availability of data to the machine learning model, as described above. For example, the system can detect the loss of several processing units. In another example, the system can detect data corruption in client data. In other examples, new client data, perhaps a new type of data, can become available during model execution.

[0445] In various embodiments, process 2100 may include determining whether a change in at least one of a resource of the system, a concept of the machine learning model, data corruption, and data availability to the machine learning model will cause the output of the prediction of the machine learning model to change by more than a predetermined amount. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, etc., as Figure 24 , and described below) can determine whether a change in at least one of the system's resources, the machine learning model's concepts, data corruption, and data availability to the machine learning model will cause the predicted output of the machine learning model to change by more than a predetermined amount, as described above. In some cases, the predetermined amount can be a percentage difference from historical output (i.e., 10%). In other cases, the predetermined amount can be compared to a KPI or QoS metric.

[0446] In various embodiments, process 2100 may include identifying one or more remedial actions for the machine learning model to correct the change when a change in at least one of a system's resources, a concept of the machine learning model, data corruption, and data availability to the machine learning model would cause a predicted output of the machine learning model to change by more than a predetermined amount. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, etc., as described herein) may include identifying one or more remedial actions for the machine learning model to correct the change when a change in at least one of a system's resources, a concept of the machine learning model, data corruption, and data availability to the machine learning model would cause a predicted output of the machine learning model to change by more than a predetermined amount. Figure 24, and described below) can identify one or more remedial actions for a machine learning model to correct a change in at least one of a system's resources, a machine learning model's concepts, data corruption, and data availability to the machine learning model, as described above, when the change would cause a predicted output of the machine learning model to change by more than a predetermined amount. The system can include a plurality of stored remedial actions. The remedial actions can be encoded with metadata identifying the one or more changes for which the remedial actions can be used.

[0447] In various embodiments, process 2100 may include displaying an alert to notify a user of a resource of the system, a concept of the machine learning model, a change in at least one of data corruption and data availability to the machine learning model, and one or more remedial actions. For example, a server system (e.g., using processing unit 2404, storage subsystem 2418, system memory 2410, communication subsystem 2424, bus 2402, and / or data feed 2424, etc., as Figure 24 and described below) can display alerts to notify a user of the system's resources, concepts of the machine learning model, data corruption, and changes to at least one of data availability to the machine learning model, and one or more remedial actions, as described above.

[0448] Process 2100 may include additional embodiments, such as any single embodiment or any combination of embodiments described below and / or in combination with one or more other processes described elsewhere herein. Figure 21 The specific steps shown in provide specific techniques for service execution and monitoring runtime service composition according to various embodiments of the present disclosure. According to alternative embodiments, other sequences of steps may also be performed. For example, alternative embodiments of the present disclosure may perform the steps outlined above in a different order. Moreover, Figure 21 The individual steps shown in the figure may include multiple sub-steps, which may be performed in various orders according to the individual steps. In addition, additional steps may be added or removed according to specific applications. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0449] In some embodiments, the predicted output includes at least one of a first metric related to the performance of the multi-objective optimization model with respect to the quality of service parameter and a second metric related to the prediction of the multi-objective optimization model compared to historical outputs of the multi-objective optimization model.

[0450] In some embodiments, process 2100 includes performing one or more remedial actions on the machine learning model to correct the change.

[0451] In some implementations, the resources of the system include at least one of available memory, processing nodes, and network bandwidth.

[0452] In some embodiments, a concept measures the statistical distribution of the performance of a machine learning model.

[0453] In some implementations, data availability includes new data for one or more new features.

[0454] In some implementations, one or more remedial actions to the machine learning model include reducing the complexity of the machine learning model.

[0455] In some implementations, one or more remedial actions on a machine learning model include eliminating one or more features affected by data corruption.

[0456] In some embodiments, one or more remedial actions to a machine learning model include evaluating the impact of new features on a predicted output.

[0457] In some implementations, one or more remedial actions on the machine learning model include rolling back the machine learning model to a previous version.

[0458] In some implementations, the one or more remedial actions include at least one of composing a new model pipeline and replacing the machine learning model microservice.

[0459] In various embodiments, the server device may include one or more memories; and one or more processors in communication with the one or more memories and configured to execute instructions stored in the one or more memories to perform the operations of the above-described methods.

[0460] In various embodiments, a computer-readable medium stores a plurality of instructions that, when executed by one or more processors of a computing device, cause the one or more processors to perform the operations of any of the methods described above.

[0461] Although Figure 21 Example steps of process 2100 are shown, but in some implementations, process 2100 may include Figure 21 2100. Additionally or alternatively, two or more of the steps of process 2100 may be performed in parallel.

[0462] XI. Exemplary Hardware and Software Configurations

[0463] Figure 22A simplified diagram of a distributed system 2200 for implementing one of the embodiments is depicted. In the illustrated embodiment, the distributed system 2200 includes one or more client computing devices 2202, 2204, 2206, and 2208 configured to execute and operate client applications, such as web browsers, proprietary clients (e.g., Oracle Forms), etc., over one or more networks 2210. A server 2212 can be communicatively coupled to the remote client computing devices 2202, 2204, 2206, and 2208 via the network 2210.

[0464] In various embodiments, the server 2212 may be adapted to run one or more services or software applications provided by one or more of the components of the system. In some embodiments, these services may be provided to users of the client computing devices 2202, 2204, 2206, and / or 2208 as web-based or cloud services or under a software as a service (SaaS) model. Users operating the client computing devices 2202, 2204, 2206, and / or 2208 may, in turn, utilize one or more client applications to interact with the server 2212 to utilize the services provided by these components.

[0465] In the configuration depicted in the figure, the software components 2218, 2220 and 2222 of system 2200 are shown as being implemented on server 2212. In other embodiments, one or more of the components of system 2200 and / or the services provided by these components may also be implemented by one or more of client computing devices 2202, 2204, 2206 and / or 2208. The user operating the client computing device can then utilize one or more client applications to use the services provided by these components. These components can be implemented in hardware, firmware, software or a combination thereof. It should be appreciated that various system configurations are possible, which may be different from distributed system 2200. The embodiment shown in the figure is therefore an example of a distributed system for implementing an embodiment system and is not intended to be limiting.

[0466] Client computing devices 2202, 2204, 2206, and / or 2208 may be portable handheld devices (e.g., Cellular phones, computing tablets, personal digital assistants (PDAs), or wearable devices (e.g., Google head-mounted display), running software such as Microsoft Windows ) and / or various mobile operating systems (such as iOS, Windows Phone, Android, BlackBerry 10, Palm OS, etc.), and enable Internet, email, short message service (SMS), or other communication protocols. The client computing device can be a general-purpose personal computer, including, for example, a computer running various versions of Microsoft Apple The client computing device may be a personal computer and / or laptop computer running various commercially available or any workstation computer running any of the UNIX-like operating systems, including but not limited to various GNU / Linux operating systems, such as, for example, Google Chrome OS. Alternatively or additionally, the client computing devices 2202, 2204, 2206, and 2208 may be any other electronic device, such as a thin client computer, an Internet-enabled gaming system (e.g., with or without a gesture input device) and / or a personal messaging device capable of communicating over network(s) 2210.

[0467] Although the exemplary distributed system 2200 is shown with four client computing devices, any number of client computing devices can be supported. Other devices (such as devices with sensors, etc.) can interact with the server 2212.

[0468] The network(s) 2210 in the distributed system 2200 may be any type of network familiar to those skilled in the art, which may support data communications using any of a variety of commercially available protocols, including but not limited to TCP / IP (Transmission Control Protocol / Internet Protocol), SNA (Systems Network Architecture), IPX (Internetwork Packet Exchange), AppleTalk, and the like. By way of example only, the network(s) 2210 may be a local area network (LAN), such as one based on Ethernet, Token Ring, and the like. The network(s) 2210 may be a wide area network and the Internet. It may include virtual networks, including but not limited to virtual private networks (VPNs), intranets, extranets, public switched telephone networks (PSTNs), infrared networks, wireless networks (e.g., over any of the Institute of Electrical and Electronics Engineers (IEEE) 802.11 protocol suites, and / or any other wireless protocol); and / or any combination of these and / or other networks.

[0469] The server 2212 may be composed of one or more general-purpose computers, dedicated server computers (for example, including PC (personal computer) servers, Servers 2212 may include servers, mid-range servers, mainframe computers, rack servers, etc.), server farms, server clusters, or any other suitable arrangement and / or combination. In various embodiments, server 2212 may be adapted to run one or more services or software app...

Claims

1. A method for automatically creating a machine learning application customized for a hardware platform for use in a production environment, the method comprising: receiving a first input, wherein the first input identifies a location of data; receiving a second input, wherein the second input describes a prediction for the machine learning application; receiving a third input, wherein the third input comprises one or more constraints for a machine learning application; accessing a memory containing one or more ontologies of data, wherein each ontology describes a hierarchical structure of one or more data sets for one or more data objects, wherein each data object exhibits one or more properties that allow classification of the data objects into one or more sets of data objects according to a rule; extracting one or more attributes of the one or more data sets to find a first ontology relevant to prediction for a machine learning application based on the one or more constraints; composing a product graph based on the first ontology stored in the memory, the one or more constraints, and one or more previous product graphs, wherein the product graph relates the one or more data objects to a set of nodes and edges, wherein the edges represent links between the nodes; wherein a node comprises a basic unit of data structure; and The product image is stored in memory.

2. The method of claim 1, further comprising: Generating an ontology based at least in part on analyzing properties of the one or more data objects, wherein the one or more properties include: one or more functional semantics for each service, the one or more functional semantics comprising at least one of feature selection, outlier detection, and an interpretability metric; one or more preconditions and one or more effects, wherein a precondition is a condition that exists before the service is performed and the one or more effects are conditions that exist after the service is performed; one or more parameters, wherein the one or more parameters relate to the provenance of the service; one or more metrics, wherein the one or more metrics include a quality of service metric; and one or more indicators, wherein the one or more indicators include key performance indicators; and The ontology is stored in memory.

3. The method of claim 2, wherein the quality of service metric comprises at least one of a latency metric, a response time metric, a throughput metric, a reliability metric, an availability metric, a success rate metric, and a privacy metric.

4. The method of claim 1 , further comprising: generating service metadata from the one or more ontologies, wherein the service metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on runtime expectations; as well as Save service metadata to storage.

5. The method of claim 1 , further comprising: searching for one or more services based at least in part on one or more descriptors from the one or more previous product graphs; correlating the one or more descriptors with a prediction of a machine learning application; generating a list of available services based at least in part on the correlation; as well as A list of available services is saved in memory. The method of claim 5 , wherein the searching uses a metaheuristic approach to determine a solution.

7. The method of claim 1 , further comprising: presenting a product graph on a display, the product graph including one or more metrics of performance of the product graph; determining a set of tradeoffs, wherein the set of tradeoffs comprises selecting the one or more constraints in light of the predictions of the machine learning application; and An eclectic collection is presented on display.

8. A computer program product tangibly embodied in a non-transitory machine-readable storage medium, comprising instructions configured to cause a data processing apparatus to perform operations for automatically creating a machine learning application for use in a production environment, the operations comprising: receiving a first input, wherein the first input identifies a location of data; receiving a second input, wherein the second input describes a prediction for the machine learning application; receiving a third input, wherein the third input comprises one or more constraints for a machine learning application; accessing a memory containing one or more ontologies of data, wherein each ontology describes one or more data sets of one or more data objects, wherein each data object exhibits one or more properties that allow classification of the data objects into one or more sets of data objects according to a rule; extracting one or more attributes of the one or more data sets to find a first ontology relevant to prediction for a machine learning application based on the one or more constraints; composing a product graph based on the first ontology stored in the memory, the one or more constraints, and one or more previous product graphs, wherein the product graph relates the one or more data objects to a set of nodes and edges, wherein the edges represent links between the nodes; wherein a node comprises a basic unit of data structure; and The product image is stored in memory.

9. The computer program product of claim 8, comprising instructions configured to cause a data processing apparatus to perform further operations, the operations comprising: Generating an ontology based at least in part on analyzing properties of the one or more data objects, wherein the one or more properties include: one or more functional semantics for each service, the one or more functional semantics comprising at least one of feature selection, outlier detection, and an interpretability metric; one or more preconditions and one or more effects, wherein a precondition is a condition that exists before the service is performed and the one or more effects are conditions that exist after the service is performed; one or more parameters, wherein the one or more parameters relate to the provenance of the service; one or more metrics, wherein the one or more metrics include a quality of service metric; and one or more indicators, wherein the one or more indicators include key performance indicators; and The ontology is stored in memory.

10. The computer program product of claim 9, wherein the quality of service metric comprises at least one of a latency metric, a response time metric, a throughput metric, a reliability metric, an availability metric, a success rate metric, and a privacy metric.

11. The computer program product of claim 8, comprising instructions configured to cause a data processing apparatus to perform further operations, the further operations comprising: generating service metadata from the one or more ontologies, wherein the service metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on runtime expectations; as well as Save service metadata to storage.

12. The computer program product of claim 8, comprising instructions configured to cause a data processing apparatus to perform further operations, the further operations comprising: searching for one or more services based at least in part on one or more descriptors from the one or more previous product graphs; correlating the one or more descriptors with a prediction of a machine learning application; generating a list of available services based at least in part on the correlation; as well as A list of available services is saved in memory.

13. The computer program product of claim 12, wherein the search uses a metaheuristic approach to determine a solution.

14. The computer program product of claim 8, comprising instructions configured to cause a data processing apparatus to perform further operations, the further operations comprising: presenting a product graph on a display, the product graph including one or more metrics of performance of the product graph; determining a set of tradeoffs, wherein the set of tradeoffs comprises selecting the one or more constraints in light of the predictions of the machine learning application; and An eclectic collection is presented on display.

15. A system for automatically creating a machine learning application for use in a production environment, comprising: one or more data processors; as well as A non-transitory computer-readable storage medium containing instructions that, when executed on the one or more data processors, cause the one or more data processors to perform operations comprising: receiving a first input, wherein the first input identifies a location of data; receiving a second input, wherein the second input describes a prediction for the machine learning application; receiving a third input, wherein the third input comprises one or more constraints for a machine learning application; accessing a memory containing one or more ontologies of data, wherein each ontology describes one or more data sets of one or more data objects, wherein each data object exhibits one or more properties that allow the data objects to be classified into one or more sets of data objects according to a rule; extracting one or more attributes of the one or more data sets to find a first ontology relevant to prediction for a machine learning application based on the one or more constraints; composing a product graph based on the first ontology stored in the memory, the one or more constraints, and one or more previous product graphs, wherein the product graph relates the one or more data objects to a set of nodes and edges, wherein the edges represent links between the nodes; wherein a node comprises a basic unit of data structure; and The product image is stored in memory.

16. The system of claim 15, wherein the non-transitory computer-readable storage medium comprises further instructions that, when executed on the one or more data processors, cause the one or more data processors to perform further operations, the further operations comprising: Generating an ontology based at least in part on analyzing properties of the one or more data objects, wherein the one or more properties include: one or more functional semantics for each service, the one or more functional semantics comprising at least one of feature selection, outlier detection, and an interpretability metric; one or more preconditions and one or more effects, wherein a precondition is a condition that exists before the service is performed and the one or more effects are conditions that exist after the service is performed; one or more parameters, wherein the one or more parameters relate to the provenance of the service; one or more metrics, wherein the one or more metrics include a quality of service metric; and one or more indicators, wherein the one or more indicators include key performance indicators; and The ontology is stored in memory.

17. The system of claim 16, wherein the quality of service metric comprises at least one of a latency metric, a response time metric, a throughput metric, a reliability metric, an availability metric, a success rate metric, and a privacy metric.

18. The system of claim 15, wherein the non-transitory computer-readable storage medium comprises additional instructions that, when executed on the one or more data processors, cause the one or more data processors to perform additional operations, the additional operations comprising: generating service metadata from the one or more ontologies, wherein the service metadata describes one or more concepts of the one or more ontologies from a machine learning perspective based at least in part on runtime expectations; as well as Save service metadata to storage.

19. The system of claim 15, wherein the non-transitory computer-readable storage medium comprises additional instructions that, when executed on the one or more data processors, cause the one or more data processors to perform additional operations, the additional operations comprising: searching for one or more services based at least in part on one or more descriptors from the one or more previous product graphs; correlating the one or more descriptors with a prediction of a machine learning application; generating a list of available services based at least in part on the correlation; as well as The list of available services is stored in memory.

20. The system of claim 15, wherein the non-transitory computer-readable storage medium comprises additional instructions that, when executed on the one or more data processors, cause the one or more data processors to perform additional operations, the additional operations comprising: presenting a product graph on a display, the product graph including one or more metrics of performance of the product graph; determining a set of tradeoffs, wherein the set of tradeoffs comprises selecting the one or more constraints in light of the predictions of the machine learning application; and An eclectic collection is presented on display.

Images