Creating and managing machine learning pipelines

Abstract

The method includes generating (701) an authoring representation (304) of the machine learning pipeline based on received input, the authoring representation configured to govern one or more machine learning operations. The method also includes receiving (703) an indication of an operation (310) to be performed on the authoring representation. The method further includes converting (705) the authoring representation to an intermediate representation (306) based on the operation and optimizing (707) the intermediate representation. The method also includes converting (709) the intermediate representation to an execution representation (307) that can be understood by one or more machine learning executors.

Description

[Technical field]

[0001] The present disclosure is directed generally to machine learning systems and, more particularly, to systems and methods for creating and managing machine learning pipelines. [Background technology]

[0002] A machine learning pipeline is a software system that provides a method for composing and executing multiple data processing and machine learning steps in an ordered procedure. Each step can take in one or more data inputs and return one or more data outputs. Machine learning pipelines are often configured to perform one or more machine learning operations. Typical operations of a pipeline may include training one or more machine learning steps, automatically adjusting training parameters of a machine learning step (e.g., during hyperparameter optimization), predicting new data given a pipeline containing one or more trained machine learning steps, measuring the performance of the prediction results (scoring), or interpreting the contribution level of different input data to the prediction results. The functionality of a basic machine learning pipeline may be extended to include additional operations or to change the logic of existing operations. Summary of the Invention

[0003] The present disclosure relates to systems and methods for creating and managing machine learning pipelines.

[0004] In a first embodiment, a method includes generating an authoring representation of a machine learning pipeline based on received input, the authoring representation configured to govern one or more machine learning operations. The method also includes receiving an indication of an operation to be performed on the authoring representation. The method further includes converting the authoring representation to an intermediate representation based on the operation and optimizing the intermediate representation. The method also includes converting the intermediate representation to an execution representation understood by one or more machine learning executors.

[0005] In a second embodiment, an apparatus includes at least one processing device configured to generate an authored representation of a machine learning pipeline based on received input, the authored representation configured to govern one or more machine learning operations. The at least one processing device is also configured to receive instructions of operations to be performed on the authored representation. The at least one processing device is further configured to convert the authored representation to an intermediate representation and optimize the intermediate representation based on the operations. The at least one processing device is also configured to convert the intermediate representation to an executable representation understood by one or more machine learning executors.

[0006] In a third embodiment, the non-transitory computer readable medium stores computer readable program code that, when executed by the one or more processors, causes the one or more processors to generate an authored representation of a machine learning pipeline based on received input, the authored representation configured to manage one or more machine learning operations. The non-transitory computer readable medium also stores computer readable program code that, when executed by the one or more processors, causes the one or more processors to receive instructions of operations to be performed on the authored representation. The non-transitory computer readable medium further stores computer readable program code that, when executed by the one or more processors, causes the one or more processors to convert the authored representation to an intermediate representation based on the operations and optimize the intermediate representation. The non-transitory computer readable medium also stores computer readable program code that, when executed by the one or more processors, causes the one or more processors to convert the intermediate representation to an executable representation understood by one or more machine learning executors.

[0007] Other technical features will be readily apparent to one skilled in the art from the following figures, descriptions, and claims. [Brief description of the drawings]

[0008] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which:

[0009] [Figure 1] 1 illustrates an example system that supports the creation and management of machine learning pipelines in accordance with the present disclosure. [Diagram 2] 1 illustrates an example of a device that supports the creation and management of machine learning pipelines in accordance with the present disclosure. [Diagram 3] 1 illustrates an example architecture for creating and managing machine learning pipelines in accordance with the present disclosure. [Figure 4A] 4 illustrates an example of an optimization that can be performed in the optimization layer in the architecture of FIG. 3 according to the present disclosure. [Figure 4B] 4 illustrates an example of an optimization that can be performed in the optimization layer in the architecture of FIG. 3 according to the present disclosure. [Figure 4C] 4 illustrates an example of an optimization that can be performed in the optimization layer in the architecture of FIG. 3 according to the present disclosure. [Figure 4D] 4 illustrates an example of an optimization that can be performed in the optimization layer in the architecture of FIG. 3 according to the present disclosure. [Diagram 5] 4 illustrates an example of machine learning pipeline composability using the architecture of FIG. 3 in accordance with the present disclosure. [Figure 6] 4 illustrates an example of a directed acyclic graph (DAG) machine learning pipeline that can be authored using the architecture shown in FIG. 3 in accordance with this disclosure. [Figure 7] 1 illustrates an example method for creating and managing machine learning pipelines in accordance with the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] 1-7 described below, and the various embodiments used herein to describe the principles of the present invention, are for illustrative purposes only and should not be construed as limiting the scope of the invention in any way. Those skilled in the art will appreciate that the principles of the present invention may be implemented in any type of suitably arranged device or system.

[0011] As described above, a machine learning pipeline is a software system that provides a method for composing and executing multiple data processing and machine learning steps in an ordered procedure. Each step can take in one or more data inputs and return one or more data outputs. A machine learning pipeline is often configured to perform one or more machine learning operations. Typical operations of a pipeline may include training one or more machine learning steps, automatically adjusting training parameters of a machine learning step (e.g., during hyperparameter optimization), predicting new data given a pipeline containing one or more trained machine learning steps, measuring the performance of the prediction results (scoring), or interpreting the contribution level of different input data to the prediction results. The functionality of a basic machine learning pipeline may be extended to include additional operations or to change the logic of existing operations.

[0012] In some systems, an "authored representation" of a machine learning pipeline represents a sequence of steps configured by a user to define the pipeline, where the authored representation is specific to one or more machine learning operations. For example, a machine learning pipeline using some systems may require a training step that is separate from the prediction step. Also, some systems do not include a Directed Acyclic Graph (DAG) topology for the authoring or execution graph.

[0013] The present disclosure provides an apparatus, method, and computer-readable medium that support a process for the creation and management of machine learning pipelines. The disclosed embodiments allow multiple operations to be performed on a single pipeline representation. This allows for a single user-defined representation ("authoring representation") of the machine learning pipeline, so that the user does not need to maintain a separate pipeline for each operation. In other words, the user (e.g., a data scientist) only needs to author the machine learning pipeline once, integrating training, prediction, scoring, tuning, and interpretation of the operations without having to repeat any of the operations. Here, the authoring representation can manage each described operation without requiring an operation-specific step or a different user-defined pipeline architecture corresponding to each operation.

[0014] The disclosed authoring expressions allow for static typing of the inputs and outputs of each step, which can be used to verify that the steps are properly connected. The authoring expressions also allow for unambiguous definition of the input / output signatures of the machine learning operations in the pipeline, thus building a well-defined interface for connecting to external software systems. The disclosed authoring expressions also support Directed Acyclic Graph (DAG) topologies, as described in more detail below.

[0015] FIG. 1 illustrates an example system 100 that supports the creation and management of a machine learning pipeline in accordance with the present disclosure. For example, the system 100 illustrated here can be used to support a three-tier machine learning pipeline architecture described below. As illustrated in FIG. 1, the system 100 includes user devices 102a-102d, one or more networks 104, one or more application servers 106, and one or more database servers 108 associated with one or more databases 110. Each of the user devices 102a-102d communicates over the network 104, for example, via a wired or wireless connection. Each of the user devices 102a-102d represents any suitable device or system used by at least one user to provide or receive information, such as a desktop computer, a laptop computer, a smartphone, and a tablet computer. However, any other or additional types of user devices may be used in the system 100.

[0016] Network 104 facilitates communication between various components of system 100. For example, network 104 may communicate Internet Protocol (IP) packets, Frame Relay frames, Asynchronous Transfer Mode (ATM) calls, or other suitable information between network addresses. Network 104 may include all or a portion of one or more local area networks (LANs), metropolitan area networks (MANs), wide area networks (WANs), a global network such as the Internet, or any other communication system or multiple communication systems in one or more locations.

[0017] The application server 106 is coupled to the network 104 and is coupled to or otherwise in communication with a database server 108. In some embodiments, the application server 106 supports a three-tiered machine learning pipeline architecture described below. For example, the application server 106 may execute one or more applications 112 that use data from the database 110 to perform operations related to the creation and management of a machine learning pipeline. It should be noted that the database server 108 may also be used within the application server 106 to store information, in which case the application server 106 may itself store the information used to perform operations related to the creation and management of a machine learning pipeline.

[0018] The database server 108 operates to store and facilitate retrieval of various information used, generated, or collected by the application server 106 and the user devices 102a-102d in the database 110. For example, the database server 108 may store various information related to the creation and management of machine learning pipelines.

[0019] FIG. 1 illustrates an example of a system 100 that supports the creation and management of machine learning pipelines, although various modifications may be made to FIG. 1. For example, system 100 may include any number of user devices 102a-102d, a network 104, an application server 106, a database server 108, and a database 110. Furthermore, these components may be located in any suitable location and may be distributed over a wide area. Additionally, while FIG. 1 illustrates an example of an operating environment in which a machine learning pipeline may be used, this functionality may be used in any other suitable system.

[0020] 2 depicts an example device 200 that supports the creation and management of a machine learning pipeline according to the present disclosure. One or more instances of the device 200 may be used to, for example, at least partially implement the functionality of the application server 106 of FIG. 1. However, the functionality of the application server 106 may be implemented in any other suitable manner. In some embodiments, the device 200 depicted in FIG. 2 may form at least a portion of the user devices 102a-102d, the application server 106, or the database server 108 of FIG. 1. However, each of these components may be implemented in any other suitable manner.

[0021] 2, device 200 represents a computing device or system that includes at least one processing device 202, at least one storage device 204, at least one communication unit 206, and at least one input / output (I / O) unit 208. Processing device 202 may execute instructions that may be loaded into memory 210. Processing device 202 may include any suitable number and types of processors or other processing devices in any suitable arrangement. Example types of processing device 202 include one or more microprocessors, microcontrollers, reduced instruction set computers (RISC), complex instruction set computers (CISC), graphics processing units (GPUs), data processing units (DPUs), virtual processing units, associative processing units (APUs), tensor processing units (TPUs), vision processing units (VPUs), neuromorphic chips, AI chips, quantum processing units (QPUs), Cerebras wafer-scale engines (WSEs), digital signal processors (DSPs), ASICs, field programmable gate arrays (FPGAs), or discrete circuits.

[0022] Memory 210 and persistent storage 212 are examples of storage device 204, which represent any structure capable of storing and facilitating retrieval of information (e.g., data, program code, and / or other suitable information, either temporary or persistent). Memory 210 may represent random access memory or any other suitable volatile or non-volatile storage device. Persistent storage 212 may include one or more components or devices that support longer-term storage of data, such as read-only memory, a hard drive, a flash memory, or an optical disk.

[0023] The communications unit 206 supports communication with other systems or devices. For example, the communications unit 206 may include a network interface card or a wireless transceiver to facilitate communication over a wired or wireless network, such as the network 104. The communications unit 206 may support communication over any suitable physical or wireless communication link.

[0024] I / O unit 208 allows for the input and output of data. For example, I / O unit 208 may provide a connection for user input through a keyboard, mouse, keypad, touch screen, or other suitable input device. I / O unit 208 may also send output to a display, printer, or other suitable output device. It should be noted that I / O unit 208 may be omitted if device 200 does not require local I / O, such as when device 200 represents a server or other device that can be accessed remotely.

[0025] Figure 2 illustrates one example of a device 200 that supports creation and management of machine learning pipelines, although various modifications may be made to Figure 2. For example, computing and communication devices and systems come in a wide variety of configurations, and Figure 2 does not limit the disclosure to any particular computing or communication device or system.

[0026] Figure 3 illustrates an example architecture 300 for creating and managing machine learning pipelines according to the present disclosure. In some cases, the architecture 300 may be implemented in a machine learning pipeline system that may execute using one or more devices, such as the application server 106 or one of the user devices 102a-102d of Figure 1. As shown in Figure 3, the architecture 300 is three-tiered, allowing a user (e.g., a data scientist, a machine learning pipeline creator, etc.) to declare what needs to be computed, while leaving it up to the system to decide how to perform the computation.

[0027] In this example, the three layers include an authoring layer 301, an optimization layer 302, and an execution layer 303. A user interacts only with the “top” layer of the architecture 300, namely the authoring layer 301. A user typically uses the authoring layer 301 to define an authoring representation 304 of a machine learning pipeline 308. In the architecture 300, the authoring representation 304 manages the various operations 310 of the machine learning pipeline 308 without requiring steps specific to the machine learning operations corresponding to each operation 301 or a different user-defined pipeline architecture. That is, the authoring representation 304 does not require a user to describe, for example, explicit training steps, then prediction steps, when authoring the machine learning pipeline 308. For example, consider a user configuring a simple machine learning pipeline 308 in the authoring layer 301: Pipeline X = Step A → Step B → Step C

[0028] Note that this machine learning pipeline 308 does not have steps specific to machine learning operations such as "train" or "predict" specified in the authoring layer 301. Instead, such steps are implicit in the configuration of the machine learning pipeline 308. When an ML operation is executed in a pipeline X, such as pipelineX.train() (which represents pipeline X + operation "train"), the following steps are executed automatically without user definition: stepA.train() to train a model for step A, stepA.process() to generate inputs for step B, stepB.train() to train a model for step B, stepB.process() to generate inputs for step C, stepC.train() to train a model for step C, and then packaging of all the trained steps.

[0029] If the user specifies an operation 310 to be performed on the authoring representation 304, the authoring representation 304 is converted to an intermediate representation 306 via a conversion operation 305. The operation 310 to be performed specified by the user may include training one or more machine learning operations, adjusting one or more training parameters of one or more machine learning operations, predicting new data, scoring the performance of the prediction result, interpreting the contribution level of different input data to the prediction result, etc. Here, scoring the performance of the prediction result refers to measuring the quality of the prediction output by the model. For example, one scoring metric for a machine learning model is accuracy, in which, given a predicted value and a ground truth value, a real number (score) is determined that indicates how well the prediction matches the ground truth. Other examples of scoring metrics for machine learning models include precision, recall, and mean absolute error. Architecture 300 allows a user to author and train a machine learning model once (e.g., using a large dataset), and the machine learning model can be run many times using fewer inputs.

[0030] The intermediate representation 306 can be optimized by the system using the optimization layer 302 to obtain other intermediate representations 306 that can produce the same result. In the optimization layer 302, the system automatically optimizes the execution of the machine learning pipeline 308 based on various criteria, such as for cost or latency, depending on the inputs provided and the outputs required. In some embodiments, various optimizations may be applied when converting the authored machine learning pipeline 308 into an executable representation. For example, Figures 4A-4D depict examples of optimizations 401-404 that can be performed in the optimization layer 302 in the architecture of Figure 3 according to the present disclosure.

[0031] FIG. 4A illustrates an example of vertex fusion 401. As shown in FIG. 4A, the intermediate representation 306 includes two vertices 411 and 412 that have compatible execution environments. Vertex fusion 401 combines the two vertices 411 and 412 into a single vertex 413. This can improve performance, for example, by minimizing marshalling and unmarshalling overhead and by minimizing passing data between steps. Although vertex fusion 401 is shown in FIG. 4A as combining two vertices into one vertex, this is only an example and other numbers of vertices may be fused into other, fewer, numbers of vertices.

[0032] FIG. 4B illustrates an example of vertex expansion 402. As shown in FIG. 4B, the intermediate representation 306 includes a vertex 421. Vertex expansion 402 distributes or splits the single vertex 421 into multiple vertices 422-426. Expanding a vertex can advantageously enable concurrent execution of different portions of the vertex in a distributed system, for example, by splitting input data into separate batches for prediction. Although vertex expansion 402 is shown in FIG. 4B as splitting one vertex into five vertices, this is merely an example and other numbers of vertices may be split into other larger numbers of vertices.

[0033] Figure 4C illustrates an example of common subexpression elimination 403. As shown in Figure 4C, intermediate representation 306 includes subgraphs 431 and 432, which in this case are overlapping or redundant with each other. Common subexpression elimination 403 identifies overlapping or redundant subgraphs 431 and 432 and eliminates one of the subgraphs 432 while leaving only subgraph 431. The transformed intermediate representation 306 remains directed and acyclic. Although common subexpression elimination 403 is illustrated in Figure 4C as eliminating one of the two overlapping or redundant subgraphs, this is merely an example and other numbers of subgraphs or other subexpressions may be eliminated.

[0034] FIG. 4D illustrates an example of redundant data conversion elimination 404. Redundant data conversion elimination refers to automatically avoiding compute-intensive data conversion. In the example illustrated in FIG. 4D, the intermediate representation 306 includes four steps 441-444. Step 443 ("step 3") is the inverse of step 442 ("step 2"), i.e., the output of step 443 is the same as the input of step 442. Redundant data conversion elimination 404 collectively removes steps 442 and 443 while leaving only steps 441 and 444. This can improve performance, for example, by minimizing the overhead of data exchange between steps. Although redundant data conversion elimination 404 is illustrated in FIG. 4D as removing two steps, this is merely an example and other numbers of steps may be removed.

[0035] 4A-4D depict various examples of optimizations 401-404 that can be performed in optimization layer 302, although various modifications may be made to FIGS. 4A-4D. For example, other or additional types of optimizations may be performed in optimization layer 302. Also, note that all optimizations preserve semantics with the authored machine learning pipeline 308 and are applied using a strategy that defines both the order of optimizations and the number of times the optimizations occur.

[0036] Returning to FIG. 3, once one or more optimizations have been performed on the intermediate representation 306, the system converts the intermediate representation 306 into an executed representation 307 that is understood by the underlying executor. The executor determines how the executed representation 307 should be executed to return the desired results. There may be multiple options for the execution of the executed representation 307 in the machine learning pipeline 308, such as execution on an executor that is best suited for training on large data, point inference with low latency requirements, etc.

[0037] The architecture 300 supports heterogeneous execution environments for various steps or operations in the machine learning pipelines 308. Among other things, this can enable user flexibility to use open source languages, frameworks, and libraries. That is, the architecture 300 is not limited to any particular system, development application, or production application. Also, each machine learning pipeline 308 can be exported or imported between different applications as needed or desired.

[0038] Further, differentiation of the machine learning pipelines 308 in the architecture 300 allows a user to remove one or more operations 310 of the machine learning pipeline 308 or to replace one or more operations 310 with one or more alternative operations 310. This differentiation may also allow different operations to be executed using different hardware, such as to support different resource requirements. In contrast, the architecture 300 allows a user to separate the execution representation 307 of the machine learning pipeline 308 into multiple parts, so that different operations 310 or different steps of one operation 310 may be executed on different (potentially more suitable or advantageous) hardware. For example, step A may be a pre-processing routine, step B may be a deep learning model, and step C may be a post-processing routine. The architecture 300 allows steps A and C to be executed on commodity hardware, while step B may utilize accelerated hardware such as a GPU. In some embodiments, a user may provide hints, parameters, or instructions to help the system determine on which hardware the operations should be executed.

[0039] To prevent unwanted or unwanted changes to the machine learning pipeline 308, the architecture 300 enables freezing some or all of the machine learning pipeline 308. As used herein, "freezing" a machine learning pipeline refers to the ability of a system to enforce immutability after a particular user action occurs. One example is freezing a machine learning pipeline after training a machine learning model. Because the machine learning pipeline is composable, freezing may only affect one or more portions of the entire machine learning pipeline without affecting one or more other portions of the machine learning pipeline.

[0040] In some embodiments, the architecture 300 allows the system to learn the minimum resources (e.g., memory, CPU, GPU, etc.) required to execute operations 310 in the machine learning pipeline 308. Once the required resources are learned, the system can restrict certain operations 310 (e.g., prediction, interpretation, etc.) to enable execution on resource-constrained devices.

[0041] In the architecture 300, each operation 310 in the machine learning pipeline 308 may be implemented in a unique language with a language version, use a unique framework and library, and run on a unique executor. Some examples of languages ​​that may be used include JAVA, JAVASCRIPT, PYTHON, etc. Some examples of frameworks and libraries that may be used include TENSORFLOW, KERAS, SCIKIT-LEARN, SPACY, HUGGINGFACE, XGBOOST, etc. Some examples of executors that may be used include SPARK, RAY, APACHE, AIRFLOW, ARGO WORKFLOWS, etc. Of course, other languages, frameworks, libraries, and executors are possible, and these examples are not intended to limit the scope of this disclosure.

[0042] The architecture 300 facilitates pipeline composability by allowing users to compose machine learning pipelines 308 from existing, independently authored machine learning pipelines while the system preserves all claimed properties. Furthermore, optimizations can be performed on the composition of multiple machine learning pipelines 308. In some embodiments, multiple machine learning pipelines 308 can be executed as a single execution graph even if the machine learning pipelines 308 do not know about each other. Also, machine learning pipelines 308 can be nested effortlessly. For example, a machine learning pipeline 308 developed by user A can be reused in a larger machine learning pipeline 308 by user B. User A's machine learning pipeline 308 can be trained and prevented from being trained for future users (e.g., by freezing), which allows sharing of interesting features without confidentiality or intellectual property infringement or performance degradation from user B. As a concrete example, user A develops a computer vision pipeline and user B wants to use the computer vision pipeline for a default detection application in manufacturing. The architecture 300 allows User B to use User A's pipeline and add pre-processing and post-processing steps for User B's particular application. Similarly, the machine learning pipeline 308 developed by User A for a given task can be shared without obfuscation. User B can decide to reuse User A's machine learning pipeline 308 but replace one or more steps with User B's own implementation. This allows User B to use his own expertise in a particular domain while leveraging previous work on pre-processing, post-processing, and any parallel tasks built by User A.

[0043] Once the machine learning pipeline 308 is configured, the architecture 300 supports persistence of the complete pipeline, including the ability for users to name, save, and retrieve the machine learning pipeline 308. This allows for graceful deployment of the machine learning pipeline 308. Additionally, the architecture 300 allows for rich query filters to search the machine learning pipeline 308 because it treats the machine learning models as reusable objects. Once the machine learning pipeline 308 is configured, the architecture 300 also allows for inspection of the machine learning pipeline 308. Pipeline inspection allows a user to trace the execution path back to the authoring level to understand how the machine learning pipeline 308 was originally authored. For example, a user can inspect the performance of the machine learning pipeline 308 or determine the time it took to make a prediction or train.

[0044] FIG. 5 illustrates an example 500 of machine learning pipeline composability using architecture 300 according to the present disclosure. As shown in FIG. 5, example 500 includes multiple machine learning pipelines 501-503, each of which may represent (or be represented by) machine learning pipeline 308 of FIG. 3. Machine learning pipeline 501 ("Dana's Pipeline") is composed of elements from a previously generated machine learning pipeline 502 ("Mike's Pipeline (v1.0)") and a previously generated machine learning pipeline 503 ("Jane's Pipeline (v2.0)"). To compose machine learning pipeline 501, Dana adds a sentiment classifier 504 to machine learning pipelines 501-502 before publishing them. Machine learning pipelines composed in one language or using one framework can later be reused in machine learning pipelines composed in other languages ​​or using other frameworks. For example, the previously generated machine learning pipeline 502 may be constructed using various PYTHON® libraries such as MINECRAFT and TESSERACT, while the sentiment classifier 504 may be constructed using the TENSORFLOW framework.

[0045] Figure 5 illustrates an example 500 of machine learning pipeline composability using architecture 300, although various modifications may be made to Figure 5. For example, the languages ​​and frameworks illustrated in Figure 5 are merely examples, and other languages ​​and / or frameworks may be used additionally or alternatively.

[0046] The architecture 300 allows for the composition of multiple steps into a directed acyclic graph (DAG) machine learning pipeline, and allows for the nesting of multiple machine learning pipelines within a higher level machine learning pipeline. A DAG machine learning pipeline can have one or more source nodes (multiple inputs) and one or more sink nodes (multiple outputs). This can be useful and important for many machine learning applications, such as when a user wants to combine the output of multiple models applied with multiple data sources to calculate one or several final scores.

[0047] The architecture 300 supports heterogeneous execution environments for different steps in the DAG machine learning pipeline to allow users the flexibility to use open source languages, frameworks, and libraries. Each step in the machine learning pipeline may include a unique language, language version, frameworks and libraries, and executor. For example, FIG. 6 depicts an example DAG machine learning pipeline 600 that may be authored by a user of the system in the authoring layer 301 of the architecture 300 depicted in FIG. 3 in accordance with the present disclosure. As depicted in FIG. 6, the DAG machine learning pipeline 600 is a computer vision pipeline that can be used for diagram analysis. The DAG machine learning pipeline 600 includes multiple source nodes 601-603 that provide structured and unstructured data, and one sink node 604 that serves as the output of the DAG machine learning pipeline 600.

[0048] The DAG machine learning pipeline 600 includes multiple steps 605-614 developed using a number of languages ​​and frameworks, including PYTHON, KERAS, and TESSERACT. For example, partial information extraction 605 may include extraction of partial information from the component list 601 and the diagram 603 (e.g., a PDF diagram), document extraction 606 may include identifying individual documents in the diagram 603, diagram preprocessing 607 may include cleaning up and filtering document data, symbol detection 608 may include detecting specific symbols in the diagram 603, symbol identification 609 may include identifying specific symbols in the diagram 603, item number identification 610 may include identifying item numbers in the diagram 603, knowledge consolidation 611 may include consolidating information obtained from the diagram 603, assembly detection 612 may include detecting one or more assemblies in the extracted documents, OCR 613 may include optical character recognition of the extracted documents, and item number identification 614 may include identifying item numbers in the extracted documents. Of course, these steps, languages, and frameworks are merely examples, and other steps, languages, and frameworks may be used in addition or alternatively.

[0049] Figure 6 represents one example of a DAG machine learning pipeline 600 that may be authored by a user of the system at authoring layer 301 of architecture 300, although various modifications may be made to Figure 6. For example, the particular DAG machine learning pipeline 600 shown here is for illustrative purposes only. Other machine learning pipelines that support directed acyclic graphs may be used without departing from the scope of this disclosure.

[0050]

[0023] Figure 3 illustrates an example of an architecture 300 for creating and managing machine learning pipelines, although various modifications may be made to Figure 3. For example, the architecture 300 shown here is for illustrative purposes only. In general, ML pipeline architectures come in a wide variety of configurations, and Figure 2 does not limit the present disclosure to any particular ML pipeline architecture. Other ML pipeline architectures may be used without departing from the scope of the present disclosure.

[0051] Figure 7 illustrates an example method 700 for creating and managing a machine learning pipeline in accordance with the present disclosure. For ease of explanation, the method 700 illustrated in Figure 7 is described as involving the use of the application server 106 illustrated in Figure 1 and the architecture 300 illustrated in Figure 3. However, the method 700 illustrated in Figure 7 may be used in any other suitable system, with any other suitable devices and architectures.

[0052] 7, in step 701, an authoring representation of the machine learning pipeline is generated based on input from a user. This may include, for example, the server 106 generating an authoring representation 304 of the machine learning pipeline 308 using the authoring layer 301. In some embodiments, the authoring representation is configured to manage one or more machine learning operations without requiring machine learning operation-specific steps or different user-defined pipeline architectures corresponding to each machine learning pipeline.

[0053] In step 703, an indication of an action to be performed on the authoring representation is received from a user. This may include, for example, the server 106 receiving an indication of the action 310 from the user. In step 705, the authoring representation is converted to an intermediate representation based on the action. This may include, for example, the server 106 performing a conversion operation 305 to convert the authoring representation 304 to the intermediate representation 306. In step 707, the intermediate representation is optimized. This may include, for example, the server 106 optimizing the intermediate representation 306 in an optimization layer 302. In step 709, the intermediate representation is converted to an executed representation that can be understood by one or more machine learning executors. This may include, for example, the server 106 converting the intermediate representation 306 to an executed representation 307 in an execution layer 303. In step 711, the executed representation is executed. This may include, for example, the server 106 executing the application or code represented by the executed representation 307.

[0054] Figure 7 illustrates one example of a method 700 for generating and managing a machine learning pipeline, although various modifications may be made to Figure 7. For example, although shown as a series of steps, the steps in Figure 7 may overlap, occur in parallel, occur in a different order, or occur any number of times.

[0055] In some embodiments, the method includes transforming the intermediate representation into an execution representation understood by one or more machine learning executors, the intermediate representation being a transformation based on operations of the authoring representation, the authoring representation being of a machine learning pipeline configuration for managing one or more machine learning operations.

[0056] In some embodiments, various functions described herein are implemented or supported by a computer program formed from computer readable program code and embodied in a computer readable medium. The phrase "computer readable program code" includes any type of computer code, including source code, object code, and executable code. The phrase "computer readable medium" includes any type of medium that can be accessed by a computer, such as read only memory (ROM), random access memory (RAM), hard disk drive (HDD), compact disk (CD), digital video disk (DVD), or any other type of memory. "Non-transitory" computer readable medium excludes wired, wireless, optical, or other communication links that carry transient electrical or other signals. Non-transitory computer readable medium includes media on which data can be stored permanently, and media on which data can be stored and later overwritten, such as rewritable optical disks or erasable storage devices.

[0057] It may be advantageous to explain definitions of certain words or phrases used throughout this specification. The terms "application" and "program" refer to one or more computer programs, software components, sets of instructions, procedures, functions, objects, classes, instances, associated data, or portions thereof that are adapted to be implemented in suitable computer code (including source code, object code, or executable code). The term "communicate" and its derivatives encompass both direct and indirect communication. The terms "including" and "having" and their derivatives are intended to include without limitation. The terms "or", "or" and "alternative" are inclusive, meaning "and / or". The phrase "related to" and its derivatives may mean "including," "contained in," "interconnected with," "containing," "contained in," "connected to," "coupled with," "communicated with," "cooperate with," "interleaved," "juxtaposed," "close to," "associated with," "having," "having the quality of," "related to," and the like. The phrase "at least one of," when used in conjunction with a list of items, means that different combinations of one or more of the items in the list may be used and that only one item in the list may be required. For example, "at least one of A, B, and C" includes any of the following combinations: A, B, C, A and B, A and C, B and C, and A, B, and C.

[0058] The description in this application should not be read as implying that any particular element, step, or function is essential or critical to inclusion in the scope of the claims. The scope of patentable subject matter is defined solely by the scope of the allowed claims. Moreover, none of the claims invoke 35 U.S.C. 112(f) with respect to any of the appended claims or claims, unless the precise terms "means for" or "step for" are expressly used in a particular claim following a participial phrase identifying the function. The use of terms such as, but not limited to, "mechanism," "module," "device," "unit," "component," "element," "member," "apparatus," "machine," "system," "processor," or "component" in the claims is understood and intended to refer to structures known to those skilled in the relevant art, as further modified or expanded by the features of the claims themselves, and is not intended to invoke 35 U.S.C. 112(f).

[0059] While the present disclosure has been described with respect to specific embodiments and generally related aspects, alterations and permutations of these embodiments and methods will be apparent to those skilled in the art. Thus, the above description of exemplary embodiments does not define or constrain the present disclosure. Other modifications, substitutions, and alterations are possible without departing from the spirit and scope of the present disclosure, as defined by the following claims.

Claims

1. generating an authoring representation of the machine learning pipeline based on the received input, the authoring representation configured to manage one or more machine learning operations; receiving an indication of an operation to be performed on the authoring representation; converting the authoring representation into an intermediate representation based on the actions; optimizing the intermediate representation; and converting the intermediate representation into an executable representation understood by one or more machine learning executors; The method according to claim 1,

2. The operation includes: training one or more machine learning operations; adjusting one or more training parameters of the one or more machine learning operations; and Predicting new data and Scoring the performance of the prediction results; and Interpret the contribution level of different input data to the forecast results having at least one of: The method of claim 1.

3. The operations include scoring the performance of the prediction results by scoring at least one of accuracy, precision, recall, and mean absolute error. The method of claim 2.

4. optimizing the intermediate representation includes combining multiple vertices of the intermediate representation into a single vertex for execution by a compatible execution environment. The method of claim 1.

5. optimizing the intermediate representation includes splitting a single vertex into multiple vertices for concurrent execution; The method of claim 1.

6. and dividing the executable representation into multiple portions for execution on different hardware. The method of claim 1.

7. The machine learning pipeline comprises a previously generated machine learning pipeline to which one or more pre-processing or post-processing operations have subsequently been added for a particular application. The method of claim 1.

8. composing the plurality of machine learning operations into a directed acyclic graph (DAG) machine learning pipeline. The method of claim 1.

9. the authoring representation is configured to manage the one or more machine learning operations independently of machine learning operation-specific steps corresponding to each machine learning operation; The method of claim 1.

10. 10. Apparatus comprising at least one processing device configured to perform the method according to any one of claims 1 to 9.

11. A non-transitory computer readable medium storing computer readable program code which, when executed, causes one or more processors to perform the method of any one of claims 1 to 9.

Images