Cloud-based data analytics system architecture

Abstract

This patent application discloses a data analysis system having a split-plane architecture that utilizes a workflow management module in the control plane for cloud-based computing. The system receives workflow files from client devices and stores those workflow files in cloud storage associated with the user. The data plane that executes the workflow files retrieves the files directly from the cloud storage or via an API provided by the workflow management module. A workflow engine wrapper, including a stager system and an execution system, prepares the workflow files for execution and stores the generated results in cloud storage.

Description

Technical Field

[0001] This application relates to the architecture of cloud-based data analysis systems.

Background Art

[0002] Cross-reference to Related Applications This application claims the benefit of U.S. Provisional Patent Application Nos. 63 / 503,368, 63 / 503,370, and 63 / 503,373, each filed on May 19, 2023, and U.S. Patent Application Nos. 18 / 403,603, 18 / 403,606, and 18 / 403,611, each filed on January 3, 2024, all of which are incorporated by reference.

[0003] Data analysis involves extracting business-valuable information from large datasets. For example, small businesses utilize third-party data analysis environments that use dedicated computing and human resources to collect, process, and analyze vast amounts of data from various data sources such as external data providers, internal data sources (e.g., local computer files, etc.), big data stores, and cloud-based data (e.g., social media information, etc.). Data analysis systems provide these services to small entities that need data analysis but lack the computing resources to perform these processes themselves.

[0004] A data analysis system may provide a software application that includes software tools for designing and executing repeatable workflows via a visual graphical user interface (GUI). For example, the GUI associated with a data analysis system may provide a drag-and-drop workflow environment for data preparation, data blending, data processing, and advanced data analysis. Furthermore, the workflow may include a set of data processing tools that perform specific processing operations or data analysis functions. The software application can be desktop-based, server-based, on-premises, or cloud-based. Data analysis systems often benefit from the computing resources available in cloud computing environments. However, significant gaps remain in workflow execution support in cloud computing environments. [Overview of the Initiative]

[0005] A data analytics system with a split-plane architecture enables cloud-based computing capabilities using a workflow management module in the control plane. The control plane of the data analytics system receives user requests to execute workflows from workflow files stored in a cloud storage account. A workflow file is a file that describes the steps of a workflow to be executed by the data analytics system. For example, a workflow file may be a JSON or XML file that describes the steps of the workflow to be executed. In some cases, a workflow file is a packaged workflow file that also stores dependency files.

[0006] The control plane publishes messages to a message queue requesting jobs to execute workflows, and the data plane consumes messages when there are sufficient resources to execute new jobs. The execution service within the data plane creates an engine wrapper container within the data plane. This wrapper container stores the execution engine for executing the workflow. The execution engine wrapper container uses the job metadata in the messages from the message queue to retrieve the packaged workflow file from the cloud storage account, and uses the execution engine to execute the workflow file and generate the workflow results. When the execution service determines that the job is complete, it deletes the execution engine wrapper container and sends the results of executing the workflow file to cloud storage for storage so that users on client devices can retrieve the results as needed.

[0007] The control plane of the data analytics system may also be used to coordinate the storage and retrieval of packaged workflow files using API endpoints. The control plane receives packaged workflow files from the user's client device and stores the packaged workflow files in a cloud storage account. The control plane receives API requests for packaged workflow files from an execution engine wrapper container running on the data plane. This data plane may be the data plane of the data analytics system or it may be run by a third-party system. The control plane retrieves the packaged workflow file from the cloud storage account using the identifier in the API request and sends the packaged workflow file to the execution engine wrapper container. The control plane receives the results of executing the workflow from the data plane and sends the execution results to the cloud storage account for storage.

[0008] The data plane may use an execution engine wrapper container to prepare workflow files for execution by the execution engine. The data plane's execution service receives a request to run a workflow with the packaged workflow file. The execution service creates an execution engine wrapper container containing a staging system, worker systems, and the execution engine. The execution service provides the job request to the worker system. The worker system calls the staging system to retrieve the packaged workflow file and unpack its contents into the local file system. The worker system then starts the execution of the packaged workflow file by the execution engine. The worker system receives the results of the workflow execution and provides the execution results to the control plane of the data analysis system. [Brief explanation of the drawing]

[0009] [Figure 1] Figure 1 is a block diagram showing a data analysis environment including a data analysis system according to one embodiment. [Figure 2] Figure 2 is a block diagram showing a workflow created using a data analysis system that processes data according to one embodiment. [Figure 3] Figure 3 shows exemplary data flows through the data analysis system environment shown in Figure 1, according to several embodiments. [Figure 4] Figure 4 is an interaction diagram illustrating exemplary data flows for executing a workflow packaged as a PEA from a client device, in several embodiments. [Figure 5] Figure 5 shows an exemplary data flow of a workflow service module in several embodiments. [Figure 6] Figure 6 is an interaction diagram showing another exemplary data flow for executing a workflow file from a client device, according to several embodiments. [Figure 7]Figure 7 shows exemplary data flows through an execution engine wrapper container in several embodiments. [Figure 8] Figure 8 is an interaction diagram showing another exemplary data flow for executing a workflow file from a client device, according to several embodiments. [Figure 9] Figure 9 is a high-level block diagram showing a functional diagram of a typical computer system for use as the data analysis system shown in Figure 1, according to several embodiments.

[0010] The drawings illustrate various embodiments for illustrative purposes only. Those skilled in the art will readily recognize from the following description that alternative embodiments of the structures and methods illustrated herein can be utilized without departing from the principles of the invention as described herein. Similar reference numerals and symbols in the various drawings indicate similar elements. [Modes for carrying out the invention]

[0011] Figure 1 is a block diagram showing a data analysis environment including a data analysis system 110 according to one embodiment. The environment further includes multiple data sources 120 and client devices 160 connected to the data analysis system 110 via a network 130. While the exemplary environment 100 includes only one data analysis system 110 coupled to multiple data sources 120, embodiments may have multiple data analysis systems and a single data source.

[0012] The data source 120 provides electronic data to the data analysis system 110. The data source 120 may be a storage device such as a hard disk drive (HDD) or solid-state drive (SSD), a computer that manages and provides access to multiple storage devices, a storage area network (SAN), a database, or a cloud storage system (e.g., a data warehouse or data lake on a cloud platform). The data source 120 may also be a computer system capable of retrieving data from other sources. The data source 120 may be remote from the data analysis system 110 and may provide data via the network 130. Furthermore, some or all of the data source 120 may be directly coupled to the data analysis system and provide data without passing data through the network 130.

[0013] The data provided by data source 120 is typically organized into data records, each containing one or more values. For example, a data record provided by data source 120 may contain a series of comma-separated values. The data describes information relevant to a company using data analysis system 110. For example, data from data source 120 may describe computer-based interactions with content accessible on a website and / or social media applications (e.g., click tracking data).

[0014] The data analysis system 110 is a computer-based system used to process and analyze large amounts of data. Data is collected, gathered, or otherwise accessed directly from multiple data sources 120 or via a network 130. The data analysis system 110 can implement scalable software tools and hardware resources used to access, prepare, blend, and analyze data from a wide variety of data sources. For example, the data analysis system 110 supports the execution of data-intensive processes and workflows.

[0015] The data analysis system 110 processes and analyzes data obtained from one or more of the data sources 120. In some cases, the data analysis system 110 provides software that supports network or cloud-based access to data analysis tools and macros by multiple end users. For example, the data analysis system 110 enables users to share, view, and utilize analyses in a manner similar to a mobile application store or other cloud-based service. Analytical data, macros, and workflows can be packaged and run as small, customizable analytical applications (i.e., apps) that can be accessed by other users of the data analysis system 110.

[0016] In some cases, access to a published analytical application can be managed by the data analysis system 110. That is, it provides access control and security functions by granting or revoking access. The data analysis system 110 can perform functions associated with analytical applications, such as creation, deployment, publication, iteration, and updating. Furthermore, the data analysis system 110 can support functions performed at various stages related to data analysis, such as the ability to handle errors that occur during data analysis, and the ability to access, prepare, blend, analyze, and output analytical results.

[0017] In one embodiment, the data analysis system 110 supports software tools for designing and executing repeatable workflows via a visual GUI. For example, the GUI associated with the data analysis system 110 provides a drag-and-drop workflow environment for data blending, data processing, and advanced data analysis. Furthermore, the workflow may include a set of data processing tools that perform specific processing operations or data analysis functions. Each tool that is part of the workflow performs a data-related function specific to that tool. For example, the workflow may include tools that implement various data analysis functions, including one or more of input / output operations, preparation operations, participation operations, predictive operations, spatial operations, investigation operations, and analysis and transformation operations. A more detailed description of the workflow is provided in relation to Figure 2.

[0018] Therefore, software tools enable users to create workflows in a low-code or no-code environment, i.e., without deep knowledge of data processing languages ​​such as SQL. For example, a software tool may be encapsulated in a desktop-based application that the user downloads and installs on their local machine, or it may be an on-premise application installed on a computer on behalf of the user or organization (e.g., business). Through the application, the user can flexibly create workflows that perform one or more processing operations on an input data source 120 to generate output data as a result of the processing operations.

[0019] However, many of these workflow input data sources require a significant amount of storage capacity (e.g., millions or billions of rows) and are stored in data source 120 configured on a cloud computing environment. For example, data source 120 can be part of a data lake or data warehouse configured on a cloud environment. There is a significant gap remaining in workflow execution support in a cloud computing environment because existing cloud capabilities may not support the editing or execution of many workflows created by software applications on a desktop. Therefore, a solution is needed that enables the execution of workflows close to the data within a cloud computing environment.

[0020] Therefore, in one embodiment, data analysis system 110 configures a cloud execution ecosystem that seamlessly facilitates the execution of workflows close to data source 120 without restricting the selected vendor or location of the cloud computing environment. The cloud execution ecosystem is a collection of patterns, components, services, and functions aimed at supporting unified engine execution across multiple cloud environments. The cloud execution ecosystem of data analysis system 110 includes a set of a control plane 140 and a data plane 150 that perform the tasks described above.

[0021] The control plane is a logical plane (e.g., a virtual private cloud) that stores and manages the shared resources of data analysis system 110. For example, control plane 140 processes tasks related to network management and configuration, message routing, and policy enforcement for data analysis system 110. In one embodiment, control plane 140 is configured as a multi-tenant plane where each account or tenant of data analysis system 110 shares the same computing resources to orchestrate these services. Control plane 140 can operate under the cloud account infrastructure of data analysis system 110.

[0022] The data plane 150 is a logical plane (e.g., a virtual private cloud) that processes data processing tasks of the data analysis system 110. For example, the data plane 150 stores, accesses, and provides data for a user, and performs calculations and analyses on the data. An account (e.g., a customer organization) of the data analysis system 110 may host one or more data plane instances 150 in a secure environment using computing resources dedicated to that account. In one embodiment, the data plane 150 within the cloud execution ecosystem described herein may be composed of a containerization engine and is established with specifications such as job orchestration, metadata exchange, data input and output contracts, as will be described in more detail below.

[0023] In this way, the supported execution environments follow a pattern in which services such as user authentication, workflow issuance, job scheduling, and result display experiences are consistent across different types of cloud environments. However, each type of execution environment may have a specific implementation based on the constraints and limitations of that environment.

[0024] Furthermore, one way to address the execution of data processing operations on a cloud platform is to perform SQL pushdown by translating and optimizing the workflow into a language such as SQL, and then running the SQL code in the native computing environment of the data source. However, even with SQL pushdown, data movement remains a problem, as data is shuffled from one computing environment to another. This is not optimal for various reasons, including cost, performance, data security, and data residency. Moreover, many data analysis processes require capabilities beyond these, such as artificial intelligence and machine learning, spatial analysis, reporting and visualization, data connectors, and custom code. Therefore, the cloud execution ecosystem described herein enables workflow execution to be carried out by bringing the execution engine closer to the source and allowing the engine to run within different ecosystems. The control plane 140 and data plane 150 are described in more detail below.

[0025] Network 130 represents the communication path between the data analysis system 110 and the data source 120. In one embodiment, network 130 is the internet and uses standard communication technologies and / or protocols. Thus, network 130 can include links using technologies such as Ethernet, 802.11, Global Interoperability Microwave Access (WiMAX), 3G, Long-Term Evolution (LTE), Digital Subscriber Line (DSL), Asynchronous Transfer Mode (ATM), InfiniBand, and PCI Express Advanced Switching. Similarly, networking protocols used in network 130 can include Multiprotocol Label Switching (MPLS), Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), Hypertext Transport Protocol (HTTP), Simple Mail Transfer Protocol (SMTP), and File Transfer Protocol (FTP).

[0026] Data exchanged over network 130 can be represented using technologies and / or formats including Hypertext Markup Language (HTML), Extended Markup Language (XML), and others. Furthermore, all or some links can be encrypted using conventional encryption technologies such as Secure Sockets Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec). In another embodiment, entities may use custom and / or proprietary data communication technologies instead of, or in addition to, the technologies described above.

[0027] The client device 160 is a device that allows a user to interact with the data analysis system 110. The client device 160 can be a personal computing device or a mobile computing device, such as a smartphone, tablet, laptop computer, or desktop computer. In some embodiments, the customer client device 160 runs a client application that communicates with the data analysis system 110 using an application programming interface (API). The client device 160 can send query requests to the data analysis system 110 and receive responses to those queries from the data analysis system 110. For example, the client device 160 can communicate directly with the data plane 150 of the data analysis system 110 using an access token. The interaction of the client device with the data analysis system is described in further detail below.

[0028] In one embodiment, a user of a client device 160 may download and install software tools or applications from the data analysis system 110 that enable the user to create workflows that perform data processing operations (e.g., extract, transform, load (ETL) operations) on a data source via a GUI. In one embodiment, the user may connect to the cloud execution ecosystem of the data analysis system 110 to schedule and run workflows using computing resources in the cloud.

[0029] Figure 2 is a block diagram showing a workflow 200 created using a data analysis system 110 to process data, according to one embodiment. In one embodiment, the workflow 200 is created using a visual workflow environment supported by the GUI of the data analysis system 110. The visual workflow environment operates a set of drag-and-drop tools, eliminating the need to provide software code for designing the workflow and the need to identify complex mathematical formulas implemented by the workflow. In another embodiment, the workflow 200 is created and described in a document such as an XML document. The workflow 200 is executed by a computer device of the data analysis system 110. However, in other embodiments, the workflow 200 is located on another computer device that can be communicated to the data analysis system 110 via a network (e.g., network 130).

[0030] A workflow can include a set of tools that perform specific processing operations or data analysis functions. As a common example, the tools in a workflow can perform one or more of the following analytical functions: input / output operations, preparation operations, participation operations, predictive operations, spatial operations, investigation operations, and analysis and transformation operations. Workflow implementation can include defining, executing, and automating a data analysis process, where data is passed to each tool in the workflow, and each tool performs its respective processing operation on the received data. Data records can be passed through the tools in a workflow. For example, passing a packet containing aggregated groups of individual data records to the tools in the workflow allows for more efficient execution of individual processing operations on the data. Such aggregation techniques can speed up workflow development and execution, even when processing large amounts of data. A workflow can define, or otherwise structure, a repeatable set of operations by specifying the sequence of operations for designated tools. In some cases, the tools included in a workflow are executed linearly. In other cases, multiple tools are executed in parallel.

[0031] As shown in the figure, the workflow 200 in Figure 2 includes input tools 205 and 206, as well as input / output tools indicated as a browse tool 230. Input tools 205 and 206 function to access data records from a specific data source 120. Input tools 205 and 206 take the accessed data records into the workflow and provide the data records to subsequent tools in workflow 200. In this example, input tool 205 provides the accessed records to the filter tool 210, and input tool 206 provides the accessed data records to the select tool 211. The browse tool 230 is located at the end of workflow 200 and receives the output resulting from each execution of the upstream tools in workflow 200. In this example, the browse tool 230 is located at the end of workflow 200, but the browse tool 230 can be added at any point in the workflow to review and verify the results of the execution of the upstream tools in the workflow.

[0032] Continuing with the example in Figure 2, workflow 200 includes preparation tools, shown as the filter tool 210, the select tool 211, the formula tool 215, and the sample tool 212. The filter tool 210 queries data records based on a formula and separates the data into two streams: a true stream containing data records that satisfy the formula and a false stream containing data records that do not satisfy the formula. The select tool 211 can be used to select, deselect, sort, and rename fields, change their type or size, and assign descriptions. The formula tool 215 creates or updates fields using one or more formulas that perform a wide variety of calculations and / or operations. The sample tool 212 limits the incoming stream of data records to a number, percentage, or random set of data records. Workflow 200 also includes a join tool 220 that blends multiple data sources. Specifically, the join tool 220 combines two input data streams based on a common field (or data record location).

[0033] The workflow 200 in Figure 2 is shown to include a summarization tool 225, which is an analysis and transformation tool that can reconstruct and reshape the data into a format used for further analysis. The summarization tool 225 can also perform data summarization by grouping, summing, counting, spatial processing, and string concatenation. In one embodiment, the output generated by the summarization tool 225 includes the results of calculations.

[0034] In some embodiments, the execution of workflow 200 results in the input tool 205 passing data records (e.g., one at a time) through the filter tool 210 and formula tool 215 until all data records have been processed and reached the join tool 220. The input tool 206 then begins passing data records one at a time through the select tool 211 and sample tool 212 until the data records are passed to the same join tool 220. Some individual tools in workflow 200 may have the ability to implement their own parallel operations, such as starting to read a block of data while processing the last block of data, or splitting computer-intensive operations such as sorting into multiple parts. However, in some existing workflow technologies, each data record from a set of data records is processed individually by each tool in the workflow, one at a time, in a pipeline manner until it reaches a tool in the workflow that requires multiple data records to perform a processing operation (e.g., sorting, join, summarizing, etc.).

[0035] The specific naming conventions, capitalization of terms, attributes, data structures, or other programming or structural aspects of the components are not mandatory or important, and the mechanisms implementing the embodiments described may have different names, formats, or protocols. Furthermore, the system may be implemented through a combination of hardware and software as described, or entirely with hardware elements. Also, the specific division of functions among the various system components described herein is merely illustrative and not mandatory. Functions performed by a single system component may instead be performed by multiple components, and functions performed by multiple components may instead be performed by a single component.

[0036] 1. Executing workflows in the cloud execution ecosystem Figure 3 shows exemplary data flows through the cloud execution ecosystem of the data analysis system 110 shown in Figure 1, according to several embodiments. Alternative embodiments may include more, fewer, or different components, and the functionality described below may be divided among components different from those described below. Furthermore, each component may perform its respective function automatically without human intervention.

[0037] In one embodiment, the cloud execution ecosystem includes a control plane 140 and a data plane 150. The control plane 140 includes a workflow management module 300 and a user interface (UI) module 314. In one embodiment, the workflow management module 300 includes components such as a platform API 302, a platform connector 304, a workflow service module 306, a scheduling module 308, and an execution service 312 (for the control plane). In one embodiment, the exemplary data plane instance 150 shown in Figure 3 is under an account owned by the user but managed by a data analytics system 110. The data plane 150 includes an execution host 334. The execution host 334 includes an execution engine wrapper container 330 and an execution service 332 (for the host).

[0038] A user of client device 160 authenticates with the ecosystem, for example, using a desktop software application. After the user creates a workflow using the software application, client device 160 sends the workflow file to the control plane 140. In one example, the workflow file is a file that describes the steps of the workflow to be performed by the data analysis system 110. For example, the workflow file may be a JSON or XML file that describes the steps of the workflow to be performed. The workflow file may have a type that indicates the information the workflow file stores. For example, the workflow file may be a baseline workflow file type in which the workflow file stores only the steps of the workflow. This file type may refer to dependency files used by the workflow, but may not include the dependency files themselves.

[0039] In one embodiment, the workflow file may also be a packaged workflow file type, also described herein as a Portable Execution Artifact (PEA). The packaged workflow file type (e.g., YXZP) can be configured to store a baseline workflow file and dependency files used when the workflow is executed. In one embodiment, a user may publish a PEA via the workflow service module 306, and the cloud execution ecosystem has a set of requirements and capabilities to support desktop workflow execution in the cloud.

[0040] The workflow service module 306 of the control plane 140 of the data analysis system 110 receives packaged workflow files (i.e., PEAs) issued from the client device 160 by the workflow management module 300. The workflow service module 306 of the workflow management module 300 manages the storage and execution of the packaged workflow files. The workflow management module 300 receives packaged workflow files from the client device 160 and stores those workflow files in cloud storage 310. The workflow management module 300 can also convert workflow files between different types. In some embodiments, cloud storage 310 is the user's private cloud storage from which the data analysis system 110 interfaces. For example, the user may establish cloud storage 310 using a third-party system, or operate their own storage for packaged workflow files. Alternatively, cloud storage 310 may be storage within the data analysis system 110 used by the user to store packaged workflow files.

[0041] The workflow management module 300 stores packaged workflow files in cloud storage 310 for later execution by the execution engine in the execution engine wrapper container 330. Through the UI generated by the UI module 314, users can browse their workspace in the cloud execution ecosystem and, in conjunction with the scheduling module 308, schedule published workflows (associated with packaged workflow files) for future execution. The workflow management module 300 may also store packaged workflow files in association with the execution time when the corresponding workflow should be executed by the workflow engine 320.

[0042] In some embodiments, the execution service 312 of the data analysis system 110 uses a message queue 360 ​​to coordinate the execution of a workflow. For example, when the control plane 140 receives a request from a client device 160 to schedule a PEA, the execution service 312 of the workflow management module 300 may issue a workflow job message to the message queue 360, which is listened to by an instance of the data plane 150 for the user. The workflow job message includes an identifier for the PEA and job metadata. Using the job metadata, the execution service 312 identifies the appropriate message topic (queue) for the job and queues the job request (message) in the message queue 360, which conforms to a predefined message schema (contract). In alternative embodiments, the user may have the option to execute the workflow immediately instead of having to schedule the workflow execution via the scheduling module 308.

[0043] When the data plane 150 has resources available to execute a workflow, the execution service 332 within the execution host 334 of the data plane 150 joins the appropriate message topic (queue) on the message queue 360 ​​and listens for jobs to execute. The execution service 332 picks up pending workflow job requests from the topic and initializes the workflow jobs. The execution service 332 creates an instance of the execution engine wrapper container 330, which is a temporary container. The execution service 332 passes job configuration details, metadata, runtime settings, etc., to the execution engine wrapper container 330.

[0044] The execution engine wrapper container 330 performs pre-workflow and post-workflow staging tasks. To execute the workflows within the packaged workflow files, the execution engine wrapper container 330 receives the packaged workflow files either from the workflow management module 300 (e.g., via an API request) or directly from the cloud storage 310 (after obtaining an access token to the cloud storage 310 from the platform API 302). The execution engine wrapper container 330 may directly retrieve the packaged workflow files from the cloud storage 310 using the workflow file identifier, or it may use the platform API 302 of the workflow management module 300 to cause the workflow management module 300 to retrieve the workflow files from the cloud storage 310 for the execution engine wrapper container 330. In some embodiments, the message queue 360 ​​manages workflow job messages using the Apache Kafka queuing system.

[0045] In response to obtaining the PEA, the execution engine wrapper container 330 initializes the execution engine and runs the job to generate results. The execution engine wrapper container 330 posts the job results to the cloud storage 310 via the platform API 302. The data plane 150 listens for the job status and, upon completion, returns this status to the control plane 140 (e.g., job succeeded or job failed).

[0046] Figure 4 is an interaction diagram illustrating exemplary data flows for executing a workflow packaged as a PEA from client device 160, according to several embodiments. Alternative embodiments may include more, fewer, or different steps, and the steps may be performed in a different order or by different components or systems than those shown in Figure 4.

[0047] The workflow management module 300 receives a packaged workflow file 400 from the client device 160. The workflow management module 300 may receive a baseline workflow file or a packaged workflow file from the client device 160. The workflow management module 300 stores the workflow file 400 in cloud storage 310, for example, via the workflow service module 306. The workflow management module 300 issues a workflow job message 410 to the message queue 360. The workflow job message 410 includes a workflow file identifier that identifies the PEA of the packaged workflow file in cloud storage 310.

[0048] When the data plane 150 has resources available to execute a workflow, the execution service 332 of the execution host 334 of the data plane 150 consumes messages for job requests 415. The execution service 332 creates a temporary execution engine wrapper container 330 420. The execution engine wrapper container 330 extracts the workflow file identifier from the workflow job message 410 and uses the workflow file identifier and access token to retrieve the workflow file 400 from cloud storage 310 440. Cloud storage 310 provides the workflow file 400 to the execution engine wrapper container 330. In some embodiments, instead of the workflow engine wrapper 330 directly requesting the workflow file from cloud storage 310, the workflow management module 300 uses its API to have the workflow management module 300 retrieve the workflow file from cloud storage 310 on behalf of the workflow engine wrapper 330.

[0049] The execution engine wrapper container 330 prepares the workflow file for execution by the workflow engine (for example, workflow steps may be separated from dependency files) and has the execution engine execute the workflow file 450. The execution engine wrapper container 330 exposes the results of the executed workflow by storing the results 460 in cloud storage 310. The workflow management module 300 can retrieve the results from cloud storage 310 460 and provide the results to the client device 160 470. Furthermore, the execution service 332 deletes the temporary container after the job is completed 465.

[0050] 2. Portable Execution Artifacts (PEAs) and Workflow Services Figure 5 shows exemplary data flows of a workflow service module in several embodiments. As mentioned above, the packaged workflow file is a PEA and can be configured to store a baseline workflow file and dependency files used when the workflow is executed. In one example, the workflow is a data transformation graph represented as an XML or JSON file. The dependencies of the PEA may include embedded datasets, DBM files, macros, plugins, images, and any other files that the workflow may reference. Users can package workflows and their dependencies created on an application (e.g., a desktop application) and expose these artifacts to the cloud execution ecosystem.

[0051] In one embodiment, the cloud execution ecosystem supports two types of workflow files: one a desktop version and the other a cloud version. The desktop version is a workflow packaged as a PEA stored in a cloud storage account 310, as illustrated in conjunction with Figure 3. The cloud version of the workflow is executed based on a JSON representation, rather than executing the packaged workflow. The desktop version of the workflow (i.e., the packaged workflow file) is executable by a first engine type, while the cloud version of the workflow is executable by a second engine type.

[0052] In one embodiment, the workflow service module 306 parses the packaged workflow file and converts the primary workflow XML definition within the package into a workflow JSON representation. As illustrated in conjunction with Figure 3, the workflow service module 306 stores the packaged workflow file in the user's cloud storage 310 (e.g., using a virtual file system 316) and may link the workflow objects (i.e., the JSON representation of the workflow) to the original binary packaged file for later execution. In one embodiment, the cloud version of the workflow is stored in the workflow database 318 of the control plane 140, and the desktop version of the workflow (i.e., PEA) is stored and abstracted in the user's cloud storage account.

[0053] In this way, the execution of existing cloud-version workflows can be integrated into the cloud execution ecosystem. Furthermore, users can view and schedule both types of workflows for execution from their workspace. For example, in the workspace UI, a user may have access to a dropdown element that allows them to select a workflow for execution (either desktop or cloud version) and a vendor for the cloud platform. In one embodiment, in response to a user initiating a request to execute the selected workflow, the scheduling module 308 may determine whether the workflow for execution is a desktop or cloud version and whether the cluster computing resources in the user's workspace support the appropriate type of execution engine. The scheduling module 308 may provide job metadata, as illustrated in conjunction with Figure 3.

[0054] During job execution, the execution engine wrapper container 330 uses job metadata to locate, download, and extract the PEA from cloud storage 310. Processing the PEA in this way supports the execution of workflows in many different types of remote execution environments. Therefore, the workflow service module 306 can function as an API endpoint to retrieve packaged workflow files from cloud storage 310 and provide the packaged workflow files from cloud storage 310 to other execution systems. For example, the execution system may be a user's remote environment managed by the data analysis system 110 (i.e., an instance of the data plane 150), or a user's remote environment managed by a third-party system that is a separate entity from the data analysis system 110. In these cases, the execution engine wrapper container 330 of the data plane 150 can use the workflow service module 306's API to retrieve the packaged workflow files from cloud storage 310 for execution.

[0055] In this way, workflows run on the private data handling (PDH) capabilities of the cloud execution ecosystem, and data access and engine execution (workflow execution) are isolated to the user's cloud account. The introduction of desktop and cloud versions of workflow types allows users to schedule both the cloud and desktop versions of workflows from the same workspace view in the data analysis system 110.

[0056] Figure 6 is an interaction diagram showing another exemplary data flow for executing a workflow file from client device 160, according to several embodiments. Alternative embodiments may include more, fewer, or different steps, and the steps may be performed in a different order or by different components or systems than those shown in Figure 6.

[0057] The workflow management module 300 receives packaged workflow files (PEAs) uploaded from the client device 160. In one example, the workflow management module 300 that receives the PEAs is the workflow service module 306. The workflow management module 300 (workflow service module 306) stores the workflow files in cloud storage 310. The cloud storage 310 account may be a user-owned storage account in the cloud.

[0058] The workflow management module 300 receives requests from client devices 160 to schedule the execution of a workflow. The workflow management module 300 schedules jobs to execute the packaged workflow. For example, jobs may be scheduled using message queue 360, as illustrated in conjunction with Figure 3.

[0059] When the data plane 150 has resources available to execute a workflow, the host execution service 332 may create an execution engine wrapper container 330, which is a temporary container 630. The execution engine wrapper container 330 extracts a workflow file identifier from the job metadata and submits a workflow file API request to the workflow management module 300 to retrieve the packaged workflow file 640. The API request is a request to an API provided by the workflow service module 306 of the workflow management module 300 to allow different execution engines to retrieve and store the workflow file. In response to receiving the API request, the workflow management module 330 retrieves the workflow file from the cloud storage 310 650. The workflow management module then provides the workflow file to the execution engine wrapper container 330.

[0060] The execution engine wrapper container 330 prepares the workflow for execution by the execution engine 660 (for example, workflow steps may be separated from dependency files), and causes the execution engine to execute the workflow file 670. The execution engine wrapper container 330 publishes the results of the executed workflow to the workflow management module 300 for storage in cloud storage 310 or other target system 680, and the execution service 322 may delete the execution engine wrapper container 330 685. The workflow management module 300 may also provide the results to the client device 160 690.

[0061] 3. Execution engine wrapper container Figure 7 shows exemplary data flows through an execution engine wrapper container in several embodiments. In one embodiment, the execution engine wrapper container 330 includes an engine wrapper 345 which includes a logger system 336, a stager system 340, and a worker system 350. The execution engine wrapper container 330 also includes an execution engine 320 and a data connection manager 325, although the execution engine wrapper container 330 may include more, fewer, or different components.

[0062] In one embodiment, the execution engine wrapper container 330 is a containerized asynchronous job execution agent that defines the specifications between a data processing engine and an engine wrapper layer 345 that provide the auxiliary engine capabilities required for execution. Specifically, the container 330 is unfolded from a container image that includes an executable file and its package dependencies for running the container on a host system. The container image is a software artifact. For example, the container image for a particular execution engine wrapper container 330 may be associated with a set of parameters indicating which version of the execution engine is included in the packaged software. The container image may also be associated with a set of resource parameters that specify hardware configurations, such as how much memory on the compute host should be allocated to the container instance, and how many CPU units on the compute host can be allocated to the container.

[0063] The components of the execution engine wrapper container 330 receive PEA as input and receive the data necessary within the data source 120 to execute the workflow. In one embodiment, the execution engine wrapper container 330 is cross-platform, supporting various operating systems such as Windows or Linux. Since the components of the execution engine container 330 are encapsulated within the container, the cloud execution ecosystem of the data analysis system 110 can execute workflows packaged as PEA close to the data source 120, as long as the cloud computing resources support the container.

[0064] As described above in conjunction with Figure 3, in response to receiving a request to execute a workflow, the host execution service 332 may start the execution engine wrapper container 330. The engine wrapper 345 posts the job execution request to the worker system 350. The worker system 350 calls the stager system 340 to retrieve the packaged workflow file (i.e., PEA) from the workflow management system 300 and prepare the corresponding workflow for execution by the execution engine 320. The stager system 340 may extract the primary workflow file and the dependency files from the PEA to the local file system. The worker system 350 then calls the execution engine 320 for the workflow job.

[0065] The worker system 350 retrieves a workflow file and interfaces with the execution engine 320 to execute the workflow described in the workflow file. The worker system 350 may receive a baseline workflow file from the stager system 340 and provide the execution engine 320 with the workflow described in the workflow file for execution. The worker system 350 may invoke the execution engine 320 within the data plane 150 (for example, via a command line instruction) for the execution of the workflow file.

[0066] The execution engine 320 executes the workflow within the container. In one embodiment, the execution of a workflow job requires the user to access one or more data sources 120 (e.g., customer data). Therefore, in one embodiment, the execution engine 320 requests credentials from the data connection manager 325 when credentials are required for the data source 120. The data connection manager 325 obtains the credentials from the workflow management module 300, for example, in the form of a connection string. Depending on the host operating system (OS), the container 330 shown in Figure 7 shows one exemplary Linux-based execution engine 320A and another exemplary Windows-based execution engine 320B. However, it is understood that in other embodiments, the execution engine wrapper container 330 may support other types of execution engines.

[0067] The worker system 350 receives the results of the executed workflow from the execution engine 320 and stores the results in cloud storage 310. More specifically, the worker system 350 sends the results of the executed workflow to the workflow management module 300, which is then stored in cloud storage 310. In some embodiments, the workflow management module 300 receives the results from the execution system 350 and stores the results in cloud storage 310 or another target storage system so that the requesting user can retrieve the results as needed.

[0068] Figure 8 is an interaction diagram showing another exemplary data flow for executing a workflow file from client device 160 in several embodiments. Alternative embodiments may include more, fewer, or different steps, and the steps may be performed in a different order or by different components or systems than those shown in Figure 8.

[0069] When the data plane 150 has resources available to execute the workflow, the worker system 350 receives a job request from the engine wrapper 345 800, calls the stager system 340 810 to retrieve and decompress the workflow file associated with the job request 820. The stager system 340 passes the workflow file 830 to the worker system 350.

[0070] The worker system 350 prepares the workflow file and the execution engine 320 for execution. The worker system 350 may also start the execution engine 320 if it is not already running. In some embodiments, the worker system 350 identifies the type of execution engine 320 to use for the workflow file and starts or uses the identified type of execution engine 320 to execute the workflow. The worker system 350 may provide the uncompressed workflow file 830 to the execution engine 320.

[0071] The execution engine 320 executes the workflow file 840 and provides the result 850 to the worker system 350. The worker system 350 may provide the result 850 to the workflow management module 300 (for example, via the virtual file system 316), and the workflow management module 300 may publish the result to the cloud storage 310.

[0072] Figure 9 is a high-level block diagram showing a functional diagram of a typical computer system 900 for use as the data analysis system 110 of Figure 1 according to one embodiment.

[0073] The illustrated computer system includes at least one processor 902 coupled to a chipset 904. The processor 902 may include multiple processor cores on the same die. The chipset 904 includes a memory controller hub 920 and an input / output (I / O) controller hub 922. Memory 906 and a graphics adapter 912 are coupled to the memory controller hub 920, and a display 918 is coupled to the graphics adapter 912. A storage device 908, a keyboard 910, a pointing device 914, and a network adapter 916 are coupled to the I / O controller hub 922. In some other embodiments, the computer system 900 may have additional components, fewer components, or different components, and the components may be coupled in different ways. For example, an embodiment of the computer system 900 may lack a display and / or a keyboard. In addition, in some embodiments, the computer system 900 may be instantiated as a rack-mount blade server or as a cloud server instance.

[0074] Memory 906 holds instructions and data used by the processor 902. In some embodiments, memory 906 is random-access memory. Storage device 908 is a non-temporary computer-readable storage medium. Storage device 908 can be an HDD, SSD, or other type of non-temporary computer-readable storage medium. Data processed and analyzed by the data analysis system 110 can be stored in memory 906 and / or storage device 908.

[0075] The pointing device 914 may be a mouse, trackball, or other type of pointing device, used in conjunction with the keyboard 910 to input data to the computer 900. The graphics adapter 912 displays images and other information on the display 918. In some embodiments, the display 918 includes touchscreen functionality for receiving user input and selections. The network adapter 916 connects the computer system 900 to the network 130.

[0076] The computer system 900 is adapted to run computer modules for providing the functions described herein. As used herein, the term “module” refers to computer program instructions and other logic for providing a particular function. Modules can be implemented in hardware, firmware, and / or software. Modules can include one or more processes and / or be provided by only a portion of a process. Modules are typically stored in a storage device 908, loaded into memory 906, and executed by a processor 902. Additional considerations

[0077] Some parts of the above description illustrate the functionality in terms of algorithms and symbolic representations of information manipulation. These descriptions and representations of algorithms are means used by those skilled in the art to most effectively communicate the nature of their work to others skilled in the art. These operations, while described functionally or logically, are understood to be implemented by computer programs. Furthermore, without loss of generality, it may be convenient to refer to arrangements of these operations as modules or by functional names.

[0078] Unless otherwise explicitly stated as is evident from the above discussion, any discussion throughout this explanation using terms such as “processing,” “computing,” “calculating,” “determining,” or “displaying” refers to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data represented as physical (electronic) quantities in the memory or registers of a computer system, or other information storage, transmission, or display devices.

[0079] The specific embodiments described herein include process steps and instructions described in the form of algorithms. Note that the process steps and instructions of the embodiments can be implemented in software, firmware, or hardware, and if embodied in software, they can be downloaded and reside on different platforms used by real-time network operating systems, from which they can be operated.

[0080] Finally, the language used in this specification has been selected primarily for readability and educational purposes, and not to define or limit the subject matter of the invention. Accordingly, the disclosure of embodiments is intended to be illustrative but not limiting.

Claims

1. The control plane of a data analysis system receives a request from a user's client device of the data analysis system to execute a workflow from a packaged workflow file, wherein the packaged workflow file includes at least a workflow for execution, and the packaged workflow file is stored in a cloud storage account. A step of publishing a message to a message queue, wherein the message requests job metadata that identifies a job for executing the workflow and the packaged workflow file, The steps include: creating an execution engine wrapper container in the data plane by an execution service in the data plane in response to consuming the aforementioned message, wherein the execution engine wrapper container houses the execution engine; The execution engine wrapper container obtains the packaged workflow file for execution using the job metadata, To generate the results of the aforementioned workflow, the steps include: executing the workflow file using the execution engine; In response to the determination that the job has been completed, the execution service deletes the execution engine wrapper container. The steps include: sending the result of executing the workflow file to the client device; Methods that include...

2. The method according to claim 1, wherein the packaged workflow file includes one or more dependency files for the workflow.

3. The method according to claim 1, wherein the received workflow file is represented as XML or JSON.

4. The job metadata of the job message includes the workflow file identifier of the packaged workflow file, The method according to claim 1, wherein the packaged workflow file is retrieved from the cloud storage using the workflow file identifier.

5. The control plane includes the step of obtaining the results from the cloud storage, The control plane transmits the result to the client device. The method according to claim 1, further comprising:

6. The method according to claim 1, wherein the control plane of the data analysis system is located in a different VPC from the virtual private cloud (VPC) of the data plane associated with the user.

7. The method according to claim 1, wherein the cloud storage account is a cloud storage system owned by an account associated with the user.

8. A non-temporary computer-readable medium, which, when executed by a processor, the processor, The control plane of a data analysis system receives a request from a user's client device of the data analysis system to execute a workflow from a packaged workflow file, wherein the packaged workflow file includes at least a workflow for execution, and the packaged workflow file is stored in a cloud storage account. A step of publishing a message to a message queue, wherein the message requests job metadata that identifies a job for executing the workflow and the packaged workflow file, In response to consuming the aforementioned message, the execution service in the data plane creates an execution engine wrapper container in the data plane, wherein the execution engine wrapper container houses the execution engine. The execution engine wrapper container obtains the packaged workflow file for execution using the job metadata, To generate the results of the aforementioned workflow, the steps include: executing the workflow file using the execution engine; In response to the determination that the job has been completed, the execution service deletes the execution engine wrapper container. The steps include: sending the result of executing the workflow file to the client device; A computer-readable medium that stores instructions for performing actions that include [a certain action].

9. The computer-readable medium according to claim 8, wherein the packaged workflow file includes one or more dependency files for the workflow.

10. The received workflow file is represented as XML or JSON in the computer-readable medium according to claim 8.

11. The job metadata of the job message includes the workflow file identifier of the packaged workflow file, The computer-readable medium according to claim 8, wherein the packaged workflow file is retrieved from the cloud storage using the workflow file identifier.

12. The aforementioned operation is, The control plane includes the step of obtaining the results from the cloud storage, The control plane transmits the result to the client device. The computer-readable medium according to claim 8, further comprising:

13. The computer-readable medium according to claim 8, wherein the control plane of the data analysis system is located in a different VPC from the virtual private cloud (VPC) of the data plane associated with the user.

14. The computer-readable medium according to claim 8, wherein the cloud storage account is a cloud storage system owned by an account associated with the user.

15. Processor and A non-temporary computer-readable medium, which, when executed by the processor, the processor, The control plane of a data analysis system receives a request from a user's client device of the data analysis system to execute a workflow from a packaged workflow file, wherein the packaged workflow file includes at least a workflow for execution, and the packaged workflow file is stored in a cloud storage account. A step of publishing a message to a message queue, wherein the message requests job metadata that identifies a job for executing the workflow and the packaged workflow file, The steps include: creating an execution engine wrapper container in the data plane by an execution service in the data plane in response to consuming the aforementioned message, wherein the execution engine wrapper container houses the execution engine; The execution engine wrapper container obtains the packaged workflow file for execution using the job metadata, To generate the results of the aforementioned workflow, the steps include: executing the workflow file using the execution engine; In response to the determination that the job has been completed, the execution service deletes the execution engine wrapper container. The steps include: sending the result of executing the workflow file to the client device; A non-temporary computer-readable medium that stores instructions to perform an action including and A system that includes these features.

16. The system according to claim 15, wherein the packaged workflow file includes one or more dependency files for the workflow.

17. The system according to claim 15, wherein the received workflow file is represented as XML or JSON.

18. The job metadata of the job message includes the workflow file identifier of the packaged workflow file, The system according to claim 15, wherein the packaged workflow file is retrieved from the cloud storage using the workflow file identifier.

19. The aforementioned operation is, The control plane includes the step of obtaining the results from the cloud storage, The control plane transmits the result to the client device. The system according to claim 15, further comprising:

20. The system according to claim 15, wherein the control plane of the data analysis system is located in a different VPC from the virtual private cloud (VPC) of the data plane associated with the user.

21. The control plane of a data analysis system includes the step of receiving a packaged workflow file from a client device of a user of the data analysis system, wherein the packaged workflow file describes at least the workflow to be executed. The control plane includes the steps of storing the packaged workflow file in a cloud storage account, The control plane receives an API request for the packaged workflow file from an execution engine wrapper container operating on the data plane, wherein the API request includes an identifier for the packaged workflow file. The control plane includes the steps of retrieving the packaged workflow file from the cloud storage account based on the identifier, The control plane sends the packaged workflow file to the execution engine wrapper container running on the data plane, The control plane receives the result of the execution of the workflow by the data plane, The control plane performs the steps of sending the results of the workflow file to the cloud storage account for storage. Methods that include...

22. The steps include extracting the workflow from the aforementioned packaged workflow file, The steps include converting the extracted workflow into a second workflow and storing the second workflow in the workflow database within the control plane. The method according to claim 21, further comprising:

23. The method according to claim 22, wherein the extracted workflow is represented in XML, and the second workflow is represented as JSON.

24. A step of generating a user interface (UI) on the client device, wherein the UI displays one or more first workflows of a first representation stored in the cloud storage account, and one or more second workflows of a second representation stored in the workflow database. The steps include receiving a request from the user to select a workflow and schedule the execution of the selected workflow, and The method according to claim 22, further comprising:

25. The method according to claim 21, wherein the packaged workflow file includes one or more dependency files for the workflow, and the one or more dependency files include at least one or a combination of a DBM file, an executable file, an image, a plugin, or a macro.

26. The method according to claim 21, wherein the execution engine wrapper container operates on cluster computing resources operated by a third-party system different from the entities of the data analysis system.

27. A non-temporary computer-readable medium, which, when executed by a processor, the processor, The control plane of a data analysis system includes the step of receiving a packaged workflow file from a client device of a user of the data analysis system, wherein the packaged workflow file describes at least the workflow to be executed. The control plane includes the steps of storing the packaged workflow file in a cloud storage account, The control plane receives an API request for the packaged workflow file from an execution engine wrapper container operating on the data plane, wherein the API request includes an identifier for the packaged workflow file. The control plane includes the steps of retrieving the packaged workflow file from the cloud storage account based on the identifier, The control plane sends the packaged workflow file to the execution engine wrapper container running on the data plane, The control plane receives the result of the execution of the workflow by the data plane, The control plane performs the steps of sending the results of the workflow file to the cloud storage account for storage. A computer-readable medium that stores instructions for performing actions that include [a certain action].

28. The aforementioned operation is, The steps include extracting the workflow from the aforementioned packaged workflow file, The steps include converting the extracted workflow into a second workflow and storing the second workflow in the workflow database within the control plane. The computer-readable medium according to claim 27, further comprising:

29. The computer-readable medium according to claim 28, wherein the extracted workflow is represented in XML and the second workflow is represented in JSON.

30. The aforementioned operation is, A step of generating a user interface (UI) on the client device, wherein the UI displays one or more first workflows of a first representation stored in the cloud storage account, and one or more second workflows of a second representation stored in the workflow database. The steps include receiving a request from the user to select a workflow and schedule the execution of the selected workflow, and The computer-readable medium according to claim 28, further comprising:

31. The computer-readable medium according to claim 27, wherein the packaged workflow file includes one or more dependency files for the workflow, and the one or more dependency files include at least one or a combination of a DBM file, an executable file, an image, a plugin, or a macro.

32. The computer-readable medium according to claim 27, wherein the execution engine wrapper container operates on cluster computing resources operated by a third-party system different from the entities of the data analysis system.

33. Processor and A non-temporary computer-readable medium, which, when executed by the processor, the processor, The control plane of a data analysis system includes the step of receiving a packaged workflow file from a client device of a user of the data analysis system, wherein the packaged workflow file describes at least the workflow to be executed. The control plane includes the steps of storing the packaged workflow file in a cloud storage account, The control plane receives an API request for the packaged workflow file from an execution engine wrapper container operating on the data plane, wherein the API request includes an identifier for the packaged workflow file. The control plane includes the steps of retrieving the packaged workflow file from the cloud storage account based on the identifier, The control plane sends the packaged workflow file to the execution engine wrapper container running on the data plane, The control plane receives the result of the execution of the workflow by the data plane, The control plane performs the steps of sending the results of the workflow file to the cloud storage account for storage. A computer-readable medium that stores instructions for performing an action including and A system that includes these features.

34. The aforementioned operation is, The steps include extracting the workflow from the aforementioned packaged workflow file, The steps include converting the extracted workflow into a second workflow and storing the second workflow in the workflow database within the control plane. The system according to claim 33, further comprising:

35. The system according to claim 34, wherein the extracted workflow is represented in XML, and the second workflow is represented in JSON.

36. The aforementioned operation is, A step of generating a user interface (UI) on the client device, wherein the UI displays one or more first workflows of a first representation stored in the cloud storage account, and one or more second workflows of a second representation stored in the workflow database. The steps include receiving a request from the user to select a workflow and schedule the execution of the selected workflow, and The system according to claim 34, further comprising:

37. The system according to claim 33, wherein the packaged workflow file includes one or more dependency files for the workflow, and the one or more dependency files include at least one or a combination of a DBM file, an executable file, an image, a plugin, or a macro.

38. The system according to claim 33, wherein the execution engine wrapper container operates on cluster computing resources operated by a third-party system different from the entities of the data analysis system.

39. In the data plane execution service, the steps include receiving a request to execute a workflow within a packaged workflow file, A step of creating an execution engine wrapper container in the data plane based on a container image, wherein the execution engine wrapper container includes at least a stager system, a worker system, and an execution engine. The steps include: obtaining the packaged workflow file using the stager system of the execution engine wrapper container; The stager system performs the steps of decompressing the components of the packaged workflow file into the local file system and providing the job request to the worker system, The steps include: executing the workflow of the packaged workflow file by the execution engine in response to the worker system's start; The worker system includes the step of receiving the result of the execution of the workflow by the execution engine, The worker system provides the results of the execution to the control plane. Methods that include...

40. The steps include accessing a container image database containing multiple container images, The steps include identifying the version of the execution engine for executing the packaged workflow file for the aforementioned request, The steps include selecting the container image having a version of the execution engine that matches the identified version for the request, The steps include: executing the workflow of the packaged workflow file using the identified version of the execution engine; The method according to claim 39, further comprising:

41. The method according to claim 40, wherein one or more container images are configured to operate on a first operating system, and one or more other container images are configured to operate on a second operating system.

42. The execution engine wrapper container includes a data connection manager, and the method is The data connection manager includes the step of receiving a request from the execution engine for credentials to one or more data sources required to execute the workflow, The steps of providing the aforementioned credentials to the execution engine and The method according to claim 39, further comprising:

43. The method according to claim 39, further comprising the step of receiving the packaged workflow file from the control plane of the data analysis system.

44. The logger system of the execution engine wrapper container logs metadata related to the execution of the workflow, The steps include: publishing the metadata to the message bus for consumption by a service in the control plane; The method according to claim 39, further comprising:

45. A non-temporary computer-readable medium, which, when executed by a processor, the processor, In the data plane execution service, the steps include receiving a request to execute a workflow within a packaged workflow file, A step of creating an execution engine wrapper container in the data plane based on a container image, wherein the execution engine wrapper container includes at least a stager system, a worker system, and an execution engine. The steps include: obtaining the packaged workflow file using the stager system of the execution engine wrapper container; The stager system performs the steps of decompressing the components of the packaged workflow file into the local file system and providing the job request to the worker system, The steps include: executing the workflow of the packaged workflow file by the execution engine in response to the worker system's start; The worker system includes the step of receiving the result of the execution of the workflow by the execution engine, The worker system provides the results of the execution to the control plane. A computer-readable medium that stores instructions for performing actions that include [a certain action].

46. The aforementioned operation is, The steps include accessing a container image database containing multiple container images, The steps include identifying the version of the execution engine for executing the packaged workflow file for the aforementioned request, The steps include selecting the container image having a version of the execution engine that matches the identified version for the request, The steps include: executing the workflow of the packaged workflow file using the identified version of the execution engine; The computer-readable medium according to claim 45, further comprising:

47. The computer-readable medium according to claim 46, wherein one or more container images are configured to operate on a first operating system, and one or more other container images are configured to operate on a second operating system.

48. The execution engine wrapper container includes a data connection manager, and the operation is as follows: The data connection manager includes the step of receiving a request from the execution engine for credentials to one or more data sources required to execute the workflow, The steps of providing the aforementioned credentials to the execution engine and The computer-readable medium according to claim 45, further comprising:

49. The aforementioned operation is, The computer-readable medium according to claim 45, further comprising the step of receiving the packaged workflow file from the control plane of the data analysis system.

50. The aforementioned operation is, The logger system of the execution engine wrapper container logs metadata related to the execution of the workflow, The steps include: publishing the metadata to the message bus for consumption by a service in the control plane; The computer-readable medium according to claim 45, further comprising:

51. Processor and A non-temporary computer-readable medium, which, when executed by a processor, the processor, In the data plane execution service, the steps include receiving a request to execute a workflow within a packaged workflow file, A step of creating an execution engine wrapper container in the data plane based on a container image, wherein the execution engine wrapper container includes at least a stager system, a worker system, and an execution engine. The steps include: obtaining the packaged workflow file using the stager system of the execution engine wrapper container; The stager system performs the steps of decompressing the components of the packaged workflow file into the local file system and providing the job request to the worker system, The steps include: executing the workflow of the packaged workflow file by the execution engine in response to the worker system's start; The worker system includes the step of receiving the result of the execution of the workflow by the execution engine, The worker system provides the results of the execution to the control plane. A non-temporary computer-readable medium that stores instructions for performing an action including and A system that includes these features.

52. The aforementioned operation is, The steps include accessing a container image database containing multiple container images, The steps include identifying the version of the execution engine for executing the packaged workflow file for the aforementioned request, The steps include selecting the container image having a version of the execution engine that matches the identified version for the request, The steps include: executing the workflow of the packaged workflow file using the identified version of the execution engine; The system according to claim 51, further comprising:

53. The system according to claim 52, wherein one or more container images are configured to operate on a first operating system, and one or more other container images are configured to operate on a second operating system.

54. The execution engine wrapper container includes a data connection manager, and the operation is as follows: The data connection manager includes the step of receiving a request from the execution engine for credentials to one or more data sources required to execute the workflow, The steps of providing the aforementioned credentials to the execution engine and The system according to claim 51, further comprising:

55. The aforementioned operation is, The system according to claim 51, further comprising the step of receiving the packaged workflow file from the control plane of the data analysis system.

56. The aforementioned operation is The logger system of the execution engine wrapper container logs metadata related to the execution of the workflow, The steps include: publishing the metadata to the message bus for consumption by a service in the control plane; The system according to claim 51, further comprising:

Images