Dataflow-driven workflow server agnostic distributed scheduling system

Abstract

A data flow-driven workflow server-insensitive distributed scheduling system includes: a directed acyclic graph resolver and a global scheduler set on the master node, and a data flow local scheduler and a local storage unit set on each worker node. This invention improves end-to-end latency and throughput by designing a data flow-based scheduler and making full use of data locality, thereby supporting out-of-order execution and a data flow-based workflow server.

Description

Technical Field

[0001] This invention relates to a technology in the field of distributed computing, specifically a data flow-driven workflow server-insensitive distributed scheduling system. Background Technology

[0002] Currently, most cloud platforms deploy workflows based on a control flow-driven architecture. This architecture reduces the parallelism of functions, introduces additional scheduling overhead, and thus increases end-to-end latency. At the same time, cloud platforms use remote storage for function data interaction. Even two functions on the same machine still have to rely on remote storage for data interaction, which does not make full use of data locality, resulting in relatively large data access latency and further increasing end-to-end latency. Summary of the Invention

[0003] This invention addresses the limitations of existing server-aware computing architectures based on control flow, which suffer from high scheduling overhead and data access latency. It proposes a data flow-driven workflow server-aware distributed scheduling system. By designing a data flow-based scheduler and fully utilizing data locality, it improves end-to-end latency and throughput, thereby supporting out-of-order execution and workflow servers based on data flow architecture.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to a dataflow-based distributed scheduling system that is unaware of data flow, comprising: a directed acyclic graph (DAG) parser and a global scheduler on the master node, and a dataflow-based local scheduler and a local storage unit on each worker node. The DAG parser generates a DAG based on a user configuration file and, based on resource and load balancing principles in the cluster, segments the DAG into subgraphs and outputs them to different worker nodes, generating metadata for local workflows and storing it in a device deploying remote storage and a data directory service unit. Each node's local scheduler, upon receiving a trigger request, retrieves the aggregated workflow metadata from the remote storage, iterates through the list of startup functions, and executes the functions sequentially while simultaneously running subsequent node functions that have data dependencies on the current function. After any function completes execution, it continues running the remaining unfinished functions, thus achieving dataflow-driven operation.

[0006] The metadata of the local workflow includes: the data key, the data size, the data location information, and the access frequency of the data location information.

[0007] The metadata of the aggregated workflow includes: the IP address of the working node where each function is located, the input and output of the function, the starting function of the workflow, and a list of startup functions of the predecessor function and its successor function.

[0008] The data stream local scheduler executes functions in an out-of-order manner, meaning that a function can still be started and run even when its predecessor function is still running.

[0009] The global scheduler includes a parser module and a graph cutting module. The parser module parses the user configuration file to generate a directed acyclic graph. The graph cutting module cuts the directed acyclic graph into subgraphs according to the cluster's resources. The parser module stores the worker node IP, function input and output, and a list of startup functions including the workflow's starting function and the successor function of the predecessor function as workflow metadata in remote storage.

[0010] The remote storage mentioned herein may be deployed, but is not limited to, on a device running CouchDB.

[0011] The local scheduler includes a gateway module and a data flow scheduling module. The gateway module receives the user's trigger request, parses it to obtain a JSON file and a workflow, and the data flow scheduling module executes the function in the local storage unit or requests the workflow's metadata from the remote storage according to the workflow.

[0012] Preferably, when a function with a predecessor function reaches a point where the data required for its computation has not yet been generated, the data flow scheduling module sets the function to a waiting state and simultaneously starts running the functions in the directed acyclic graph and their successor node functions without waiting for the list of started functions to finish executing; when the predecessor function or any of the current function finishes running, the data flow scheduling module sets the function to continue starting and running the remaining unfinished functions, thereby achieving a balance between resource utilization and function parallelism, specifically implemented in the following ways:

[0013] ① The data flow scheduling module triggers the corresponding function and starts the container according to the data dependency relationship, so that the function runs in the container. It also checks whether the function is located in the local storage unit by calling the get interface: if it is located in the local storage unit, the container directly copies and executes it from the local storage unit; otherwise, it queries the data directory service unit through the local storage unit and copies the data from the remote storage, and then copies and executes it from the local storage unit through the container.

[0014] ② After calling the put interface and saving the data obtained from the execution to the local storage unit, the local storage unit outputs the metadata information corresponding to the data to the remote storage.

[0015] The data location information in the metadata includes the IP address of the worker node that owns the data and the current generation status of the data. In workflow server-side computation, there might be a node whose function A has many successor functions, distributed across different devices. Because these successor functions require the output of function A as input for the current worker node, node 1 must send the data to many nodes, potentially becoming a performance bottleneck. To alleviate this bottleneck, the data directory service maintains a data structure that records the access frequency of each piece of data. When a client queries the data directory service to retrieve the location information of a piece of data, the data directory responds with the IP address of the worker node with the lowest access frequency for that data.

[0016] The local storage unit and data directory service unit on each worker node constitute a hybrid distributed key-value storage system (DStore). The data directory service unit manages the metadata of workflow data and provides query services to facilitate clients in querying the location and size of data. The local storage unit, as a local key-value storage system, manages local data and writes the size of the data and the device IP address where the data is located to the data directory service.

[0017] Preferably, the key-value storage system decouples data and metadata, meaning that each worker node stores the data locally, while the metadata is stored on another device where the data catalog service is deployed. Simultaneously, to support efficient, data-driven workflow server-side computation, the data catalog service integrates a frequency access mechanism and an access frequency mechanism to prevent any single worker node from becoming a performance bottleneck.

[0018] Preferably, the data directory service unit optimizes the function execution order through an auto blocking (waking) mechanism. Specifically, when a process on a worker node queries the data directory service to retrieve metadata for certain data, and the data directory service does not yet contain that metadata, the data directory service automatically blocks the process to ensure successful function execution. When the metadata is available in the data directory, the process is automatically woken up. With this auto blocking (waking) mechanism, the present invention can successfully support out-of-order function execution, ensuring that all functions in a workflow execute correctly.

[0019] Preferably, the direct copying from the local storage unit means that when a function with data dependencies exists in the local storage unit, the data flow local scheduler directly reads the required data from the local storage unit.

[0020] Preferably, the direct copying from the local storage unit is further implemented using an intra-node pipeline method. Specifically, by setting up a data structure in the local storage unit to record the data generation progress, even if the data is still being copied from the container to the local storage unit, the container of the current working node can simultaneously copy the currently generated data from the local storage unit. That is, one container copies data from its memory to the local storage unit while another container copies and reads the data from the local storage unit to the container of the current working node.

[0021] Preferably, when two functions that depend on the data are assigned to different worker nodes, the function data interaction is achieved through inter-node pipeline replication. Specifically, when a container copies data from the container memory of the current worker node to the local storage unit, the local storage unit asynchronously outputs the metadata containing the current generation progress of the data to the data catalog service unit. When other worker nodes query the data catalog service unit and receive the metadata, they then copy the currently generated data from the local storage units of other worker nodes.

[0022] Preferably, the key-value storage system sets each piece of data to read-only, meaning that functions that need the data can only read it and cannot update it. When they want to update the data, they must use a different ID to call the Put interface.

[0023] Preferably, the key-value storage system achieves decentralized reception between different working nodes through a coordination mechanism data exchange (Decentralized Receriver-Driven). Specifically, in DStore, data transmission between nodes via the network only occurs in one scenario: a function on a device discovers that the required data is stored remotely, meaning there is a data dependency between functions on different devices. When a function on device 1 needs the output of its preceding function as input, it first checks if the data exists in its local storage. If not, it queries the data directory service to obtain the location information of the data. After obtaining the location information, the data directory service mechanism is no longer needed to coordinate data transmission. Device 1 can directly read the data from device 2, which has the data. Attached Figure Description

[0024] Figure 1 This is a flowchart of the present invention;

[0025] Figure 2 This is a schematic diagram of the system of the present invention;

[0026] Figure 3 This is a diagram illustrating the system scheduling effect;

[0027] Figure 4 This is a schematic diagram of a hybrid inter-node and intra-node pipeline in DStore.

[0028] Figure 5 A diagram illustrating the data structure maintained for DStore's data directory service;

[0029] Figure 6 This is a diagram illustrating the throughput of a DStore's get operation.

[0030] Figure 7 This is a schematic diagram illustrating the throughput of DStore's put operations;

[0031] Figure 8 and Figure 9 This is a schematic diagram illustrating the effect of an example. Detailed Implementation

[0032] like Figure 1 This embodiment relates to a two-layer seamless scheduling system for workflow servers based on a dataflow architecture. It includes: a master node with a global scheduler equipped with a directed acyclic graph (DAG) parser, and several worker nodes equipped with dataflow-based local schedulers. Specifically: the DAG parser generates a DAG based on the user's configuration file, then segments it into subgraphs and distributes them to different worker nodes based on resource and load balancing principles within the cluster; the global scheduler stores the aggregated workflow metadata in remote storage on the device running CouchDB; the local schedulers receive trigger requests from users, first retrieving the aggregated workflow metadata from the remote storage, then traversing the list of startup functions and running them sequentially, while simultaneously running subsequent node functions that have data dependencies on the current function, and continuing to run the remaining unfinished functions after any function completes its execution, thus achieving dataflow-driven operation.

[0033] This embodiment uses multiple applications as examples, including: 2 real-world applications and 4 scientific computing tasks. The real-world applications include: 1) an application widely used in the field of big data analysis to count the number of characters in a text; 2) a text processing application to convert a file to its desired format and store it in a database; the scientific computing tasks each contain dozens of functions.

[0034] The system environment in this embodiment is as follows: eight ECSg7.2 devices, each equipped with 8 cores, 32GB of memory, and a 100GB SSD. One device acts as the master node, deploying the global scheduler and data directory service; the other seven devices act as worker devices, receiving HTTP requests from users or trigger requests from their respective worker devices. Each worker device deploys a data flow-driven local scheduler and a local storage unit aware of data locality. To simulate different bandwidths, this embodiment uses Wondershaper to set different network bandwidths: 25MB / s, 50MB / s, 75MB / s, and 100MB / s, to demonstrate the experimental effects of this invention under different network bandwidths.

[0035] like Figure 2 As shown, the embodiment of the distributed scheduling system for workflow servers that supports out-of-order execution of functions based on a dataflow architecture includes: a global scheduler (GlobalScheduler) with a directed acyclic graph (DAG) parser as the master node, and several local schedulers (Dataflow-basedLocalSchedulers) as worker nodes. Specifically: the DAG parser generates a DAG based on the user configuration file, divides it into subgraphs based on resource and load balancing principles in the cluster, and distributes them to different worker nodes, generating workflow metadata and storing it in remote storage; the local scheduler retrieves the entire workflow metadata from the remote storage based on the trigger request, traverses the list of startup functions and runs the functions sequentially, while simultaneously running subsequent node functions that have data dependencies on the current function, and continues running the remaining unfinished functions after any function has completed its execution, thus achieving dataflow-driven execution.

[0036] like Figure 2 As shown, the user first uploads the workflow configuration file to the master node. The directed acyclic graph (DAG) parser on the master node parses the configuration file into a DAG. The global scheduler on the master node then segments the DAG into subgraphs and assigns them to different worker nodes. Workflow metadata (such as workflow entry points, function inputs and outputs, etc.) is stored in remote storage.

[0037] like Figure 2As shown, only function A is the entry point of this workflow. Functions A and B are on worker node 1, and function C is on node 2. When the local scheduler based on the data flow receives a trigger request from an event trigger, it first triggers the startup of functions A and B. Because function C is on device 2, the local scheduler on device 1 sends a trigger request to the local scheduler on device 2, and then function C is also started. Since function A is the startup node of the workflow, it will run until completion and store the data in the distributed cache. For function B, after executing for a period of time, it will be automatically blocked after querying the data catalog service and finding that the required data is unavailable, until the metadata is located in the data catalog service. When function B is awakened and finds that the data is in the local storage of node 1, it directly reads the data from the local storage into the container, executes until completion, stores the data, and synchronizes the metadata to the data catalog service. For function C, it is automatically blocked because the required data metadata is not shown in the data catalog service, and it queries the data catalog service. When function C is awakened and knows that the data is on remote worker node 1, it reads the data, copies it to its local storage, copies the data from its local storage to the container, and then executes the function. Finally, after all functions in the workflow have executed, the generated data is stored in the distributed key-value system.

[0038] like Figure 2 As shown, ABC is the predecessor function of D. In this system, even if ABC is still running, D can still be started and run, so the running time of D and the running time of ABC overlap.

[0039] like Figure 4 As shown, this is a hybrid pipeline optimization for distributed storage system integration, including intra-node pipelined data replication and inter-node pipelined data replication. Intra-node pipeline optimization provides low-latency data interaction between functions on the same node, while inter-node pipeline optimization provides efficient data exchange across nodes. When operations that generate data copy it from their local storage or the local storage of other nodes, the data may be incomplete. To support the replication of incomplete data, this embodiment uses a data catalog service to maintain the metadata of incomplete data. Using pipeline mechanisms to achieve low latency is crucial for functions on different nodes, especially for large data transfers.

[0040] The key-value store system (DStore) in this embodiment achieves cross-node storage in the following way: data is first copied from the local storage of another node to the local storage of the first node, and then copied into the container. By using inter-node pipelining, memory copy latency can be hidden through network transmission. In this case, local storage retrieves data from another storage. When a process copies data from its local storage, it accesses the node via the network. Therefore, overall latency can be reduced.

[0041] like Figure 4 As shown, an example is used to illustrate hybrid pipeline optimization: Container A of node 1 has incomplete data S1; such as Figure 4 As shown in diagram b, with the support of intra-node pipeline, container A copies data to the local storage of node 1. Simultaneously, container B on node 1 copies the data S1 from its local storage to the memory of the current working node. With the support of inter-node pipeline, the local storage of node 2 first copies data S1 from the local storage of node 1, and then container C copies the data from the local storage of node 2 to the memory of the current working node. Figure 4 As shown in diagram c, container A generates complete data S and copies it to its local storage 1. With the support of the intra-node pipeline mechanism, container B copies data S2 from local storage 1. Simultaneously, with the support of inter-node pipeline, node 2 copies data from local storage 1 to the local memory of its current working node. Then, container C copies data from local storage 2 into its container memory. Finally, both function B and function C efficiently retrieve data S.

[0042] like Figure 5 As shown, the data structure of the metadata maintained by the data catalog service includes: 1) data size; 2) IP address of the worker node where the data resides; 3) access frequency of the worker node's IP; 4) whether the data is large or small; and 5) the data's ID. Meanwhile, to reduce the cross-node data transfer of small data, the data catalog service maintains a small caching system that caches small data.

[0043] like Figure 6 The figure shows the throughput and latency performance of DStore's put operation. In this experiment, one device is deployed as a data directory service to store metadata. Each worker node runs 8 clients simultaneously executing put requests, with a maximum of 5 worker nodes running a total of 40 clients. The key size is 64 bytes, and the value size is 1MB, as shown in the figure. 1) In put operations, DStore exhibits better performance, offering up to 22 times the improvement compared to CouchDB; 2) DStore demonstrates good scalability, and its performance does not decrease with an increase in the number of clients.

[0044] like Figure 7 The figure shows the throughput experiment results of DStore's get operation. In this experiment, one device is deployed as a data directory to store metadata, another worker node performs put operations, and five other devices simultaneously execute get requests from eight clients, for a total of 40 clients. The key size is 64 bytes and the value size is 1MB.

[0045] As shown in the figure, in the get operation, DStore shows better performance than CouchDB, with a maximum improvement of 6 times. This is because in CouchDB, the get request of the function has to compete for the limited network bandwidth, CPU and other shared resources of the master node. When the number of clients increases, CouchDB will not bring performance improvement; 2) DStore shows good scalability and will not reduce performance due to the increase of clients.

[0046] like Figure 8 As shown, this embodiment achieves 99% tail latency when the network bandwidth is 50MB / s and the request volume is 6 / min, which means it brings tail latency optimization.

[0047] like Figure 9 As shown, this embodiment operates in two modes: 1) each workflow is run individually; 2) all workflows run simultaneously, and then their end-to-end latency is tested, meaning that performance is basically not affected by running workflows simultaneously.

[0048] In summary, compared with existing technologies, this invention, by integrating a two-layer scheduler based on data flow and a hybrid distributed key-value storage system, enables seamless computation of workflow servers within a data flow architecture. The two-layer scheduler reduces scheduling overhead, while the distributed key-value storage system acts as a storage backend for data exchange between functions. This supports out-of-order execution of functions based on data dependencies, improving function parallelism. Furthermore, the decentralized network transmission mechanism increases network bandwidth utilization. Experiments demonstrate that the key-value storage system designed in this invention exhibits good throughput, efficient end-to-end latency, and good scalability, fully utilizing network bandwidth. Simultaneously, the server-aware computation workflow system based on a data-driven architecture proposed in this invention demonstrates significant throughput improvements, end-to-end latency optimization, and tail latency optimization.

[0049] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. A non-aware distributed scheduling system based on a data flow architecture, characterized in that, The application relates to a data flow scheduling method and device based on a distributed storage system. The application comprises a directed acyclic graph parser and a global scheduler arranged on a master node and a data flow local scheduler and a local storage unit arranged on each working node, wherein the directed acyclic graph parser generates a directed acyclic graph according to a user configuration file, cuts the directed acyclic graph into subgraphs based on resources and load balancing principles in a cluster, and outputs the subgraphs to different working nodes, generates metadata of a local work flow, and saves the metadata in a device of a deployment remote storage and a data directory service unit; the data flow local scheduler of each node obtains the summarized metadata of the work flow from the remote storage according to a trigger request, traverses a start function list in the metadata, runs functions in the list in sequence, runs subsequent node functions which have data dependency with the current functions, and continues to run remaining unfinished functions after any function is completed, so that data flow driving is realized. The metadata of the local work flow comprises a data key, a data size, data position information and access frequency of the data position information. The summarized metadata of the work flow comprises an IP of a working node where each function is located, input and output of the function, a start function list of a start function, a successor function of a predecessor function, and a work flow.

2. The data flow architecture based non-cognitive distributed scheduling system according to claim 1, characterized in that, The global scheduler comprises a parser module and a graph cutting module, wherein the parser module parses a user configuration file to generate a directed acyclic graph, the graph cutting module cuts the directed acyclic graph to obtain subgraphs according to resources of a cluster, and saves an IP of a working node where each function in the subgraphs is located, input and output of the function, a start function list of a start function, a successor function of a predecessor function and a work flow as metadata of the work flow in a remote storage.

3. The data flow architecture based non-cognitive distributed scheduling system according to claim 1, characterized in that, The data flow local scheduler adopts an out-of-order trigger mode to execute functions, that is, when a predecessor function of a function is still running, the function can still be started and run, the data flow local scheduler comprises a gateway module and a data flow scheduling module, wherein the gateway module receives a trigger request of a user, obtains a json file and a work flow after analysis, and the data flow scheduling module executes functions in a local storage unit or requests metadata of the work flow from a remote storage according to the work flow.

4. The data flow architecture based non-cognitive distributed scheduling system according to claim 3, characterized in that, When a function with a predecessor function runs to a state that data required by the function has not been generated, the data flow scheduling module sets the function to a waiting state, and simultaneously starts and runs functions in a directed acyclic graph and successor node functions without waiting for execution of a start function list to be completed; when any of the predecessor function or the function is completed, the data flow scheduling module sets to continue to start and run remaining unfinished functions, so that balance between resource utilization and function parallelism is realized.

5. The data flow architecture based non-cognitive distributed scheduling system according to claim 3 or 4, characterized in that, The data flow scheduling module: ① triggers corresponding functions and starts a container according to a data dependency relationship, so that the functions run in the container, and a get interface is called to query whether the functions are located in a local storage unit: when the functions are located in the local storage unit, the container directly copies and executes the functions from the local storage unit, otherwise, data directory service unit is queried through the local storage unit, and data is copied from a remote storage, and then the container copies and executes the functions from the local storage unit. ②Call the put interface, save the data obtained by execution to the local storage unit, and output the metadata information corresponding to the data from the local storage unit to the remote storage.

6. The data flow architecture based non-cognitive distributed scheduling system according to claim 1, wherein, The local storage unit and the data directory service unit on each worker node constitute a hybrid distributed key-value storage system, wherein: the data directory service unit manages the metadata of the workflow data, and provides query services to facilitate the client to query the location and size of the data; and the local storage unit serves as a local key-value storage system, manages the local data, and writes the device IP where the data is located to the data directory service. The key-value storage system decouples data and metadata storage, that is, each worker node stores data in the local storage, and then stores the metadata of the data in another device that deploys the data directory service; and to support efficient data flow-driven workflow server non-aware computing, the data directory service integrates a node frequency access mechanism and an access frequency mechanism to avoid a certain worker node becoming a performance bottleneck.

7. The data flow architecture based non-cognitive distributed scheduling system according to claim 1 or 6, characterized in that, The data directory service unit optimizes the function execution order through an automatic blocking and waking mechanism, specifically: when a process of a worker node queries the data directory service to obtain the metadata of certain data, the data directory service does not have the metadata at this time, to ensure smooth function running, the data directory service will automatically block the process, and automatically wake up the process when the data directory has the metadata.

8. The data flow architecture based non-cognitive distributed scheduling system according to claim 5, wherein, The direct copying from the local storage unit is achieved through in-node pipelined copying, specifically: a data structure is set in the local storage unit to record the generation progress of the data, so that the data is still being copied from the container to the local storage unit, and the container of the current worker node can simultaneously copy the currently generated data from the local storage unit, that is, one container copies data from the container memory to the local storage unit, and the other container reads the data from the local storage unit to the container of the current worker node.

9. The data flow architecture based non-sensor distributed scheduling system according to claim 8, characterized in that, When the data dependency of two functions is allocated to different worker nodes, the function data interaction is achieved through cross-node pipelined copying, specifically: when the container copies data from the container memory of the current worker node to the local storage unit, the local storage unit asynchronously outputs the metadata of the data containing the current generation progress of the data to the data directory service unit. When other worker nodes query the data directory service unit and receive the metadata, the currently generated data is copied from the local storage unit of the other worker nodes.

10. The data flow architecture based non-cognitive distributed scheduling system according to claim 6, wherein, The key-value storage system realizes the decentralization of receiving between different worker nodes through a coordination mechanism data exchange, specifically: the key-value storage system performs node-to-node data transmission through the network only in one case: a function of a device finds that the required data is on the remote storage, that is, there is data dependency between functions on different devices.

Images