Cross-cluster traffic optimization scheduling method and system for data processing workflow

Abstract

This invention discloses a cross-cluster traffic optimization scheduling method and system for data processing workflows. It provides a strategy for optimizing job scheduling of cross-cluster data processing workflows by analyzing bottleneck points and migrating jobs across clusters at those bottlenecks. The main steps include: constructing a branching-merging directed acyclic graph of the data processing workflow; performing a depth-first post-order traversal of the graph; analyzing the bottleneck points and bottleneck traffic of each job during the traversal, and determining the execution cluster for that job at the bottleneck point. This invention can reduce cross-cluster data traffic in data pipelines and container workflows.

Description

Technical Field

[0001] This invention relates to the fields of distributed computing, data pipelines, container workflows, and more specifically to a method and system for optimizing and scheduling data processing workflows running across clusters. Background Technology

[0002] In this invention, "data processing workflow" refers to a data pipeline or container workflow running across multiple clusters. A cluster refers to a set of low-latency, high-speed interconnected servers. The network latency between servers within a cluster is generally no more than 5ms, and the point-to-point interconnection bandwidth between servers is at least 10Gbps. In contrast, the bandwidth between different clusters is at least 1-2 orders of magnitude lower than the interconnection bandwidth within a cluster (e.g., 1Gbps, 100Mbps), and is shared by the servers within the cluster. Generally, servers located in the same data center (i.e., server room) can form a cluster; servers located in different data centers belong to different clusters. However, if high-speed communication lines are established between data centers, servers located in different data centers may also form a cluster. In this invention, server types include physical machines and virtual machines.

[0003] A data pipeline is a multi-step data processing flow consisting of multiple data processing jobs with upstream and downstream dependencies. Here, a data processing job (or simply job) broadly refers to the querying, reading, modifying, and analyzing of a given input dataset (or input), which outputs a corresponding processed dataset (or output). Typical jobs include a Python data processing script, a MapReduce big data processing job, and an Apache Hive big data query. Big data jobs are typically broken down into multiple tasks, and all tasks are usually scheduled within the same cluster for distributed concurrent execution. To simplify the problem, jobs in this invention are limited to not running across clusters. The upstream and downstream dependencies between jobs mean that the input of a downstream job is the output of an upstream job. Each job constitutes a step in the data processing flow, and enters a stopped state upon completion. Data pipelines are widely used to represent ETL (Extract, Transform, Load) preprocessing of data, or data product processing flows. Typical application scenarios include data warehouses and data lakes. In practice, various big data systems can be used to execute data pipelines, such as Apache NiFi and PiFlow.

[0004] A container workflow is a multi-step execution process consisting of multiple containers with upstream and downstream dependencies. Here, a container is a widely used, lightweight virtualization technology. The upstream and downstream dependencies between containers mean that the input data of a downstream container is the output data of an upstream container. Each container constitutes a step in the container workflow, and enters a stopped state after execution. Container workflows are widely used in cloud computing environments for microservice instantiation and process-oriented data analysis. For example, Terra.BIO uses container workflows for multi-step life science data analysis.

[0005] With the widespread adoption of distributed big data processing systems such as Apache Spark and Apache Hadoop, and the extensive use of cross-datacenter big data collaborative analysis, federated learning, multi-cloud computing, functional computing, and container workflow technologies, cross-datacenter data processing workflows are becoming increasingly common. Since servers located in different data centers typically belong to different clusters, the cross-cluster operation of data processing workflows is becoming more prevalent, inevitably leading to cross-cluster data migration. Because inter-cluster network bandwidth is generally 1-2 orders of magnitude lower than intra-cluster interconnection bandwidth, and because there is a pay-per-traffic model (e.g., Alibaba Cloud and Amazon Web Services offer public network bandwidth pay-per-traffic policies), a reasonable scheduling strategy should be designed to minimize cross-cluster traffic.

[0006] This invention uses "data processing workflow" to refer to a data pipeline or container workflow, and "job" to refer to a job in a data pipeline or a container in a container workflow.

[0007] The logical structure of a data processing workflow can typically be represented as a Fork-Merge DAG (Fork-Merge Directed Acyclic Graph, FMDAG for short). In an FMDAG, vertices represent jobs, and directed edges represent upstream and downstream dependencies between jobs; that is, the output of the starting vertex of an edge is used as the input of the target vertex of that edge. Leaf vertices represent the loading process of the original input data, also known as the initial job; the root vertex represents the final step of the workflow, also known as the final job. FMDAG can effectively express various dependencies between different jobs in a data processing workflow, such as upstream and downstream dependencies, fork dependencies, and merge dependencies. Among them:

[0008] (1) Upstream and downstream dependency refers to the output of one job (numbered J_A) being directly used as part or all of the input of another job (numbered J_B). In this case, the present invention refers to J_A as the direct upstream of J_B and J_B as the direct downstream of J_A. J_A itself and its direct upstream are referred to as the upstream of J_B, J_B itself and its direct downstream are referred to as the downstream of J_A, the upstream of J_A's upstream is also referred to as the upstream of J_B, and the downstream of J_B's downstream is also referred to as the downstream of J_A.

[0009] (2) The Fork dependency relationship is used to refer to the relationship between jobs (numbered J_B, J_C, etc.) that use the output of the same job (numbered J_A) as part or all of their input. In this case, the present invention refers to J_B, J_C, etc. as each other's Fork jobs.

[0010] (3) The Merge dependency relationship is used to refer to "the relationship between two (or more) jobs (numbered J_B, J_C, etc.) when the output of two jobs (numbered J_B, J_C, etc.) is simultaneously used as the input of another job (numbered J_A)". In this case, the present invention refers to J_B, J_C, etc. as each other's Merge jobs, and J_A as the Merge point.

[0011] In this invention, "flow" refers to "the amount of data (assumed to be s) from a direct upstream node (J_B) of a certain job (numbered J_A) in the input of that job," and states that "the flow from J_A to J_B is s," "the flow from J_B to J_A is s," or "the flow between J_A and J_B is s." On FMDAG, flow is represented as the weight of an edge.

[0012] If two upstream and downstream jobs (numbered J_A and J_B) are not located in the same cluster (i.e., the data processing workflow is cross-cluster), then cross-cluster data migration is inevitable. This invention refers to the traffic between jobs located in different clusters as "cross-cluster traffic," uses "total cross-cluster traffic" to refer to the sum of all cross-cluster traffic generated during the execution of the data processing workflow, and uses "cross-cluster traffic overhead" to refer to the time and network traffic costs (referred to as network overhead) incurred due to cross-cluster data migration.

[0013] The data processing workflow scheduling strategy refers to selecting appropriate execution clusters for all jobs in the workflow and determining the job submission order. It should be noted that the execution cluster of the starting job (i.e., the leaf vertex in FMDAG) is known. This is because the starting job represents the loading process of the original input data and can only be executed on the cluster where the original input data resides. For a given data processing workflow, the location of the original input data is a known condition.

[0014] Popular job scheduling strategies available for FMDAG workflow scheduling include those proposed by Blythe et al. (see BLYTHE J, JAIN S, DEELMAN E, et al. Task scheduling strategies for workflow-based applications in grids[J / OL]. 2005 IEEE International Symposium on Cluster Computing and the Grid, CCGrid 2005, 2005, 2:759-767[2022-12-05]. https: / / doi.org / 10.1109 / CCGRID.2005.1558639.), the Kubernetes scheduler, and the Skipy scheduler (see RAUSCH T, RASHED A, DUSTD AR S. Optimized container scheduling for data-intensive serverless edge computing[J / OL]. Future Generation). ComputerSystems, 2021, 114:259-271 [2022-12-05]. https: / / doi.org / 10.1016 / J.FUTURE.2020.07.017. These scheduling strategies generally aim to minimize workflow runtime. However, they are designed for single-cluster data processing workflows. Using them for job scheduling across cross-cluster data processing workflows can lead to additional cross-cluster data transfer overhead. For example, the Kubernetes scheduler prioritizes new jobs (containers) for execution on clusters with sufficient CPU / memory resources. However, if the cluster lacks the input data required by the new job, additional time and network traffic costs are incurred to migrate the input data from other clusters.

[0015] A naive scheduling strategy is readily conceived to optimize cross-cluster traffic in data processing workflows. This strategy starts from the upstream of the workflow and schedules new jobs to be executed step-by-step, prioritizing the allocation of the job to the cluster containing its immediate upstream node. This invention refers to this naive scheduling strategy as... Scheduling Strategy. A job may have multiple direct upstreams. If all its direct upstreams are located in the same cluster, this scheduling strategy does not generate additional cross-cluster traffic. If its direct upstreams are located in different clusters, this scheduling strategy selects a job allocation scheme that minimizes cross-cluster traffic. For example, consider a data processing workflow W, where a job J has two direct upstreams J_A and J_B, located in clusters A and B respectively. The traffic from the two direct upstreams to J is f... A f B Obviously, if f A <f B In this case, job J should be scheduled to be executed on cluster B. The additional cross-cluster traffic at this time is f. A Obviously The scheduling strategy still has room for improvement. For example, if J_A in the above example has exactly one direct upstream J_C, and J_C is known to be executing on cluster A, the traffic from J_C to J_A is f. C→A And f C→A <f A If according to The scheduling strategy dictates that jobs J_A and J will be assigned to cluster A and cluster B for execution, respectively, resulting in cross-cluster traffic of f. A It is evident that its drawback is that it generates significant cross-cluster traffic. Summary of the Invention

[0016] This invention proposes a new technical concept for The scheduling strategy is improved. Taking a data processing workflow as an example, this new technology pre-schedules job J_A to cluster B for execution, thereby reducing the cross-cluster traffic generated by jobs J_A and J_B to f. C→A The principle behind this improvement is to find the upstream job with the lowest traffic (i.e., the bottleneck) among all upstream jobs of the current job, and then "reroute" (i.e., migrate to another cluster) the execution from the bottleneck point. This invention refers to the "bottleneck" of a job as the job with the lowest input data volume among the current job and all its upstream jobs (excluding the starting job), and the input data volume of the bottleneck is called the bottleneck traffic.

[0017] The purpose of this invention is to propose a cross-cluster traffic optimization scheduling method and system for data processing workflows based on the above-mentioned improved technical concept. This scheduling method can allocate a reasonable execution cluster to each job in the data processing workflow to minimize the total cross-cluster traffic of the data processing workflow, thereby reducing the time and network overhead caused by cross-cluster data migration.

[0018] The technical solution adopted by the present invention to achieve the above objectives is as follows:

[0019] This invention proposes a cross-cluster traffic optimization scheduling method for data processing workflows, comprising the following steps:

[0020] 1) For the data processing workflow, construct a branching-merging directed acyclic graph (FMDAG), where vertices in the FMDAG represent jobs and edges represent upstream and downstream dependencies between jobs;

[0021] 2) Reverse the direction of all edges in FMDAG; starting from the root vertex, perform a depth-first postorder traversal of FMDAG;

[0022] 3) If the currently traversed vertex is a leaf vertex, its execution cluster is known, and its bottleneck flow is set to infinity; if the currently traversed vertex is not a leaf vertex, determine whether the vertex has one or more direct upstreams.

[0023] 4) If the currently traversed vertex has only one direct upstream, then determine the bottleneck point and bottleneck flow of the vertex based on the relationship between the flow from the direct upstream and the bottleneck flow of the direct upstream.

[0024] 5) If the currently traversed vertex has only one direct upstream vertex and that vertex is the root vertex, then use... The scheduling strategy determines the execution cluster of the vertex and its upstream execution cluster to be assigned;

[0025] 6) If the currently traversed vertex has multiple direct upstreams, calculate the component bottleneck flow and component bottleneck point of each direct upstream to the vertex, and determine the direct upstream with the largest component bottleneck flow as the main stream of the vertex.

[0026] 7) If the main stream of the currently traversed vertex is yet to be assigned, then first use... The scheduling strategy determines the main stream of the vertex and the execution cluster C of the upstream of the main stream to be assigned, then sets the execution cluster of the vertex to C, and finally sets the bottleneck flow of the vertex to infinity.

[0027] 8) If the currently traversed vertex has a non-mainstream direct upstream to be assigned, and the bottleneck point of the component from that direct upstream to the vertex is B, then perform a depth-first postorder traversal of the vertex, the direct upstream, and the upstream of that direct upstream. When B is visited, stop the traversal and start backtracking. During backtracking, set the execution cluster of each traversed job to execution cluster C, and finally use... The scheduling strategy determines the execution cluster of the immediate upstream and the remaining jobs to be assigned in the upstream of the immediate upstream;

[0028] 9) If the currently traversed vertex has a non-mainstream direct upstream that needs to be assigned, return to step 8) and repeat the execution.

[0029] Preferably, the data processing workflow includes a data pipeline and a container workflow.

[0030] Preferably, in step 4), for a vertex J with only one upstream, the flow from J to its only upstream is known to be p1. Let the bottleneck point of the only upstream be B1 and the bottleneck flow be ρ1. The method for determining the bottleneck point B and the bottleneck flow ρ of J is as follows: if ρ1≥p1, then B=J, ρ=p1; otherwise, B=B1, ρ=ρ1.

[0031] Preferably, in step 5), for a root vertex J with only one upstream vertex, the method of using... The scheduling strategy determines the execution cluster of job J as follows: starting from job J, perform a depth-first, post-order traversal of job J and its upstream jobs, until the first execution cluster C is visited. p When the job is known, stop traversing and start backtracking, setting the execution cluster of each traversed job to C during backtracking. p .

[0032] Preferably, in step 6), let the direct upstream of the currently traversed vertex J be F1, F2, ..., F n F i Let J be any of the direct upstreams, and J be known to come from F. i The flow rate is p i Let F i Bottleneck point B i The bottleneck flow rate is ρ i The F i Bottleneck flow to J Bottleneck The calculation method is as follows: if ρ i ≥p i ,but otherwise

[0033] Preferably, in step 7), the adoption The scheduling strategy determines the main stream and all upstream execution clusters to be assigned to the main stream as follows: Starting from the main stream, the main stream and its upstream jobs are traversed in a depth-first, post-order manner. When the first execution cluster C of a job is encountered, the traversal is stopped and backtracking begins. During backtracking, the execution cluster of each traversed job is set to C.

[0034] Preferably, in step 8), the adoption The scheduling strategy determines the execution cluster of the direct upstream and the remaining unassigned jobs in its upstream by using the component bottleneck point from the direct upstream to the currently traversed vertex J. direct upstream Starting from the first position, perform a depth-first postorder traversal. And its upstream operations, upon encountering the first execution cluster When the job is known, stop traversing and start backtracking, setting the execution cluster of each traversed job to [a specific cluster].

[0035] The present invention also proposes a cross-cluster traffic optimization scheduling system for data processing workflows, including a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the steps of the above method.

[0036] The technical effects achieved by this invention are as follows:

[0037] This invention, by traversing the FMDAG corresponding to the data processing workflow, ensures that each job in the workflow is assigned to an execution cluster. It is worth noting that the job submission order does not affect the total cross-cluster traffic; therefore, any job submission order can be used as the bottleneck scheduling strategy. This invention calls... The scheduling strategy involves migrating jobs at their bottlenecks to ensure that the total cross-cluster traffic generated does not exceed [a certain threshold]. The total cross-cluster traffic generated by the scheduling strategy. Since cross-cluster traffic overhead is generally positively correlated with cross-cluster traffic, reducing cross-cluster traffic can reduce cross-cluster traffic overhead; and since cross-center traffic is generally also cross-cluster, reducing cross-cluster traffic can also reduce potential cross-center traffic. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below.

[0039] Figure 1 This is a flowchart of a cross-cluster traffic optimization scheduling method for data processing workflow proposed in this invention.

[0040] Figure 2 This is a schematic diagram of the FMDAG corresponding to the data processing workflow in an embodiment of the present invention.

[0041] Figures 3 to 5 This is a specific operational example diagram of an embodiment of the present invention. Detailed Implementation

[0042] To make the above features and advantages of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0043] This invention proposes a cross-cluster traffic optimization scheduling method for data processing workflows (referred to as the bottleneck scheduling strategy). For a data processing workflow, a corresponding bifurcation-merge directed acyclic graph (FMDAG) is constructed. The running environment of the data processing workflow is a set of clusters N = {C1, C2, ..., C}. n}, where C i is the execution cluster. The cross-cluster traffic optimization scheduling method proposed by the present invention can be used to find a mapping relationship from the data processing workflow to N. Under this relationship, each job in the data processing workflow is assigned to run on a certain cluster in N, and the total cross-cluster traffic generated does not exceed (usually less than) the total cross-cluster traffic generated by the scheduling policy mentioned above.

[0044] The bottleneck point scheduling policy first reverses the direction of the edges in the FMDAG, and then performs a depth-first post-order traversal of the FMDAG. By analyzing the bottleneck points and bottleneck traffic of each job, and determining the execution cluster of the job at the bottleneck point, an execution cluster is assigned to each vertex of the FMDAG. The traversal starts from the root vertex of the FMDAG. Among them, the calculation process of the execution cluster C of the currently traversed vertex J is as Figure 1 shown. The specific steps are as follows (using hierarchical numbers to represent the hierarchical relationship between steps):

[0045] Step 1. If J has no direct upstream, then J is a leaf vertex, and the execution cluster C of J is known; let the bottleneck traffic ρ of J be +∞ (that is, J cannot be a bottleneck point).

[0046] Step 2. If J has only one direct upstream (numbered F1), and the known traffic from this upstream to J is p1, let the bottleneck point of F1 be B1 and the bottleneck traffic be ρ1:

[0047] Step 2.1. If ρ1≥p1, then the bottleneck point B of J is J, and the bottleneck traffic ρ is p1.

[0048] Step 2.2. If ρ1<p1, then the bottleneck point B of J is B1, and the bottleneck traffic ρ is ρ1.

[0049] Step 2.3. If J is the root vertex, then use the scheduling policy to determine the execution cluster of J and the execution clusters of the unassigned upstream of J: Starting from J, perform a depth-first post-order traversal of job J and its upstream jobs. When accessing the first job whose execution cluster is known (that is, this job is an assigned job, and let the execution cluster of this job be C p ), stop continuing the traversal, start backtracking, and set the execution cluster of each traversed job to C p .

[0050] Step 3. If J has multiple direct upstreams, and the known traffic from each direct upstream (numbered F1~F n ) to J is p1,…,p n , let the bottleneck points of the direct upstreams be B1,…,B n respectively, and the corresponding bottleneck traffic be ρ1,…,ρ n:

[0051] Step 3.1. For each direct upstream F i (i = 1, ..., n), calculate the bottleneck flow of J from F i The amount and the bottleneck point corresponding to this component.

[0052] Step 3.1.1. If ρ i ≥p i ,make

[0053] Step 3.1.2. If ρ i <p i ,make

[0054] Step 3.2. Calculation The maximum value in the range, let the index of this maximum value be m, is called F. m The main flow operation is J.

[0055] Step 3.3. Use Scheduling strategy determines mainstream operation F m and its entire upstream execution cluster: with mainstream job F m Starting from point F, the depth-first postorder traversal operation is performed. m And its upstream jobs, upon accessing the first job whose execution cluster is known (let the execution cluster of this job be C). M When the time reaches 0, stop traversing and start backtracking, setting the execution cluster of each traversed job to C during backtracking. M .

[0056] Step 3.4. Set the execution cluster of J to C. M And set the bottleneck flow ρ of J to +∞ (that is, the execution cluster of J has been determined, and J cannot be the bottleneck point).

[0057] Step 3.5. Determine other direct upstream F of J. i ′(i=1,…,n,i≠m) and F i The entire upstream execution cluster of ':

[0058] Step 3.5.1. Starting from J, perform a depth-first postorder traversal of J and F. i ′ and F i Upstream of ′, when accessing the job When the time comes, stop traversing and start backtracking, setting the execution cluster of each traversed job to C during backtracking. M .

[0059] Step 3.5.2. Use Scheduling strategy determination F i The execution cluster of ′ and all unassigned jobs in its entire upstream: by job direct upstream Starting from the first position, perform a depth-first postorder traversal. And its upstream jobs, when encountering the first job whose execution cluster is known (let the execution cluster of this job be...), When the iteration stops and backtracking begins, the execution cluster of each traversed job is set to [a specific value].

[0060] This invention's method, by traversing the FMDAG corresponding to the data processing workflow, ensures that each job in the workflow is assigned to an execution cluster. It is worth noting that the job submission order does not affect the total cross-cluster traffic; therefore, this invention's method can use any job submission order. This invention's method calls [the relevant steps] in steps 2.3, 3.3, and 3.5.2. The scheduling strategy is implemented, and jobs are migrated at bottleneck points (step 3.5.1) to ensure that the total cross-cluster traffic generated does not exceed [a certain threshold]. Total cross-cluster traffic generated by the scheduling strategy.

[0061] The following is a specific example:

[0062] This embodiment uses respectively Scheduling strategies and bottleneck scheduling strategies are used to optimize cross-center traffic scheduling for a specific data processing workflow, and the total cross-center traffic volume under the two strategies is compared.

[0063] The data processing workflow in this embodiment consists of 10 jobs, with corresponding FMDAGs as follows: Figure 2 As shown in the diagram. Jobs J_1, J_2, J_3, and J_4 read object storage data from clusters C_1, C_2, C_3, and C_4, respectively. Jobs J_5, J_6, and J_7 are Apache Spark data processing jobs, and jobs J_8, J_9, and J_10 are Python data processing scripts running in containers. Figure 2 It can be seen that J_1 and J_2 are both direct upstreams of J_5 and upstreams of J_10. J_7 and J_8 are each other's Fork operations, J_1 and J_2 are each other's Merge operations, and the traffic from J_6 to J_7 is 1000MB.

[0064] Table 1 The scheduling policy generates the execution cluster allocation scheme and the corresponding cross-cluster traffic.

[0065]

[0066] Figures 3 to 5 This is a specific operational example diagram of this embodiment. Wherein, Figure 3 The BottleneckScheduler function in the code is a bottleneck scheduling strategy implemented in Python. The input J of the function is the index of the root vertex of the FMDAG corresponding to the data processing workflow. This function depends on... Figure 4 The NaiveScheduler function and the BottleneckRedirect function in the code. Figure 4 The NaiveScheduler function in the code is implemented in Python. The scheduling strategy takes the input F of the function as the index of the root vertex of the FMDAG corresponding to the data processing workflow. This FMDAG can be... Figure 2 The function is a subgraph of FMDAG and can only be called by the BottleneckScheduler function; it cannot be run directly. Figure 5 The BottleneckRedirect function is a bottleneck job migration strategy implemented in Python. This function can only be called by the BottleneckScheduler function and cannot be run directly.

[0067] The specific implementation steps of this embodiment are as follows:

[0068] Step 1. According to The scheduling strategy was manually calculated, resulting in the scheduling scheme shown in Table 1. Under this scheme, the total cross-cluster traffic generated by the data processing workflow is 1000MB.

[0069] Step 2. Number the assignments starting from zero. Assignment J_1 is numbered 0, and so on. Assignment J_10 is numbered 9.

[0070] Step 3. Construct a global variable "adjacency matrix M" using Python. M[i][j] represents the flow from job i to job j. M[i][j] = 0 indicates that job j is not an upstream of job i (i and j are the job numbers, the same below).

[0071] Step 4. Construct a global variable "allocation array C" using Python. C[i] is used to store the cluster code of job i. The element values ​​in C are initialized to 0, and the cluster code corresponding to the starting job is set.

[0072] Step 5. Construct a global variable “bottlenecks_point array” using Python, initialize the element values ​​to -1, and use bottlenecks_point[i] to store the bottleneck point of job i.

[0073] Step 6. Construct a global variable “bottlenecks_flow” using Python. Initialize the element values ​​to infinity. bottlenecks_flow[i] is used to store the bottleneck flow of job i.

[0074] Step 7. Execute Figure 3 The BottleneckScheduler function shown has an input value of J = 9 (i.e., ... Figure 2 (The index of the root vertex J_10)

[0075] Step 8. Print the allocated array C using Python: print(C). The output is:

[0076] [1,2,3,4,1,4,4,4,4,4]

[0077] The result indicates that the execution clusters for jobs J_1, ..., J_10 are cluster 1, cluster 2, cluster 3, cluster 4, cluster 1, cluster 4, cluster 4, cluster 4, cluster 4, cluster 4, cluster 4, respectively.

[0078] Step 9. Based on the output of Step 8, the total cross-cluster traffic generated by the data processing workflow under the bottleneck scheduling strategy is calculated to be 500MB (as shown in Table 2).

[0079] Step 10. Conclusion: The bottleneck scheduling strategy significantly reduced the total cross-cluster traffic of the data processing workflow in this embodiment.

[0080] Table 2 shows the execution cluster allocation scheme and corresponding cross-cluster traffic generated by the bottleneck scheduling strategy.

[0081]

[0082]

[0083] It should be noted that, for the sake of simplicity, the embodiments of the above methods are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the present invention.

[0084] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-cluster traffic optimization scheduling method for data processing workflows, characterized in that, Includes the following steps: 1) For the data processing workflow, construct a branching-merging directed acyclic graph (FMDAG), where vertices in the FMDAG represent jobs and edges represent upstream and downstream dependencies between jobs; 2) Reverse the direction of all edges in FMDAG; Starting from the root vertex, perform a depth-first postorder traversal of FMDAG; 3) If the currently traversed vertex is a leaf vertex, its execution cluster is known, and its bottleneck flow is set to infinity; if the currently traversed vertex is not a leaf vertex, determine whether the vertex has one or more direct upstreams. 4) If the currently traversed vertex has only one direct upstream, then determine the bottleneck point and bottleneck flow of the vertex based on the relationship between the flow from the direct upstream and the bottleneck flow of the direct upstream. 5) If the currently traversed vertex has only one direct upstream vertex and that vertex is the root vertex, then use... The scheduling strategy determines the execution cluster of the vertex and its upstream execution cluster to be assigned; 6) If the currently traversed vertex has multiple direct upstreams, calculate the component bottleneck flow and component bottleneck point of each direct upstream to the vertex, and determine the direct upstream with the largest component bottleneck flow as the main stream of the vertex. 7) If the main stream of the currently traversed vertex is yet to be assigned, then first use... The scheduling strategy determines the main stream of the vertex and the execution cluster C of the upstream of the main stream to be assigned, then sets the execution cluster of the vertex to C, and finally sets the bottleneck flow of the vertex to infinity. 8) If the currently traversed vertex has a non-mainstream direct upstream to be assigned, and the bottleneck point of the component from that direct upstream to the vertex is B, then perform a depth-first postorder traversal of the vertex, the direct upstream, and the upstream of that direct upstream. When B is visited, stop the traversal and start backtracking. During backtracking, set the execution cluster of each traversed job to execution cluster C, and finally use... The scheduling strategy determines the execution cluster of the immediate upstream and the remaining jobs to be assigned in the upstream of the immediate upstream; 9) If the currently traversed vertex has a non-mainstream direct upstream that needs to be assigned, return to step 8) and repeat the execution.

2. The method as described in claim 1, characterized in that, The data processing workflow includes data pipelines and container workflows.

3. The method as described in claim 1, characterized in that, In step 4), for a vertex J with only one upstream, the flow from J to its only upstream is known to be p1. Let the bottleneck point of the only upstream be B1 and the bottleneck flow be ρ1. The bottleneck point B and bottleneck flow ρ of J are determined as follows: if ρ1≥p1, then B=J, ρ=p1; otherwise, B=B1, ρ=ρ1.

4. The method as described in claim 1, characterized in that, In step 5), for a root vertex J with only one upstream vertex, the method adopted... The scheduling strategy determines the execution cluster of job J as follows: starting from job J, perform a depth-first, post-order traversal of job J and its upstream jobs, until the first execution cluster C is visited. p When the job is known, stop traversing and start backtracking, setting the execution cluster of each traversed job to C during backtracking. p .

5. The method as described in claim 1, characterized in that, In step 6), let the direct upstream of the currently traversed vertex J be F1, F2, ..., F n F i Let J be any of the direct upstreams, and J be known to come from F. i The flow rate is p i Let F i Bottleneck point B i The bottleneck flow rate is ρ i The F i Bottleneck flow to J Bottleneck The calculation method is as follows: if ρ i ≥p i ,but otherwise 6. The method as described in claim 1, characterized in that, In step 7), the adoption The scheduling strategy determines the main stream and all upstream execution clusters to be assigned to the main stream as follows: Starting from the main stream, the main stream and its upstream jobs are traversed in a depth-first, post-order manner. When the first execution cluster C of a job is encountered, the traversal is stopped and backtracking begins. During backtracking, the execution cluster of each traversed job is set to C.

7. The method as described in claim 1, characterized in that, In step 8), the adoption The scheduling strategy determines the execution cluster of the direct upstream and the remaining unassigned jobs in its upstream by using the component bottleneck point from the direct upstream to the currently traversed vertex J. direct upstream Starting from the first position, perform a depth-first postorder traversal. And its upstream operations, upon encountering the first execution cluster When the job is known, stop traversing and start backtracking, setting the execution cluster of each traversed job to [a specific cluster].

8. A cross-cluster traffic optimization scheduling system for data processing workflows, characterized in that, It includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the program to implement the steps of the method according to any one of claims 1-7.

Images