A task scheduling method and system based on Spark load balancing

Abstract

The application discloses a task scheduling method based on Spark load balancing, which firstly acquires computing resource information, data storage information, and defines a communication cost calculation mode, and then uniformly models the three; then the Hungarian algorithm is adopted to obtain the optimal solution under the global state of the established unified model; finally, the obtained task scheduling table is embedded in the task scheduler of Spark, and the original scheduling scheme based on the greedy algorithm is shielded. The application can solve the technical problems that the existing scheduling method using the greedy algorithm selects the current optimal solution at each step, lacks a global perspective, and thus leads to the final scheduling strategy falling into a local optimal situation, thereby affecting the scheduling performance, and the technical problems that the method only considers the performance improvement caused by the high locality level, and does not consider the performance loss caused by the waiting time, so that the overall communication consumption is large, and the CPU utilization is low.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed parallel computing performance optimization, and more specifically, relates to a task scheduling method and system based on Spark load balancing. Background Art

[0002] With the development of the internet, data volumes have exploded, and the amount of data that needs to be processed is increasing. However, the growth of CPU computing power has lagged far behind this growth. The open source ecosystem for big data is also becoming increasingly diverse, but the diversity of computing engines and open source components also presents the challenge of improving data processing performance throughout the entire lifecycle. How to process as much data as possible within limited resources and achieve load balancing in the system is an area where the current task scheduling algorithm of the big data computing engine Spark urgently needs optimization.

[0003] The mainstream scheduling method currently used by Spark is based on a greedy strategy. The system sets a default waiting time of 3 seconds. That is, whenever a new task arrives, it will be given priority for execution on an executor with a higher locality level (data closer to the computing resources). If the executors with a higher locality level are all busy, the system will wait for 3 seconds. If there is still no executor that meets the level requirement, the locality level of the task will be downgraded.

[0004] However, the above scheduling method based on greedy strategy has some non-negligible defects:

[0005] First, because this method selects the current optimal solution at each step and lacks a global perspective, the final scheduling strategy falls into a local optimal state, which in turn affects scheduling performance.

[0006] Second, this method only considers the performance improvement brought by high locality level, but does not consider the performance loss caused by waiting time (that is, even if the node is idle, it cannot run tasks due to insufficient locality level). Therefore, it will cause the overall communication consumption to be high and the CPU utilization to be low.

[0007] Third, this method attempts to schedule tasks as long as there is an idle executor, without considering the compatibility between the task and the executor. As a result, when a subsequent task compatible with the executor (i.e., with a high locality level) appears, the executor is already occupied, ultimately extending the overall program runtime. Summary of the Invention

[0008] In response to the above defects or improvement needs of the prior art, the present invention provides a task scheduling method and system based on Spark load balancing, which aims to solve the technical problems that the existing scheduling method using a greedy algorithm selects the current optimal one at each step of execution and lacks a global perspective, thus causing the final scheduling strategy to fall into a local optimal situation, thereby affecting the scheduling performance; and the technical problems that the method only considers the performance improvement brought by the high level of locality but does not consider the performance loss brought by the waiting time, thus causing the overall communication consumption to be large and the CPU utilization to be low; and the method will try to arrange the task to run as long as there is an idle executor, without considering the adaptation between the task and the executor, so that when a task adapted to the executor appears later, the executor is already occupied, which ultimately leads to the technical problem of extending the overall running time of the program.

[0009] To achieve the above object, according to one aspect of the present invention, a task scheduling method based on Spark load balancing is provided, comprising the following steps:

[0010] (1) Get each executor on each node j in the Spark cluster And the data block b on each node j ji ={b ji0, b ji1 ,b ji2}, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j];

[0011] (2) For each executor l on each node j in the Spark cluster obtained in step (1), obtain the communication overhead of the executor l processing each data block p in the Spark cluster;

[0012] c pl =d(p,j)×db p

[0013] where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p Represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2];

[0014] (3) Create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lpIt is the communication overhead of executor l obtained in step (2) processing the p-th data block in the Spark cluster;

[0015] (4) Create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution;

[0016] (5) According to the matrix Matrix1 obtained in step (3) and the matrix Matrix2 obtained in step (4), the objective function MinCost(Matrix2,Matrix1) of the minimum total communication cost is obtained;

[0017] (6) Use the Hungarian algorithm to solve the objective function of the minimum total communication cost obtained in step (5) to obtain the task scheduling allocation table.

[0018] (7) According to the task scheduling allocation table obtained in step (6), all pending tasks are scheduled to all executors in the Spark cluster to obtain the scheduling results.

[0019] Preferably, the network distance from the pth data block to node j is:

[0020] d(p,j)=min{d(p0,j),d(p1,j),d(p2,j)}

[0021] Where min means taking the minimum value, d(p0,j) represents the network distance from the first replica in the p-th data block to node j, d(p1,j) represents the network distance from the second replica in the p-th data block to node j, and d(p2,j) represents the network distance from the third replica in the p-th data block to node j.

[0022] Preferably, the matrix element Matri2 pl The constraints are as follows:

[0023]

[0024] Preferably, the objective function of minimum total communication cost is equal to:

[0025]

[0026] The vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the p-th column in matrix Matrix1.

[0027] Preferably, step (7) comprises the following steps:

[0028] (7-1) Set counter i = 0;

[0029] (7-2) Determine whether i is greater than the total number of tasks to be processed. If so, the process ends; otherwise, proceed to step (7-3);

[0030] (7-3) Determine whether the total number of idle executors in the current Spark cluster is greater than 80% of the total number of executors in the Spark cluster. If so, proceed to step (7-4), otherwise return to step (7-1);

[0031] (7-4) Allocate the i-th pending task to the corresponding executor for execution according to the task scheduling allocation table, delete the i-th pending task from the task scheduling allocation table, and then proceed to step (7-5);

[0032] (7-5) Set counter i=i+1 and return to step (7-2).

[0033] According to another aspect of the present invention, a task scheduling system based on Spark load balancing is provided, comprising:

[0034] The first module is used to obtain each executor on each node j in the Spark cluster And the data block b on each node j ji ={b ji0, b ji1, b ji2}, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j];

[0035] The second module is used to obtain the communication overhead of each executor l on each node j in the Spark cluster obtained in the first module for processing each data block p in the Spark cluster;

[0036] c pl =d(p,j)×db p

[0037] where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p Represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2];

[0038] The third module is used to create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp This is the communication overhead of executor l processing the p-th data block in the Spark cluster obtained in the second module;

[0039] The fourth module is used to create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution;

[0040] The fifth module is used to obtain the objective function MinCost(Matrix2,Matrix1) of the minimum total communication cost according to the matrix Matrix1 obtained by the third module and the matrix Matrix2 obtained by the fourth module;

[0041] The sixth module is used to solve the objective function of the minimum total communication cost obtained in the fifth module using the Hungarian algorithm to obtain a task scheduling allocation table.

[0042] The seventh module is used to schedule all pending tasks to all executors in the Spark cluster according to the task scheduling allocation table obtained in the sixth module to obtain the scheduling results.

[0043] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0044] (1) Due to the adoption of steps (1) and (2), the present invention obtains the information of all executors and data blocks of the Spark cluster and calculates the communication overhead between them, thereby obtaining scheduling information from a global perspective. Therefore, it can solve the technical problem of local optimality existing in the existing scheduling method based on the greedy strategy;

[0045] (2) The present invention adopts steps (3) to (6), which obtains the communication consumption matrix from a global perspective and obtains a unique solution through the Hungarian algorithm to form a task scheduling allocation table, so that the tasks to be assigned correspond to the executors one by one, and there is no problem of executor idling (the CPU is in an idle state but no tasks are executed). Therefore, it can solve the technical problems of large overall communication consumption and low CPU utilization in the existing scheduling method based on the greedy strategy;

[0046] (3) Due to the adoption of step (7), the present invention generates a new round of task scheduling allocation table by controlling the number of idle executors to reach 80%. This ensures that the majority of executors in the cluster participate in the allocation process during the generation of the task allocation table, thereby resolving the problem of executors and tasks not being compatible, which causes the overall program running time to be too long. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the task scheduling method based on Spark load balancing of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0049] The basic idea of ​​the present invention is to first obtain computing resource information, data storage information, and define the communication cost calculation method, and then unify the three into a model; then use the Hungarian algorithm to obtain the optimal solution under the global state of the established unified model; finally, the obtained task scheduling table is built into Spark's built-in task scheduler, and the original scheduling scheme based on the greedy algorithm is blocked.

[0050] like Figure 1 As shown, the present invention provides a task scheduling method based on Spark load balancing, comprising the following steps:

[0051] (1) Get each executor on each node j in the Spark cluster And the data block b on each node j ji ={b ji0, b ji1, b ji2}, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j];

[0052] It should be noted that each data block is backed up three times by default.

[0053] (2) For each executor l on each node j in the Spark cluster obtained in step (1), obtain the communication overhead of the executor l processing each data block p in the Spark cluster;

[0054] cpl =d(p,j)×db p

[0055] where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p Represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2];

[0056] Since there are three copies of each individual data block, the network distance is:

[0057] d(p,j)=min{d(p0,j),d(p1,j),d(p2,j)}

[0058] The advantage of the above steps (1) to (2) is that the executor and data block information of the Spark cluster is obtained, and the communication overhead between them is obtained, providing a global perspective.

[0059] (3) Create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp It is the communication overhead of executor l obtained in step (2) processing the p-th data block in the Spark cluster;

[0060] (4) Create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution;

[0061] It should be noted that a data block can only be executed on a certain executor. Similarly, an executor can only execute one data block at a time. Therefore, for the matrix element Matri2 pl The constraints are as follows:

[0062]

[0063] (5) According to the matrix Matrix1 obtained in step (3) and the matrix Matrix2 obtained in step (4), the objective function MinCost(Matrix2,Matrix1) of the minimum total communication cost is obtained;

[0064] Specifically, the objective function of minimum total communication cost is equal to:

[0065]

[0066] The vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the p-th column in matrix Matrix1;

[0067] (6) Use the Hungarian algorithm to solve the objective function of the minimum total communication cost obtained in step (5) to obtain the task scheduling allocation table.

[0068] The advantage of the above steps (3) to (6) is that the optimal solution of the communication consumption matrix is ​​obtained through the Hungarian algorithm, and a task scheduling allocation table is generated, so that the tasks to be assigned correspond to the executors one by one, avoiding the problem of executor idling.

[0069] (7) According to the task scheduling allocation table obtained in step (6), all pending tasks are scheduled to all executors in the Spark cluster to obtain the scheduling results.

[0070] Specifically, step (7) includes the following steps:

[0071] (7-1) Set counter i = 0;

[0072] (7-2) Determine whether i is greater than the total number of tasks to be processed. If so, the process ends; otherwise, proceed to step (7-3);

[0073] (7-3) Determine whether the total number of idle executors in the current Spark cluster is greater than 80% of the total number of executors in the Spark cluster. If so, proceed to step (7-4), otherwise return to step (7-1);

[0074] (7-4) Allocate the i-th pending task to the corresponding executor for execution according to the task scheduling allocation table, delete the i-th pending task from the task scheduling allocation table, and then proceed to step (7-5);

[0075] (7-5) Set counter i=i+1 and return to step (7-2);

[0076] The advantage of this step is that a new round of task scheduling allocation table is generated when the idle number of idle executors reaches 80%, thereby ensuring a high degree of adaptation between tasks and executors.

[0077] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A task scheduling method based on Spark load balancing, characterized in that: The following steps are involved: (1) Get each executor on each node j in the Spark cluster And the data block b on each node j ji ={b ji0 ,b ji1 ,b ji2 }, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j]; (2) For each executor l on each node j in the Spark cluster obtained in step (1), obtain the communication overhead of the executor l processing each data block p in the Spark cluster; c pl =d(p,j)×db p where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p Represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2]; (3) Create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp It is the communication overhead of executor l obtained in step (2) processing the p-th data block in the Spark cluster; (4) Create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution; (5) According to the matrix Matrix1 obtained in step (3) and the matrix Matrix2 obtained in step (4), the objective function of the minimum total communication cost MinCost(Matrix2,Matrix1) is obtained; the objective function of the minimum total communication cost is equal to: The vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the p-th column in matrix Matrix1; (6) Using the Hungarian algorithm to solve the objective function of the minimum total communication cost obtained in step (5) to obtain the task scheduling allocation table; (7) According to the task scheduling allocation table obtained in step (6), all pending tasks are scheduled to all executors in the Spark cluster to obtain the scheduling results.

2. The task scheduling method based on Spark load balancing according to claim 1, characterized in that: The network distance from the pth data block to node j is: d(p,j)=min{d(p0,j),d(p1,j),d(p2,j)} Where min means taking the minimum value, d(p0,j) represents the network distance from the first replica in the p-th data block to node j, d(p1,j) represents the network distance from the second replica in the p-th data block to node j, and d(p2,j) represents the network distance from the third replica in the p-th data block to node j.

3. The task scheduling method based on Spark load balancing according to claim 1 or 2, characterized in that: Matrix element Matri2 lp The constraints are as follows:

4. The task scheduling method based on Spark load balancing according to claim 3 is characterized in that: Step (7) comprises the following steps: (7-1) Set counter i = 0; (7-2) Determine whether i is greater than the total number of tasks to be processed. If so, the process ends; otherwise, proceed to step (7-3); (7-3) Determine whether the total number of idle executors in the current Spark cluster is greater than 80% of the total number of executors in the Spark cluster. If so, proceed to step (7-4), otherwise return to step (7-1); (7-4) Allocate the i-th pending task to the corresponding executor for execution according to the task scheduling allocation table, delete the i-th pending task from the task scheduling allocation table, and then proceed to step (7-5); (7-5) Set counter i=i+1 and return to step (7-2).

5. A task scheduling system based on Spark load balancing, characterized in that: include: The first module is used to obtain each executor on each node j in the Spark cluster And the data block b on each node j ji ={b ji0, b ji1, b ji2 }, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j]; The second module is used to obtain the communication overhead of each executor l on each node j in the Spark cluster obtained in the first module for processing each data block p in the Spark cluster; c pl =d(p,j)×db p where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p Represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2]; The third module is used to create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp This is the communication overhead of executor l processing the p-th data block in the Spark cluster obtained in the second module; The fourth module is used to create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution; The fifth module is used to obtain the objective function MinCost(Matrix2,Matrix1) of the minimum total communication cost based on the matrix Matrix1 obtained in the third module and the matrix Matrix2 obtained in the fourth module. The objective function of the minimum total communication cost is equal to: The vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the p-th column in matrix Matrix1; The sixth module is used to solve the objective function of the minimum total communication cost obtained in the fifth module using the Hungarian algorithm to obtain a task scheduling allocation table; The seventh module is used to schedule all pending tasks to all executors in the Spark cluster according to the task scheduling allocation table obtained in the sixth module to obtain the scheduling results.

Images