Task scheduling method and system based on Spark load balancing

Abstract

The invention discloses a task scheduling method based on Spark load balancing, which comprises the following steps: firstly, acquiring computing resource information and data storage information, defining a communication cost computing mode, and then carrying out unified modeling on the computing resource information, the data storage information and the communication cost computing mode; then solving an optimal solution of the established unified model in a global state by adopting a Hungary algorithm; and finally, the obtained task scheduling table is built in a task scheduler of the Spark, and an original scheduling scheme based on a greedy algorithm is shielded. According to the method, the technical problem that the final scheduling strategy is caught in local optimum and further the scheduling performance is influenced due to the fact that the current optimum is selected in each step of execution in the existing scheduling method using a greedy algorithm and a global perspective is lacked can be solved; and the method only considers performance improvement caused by high locality level, but does not consider performance loss caused by waiting time, so that the technical problems of high total communication consumption and low CPU (Central Processing Unit) utilization rate can be caused.

Description

Technical Field

[0001] The present invention belongs to the technical field of distributed parallel computing performance optimization, and more specifically, relates to a task scheduling method and system based on Spark load balancing. Background Art

[0002] With the development of the Internet, the scale of data has exploded, the amount of data that needs to be processed is getting bigger and bigger, and the growth of CPU computing power lags far behind the growth of data. The big data open source ecosystem is becoming more and more abundant, but the diversified computing engines and open source components also bring the problem of difficulty in improving data processing performance throughout the life cycle. How to process as much data as possible with limited resources and make the system load balanced is an area where the current task scheduling algorithm of the big data computing engine Spark urgently needs to be optimized.

[0003] The mainstream scheduling method used by Spark at the current stage is a scheduling method based on a greedy strategy. The system sets a default waiting time of 3s, that is, whenever a new task arrives, it will be given priority to execute on an executor with a higher local row level (data is closer to the computing resources). If the executors with a higher locality level are currently busy, they will wait for 3s. If there is still no executor that meets this level released, the locality level of the task will be downgraded.

[0004] However, the above scheduling method based on greedy strategy has some non-negligible defects:

[0005] First, since this method selects the current optimal solution at each step and lacks a global perspective, the final scheduling strategy falls into a local optimal situation, which in turn affects the scheduling performance.

[0006] Second, this method only considers the performance improvement brought by the high locality level, but does not consider the performance loss caused by the waiting time (that is, even if the node is idle, it cannot run the task due to insufficient locality level), which will cause the overall communication consumption to be large and the CPU utilization to be low;

[0007] Third, this method will try to schedule tasks to run as long as there is an idle executor, without considering the compatibility between the task and the executor. As a result, when a subsequent task that is compatible with the executor (i.e., a high locality level) appears, the executor is already occupied, ultimately extending the overall running time of the program. Summary of the invention

[0008] In response to the above defects or improvement needs of the prior art, the present invention provides a task scheduling method and system based on Spark load balancing, which aims to solve the technical problems that the existing scheduling method using a greedy algorithm selects the current optimal one at each step of execution and lacks a global perspective, thus causing the final scheduling strategy to fall into a local optimal situation, thereby affecting the scheduling performance; and the technical problems that the method only considers the performance improvement brought by the high level of locality but does not consider the performance loss brought by the waiting time, thus causing the overall communication consumption to be large and the CPU utilization to be low; and the method will try to arrange the task to run as long as there is an idle executor, without considering the adaptation between the task and the executor, so that when a subsequent task adapted to the executor appears, the executor has been occupied, which ultimately leads to the technical problem of extending the overall running time of the program.

[0009] To achieve the above object, according to one aspect of the present invention, a task scheduling method based on Spark load balancing is provided, comprising the following steps:

[0010] (1) Get each executor on each node j in the Spark cluster And the data block b on each node j ji = {b ji0, b ji1 ,b ji2}, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j];

[0011] (2) For each executor l on each node j in the Spark cluster obtained in step (1), obtain the communication overhead of the executor l in processing each data block p in the Spark cluster;

[0012] c pl =d(p,j)×db p

[0013] where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2];

[0014] (3) Create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lpIt is the communication overhead of executor l processing the pth data block in the Spark cluster obtained in step (2);

[0015] (4) Create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution;

[0016] (5) According to the matrix Matrix1 obtained in step (3) and the matrix Matrix2 obtained in step (4), obtain the objective function MinCost (Matrix2, Matrix1) of the minimum total communication cost;

[0017] (6) Use the Hungarian algorithm to solve the objective function of the minimum total communication cost obtained in step (5) to obtain the task scheduling allocation table.

[0018] (7) According to the task scheduling allocation table obtained in step (6), all pending tasks are scheduled to all executors in the Spark cluster to obtain a scheduling result.

[0019] Preferably, the network distance from the pth data block to node j is:

[0020] d(p,j)=min{d(p0,j),d(p1,j),d(p2,j)}

[0021] Where min means taking the minimum value, d(p0,j) represents the network distance from the first replica in the p-th data block to node j, d(p1,j) represents the network distance from the second replica in the p-th data block to node j, and d(p2,j) represents the network distance from the third replica in the p-th data block to node j.

[0022] Preferably, the matrix element Matri2 pl The constraints are as follows:

[0023]

[0024] Preferably, the objective function of the minimum total communication cost is equal to:

[0025]

[0026] Where the vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the p-th column in matrix Matrix1.

[0027] Preferably, step (7) comprises the following steps:

[0028] (7-1) Set counter i=0;

[0029] (7-2) Determine whether i is greater than the total number of tasks to be processed. If so, the process ends, otherwise proceed to step (7-3);

[0030] (7-3) Determine whether the total number of idle executors in the current Spark cluster is greater than 80% of the total number of executors in the Spark cluster. If so, proceed to step (7-4), otherwise return to step (7-1);

[0031] (7-4) Allocate the i-th pending task to the corresponding executor for execution according to the task scheduling allocation table, delete the i-th pending task from the task scheduling allocation table, and then proceed to step (7-5);

[0032] (7-5) Set counter i=i+1 and return to step (7-2).

[0033] According to another aspect of the present invention, a task scheduling system based on Spark load balancing is provided, comprising:

[0034] The first module is used to obtain each executor on each node j in the Spark cluster And the data block b on each node j ji = {b ji0, b ji1, b ji2}, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j];

[0035] The second module is used to obtain the communication overhead of each executor l on each node j in the Spark cluster obtained in the first module for processing each data block p in the Spark cluster by the executor l;

[0036] c pl =d(p,j)×db p

[0037] where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2];

[0038] The third module is used to create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp It is the communication overhead of executor l processing the pth data block in the Spark cluster obtained in the second module;

[0039] The fourth module is used to create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution;

[0040] The fifth module is used to obtain the objective function MinCost (Matrix2, Matrix1) of the minimum total communication cost according to the matrix Matrix1 obtained by the third module and the matrix Matrix2 obtained by the fourth module;

[0041] The sixth module is used to solve the objective function of the minimum total communication cost obtained in the fifth module using the Hungarian algorithm to obtain a task scheduling allocation table.

[0042] The seventh module is used to schedule all pending tasks to all executors in the Spark cluster according to the task scheduling allocation table obtained in the sixth module to obtain a scheduling result.

[0043] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects compared with the prior art:

[0044] (1) Due to the adoption of steps (1) and (2), the present invention obtains the information of all executors and data blocks of the Spark cluster and calculates the communication overhead between them, thereby obtaining scheduling information from a global perspective, thereby solving the technical problem of local optimality existing in the existing scheduling method based on the greedy strategy;

[0045] (2) Since the present invention adopts steps (3) to (6), it obtains the communication consumption matrix from a global perspective and obtains a unique solution through the Hungarian algorithm to form a task scheduling allocation table, so that the tasks to be assigned correspond to the executors one by one, and there is no problem of executor idling (the CPU is in an idle state, but no tasks are executed). Therefore, it can solve the technical problems of large overall communication consumption and low CPU utilization in the existing scheduling method based on the greedy strategy;

[0046] (3) The present invention adopts step (7), which generates a new round of task scheduling allocation table by controlling the idle number of executors to reach 80%. This ensures that most executors in the cluster participate in the allocation process during the generation of the task allocation table, thereby solving the problem of executors and tasks not matching each other, causing the overall program running time to be too long. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of the task scheduling method based on Spark load balancing of the present invention. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0049] The basic idea of ​​the present invention is to first obtain computing resource information, data storage information, and define the communication cost calculation method, and then unify the three models; then use the Hungarian algorithm to obtain the optimal solution under the global state for the established unified model; finally, the obtained task scheduling table is built into the task scheduler that comes with Spark, and the original scheduling scheme based on the greedy algorithm is shielded.

[0050] like Figure 1 As shown, the present invention provides a task scheduling method based on Spark load balancing, comprising the following steps:

[0051] (1) Get each executor on each node j in the Spark cluster And the data block b on each node j ji = {b ji0, b ji1, b ji2}, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j];

[0052] It should be noted that each data block is backed up three times by default.

[0053] (2) For each executor l on each node j in the Spark cluster obtained in step (1), obtain the communication overhead of the executor l in processing each data block p in the Spark cluster;

[0054] cpl =d(p,j)×db p

[0055] where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2];

[0056] Since there are 3 copies of each individual data block, the network distance is:

[0057] d(p,j)=min{d(p0,j),d(p1,j),d(p2,j)}

[0058] The advantage of the above steps (1) to (2) is that the executor and data block information of the Spark cluster is obtained, and the communication overhead between them is obtained, providing a global perspective.

[0059] (3) Create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp It is the communication overhead of executor l processing the pth data block in the Spark cluster obtained in step (2);

[0060] (4) Create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution;

[0061] It should be noted that a data block can only be executed on a certain executor. Similarly, an executor can only execute one data block at a time. Therefore, for the matrix element Matri2 pl The constraints are as follows:

[0062]

[0063] (5) According to the matrix Matrix1 obtained in step (3) and the matrix Matrix2 obtained in step (4), obtain the objective function MinCost (Matrix2, Matrix1) of the minimum total communication cost;

[0064] Specifically, the objective function of minimum total communication cost is equal to:

[0065]

[0066] Where the vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the pth column in matrix Matrix1;

[0067] (6) Use the Hungarian algorithm to solve the objective function of the minimum total communication cost obtained in step (5) to obtain the task scheduling allocation table.

[0068] The advantage of the above steps (3) to (6) is that the optimal solution of the communication consumption matrix is ​​obtained through the Hungarian algorithm, and a task scheduling allocation table is generated, so that the tasks to be assigned correspond to the executors one by one, avoiding the problem of executor idling.

[0069] (7) According to the task scheduling allocation table obtained in step (6), all pending tasks are scheduled to all executors in the Spark cluster to obtain a scheduling result.

[0070] Specifically, step (7) includes the following steps:

[0071] (7-1) Set counter i=0;

[0072] (7-2) Determine whether i is greater than the total number of tasks to be processed. If so, the process ends, otherwise proceed to step (7-3);

[0073] (7-3) Determine whether the total number of idle executors in the current Spark cluster is greater than 80% of the total number of executors in the Spark cluster. If so, proceed to step (7-4), otherwise return to step (7-1);

[0074] (7-4) Allocate the i-th pending task to the corresponding executor for execution according to the task scheduling allocation table, delete the i-th pending task from the task scheduling allocation table, and then proceed to step (7-5);

[0075] (7-5) Set counter i=i+1 and return to step (7-2);

[0076] The advantage of this step is that a new round of task scheduling allocation table is generated when the idle number of idle executors reaches 80%, thereby ensuring a high degree of adaptation between tasks and executors.

[0077] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A task scheduling method based on Spark load balancing, characterized in that: The following steps are involved: (1) Get each executor on each node j in the Spark cluster And the data block b on each node j ji = {b ji0, b ji1, b ji2 }, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j]; (2) For each executor l on each node j in the Spark cluster obtained in step (1), obtain the communication overhead of the executor l in processing each data block p in the Spark cluster; c pl =d(p,j)×db p where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2]; (3) Create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp It is the communication overhead of executor l processing the pth data block in the Spark cluster obtained in step (2); (4) Create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution; (5) According to the matrix Matrix1 obtained in step (3) and the matrix Matrix2 obtained in step (4), obtain the objective function MinCost (Matrix2, Matrix1) of the minimum total communication cost; (6) Use the Hungarian algorithm to solve the objective function of the minimum total communication cost obtained in step (5) to obtain the task scheduling allocation table. (7) According to the task scheduling allocation table obtained in step (6), all pending tasks are scheduled to all executors in the Spark cluster to obtain a scheduling result.

2. The task scheduling method based on Spark load balancing according to claim 1 is characterized in that: The network distance from the pth data block to node j is: d(p,j)=min{d(p0,j),d(p1,j),d(p2,j)} Where min means taking the minimum value, d(p0,j) represents the network distance from the first replica in the p-th data block to node j, d(p1,j) represents the network distance from the second replica in the p-th data block to node j, and d(p2,j) represents the network distance from the third replica in the p-th data block to node j.

3. The task scheduling method based on Spark load balancing according to claim 1 or 2, characterized in that: Matrix element Matri2 pl The constraints are as follows:

4. The task scheduling method based on Spark load balancing according to any one of claims 1 to 3, characterized in that: The objective function for minimum total communication cost is equal to: Where the vector Marix2V p Represents the pth column in matrix Matrix2, Matrix1V p T Represents the transposed matrix of the p-th column in matrix Matrix1.

5. The task scheduling method based on Spark load balancing according to claim 4 is characterized in that: Step (7) comprises the following steps: (7-1) Set counter i=0; (7-2) Determine whether i is greater than the total number of tasks to be processed. If so, the process ends, otherwise proceed to step (7-3); (7-3) Determine whether the total number of idle executors in the current Spark cluster is greater than 80% of the total number of executors in the Spark cluster. If so, proceed to step (7-4), otherwise return to step (7-1); (7-4) Allocate the i-th pending task to the corresponding executor for execution according to the task scheduling allocation table, delete the i-th pending task from the task scheduling allocation table, and then proceed to step (7-5); (7-5) Set counter i=i+1 and return to step (7-2).

6. A task scheduling system based on Spark load balancing, characterized in that: include: The first module is used to obtain each executor on each node j in the Spark cluster And the data block b on each node j ji = {b ji0, b ji1, b ji2 }, where l∈[1, the total number of all executors Num1 on node j], j∈[1, the total number of nodes in the Spark cluster], i∈[1, the total number of data blocks on node j]; The second module is used to obtain the communication overhead of each executor l on each node j in the Spark cluster obtained in the first module for processing each data block p in the Spark cluster by the executor l; c pl =d(p,j)×db p where c pl represents the communication overhead of executor l processing the pth data block in the Spark cluster; d(p,j) represents the network distance from the pth data block to node j, db p represents the size of the pth data block, and p∈[1, the total number of data blocks in the Spark cluster Num2]; The third module is used to create a matrix Matrix1 with Num1 rows and Num2 columns. The element Matri1 in the lth row and pth column of the matrix Matrix1 is lp It is the communication overhead of executor l processing the pth data block in the Spark cluster obtained in the second module; The fourth module is used to create a matrix Matrix2 with Num1 rows and Num2 columns. The element Matri2 in the lth row and pth column of the matrix lp The value is 0 or 1, where 1 means that the data block p is placed on executor l for execution, and 0 means that the data block p is not placed on executor l for execution; The fifth module is used to obtain the objective function MinCost (Matrix2, Matrix1) of the minimum total communication cost according to the matrix Matrix1 obtained by the third module and the matrix Matrix2 obtained by the fourth module; The sixth module is used to solve the objective function of the minimum total communication cost obtained in the fifth module using the Hungarian algorithm to obtain a task scheduling allocation table. The seventh module is used to schedule all pending tasks to all executors in the Spark cluster according to the task scheduling allocation table obtained in the sixth module to obtain a scheduling result.

Images