A low-cost Spark executor placement method based on BFD-VNS algorithm

By constructing a cost model for the Spark cluster environment and the BFD-VNS algorithm, the placement strategy of Spark executors was optimized, solving the problems of high cost and low resource utilization, and achieving cost minimization and improved resource utilization.

CN114064281BActive Publication Date: 2024-11-19北京奕辰科技有限公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111382902.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-11-22
Publication Date
2024-11-19
Estimated Expiration
2041-11-22

AI Technical Summary

Technical Problem

Existing Spark executor placement strategies suffer from high costs and low resource utilization, necessitating optimization of resource usage costs and improvement of resource utilization.

Method used

A cost model for a cluster environment is constructed using a BFD-VNS algorithm. Feasible solutions are obtained through a descending best-fit algorithm, and a variable neighborhood search algorithm is used for deep optimization to finally determine the optimal Spark executor placement strategy.

Benefits of technology

It reduces the cost of using Spark clusters, improves resource utilization, and enhances the efficiency of native placement strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114064281B_ABST
    Figure CN114064281B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of big data processing, and specifically relates to a low-cost Spark executor placement method based on a BFD-VNS algorithm, comprising constructing a cost model of Spark jobs in a cluster environment to establish an objective function to minimize the use cost of a Spark cluster; using a descending best fit algorithm to obtain a feasible solution for Spark executor placement; using the obtained feasible solution as an initial solution, and using a variable neighborhood search algorithm for deep optimization to obtain the optimal Spark executor placement; placing executors according to the obtained optimal executor placement; the present invention reduces the use cost of the entire cluster and improves the problems existing in the Spark native placement strategy.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of big data processing, and in particular relates to a low-cost Spark executor placement method based on a BFD-VNS algorithm. Background Art

[0002] As the Internet develops rapidly, the amount of data in various countries and even the world is growing exponentially. According to statistics, in 2010, my country's share of global data exceeded 10%. It is expected that by 2025, my country's total data volume will jump to the first place in the world, and the global share is expected to reach more than 27%. China is striving to become a real data resource power. Major enterprises and institutions and various business fields generate massive amounts of data every minute and every second, which brings with it a large amount of data analysis needs. Therefore, fast and effective data analysis and processing is crucial, which is inseparable from the increasingly mature data processing framework.

[0003] Currently, Spark has become the mainstream data processing framework, and data processing cannot be separated from the support of computing resources. It is precisely because computing resources play a pivotal role in the data processing process that it is crucial to use computing resources reasonably and effectively and improve resource utilization. It is also meaningful to optimize the cost of resource use in the data processing process.

[0004] Although there are many studies on cost optimization, few studies have been conducted in depth from the perspective of executor placement strategy. In addition, in actual production, Spark's native executor placement strategy also has the problems of high cost and low resource utilization. Therefore, optimizing the resource usage cost of the cluster by adjusting the executor placement strategy is an urgent problem to be solved. Summary of the invention

[0005] To solve the above problems, the present invention proposes a low-cost Spark executor placement method based on the BFD-VNS algorithm.

[0006] S1: Build a cost model for Spark jobs in a cluster environment and establish an objective function to minimize the cost of using the Spark cluster;

[0007] S2: Use the descending best fit algorithm to find a feasible solution for Spark executor placement;

[0008] S3: The feasible solution is used as the initial solution, and the variable neighborhood search algorithm is used for deep optimization to obtain the optimal Spark executor placement;

[0009] S4: placing the actuators according to the obtained optimal actuator placement.

[0010] Furthermore, the objective function established to minimize the Spark cluster usage cost includes:

[0011]

[0012] Constraints:

[0013]

[0014]

[0015] Among them, F k represents the fixed cost of using a virtual node of type k, η represents the set of types of all virtual nodes in the Spark cluster, and δ k represents the set of all virtual nodes of type k; m jk is a binary decision variable indicating whether the j-th virtual node of type k is used, which takes the value 1 if it is used and 0 otherwise; n ijk To indicate whether executor i is placed on the jth virtual node of type k, if yes, it takes the value 1, otherwise it takes the value 0; Represents the set of executors required to run the current Spark job; μ cpu Indicates the CPU resource requirements of a certain executor of the current Spark job. represents the available CPU resources of the jth virtual node of type k; μ mem Indicates the memory resource requirement of a certain executor of the current Spark job. Represents the available memory resources of the j-th virtual node of type k.

[0016] Furthermore, the following steps are used to find a feasible solution for Spark executor placement using the descending best fit algorithm:

[0017] 101. Arrange all virtual nodes in the cluster in ascending order according to resource availability;

[0018] 102. Put the executors in order according to the established order of the virtual nodes. For each virtual node, minimize the remaining available resources of the node under the premise of satisfying the resource capacity constraint. If the remaining available resources of the current node cannot meet the resource requirements of the next executor, put the next executor into the next virtual node that meets its resource requirements.

[0019] 103. If all executors required by the current Spark job can be placed in the virtual nodes of the cluster, the virtual node order of the placed executors is returned; otherwise, failure is returned and the available resources of the cluster need to be adjusted before re-execution.

[0020] Furthermore, the calculation of the resource availability of the virtual node includes the following steps:

[0021] Normalize the available CPU resources and available memory resources of the virtual node;

[0022] Determine the objective weights of available CPU resources and available memory resources based on the standard deviation method;

[0023] The CPU resources and available memory resources are weighted and fused with the corresponding objective weights to obtain the resource availability of the virtual node.

[0024] Furthermore, the available CPU resources and available memory resources of the virtual node are normalized, including:

[0025]

[0026]

[0027] in, Indicates the available CPU resources of the jth virtual node of type k; Represents the normalized available memory resources of the j-th virtual node of type k.

[0028] Furthermore, the resource availability of a virtual node is expressed as:

[0029]

[0030] Among them, RA jk W represents the resource availability of the jth virtual node of type k; cpu Indicates the objective weight of CPU resources; W mem Indicates the objective weight of memory resources.

[0031] Furthermore, the objective weight of CPU resources W cpu It is expressed as:

[0032]

[0033] The objective weight of memory resources W mem It is expressed as:

[0034]

[0035]

[0036] in, Indicates the average value of available CPU resources of all virtual nodes; Represents the average value of available memory resources of all virtual nodes; |δ k| represents the number of virtual nodes in the set consisting of all virtual nodes of type k.

[0037] The beneficial effects of the present invention are: building a cost model for Spark jobs in a cluster environment, with the optimization goal of minimizing the cost of using the Spark cluster; obtaining the optimal solution for the Spark executor placement strategy through the BFD algorithm and the VNS algorithm, reducing the cost of using the entire cluster and improving the problems existing in the Spark native placement strategy. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the Spark executor placement method based on the BFD-VNS algorithm of the present invention;

[0039] Figure 2 It is a flow chart of the BFD algorithm in the BFD-VNS algorithm of the present invention;

[0040] Figure 3 It is a flow chart of the VNS algorithm in the BFD-VNS algorithm of the present invention;

[0041] Figure 4 It is a flow chart of the VND process in the VNS algorithm in the BFD-VNS algorithm of the present invention;

[0042] Figure 5 This is the Spark cluster node architecture diagram;

[0043] Figure 6 Place a strategy graph for the Spark executor. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The present invention provides a low-cost Spark executor placement method based on the BFD-VNS algorithm, comprising the following steps:

[0046] S1: Build a cost model for Spark jobs in a cluster environment and establish an objective function to minimize the cost of using the Spark cluster;

[0047] S2: Use the descending best fit algorithm to find a feasible solution for Spark executor placement;

[0048] S3: The feasible solution is used as the initial solution, and the variable neighborhood search algorithm is used for deep optimization to obtain the optimal Spark executor placement;

[0049] S4: placing the actuators according to the obtained optimal actuator placement.

[0050] Figure 6 The figure shows the Spark executor placement strategy diagram. Different placement strategies will have different degrees of impact on the cluster usage cost and resource utilization. Then, in order to better adjust the Spark executor placement strategy, reduce the cluster usage cost, and improve the cluster resource utilization, the present invention proposes a low-cost Spark executor placement method based on the BFD-VNS algorithm. This embodiment divides the method into four parts, such as Figure 1 As shown, including:

[0051] 1. Build a cost model for Spark jobs in a cluster environment.

[0052] In this embodiment, a cost model of Spark jobs in a cluster environment is constructed. The overall optimization goal is to minimize the cost of using the Spark cluster. The cost of using the Spark cluster can be summarized as the total cost of all virtual nodes used. The virtual nodes are virtual machines. Virtual nodes have resource availability restrictions. The resources include CPU resources and memory resources. Therefore, consider setting constraints on the objective function under the cost model. The overall optimization goal is

[0053]

[0054] Where minC represents the usage cost of the entire Spark cluster, that is, the sum of the costs of all used virtual nodes; F k represents the fixed cost of using a virtual node of type k; m jk is a binary decision variable indicating whether the j-th virtual node of type k is used. If yes, it takes the value 1, otherwise it takes the value 0.

[0055] The constraints set by the objective function include Spark executor placement constraints and resource capacity constraints. The Spark executor placement constraints are:

[0056]

[0057] Among them, η represents the set of types of all virtual nodes in the Spark cluster, δ k Represents the set of all virtual nodes of type k; Indicates the set of executors required to run the current Spark job; n ijkIndicates whether executor i is placed on the jth virtual node of type k. If yes, it takes the value 1, otherwise it takes the value 0.

[0058] The resource capacity constraints are:

[0059]

[0060]

[0061] Among them, μ cpu Indicates the CPU resource requirements of a certain executor of the current Spark job. represents the available CPU resources of the jth virtual node of type k; μ mem Indicates the memory resource requirement of a certain executor of the current Spark job. Represents the available memory resources of the j-th virtual node of type k.

[0062] Second, use the descending best fit algorithm to find a feasible solution for the Spark executor placement strategy.

[0063] The following steps are used to find a feasible solution for Spark executor placement using the descending best fit algorithm:

[0064] 101. Arrange all virtual nodes in the cluster in ascending order according to resource availability;

[0065] 102. Put the executors in order according to the established order of the virtual nodes. For each virtual node, minimize the remaining available resources of the node under the premise of satisfying the resource capacity constraint. If the remaining available resources of the current node cannot meet the resource requirements of the next executor, put the next executor into the next virtual node that meets its resource requirements.

[0066] 103. If all executors required by the current Spark job can be placed in the virtual nodes of the cluster, the virtual node order of the placed executors is returned; otherwise, failure is returned and the available resources of the cluster need to be adjusted before re-execution.

[0067] In this embodiment, when this step is specifically implemented, Figure 2 , which can be briefly summarized as the following steps:

[0068] Step 1: Sorting;

[0069] Arrange all virtual nodes in the cluster in ascending order according to resource availability.

[0070] Step 2: Actuator placement;

[0071] The executors are placed in sequence according to the established order of virtual nodes. For each virtual node, the remaining available resources of the node are made as small as possible while satisfying the resource capacity constraint. If the remaining available resources of the current node cannot meet the resource requirements of the next executor, the next executor is placed in the next virtual node that meets its resource requirements.

[0072] Step 3: Get the solution;

[0073] If all executors required by the current Spark job can be placed in the virtual nodes of the cluster, the virtual node order of the placed executors is returned; otherwise, failure is returned and the available resources of the cluster need to be adjusted before re-execution.

[0074] The resource availability index in step 1 is obtained by normalizing and objectively weighting the available CPU resources and available memory resources of the current virtual node.

[0075] Furthermore, the data normalization process can be expressed as:

[0076]

[0077]

[0078] The method for determining objective weights can be expressed as:

[0079]

[0080]

[0081] in, Indicates the average value of available CPU resources of all virtual nodes; Represents the average value of available memory resources of all virtual nodes; |δ k | represents the number of virtual nodes in the set consisting of all virtual nodes of type k.

[0082] The resource availability index can be expressed as:

[0083]

[0084] 3. Use the variable neighborhood search algorithm to deeply optimize the better solution of the descending best adaptation algorithm.

[0085] In this embodiment, the variable neighborhood search algorithm is used to deeply optimize the better solution of the descending best adaptation algorithm. Figure 3 As shown, it can be briefly summarized into the following steps:

[0086] S311: Define the neighborhood structure set N for shaking s(s=1,2,……,s max ), let the optimal solution obtained by the BFD algorithm be the initial solution x=x0, and the objective function be y(x);

[0087] S312: Assume the global optimal solution x m =x0,s=1;

[0088] S313: Shaking process: from x m Randomly select a solution x′ from the sth neighborhood structure;

[0089] S314: Local search (VND) process: perform local search on the solution x′ to find the local optimal solution x″;

[0090] S315: If y(x″)<y(x m ), then x m =x″, s=1, return to S313; otherwise, s=s+1, execute S316;

[0091] S316: If s≤s max , then return to S313; otherwise, execute S317;

[0092] S317: Output the global optimal solution x m .

[0093] The specific VND process described in step S314 is as follows: Figure 4 As shown, it can be briefly summarized into the following steps:

[0094] S3141: Define the neighborhood structure set N for VND l (l=1,2,……,l max );

[0095] S3142: set l=1;

[0096] S3143: traverse and search for the local optimal solution x″ from the lth neighborhood structure of the solution x′;

[0097] S3144: If y(x″)<y(x′), then x′=x″, l=1, and return to S3143; otherwise, l=l+1, and execute S3145;

[0098] l≤l max

[0099] S3145: If yes, return to S3143; otherwise, execute S3146;

[0100] S3146: Output the local optimal solution x″=x′.

[0101] The neighborhood structure can be obtained by the following neighborhood action perturbations:

[0102] 2-optimization neighborhood action: Randomly select two points a and b, and reverse the order of the virtual nodes between a and b (including a and b). If the order of the virtual nodes is 1→2→3→4→5, randomly select nodes 2 and 4, and the new order after the reverse processing is 1→4→3→2→5;

[0103] Exchange neighborhood action: randomly select element a from the set of virtual nodes where executors have been placed, randomly select element b from the set of virtual nodes where executors have not been placed, and exchange a and b. If the set of virtual nodes where executors have been placed is {1,2,3,4}, and the set of virtual nodes where executors have not been placed is {5,6}, randomly select nodes 2 and 6, then the set of virtual nodes where executors have been placed after the exchange is {1,6,3,4}, and the set of virtual nodes where executors have not been placed is {5,2}.

[0104] Insert neighborhood action: randomly select element a from the set of virtual nodes where no actuators are placed, delete it from the set and add it to the set of virtual nodes where actuators are placed. For example, if the set of placed virtual nodes is {1,2,3,4} and the set of unplaced virtual nodes is {5,6}, and node 5 is randomly selected, then the set of placed virtual nodes after insertion is {1,2,3,4,5} and the set of unplaced virtual nodes is {6}.

[0105] Remove neighborhood action: randomly select element a from the set of virtual nodes where the executor has been placed, delete it from the set and add it to the set of virtual nodes where the executor has not been placed. For example, if the set of placed virtual nodes is {1,2,3,4} and the set of unplaced virtual nodes is {5,6}, and node 1 is randomly selected, the set of placed virtual nodes after deletion is {2,3,4} and the set of unplaced virtual nodes is {5,6,1}.

[0106] In addition, all executor transfers between virtual nodes caused by neighborhood actions must meet resource capacity constraints.

[0107] 4. Place actuators according to the optimal actuator placement strategy.

[0108] like Figure 6 As shown in the figure, the different executor placement strategies are shown, among which the optimal placement strategy will be applied to the cluster for deployment. The node architecture diagram of the entire cluster is as follows Figure 5 shown.

[0109] The method of the present invention deeply optimizes the better solution of the placement strategy on the basis of the descending best adaptation algorithm, obtains different neighborhood structures through different neighborhood actions, and then alternately searches the neighborhood structures, and finally tends to the global optimal solution, so as to achieve the purpose of reducing the cluster usage cost and improving resource utilization.

[0110] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A low-cost Spark executor placement method based on the BFD-VNS algorithm, characterized in that: The following steps are involved: S1: Build a cost model for Spark jobs in a cluster environment to establish an objective function to minimize the cost of using the Spark cluster, namely: Constraints: Among them, F k represents the fixed cost of using a virtual node of type k, η represents the set of types of all virtual nodes in the Spark cluster, and δ k represents the set of all virtual nodes of type k; m jk is a binary decision variable indicating whether the j-th virtual node of type k is used, which takes the value 1 if it is used and 0 otherwise; n ijk To indicate whether executor i is placed on the jth virtual node of type k, if yes, it takes the value 1, otherwise it takes the value 0; Represents the set of executors required to run the current Spark job; μ cpu Indicates the CPU resource requirements of a certain executor of the current Spark job. represents the available CPU resources of the jth virtual node of type k; μ mem Indicates the memory resource requirement of a certain executor of the current Spark job. Represents the available memory resources of the j-th virtual node of type k; S2: Use the descending best fit algorithm to find a feasible solution for Spark executor placement, including the following steps:

101. Arrange all virtual nodes in the cluster in ascending order according to resource availability; 102. Put the executors in order according to the established order of the virtual nodes. For each virtual node, minimize the remaining available resources of the node under the premise of satisfying the resource capacity constraint. If the remaining available resources of the current node cannot meet the resource requirements of the next executor, put the next executor into the next virtual node that meets its resource requirements.

103. If all executors required by the current Spark job can be placed in the virtual nodes of the cluster, the virtual node order of the placed executors is returned; otherwise, failure is returned and the available resources of the cluster need to be adjusted before re-execution; S3: The feasible solution is used as the initial solution, and the variable neighborhood search algorithm is used for deep optimization to obtain the optimal Spark executor placement, including:

201. Obtain the neighborhood structure set N for disturbance s , the initial solution is taken as the optimal solution obtained by the BFD algorithm, and the objective function is y(x), where y(x) represents the cost of using the Spark cluster when the placement strategy is x; 202. Let x m As the global optimal solution, and the initial solution as the current global optimal solution x m The value of , let s = 1; 203. From the global optimal solution x m The sth neighborhood structure set N s Randomly select a solution x'; 204. Perform a local search on x' to find the local optimal solution x', that is, find a placement strategy in the neighborhood structure set that makes the Spark cluster use the lowest cost as the local optimal solution; 205. Calculate and determine whether y(x') < y(x m ), if so, let x m = x" and return to step 203; 206. Otherwise, let s = s + 1 and determine whether s ≤ s max If yes, return to step 203; otherwise, output the current global optimal solution; S4: placing the actuators according to the obtained optimal actuator placement.

2. According to a low-cost Spark executor placement method based on the BFD-VNS algorithm according to claim 1, it is characterized in that: The calculation of resource availability of a virtual node includes the following steps: Normalize the available CPU resources and available memory resources of the virtual node; Determine the objective weights of available CPU resources and available memory resources based on the standard deviation method; The CPU resources and available memory resources are weighted and fused with the corresponding objective weights to obtain the resource availability of the virtual node.

3. According to the low-cost Spark executor placement method based on the BFD-VNS algorithm of claim 2, it is characterized in that: Normalize the available CPU resources and available memory resources of the virtual node, including: Among them, η represents the set of types of all virtual nodes in the Spark cluster, δ k Represents the set of all virtual nodes of type k; represents the available CPU resources of the j-th virtual node of type k, Indicates the normalized available CPU resources of the j-th virtual node of type k; represents the normalized available memory resources of the j-th virtual node of type k, Represents the available memory resources of the j-th virtual node of type k.

4. According to a low-cost Spark executor placement method based on the BFD-VNS algorithm according to claim 2, it is characterized in that: The resource availability of a virtual node is expressed as: Among them, RA jk W represents the resource availability of the jth virtual node of type k; cpu Indicates the objective weight of CPU resources; W mem Represents the objective weight of memory resources; Indicates the available CPU resources of the jth virtual node of type k; represents the normalized available memory resources of the jth virtual node of type k; η represents the set of types of all virtual nodes in the Spark cluster, δ k Represents the set of all virtual nodes of type k.

5. A low-cost Spark executor placement method based on the BFD-VNS algorithm according to claim 2 or 4, characterized in that: The objective weight of CPU resources W cpu It is expressed as: The objective weight of memory resources W mem It is expressed as: in, Indicates the average value of available CPU resources of all virtual nodes; Represents the average value of available memory resources of all virtual nodes; |δ k | represents the number of virtual nodes in the set consisting of all virtual nodes of type k.

6. The low-cost Spark executor placement method based on the BFD-VNS algorithm according to claim 1 is characterized in that: The neighborhood structure set of a solution is obtained by performing neighborhood action perturbations on the solution. The neighborhood actions include at least 2-optimization neighborhood actions, Exchange neighborhood actions, Insert neighborhood actions, and Remove neighborhood actions. The executor transfers between virtual nodes caused by all neighborhood actions must meet resource capacity constraints before they can be added to the neighborhood structure set.