Low-cost Spark task scheduling method based on NOA-LightGBM hybrid algorithm

Abstract

The invention belongs to the field of big data, and particularly relates to a low-cost Spark task scheduling method based on an NOA-LightGBM hybrid algorithm, and the method comprises the steps: building a cost model of Spark operation, and taking the improvement of the utilization rate of cluster resources and the minimization of the use cost of a Spark cluster as objective functions; the hyper-parameter combination of the LightGBM algorithm is optimized by adopting a star graffiti optimization algorithm (NOA), and the optimal combination of the LightGBM hyper-parameters is determined; using the obtained optimal parameter combination as a hyper-parameter of a LightGBM algorithm, training a prediction model, and using the prediction model to obtain a node of a person who may fall behind in the running process of the Spark application; in combination with node performance, performing priority ranking on all nodes, and performing Spark task scheduling according to node priorities; according to the method, the utilization rate of cluster resources is increased, the use cost of the cluster is reduced, and the problems existing in a native Spark task scheduling strategy are solved.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm. Background Art

[0002] As an emerging strategic resource, big data has become a crucial engine driving socioeconomic development. In recent years, with the commercial application of 5G technology, the rapid development of artificial intelligence, cloud computing, the Internet of Things, and the accelerated transformation of traditional industries towards digitalization and informatization, data growth has shown an exponential upward trend. These technological innovations have not only driven social progress but also led to a massive explosion in global data volumes, presenting unprecedented opportunities and challenges across all industries. IDC's latest report, which provides a five-year forecast for the global datasphere, predicts that global data volume will reach 393.8ZB by 2028, a 9.8-fold increase compared to 2018. The amount of data generated between 2024 and 2028 will be at least 2.2 times the total amount generated over the previous decade. China's data volume is projected to grow from 23.88ZB in 2022 to 76.6ZB in 2027, with an average annual growth rate of 26.3%. Government, media, professional services, retail, healthcare, and finance are the main distribution areas. They have more data, which also brings greater pressure on storage governance and analysis management. This also creates more opportunities for data management services to activate data and tap into commercial and social value.

[0003] Spark is a fast and versatile large-scale data processing engine. With its fast memory-based processing speed and efficient iterative computing features, it has become an indispensable member of the big data processing framework. It is deeply favored by many enterprises and is widely used in actual production.

[0004] During the running of a Spark job, some tasks in certain Spark tasks may run slowly, thus slowing down the overall task running speed. During the running of an application, Spark's inherent speculative execution mechanism can help identify and speculate on slow or failed tasks. However, excessive use of speculation may lead to duplicate calculations. Spark may execute the original task and the speculative task simultaneously, resulting in duplicate calculations and waste of resources, causing a decrease in resource utilization. Spark's built-in FIFO and FAIR scheduling policies use an indifferent random placement method when scheduling tasks, distributing tasks to all available computing nodes. This method ignores the heterogeneity between nodes, the distribution of data to be calculated, and the problem of cluster resource usage costs. Simply allocating tasks and occupying resources without considering the full utilization of resources will not only lead to waste of resources but also increase the resource usage cost during job execution. Therefore, how to minimize the resource usage cost while ensuring good performance has become an urgent problem to be solved. Summary of the Invention

[0005] To solve the above problems of the prior art, the present invention proposes a low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm, which includes:

[0006] S1: Construct a cost model for the Spark job, and establish an objective function to improve cluster resource utilization and minimize cluster usage costs;

[0007] S2: Obtain Spark running logs and node running data, including executor running time, virtual machine garbage collection time, data locality, and the usage of CPU and memory resources;

[0008] S3: Clean and analyze the obtained data, select potentially relevant features, and mark the data set using formulas;

[0009] S4: Optimize the selection of hyperparameter combinations of the LightGBM algorithm using the Starling Optimization Algorithm (NOA);

[0010] S5: Use the obtained optimal parameter combination as the hyperparameters of the LightGBM algorithm, train a prediction model, and use the prediction model to find out the possible stragglers of Spark tasks;

[0011] S6: According to the nodes that may produce stragglers predicted, combined with node performance, sort the node priorities, and perform Spark task scheduling according to the node priorities;

[0012] Furthermore, determine the parameters of the energy consumption calculation model during the running of the Spark application, including:

[0013] App = {Job0, Job1, Job2, …, Job m-1}

[0014] Job i = {Stage i0 , Stage i1 , Stage i2 , …, Stage i(n-1)}

[0015]

[0016] Exe = {ex0, ex1, ex2, …, ex p-1}

[0017] Among them, App represents the application program submitted by the user to Spark, which consists of multiple jobs; Job i represents the i-th job in the application program; Stage ij represents the j-th stage in the i-th job; represents the k-th task in Stage ij ; Exe represents the set of executors in the Spark cluster.

[0018] Furthermore, determine the resource usage cost model when the Spark application is running:

[0019]

[0020] Among them, η represents the set composed of the types of all virtual machines in the Spark cluster; δ k represents the set composed of all virtual machines of type k; p k represents the fixed cost required to use a virtual machine of type k; a jk is a binary decision variable, indicating whether the j-th virtual machine of type k is used; t App represents the execution time of the application program; represents the execution time of the i-th job in the application program.

[0021] Furthermore, analyze the collected data set and use a formula to mark whether a task is a straggler:

[0022]

[0023] Among them, S i indicates whether the i-th task in job J is a straggler. If it is, the value is 1; otherwise, the value is 0; nd(t i ) represents the ratio of the task execution time to the amount of work completed by task t (the number of bytes read or written); M erepresents the median in a set; β is a threshold variable, usually taking the value of 1.3.

[0024] Furthermore, the starling optimization algorithm (NOA) is used to find the optimal combination of LightGBM algorithm hyperparameters, including the following steps:

[0025] Step 1: Initialize the starling population, define the population size, problem dimension, and randomly generate the initial position set of starlings;

[0026] Step 2: Simulate the behavior of starlings storing food to perform a global search and find a new position in the solution space;

[0027] Step 3: Simulate the behavior of starlings looking for food to perform a local search and update the current solution according to the existing optimal solution and the position of the reference point of the starling individual;

[0028] Step 4: Calculate its fitness value based on the objective function to obtain the current individual's optimal fitness value;

[0029] Step 5: Update a better fitness value according to the fitness value;

[0030] Step 6: Repeat Step 2 to Step 5 until the set number of iterations is reached, and select it as the optimal solution;

[0031] Furthermore, the formula for calculating the fitness value of each starling is:

[0032]

[0033] where Loss represents the cross-entropy loss function, which is used as an evaluation index for the model prediction accuracy; y i represents the label value of sample i; p i represents the probability that sample i is predicted as a straggler; N represents the number of samples.

[0034] Furthermore, simulate the behavior of starlings searching for and storing food storage space to find a new position in the solution space:

[0035]

[0036] where t represents the current iteration number, j = 1, 2, 3,..., d; represents the position information of the i-th starling in the j-th dimension in the t-th iteration, U j and L j are the upper and lower bounds of the optimization problem in the j-th dimension, and γ is a random number generated according to the Levy flight function, is the optimal individual of the current population, A, B, and C are three randomly selected nutcracker individuals from the population; τ1, τ2, τ3, r1, r2, r3 are random numbers between [0, 1]; τ4 is a random number based on the normal distribution; τ5 is a random number generated according to the Levy flight function; μ is a number generated based on τ3, τ4, τ5; t is the current iteration number; T max is the maximum number of iterations; δ is set to 0.05; l is a number that linearly decays from 1 to 0.

[0037] Furthermore, to simulate the behavior of nutcrackers searching for and retrieving their previously stored food, two positions are set as reference points for finding the stored food:

[0038]

[0039] where t represents the current iteration number, represents the first reference point of the i-th nutcracker in the t-th iteration, represents the second reference point of the i-th nutcracker in the t-th iteration; RP is a random position; P rp is the probability of globally exploring other regions within the search space; and are the upper and lower bounds of the parameters to be found in the optimization problem; is the upper bound of the second dimension in the optimization problem; θ is a random radian between [0, 1]; r1, r2 are random numbers between [0, 1].

[0040] According to the reference points of the nutcracker's stored food, explore and develop the optimal positions where food may be stored:

[0041]

[0042]

[0043] where, represents the position information of the i-th nutcracker in the j-th dimension in the (t + 1)-th iteration;

[0044] is the optimal individual of the current population; is the position information randomly selected from the population; is the value of the objective function when the nutcracker individual is located at ; is the value of the objective function when the nutcracker individual is at the first reference point; is the value of the objective function when the nutcracker individual is at the second reference point.

[0045] Furthermore, the LightGBM algorithm includes the following steps:

[0046] Step 1: Obtain the optimal hyperparameter combination obtained by the Nutcracker Optimization Algorithm (NOA) and use it as the hyperparameter input for the LightGBM algorithm;

[0047] Step 2: Construct the decision tree function:

[0048]

[0049] where K is the total number of trees, and f k represents the k-th tree, represents the prediction result of sample x i ;

[0050] Step 3: Determine the loss function and construct the optimal model by minimizing the loss function:

[0051]

[0052] where is the training error, and Ω(f k ) represents the regularization term of the k-th tree;

[0053] Step 4: The complexity of each tree is expressed as:

[0054]

[0055] where Ω(f t ) represents the regularization term of the model at the t-th iteration, which is used to prevent overfitting; γ and λ represent the regularization term coefficients to prevent the decision tree from being too complex; T represents the number of leaf nodes of the model;

[0056] Step 5: Determine the objective function as:

[0057]

[0058] where Obj(θ) t represents the objective function at the t-th iteration; represents the predicted value of the previous t - 1 iterations;

[0059] The Taylor expansion of the objective function is:

[0060]

[0061] Simplify the objective function to:

[0062]

[0063] where Obj(θ) t represents the objective function at the t-th iteration; g i represents the first-order derivative of each sample; h i represents the second-order derivative of each sample; ft (x i ) represents the predicted value of the previous t - 1 iterations; Ω(f t ) represents the regularization term of the model at the t - th iteration.

[0064] Furthermore, the node priority index is obtained by normalizing and objectively weighting the performance scores of virtual nodes and whether the nodes produce stragglers.

[0065] Furthermore, the evaluation index of node performance scores is expressed as:

[0066]

[0067] Among them, represents the CPU main frequency of the i - th virtual node in the Spark cluster; cupFre stand represents the lowest CPU main frequency of all virtual nodes in the Spark cluster; represents the IO write speed of the i - th virtual node in the Spark cluster; iow stand represents the lowest IO write speed among all nodes; represents the IO read speed of the i - th virtual node in the Spark cluster, ior stand represents the lowest IO write speed among all nodes; w cpu represents the weight of node CPU performance; w io represents the weight of node IO performance.

[0068] The performance scores of nodes are normalized as:

[0069]

[0070] Among them, represents the performance score of the i - th node after data normalization; represents the performance score of the i - th node; N represents the set of all nodes in the Spark cluster.

[0071] The method for determining the objective weight can be expressed as:

[0072]

[0073] Among them, ω PS represents the objective weight of node performance scores; ω ST represents the objective weight of nodes producing stragglers; σ PS represents the standard deviation of node performance scores; σ ST represents the standard deviation of nodes producing stragglers;

[0074] is a Boolean variable, with a value of 1 when it is predicted that the i-th node will not produce a straggler, and 0 otherwise; represents the average value of the performance scores of all nodes; represents all nodes The average value N of the values represents all virtual nodes in the Spark cluster.

[0075] The node priority metric can be expressed as:

[0076]

[0077] 1) Since many hyperparameters of the LightGBM algorithm have a great impact on accuracy, the present invention utilizes the powerful global search ability and flexibility of the Nutcracker Optimization Algorithm (NOA) to reasonably select parameters, thereby significantly improving the prediction accuracy.

[0078] 2) Construct a resource cost model for Spark jobs, and utilize the prediction results of the LightGBM algorithm to optimize the task scheduling of Spark in combination with the performance of each node. This optimization not only significantly improves the resource utilization rate of the cluster, but also effectively reduces the usage cost of the cluster. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 is a flowchart of the low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm of the present invention;

[0080] Figure 2 is a flowchart of the NOA-LightGBM hybrid algorithm of the present invention;

[0081] Figure 3 is a module diagram added or modified based on the Spark framework in the present invention;

[0082] Figure 4 is an architecture diagram of a Spark cluster node; DETAILED DESCRIPTION OF THE EMBODIMENTS

[0083] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0084] In order to successfully implement the low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm provided by the present invention in Spark, certain modifications need to be made to the Spark module, such as Figure 3As shown in the figure, the Model Building module is a newly added model training module that trains the NOA-LightGBM hybrid algorithm model through Spark logs and monitoring script data. The Model Building module is a newly added model training module, and the Predicter is a newly added straggler identification and prediction module, that is, the trained NOA-LightGBM hybrid algorithm model; the TaskScheduler and TaskSetManager are modules that need to be modified, which are used to implement the modification of the scheduling algorithm and the scheduling management of TaskSet respectively.

[0085] The present invention proposes a low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm, as Figure 1 shown, the method includes:

[0086] S1: Construct a cost model for Spark jobs to improve cluster resource utilization and minimize the cluster usage cost to establish an objective function;

[0087] S2: Obtain Spark running logs and node running data, including executor running time, virtual machine garbage collection time, data locality, and the usage of CPU and memory resources;

[0088] S3: Clean and analyze the obtained data, select potentially relevant features, and mark the data set using formulas;

[0089] S4: Use the Starling optimization algorithm (NOA) to optimize the selection of hyperparameter combinations of the LightGBM algorithm;

[0090] S5: Use the obtained optimal parameter combination as the hyperparameters of the LightGBM algorithm to train a prediction model, and use the prediction model to obtain the potential stragglers that may be generated by Spark tasks;

[0091] S6: According to the nodes that may generate stragglers predicted, combined with the node performance, sort the node priorities, and perform Spark task scheduling according to the node priorities.

[0092] This embodiment divides the method into three parts, including:

[0093] I. Construct a resource cost model for Spark applications in the cluster environment.

[0094] In this embodiment, a cost model of Spark in a cluster environment is constructed. The minimum cost of the cluster is used as the objective function. The cost of using the Spark cluster can be summarized as the total cost of all virtual nodes used, and the virtual nodes are virtual machines. The virtual nodes have resource availability limitations, and the resources include CPU resources and memory resources. Therefore, constraint conditions are considered for the objective function in this cost model. The overall optimization objective is:

[0095]

[0096] where η represents the set composed of the types of all virtual machines in the Spark cluster; δ k represents the set composed of all virtual machines of type k; p k represents the fixed cost required to use a virtual machine of type k; a jk is a binary decision variable indicating whether the j-th virtual machine of type k is used; t App represents the execution time of the application; represents the execution time of the i-th job in the application.

[0097] II. Use the NOA-LightGBM hybrid algorithm to predict the nodes that may become stragglers.

[0098] As Figure 2 shown, the steps of optimizing the LightGBM algorithm with the Nutcracker Optimization Algorithm (NOA) are as follows:

[0099] Step 1: Select the hyperparameters to be optimized by the LightGBM algorithm. For example, n_estimators represents the number of sub-models, max_depth represents the depth of the tree, learning_rate represents the ratio of the weight of the model generated by each iteration to the learning rate, and eta represents the learning rate.

[0100] Step 2: Obtain the Spark running logs and monitoring script data, including the executor running time, virtual machine garbage collection time, data locality, and the usage of CPU and memory resources, and clean the obtained data according to certain rules;

[0101] Step 3: Select the features as the executor running time, virtual machine garbage collection time, data locality, and the usage of CPU and memory resources;

[0102] Step 4: Use the Nutcracker Optimization Algorithm (NOA) to find the optimal combination of the LightGBM algorithm hyperparameters;

[0103] Step 5: Initialize the nutcracker population, define the population size, problem dimension, and randomly generate the initial position set of the nutcrackers;

[0104] Step 6: Conduct a global search by simulating the behavior of nutcrackers storing food, and find a new position within the solution space;

[0105] Step 7: Conduct a local search by simulating the behavior of nutcrackers searching for food, and update the current solution based on the existing optimal solution and the position of the reference point of the nutcracker individual;

[0106] Step 8: Calculate its fitness value based on the objective function to obtain the current individual's optimal fitness value;

[0107] Step 9: Update a better fitness value according to the fitness value;

[0108] Step 10: Conduct iterations until the set number of iterations is reached, and select it as the optimal solution;

[0109] Step 11: The formula for calculating the fitness value of each nutcracker is:

[0110]

[0111] Among them, Loss represents the cross-entropy loss function, which is used as an evaluation index for the prediction accuracy of the model; y i represents the label value of sample i; p i represents the probability that sample i is predicted as a straggler; N represents the number of samples.

[0112] Simulate the behavior of nutcrackers searching for and storing food storage space, and find a new position in the solution space:

[0113]

[0114] Among them, t represents the current iteration number, j = 1, 2, 3,..., d; represents the position information of the i-th nutcracker in the j-th dimension in the t-th iteration, U j and L j are the upper and lower bounds of the optimization problem in the j-th dimension, γ is a random number generated according to the Levy flight function, is the current optimal individual in the population, A, B, C are three randomly selected nutcracker individuals from the population; τ1, τ2, τ3, r1, r2, r3 are random numbers between [0, 1]; τ4 is a random number based on the normal distribution; τ5 is a random number generated according to the Levy flight function; μ is a number generated based on τ3, τ4, τ5; t is the current iteration number; T max is the maximum number of iterations; δ is set to 0.05; l is a number linearly decaying from 1 to 0.

[0115] Simulate the behavior of nutcrackers searching for and retrieving their previously stored food, and set two positions as reference points for searching for stored food:

[0116]

[0117]

[0118] Among them, t represents the current iteration number, represents the first reference point of the i-th nutcracker in the t-th iteration, represents the second reference point of the i-th nutcracker in the t-th iteration; RP is a random position; P rp is the probability of globally exploring other regions within the search space; and are the upper and lower bounds of the parameters to be obtained in the optimization problem; is the upper bound of the second dimension in the optimization problem; θ is a random radian in [0,1]; r1, r2, are random numbers between [0,1].

[0119] Explore and develop the optimal positions where food may be stored according to the reference points of the nutcrackers storing food:

[0120]

[0121] Among them, represents the position information of the i-th nutcracker in the j-th dimension in the (t + 1)-th iteration;

[0122] is the optimal individual of the current population; is the position information randomly selected from the population; is the value of the objective function when the nutcracker individual is located at ; is the value of the objective function when the nutcracker individual is located at the first reference point; is the value of the objective function when the nutcracker individual is located at the second reference point.

[0123] Step 12: Obtain the optimal hyperparameter combination obtained by the Nutcracker Optimization Algorithm (NOA) and use it as the hyperparameter input for the LightGBM algorithm;

[0124] Construct the LightGBM decision tree function:

[0125]

[0126] Among them, K is the total number of trees, f k represents the k-th tree, represents the prediction result of the sample x i ;

[0127] Determine its loss function and construct the optimal model by minimizing the loss function:

[0128]

[0129] where is the training error, and Ω(f k ) represents the regularization term of the k-th tree;

[0130] The complexity of each tree can be expressed as:

[0131]

[0132] where Ω(f t ) represents the regularization term of the model at the t-th iteration, which is used to prevent overfitting; γ and λ represent the regularization term coefficients to prevent the decision tree from being too complex; T represents the number of leaf nodes of the model;

[0133] The objective function is determined as:

[0134]

[0135] where Obj(θ) t represents the objective function at the t-th iteration; represents the predicted values of the previous t - 1 iterations;

[0136] The Taylor expansion of the objective function is:

[0137]

[0138] The simplified objective function is:

[0139]

[0140] where Obj(θ) t represents the objective function at the t-th iteration; g i represents the first-order derivative of each sample; h i represents the second-order derivative of each sample; f t (x i ) represents the predicted values of the previous t - 1 iterations; Ω(f t ) represents the regularization term of the model at the t-th iteration.

[0141] III. Task scheduling is performed based on the prediction results in combination with the node performance metrics.

[0142] After obtaining the prediction results by relying on the NOA-LightGBM hybrid algorithm, the node priorities are sorted based on the performance of the cluster nodes:

[0143] Node performance score evaluation metric is expressed as:

[0144]

[0145] Normalize the performance scores of the nodes as follows:

[0146]

[0147] The method for determining the objective weight can be expressed as:

[0148]

[0149] The node priority index can be expressed as:

[0150]

[0151] According to the node priority, tasks are preferentially assigned to nodes with higher priorities. The node architecture diagram of the entire cluster is as shown in Figure 4 shown.

[0152] The method of the present invention optimizes the selection of hyperparameter combinations in the LightGBM algorithm, uses the global optimization ability of the Nutcracker Optimization Algorithm (NOA) to search for the best parameter combination of the LightGBM algorithm, thereby improving the prediction accuracy of the LightGBM algorithm. Then, the node priorities are sorted based on the prediction results, and the Spark task scheduling is performed according to the node priorities, so as to achieve the purpose of reducing the cluster usage cost and improving the cluster utilization rate.

[0153] The above-mentioned embodiments further elaborate on the purpose, technical solution, and advantages of the present invention. It should be understood that the above-mentioned embodiments are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made to the present invention within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm, characterized in that It includes the following steps: S1: Build a cost model for the Spark job, and establish an objective function to improve cluster resource utilization and minimize the cluster usage cost; S2: Obtain the Spark running logs and node running data, including the executor running time, virtual machine garbage collection time, data locality, and the usage of CPU and memory resources; S3: Clean and analyze the obtained data, select the potentially relevant features, and label the data set using a formula; S4: Optimize the selection of hyperparameter combinations of the LightGBM algorithm using the Starling Optimizer Algorithm (NOA); S5: Use the obtained optimal parameter combination as the hyperparameters of the LightGBM algorithm to train a prediction model, and use the prediction model to find the potential stragglers that may occur in the Spark task; S6: Based on the nodes that may produce stragglers predicted, combined with the node performance, sort the node priorities, and perform Spark task scheduling according to the node priorities.

2. The low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, wherein Determine the parameters of the runtime cost calculation model for the Spark application, including: App = {Job0, Job1, Job2, …, Job m-1} Job i = {Stage i0 , Stage i1 , Stage i2 , …, Stage i(n-1)} Exe = {ex0, ex1, ex2, …, ex p-1} Among them, App represents the application submitted by the user to Spark, which consists of multiple jobs; Job i represents the i-th job in the application; Stage ij represents the j-th stage in the i-th job; represents Stage ij the k-th task in it; Exe represents the set of executors in the Spark cluster.

3. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that Determine the resource usage cost model during the runtime of the Spark application: Among them, η represents the set composed of the types of all virtual machines in the Spark cluster; δ k represents the set composed of all virtual machines of type k; p k represents the fixed cost required to use a virtual machine of type k; a jk is a binary decision variable indicating whether the j-th virtual machine of type k is used; t App represents the execution time of the application; represents the execution time of the i-th job in the application.

4. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that, Analyze the collected data set and use a formula to label whether a task is a straggler: Among them, S i indicates whether the i-th task in job J is a straggler. If so, the value is 1; otherwise, the value is 0. nd(t i ) represents the ratio of the task execution time to the amount of work completed by task t (the number of bytes read or written); M e represents the median in a set; β is a threshold variable, usually taking the value of 1.

3.

5. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that Use the Starling Optimizer Algorithm (NOA) to find the optimal combination of hyperparameters of the LightGBM algorithm, including the following steps: Step 1: Initialize the starling population, define the population size, problem dimension, and randomly generate the initial position set of the starlings; Step 2: Simulate the behavior of starlings storing food to perform a global search and find new positions in the solution space; Step 3: Simulate the behavior of starlings looking for food to perform a local search, and update the current solution according to the existing optimal solution and the position of the reference point of the starling individual; Step 4: Calculate its fitness value based on the objective function to obtain the current individual's optimal fitness value; Step 5: Update the better fitness value according to the fitness value; Step 6: Repeat steps 2 to 5 until the set number of iterations is reached, and select it as the optimal solution.

6. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that, The formula for calculating the fitness value of each starling is: Among them, Loss represents the cross-entropy loss function, which is used as an evaluation metric for the prediction accuracy of the model; y i represents the label value of sample i; p i represents the probability that sample i is predicted as a straggler; N represents the number of samples.

7. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that Simulate the behavior of starlings searching for and storing food storage space to find new positions in the solution space: where \(t\) represents the current iteration number, and \(j = 1, 2, 3, \cdots, d\); represents the position information of the \(i\)-th nutcracker in the \(j\)-th dimension at the \(t\)-th iteration, \(U\) j and \(L\) j are the upper and lower bounds of the optimization problem in the \(j\)-th dimension, \(\gamma\) is a random number generated according to the Levy flight function, is the optimal individual of the current population, \(A\), \(B\), \(C\) are three randomly selected nutcracker individuals from the population; \(\tau_1\), \(\tau_2\), \(\tau_3\), \(r_1\), \(r_2\), \(r_3\) are random numbers between \([0, 1]\); \(\tau_4\) is a random number based on the normal distribution; \(\tau_5\) is a random number generated according to the Levy flight function; \(\mu\) is a number generated based on \(\tau_3\), \(\tau_4\), \(\tau_5\); \(t\) is the current iteration number; \(T\) max is the maximum number of iterations; \(\delta\) is set to \(0.05\); \(l\) is a number linearly decaying from \(1\) to \(0\).

8. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that, Simulate the behavior of starlings searching for and retrieving their previously stored food. To find the position of the stored food, two reference points are selected: where t represents the current iteration number, represents the first reference point of the i-th nutcracker in the t-th iteration, represents the second reference point of the i-th nutcracker in the t-th iteration; RP is a random position; P rp is the probability of globally exploring other regions within the search space; and are the upper and lower bounds of the parameters to be obtained in the optimization problem; is the upper bound of the second dimension in the optimization problem; θ is a random radian in [0, 1]; r1, r2 are random numbers between [0, 1].

9. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that Explore and develop the optimal positions where food may be stored based on the reference points of starling food storage: Among them, represents the position information of the i-th nutcracker in the j-th dimension at the (t + 1)-th iteration; is the optimal individual of the current population; is the position information randomly selected from the population; is the value of the objective function when the nutcracker individual is located at ; is the value of the objective function when the nutcracker individual is located at the reference point one; is the value of the objective function when the nutcracker individual is located at the reference point two.

10. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that, Use the LightGBM algorithm as the prediction algorithm to find the nodes that may produce stragglers, including: Step 1: Obtain the optimal hyperparameter combination obtained by the Starling Optimizer Algorithm (NOA) and use it as the hyperparameter input of the LightGBM algorithm; Step 2: Build a decision tree function: where K is the total number of trees, and f k represents the k-th tree, represents the prediction result of the sample x i ; Step 3: Determine the loss function and build an optimal model by minimizing the loss function; Among them is the training error, and Ω(f k ) represents the regularization term of the k-th tree; Step 4: The complexity of each tree is expressed as: where Ω(f t ) represents the regularization term of the model in the t-th iteration, which is used to prevent overfitting; γ and λ represent the regularization term coefficients to prevent the decision tree from being too complex; T represents the number of leaf nodes of the model; Step 5: Determine the objective function as: where Obj(θ) t represents the objective function of the t-th iteration; represents the predicted values of the previous t - 1 iterations; Perform a Taylor expansion on the objective function as: Simplify the objective function as: where Obj(θ) t represents the objective function of the t-th iteration; g i represents the first-order derivative of each sample; h i represents the second-order derivative of each sample; f t (x i ) represents the predicted value of the previous t-1 iterations; Ω(f t ) represents the regularization term of the model at the t-th iteration.

11. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that Node performance score evaluation index Expressed as: w cpu +w io = 1 Among them, represents the CPU main frequency of the i-th virtual node in the Spark cluster; cupFre stand Represents the lowest CPU main frequency of all virtual nodes in the Spark cluster; Represents the IO write speed of the i-th virtual node in the Spark cluster; iow stand Represents the lowest IO write speed among all nodes; Represents the IO read speed of the i-th virtual node in the Spark cluster, ior stand Represents the lowest IO write speed among all nodes; w cpu Represents the weight of the node CPU performance; w io Represents the weight of the node IO performance.

12. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that Normalize the performance scores of the nodes: Among them, represents the performance score of the i-th node after data normalization; represents the performance score of the i-th node; N represents the set of all nodes in the Spark cluster.

13. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that, The objective weight ω of the node performance score PS Expressed as: The objective weight ω of the node generating stragglers ST It is expressed as: Among them, ω PS represents the objective weight of the node performance score; ω ST represents the objective weight of the node producing stragglers; σ PS represents the standard deviation of the node performance score; σ ST represents the standard deviation of the node producing stragglers; is a boolean variable, which is 1 when it is predicted that the i-th node will not produce stragglers, and 0 otherwise; represents the average value of all node performance scores; represents all nodes The average value of the values N represents all virtual nodes in the Spark cluster.

14. A low-cost Spark task scheduling method based on the NOA-LightGBM hybrid algorithm according to claim 1, characterized in that The priority of the nodes in the cluster is expressed as: Among them, represents the priority of the i-th node in the cluster; ω PS represents the objective weight of the node performance score; ω ST represents the objective weight of the node producing stragglers; represents the normalized performance score of the i-th node; is a boolean variable, which is 1 when it is predicted that the i-th node will not produce stragglers, otherwise 0; N represents the set of all nodes in the Spark cluster.

Images