A hadoop parameter optimization method based on improved sparrow search algorithm

Abstract

The application provides a Hadoop parameter optimization method based on an improved sparrow search algorithm, and relates to the technical field of distributed computing. First, population parameters are set and the population is initialized, and the parameter value of each sparrow in the initialized population is taken as the value of the configuration parameter; a job is submitted to a Hadoop cluster, the execution time of the job is taken as the fitness of the sparrow, and each sparrow in the population is sorted according to the fitness value; the finder update formula is optimized, and the positions of the finder, the follower and the early warning person in the population are updated; the sparrow parameters in the iteratively updated population are taken as the configuration parameter values, the job is submitted to the Hadoop cluster again, the fitness of the sparrow in the population is obtained, and each sparrow in the population is sorted according to the fitness value; if the iteration termination condition is reached, the position vector of the sparrow with the optimal fitness in the current population is output as the optimal solution of the configuration parameter; if the iteration termination condition is not reached, the iteration is continued.

Description

Technical Field

[0001] The present invention relates to the field of distributed computing technology, and in particular to a Hadoop parameter optimization method based on an improved sparrow search algorithm. Background Art

[0002] With the rapid development of next-generation information technologies such as the internet and cloud computing, the amount of data required to be processed across various industries has grown exponentially, reaching terabytes. Simultaneously, data processing capabilities in many industrial sectors have increased from terabytes to petabytes. Due to the need to integrate and analyze large amounts of data, data processing has experienced explosive growth. The emergence of distributed data processing frameworks such as Hadoop has fundamentally addressed the problem of distributed massive data processing. Hadoop primarily consists of three components: HDFS, MapReduce, and YARN. HDFS addresses the distributed storage of massive data, MapReduce addresses distributed computing of massive data, and YARN manages resource scheduling. Currently, most internet companies are using Hadoop to address their data processing challenges. In a Hadoop cluster, job execution performance can be optimized by adjusting various parameters. Parameter optimization can improve job execution efficiency, reduce resource usage, and shorten job execution time. Optimizing multiple MapReduce configuration parameters can improve MapReduce job performance, thereby enhancing overall efficiency. Therefore, optimizing MapReduce configuration parameters is a key research area in Hadoop performance optimization. Traditional parameter optimization methods require manual adjustments based on experience. Due to the large and complex number of Hadoop parameters, this method is extremely labor-intensive and difficult to obtain the optimal configuration parameter values. Therefore, an automatic Hadoop parameter optimization method is urgently needed to solve this problem.

[0003] The Sparrow Search Algorithm (SSA) is a swarm intelligence optimization algorithm proposed in 2020 that mimics the foraging and anti-predation behaviors of sparrows. By simulating the strategies of sparrows in their foraging process and combining randomness and local search, it can quickly find optimal solutions in the search space. However, the Sparrow Search Algorithm suffers from slow convergence and is prone to falling into local optimal solutions. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to address the deficiencies of the above-mentioned existing technologies and provide a Hadoop parameter optimization method based on an improved sparrow search algorithm. The sparrow search algorithm is improved and used to find appropriate parameter values ​​for MapReduce configuration parameter optimization problems.

[0005] A Hadoop parameter optimization method based on an improved sparrow search algorithm includes the following steps:

[0006] Step 1: Determine the value range of the parameter to be optimized and set the population parameters according to the value range;

[0007] The population parameters include the parameter item to be optimized, the total number of sparrows in the population, the maximum number of iterations, the proportion of discoverers, the proportion of early warnings and the safety value;

[0008] Step 2: Initialize the population by initializing the parameters to be optimized;

[0009] Assume the total number of sparrows in the initial population and the dimension of the solution space for the optimization problem of job execution efficiency. The expression of each sparrow in the solution space is as follows:

[0010] X i =(X i,1 ,X i,2 ,···,X i,D ),i∈[1,N]

[0011] Among them, N is the total number of sparrows in the initial population, X i represents the parameter value of the i-th sparrow in the solution space, D is the dimension of the solution space of the problem to be optimized, X i,D Indicates the position of the i-th sparrow in the D-dimensional solution space;

[0012] Step 3: Submit the job to the Hadoop cluster using the parameter values ​​of each sparrow in the initialized population as the configuration parameter values. Use the job execution time as the sparrow's fitness and sort each sparrow in the population by fitness value.

[0013] Step 4: Based on the relationship between the warning value and the safety value, determine whether the current position of the population is safe, determine the principle for updating the discoverer position, update the discoverer position according to the update principle, use the first M sparrows in the population as discoverers, optimize the discoverer update formula, and update the discoverer position in the population according to the optimized discoverer update formula;

[0014] Based on the relationship between the warning value and the safety value, the principle for determining the update of the discoverer's location is:

[0015] When the warning value is less than the safety value, it means that the population is in a safe position and the discoverers in the population will actively expand their search range to search for food;

[0016] When the warning value is greater than or equal to the safety value, it means that the population is in a poor position and there is danger around, so the current position should be left;

[0017] The discoverer update formula is optimized by integrating the sparrow search algorithm and the butterfly optimization algorithm. The optimized discoverer update formula is shown as follows:

[0018]

[0019] in, is the parameter value of the i-th sparrow in the j-dimensional solution space in the t+1-th iteration; λ is a random number of 0 or 1; is the parameter value of the i-th sparrow in the j-dimensional solution space in the t-th iteration; α is a random number; R2 is the warning value; iter max is the maximum number of iterations; ST is the safety value; r is a random number between (0,1); Q is a random number that obeys the standard normal distribution; is the parameter value of the sparrow with the best fitness value in the t-th iteration in the j-th dimension space; is the fitness value of the i-th sparrow in the t-th iteration;

[0020] Step 5: Take the sparrows in the population except the discoverer as followers and update the positions of the followers;

[0021] The follower update formula is as follows:

[0022]

[0023] in, is the value of the sparrow with the worst fitness in the t-th iteration in the j-dimensional solution space; is the value of the sparrow with the best fitness in the j-dimensional solution space in the t+1th iteration; A+ is the calculation matrix, A + =A T ·(A·A T ) -1 , A is a matrix with 1 row and D columns; L is a matrix with all values ​​1; n is the number of followers;

[0024] Step 6: Randomly select T sparrows from the population as early warning detectors and update their positions;

[0025] The update formula of the early warning system is as follows:

[0026]

[0027] Among them, β is a random number that obeys the standard normal distribution; K is a random number between (-1,1); f i 、f worst and f best are the fitness value of the i-th sparrow in the population, the worst fitness value of the population, and the best fitness value of the population respectively; ε is the minimum value;

[0028] Step 7: Submit the job to the Hadoop cluster again using the iteratively updated parameters of the sparrows in the population as the configuration parameter values ​​to obtain the fitness of the sparrows in the population and sort each sparrow in the population by the fitness value;

[0029] Step 8: Determine whether the iteration termination condition is met. If so, output the position vector of the sparrow with the best fitness in the current population as the optimal solution for the configuration parameters; if not, return to step 4 and continue iteration.

[0030] The beneficial effects of adopting the above technical solution are:

[0031] The present invention improves the discoverer update formula of the sparrow search algorithm and proposes an improved sparrow search algorithm. The improved sparrow search algorithm is used to optimize Hadoop configuration parameters to shorten the Hadoop job execution time and improve the actual work efficiency of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A main flow chart of a Hadoop parameter optimization method based on an improved sparrow search algorithm provided by an embodiment of the present invention;

[0033] Figure 2 The iterative optimization results of the WordCount test program provided by the embodiment of the present invention, where (a) is the iterative optimization result when the data set size is 120M, (b) is the iterative optimization result when the data set size is 1G, and (c) is the iterative optimization result when the data set size is 2G;

[0034] Figure 3 Comparison results of iterative optimization of the Improved Sparrow Search Algorithm (ISSA) and the Sparrow Search Algorithm (SSA) in the WordCount test program provided by the embodiments of the present invention;

[0035] Figure 4 The comparison results of the execution time of the optimized parameters and the default parameters of the algorithm provided by the embodiment of the present invention are shown. DETAILED DESCRIPTION

[0036] The following embodiments of the present invention are described in further detail with reference to the accompanying drawings and examples. The following examples are used to illustrate the present invention but are not intended to limit the scope of the present invention.

[0037] like Figure 1 As shown, a Hadoop parameter optimization method based on an improved sparrow search algorithm includes the following steps:

[0038] Step 1: Determine the value range of the parameter to be optimized, and reasonably set the population parameters according to the value range to ensure the correct conduct of the experiment;

[0039] The population parameters include the parameter item to be optimized, the total number of sparrows in the population (population size), the maximum number of iterations, the proportion of discoverers, the proportion of early warnings and the safety value;

[0040] In this embodiment, the parameter items are MapReduce parameter items;

[0041] Step 2: Initialize the population by initializing the parameters to be optimized;

[0042] Assume the total number of sparrows in the initial population and the dimension of the solution space for the optimization problem of job execution efficiency. The expression of each sparrow in the solution space is as follows:

[0043] X i =(X i,1 ,X i,2 ,···,X i,D ),i∈[1,N]

[0044] Among them, N is the total number of sparrows in the initial population, X i represents the parameter value of the i-th sparrow in the solution space, D is the dimension of the solution space of the problem to be optimized, X i,D Indicates the position of the i-th sparrow in the D-dimensional solution space;

[0045] Step 3: Submit the job to the Hadoop cluster using the parameter values ​​of each sparrow in the initialized population as the configuration parameter values. Use the job execution time as the sparrow's fitness and sort each sparrow in the population by fitness value.

[0046] Step 4: Based on the relationship between the warning value and the safety value, determine whether the current position of the population is safe, determine the principle of updating the discoverer position, and update the discoverer position according to the update principle. The first M sparrows in the population are regarded as discoverers. The discoverer determines the movement direction of the population. The discoverer update formula is optimized and the positions of the discoverers in the population are updated according to the optimized discoverer update formula.

[0047] Based on the relationship between the warning value and the safety value, the principle for determining the update of the discoverer's location is:

[0048] When the warning value is less than the safety value, it means that the population is in a safe position and the discoverers in the population will actively expand their search range to search for food;

[0049] When the warning value is greater than or equal to the safety value, it means that the population is in a poor position and there is danger around, so the current position should be left;

[0050] The expression for the number of discoverers in a population is as follows:

[0051] M=N·PD

[0052] Wherein, PD is the proportion of discoverer in the population;

[0053] The position update formula of the discoverer in the population of sparrow search algorithm is as follows:

[0054]

[0055] Wherein, is the parameter value of the ith sparrow in the jth dimensional solution space in the t+1th iteration, is the parameter value of the ith sparrow in the jth dimensional solution space in the tth iteration; a is a random number, R2 is a warning value, both of which are random numbers uniformly distributed between (0, 1); iter max is the maximum number of iterations; ST is a safety value, which is a random number uniformly distributed between (0.5, 1); Q is a random number following a standard normal distribution;

[0056] Since the formula is easy to fall into local optimal solution, the sparrow search algorithm and the butterfly optimization algorithm are fused to optimize the discoverer update formula, and the optimized discoverer update formula is as follows:

[0057]

[0058] Wherein, λ is a random number of 0 or 1; r is a random number between (0, 1); is the parameter value of the sparrow with the optimal fitness value in the jth dimensional space in the tth iteration; is the fitness value of the ith sparrow in the tth iteration; through the formula, the discoverer is updated, which can improve the global search ability of the sparrow;

[0059] Step 5: the sparrows in the population except the discoverer are taken as followers, and the position of the follower is updated; the follower monitors the movement of the discoverer and competes for the position, when the follower fails to compete for the position, the follower will fly to update to other position, and the follower update formula is as follows:

[0060]

[0061] Wherein, is the value of the sparrow with the worst fitness in the jth dimensional solution space in the tth iteration; is the value of the sparrow with the optimal fitness in the jth dimensional solution space in the t+1th iteration; A+ is a calculation matrix, A + = A T · (A·A T ) -1 , A is a 1 row D column matrix, each element of the matrix A is randomly selected from {-1, 1}; L is a matrix whose values are all 1; n is the number of followers;

[0062] Step 6: Randomly select T sparrows from the population as early warning agents and update their positions. When an early warning agent detects danger, it will leave its current position and fly to a position near the optimal value of the group.

[0063] The expression for the number of early warning persons in a population is as follows:

[0064] T=N·SD

[0065] Wherein, SD is the proportion of early warning individuals in the population;

[0066] The update formula of the early warning system is as follows:

[0067]

[0068] Among them, β is a random number that obeys the standard normal distribution; K is a random number between (-1,1); f i 、f worst and f best They are the fitness value of the i-th sparrow in the population, the worst fitness value of the population, and the best fitness value of the population; ε is the minimum value. In order to avoid the denominator being 0, e is taken in the program. -10 ;

[0069] Step 7: Submit the job to the Hadoop cluster again using the iteratively updated parameters of the sparrows in the population as the configuration parameter values ​​to obtain the fitness of the sparrows in the population and sort each sparrow in the population by the fitness value;

[0070] Step 8: Determine whether the iteration termination condition is met. If so, output the position vector of the sparrow with the best fitness in the current population as the optimal solution for the configuration parameters; if not, return to step 4 and continue iteration.

[0071] The following will illustrate the Hadoop parameter optimization method based on the improved sparrow search algorithm of the present invention in conjunction with specific experiments. The experiments of the present invention were carried out on three CentOS operating systems, with machine memory of 8G, 3G, and 3G respectively. The Hadoop version used was 3.1.3, the JDK version used was 1.8, and the MapReduce job test program used was WordCount. To ensure that the experimental time was not too long, the generated test data sets were of sizes of 120M, 1G, and 2G respectively. Other experimental parameter settings are shown in Table 1 below, and the parameter information of the MapReduce parameters to be optimized is shown in Table 2;

[0072] Table 1 Experimental parameter settings

[0073] Parameter items Value maximum number of iterations iter max ]] 30 Population size N 20 Number of parameters to be optimized 6 Proportion of discoverers (PD) 0.7 Proportion of early warning SD 0.2 Safety value ST 0.6

[0074] Table 2 MapReduce parameter information

[0075]

[0076] The iterative optimization results of the WordCount test program are as follows Figure 2 Figures (a), (b), and (c) show the iterative optimization results for datasets of 120MB, 1GB, and 2GB, respectively. The horizontal axis represents the number of iterations, the vertical axis represents fitness (job execution time), the dotted line represents the average job execution time per iteration, and the solid line represents the minimum job execution time per iteration. As can be seen, due to the uncertainty in the direction of parameter updates, the average job execution time varies, but the downward trend is more pronounced.

[0077] Figure 3 Comparison results of iterative optimization of the improved sparrow search algorithm (ISSA) and the sparrow search algorithm (SSA) on the WordCount test program. As can be seen, the ISSA algorithm achieves the shortest job execution time faster than the SSA algorithm, and the average job execution time of the ISSA algorithm is significantly lower than that of the SSA algorithm.

[0078] Figure 4 The following table compares the execution time of MapReduce jobs using the optimized parameters of the improved sparrow search algorithm with the default parameters for three datasets. As can be seen, when the dataset sizes are 120MB, 1GB, and 2GB, the optimized algorithm reduces the execution time by 28.6%, 34.4%, and 22.7%, respectively, compared to the default execution time. This significantly reduces the execution time.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A Hadoop parameter optimization method based on an improved sparrow search algorithm, characterized by: Step 1: Determine the value range of the parameter to be optimized and set the population parameters according to the value range; Step 2: Initialize the population by initializing the parameters to be optimized; Step 3: Submit the job to the Hadoop cluster using the parameter values ​​of each sparrow in the initialized population as the configuration parameter values. Use the job execution time as the sparrow's fitness and sort each sparrow in the population by fitness value. Step 4: Based on the relationship between the warning value and the safety value, determine whether the current position of the population is safe, determine the principle for updating the discoverer position, update the discoverer position according to the update principle, use the first M sparrows in the population as discoverers, optimize the discoverer update formula, and update the discoverer position in the population according to the optimized discoverer update formula; Step 5: Take the sparrows in the population except the discoverer as followers and update the positions of the followers; Step 6: Randomly select T sparrows from the population as early warning detectors and update their positions; Step 7: Submit the job to the Hadoop cluster again using the iteratively updated parameters of the sparrows in the population as the configuration parameter values ​​to obtain the fitness of the sparrows in the population and sort each sparrow in the population by the fitness value; Step 8: Determine whether the iteration termination condition is met. If so, output the position vector of the sparrow with the best fitness in the current population as the optimal solution for the configuration parameters. If not, return to step 4 and continue iterating. In step 2, the total number of sparrows in the initial population and the dimension of the solution space for the optimization problem of job execution efficiency are set. The expression of each sparrow in the solution space is shown as follows: X i =(X i,1 ,X i,2 ,···,X i,D ),i∈[1,N] Among them, N is the total number of sparrows in the initial population, X i represents the parameter value of the i-th sparrow in the solution space, D is the dimension of the solution space of the problem to be optimized, X i,D Indicates the position of the i-th sparrow in the D-dimensional solution space; In step 4, the principle for determining the update of the discoverer's location based on the relationship between the warning value and the safety value is: When the warning value is less than the safety value, it means that the population is in a safe position and the discoverers in the population will actively expand their search range to search for food; When the warning value is greater than or equal to the safety value, it means that the population is in a poor position and there is danger around, so the current position should be left; In step 4, the sparrow search algorithm and the butterfly optimization algorithm are integrated to optimize the discoverer update formula. The optimized discoverer update formula expression is shown as follows: in, is the parameter value of the i-th sparrow in the j-dimensional solution space in the t+1-th iteration; λ is a random number of 0 or 1; is the parameter value of the i-th sparrow in the j-dimensional solution space in the t-th iteration; α is a random number; R2 is the warning value; iter max is the maximum number of iterations; ST is the safety value; r is a random number between (0,1); Q is a random number that obeys the standard normal distribution; is the parameter value of the sparrow with the best fitness value in the t-th iteration in the j-th dimension space; f i t is the fitness value of the i-th sparrow in the t-th iteration.

2. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 1, characterized in that: The population parameters in step 1 include the parameter item to be optimized, the total number of sparrows in the population, the maximum number of iterations, the proportion of discoverers, the proportion of early warnings, and the safety value.

3. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 1, characterized in that: The follower update formula in step 5 is as follows: in, is the value of the sparrow with the worst fitness in the t-th iteration in the j-dimensional solution space; A is the value of the sparrow with the best fitness in the j-dimensional solution space in the t+1th iteration; + To calculate the matrix, A + =A T ·(A·A T ) -1 , A is a matrix with 1 row and D columns; L is a matrix with all values ​​1; n is the number of followers.

4. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 3, characterized in that: The update formula of the early warning system in step 6 is as follows: Among them, β is a random number that obeys the standard normal distribution; K is a random number between (-1,1); f i 、f worst and f best are the fitness value of the i-th sparrow in the population, the worst fitness value of the population, and the optimal fitness value of the population; ε is the minimum value.

Images