Hadoop parameter optimization method based on improved sparrow search algorithm

Abstract

The invention provides a Hadoop parameter optimization method based on an improved sparrow search algorithm, and relates to the technical field of distributed computing. Firstly, population parameters are set, a population is initialized, the parameter value of each sparrow in the initialized population is used as the value of a configuration parameter, a job is submitted to a Hadoop cluster, the execution time of the job is used as the fitness of the sparrows, and the sparrows in the population are sorted according to the fitness value; optimizing a discoverer updating formula, updating the positions of discoverers, followers and early warners in the population, taking parameters of sparrows in the population after iteration updating as configuration parameter values, submitting the values to a Hadoop cluster again, obtaining the fitness of the sparrows in the population, and sorting each sparrow in the population according to the fitness value; if the iteration termination condition is met, outputting the position vector of the sparrow with the optimal fitness in the current population as the optimal solution of the configuration parameter, and if the iteration termination condition is not met, continuing iteration.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed computing, and in particular to a Hadoop parameter optimization method based on an improved sparrow search algorithm. Background Art

[0002] With the rapid development of new generation information technologies such as the Internet and cloud computing, the amount of data that needs to be processed in various industries has grown exponentially to the TB level. At the same time, in many industrial fields, data processing capabilities have been upgraded from the TB level to the PB level. As a large amount of data needs to be integrated and counted, data processing has experienced explosive growth. The emergence of distributed data processing frameworks such as Hadoop has fundamentally solved the problem of processing distributed massive data. Hadoop mainly includes three parts: HDFS, MapReduce, and YARN. HDFS solves the problem of distributed storage of massive data, MapReduce solves the problem of distributed computing of massive data, and YARN is responsible for resource scheduling. At present, most Internet companies are using Hadoop to solve data processing problems. In the Hadoop cluster, the performance of job execution can be optimized by adjusting different parameters. Optimizing parameters can help improve the execution efficiency of jobs, reduce resource usage, and reduce job execution time. In Hadoop parameter optimization, optimizing multiple configuration parameters of MapReduce can improve the performance of MapReduce jobs, thereby improving overall efficiency. Therefore, in Hadoop performance optimization, MapReduce configuration parameter optimization is an important research field. The traditional parameter optimization method requires manual adjustment based on experience. Due to the large and complex number of Hadoop parameters, this method is extremely labor-intensive and it is difficult to obtain the optimal configuration parameter values. Therefore, there is an urgent need for an automatic Hadoop parameter optimization method to solve this problem.

[0003] The Sparrow Search Algorithm (SSA) is a swarm intelligence optimization algorithm proposed in 2020 that imitates the foraging and anti-predation behaviors of sparrows. The algorithm simulates the strategies of sparrows in the process of searching for food, combines randomness and local search, and can quickly find a better solution in the search space. The Sparrow Search Algorithm has the problem of slow convergence and easy to fall into the local optimal solution. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a Hadoop parameter optimization method based on an improved sparrow search algorithm in view of the deficiencies of the above-mentioned prior art, improve the sparrow search algorithm, and use it to find suitable parameter values ​​for MapReduce configuration parameter optimization problems.

[0005] A Hadoop parameter optimization method based on an improved sparrow search algorithm comprises the following steps:

[0006] Step 1: Determine the value range of the parameter to be optimized and set the population parameters according to the value range;

[0007] The population parameters include the parameter item to be optimized, the total number of sparrows in the population, the maximum number of iterations, the proportion of discoverers, the proportion of early warnings and the safety value;

[0008] Step 2: Initialize the population by initializing the parameters to be optimized;

[0009] Set the total number of sparrows in the initial population and the dimension of the solution space of the optimization problem of job execution efficiency. The expression of each sparrow in the solution space is as follows:

[0010] X i =(X i,1 ,X i,2 ,···,X i,D ),i∈[1,N]

[0011] Among them, N is the total number of sparrows in the initial population, X i represents the parameter value of the i-th sparrow in the solution space, D is the dimension of the solution space of the problem to be optimized, X i,D represents the position of the i-th sparrow in the D-dimensional solution space;

[0012] Step 3: Take the parameter value of each sparrow in the initialized population as the value of the configuration parameter, submit the job to the Hadoop cluster, take the execution time of the job as the fitness of the sparrow, and sort each sparrow in the population according to the fitness value;

[0013] Step 4: According to the relationship between the warning value and the safety value, determine whether the current position of the population is safe, determine the principle of updating the discoverer position, update the discoverer position according to the update principle, take the first M sparrows of the population as discoverers, optimize the discoverer update formula, and update the position of the discoverer in the population according to the optimized discoverer update formula;

[0014] According to the relationship between the warning value and the safety value, the principle for determining the update of the discoverer's position is:

[0015] When the warning value is less than the safety value, it means that the population is in a safe position, and the discoverers in the population will actively expand the search range to search for food;

[0016] When the warning value is greater than or equal to the safety value, it means that the population is in a bad position and there is danger around, so the current position should be left;

[0017] The discoverer update formula is optimized by integrating the sparrow search algorithm and the butterfly optimization algorithm. The optimized discoverer update formula is shown as follows:

[0018]

[0019] in, is the parameter value of the i-th sparrow in the j-dimensional solution space in the t+1-th iteration; λ is a random number of 0 or 1; is the parameter value of the i-th sparrow in the j-dimensional solution space in the t-th iteration; α is a random number; R2 is the warning value; iter max is the maximum number of iterations; ST is the safety value; r is a random number between (0,1); Q is a random number that follows a standard normal distribution; is the parameter value of the sparrow with the best fitness value in the t-th iteration in the j-th dimension space; is the fitness value of the i-th sparrow in the t-th iteration;

[0020] Step 5: Take the sparrows in the population except the discoverer as followers and update the positions of the followers;

[0021] The follower update formula is as follows:

[0022]

[0023] in, is the value of the sparrow with the worst fitness in the t-th iteration in the j-dimensional solution space; is the value of the sparrow with the best fitness in the j-dimensional solution space in the t+1th iteration; A+ is the calculation matrix, A + =A T ·(A·A T ) -1 , A is a matrix with 1 row and D columns; L is a matrix with all values ​​1; n is the number of followers;

[0024] Step 6: Randomly select T sparrows from the population as early warning detectors and update the positions of the early warning detectors;

[0025] The update formula of the early warning system is as follows:

[0026]

[0027] Among them, β is a random number that follows a standard normal distribution; K is a random number between (-1,1); f i 、f worst and f best are the fitness value of the i-th sparrow in the population, the worst fitness value of the population, and the best fitness value of the population respectively; ε is the minimum value;

[0028] Step 7: Submit the job to the Hadoop cluster again using the iteratively updated parameters of the sparrows in the population as the configuration parameter values, obtain the fitness of the sparrows in the population, and sort each sparrow in the population according to the fitness value;

[0029] Step 8: Determine whether the iteration termination condition is met. If so, output the position vector of the sparrow with the best fitness in the current population as the optimal solution for the configuration parameters; if not, return to step 4 to continue the iteration.

[0030] The beneficial effects of adopting the above technical solution are:

[0031] The present invention improves the finder update formula of the sparrow search algorithm, proposes an improved sparrow search algorithm, and uses the improved sparrow search algorithm to optimize Hadoop configuration parameters to shorten the Hadoop job execution time and improve the actual work efficiency of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A main flow chart of a Hadoop parameter optimization method based on an improved sparrow search algorithm provided by an embodiment of the present invention;

[0033] Figure 2 The iterative optimization results of the WordCount test program provided by the embodiment of the present invention, wherein (a) is the iterative optimization result when the data set size is 120M, (b) is the iterative optimization result when the data set size is 1G, and (c) is the iterative optimization result when the data set size is 2G;

[0034] Figure 3 Iterative optimization comparison results of the Improved Sparrow Search Algorithm (ISSA) and the Sparrow Search Algorithm (SSA) provided in the embodiment of the present invention in the WordCount test program;

[0035] Figure 4 The comparison results of the execution time of the algorithm optimized parameters and the default parameters provided in the embodiment of the present invention. DETAILED DESCRIPTION

[0036] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0037] like Figure 1 As shown, a Hadoop parameter optimization method based on an improved sparrow search algorithm includes the following steps:

[0038] Step 1: Determine the value range of the parameter to be optimized, and reasonably set the population parameters according to the value range to ensure the correct conduct of the experiment;

[0039] The population parameters include the parameter item to be optimized, the total number of sparrows in the population (population size), the maximum number of iterations, the proportion of discoverers, the proportion of early warnings and the safety value;

[0040] In this embodiment, the parameter items are MapReduce parameter items;

[0041] Step 2: Initialize the population by initializing the parameters to be optimized;

[0042] Set the total number of sparrows in the initial population and the dimension of the solution space of the optimization problem of job execution efficiency. The expression of each sparrow in the solution space is as follows:

[0043] X i =(X i,1 ,X i,2 ,···,X i,D ),i∈[1,N]

[0044] Among them, N is the total number of sparrows in the initial population, X i represents the parameter value of the i-th sparrow in the solution space, D is the dimension of the solution space of the problem to be optimized, X i,D represents the position of the i-th sparrow in the D-dimensional solution space;

[0045] Step 3: Take the parameter value of each sparrow in the initialized population as the value of the configuration parameter, submit the job to the Hadoop cluster, take the execution time of the job as the fitness of the sparrow, and sort each sparrow in the population according to the fitness value;

[0046] Step 4: According to the relationship between the warning value and the safety value, determine whether the current position of the population is safe, determine the principle of updating the discoverer position, update the discoverer position according to the update principle, take the first M sparrows of the population as discoverers, and the discoverer determines the movement direction of the population. Optimize the discoverer update formula and update the position of the discoverer in the population according to the optimized discoverer update formula;

[0047] According to the relationship between the warning value and the safety value, the principle for determining the update of the discoverer's position is:

[0048] When the warning value is less than the safety value, it means that the population is in a safe position, and the discoverers in the population will actively expand the search range to search for food;

[0049] When the warning value is greater than or equal to the safety value, it means that the population is in a bad position and there is danger around, so the current position should be left;

[0050] The expression for the number of discoverers in a population is as follows:

[0051] M=N·PD

[0052] Among them, PD is the proportion of discoverers in the population;

[0053] The position update formula of the discoverer in the population of the sparrow search algorithm is as follows:

[0054]

[0055] in, is the parameter value of the i-th sparrow in the j-dimensional solution space in the t+1-th iteration, is the parameter value of the i-th sparrow in the j-dimensional solution space in the t-th iteration; α is a random number, R2 is the warning value, both of which are random numbers between (0,1) and obey uniform distribution; iter max is the maximum number of iterations; ST is a safety value, which is a random number between (0.5, 1) that follows a uniform distribution; Q is a random number that follows a standard normal distribution;

[0056] Since this formula is prone to fall into a local optimal solution, the discoverer update formula is optimized by integrating the sparrow search algorithm and the butterfly optimization algorithm. The optimized discoverer update formula is shown as follows:

[0057]

[0058] Among them, λ is a random number of 0 or 1; r is a random number between (0,1); is the parameter value of the sparrow with the best fitness value in the t-th iteration in the j-th dimension space; is the fitness value of the i-th sparrow in the t-th iteration; by updating the discoverer through this formula, the global search ability of the sparrow can be improved;

[0059] Step 5: Take the sparrows in the population other than the discoverer as followers and update the positions of the followers; the followers monitor the movement of the discoverer and compete for the position. When the follower fails to compete for the position, the follower will fly and update to other positions. The follower update formula is shown as follows:

[0060]

[0061] in, is the value of the sparrow with the worst fitness in the t-th iteration in the j-dimensional solution space; is the value of the sparrow with the best fitness in the j-dimensional solution space in the t+1th iteration; A+ is the calculation matrix, A + =A T ·(A·A T ) -1 , A is a 1-row, D-column matrix, each element of matrix A is randomly selected from {-1,1}; L is a matrix with all values ​​1; n is the number of followers;

[0062] Step 6: Randomly select T sparrows from the population as early warning detectors and update their positions. When the early warning detector finds danger coming, it will leave its current position and fly to the position near the optimal value of the group.

[0063] The expression of the number of early warning persons in a population is as follows:

[0064] T=N·SD

[0065] Among them, SD is the proportion of early warning persons in the population;

[0066] The update formula of the early warning system is as follows:

[0067]

[0068] Among them, β is a random number that follows a standard normal distribution; K is a random number between (-1,1); f i 、f worst and f best are the fitness value of the i-th sparrow in the population, the worst fitness value of the population, and the best fitness value of the population; ε is the minimum value. In order to avoid the denominator being 0, e is taken in the program. -10 ;

[0069] Step 7: Submit the job to the Hadoop cluster again using the iteratively updated parameters of the sparrows in the population as the configuration parameter values, obtain the fitness of the sparrows in the population, and sort each sparrow in the population according to the fitness value;

[0070] Step 8: Determine whether the iteration termination condition is met. If so, output the position vector of the sparrow with the best fitness in the current population as the optimal solution for the configuration parameters; if not, return to step 4 to continue the iteration.

[0071] The following will combine specific experiments to illustrate the Hadoop parameter optimization method based on the improved sparrow search algorithm of the present invention. The experiment of the present invention was carried out on 3 CentOS operating systems, and the machine memory was 8G, 3G and 3G respectively. The Hadoop version used was 3.1.3, the JDK version used was 1.8, and the MapReduce job test program used was WordCount. In order to ensure that the experimental time is not too long, the generated test data set sizes are 120M, 1G and 2G respectively. Other experimental parameter settings are shown in Table 1 below, and the parameter information of the MapReduce parameter items to be optimized is shown in Table 2;

[0072] Table 1 Experimental parameter settings

[0073] Parameters Value <![CDATA[Maximum number of iterations iter max > 30 Population size N 20 Number of parameters to be optimized 6 Proportion of discoverers PD 0.7 Proportion of early warning SD 0.2 Safety value ST 0.6

[0074] Table 2 MapReduce parameter information

[0075]

[0076] The iterative optimization results of the WordCount test program are as follows Figure 2 As shown in the figure, (a), (b), and (c) are the iterative optimization results when the data set size is 120M, 1G, and 2G, respectively. The horizontal axis represents the number of iterations, the vertical axis represents the fitness (job execution time), the dotted line represents the average execution time of each round of iteration, and the solid line represents the shortest execution time of each round of iteration. It can be seen that due to the uncertainty of the direction of parameter update, the average execution time of the job has increased and decreased, but the downward trend is more obvious.

[0077] Figure 3 The comparison results of iterative optimization of the improved sparrow search algorithm (ISSA) and the sparrow search algorithm (SSA) in the WordCount test program. It can be seen that the ISSA algorithm can obtain the shortest execution time of the job faster than the SSA algorithm, and the average execution time of the ISSA algorithm job is significantly lower than that of the SSA algorithm.

[0078] Figure 4 The results are the comparison of the execution time of MapReduce jobs executed with the parameter values ​​optimized by the improved sparrow search algorithm and the default parameter values ​​under three data sets. It can be seen that when the data set size is 120M, 1G, and 2G, the execution time after the algorithm optimization of the present invention is shortened by 28.6%, 34.4%, and 22.7% respectively compared with the default execution time, and the execution time is greatly shortened.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope defined by the claims of the present invention.

Claims

1. A Hadoop parameter optimization method based on an improved sparrow search algorithm, characterized in that: Step 1: Determine the value range of the parameter to be optimized and set the population parameters according to the value range; Step 2: Initialize the population by initializing the parameters to be optimized; Step 3: Take the parameter value of each sparrow in the initialized population as the value of the configuration parameter, submit the job to the Hadoop cluster, take the execution time of the job as the fitness of the sparrow, and sort each sparrow in the population according to the fitness value; Step 4: According to the relationship between the warning value and the safety value, determine whether the current position of the population is safe, determine the principle of updating the discoverer position, update the discoverer position according to the update principle, take the first M sparrows of the population as discoverers, optimize the discoverer update formula, and update the position of the discoverer in the population according to the optimized discoverer update formula; Step 5: Take the sparrows in the population except the discoverer as followers and update the positions of the followers; Step 6: Randomly select T sparrows from the population as early warning detectors and update the positions of the early warning detectors; Step 7: Submit the job to the Hadoop cluster again using the iteratively updated parameters of the sparrows in the population as the configuration parameter values, obtain the fitness of the sparrows in the population, and sort each sparrow in the population according to the fitness value; Step 8: Determine whether the iteration termination condition is met. If so, output the position vector of the sparrow with the best fitness in the current population as the optimal solution for the configuration parameters; if not, return to step 4 to continue the iteration.

2. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 1, characterized in that: The population parameters in step 1 include parameter items to be optimized, the total number of sparrows in the population, the maximum number of iterations, the proportion of discoverers, the proportion of early warnings, and a safety value.

3. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 2, characterized in that: In step 2, the dimension of the solution space of the total number of sparrows in the initialization population and the optimization problem of the job execution efficiency is set, and the expression of each sparrow in the solution space is shown as follows: X i =(X i,1 ,X i,2 ,···,X i,D ),i∈[1,N] Among them, N is the total number of sparrows in the initial population, X i represents the parameter value of the i-th sparrow in the solution space, D is the dimension of the solution space of the problem to be optimized, X i,D Represents the position of the i-th sparrow in the D-dimensional solution space.

4. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 3, characterized in that: In step 4, according to the relationship between the warning value and the safety value, the principle for determining the update of the discoverer's location is: When the warning value is less than the safety value, it means that the population is in a safe position, and the discoverers in the population will actively expand the search range to search for food; When the warning value is greater than or equal to the safety value, it means that the population is in a poor position and there is danger around, so the current position should be left.

5. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 4 is characterized in that: In step 4, the sparrow search algorithm and the butterfly optimization algorithm are integrated to optimize the discoverer update formula. The optimized discoverer update formula expression is shown as follows: in, is the parameter value of the i-th sparrow in the j-dimensional solution space in the t+1-th iteration; λ is a random number of 0 or 1; is the parameter value of the i-th sparrow in the j-dimensional solution space in the t-th iteration; α is a random number; R2 is the warning value; iter max is the maximum number of iterations; ST is the safety value; r is a random number between (0,1); Q is a random number that follows a standard normal distribution; is the parameter value of the sparrow with the best fitness value in the tth iteration in the jth dimension space; f i t is the fitness value of the i-th sparrow in the t-th iteration.

6. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 5, characterized in that: The follower update formula in step 5 is as follows: in, is the value of the sparrow with the worst fitness in the t-th iteration in the j-dimensional solution space; A is the value of the sparrow with the best fitness in the j-dimensional solution space in the t+1th iteration; + To calculate the matrix, A + =A T ·(A·A T ) -1 , A is a matrix with 1 row and D columns; L is a matrix with all values ​​1; n is the number of followers.

7. The Hadoop parameter optimization method based on the improved sparrow search algorithm according to claim 6, characterized in that: The update formula of the early warning in step 6 is as follows: Among them, β is a random number that follows a standard normal distribution; K is a random number between (-1,1); f i 、f worst and f best are the fitness value of the i-th sparrow in the population, the worst fitness value of the population and the optimal fitness value of the population respectively; ε is the minimum value.

Images