Energy consumption monitoring and optimizing method of HADOOP cluster

Abstract

The invention discloses an energy consumption monitoring and optimizing method of an HADOOP cluster, and belongs to the technical field of computers. The problems that an existing power consumption prediction model is poor in generalization ability, an existing energy consumption prediction method does not consider the influence of CPU frequency, and an existing energy consumption optimization method is difficult to recognize an optimal configuration parameter combination are solved. According to the method, the power missing value is firstly filled, then the filled data is utilized to train the power consumption prediction model, and the hardware resource use condition data is considered in the training process, so that the generalization ability of the trained power consumption prediction model is improved. According to the method, the mapping relation between the hardware parameters including the CPU frequency of the cluster server and the HADOOP cluster configuration parameter list and the cluster energy consumption is qualitatively studied, the parameter selection algorithm based on reinforcement learning is utilized to select the configuration parameter list, and adjustment and optimization are applied on the basis, so that the optimal configuration parameter combination can be identified; and optimization of energy consumption is realized. The method can be applied to energy consumption monitoring and optimization of the HADOOP cluster.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a method for monitoring and optimizing energy consumption of a HADOOP cluster. Background Art

[0002] Energy consumption optimization plays a key role in data center energy efficiency. At present, there are a lot of research results in the field of energy consumption optimization of big data computing frameworks (He, W., et al., Analysis on data center power supply system based on multiple renewable power configurations and multi-objective optimization. 2024. 222: p. 119865.). The global big data market size is expected to grow from US$138.9 billion in 2020 to US$229.4 billion in 2025 (Honar Pajooh, H., et al., IoT Big Data provenance scheme using blockchain on HADOOP ecosystem. 2021. 8: p. 1-26.). Therefore, there is a huge energy consumption problem behind the rapidly growing big data market size that needs to be studied in depth. In some application scenarios, energy consumption needs to be optimized to meet this challenge.

[0003] At the same time, large AI models similar to GPT-4 consume a huge amount of energy, with training consuming about 7.2 million kWh and annual deployment consuming about 180 million kWh. The training energy consumption of large AI models doubles every 4 to 6 months.

[0004] The energy consumption of the digital intelligence center mainly comes from the computing of IT equipment. How to intelligently schedule computing resources and optimize management to save energy and reduce emissions and increase economic benefits has become the key to achieving green computing.

[0005] At present, there is a lack of research in China on intelligent scheduling and optimized management of computing centers to reduce energy consumption. It is necessary to build a green computing theory and technology system and develop intelligent green optimization systems for the new generation of data centers to meet energy consumption challenges and promote the development of green computing.

[0006] Energy consumption measurement is the basis for optimizing energy consumption. Data center servers have various application scenarios and energy consumption measurement methods. Lin Weiwei et al. summarized four energy consumption measurement methods under cloud computing, including direct measurement based on software and hardware, estimation based on energy consumption models, measurement based on virtualization technology, and evaluation based on simulation. Jiang Weixiang divided the energy consumption measurement of data centers into four categories according to the granularity of energy consumption measurement: component layer, server layer, application layer, and virtual machine layer. Direct measurement methods usually use physical instruments (such as IBM's Power Executive), which can continuously monitor server energy consumption independently of the operating system and application load. In addition, power distribution units (PDUs) can also measure energy consumption while distributing power. With the development of cloud computing, energy consumption measurement of virtualization technology has attracted attention, such as Microsoft's Joulemeter and Xen-based VMeter, which measure virtual machine system component indicators through software tools and perform energy consumption modeling to estimate energy consumption. However, there is an error between the data of these built-in tools and the actual energy consumption. Considering the limitations and risks of the experimental environment, simulation methods are also used for energy consumption estimation. Frameworks such as CloudSim and GreenCloud allow fine-grained modeling and evaluation of data center energy consumption. Although the simulation method is not limited by hardware, the data credibility is low.

[0007] Energy consumption optimization work mainly focuses on computing resources such as CPU, cache, and memory, while also considering power resource management.

[0008] But in general, the existing HADOOP cluster energy efficiency monitoring and optimization methods have the following three problems:

[0009] 1. Regarding the power consumption monitoring problem under the big data computing framework, previous work lacked research on energy consumption optimization in actual scenarios. Therefore, the power consumption model has the problem of low prediction accuracy and there are blind spots in the monitoring of instantaneous power consumption and process power consumption. In addition, data collection relies on professional measurement platforms, which brings high cost and low efficiency problems to cluster power consumption monitoring. Finally, current research fails to comprehensively consider hardware resource usage, software configuration parameters and workload characteristics, resulting in the model lacking a certain generalization ability and being difficult to adapt to the power consumption prediction needs under the big data computing framework and hardware configuration conditions.

[0010] 2. There are many configuration parameters for big data systems, including cluster size, node configuration, storage method, etc. Different combinations of these parameters will have a significant impact on system energy consumption. In addition, the impact of CPU frequency adjustment on energy consumption cannot be ignored. By dynamically adjusting the CPU frequency, energy consumption can be reduced while ensuring performance. Therefore, for the configuration parameter combination and CPU frequency of the big data computing system, designing a method that can effectively and quantitatively predict the energy consumption level to achieve a balance between energy consumption and performance is an urgent problem to be solved.

[0011] 3. With regard to the hundreds of configuration parameters of the big data system itself, the existing technology lacks in-depth analysis of the internal operating mechanism of the big data system, making it difficult to identify the configuration parameter space that is closely related to energy consumption, and lacks an effective parameter space search method. Therefore, it is difficult for the existing technology to find the optimal configuration parameter combination in the parameter space, so the performance of the energy consumption optimization method based on configuration parameters is poor. Summary of the invention

[0012] The purpose of the present invention is to solve the problems that the generalization ability of the existing power consumption prediction model is poor, the existing energy consumption prediction method does not consider the influence of CPU frequency, and the existing energy consumption optimization method is difficult to identify the optimal configuration parameter combination, and proposes an energy consumption monitoring and optimization method for a HADOOP cluster.

[0013] The technical solution adopted by the present invention to solve the above technical problems is: a method for monitoring and optimizing energy consumption of a HADOOP cluster, the method specifically comprising the following steps:

[0014] Step 1: For the task to be optimized for energy consumption, collect the system hardware resource status log file, active power log file and task execution timestamp log file of the HADOOP cluster composed of N servers during the historical execution of the task;

[0015] The system hardware resource status log file includes the CPU usage and hard disk usage data corresponding to each collection moment;

[0016] Step 2: preprocessing the active power log file based on the system hardware resource status log file and the task execution timestamp log file to obtain the preprocessed active power log file;

[0017] Step 3: Initialize server n=1;

[0018] Step 4: training the power consumption prediction model of the nth server using the system hardware resource status log file of the nth server and the preprocessed active power log file;

[0019] Step 5: Establish a server energy consumption model, and train the server energy consumption model and reinforcement learning model of the nth server based on the trained power consumption prediction model of the nth server and the HADOOP cluster configuration parameter list C;

[0020] Step 6: Determine whether n=N is satisfied;

[0021] If satisfied, proceed to step seven;

[0022] If not satisfied, set n=n+1 and return to step 4;

[0023] Step 7: Set the HADOOP cluster configuration parameter list C when the nth server executes the task task to the HADOOP cluster configuration parameter list C output by the reinforcement learning model trained on the nth server, so as to optimize the energy consumption when the nth server executes the task task;

[0024] The optimized energy consumption of the nth server is calculated through the actual CPU usage, actual hard disk usage data and actual CPU frequency of the nth server.

[0025] The beneficial effects of the present invention are:

[0026] The present invention proposes a method for filling missing power values ​​in the data processing stage to solve the problem of uneven data collection caused by blind spots in monitoring instantaneous power consumption and process power consumption. The power consumption prediction model is trained using the filled data, and the hardware resource usage data is considered during the training process. The trained power consumption prediction model has strong generalization ability and can monitor power consumption data well. The present invention qualitatively studies the mapping relationship between the hardware parameters including the CPU frequency of the cluster server and the HADOOP cluster configuration parameter list for cluster energy consumption, and uses a parameter selection algorithm based on reinforcement learning to select the configuration parameter list and use it as the basis for application tuning, which can identify the optimal configuration parameter combination. The energy consumption optimization method of the present invention, which combines the CPU frequency and the optimal configuration parameter combination, can realize energy consumption monitoring while optimizing energy consumption, and has good scalability and application capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a schematic diagram of the HADOOP cluster power consumption monitoring device;

[0028] Figure 2 It is a structural diagram of the server energy consumption model;

[0029] Figure 3 is a flow chart of the method of the present invention;

[0030] Figure 4 It is a flow chart of server power consumption prediction;

[0031] Figure 5 It is a structural diagram of the server power consumption prediction model;

[0032] Figure 6 It is a flowchart of the parameter selection algorithm based on reinforcement learning. DETAILED DESCRIPTION

[0033] Specific implementation method 1: A method for monitoring and optimizing energy consumption of a HADOOP cluster described in this implementation method specifically includes the following steps:

[0034] Step 1: For the task to be optimized for energy consumption, collect the system hardware resource status log file, active power log file and task execution timestamp log file of the HADOOP cluster composed of N servers during the historical execution of the task;

[0035] The system hardware resource status log file includes the CPU usage and hard disk usage data corresponding to each collection moment;

[0036] Step 2: preprocessing the active power log file based on the system hardware resource status log file and the task execution timestamp log file to obtain the preprocessed active power log file;

[0037] Step 3: Initialize server n=1;

[0038] Step 4: training the power consumption prediction model of the nth server using the system hardware resource status log file of the nth server and the preprocessed active power log file;

[0039] Step 5: Establish a server energy consumption model, and train the server energy consumption model and reinforcement learning model of the nth server based on the trained power consumption prediction model of the nth server and the HADOOP cluster configuration parameter list C;

[0040] Step 6: Determine whether n=N is satisfied;

[0041] If satisfied, proceed to step seven;

[0042] If not satisfied, set n=n+1 and return to step 4;

[0043] Step 7: Set the HADOOP cluster configuration parameter list C when the nth server executes the task task to the HADOOP cluster configuration parameter list C output by the reinforcement learning model trained on the nth server, so as to optimize the energy consumption when the nth server executes the task task;

[0044] The optimized energy consumption of the nth server is calculated through the actual CPU usage rate, actual hard disk usage rate data and actual CPU frequency of the nth server (that is, by substituting the hardware parameters into the power consumption prediction model and the server energy consumption model, and then performing weighted summation on the outputs of the power consumption prediction model and the server energy consumption model, the present invention can optimize the energy consumption and obtain the monitoring value of the optimized energy consumption at the same time).

[0045] Specific implementation method 2: This implementation method is different from specific implementation method 1 in that the timestamp log file of task execution includes the timestamps corresponding to the task in each historical execution process.

[0046] The other steps and parameters are the same as those in the first embodiment.

[0047] The timestamp in this implementation is defined as follows: first, the sum of the end time of the current task execution and the start time of the current task execution is calculated, then the calculated sum is divided by 2, and the result of the division operation is used as the timestamp of the current task execution.

[0048] Specific implementation method three: This implementation method is different from specific implementation methods one or two in that the step one is specifically as follows:

[0049] For each server in the HADOOP cluster, the system hardware resource status log file, active power log file and task execution timestamp log file of the server during the historical task execution process are collected.

[0050] The other steps and parameters are the same as those in the first or second embodiment.

[0051] Specific implementation method 4: This implementation method is different from any one of specific implementation methods 1 to 3 in that the preprocessing in step 2 includes filling edge data and internal data of the active power log file;

[0052] Taking the nth server in the HADOOP cluster as an example, the specific process of edge data filling is as follows:

[0053] Compare the data collection time in the system hardware resource status log file and the active power log file of the nth server, and record the minimum collection time corresponding to the data in the system hardware resource status log file as x 1,min , record the maximum collection time corresponding to the data in the system hardware resource status log file as x 1,max , record the minimum acquisition time corresponding to the data in the active power log file as y 1,min , record the maximum acquisition time corresponding to the data in the active power log file as y 1,max ;

[0054] If y 1,min ≤x 1,min And y 1,max ≥x 1,max , then there is no need to fill the active power log file with edge data;

[0055] If y 1,min ≥x 1,min , then use the collection time y in the active power log file 1,min The corresponding data fills the edge data until the minimum collection time corresponding to the data in the active power log file is equal to x 1,min Alignment;

[0056] If y1,max ≤x 1,max , then use the collection time y in the active power log file 1,max The corresponding data fills the edge data until the maximum acquisition time corresponding to the data in the active power log file is equal to x 1,max Alignment;

[0057] The specific process of internal data filling is as follows:

[0058] For any timestamp in the task execution timestamp log file of the nth server, determine whether there is corresponding active power for the timestamp in the active power log file. If so, there is no need to fill the active power of the timestamp. If not, use the active power in the active power log file that is closest in time to the timestamp to fill the active power of the timestamp in the active power log file.

[0059] Similarly, each timestamp in the task execution timestamp log file of the nth server is processed separately.

[0060] The other steps and parameters are the same as those in Specific Embodiments 1 to 3.

[0061] Specific implementation method five: Combination Figure 5 This embodiment is described. The difference between this embodiment and any one of the specific embodiments 1 to 4 is that the power consumption prediction model includes a deep belief network, a back propagation network and a predictor; wherein: the deep belief network includes a fully connected layer, a batch normalization layer and a ReLU activation function layer, and the predictor includes a ReLU activation function layer and an ELU function layer;

[0062] The input of the power consumption prediction model is used as the input of the deep belief network. In the deep belief network, the input passes through the fully connected layer, batch normalization layer, and ReLU activation function layer in sequence, and then the feature vector output by the ReLU activation function layer is used as the input of the back propagation network.

[0063] In the back propagation network, the mean of each element in the input feature vector is first calculated, and then the calculated mean is smoothed by the kernel function. Finally, the input feature vector is calibrated according to the kernel function smoothing result and the mean to obtain the calibrated feature vector.

[0064] The calibrated feature vector then passes through the ReLU activation function layer and the ELU function layer in the predictor in turn, and outputs the power consumption prediction result through the ELU function layer.

[0065] The other steps and parameters are the same as those in Specific Embodiments 1 to 4.

[0066] Specific implementation method 6: This implementation method is different from the specific implementation methods 1 to 5 in that the input feature vector is calibrated according to the kernel function smoothing result and the mean value to obtain the calibrated feature vector; specifically:

[0067]

[0068] Among them, z is the feature vector of the back propagation network input, is the calibrated eigenvector, μ b is the mean of the elements in the eigenvector z, is for μ b The kernel function smoothing result is .

[0069] The other steps and parameters are the same as those in Specific Implementation Methods 1 to 5.

[0070] Specific implementation method 7: This implementation method is different from any one of specific implementation methods 1 to 6 in that the function form of the power consumption prediction model is:

[0071]

[0072] Where P is the output of the power consumption prediction model, net(·) represents the power consumption prediction model, and w cpu is the CPU usage, w io is the hard disk usage rate, f cpu is the CPU frequency.

[0073] The other steps and parameters are the same as those in Specific Embodiments 1 to 6.

[0074] Specific implementation method eight: Combination Figure 2 This embodiment is different from any one of the specific embodiments 1 to 7 in that the server energy consumption model includes a Backbone unit and a Neck unit, and the Backbone unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and an average pooling layer in sequence, and the Neck unit includes a first fully connected layer and a second fully connected layer;

[0075] The working process of the server energy consumption model is as follows:

[0076] The input of the server energy consumption model is used as the input of the Backbone unit. In the Backbone unit, the input passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer and the average pooling layer in sequence, and then the output of the average pooling layer is used as the input of the Neck unit;

[0077] In the Neck unit, the input passes through the first fully connected layer and the second fully connected layer in sequence; the output of the first fully connected layer is used as the input of the third fully connected layer, and the output of the third fully connected layer passes through the fifth fully connected layer and the sigmoid activation function layer in sequence; the output of the second fully connected layer is used as the input of the fourth fully connected layer, and the output of the fourth fully connected layer passes through the sixth fully connected layer and the softmax activation function layer in sequence;

[0078] Finally, the output of the sigmoid activation function layer is connected to the output of the softmax activation function layer, and the connection result is used as the output of the server energy consumption model.

[0079] The other steps and parameters are the same as those in Specific Embodiments 1 to 7.

[0080] Specific implementation method 9: This implementation method is different from any one of specific implementation methods 1 to 8 in that the function form of the server energy consumption model is:

[0081] F(C,f cpu ,task)=T

[0082] Where F is the server energy consumption model, and T is the task running time output by the server energy consumption model.

[0083] The other steps and parameters are the same as those in Specific Embodiments 1 to 8.

[0084] Specific implementation method 10: This implementation method is different from specific implementation methods 1 to 9 in that the specific process of step 5 is as follows:

[0085] Step 51: Using 100 / M% as the group interval, the CPU usage range of 0% to 100% is divided into M intervals, and using 100 / N% as the group interval, the hard disk usage range of 0% to 100% is divided into N intervals, and M*N hardware resource interval segments are obtained;

[0086] And count the number of samples in each hardware resource interval in the system hardware resource status log file of the nth server, and use the number of samples in the ith hardware resource interval as the weight value of the ith hardware resource interval;

[0087] Step 52: Output the predicted power consumption of each hardware resource interval segment through the power consumption prediction model (for all samples in a hardware resource interval segment, the output of the power consumption prediction model is the same), and then multiply the predicted power consumption of the i-th hardware resource interval segment by the weight value of the i-th hardware resource interval segment to obtain the final power consumption P1 of the i-th hardware resource interval segment, i=1,2,…,M*N;

[0088] Step 53: Use the gated recurrent unit (GRU) of the recurrent neural network as the agent of the reinforcement learning model, and use the agent to generate the initial configuration parameter list C of the HADOOP cluster;

[0089] Step 54: Initialize the number of training times l=1;

[0090] Step 5: The server energy consumption model generates the task running time of each hardware resource interval according to the HADOOP cluster configuration parameter list C, and the task running time of the i-th hardware resource interval is recorded as t i ;

[0091] Step 56: Perform a weighted sum of the final power consumption and task running time of each hardware resource interval segment, and use the weighted sum result as the reward value;

[0092]

[0093] Step 57: Update the HADOOP cluster configuration parameter list C output by the agent according to the reward value (let the model agent learn to make decisions in the direction of reducing energy consumption);

[0094] Step 58: Determine whether the maximum number of iterations has been reached;

[0095] If it is reached, the iteration stops, and the HADOOP cluster configuration parameter list C outputted in step 57 of the last iteration is used as the final HADOOP cluster configuration parameter list C of the nth server;

[0096] If not reached, set l=l+1 and return to step 55.

[0097] The other steps and parameters are the same as those in Specific Embodiments 1 to 9.

[0098] Example

[0099] The energy consumption monitoring and optimization method of a HADOOP cluster proposed by the present invention is further described in detail below in conjunction with the accompanying drawings. Figure 3 As shown, the method specifically comprises the following steps:

[0100] Step 1: Figure 1The figure shows a schematic diagram of the HADOOP cluster power consumption monitoring device. The HADOOP cluster consists of servers built on three real physical machines. For the tasks to be optimized for energy consumption, each HADOOP machine uses a power meter to collect its own power during the historical execution of the tasks to be optimized by the HADOOP cluster. The power meter is connected to the machine through the power cord of the machine, and the power cord is connected to the socket through a power meter. The power meter collects the power value of the HADOOP cluster every 1 second and transmits it to the terminal via a USB data cable. After the data reaches the terminal, it will enter the data processing module. At the same time, the hardware resource status log file of each HADOOP machine and the timestamp log file of the task execution are collected during the historical execution of the tasks to be optimized by the HADOOP cluster;

[0101] Among them, the timestamp log file of task execution includes the timestamps corresponding to each historical execution process of the task; the hardware resource status log file includes the CPU usage and hard disk usage data corresponding to each collection moment;

[0102] Step 2: Data may be missing due to equipment problems during the active power collection process. Since the active power data is obtained through manual operation, there may be a time difference between the start time of the power meter collection platform and the start time of the system hardware resource status log when the workload is running (executing the task), or there may be a time difference between the end time of the power meter collection platform and the end time of the system hardware resource status log when the workload is running (executing the task). Therefore, it is necessary to fill the edge missing values ​​of the active power log file in a time dimension, and then interpolate and fill the internal missing values ​​of the active power log file respectively, and finally obtain the active power log file after the edge missing values ​​and internal missing values ​​are filled.

[0103] The specific process of edge data filling is as follows:

[0104] For the nth server in the HADOOP cluster, compare the data collection time in the system hardware resource status log file and the active power log file of the nth server, and record the minimum collection time corresponding to the data in the system hardware resource status log file as x 1,min , record the maximum collection time corresponding to the data in the system hardware resource status log file as x 1,max , record the minimum acquisition time corresponding to the data in the active power log file as y 1,min , record the maximum acquisition time corresponding to the data in the active power log file as y 1,max ;

[0105] If y 1,min ≤x 1,min And y 1,max ≥x 1,max, then there is no need to fill the active power log file with edge data;

[0106] If y 1,min ≥x 1,min , then use the collection time y in the active power log file 1,min The corresponding data fills the edge data (that is, the data at the missing position is filled with the corresponding data at time y 1,min Same data) until the minimum acquisition time corresponding to the data in the active power log file is equal to x 1,min Alignment;

[0107] If y 1,max ≤x 1,max , then use the collection time y in the active power log file 1,max The corresponding data fills the edge data (that is, the data at the missing position is filled with the corresponding data at time y 1,max same data) until the maximum acquisition time corresponding to the data in the active power log file is equal to x 1,max Alignment;

[0108] The specific process of internal data filling is as follows:

[0109] For any timestamp in the timestamp log file of task execution, determine whether there is corresponding active power in the active power log file for this timestamp. If so, there is no need to fill in the active power of this timestamp. If not, use the active power in the active power log file that is closest in time to this timestamp (if there are two active power values ​​closest in time to this timestamp, take the average of the two for filling), and fill in the active power of this timestamp in the active power log file.

[0110] Similarly, each timestamp in the timestamp log file of task execution is processed separately.

[0111] Step 3: Initialize server n=1;

[0112] Step 4: training the power consumption prediction model of the nth server using the system hardware resource status log file of the nth server and the preprocessed active power log file;

[0113] The power consumption prediction model includes a deep belief network, a back propagation network and a predictor; wherein: the deep belief network includes a fully connected layer, a batch normalization layer and a ReLU activation function layer, and the predictor includes a ReLU activation function layer and an ELU function layer;

[0114] The input of the power consumption prediction model is used as the input of the deep belief network. In the deep belief network, the input passes through the fully connected layer, batch normalization layer, and ReLU activation function layer in sequence, and then the feature vector output by the ReLU activation function layer is used as the input of the back propagation network.

[0115] In the back propagation network, the mean of each element in the input feature vector is first calculated, and then the calculated mean is smoothed by the kernel function. Finally, the input feature vector is calibrated according to the kernel function smoothing result and the mean to obtain the calibrated feature vector.

[0116]

[0117] Among them, z is the feature vector of the back propagation network input, is the calibrated eigenvector, μ b is the mean of the elements in the eigenvector z, is for μ b The kernel function smoothing result is .

[0118] The calibrated feature vector then passes through the ReLU activation function layer and the ELU function layer in the predictor in turn, and outputs the power consumption prediction result through the ELU function layer.

[0119] The functional form of the power consumption prediction model is expressed as:

[0120]

[0121] Where P is the output of the power consumption prediction model, net(·) represents the power consumption prediction model, and w cpu is the CPU usage, w i / o is the hard disk usage rate, f cpu It is the CPU frequency (the value given by the system when the server executes tasks).

[0122] For a certain big data computing task, the energy consumption is calculated as follows:

[0123]

[0124] Among them, E is the energy consumption value, T is the total duration, and P(t) is the power consumption value of the server at time t. The single-point server power consumption prediction model predicts the power consumption value P through the hardware resource status, so the energy consumption E can be calculated by estimating the total duration T. Since the energy consumption E is not only related to the hardware conditions, but also closely related to the HADOOP cluster configuration parameter C of the running task task, therefore, a cluster server energy consumption model is further established.

[0125] Step 5: Establish a server energy consumption model, and train the server energy consumption model and reinforcement learning model of the nth server based on the trained power consumption prediction model of the nth server and the HADOOP cluster configuration parameter list C;

[0126] The server energy consumption model includes a Backbone unit and a Neck unit, and the Backbone unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and an average pooling layer in sequence, and the Neck unit includes a first fully connected layer and a second fully connected layer;

[0127] The working process of the server energy consumption model is as follows:

[0128] The input of the server energy consumption model is used as the input of the Backbone unit. In the Backbone unit, the input passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer and the average pooling layer in sequence, and then the output of the average pooling layer is used as the input of the Neck unit;

[0129] In the Neck unit, the input passes through the first fully connected layer and the second fully connected layer in sequence; the output of the first fully connected layer is used as the input of the third fully connected layer, and the output of the third fully connected layer passes through the fifth fully connected layer and the sigmoid activation function layer in sequence; the output of the second fully connected layer is used as the input of the fourth fully connected layer, and the output of the fourth fully connected layer passes through the sixth fully connected layer and the softmax activation function layer in sequence;

[0130] Finally, the output of the sigmoid activation function layer is connected to the output of the softmax activation function layer, and the connection result is used as the output of the server energy consumption model.

[0131] Step 6: Determine whether n=N is satisfied;

[0132] If satisfied, proceed to step seven;

[0133] If not satisfied, set n=n+1 and return to step 4;

[0134] Step 7: Set the HADOOP cluster configuration parameter list C when the nth server executes the task task to the HADOOP cluster configuration parameter list C output by the reinforcement learning model trained on the nth server, so as to optimize the energy consumption when the nth server executes the task task;

[0135] The optimized energy consumption of the nth server is calculated through the actual CPU usage, actual hard disk usage data and actual CPU frequency of the nth server (that is, by substituting the hardware parameters into the power consumption prediction model and the server energy consumption model, and then performing weighted summation on the outputs of the power consumption prediction model and the server energy consumption model).

[0136] like Figure 4 and Figure 6 As shown, the training process of step five of the present invention is specifically described as follows:

[0137] Step 51: Using 100 / M% as the group interval, the CPU usage range of 0% to 100% is divided into M intervals, and using 100 / N% as the group interval, the hard disk usage range of 0% to 100% is divided into N intervals, and M*N hardware resource interval segments are obtained;

[0138] And count the number of samples in each hardware resource interval in the system hardware resource status log file of the nth server, and use the number of samples in the ith hardware resource interval as the weight value of the ith hardware resource interval;

[0139] Step 52: Output the predicted power consumption of each hardware resource interval segment through the power consumption prediction model (for all samples in a hardware resource interval segment, the output of the power consumption prediction model is the same), and then multiply the predicted power consumption of the i-th hardware resource interval segment by the weight value of the i-th hardware resource interval segment to obtain the final power consumption P1 of the i-th hardware resource interval segment, i=1,2,…,M*N;

[0140] Step 53: Use the gated recurrent unit (GRU) of the recurrent neural network as the agent of the reinforcement learning model, and use the agent to generate the initial configuration parameter list C of the HADOOP cluster;

[0141] Step 54: Initialize the number of training times l=1;

[0142] Step 5: The server energy consumption model generates the task running time of each hardware resource interval according to the HADOOP cluster configuration parameter list C, and the task running time of the i-th hardware resource interval is recorded as t i ;

[0143] Step 56: Perform a weighted sum of the final power consumption and task running time of each hardware resource interval segment, and use the weighted sum result as the reward value;

[0144]

[0145] Step 57: Update the HADOOP cluster configuration parameter list C output by the agent according to the reward value (let the model agent learn to make decisions in the direction of reducing energy consumption);

[0146] Step 58: Determine whether the maximum number of iterations has been reached;

[0147] If it is reached, the iteration stops, and the HADOOP cluster configuration parameter list C outputted in step 57 of the last iteration is used as the final HADOOP cluster configuration parameter list C of the nth server;

[0148] If not reached, set l=l+1 and return to step 55.

[0149] The CPU frequency value range to be searched for the reinforcement learning energy consumption tuning in the present invention is: [800, 1000, 1100, 1300, 1500, 1700, 1800, 2000, 2200, 2300, 2500, 2700, 2900, 3000, 3200], in MHz.

[0150] There are eight HADOOP configuration parameters to be searched, namely mapreduce.task.io.sort.factor, mapreduce.reduce.shuffle.merge.percent, mapreduce.reduce.merge.inmem.threshold, mapreduce.output.fileoutputformat.compress, mapreduce.map.sort.spill.percent, mapreduce.task.io.sort.mb, mapreduce.reduce.shuffle.input.buffer.percent and mapreduce.map.output.compress. For the Sort task, the energy consumption value optimization rate of the present invention is 34.91%; for the WordCount task, the energy consumption value optimization rate of the present invention is 22.40%.

[0151] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for monitoring and optimizing energy consumption of a HADOOP cluster, characterized in that: The method specifically comprises the following steps: Step 1: For the task to be optimized for energy consumption, collect the system hardware resource status log file, active power log file and task execution timestamp log file of the HADOOP cluster composed of N servers during the historical execution of the task; The system hardware resource status log file includes the CPU usage and hard disk usage data corresponding to each collection moment; Step 2: preprocessing the active power log file based on the system hardware resource status log file and the task execution timestamp log file to obtain the preprocessed active power log file; Step 3: Initialize server n=1; Step 4: training the power consumption prediction model of the nth server using the system hardware resource status log file of the nth server and the preprocessed active power log file; Step 5: Establish a server energy consumption model, and train the server energy consumption model and reinforcement learning model of the nth server based on the trained power consumption prediction model of the nth server and the HADOOP cluster configuration parameter list C; Step 6: Determine whether n=N is satisfied; If satisfied, proceed to step seven; If not satisfied, set n=n+1 and return to step 4; Step 7: Set the HADOOP cluster configuration parameter list C when the nth server executes the task task to the HADOOP cluster configuration parameter list C output by the reinforcement learning model trained on the nth server, so as to optimize the energy consumption when the nth server executes the task task; The optimized energy consumption of the nth server is calculated through the actual CPU usage, actual hard disk usage data and actual CPU frequency of the nth server.

2. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 1, characterized in that: The timestamp log file of the task execution includes the timestamps corresponding to each historical execution process of the task.

3. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 2, characterized in that: The step 1 is specifically as follows: For each server in the HADOOP cluster, the system hardware resource status log file, active power log file and task execution timestamp log file of the server during the historical task execution process are collected.

4. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 3, characterized in that: The preprocessing of step 2 includes filling edge data and internal data of the active power log file; Taking the nth server in the HADOOP cluster as an example, the specific process of edge data filling is as follows: Compare the data collection time in the system hardware resource status log file and the active power log file of the nth server, and record the minimum collection time corresponding to the data in the system hardware resource status log file as x 1,min , record the maximum collection time corresponding to the data in the system hardware resource status log file as x 1,max , record the minimum acquisition time corresponding to the data in the active power log file as y 1,min , record the maximum acquisition time corresponding to the data in the active power log file as y 1,max ; If y 1,min ≤x 1,min And y 1,max ≥x 1,max , then there is no need to fill the active power log file with edge data; If y 1,min ≥x 1,min , then use the collection time y in the active power log file 1,min The corresponding data fills the edge data until the minimum collection time corresponding to the data in the active power log file is equal to x 1,min Alignment; If y 1,max ≤x 1,max , then use the collection time y in the active power log file 1,max The corresponding data fills the edge data until the maximum acquisition time corresponding to the data in the active power log file is equal to x 1,max Alignment; The specific process of internal data filling is as follows: For any timestamp in the task execution timestamp log file of the nth server, determine whether there is a corresponding active power in the active power log file for the timestamp. If so, there is no need to fill the active power of the timestamp. If not, use the active power in the active power log file that is closest to the timestamp in time to fill the active power of the timestamp in the active power log file. Similarly, each timestamp in the task execution timestamp log file of the nth server is processed separately.

5. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 4, characterized in that: The power consumption prediction model includes a deep belief network, a back propagation network and a predictor; wherein: the deep belief network includes a fully connected layer, a batch normalization layer and a ReLU activation function layer, and the predictor includes a ReLU activation function layer and an ELU function layer; The input of the power consumption prediction model is used as the input of the deep belief network. In the deep belief network, the input passes through the fully connected layer, batch normalization layer, and ReLU activation function layer in sequence, and then the feature vector output by the ReLU activation function layer is used as the input of the back propagation network. In the back propagation network, the mean of each element in the input feature vector is first calculated, and then the calculated mean is smoothed by the kernel function. Finally, the input feature vector is calibrated according to the kernel function smoothing result and the mean to obtain the calibrated feature vector. The calibrated feature vector then passes through the ReLU activation function layer and the ELU function layer in the predictor in turn, and outputs the power consumption prediction result through the ELU function layer.

6. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 5, characterized in that: The input feature vector is calibrated according to the kernel function smoothing result and the mean value to obtain the calibrated feature vector; specifically: Among them, z is the feature vector of the back propagation network input, is the calibrated eigenvector, μ b is the mean of the elements in the eigenvector z, is for μ b The kernel function smoothing result is .

7. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 6, characterized in that: The functional form of the power consumption prediction model is: Where P is the output of the power consumption prediction model, net(·) represents the power consumption prediction model, and w cpu is the CPU usage, w i / o is the hard disk usage rate, f cpu is the CPU frequency.

8. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 7, characterized in that: The server energy consumption model includes a Backbone unit and a Neck unit, and the Backbone unit includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a fourth convolutional layer, a fifth convolutional layer and an average pooling layer in sequence, and the Neck unit includes a first fully connected layer and a second fully connected layer; The working process of the server energy consumption model is as follows: The input of the server energy consumption model is used as the input of the Backbone unit. In the Backbone unit, the input passes through the first convolution layer, the second convolution layer, the third convolution layer, the fourth convolution layer, the fifth convolution layer and the average pooling layer in sequence, and then the output of the average pooling layer is used as the input of the Neck unit; In the Neck unit, the input passes through the first fully connected layer and the second fully connected layer in sequence; the output of the first fully connected layer is used as the input of the third fully connected layer, and the output of the third fully connected layer passes through the fifth fully connected layer and the sigmoid activation function layer in sequence; the output of the second fully connected layer is used as the input of the fourth fully connected layer, and the output of the fourth fully connected layer passes through the sixth fully connected layer and the softmax activation function layer in sequence; Finally, the output of the sigmoid activation function layer is connected to the output of the softmax activation function layer, and the connection result is used as the output of the server energy consumption model.

9. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 8, characterized in that: The functional form of the server energy consumption model is: F(C,f cpu ,task)=T Where F is the server energy consumption model, and T is the task running time output by the server energy consumption model.

10. The method for monitoring and optimizing energy consumption of a HADOOP cluster according to claim 9, characterized in that: The specific process of step five is as follows: Step 51: Using 100 / M% as the group interval, the CPU usage range of 0% to 100% is divided into M intervals, and using 100 / N% as the group interval, the hard disk usage range of 0% to 100% is divided into N intervals, and M*N hardware resource interval segments are obtained; And count the number of samples in each hardware resource interval in the system hardware resource status log file of the nth server, and use the number of samples in the ith hardware resource interval as the weight value of the ith hardware resource interval; Step 52: output the predicted power consumption of each hardware resource interval segment respectively through the power consumption prediction model, and then multiply the predicted power consumption of the i-th hardware resource interval segment by the weight value of the i-th hardware resource interval segment to obtain the final power consumption P1 of the i-th hardware resource interval segment, i=1,2,…,M*N; Step 53: Use the recurrent neural network gated recurrent unit as the intelligent agent of the reinforcement learning model, and use the intelligent agent to generate the initial configuration parameter list C of the HADOOP cluster; Step 54: Initialize the number of training times l=1; Step 5: The server energy consumption model generates the task running time of each hardware resource interval according to the HADOOP cluster configuration parameter list C, and the task running time of the i-th hardware resource interval is recorded as t i ; Step 56: Perform a weighted sum of the final power consumption and task running time of each hardware resource interval segment, and use the weighted sum result as the reward value; Step 57: Update the HADOOP cluster configuration parameter list C output by the agent according to the reward value; Step 58: Determine whether the maximum number of iterations has been reached; If it is reached, the iteration stops, and the HADOOP cluster configuration parameter list C outputted in step 57 of the last iteration is used as the final HADOOP cluster configuration parameter list C of the nth server; If not reached, set l=l+1 and return to step 55.

Images