Energy-saving scheduling method and system based on deep reinforcement learning and heterogeneous Spark cluster
Through the combination of deep reinforcement learning and heterogeneous Spark clusters, resource optimization is carried out for the energy consumption differences of heterogeneous clusters, and the balance of energy consumption and response time is achieved, solving the problem that the existing technology cannot optimize the resource allocation of heterogeneous clusters, reducing the problem of the existing technology being unable to optimize the resource allocation of heterogeneous clusters, and Cloud service costs.
Patent Information
- Application Number
- CN202111505917.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-10
- Publication Date
- 2024-12-13
- Estimated Expiration
- 2041-12-10
AI Technical Summary
The existing Spark scheduling strategies cannot prioritize resource allocation for the energy consumption differences in heterogeneous clusters, resulting in the inability to effectively optimize energy consumption and response time, affecting the cost balance of cloud service providers and users.
Using a method based on deep reinforcement learning, the energy consumption-time target prediction of online data information is performed through the Q network, and the resource allocation scheme with the lowest energy consumption-time target is selected, and the task scheduling is combined with the joint strategy of ε-Greedy and Boltzmann to build energy consumption - Time model and reward model, update DQN parameters to optimize resource allocation.
While ensuring user response time, it is possible to optimize resource utilization, reduce energy consumption, achieve multiple optimizations for energy consumption targets and SLA targets, balance the costs of cloud service providers and users, and has good economic benefits.
Smart Images
Figure CN114281528B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of reinforcement learning and big data processing, and specifically relates to an energy-saving scheduling method and system based on deep reinforcement learning and heterogeneous Spark clusters. Background Art
[0002] Spark, a distributed big data processing framework, has been widely used in research and industry analysis. It stores intermediate results in memory to speed up processing and has higher scalability than other frameworks, making it suitable for running various complex analysis tasks. Cloud computing provides cheaper and easier-to-manage computing resources. Many companies are turning to cloud deployment of big data computing clusters. Efficient use of computing clusters is important for companies. Even a small improvement in utilization can avoid the waste of tens of millions of dollars on a large scale. The implementation of a good cluster scheduler is the key to avoiding these wastes.
[0003] Therefore, it is very necessary to optimize the energy consumption generated by the Spark job running scheduling mechanism. After the Spark cluster is deployed, the scheduling process of the Spark task scheduler can be simply abstracted as the scheduler allocates resource blocks executors to the job, which include physical resources such as CPU and memory. The existing Spark uses simple heuristic algorithms FIFO and Fair for scheduling by default, creating executors in a distributed manner, with the goal of using the cluster in a balanced manner and taking into account the versatility of use. However, the existing Spark does not take into account the resource priority allocation problem caused by different energy consumption due to cluster heterogeneity. Therefore, the default scheduling strategy of Spark cannot optimize scheduling for specific SLA goals.
[0004] In summary, finding a method that can improve resource utilization in a spark cluster environment, optimize energy consumption targets or multiple SLA targets and save energy and reduce emissions as much as possible while ensuring that user response time is met, is of great significance for achieving a balance between cloud service provider (CSP) costs and user response time through this method. Summary of the invention
[0005] In view of the shortcomings of the prior art, the present invention proposes an energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark clusters, which includes: acquiring online data information in real time, inputting the data information into a trained Q network, the Q network predicting the energy consumption-time target for the data information, and the system selecting the solution with the lowest energy consumption-time target for resource allocation according to the energy consumption-time target prediction;
[0006] The training process of the Q network is:
[0007] S1: Get the relevant configuration parameters and execution parameters of the job; initialize DQN parameters; DQN stands for Deep Q-Network, or Q network;
[0008] S2: Calculate the weight coefficient according to the obtained relevant configuration parameters;
[0009] S3: Generate ε-Greedy and Boltzmann joint strategy based on DQN parameters;
[0010] S4: The task scheduler schedules tasks on the working nodes according to the ε-Greedy and Boltzmann joint strategy;
[0011] S5: construct an energy consumption-time model according to the weight coefficient and the execution parameter; construct a reward model according to the task scheduling and the energy consumption-time model, and generate a reward value according to the reward model;
[0012] S6: Update the DQN parameters according to the reward value to obtain updated DQN parameters;
[0013] S7: Repeat steps S3-S6. When the energy consumption-time target converges, the training is completed.
[0014] Preferably, the relevant configuration parameters of the job running include: the number of executors, CPU resources and memory resources; the execution parameters of the job running include: job arrival time, job identification, job completion time and job duration.
[0015] Preferably, the ε-Greedy and Boltzmann combined strategy is:
[0016]
[0017]
[0018] Among them, s represents the resource status of the cluster; a is an action, which means selecting a specific physical machine to create an executor and allocate resources; a' represents the action with the maximum Q value; Q(s,a) represents a cumulative reward value that may be obtained when the state is s and the action is a; A represents a random action; ε represents probability, and step represents the time step of the task scheduler exploration.
[0019] Preferably, scheduling tasks for working nodes includes:
[0020] Generate actions based on the ε-Greedy and Boltzmann joint strategy and assign resources to the working nodes accordingly;
[0021] If the task is partially assigned or the task is invalidly assigned, a large negative reward calculated under the energy consumption model will be fed back to the task scheduler;
[0022] If the task is successfully assigned, the positive reward calculated under the energy consumption model will be fed back to the task scheduler.
[0023] Preferably, the formula of the energy consumption-time model is:
[0024]
[0025]
[0026]
[0027] Among them, C0, C1, and C2 represent weight coefficients respectively. Indicates the CPU utilization of the i-th node, represents the memory utilization of the i-th node, t represents the start time of the work node i, t' represents the end time of the work under the current CPU utilization and memory utilization, EA total Indicates the energy consumption generated by the cluster, Avg T The average time for all jobs to run. represents the running time of job j, M represents the number of jobs, n represents the number of nodes, φ represents all jobs, target represents the energy consumption-time target value, Represents the weight of the target.
[0028] Furthermore, the calculation formula for CPU utilization is:
[0029]
[0030] in, Indicates the CPU utilization of the i-th node, n indicates the number of nodes, and i indicates the specific node. Indicates the CPU usage on the i-th node. represents the total amount of CPU on the i-th node, and t represents the running time under the current CPU usage of the i-th node.
[0031] Furthermore, the calculation formula for memory utilization is:
[0032]
[0033] in, represents the memory utilization of the i-th node, Indicates the memory usage on the i-th node, represents the total amount of memory on the i-th node, and t represents the running time under the current memory usage of the i-th node.
[0034] Preferably, the reward model is:
[0035]
[0036]
[0037]
[0038]
[0039]
[0040]
[0041] Among them, EA total It represents the energy consumption of a complete scheduling cluster, EA max Indicates that all the working nodes in the cluster consume energy when running the job at full load. normalized represents the normalized energy consumption, EA epi It represents the energy consumption-related part of the reward value in a trajectory generated by the task scheduler exploration, that is, in an episode; represents the running time of job j, M represents the number of jobs, represents the minimum average completion time of all jobs, represents the normalized average completion time of the job, represents the weight of the target, Indicates the part of the reward value related to the average completion time of the job in an episode, R fixed represents a fixed reward value, R epi Represents the true reward value for successful task assignment.
[0042] Preferably, the formula for generating the reward value is:
[0043]
[0044] Among them, Reward represents the generated reward value, R epi Represents the true reward value for successful task assignment.
[0045] An energy-saving scheduling system based on deep reinforcement learning and heterogeneous Spark clusters, including: a task scheduling module, an energy consumption calculation module, a reward generation module and a DQN parameter update module;
[0046] The task scheduling module is used to explore the cluster environment and schedule jobs according to DQN parameters;
[0047] The energy consumption calculation module is used to calculate the system energy consumption according to the job scheduling result;
[0048] The reward generation module is used to calculate the reward value according to the system energy consumption;
[0049] The DQN parameter updating module is used to update the network DQN parameters using the reward value and feed the DQN parameters back to the task scheduling module.
[0050] The beneficial effects of the present invention are as follows: the present invention takes into account the resource priority allocation problem caused by different energy consumption due to cluster heterogeneity, calculates the energy consumption-time target in the system resource allocation process in a heterogeneous spark cluster environment based on deep reinforcement learning, finds the minimum energy consumption-time target while ensuring that the user response time is met, and schedules resources according to the minimum energy consumption-time target. The present invention can optimize energy consumption targets or multiple SLA targets and save energy and reduce emissions as much as possible, which is of great significance for achieving a balance between cloud service provider (CSP) costs and user response time through this method, and has good economic benefits. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 The system model diagram based on deep reinforcement learning of the present invention;
[0052] Figure 2 Scheduling resource architecture diagram for Spark nodes;
[0053] Figure 3 This is a flowchart of the energy-efficient Spark task scheduling based on deep reinforcement learning in the present invention. DETAILED DESCRIPTION
[0054] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0055] The present invention proposes an energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark clusters, such as Figure 1 As shown, the method includes:
[0056] Obtain online data information under real load on the Spark cluster in real time, input the data information into the trained Q network, and the Q network predicts the energy consumption-time target of the data information. The system selects the solution with the lowest energy consumption-time target for resource allocation based on the energy consumption-time target prediction;
[0057] The training process of the Q network is:
[0058] S1: Get the relevant configuration parameters and execution parameters of the job; initialize DQN parameters; DQN stands for Deep Q-Network, or Q network;
[0059] S2: Calculate the weight coefficient according to the obtained relevant configuration parameters;
[0060] S3: Generate ε-Greedy and Boltzmann joint strategy based on DQN parameters;
[0061] S4: The task scheduler schedules tasks on the working nodes according to the ε-Greedy and Boltzmann joint strategy;
[0062] S5: construct an energy consumption-time model according to the weight coefficient and the execution parameter; construct a reward model according to the task scheduling and the energy consumption-time model, and generate a reward value according to the reward model;
[0063] S6: Update the DQN parameters according to the reward value to obtain updated DQN parameters;
[0064] S7: Repeat steps S3-S6. When the energy consumption-time target converges, the training is completed.
[0065] The task energy efficiency scheduling environment includes: task scheduling module, energy consumption calculation module, reward generation module and DQN parameter update module. In the task energy efficiency scheduling environment, the specific process of training the Q network is as follows:
[0066] In a real Spark cluster environment, different applications in the BigDataBench benchmark toolkit are used as execution loads to collect relevant configuration parameters and execution parameters of job running; the relevant configuration parameters of job running include: the number of executors (executor), CPU resources (cpu) and memory resources (men); the execution parameters of job running include: job arrival time (arrive_time), job identification (job_id), job completion time (finishtime) and job duration (duration).
[0067] like Figure 2As shown in the figure, the cluster manager (master) in the task scheduling module issues allocation instructions to the cluster driver process, and the driver process controls the task scheduler (agent) to allocate resources to each node; the resource allocation includes: CPU usage, memory usage, task type and average running time of the application; initialize the system cluster environment and DQN parameters, and the agent obtains the state S by observing the cluster; the agent explores the environment through the ε-Greedy strategy and generates actions, that is, creating executors for the tasks on the working nodes. Different delay reward values will be obtained when the tasks are completed or failed; in order to increase the agent's exploration ability, the Boltzmann action exploration strategy is used to sample actions according to the probability distribution; first, the number of working nodes is determined to be N, and the value of each action when the state is S is calculated, that is, the Q value of each action, and the Q value is limited to [-100,100], and then P is calculated. n =e Q(s,a) and Finally follow Perform non-uniform sampling output actions on N working nodes; the task scheduler schedules tasks for the working nodes according to the ε-Greedy and Boltzmann joint strategy, and the ε-Greedy and Boltzmann joint strategy is:
[0068]
[0069]
[0070] Among them, s represents the resource state of the cluster; a is an action, which means selecting a specific physical machine to create an executor and allocate resources; a' represents the action with the maximum Q value; Q(s,a) represents a cumulative reward value that may be obtained when the state is s and the action is a, which is an expectation; A represents a random action; ε represents that there is a probability of ε selecting a random action, and a probability of 1-ε selecting the action corresponding to the maximum Q value; ε is a value between 0 and 1. At the beginning, ε will be set relatively large, with a probability of 1-ε selecting the action a generated by the largest value in the Q function, and a probability of ε increasing with time. The machine randomly selects action a in the action space A and enters s' in a state space S; the Q function is actually a neural network, which means that when an action a is taken in a state s, the expected value of the cumulative reward may be obtained. The state space S is the idle physical resources of the working node and the state of the currently running job; step represents the time step of the agent exploration. ε is reduced to the original 0.75 with every 2000 step iterations, so as to ensure that when the agent explores enough episodes, the random exploration probability is reduced to avoid convergence instability. Episode refers to a trajectory generated by the task scheduler exploration.
[0071] The idle physical resources of the working node are expressed as:
[0072]
[0073] The status of the currently running job is represented as:
[0074] {id,cpu,mem,executor}
[0075] like Figure 3 As shown, the task scheduling of the working nodes of the job includes:
[0076] Generate actions based on the ε-Greedy and Boltzmann joint strategy, and dispatch resources to working nodes based on the actions; the actions can be understood as resource scheduling commands;
[0077] If a task is partially assigned or invalidly assigned, a large negative reward will be fed back to the task scheduler;
[0078] If the task is successfully assigned, the positive reward calculated under the energy consumption model will be fed back to the task scheduler.
[0079] The formula of the energy consumption-time model is:
[0080]
[0081]
[0082]
[0083] Among them, C0, C1, and C2 represent weight coefficients respectively. Indicates the CPU utilization of the i-th node, represents the memory utilization of the i-th node, t represents the start time of the work node i, t' represents the end time of the work under the current CPU utilization and memory utilization, EA total Indicates the energy consumption generated by the cluster, Avg T The average time for all jobs to run. represents the running time of job j, M represents the number of jobs, n represents the number of nodes, φ represents all jobs, target represents the energy consumption-time target value, Represents the weight of the target.
[0084] For example, for some IO-intensive tasks:
[0085] EA total =112+9.17U cpu -19.46U mem
[0086] For some kind of cpu intensive task:
[0087] EA total =103+1.97U cpu +2.53U mem
[0088] The calculation formula for cpu utilization is:
[0089]
[0090] in, Indicates the CPU utilization of the i-th node, n indicates the number of nodes, and i indicates the specific node. Indicates the CPU usage on the i-th node. represents the total amount of CPU on the i-th node, and t represents the running time under the current CPU usage of the i-th node.
[0091] The calculation formula for memory utilization is:
[0092]
[0093] in, represents the memory utilization of the i-th node, Indicates the memory usage on the i-th node, represents the total amount of memory on the i-th node, and t represents the running time under the current memory usage of the i-th node.
[0094] The least square method and stepwise regression method are used to calculate the weight coefficient according to the relevant configuration parameters obtained; the specific process is:
[0095] Using multiple linear regression model:
[0096]
[0097] Among them, C0, C1, C2 are regression coefficients, that is, weight coefficients that need to be determined; ξ is an unobservable random error; Assume that there are n sets of energy consumption observations Then we have:
[0098]
[0099] The stepwise regression method can solve the problem that it is difficult to find the optimal solution of the least squares method. The idea is to introduce variables one by one, and the condition for the introduction is that the partial F test of the variable is significant. At the same time, after each variable is introduced, the existing variables are tested. The present invention adopts the least squares method + stepwise regression method to substitute the system usage rate data into the multivariate linear regression model, and the objective function is:
[0100]
[0101] By using the least squares method to estimate the regression coefficients and minimize the objective function, we can get the specific regression equation and thus the values of the weight coefficients C0, C1, and C2.
[0102] The reward model is:
[0103]
[0104]
[0105]
[0106]
[0107]
[0108]
[0109] Among them, EA total It represents the energy consumption of a complete scheduling cluster, EA max Indicates that all the working nodes in the cluster consume energy when running the job at full load. normalized represents the normalized energy consumption, EA epiIt represents the trajectory generated by a task scheduler exploration, that is, the part of the reward value related to energy consumption in an episode; represents the running time of job j, M represents the number of jobs, represents the minimum average completion time of all jobs, represents the normalized average completion time of the job, represents the weight of the target, Indicates the part of the reward value related to the average completion time of the job in an episode, R fixed represents a fixed reward value, R epi Represents the true reward value of successful task assignment; since R epi ∈(0,1), in order to make the agent have a better quantified positive value after selecting the correct schedule a, R fixed Designed to be a large number, for example, R fixed Take 10000.
[0110] The formula for generating reward value is:
[0111]
[0112] For a calculation of the true reward value R epi Another preferred embodiment is: the real reward value R epi Calculated according to the energy efficiency model, the energy efficiency model expression is:
[0113]
[0114] Among them, C1 and C2 represent weight coefficients. represents the average utilization of the CPU, T represents the average running time of the job, Indicates the average memory utilization.
[0115] Feed the calculated reward value back to the agent; update the DQN parameters according to the reward value to obtain the updated DQN parameters; the agent fits the value of the Q function according to the DQN parameters; continuously iterate this process until the current energy consumption-time target converges or exceeds the set total number of times, and obtains the trained Q network according to the latest DQN parameters. DQN saves the current network parameters and ends the iteration. At this time, the agent selects the correct action to execute task scheduling according to the value of the Q function.
[0116] An energy-saving scheduling system based on deep reinforcement learning and heterogeneous Spark clusters, including: a task scheduling module, an energy consumption calculation module, a reward generation module and a DQN parameter update module;
[0117] The task scheduling module is used to explore the cluster environment and schedule jobs according to DQN parameters;
[0118] The energy consumption calculation module is used to calculate the system energy consumption according to the job scheduling result;
[0119] The reward generation module is used to calculate the reward value according to the system energy consumption;
[0120] The DQN parameter updating module is used to update the network DQN parameters using the reward value and feed the DQN parameters back to the task scheduling module.
[0121] The present invention takes into account the resource priority allocation problem caused by different energy consumption due to cluster heterogeneity, calculates the energy consumption-time target in the system resource allocation process in a heterogeneous spark cluster environment based on deep reinforcement learning, finds the minimum energy consumption-time target while ensuring that the user response time is met, and schedules resources according to the minimum energy consumption-time target. The present invention can optimize energy consumption targets or multiple SLA targets and save energy and reduce emissions as much as possible, which is of great significance for achieving a balance between cloud service provider (CSP) costs and user response time through this method, and has good economic benefits.
[0122] It should be noted that the functional modules in the various embodiments of the present disclosure may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules; if the integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or all or part of the technical solution, may be embodied in the form of a software product.
[0123] The above embodiments further illustrate the purpose, technical solutions and advantages of the present invention in detail. It should be understood that the above embodiments are only preferred implementation modes of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. An energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark clusters, It is characterized in that include: Acquire online data information in real time and input the data information into the trained Q network. The Q network predicts the energy consumption-time target of the data information. The system selects the solution with the lowest energy consumption-time target for resource allocation based on the energy consumption-time target prediction. The training process of the Q network is: S1: Get the relevant configuration parameters and execution parameters of the job; initialize DQN parameters; DQN stands for Deep Q-Network, or Q network; S2: Calculate the weight coefficient according to the obtained relevant configuration parameters; S3: Generate ε-Greedy and Boltzmann joint strategy based on DQN parameters; S4: The task scheduler schedules tasks for the working nodes according to the ε-Greedy and Boltzmann joint strategy; the ε-Greedy and Boltzmann joint strategy is: Where s represents the resource state of the cluster; a is an action, which means selecting a specific physical machine to create an executor and allocate resources; a' represents the action with the maximum Q value; Q(s,a) represents a cumulative reward value that may be obtained when the state is s and the action is a; A represents a random action; ε represents the probability, and step represents the time step of the task scheduler exploration; S5: construct an energy consumption-time model according to the weight coefficient and the execution parameter; construct a reward model according to the task scheduling and the energy consumption-time model, and generate a reward value according to the reward model; The formula of the energy consumption-time model is: Among them, C 0 , C 1 , C 2 They represent weight coefficients respectively, Indicates the CPU utilization of the i-th node, represents the memory utilization of the i-th node, t represents the start time of the work node i, t' represents the end time of the work under the current CPU utilization and memory utilization, EA total Indicates the energy consumption generated by the cluster, Avg T The average time for all jobs to run. represents the running time of job j, M represents the number of jobs, n represents the number of nodes, φ represents all jobs, target represents the energy consumption-time target value, Indicates the weight of the target; The reward model is: Among them, EA max Indicates that all the working nodes in the cluster consume energy when running the job at full load. normalized represents the normalized energy consumption, EA epi It represents the energy consumption-related part of the reward value in a trajectory generated by the task scheduler exploration, that is, in an episode; represents the minimum average completion time of all jobs, represents the normalized average completion time of the job, Indicates the part of the reward value related to the average completion time of the job in an episode, R fixed represents a fixed reward value, R epi Indicates the true reward value for successful task assignment; The formula for generating the reward value is: Among them, Reward represents the generated reward value; S6: Update the DQN parameters according to the reward value to obtain updated DQN parameters; S7: Repeat steps S3-S6. When the energy consumption-time target converges, the training is completed.
2. According to claim 1, an energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark clusters, It is characterized in that The relevant configuration parameters for job running include: the number of executors, CPU resources and memory resources; the execution parameters for job running include: job arrival time, job identification, job completion time and job duration.
3. According to claim 1, an energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark clusters, It is characterized in that Task scheduling for working nodes includes: Generate actions based on the ε-Greedy and Boltzmann joint strategy, and dispatch resources to working nodes based on the actions; If the task is partially assigned or the task is invalidly assigned, a large negative reward calculated under the energy consumption model will be fed back to the task scheduler; If the task is successfully assigned, the positive reward calculated under the energy consumption model will be fed back to the task scheduler.
4. The energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark cluster according to claim 1, It is characterized in that The calculation formula for cpu utilization is: in, Indicates the CPU utilization of the i-th node, n indicates the number of nodes, and i indicates the specific node. Indicates the CPU usage on the i-th node. represents the total amount of CPU on the i-th node, and t represents the running time under the current CPU utilization rate of the i-th node.
5. The energy-saving scheduling method based on deep reinforcement learning and heterogeneous Spark cluster according to claim 1, It is characterized in that The calculation formula for memory utilization is: in, represents the memory utilization of the i-th node, Indicates the memory usage on the i-th node, represents the total amount of memory on the i-th node, and t represents the running time under the current memory usage of the i-th node.
6. An energy-saving scheduling system based on deep reinforcement learning and heterogeneous Spark clusters, the system is implemented based on the method according to claim 1, It is characterized in that include: Task scheduling module, energy consumption calculation module, reward generation module and DQN parameter update module; The task scheduling module is used to explore the cluster environment and schedule jobs according to DQN parameters; The energy consumption calculation module is used to calculate the system energy consumption according to the job scheduling result; The reward generation module is used to calculate the reward value according to the system energy consumption; The DQN parameter updating module is used to update the network DQN parameters using the reward value and feed the DQN parameters back to the task scheduling module.