Multi-agent server energy saving method

Abstract

The invention belongs to the technical field of computers, and provides a multi-agent server energy-saving method which autonomously learns an efficient control strategy through interactive learning with an actual server without depending on any preset model; meanwhile, aiming at the problems of huge state and action space faced by the multi-core server, the algorithm structure is optimized, and the complexity of a solution is effectively reduced; besides, an asynchronous updating strategy is adopted, the collaborative optimization effect of the DVFS and the fan is achieved, the load and the resource utilization rate of all assemblies of the server system are effectively balanced, the use loss and the electric energy of service components are reduced, the operation cost is reduced on the whole, and then the competitiveness of the whole server is improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to an energy-saving method for a multi-agent server. Background Art

[0002] With the rapid development of cloud computing technology, the energy consumption of servers is also increasing. Energy consumption is becoming the main cost of data center operations, even exceeding the initial cost of purchasing equipment. In this context, the industry will no longer focus on performance indicators alone, and more and more companies will turn their attention to low-power, high-performance server architecture.

[0003] At present, the energy consumption of cloud data centers is mainly reflected in two aspects: hardware and software. Among them, the hardware aspect includes IT equipment (servers, network equipment) and infrastructure (cooling, power regulation system). According to statistics, the energy consumption of each module is: 56% for servers, 5% for network equipment, 30% for cooling systems, 8% for power regulation systems, and 1% for others (such as lighting). The normal CPU utilization rate of the host is only 10% to 50%. Studies have shown that the energy consumption of server fans accounts for about 6% of the total cooling energy consumption. Usually, in order to control the temperature below the safety threshold, excessive cooling is provided. Although this can prevent damage caused by overheating, it also causes energy waste. Therefore, it is particularly important to reasonably adjust the fan speed to reduce energy consumption and avoid violating thermal constraints.

[0004] Dynamic voltage and frequency scaling (DVFS) is a key technology in dynamic power management (DPM), which achieves a balance between system performance and energy consumption by adjusting the operating voltage and frequency. Although many studies have proposed polynomial time algorithms and heuristic methods to select the optimal V / F level, these methods often rely on the physical model of the chip and workload information, which may sometimes be difficult to obtain.

[0005] In addition, both voltage / frequency and fan speed have a significant impact on the energy efficiency and thermal state of the server. Their joint control can achieve better energy efficiency optimization while meeting temperature constraints. For example, under light load conditions, the DVFS algorithm can reduce the voltage / frequency level to reduce power consumption while reducing heat generation, thereby reducing the cooling demand of the fan. Under high load conditions, the fan control mechanism can actively increase cooling capacity to prevent the voltage / frequency from being reduced due to thermal protection, thereby indirectly improving server performance.

[0006] Based on the above, a multi-agent reinforcement learning method is proposed and applied to multi-core servers to achieve the coordinated optimization of dynamic voltage and frequency scaling (DVFS) and dynamic fan control. Under the premise of ensuring the thermal constraints of the server, power consumption and performance are effectively balanced. Summary of the invention

[0007] In order to solve the above technical problems, the present invention proposes a multi-agent server energy-saving method, which effectively balances the load and resource utilization of each component of the server system, reduces the usage loss and electricity of service components, thereby reducing the overall operating cost, and can be integrated into the system as a software, thereby enhancing the competitiveness of the entire server.

[0008] The technical solution of the present invention is:

[0009] The present invention proposes a multi-agent server energy saving method, comprising the following steps:

[0010] The energy efficiency optimization problem of the server is modeled as a multi-agent reinforcement learning training process. The multi-agent includes a DVFS control agent and a fan control agent that interact with the cooling fans and multi-core processors in the server system environment.

[0011] Constructing a fan control algorithm based on the DQN algorithm to train the fan control agent;

[0012] Constructing a DVFS control algorithm based on an improved Q-learning algorithm to train the DVFS control agent;

[0013] The multi-agent reinforcement learning training process jointly optimizes the DVFS control agent and the fan control agent that have undergone the above training to complete the tuning of server performance and energy consumption.

[0014] Preferably, the state space of the fan control agent is defined as:

[0015] T io , is the latest temperature of the fan inlet and outlet;

[0016] ΔT io , is the temperature difference between the fan inlet and outlet in the most recent time interval;

[0017] T is the temperature change rate of the fan inlet and outlet in the most recent time interval; c_i , is the real-time temperature of the i-th CPU;

[0018] ΔT c_i , is the temperature difference of the i-th CPU;

[0019] is the change rate of the i-th CPU;

[0020] n, is the number of CPUs in the server;

[0021] The action space of the fan control agent is defined as:

[0022] Afan ={FA1,,...,FA m},in:

[0023] FA j , indicating that the fan speed is set to the jth level;

[0024] m, is the number of levels for a given fan speed;

[0025] The incentive function of the fan control agent is defined as:

[0026]

[0027] β and α are super parameters, β is greater than α;

[0028] P fan_cur , is the current power consumption of the fan;

[0029] P fan_max , is the maximum power consumption of the fan;

[0030] p, is a variable determined by the CPU temperature,

[0031] When the CPU temperature is greater than the set threshold T threshold , p = -C;

[0032] When the CPU temperature is less than or equal to the set threshold,

[0033] C, is the target Q network Q′ parameter update frequency.

[0034] Preferably, the fan control agent controls the cooling fan action and calculates the incentive value according to the fan control algorithm, and the fan control algorithm includes:

[0035] S101. Randomly initialize the values ​​Q corresponding to all states and actions, randomly initialize all parameters w of the current Q network, initialize the parameters w′=w of the target Q network Q′, and clear the set D of memory playback units;

[0036] S102.for i from 1 to T, iterate:

[0037] a) Initialize the environment state S to the first state of the current state sequence;

[0038] b) Use the preprocessed environment state S as input in the Q network to obtain the Q value output corresponding to all actions of the Q network, and select the action A with the highest probability in the current Q value output;

[0039] c) Execute the current action A in state S to obtain the preprocessed new state S′ and the environmental reward R;

[0040] d) storing {S, A, R, S′} into the memory playback set D;

[0041] e) S = S′;

[0042] f) Sample m samples from the memory playback set D {S j , A j , R j , S′ j}, calculate the current target Q value y j =R j +γmax a′ Q′(S′ j , A′ j , w′)

[0043] g) Use mean square error loss function Update all parameters w of the Q network through gradient backpropagation of the neural network;

[0044] h) If i%C=1, update the target Q network parameter w′=w;

[0045] i) If the reduction of the loss function is less than the set threshold, the current round of iteration is completed, otherwise go to step b);

[0046] Where T is the number of iterations, S is the environment state, A is the action set, γ is the decay factor, Q is the current Q network, Q′ is the target Q network, m is the number of samples of batch gradient descent, and C is the parameter update frequency of the target Q network Q′.

[0047] Preferably, the state space of the DVFS control agent is defined as:

[0048] S dvfs = {U core_1 , ..., U core_k},in:

[0049] k, is the total number of cores in the server;

[0050] U core_i is the utilization of the kth core;

[0051] The action space of the DVFS control agent is defined as:

[0052] A dvfs ={a core_1 , ..., a core_k}in:

[0053] Each core can choose a set of actions a core_x ∈{Freq1,Freq p}, x∈{1,...,k}, p is the number of different frequencies for each core;

[0054] The incentive function of the DVFS control agent is defined as:

[0055] in:

[0056] Q, is the performance of the server;

[0057] P, is the energy consumption of the server;

[0058] w, is a constant coefficient used to balance power consumption and performance;

[0059] When w>1, the optimization tends to improve performance; when 0<w<1, the optimization tends to reduce power consumption.

[0060] Preferably, the DVFS control agent controls the DVFS action, calculates the incentive value, and evaluates the actual contribution of the DVFS action to the incentive value according to the DVFS control algorithm. The strategy of the DVFS control algorithm is expressed by the following formula:

[0061] new_state={s1→a1,...,s t →a t ; i∈[1,t],a i ∈[Freq1, ..., Freq p ]},

[0062] Q new_state =μ*R dvfs +(1-μ)*Q new_state ,

[0063] Here, μ represents the learning rate.

[0064] Preferably, the multi-agent reinforcement learning training process includes:

[0065] S201. Initialize the DVFS control agent's Q new_state ;

[0066] Initialize the fan control agent to evaluate the Q-network Q fan (S fan , a fan ; θ) weight parameter θ;

[0067] Initialize the fan control agent target Q-network Q′ fan (S fan , a fan ; θ′) weight parameter θ′=θ;

[0068] Initialize memory playback unit D;

[0069] S202. Obtain the initial state S of the fan control agent and the DVFS control agent fan_0 and S dvfs_0 ;

[0070] S203.for i from 1 to T, iterate:

[0071] a) If t%L == 0, Execute a fan_t ;

[0072] b) If t%L==1, get R fan_t and S fan_t+1 , stored in D (S fan_t ,

[0073] a fan_t , R fan_t , S fan_t+1 ), from which a sample (S fan_j , a fan_j , R fan_j , S fan_j+1 ),calculate calculate Error, update Q network parameters;

[0074] c) According to Q new_state Select the corresponding action a dvfs_t , execute the action and obtain the corresponding incentive R dvfs_t , and enter the new state S dvfs_t+1 , according to the formula Q new_state =μ*R dvfs +(1-μ)*Q new_state Update Q new_state State value function.

[0075] Compared with the prior art, the present invention has the following advantages and effects:

[0076] This method does not rely on any preset model. Through interactive learning with the actual server, it can autonomously learn an efficient control strategy. In order to solve the redundant energy consumption problem of multi-core servers when processing user tasks in actual applications, the combined control of processor voltage / frequency and fan can achieve better energy efficiency optimization while meeting temperature constraints.

[0077] In order to solve the huge state and action space problem faced by multi-core servers, the traditional deep Q-learning algorithm is improved to reduce the complexity of the solution. By optimizing the algorithm structure, the complexity of the solution is effectively reduced.

[0078] In addition, to address the problem of asynchronous changes between fans and processor cores, an asynchronous update strategy is adopted to achieve the coordinated optimization effect of DVFS and fans. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 It is a structural diagram of the DQN reinforcement learning algorithm framework in the multi-agent server energy saving method of the present invention;

[0080] Figure 2 A schematic diagram of the interaction between multi-agents and the server system environment in the multi-agent server energy saving method of the present invention;

[0081] Figure 3 It is a flow chart of the energy-saving method of the multi-agent server of the present invention. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the present invention, the present invention is further described in conjunction with specific implementation methods.

[0083] In order to enable those skilled in the art to have a clearer understanding of the technical solution of the present invention, some technical terms appearing in the present invention are briefly described here:

[0084] DVFS (Dynamic Voltage and Frequency Scaling) dynamic voltage and frequency scaling.

[0085] Reinforcement learning is a different algorithm from supervised learning and unsupervised learning. Its basic principle is that the agent learns in the process of interaction with the environment, obtains rewards from environmental feedback, and continuously optimizes its own strategy. The five basic elements that make up reinforcement learning are:

[0086] 1) Environmental state set S (State): describes the environment s in which the agent is at time t t , where s t ∈S, S is the set of all states;

[0087] 2) The action set A (Action) of the agent: describes the action a taken by the agent at time t t , where a t ∈A, A is the set of all actions;

[0088] 3) Environment reward R (Reward): represents the agent in state s at time t t Execute action a t The instantaneous reward R t+1 , used to measure the contribution of current behavior to the ultimate goal;

[0089] 4) Agent’s strategy π: describes a function with input st , output action a t The probability of being executed in the state, that is, π = p(a t |s t );

[0090] 5) Value function: Instantaneous incentives are contributions to a single behavior or environmental state. The goal of reinforcement learning is to expect the best long-term return. This function based on long-term returns is called a value function. Based on the evaluation object, it is divided into state value function and action-state value function. The corresponding expressions are as follows:

[0091] v(s t )=E(R t+1 +γv(s t+1 )|s t )

[0092] q(s t , a t )=E(R t+1 +γq(s t+1 , a t+1 )|s t , a t )

[0093] Among them, γ is the discount factor, which calculates the long-term return.

[0094] The specific process of reinforcement learning training is as follows: the agent obtains the environment state information s t , according to the strategy π, select the execution action a t ; After executing the action, the environment enters a new state s t+1 At the same time, the instantaneous stimulus R corresponding to the action is given t+1 ; The agent optimizes the strategy parameters according to the incentives obtained. Under the new strategy parameters, the state s t+1 Execute a new action selection strategy; the above process is repeated until the value function converges.

[0095] The Q-learning algorithm, as one of the most popular algorithms in the field of reinforcement learning, is centered on the value function Q(s,a), which is used to measure the expected reward of performing action a in a specific state s. The higher the Q value, the greater the expected reward of the action. In order to update these Q values ​​efficiently, we usually construct a Q table to record the Q value corresponding to each state-action combination. During the training process, the agent continuously updates the Q table through continuous interaction with the environment using the following formula:

[0096] Q(s,a)←Q(s,a)+α[r+γmax a′ Q(s′, a′)-Q(s, a)]

[0097] In this formula, α represents the learning rate, which determines the degree of influence of new information on the Q value update; r is the immediate reward after executing the action; γ is the discount factor of future rewards, which reflects the value of future rewards relative to immediate rewards; s′ represents the new state reached after the action is executed; and a′ is the optimal action that can be taken in the new state. Although Q-learning is highly favored in reinforcement learning, it also faces some limitations, such as poor generalization ability and dimensionality curse.

[0098] like Figure 1 to Figure 3 As shown, the present invention provides a multi-agent server energy saving method, which specifically includes the following steps:

[0099] The energy efficiency optimization problem of the server is modeled as a multi-agent reinforcement learning training process. The multi-agent includes a DVFS control agent and a fan control agent that interact with the cooling fans and multi-core processors in the server system environment.

[0100] Construct a fan control algorithm based on the DQN algorithm to train the fan control agent;

[0101] Construct a DVFS control algorithm based on the improved Q-learning algorithm to train the DVFS control agent;

[0102] The multi-agent reinforcement learning training process optimizes the DVFS control agent and fan control agent trained above to optimize server performance and energy consumption. Figure 2 As shown, the method provided by the present invention combines dynamic voltage frequency scaling (DVFS) control with fan dynamic control. The server system environment consists of a multi-core processor server equipped with a cooling fan. The multi-agent includes a fan control agent and a DVFS control agent. The fan control agent will control the fan action (adjust the fan speed) according to the current temperature state of the multi-core server, and calculate the reward (incentive value) obtained based on the fan power and temperature state after the action is executed. The DVFS control agent will dynamically adjust the frequency of each processor core in the multi-core server according to the core state, and calculate the reward (incentive value) based on the energy consumption and performance of the server.

[0103] Since the power consumption of cooling fans is also part of the total server power, the impact of fan actions on the total power of multi-core servers needs to be excluded when calculating the reward of DVFS actions. Therefore, the DVFS control agent receives the most recently executed fan actions (i.e., the fan speed selected by the fan control agent) from the fan control agent and evaluates the actual contribution of the DVFS action to the overall reward.

[0104] The thermal state of the server is related to the temperature and the temperature change. Therefore, the state space of the fan control agent is defined as:

[0105] in:

[0106] T io , is the latest temperature of the fan inlet and outlet;

[0107] ΔT io , is the temperature difference between the fan inlet and outlet in the most recent time interval;

[0108] The temperature change rate of the fan inlet and outlet in the most recent time interval;

[0109] T c_i , is the real-time temperature of the i-th CPU;

[0110] ΔT c_i , is the temperature difference of the i-th CPU;

[0111] is the change rate of the i-th CPU;

[0112] n is the number of CPUs in the server.

[0113] Assuming that the fan speed has m levels, the action space of the fan control agent is defined as:

[0114] A fan ={FA1,,...,FA m},in:

[0115] FA j , indicating that the fan speed is set to the jth level;

[0116] m, is the number of levels for a given fan speed.

[0117] The incentive function of the fan control agent is defined as:

[0118] in:

[0119] β and α are hyperparameters that need to be adjusted according to the specific application, and β should be greater than α to ensure that the fan control agent should receive negative incentives when the temperature exceeds the set threshold;

[0120] P fan_cur , is the current power consumption of the fan;

[0121] P fan_max , is the maximum power consumption of the fan;

[0122] p, is a variable determined by the CPU temperature,

[0123] When the CPU temperature is greater than the set threshold T threshold , p = -C;

[0124] When the CPU temperature is less than or equal to the set threshold T threshold hour,

[0125] C is the parameter update frequency of the target Q network Q′.

[0126] The fan control agent controls the cooling fan action and calculates the incentive value according to the fan control algorithm. The fan control algorithm based on the DQN algorithm is used to calculate the state space (S fan ) and action space (A fan ) is processed. Specifically, the fan control algorithm includes:

[0127] S101. Randomly initialize the values ​​Q corresponding to all states and actions, randomly initialize all parameters w of the current Q network, initialize the parameters w′=w of the target Q network Q′, and clear the set D of memory playback units;

[0128] S102.for i from 1 to T, iterate:

[0129] a) Initialize the environment state S to the first state of the current state sequence;

[0130] b) Use the preprocessed environment state S as input in the Q network to obtain the Q value output corresponding to all actions of the Q network, and select the action A with the highest probability in the current Q value output;

[0131] c) Execute the current action A in state S to obtain the preprocessed new state S′ and the environmental reward R;

[0132] d) storing {S, A, R, S′} into the memory playback set D;

[0133] e) S = S′;

[0134] f) Sample m samples from the memory playback set D {S j , A j , R j , S′ j}, calculate the current target Q value y j =R j +γmax a′ Q′(S′ j , A′ j , w′);

[0135] g) Use mean square error loss function Update all parameters w of the Q network through gradient backpropagation of the neural network;

[0136] h) If i%C=1, update the target Q network parameter w′=w;

[0137] i) If the reduction of the loss function is less than the set threshold, the current round of iteration is completed, otherwise go to step b);

[0138] Where T is the number of iterations, S is the environment state, A is the action set, γ is the decay factor, Q is the current Q network, Q′ is the target Q network, m is the number of samples of batch gradient descent, and C is the parameter update frequency of the target Q network Q′.

[0139] The above fan control algorithm is based on the DQN algorithm, and its specific execution process is as follows:

[0140] Input: number of iterations T, environment state S (S fan ), action set A(A fan ), decay factor γ, current Q network Q, target Q network Q′, number of samples for batch gradient descent m, target Q network Q′ parameter update frequency C;

[0141] Execution: Run according to the above steps S101 and S102;

[0142] Output: Q network parameters (Q-network).

[0143] The state space of the DVFS control agent is the state of each core processor in the multi-core server. The utilization of each core will be used to represent its state. Therefore, the state space of the DVFS control agent is defined as the vector S dvfs = {U core_1 , ..., U core_k},in:

[0144] k, is the total number of cores in the server;

[0145] U core_i , is the utilization of the kth core.

[0146] Assuming that each core processor has p levels of frequency, the action space of the DVFS control agent is expressed as: A dvfs ={a core_1 , ..., a core_k},in:

[0147] Each core can choose a set of actions a core_x ∈{Freq1, ..., Freq p}, x∈{1,...,k}, and p is the number of different frequencies for each core.

[0148] Given that the adjustment of the core voltage / frequency of the core processor (CPU) affects both its performance and energy consumption, the design of the DVFS control agent incentive needs to consider both. Therefore, the incentive function of the DVFS control agent is defined as: in:

[0149] Q, is the performance of the server;

[0150] P, is the energy consumption of the server;

[0151] w, is a constant coefficient used to balance power consumption and performance;

[0152] When w>1, the optimization tends to improve performance; when 0<w<1, the optimization tends to reduce power consumption.

[0153] The DVFS control agent controls the DVFS action, calculates the incentive value, and evaluates the actual contribution of the DVFS action to the incentive value according to the DVFS control algorithm. Since the DVFS control agent selects an action for each core in each learning, its action space will grow exponentially with the increase of the total number of cores, resulting in huge storage costs and training overhead. Therefore, how to make the action space independent of the number of cores, thereby reducing the overhead, is an urgent problem to be solved by the existing DVFS control algorithm. In view of this problem, the present invention proposes an improved Q learning method based on the state-action transition strategy to replace the existing DVFS control algorithm.

[0154] Specifically, in a multi-core server system, the DVFS control algorithm needs to make decisions on the actions of each core. The traditional processing method considers the joint state and action space of all cores. For a system with k cores, each core can be set to p different frequency levels {Freq1, ..., Freq p If the core utilization is evenly divided into t intervals from 0% to 100%, and each interval corresponds to a core state, there are t core states {s1, ..., s t}, at this point, the Q table will contain t k *p k The increase of state-action value will lead to a significant increase in storage cost and training cost, which may affect the convergence speed and decision quality of the algorithm.

[0155] To this end, the DVFS control algorithm provided by the present invention is based on a Q-learning algorithm of a state-action transition strategy, which includes all possible core states and corresponding actions specified for each state, independent of the core of the system. The specific strategy is expressed by the following formula:

[0156] new_state={s1→a1,...,s t →a t; i∈[1,t],a i ∈[Freq1, ..., Freq p ]},

[0157] At this point, the current total number of states becomes p t , and its corresponding value update strategy is as follows:

[0158] Q new_state =μ*R dvfs +(1-μ)*Q new_state ,

[0159] Here, μ represents the learning rate.

[0160] As mentioned above, since the power consumption of the cooling fan is also part of the total power of the server, DVFS (dynamic voltage frequency scaling) control and dynamic fan control need to be jointly optimized to complete the tuning of server performance and energy consumption.

[0161] However, there is a problem of asynchronous response frequency between multi-core server systems and cooling devices (cooling fans). Usually, the cooling effect of the fan takes several seconds to stabilize, while the change of DVFS may be at the ms or even ns level.

[0162] To this end, in the method provided by the present invention, a decision is made for the DVFS control agent at each time step, while a decision is made for the fan control agent at every L time steps, so as to achieve a collaborative optimization effect of DVFS and cooling fans through an asynchronous update strategy.

[0163] Specifically, the multi-agent reinforcement learning training process in the method provided by the present invention includes:

[0164] First, enter the server core count and cooling fan speed information;

[0165] Next, follow these process steps:

[0166] S201. Initialize the DVFS control agent's Q new_state ;

[0167] Initialize the fan control agent to evaluate Q-networkQ fan (S fan , a fan ; θ) weight parameter θ;

[0168] Initialize the fan control agent target Q-network Q′ fan (S fan , a fan ; θ′) weight parameter θ′=θ;

[0169] Initialize memory playback unit D;

[0170] S202. Obtain the initial state S of the fan control agent and the DVFS control agent fan_0 and S dvfs_0 ;

[0171] S203.for i from 1 to T, iterate:

[0172] a) If t%L == 0,

[0173] Execute a fan_t ;

[0174] b) If t%L==1, get R fan_t and S fan_t+1 , stored in D (S fan_t ,

[0175] a fan_t , R fan_t , S fan_t+1 ), from which a sample (S fan_j , a fan_j , R fan_j , S fan_j+1 ),calculate calculate Error, update Q network parameters;

[0176] c) According to Q new_state Select the corresponding action a dvfs_t , execute the action and obtain the corresponding incentive R dvfs_t , and enter the new state S dvfs_t+1 , according to the formula Q new_state =μ*R dvfs +(1-μ)*Q new_state Update Q new_state State value function;

[0177] Finally, the output of DVFS controls the agent’s Q new_state and the Q-network of the fan control agent.

[0178] The multi-agent server energy-saving method provided by the present invention effectively balances the load and resource utilization of each component of the server system, reduces the usage loss and electricity of service components, thereby reducing the overall operating cost, and can be integrated into the system as a software, thereby enhancing the competitiveness of the entire server.

[0179] It should be noted that the multi-agent server energy-saving method provided by the present invention can be used not only for multi-core servers, but also for energy saving in data centers, as well as for energy saving in personal notebooks, industrial electricity and other fields.

[0180] In summary, the multi-agent server energy-saving method provided by the present invention does not need to rely on any preset model, and can autonomously learn efficient control strategies through interactive learning with actual servers. At the same time, in response to the huge state and action space problems faced by multi-core servers, an improved Q-learning algorithm is proposed, which effectively reduces the complexity of the solution by optimizing the algorithm structure. In addition, in view of the problem of asynchronous changes between fans and processor cores, an asynchronous update strategy is adopted to achieve the coordinated optimization effect of DVFS and fans.

[0181] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. All equivalent changes and modifications made within the scope of the present invention should still fall within the scope of the present invention.

Claims

1. A multi-agent server energy saving method, It is characterized in that The following steps are involved: The energy efficiency optimization problem of the server is modeled as a multi-agent reinforcement learning training process. The multi-agent includes a DVFS control agent and a fan control agent that interact with the cooling fans and multi-core processors in the server system environment. Constructing a fan control algorithm based on the DQN algorithm to train the fan control agent; Constructing a DVFS control algorithm based on an improved Q-learning algorithm to train the DVFS control agent; The multi-agent reinforcement learning training process jointly optimizes the DVFS control agent and the fan control agent that have undergone the above training to complete the tuning of server performance and energy consumption.

2. The multi-agent server energy saving method according to claim 1, It is characterized in that The state space of the fan control agent is defined as: in: T io , is the latest temperature of the fan inlet and outlet; ΔT io , is the temperature difference between the fan inlet and outlet in the most recent time interval; The temperature change rate of the fan inlet and outlet in the most recent time interval; T c_i , is the real-time temperature of the i-th CPU; ΔT c_i , is the temperature difference of the i-th CPU; is the change rate of the i-th CPU; n, is the number of CPUs in the server; The action space of the fan control agent is defined as: A fan ={FA 1 , , ..., F.A. m }, in which: FA j , indicating that the fan speed is set to the jth level; m, is the number of levels for a given fan speed; The incentive function of the fan control agent is defined as: in: β and α are super parameters, β is greater than α; P fan_cur , is the current power consumption of the fan; P fan_max , is the maximum power consumption of the fan; p, is a variable, When the CPU temperature is greater than the set threshold T threshold , p = -C; When the CPU temperature is less than or equal to the set threshold, C, is the target Q network Q′ parameter update frequency.

3. The multi-agent server energy saving method according to claim 2, It is characterized in that The fan control agent controls the cooling fan action and calculates the incentive value according to the fan control algorithm, and the fan control algorithm includes: S101. Randomly initialize the values ​​Q corresponding to all states and actions, randomly initialize all parameters w of the current Q network, initialize the parameters w′=w of the target Q network Q′, and clear the set D of memory playback units; S102.for i from 1 to T, iterate: a) Initialize the environment state S to the first state of the current state sequence; b) Use the preprocessed environment state S as input in the Q network to obtain the Q value output corresponding to all actions of the Q network, and select the action A with the highest probability in the current Q value output; c) Execute the current action A in state S to obtain the preprocessed new state S′ and the environmental reward R; d) storing {S, A, R, S′} into the memory playback set D; e) S = S′; f) Sample m samples from the memory playback set D {S j , A j , R j , S′ j }, calculate the current target Q value y j =R j +γmax a′ Q′(S′ j , A′ j , w′); g) Use mean square error loss function Update all parameters w of the Q network through gradient backpropagation of the neural network; h) If i%C=1, update the target Q network parameter w′=w; i) If the reduction of the loss function is less than the set threshold, the current round of iteration is completed, otherwise go to step b); Among them, T is the number of iterations, S is the environment state, A is the action set, γ is the decay factor, Q is the current Q network, Q′ is the target Q network, m is the number of samples of batch gradient descent, and C is the parameter update frequency of the target Q network Q′.

4. The multi-agent server energy saving method according to claim 3, It is characterized in that The state space of the DVFS control agent is defined as: S dvfs ={U core_1 , ...U core_k }, in which: k, is the total number of cores in the server; U core_i , is the utilization of the kth core; The action space of the DVFS control agent is defined as: A dvfs ={a core_1 ,...,a core_k }, in which: Each core can choose a set of actions a core_x ∈{Freq 1 ,...,Freq p }, x∈{1,...,k}, p is the number of different frequencies for each core; The incentive function of the DVFS control agent is defined as: in: Q, is the performance of the server; P, is the energy consumption of the server; w, is a constant coefficient used to balance power consumption and performance; When w>1, optimization tends to improve performance; when 0<w<1, optimization tends to reduce power consumption.

5. The multi-agent server energy saving method according to claim 4, It is characterized in that The DVFS control agent controls the DVFS action, calculates the incentive value, and evaluates the actual contribution of the DVFS action to the incentive value according to the DVFS control algorithm. The strategy of the DVFS control algorithm is expressed by the following formula: new_state={s 1 →a 1 ,...,s t →a t ;i∈[1,t],a i ∈[Freq 1 ,...,Freq p ]}, Q new_state =μ*R dvfs +(1-μ)*Q new_state , Here, μ represents the learning rate.

6. The multi-agent server energy saving method according to claim 5, It is characterized in that The multi-agent reinforcement learning training process includes: S201. Initialize the DVFS control agent's Q new_state ; Initialize the fan control agent to evaluate the Q-network Q fan (S fan , a fan ; θ) weight parameter θ; Initialize the fan control agent target Q-network Q′ fan (S fan , a fan ; θ′) weight parameter θ′=θ; Initialize memory playback unit D; S202. Obtain the initial state S of the fan control agent and the DVFS control agent fan_0 and S dvfs_0 ; S203.for i from 1 to T, iterate: a) If t%L == 0, Execute a fan_t ; b) If t%L==1, get R fan_t and S fan_t+1 , stored in D (S fan_t , a fan_t , R fan_t , S fan_t+1 ), from which a sample (S fan_j , a fan_j , R fan_j , S fan_j+1 ),calculate calculate Error, update Q network parameters; c) According to Q new_state Select the corresponding action a dvfs_t , execute the action and obtain the corresponding incentive R dvfs_t , and enter the new state S dvfs_t+1 , according to the formula Q new_state =μ*R dvfs +(1-μ)*Q new_state Update Q new_state State value function.

Images