Server cluster adaptive energy consumption control method and system

Abstract

The invention discloses a self-adaptive energy consumption control method and system for a server cluster, relates to the technical field of computer system energy consumption optimization, and aims to process state data and update a model through a deep reinforcement learning agent module to realize self-adaptive control. The method has the advantages that complex environment laws can be learned autonomously, and accurate control instructions can be generated in real time, so that refined energy consumption regulation and control of the server cluster can be realized on the premise of guaranteeing the service quality, the energy utilization efficiency is improved, and waste is reduced.

Description

Technical Field

[0001] This application relates to the field of computer system energy consumption optimization technology, and in particular to a method and system for adaptive energy consumption control of server clusters. Background Technology

[0002] In the current wave of technological development, server clusters, as the core infrastructure for data processing and storage, are increasingly facing challenges in energy consumption. With the explosive growth of data volume and constantly changing business demands, server clusters are expanding in scale, leading to a sharp increase in energy consumption. High energy consumption not only increases operating costs for enterprises but also has a significant impact on the environment. Therefore, effectively reducing energy consumption while ensuring server cluster performance has become a pressing technical challenge. Traditional energy control methods are often based on fixed rules and strategies, making it difficult to adapt to the complex and ever-changing operating environment of server clusters. Adaptive energy control methods, on the other hand, can dynamically adjust control strategies based on the real-time status and operational needs of the server cluster, achieving optimized energy management. However, existing adaptive energy control methods still suffer from low efficiency and insufficient accuracy when dealing with large-scale, highly complex server clusters.

[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0004] The main purpose of this application is to provide a server cluster adaptive energy consumption control method and system, which aims to improve the energy utilization efficiency of server clusters.

[0005] To achieve the above objectives, this application proposes an adaptive energy consumption control method for server clusters, the method comprising: The raw status data of the server cluster is obtained through the status monitoring module; The original state data is input into the deep reinforcement learning agent module for processing to generate control action data; The strategy execution module converts the control action data into control commands and applies them to the server cluster. After the control command is executed, new server cluster status data is obtained; Based on the original state data, control action data, and new server cluster state data, instant reward data is generated through the reward calculation module. The deep reinforcement learning agent module is updated using the original state data, control action data, instant reward data, and new server cluster state data through the model training module.

[0006] In one embodiment, the step of inputting the raw state data into a deep reinforcement learning agent module for processing to generate control action data includes: The original state data is standardized to generate standardized state data; Standardized state data is input into a neural network model for processing to generate high-level feature representation data. High-level feature representation data is mapped to action space data to generate control action data.

[0007] In one embodiment, the step of inputting standardized state data into a neural network model for processing to generate high-level feature representation data includes: Standardized state data is input into the evaluation network for processing, generating evaluation network output data. The evaluation network output data is input into a noise-adding layer for processing, generating noisy feature data as high-level feature representation data.

[0008] In one embodiment, the step of converting the control action data into control commands through the policy execution module and applying them to the server cluster includes: The control action data is parsed to generate frequency adjustment instructions, which are used to adjust the server CPU operating frequency; Parse control action data to generate core management instructions, which are used to enable or disable CPU cores; Parse control action data to generate migration instructions, which are used to migrate virtual machines between physical servers; The control action data is parsed to generate power control commands, which are used to turn the server equipment on or off.

[0009] In one embodiment, the step of generating instant reward data through the reward calculation module based on the original state data, control action data, and new server cluster state data includes: Extract the first total power consumption data from the original state data; Extract the second total power consumption data from the new server cluster status data; Calculate the difference between the second total power consumption data and the first total power consumption data to generate power consumption change data; Extract task response time data from the new server cluster status data; The task response time data is compared with the service level agreement threshold to generate service evaluation data; By combining the power consumption change data and the service evaluation data, real-time reward data is generated.

[0010] In one embodiment, the step of comparing the task response time data with a service level agreement threshold to generate service evaluation data includes: Set the upper and lower thresholds for task response time; When the task response time data is less than the lower limit of the threshold, positive service evaluation data is generated; When the task response time data exceeds the upper limit of the threshold, negative service evaluation data is generated; Neutral service evaluation data is generated when the task response time data is between the upper and lower limits of the threshold.

[0011] In one embodiment, the step of updating the deep reinforcement learning agent module using the original state data, control action data, immediate reward data, and new server cluster state data through the model training module includes: Combine the original state data, control action data, instant reward data, and new server cluster state data to form an experience data tuple; Store the experience data tuple into the experience replay buffer; Multiple sets of empirical data tuples are extracted from the buffer to form batch training data; The model gradient data is calculated using the batch training data. The deep reinforcement learning agent module is updated using the model gradient data.

[0012] In one embodiment, the step of calculating model gradient data using the batch training data includes: New server cluster status data is processed through the target network to generate value estimation data. By combining the instant reward data and the value estimation data, target value data is generated; By evaluating the network's processing of raw state data and control action data, practical value data is generated. By comparing the target value data with the actual value data, value difference data is generated; Calculate model gradient data based on value difference data.

[0013] In one embodiment, the raw status data includes CPU utilization data of each server, memory utilization data of each server, real-time power consumption data of each server, temperature data of each server, and task queue length data of the task scheduling system.

[0014] Furthermore, to achieve the above objectives, this application also proposes a server cluster adaptive energy consumption control system, which includes: a memory, a processor, and a server cluster adaptive energy consumption control program stored in the memory and executable on the processor. The server cluster adaptive energy consumption control program is configured to implement the steps of the server cluster adaptive energy consumption control method.

[0015] The adaptive energy consumption control method and system for server clusters proposed in this application achieve adaptive control by processing state data and updating the model through a deep reinforcement learning intelligent agent module. It has the ability to autonomously learn the laws of complex environments and generate precise control commands in real time, thereby enabling refined energy consumption control of server clusters while ensuring service quality, improving their energy utilization efficiency and reducing waste. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart illustrating an embodiment of the server cluster adaptive energy consumption control method of this application; Figure 2 This is a schematic diagram of a structural embodiment of the server cluster adaptive energy consumption control system of this application.

[0019] Explanation of icon numbers: 10. Memory; 20. Processor.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of this application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0022] It should be understood that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] In existing technologies, server cluster energy consumption control methods are often based on fixed rules and strategies, making it difficult to adapt to the complex and ever-changing operating environment of server clusters. When dealing with large-scale, highly complex server clusters, these methods suffer from low efficiency and insufficient accuracy, resulting in poor energy consumption optimization and management, increased operating costs, and environmental impact.

[0024] Based on this, embodiments of this application provide a server cluster adaptive energy consumption control method, referring to... Figure 1 The server cluster adaptive energy consumption control method includes steps S100 to S600, wherein: Step S100: Obtain the raw status data of the server cluster through the status monitoring module; Step S200: Input the original state data into the deep reinforcement learning agent module for processing to generate control action data; Step S300: The control action data is converted into control commands by the strategy execution module and applied to the server cluster; Step S400: After the control command is executed, obtain new server cluster status data; Step S500: Based on the original state data, control action data, and new server cluster state data, generate instant reward data through the reward calculation module; Step S600: The deep reinforcement learning agent module is updated using the original state data, control action data, instant reward data, and new server cluster state data through the model training module.

[0025] In this embodiment, the state monitoring module is responsible for collecting real-time operational information of the server cluster, such as CPU utilization, memory usage, real-time power consumption, temperature, and task queue length of each server. This data constitutes the raw state data of the server cluster, providing a basis for subsequent decision-making. The deep reinforcement learning agent module contains one or more deep neural networks, capable of receiving the raw state data of the server cluster and generating corresponding control action data through learned strategies. This agent continuously learns and optimizes through interaction with the environment and reward feedback.

[0026] In this embodiment, the control action data is the decision output generated by the deep reinforcement learning agent module based on the current server cluster state, representing specific operational suggestions for optimizing the energy consumption of the server cluster. For example, this data may indicate adjusting CPU frequency, enabling or disabling CPU cores, migrating virtual machines, or controlling server power status. The policy execution module is responsible for parsing the control action data generated by the deep reinforcement learning agent module and converting it into control instructions that the server cluster can recognize and execute. These instructions are then sent to various components in the server cluster to achieve dynamic adjustment of energy consumption. The control instructions are specific operation commands generated by the policy execution module based on the control action data, directly acting on the hardware or software resources of the server cluster. For example, these instructions may include frequency adjustment instructions, core management instructions, virtual machine migration instructions, or power control instructions.

[0027] In this embodiment, the reward calculation module calculates and generates instant reward data based on the state changes of the server cluster before and after executing the control command, as well as the control action itself. This reward data is used to evaluate the quality of the current control action, providing feedback signals for the policy learning of the deep reinforcement learning agent module. The instant reward data is a value generated by the reward calculation module after comprehensively evaluating factors such as the energy consumption changes and service performance of the server cluster. A positive instant reward indicates that the control action has achieved good results, while a negative instant reward indicates poor results. This data is a key basis for the agent's learning and optimization. The model training module is responsible for continuously training and updating the deep reinforcement learning agent module using the collected original state data, control action data, instant reward data, and new server cluster state data. Through training, the agent can continuously optimize its decision-making strategy to achieve more efficient energy consumption control.

[0028] In this embodiment, the server cluster adaptive energy consumption control method dynamically optimizes the energy consumption of the server cluster through a deep reinforcement learning agent. First, the raw state data of the server cluster is acquired through a state monitoring module. This process may involve deploying lightweight agents on various physical or virtual servers in the server cluster. These agents periodically collect server operating metrics, such as CPU load, memory usage, network throughput, and system logs. This collected data is then aggregated and transmitted to the state monitoring module to form the raw state data of the server cluster. Alternatively, an integrated hardware monitoring interface can be used to directly obtain physical parameters such as power consumption and temperature from the server's hardware level and incorporate them as part of the raw state data.

[0029] Next, the raw state data is input into a deep reinforcement learning agent module for processing to generate control action data. The raw state data is directly input into the input layer of the deep reinforcement learning agent module. This agent module contains one or more neural network structures, which are pre-trained to recognize state patterns and output corresponding actions. For example, this neural network could be a multilayer perceptron, which receives the raw state data as input, processes it through a series of hidden layers, and generates an action vector at the output layer. Each component of this action vector represents a potential control action or its intensity.

[0030] Next, the policy execution module converts the control action data into control commands and applies them to the server cluster. After receiving the control action data generated by the deep reinforcement learning agent module, the policy execution module parses this abstract action data into specific control commands that the server cluster can directly understand and execute, based on preset rules or mapping tables. For example, if the control action data indicates reducing the load on a server, the policy execution module can generate a command instructing the server's virtualization management program to adjust its virtual machine scheduling policy or send a command to the operating system to reduce the CPU frequency. These commands are then sent to the target server or its management system through a standardized management interface or API.

[0031] Subsequently, after the control commands are executed, new server cluster status data is acquired. After the control commands sent by the policy execution module have been executed by the server cluster, or after a preset waiting time, the status monitoring module restarts to comprehensively collect data on the current operating status of the server cluster. This newly collected data, including CPU utilization, memory usage, real-time power consumption, temperature, and task queue length, is considered the new server cluster status data. This step ensures timely feedback on the effects of control actions, providing the latest environmental information for subsequent decision-making cycles.

[0032] Furthermore, based on the original state data, control action data, and new server cluster state data, the reward calculation module generates real-time reward data. The reward calculation module receives server cluster state data before and after the execution of the control action, as well as the executed control action data. This module can use a preset reward function, for example, simply comparing the total energy consumption change before and after the control action execution; if energy consumption decreases, a positive reward is given; if energy consumption increases, a negative reward is given. Alternatively, the reward function can also comprehensively consider the balance between energy consumption changes and service quality (such as task response time) to generate real-time reward data reflecting the overall effect of the control action.

[0033] Finally, the deep reinforcement learning agent module is updated using the original state data, control action data, immediate reward data, and new server cluster state data through the model training module. The model training module treats the collected original state data, control action data, immediate reward data, and new server cluster state data as a complete set of experience tuples. These experience tuples are used to train the deep reinforcement learning agent module. For example, algorithms based on policy gradients or value functions can be used to adjust the weights and biases of the agent's internal neural network according to the immediate reward data, enabling it to learn which control actions to take in different states to obtain higher rewards. This training process can be offline batch processing or online incremental processing to continuously optimize the agent's decision-making ability.

[0034] This method achieves adaptive control of server cluster energy consumption by introducing a deep reinforcement learning agent. It dynamically generates and executes energy consumption optimization control commands based on the real-time operating status of the server cluster, and continuously optimizes the control strategy using a reward feedback mechanism. Therefore, this method effectively solves the problems of low efficiency and insufficient accuracy of traditional methods in complex and variable server cluster environments, thus achieving refined management and optimization of server cluster energy consumption while ensuring service performance, thereby reducing operating costs.

[0035] In one feasible implementation, the raw state data includes CPU utilization data of each server, memory utilization data of each server, real-time power consumption data of each server, temperature data of each server, and task queue length data of the task scheduling system.

[0036] Specifically, CPU utilization data for each server indicates the percentage of the server's processor currently under load. This data can be periodically collected through performance monitoring interfaces provided by the operating system (e.g., the ` / proc / stat` file in Linux or performance counters in Windows). CPU utilization is a key indicator for assessing server computing resource requirements and potential energy consumption; high utilization usually means high energy consumption and high computing pressure, while low utilization may indicate resource waste. Memory utilization data for each server indicates the percentage of physical memory currently used. This data can also be obtained through performance monitoring tools provided by the operating system (e.g., the `free -m` command in Linux or Task Manager in Windows). Memory utilization reflects the application's demand for memory resources; excessively high memory utilization may lead to performance bottlenecks, while excessively low memory utilization indicates that memory resources are not being fully utilized. Real-time power consumption data for each server indicates the server's current power consumption. This data can be monitored and collected in real time through power sensors integrated into the server hardware (e.g., Intel RAPL interface, IPMI interface) or external power management units (PDUs). Real-time power consumption data is a direct indicator of server energy consumption and is crucial for precise energy control. Temperature data for each server indicates the operating temperature of critical internal components (e.g., CPU, GPU, hard drive). This data can be obtained through temperature sensors (e.g., LM-Sensors) on the server motherboard. Temperature data is an important indicator for assessing server heat dissipation and potential hardware risks; excessively high temperatures can lead to performance degradation or even hardware damage, and indirectly reflect the server's energy consumption level. Task queue length data from the task scheduling system indicates the number of tasks waiting to be processed. This data can be extracted through the API interface or log files of the task scheduling system (e.g., Kubernetes, OpenStackNova, Slurm, etc.). Task queue length reflects the current task load and processing capacity of the server cluster and is an important basis for predicting future resource needs and adjusting energy consumption strategies.

[0037] In this embodiment, through the above technical solution, the deep reinforcement learning agent module can obtain more comprehensive, detailed, and multi-dimensional information on the server cluster's operating status. CPU utilization data and memory usage data directly reveal the load on computing and storage resources, enabling the agent to accurately judge the server's resource utilization efficiency; real-time power consumption data provides direct energy consumption feedback, helping the agent to directly optimize energy consumption targets; temperature data provides the agent with an important reference for server health and heat dissipation pressure, avoiding the risk of overheating during energy consumption optimization; and task queue length data from the task scheduling system provides the agent with forward-looking load trend information, enabling it to predict future resource needs. These rich and crucial raw state data work together to enable the deep reinforcement learning agent to establish a more accurate state-action-reward mapping relationship, thereby generating a more intelligent and adaptive control strategy. This not only significantly improves the accuracy and efficiency of energy consumption control, ensuring that service quality is maintained or optimized while reducing energy consumption, but also effectively solves the problem of difficulty in achieving refined energy consumption management with limited data.

[0038] In one feasible implementation, the step of inputting the original state data into a deep reinforcement learning agent module for processing to generate control action data includes: standardizing the original state data to generate standardized state data; inputting the standardized state data into a neural network model for processing to generate high-level feature representation data; and mapping the high-level feature representation data into action space data to generate control action data.

[0039] In this embodiment, the original state data includes various types of data, such as CPU utilization, memory usage, real-time power consumption, temperature, and task queue length of each server. These data often have different dimensions and numerical ranges. To eliminate the differences in dimensions between the data and ensure that all features are equally important for model training, the original state data needs to be standardized. Standardization can be performed using various methods, such as scaling the data to a specific interval (e.g., [0, 1] or [-1, 1]) through min-max normalization, or converting the data to a distribution with a mean of 0 and a standard deviation of 1 through Z-score standardization. After standardization, standardized state data can be generated, which helps improve the training efficiency and stability of the neural network model in the deep reinforcement learning agent module and prevents certain features from dominating the learning process due to excessively large values.

[0040] In this embodiment, although the standardized state data eliminates dimensional differences, it may still contain redundant information or low-level features that are difficult to directly utilize by the deep reinforcement learning agent module. Therefore, the standardized state data needs to be input into one or more neural network models for processing. This neural network model can consist of fully connected layers, convolutional layers, or recurrent layers, and its role is to extract more abstract and representative high-level feature representations from the standardized state data. These high-level feature representations can capture the deep correlations and patterns of the server cluster state, providing more effective information input for subsequent generation of control actions. For example, the neural network model can learn the mutual influence between different server resource utilization rates, or the nonlinear relationship between power consumption and temperature, thereby forming a more comprehensive understanding of the overall operating status of the server cluster.

[0041] In this embodiment, the high-level feature representation data is an abstract description of the server cluster state, which needs to be converted into specific, executable control actions. This step is accomplished by mapping the high-level feature representation data to a predefined action space. The action space defines the set of all possible control actions that the deep reinforcement learning agent module can take, such as adjusting the server CPU operating frequency, enabling or disabling CPU cores, migrating virtual machines between physical servers, or turning server devices on or off. The mapping process is typically implemented by the output layer of the neural network model, which outputs the probability distribution or specific action value corresponding to a certain action in the action space based on the high-level feature representation data. For example, for a discrete action space, the output layer can use the Softmax activation function to generate the probability of each action; for a continuous action space, the output layer can directly output the continuous values ​​of the actions. The final generated control action data is the decision made by the deep reinforcement learning agent module based on the current server cluster state.

[0042] In this embodiment, the above-described technical solution standardizes the original state data, eliminating the dimensional differences between different types of data. This ensures that all state features have similar weights when input into the neural network model, preventing certain features from dominating the learning process due to excessively large values. This significantly improves the training efficiency and stability of the deep reinforcement learning agent module. Furthermore, inputting the standardized state data into the neural network model allows for the extraction of more abstract and representative high-level feature representations from the original, potentially redundant, low-level data. These high-level features can more effectively capture the deep correlations and patterns of the server cluster state, providing more refined and insightful information for the agent module's decision-making. Finally, by mapping these high-level feature representations to action space data, it ensures that the control action data generated by the deep reinforcement learning agent module is based on a deep understanding and effective abstraction of the server cluster state, resulting in more precise and reasonable control actions. This processing effectively addresses the challenge posed by the heterogeneity of the original data to the training of deep reinforcement learning models, enabling the deep reinforcement learning agent module to learn optimal energy consumption control strategies more stably and efficiently, thereby improving the adaptive energy consumption control performance of the server cluster.

[0043] In one feasible implementation, the step of inputting standardized state data into a neural network model for processing to generate high-level feature representation data includes: inputting standardized state data into an evaluation network for processing to generate evaluation network output data; and inputting the evaluation network output data into a noise-adding layer for processing to generate noisy feature data as high-level feature representation data.

[0044] In this embodiment, the evaluation network is a key component of the deep reinforcement learning agent. Its role is to receive state information from the environment, i.e., standardized state data, and output a value assessment or feature representation of these states. The evaluation network is typically a deep neural network capable of extracting meaningful, high-dimensional features from the standardized state data. These features capture the operational status, load patterns, and potential energy consumption factors of the server cluster. The evaluation network can employ structures such as multilayer perceptrons, convolutional neural networks, or recurrent neural networks, depending on the type and complexity of the original state data. Its output data is a feature vector, abstracted and transformed layer by layer by the network, representing the agent's deep understanding of the current state of the server cluster.

[0045] Building upon this, a noise-adding layer is a mechanism used to introduce randomness into the feature data of the evaluation network output. Its role is to increase the exploratory nature of the agent's learning process, preventing the policy from prematurely converging to a suboptimal solution. By superimposing a certain form of noise, such as Gaussian noise, uniform noise, or more complex exploratory noise, onto the deterministic features of the evaluation network output, the agent can generate slightly different high-level feature representations when facing similar states. This randomness prompts the agent to try different control actions, thereby exploring the environment more comprehensively and discovering potentially better strategies. The noise-adding layer can be implemented simply by adding a randomly sampled noise vector to the evaluation network output data, or through more complex mechanisms, such as using a parameterized noise generator whose noise intensity can be dynamically adjusted according to the training process. The resulting noisy feature data serves as the high-level feature representation data required by the deep reinforcement learning agent module for subsequent action generation.

[0046] In this embodiment, the standardized state data is first input into the evaluation network for processing. This process extracts more critical and abstract high-level features for energy consumption control decisions from the original standardized state data, which may contain redundant or low-relevance information. Based on this, the output data of the evaluation network is further processed by a noise-adding layer to generate noisy feature data as high-level feature representation data. This noise-introduction mechanism allows the deep reinforcement learning agent to maintain necessary exploratory behavior during the learning process. Specifically, even under similar server cluster conditions, the introduction of noise allows the agent to generate slightly different high-level feature representations, prompting it to try different control actions. This helps the agent escape local optima, more comprehensively explore the energy consumption optimization space of the server cluster, and discover more robust and adaptive energy consumption control strategies. For example, when facing complex load fluctuations or sudden events, noisy feature representations can prompt the agent to try more diverse frequency adjustment, core management, or virtual machine migration strategies, thereby achieving deeper energy consumption optimization while ensuring service quality and improving the adaptive energy consumption control performance of the entire server cluster.

[0047] In one feasible implementation, the step of converting the control action data into control instructions through the policy execution module and applying them to the server cluster includes: parsing the control action data to generate frequency adjustment instructions for adjusting the server CPU operating frequency; parsing the control action data to generate core management instructions for enabling or disabling CPU cores; parsing the control action data to generate migration instructions for migrating virtual machines between physical servers; and parsing the control action data to generate power control instructions for turning server devices on or off.

[0048] In this embodiment, the frequency adjustment command is used to dynamically adjust the operating frequency of the server's central processing unit (CPU). The CPU's operating frequency directly affects its computing power and power consumption. When control action data indicates a need to reduce power consumption, the policy execution module can parse the data and generate a command to reduce the CPU frequency; when performance needs to be improved to cope with high load, a command to increase the CPU frequency is generated. This can be achieved by calling the CPU frequency management interface provided by the operating system (e.g., through the `cpufreq` subsystem in Linux) or through the interface provided by the server hardware management controller (such as BMC / IPMI). Specific values ​​or codes in the control action data can be mapped to predefined frequency levels (such as low, medium, and high) or specific frequency values.

[0049] In this embodiment, core management instructions are used to dynamically control the enabling and disabling of physical or logical cores in the server CPU. Disabling currently unnecessary CPU cores can significantly reduce the server's power consumption, while enabling more cores can improve its parallel processing capabilities. The policy execution module can generate instructions to enable or disable specific CPU cores based on control action data. This is typically achieved through CPU hot-swapping functionality provided by the operating system (e.g., via the `sysfs` file system in Linux) or APIs provided by the virtualization management platform. The control action data may include a bitmask or a core count value, indicating which cores should be activated or deactivated.

[0050] In this embodiment, migration commands are used to seamlessly move running virtual machines from one physical server to another. This virtual machine migration is a key means of achieving server cluster load balancing, energy optimization, and fault recovery. The policy execution module can generate migration commands containing the source server, target server, and the identifier of the virtual machine to be migrated, based on control action data. This is typically implemented through the application programming interface (API) or command-line tools provided by virtualization management software (such as VMware vMotion, KVM LiveMigration, etc.). For example, when the load on a physical server is too high or its energy consumption exceeds expectations, some virtual machines can be migrated to other servers with lower load or better energy efficiency.

[0051] In this embodiment, power control commands are used to remotely control the power status of physical servers, including power-on, power-off, restart, or hibernation. This is one of the most direct and effective energy consumption control methods. The policy execution module can generate power control commands for the target server based on the control action data. This is typically achieved through the server's out-of-band management interface (such as IPMI, BMC) or the interface provided by the Data Center Infrastructure Management (DCIM) system. The control action data can contain a Boolean value or an enumerated value indicating the power state the target server should be in. For example, during off-peak periods when cluster load is extremely low, commands can be generated to shut down some idle physical servers to maximize energy savings; when the load recovers, power-on commands are generated to meet computing demands.

[0052] In this embodiment, the abstract control action data generated by the deep reinforcement learning agent module is refined and transformed into various specific control instructions that the server cluster can directly execute through the policy execution module. This effectively solves the problem that abstract control data is difficult to directly apply to complex heterogeneous server clusters. The introduction of frequency adjustment instructions, core management instructions, migration instructions, and power control instructions enables the control strategy to intervene in a multi-dimensional and refined manner from the micro level of the CPU to the macro level of the entire server device. This not only ensures that the control actions can be executed accurately and efficiently by the server cluster, but also achieves adaptive and dynamic optimization of the server cluster's energy consumption and performance through these diverse control methods. For example, while ensuring service quality, the system can accurately adjust the CPU frequency, enable or disable CPU cores, and even migrate virtual machines between physical servers or directly shut down idle servers based on real-time status, thereby significantly reducing overall energy consumption. When the load increases, it can quickly perform the opposite operation to ensure service performance and improve resource utilization. This refined and comprehensive control capability greatly enhances the effectiveness and flexibility of adaptive energy consumption control of the server cluster.

[0053] In one feasible implementation, the step of generating instant reward data through the reward calculation module based on the original state data, control action data, and new server cluster state data includes: extracting first total power consumption data from the original state data; extracting second total power consumption data from the new server cluster state data; calculating the difference between the second total power consumption data and the first total power consumption data to generate power consumption change data; extracting task response time data from the new server cluster state data; comparing the task response time data with a service level agreement threshold to generate service evaluation data; and combining the power consumption change data and the service evaluation data to generate instant reward data.

[0054] In this embodiment, extracting the first total power consumption data from the raw state data refers to the overall energy consumption information of the server cluster obtained from the raw state data of the server cluster before the deep reinforcement learning agent module generates control action data and the policy execution module converts it into control commands to act on the server cluster. The raw state data is usually collected in real time by the state monitoring module, which includes the real-time power consumption data of each server. The first total power consumption data can be obtained by summing, aggregating, or weighting the real-time power consumption data of all servers in the cluster. For example, power consumption readings can be obtained from the power management unit (PMU) or intelligent PDU of each server, and then these readings can be accumulated to reflect the baseline energy consumption level before the control action is executed.

[0055] In this embodiment, extracting the second total power consumption data from the new server cluster status data refers to extracting the overall energy consumption information of the cluster from the new server cluster status data obtained by the status monitoring module after the control command is executed. The second total power consumption data is also obtained by aggregating the real-time power consumption data of all servers within the cluster after the control action is executed; this data can be obtained from the PMU or intelligent PDU. This step aims to obtain the actual energy consumption information after the control action is executed, so as to compare it with the energy consumption before execution.

[0056] In this embodiment, the difference between the second total power consumption data and the first total power consumption data is calculated to generate power consumption change data. This step directly quantifies the impact of control actions on the energy consumption of the server cluster. Power consumption change data is a key indicator; its positive and negative values ​​represent increases or decreases in energy consumption, respectively, thus directly reflecting the energy-saving effect of the control actions. This calculation is typically performed internally within the reward calculation module, for example, through simple subtraction.

[0057] In this embodiment, task response time data is extracted from the new server cluster status data. This step aims to obtain an important metric for measuring the Quality of Service (QoS) of the server cluster. Task response time data reflects the time required for a user request or task to be submitted and completed, and is key to assessing whether energy-saving measures affect service performance. Task response time data can be obtained from task scheduling systems, application performance monitoring (APM) tools, or log systems; for example, by recording the start and end times of each task and calculating their differences.

[0058] In this embodiment, the task response time data is compared with Service Level Agreement (SLA) thresholds to generate service evaluation data. This step transforms continuous task response time data into discrete service evaluation data that can be used for reward calculation. The SLA thresholds are preset, defining acceptable service quality standards. By comparing the data with these thresholds, it can be determined whether the current service quality meets the standards. For example, different evaluation results, such as "Good," "Average," or "Poor," or corresponding numerical scores, are generated based on whether the task response time data exceeds the upper limit, falls below the lower limit, or falls between the two.

[0059] In this embodiment, the generation of instant reward data by combining the power consumption change data and the service evaluation data is the core step in reward calculation. It integrates the two key indicators of energy consumption and performance into a single instant reward signal to guide the learning of the deep reinforcement learning agent module. The combination method can employ weighted summation, piecewise functions, or penalty mechanisms. For example, a weighted summation method can be used: Instant Reward = w1 × (negative power consumption change) + w2 × (service evaluation score), where w1 and w2 are weights. A negative power consumption change indicates greater energy saving and a higher reward, and a higher service evaluation score indicates better service quality and a higher reward.

[0060] In this embodiment, the above technical solution not only considers the energy consumption changes caused by control actions when generating instant reward data, but also introduces an evaluation of service quality. Specifically, by extracting the total power consumption data before and after the control action and calculating the difference, the increase or decrease in energy consumption can be directly quantified. Simultaneously, by monitoring task response time data and comparing it with a preset service level agreement threshold, the current service quality can be objectively assessed. Combining the power consumption change data and service evaluation data forms a comprehensive instant reward signal that more comprehensively and accurately reflects the quality of the control actions. This enables the deep reinforcement learning agent module to learn strategies for effectively ensuring or even optimizing service quality while reducing server cluster energy consumption during training, avoiding performance degradation or energy consumption rebound problems that may result from single-objective optimization, thereby achieving a dynamic balance and adaptive optimization between energy consumption and performance.

[0061] In one feasible implementation, the step of comparing the task response time data with a service level agreement (SLA) threshold to generate service evaluation data includes: setting an upper threshold and a lower threshold for the task response time; generating positive service evaluation data when the task response time data is less than the lower threshold; generating negative service evaluation data when the task response time data is greater than the upper threshold; and generating neutral service evaluation data when the task response time data is between the upper threshold and the lower threshold.

[0062] In this embodiment, the upper and lower thresholds for task response time together define an acceptable service performance range. The lower threshold typically represents an ideal or excellent service response time, while the upper threshold represents the maximum response time allowed by the Service Level Agreement (SLA). By setting these two thresholds, the system can divide task response time data into three different performance zones: excellent, acceptable, and unacceptable. These thresholds can be flexibly configured based on the server cluster's historical operating data, business load characteristics, user experience requirements, and preset SLA standards. For example, the lower threshold can be set to 100 milliseconds and the upper threshold to 500 milliseconds based on the business type to ensure the response speed of critical services.

[0063] In this embodiment, when the server cluster's task response time data performs exceptionally well, falling below a preset threshold, it indicates that the system provides outstanding service performance, far exceeding user expectations. In this case, the system generates positive service evaluation data. This positive data, as part of the immediate reward data, sends a strong positive signal to the deep reinforcement learning agent module, encouraging it to maintain or further optimize the current control strategy that leads to high performance. For example, a large positive value (such as +100) can be generated to significantly enhance the agent's preference for the current strategy.

[0064] In this embodiment, conversely, when the task response time data exceeds a preset threshold, it typically signifies that the service quality of the server cluster has deteriorated to an unacceptable level, failing to meet SLA requirements and severely impacting user experience. At this point, the system generates negative service evaluation data. This negative data, as part of the immediate reward data, sends a strong penalty signal to the deep reinforcement learning agent module, prompting it to quickly adjust its current control strategy to avoid recurring service quality issues. For example, a large negative value (such as -100) can be generated to strongly suppress the agent from repeating strategies that lead to performance degradation.

[0065] In this embodiment, when the task response time data falls between the lower and upper thresholds, it indicates that the server cluster's service performance is within an acceptable range, meeting basic service requirements but not yet reaching an excellent level. In this case, the system generates neutral service evaluation data. This neutral data, as part of the immediate reward data, provides a relatively mild feedback to the deep reinforcement learning agent module, neither strongly encouraging nor strongly penalizing it. This allows the agent to explore and optimize energy consumption control strategies while meeting basic service quality, such as further reducing power consumption without sacrificing service quality. The neutral evaluation data can be a zero value, or a small positive or negative value close to zero, to provide guidance for fine-tuning.

[0066] In this embodiment, the above-described technical solution introduces an upper and lower threshold for task response time. Based on the comparison between the task response time data and these two thresholds, positive, negative, or neutral service evaluation data is generated, providing the deep reinforcement learning agent module with more refined and multi-dimensional reward signals. This tiered service evaluation mechanism overcomes the limitations of single threshold comparisons, enabling the agent to more accurately understand the impact of its control actions on service quality. When service performance is excellent, the agent receives strong positive feedback, prompting it to consolidate efficient strategies; when service performance deteriorates, the agent receives strong negative feedback, prompting it to quickly correct erroneous strategies; when service performance is within an acceptable range, the agent receives neutral feedback, allowing it to focus more on energy consumption optimization while meeting basic service quality requirements. Therefore, this solution significantly improves the learning efficiency and control accuracy of the deep reinforcement learning agent module in balancing server cluster energy consumption and service quality, enabling it to more intelligently and adaptively adjust the server cluster's operating state, ultimately achieving better overall performance and energy efficiency.

[0067] In one feasible implementation, the step of updating the deep reinforcement learning agent module using the original state data, control action data, immediate reward data, and new server cluster state data through the model training module includes: combining the original state data, control action data, immediate reward data, and new server cluster state data to form an experience data tuple; storing the experience data tuple in an experience replay buffer; extracting multiple sets of experience data tuples from the buffer to form batch training data; calculating model gradient data using the batch training data; and applying the model gradient data to update the deep reinforcement learning agent module.

[0068] In this embodiment, the experience data tuple is the basic unit used in deep reinforcement learning to record the interaction process between the agent and the environment. It integrates the agent's original state data at a given moment, the control action data taken based on that state, the immediate reward data obtained after executing the action, and the new server cluster state data after the environment transitions, forming a complete four-tuple. This combination ensures that complete information from each interaction is captured, providing the necessary data foundation for subsequent model training. The experience replay buffer is a storage structure used to store experience data tuples generated by the agent's interaction with the environment. Its function is to break the temporal correlation between data, preventing the model from overfitting to the most recent experience during training, thereby improving the stability and efficiency of training. When the agent interacts with the environment and generates a new experience data tuple, this tuple is added to the experience replay buffer, typically using a first-in, first-out (FIFO) strategy; that is, when the buffer reaches its maximum capacity, the oldest experience data tuple is replaced by the newest tuple.

[0069] In this embodiment, to achieve efficient and stable model training, instead of directly using a single empirical data tuple for updates, multiple sets of empirical data tuples are randomly sampled from the empirical replay buffer and combined into a batch of training data. This batch sampling method helps reduce the variance of the training data, making the model update direction more stable, while also fully utilizing the parallel processing capabilities of modern computing hardware (such as GPUs) to accelerate the training process. Model gradient data is crucial information used to guide the direction and magnitude of parameter updates for the deep reinforcement learning agent module. After acquiring the batch training data, the gradients of the neural network parameters in the agent module relative to the loss function are calculated using the backpropagation algorithm of deep learning, combined with a predefined loss function (e.g., a loss function based on temporal difference error). These gradient data indicate how to adjust the model parameters to reduce prediction error or increase expected reward. After calculating the model gradient data, optimization algorithms (e.g., stochastic gradient descent SGD, Adam, RMSprop, etc.) are used to apply these gradient data to the neural network parameters of the deep reinforcement learning agent module. The optimization algorithm adjusts the weights and biases of the neural network based on gradient data, enabling the agent module to better predict the optimal control action or state value, thereby gradually improving its performance in server cluster energy consumption control tasks.

[0070] In this embodiment, through the above technical solution, this application introduces an experience replay mechanism. This mechanism combines the original state data, control action data, immediate reward data, and new server cluster state data generated by the agent's interaction with the environment into experience data tuples, which are then stored in an experience replay buffer. This mechanism effectively breaks the temporal correlation between continuous experience data, avoiding instability during model training. By randomly extracting multiple sets of experience data tuples from the buffer to form batch training data, not only can the variance of the training data be reduced, making model gradient calculation more accurate and stable, but parallel computing resources can also be fully utilized, significantly improving the training efficiency of the deep reinforcement learning agent module. Finally, the calculated model gradient data is used to update the agent module, enabling the agent to learn from diverse historical experiences, thereby obtaining a more robust and generalized energy consumption control strategy and effectively improving the adaptive energy consumption control performance of the server cluster.

[0071] In one feasible implementation, the step of calculating model gradient data using the batch training data includes: processing new server cluster state data through a target network to generate value estimation data; combining instantaneous reward data and the value estimation data to generate target value data; processing raw state data and control action data through an evaluation network to generate actual value data; comparing the target value data and the actual value data to generate value difference data; and calculating model gradient data based on the value difference data.

[0072] In this embodiment, the target network processes new server cluster state data to generate value estimation data. This target network, used as a mechanism for stable training in deep reinforcement learning, is typically a delayed or periodically updated copy of the evaluation network. Its role is to provide a relatively stable value function estimation target, avoiding frequent fluctuations in the value estimation target caused by rapid changes in the evaluation network's own parameters during training. When new server cluster state data is input into the target network, the target network predicts the future potential rewards for these states based on its current parameters, thereby generating value estimation data. This value estimation data represents the long-term cumulative reward that the agent is expected to obtain given a new state.

[0073] The target value data, generated by combining immediate reward data and the value estimation data, refers to the "real" or "expected" value used in deep reinforcement learning to guide the evaluation network's learning. It is calculated by combining the immediate reward data obtained by the agent after performing a control action with the discounted value estimation data of the next state predicted by the target network. Specifically, the target value data can be represented as the immediate reward data plus a discount factor multiplied by the value estimation data. This combination allows the target value to consider not only immediate feedback but also predictions of future gains, thus providing the evaluation network with a more comprehensive and stable learning objective.

[0074] In this embodiment, the evaluation network processes the original state data and control action data to generate actual value data. This means the evaluation network is a core component of the deep reinforcement learning agent module, and its parameters are continuously optimized during training. When the original state data and the agent's chosen control action data are input into the evaluation network, the network predicts the value of this state-action combination based on its current parameters, generating actual value data. This actual value data represents the evaluation network's current estimate of the expected cumulative reward obtainable by performing a specific action in a specific state. Comparing the target value data and the actual value data generates value difference data, also known as temporal difference (TD) error, which is the difference between the target value data and the actual value data. This difference quantifies the degree of inconsistency between the value currently predicted by the evaluation network and a more stable target composed of immediate rewards and the future value estimated by the target network. The magnitude and direction of the value difference data directly reflect the evaluation network's prediction accuracy on the current state-action pair and are crucial for subsequent model updates. Calculating model gradient data based on the value difference data means the model gradient data is used to update the direction and step size information of the evaluation network parameters. Once the value difference data is generated, it is backpropagated through the evaluation network. This process computes the gradient of each trainable parameter in the evaluation network relative to the value difference. These gradients indicate how to adjust the parameters of the evaluation network to reduce the difference between the actual value data and the target value data, thereby making the evaluation network's predictions more accurate. For example, optimization algorithms such as stochastic gradient descent (SGD) or its variants (e.g., Adam, RMSprop) can be used to update the weights and biases of the evaluation network.

[0075] In this embodiment, by introducing a target network to process new server cluster state data to generate value estimation data, and combining it with real-time reward data to form stable target value data, this application effectively solves the problem of large fluctuations in target value during the training process of the evaluation network. The evaluation network generates actual value data by processing raw state data and control action data, and compares it with target value data to accurately generate value difference data. Based on this value difference data, the model gradient data is calculated, which can provide a stable and accurate parameter update direction for the deep reinforcement learning agent module. This mechanism significantly improves the stability and convergence speed of agent learning, avoids training difficulties caused by target drift, and enables the agent to learn the optimal energy consumption control strategy more effectively. Ultimately, this helps the server cluster achieve more refined and robust adaptive energy consumption management, minimizing energy consumption while ensuring service quality.

[0076] In the embodiments of this application, the server cluster adaptive energy consumption control method and system process state data and update the model through a deep reinforcement learning intelligent agent module to achieve adaptive control. It has the ability to autonomously learn the laws of complex environments and generate precise control commands in real time, thereby enabling refined energy consumption control of the server cluster while ensuring service quality, improving its energy utilization efficiency and reducing waste.

[0077] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the adaptive energy consumption control method for server clusters in this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0078] This application also provides a server cluster adaptive energy consumption control system, referenced... Figure 2 The server cluster adaptive energy consumption control system includes: a memory 10, a processor 20, and a server cluster adaptive energy consumption control program stored on the memory 10 and executable on the processor 20. The server cluster adaptive energy consumption control program is configured to implement the steps of the server cluster adaptive energy consumption control method.

[0079] The server cluster adaptive energy consumption control system provided in this application, employing the server cluster adaptive energy consumption control method in the above embodiments, can improve the energy utilization efficiency of the server cluster. Compared with the prior art, the beneficial effects of the server cluster adaptive energy consumption control system provided in this application are the same as those of the server cluster adaptive energy consumption control method provided in the above embodiments, and other technical features in the server cluster adaptive energy consumption control system are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0080] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0081] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. All equivalent structural transformations made under the technical concept of this application using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included within the scope of patent protection of this application.

Claims

1. A server cluster adaptive energy consumption control method, characterized in that, The method includes: The raw status data of the server cluster is obtained through the status monitoring module; The original state data is input into the deep reinforcement learning agent module for processing to generate control action data; The strategy execution module converts the control action data into control commands and applies them to the server cluster. After the control command is executed, new server cluster status data is obtained; Based on the original state data, control action data, and new server cluster state data, instant reward data is generated through the reward calculation module. The deep reinforcement learning agent module is updated using the original state data, control action data, instant reward data, and new server cluster state data through the model training module.

2. The server cluster adaptive energy consumption control method as described in claim 1, characterized in that, The steps of inputting the raw state data into the deep reinforcement learning agent module for processing to generate control action data include: The original state data is standardized to generate standardized state data; Standardized state data is input into a neural network model for processing to generate high-level feature representation data. High-level feature representation data is mapped to action space data to generate control action data.

3. The server cluster adaptive energy consumption control method as described in claim 2, characterized in that, The steps involved in inputting standardized state data into a neural network model to generate high-level feature representation data include: Standardized state data is input into the evaluation network for processing, generating evaluation network output data. The evaluation network output data is input into a noise-adding layer for processing, generating noisy feature data as high-level feature representation data.

4. The server cluster adaptive energy consumption control method as described in claim 1, characterized in that, The steps of converting the control action data into control commands through the policy execution module and applying them to the server cluster include: The control action data is parsed to generate frequency adjustment instructions, which are used to adjust the server CPU operating frequency; Parse control action data to generate core management instructions, which are used to enable or disable CPU cores; Parse control action data to generate migration instructions, which are used to migrate virtual machines between physical servers; The control action data is parsed to generate power control commands, which are used to turn the server equipment on or off.

5. The server cluster adaptive energy consumption control method as described in claim 1, characterized in that, The steps for generating instant reward data through the reward calculation module based on the original state data, control action data, and new server cluster state data include: Extract the first total power consumption data from the original state data; Extract the second total power consumption data from the new server cluster status data; Calculate the difference between the second total power consumption data and the first total power consumption data to generate power consumption change data; Extract task response time data from the new server cluster status data; The task response time data is compared with the service level agreement threshold to generate service evaluation data; By combining the power consumption change data and the service evaluation data, real-time reward data is generated.

6. The server cluster adaptive energy consumption control method as described in claim 5, characterized in that, The steps for comparing the task response time data with service level agreement thresholds to generate service evaluation data include: Set the upper and lower thresholds for task response time; When the task response time data is less than the lower limit of the threshold, positive service evaluation data is generated; When the task response time data exceeds the upper limit of the threshold, negative service evaluation data is generated; Neutral service evaluation data is generated when the task response time data is between the upper and lower limits of the threshold.

7. The server cluster adaptive energy consumption control method as described in claim 1, characterized in that, The steps of updating the deep reinforcement learning agent module using the original state data, control action data, immediate reward data, and new server cluster state data through the model training module include: Combine the original state data, control action data, instant reward data, and new server cluster state data to form an experience data tuple; Store the experience data tuple into the experience replay buffer; Multiple sets of empirical data tuples are extracted from the buffer to form batch training data; The model gradient data is calculated using the batch training data. The deep reinforcement learning agent module is updated using the model gradient data.

8. The server cluster adaptive energy consumption control method as described in claim 7, characterized in that, The steps for calculating model gradient data using the batch training data include: New server cluster status data is processed through the target network to generate value estimation data. By combining the instant reward data and the value estimation data, target value data is generated; By evaluating the network's processing of raw state data and control action data, it generates data of practical value. By comparing the target value data with the actual value data, value difference data is generated; Calculate model gradient data based on value difference data.

9. The server cluster adaptive energy consumption control method as described in claim 1, characterized in that, The raw status data includes CPU utilization data, memory utilization data, real-time power consumption data, temperature data, and task queue length data of each server.

10. A server cluster adaptive energy consumption control system, characterized in that, The server cluster adaptive energy consumption control system includes: a memory, a processor, and a server cluster adaptive energy consumption control program stored in the memory and executable on the processor, wherein the server cluster adaptive energy consumption control program is configured to implement the steps of the server cluster adaptive energy consumption control method as described in any one of claims 1 to 9.

Images