Regulation method and apparatus for air conditioners in data center room, and computer device and storage medium

Abstract

A regulation method and apparatus for air conditioners in a data center room, and a computer device and a storage medium. The regulation method for air conditioners in a data center room comprises: acquiring data in an experience replay buffer; in response to there being no historical regulation data of air conditioners in a first data center room among data in the experience replay buffer, acquiring expert experience data of the first data center room, and on the basis of the expert experience data of the first data center room, using a reinforcement learning algorithm to obtain a first model; and on the basis of the first model, regulating the air conditioners in the first data center room.

Description

Control methods and devices for computer room air conditioning, computer equipment and storage media

[0001] This application claims priority to Chinese patent application No. 202410924435.9, filed on July 10, 2024, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of control technology for computer room air conditioning, and in particular to a control method and apparatus for computer room air conditioning, computer equipment, and readable storage medium. Background Technology

[0003] With the development of the times, the advancement of technology, and the rise of the Artificial Intelligence (AI) wave, hardware devices, as a crucial resource at the bottom physical layer, play a particularly important role. Data center facilities are the essential foundation for ensuring the normal operation of various bottom-level hardware. For example, in the era of big data, data center server rooms play a vital role, but the high energy consumption of data center server rooms has always been a problem to be solved. Furthermore, the data center air conditioning system (or server room air conditioning system) is one of the main energy-consuming devices. While maintaining the operation and stable functioning of servers and equipment, the server room air conditioning system requires a large amount of energy to maintain a suitable temperature and humidity inside the server room to ensure the normal operation of the equipment within the server room. Summary of the Invention

[0004] In a first aspect, this disclosure provides a method for controlling a computer room air conditioner, the method comprising: acquiring data in an experience playback buffer; in response to the absence of historical control data for a first computer room air conditioner in the experience playback buffer, acquiring expert experience data for the first computer room, and obtaining a first model based on the expert experience data for the first computer room using a reinforcement learning algorithm; and controlling the air conditioner in the first computer room according to the first model.

[0005] Optionally, the expert experience data for the first computer room includes expert experience action data and expert experience environmental data. The expert experience action data for the first computer room includes action data for each adjustment made by the expert to the air conditioning system during multiple adjustments. This action data includes temperature setting data and power on / off action data. The expert experience environmental data for the first computer room includes environmental data corresponding to each adjustment action made by the expert to the air conditioning system during multiple adjustments. This environmental data includes environmental data for the first computer room within a first preset time period before each adjustment action and environmental data for the first computer room within a second preset time period after each adjustment action. The environmental data for the first computer room includes the temperature within the first computer room, the return air temperature of the air conditioning system, the set temperature of the air conditioning system, the on / off status of the air conditioning system, the real-time power of the air conditioning system, and the real-time power of the equipment within the first computer room.

[0006] Optionally, based on the expert experience data of the first computer room, a first model is obtained using a reinforcement learning algorithm, including: defining a state space, an action space, and a reward space; and obtaining the first model based on the definitions of the reward space, the action space, and the reward space. The state space includes the expert experience environment data of the first computer room. The action space includes the expert experience action data of the first computer room. The reward space includes a reward function. The reward function includes a penalty term and a reward term. The penalty term includes: the temperature in the first computer room exceeds the highest value of a preset temperature range in the expert experience environment data of the first computer room. The reward term includes: the temperature in the first computer room does not rise significantly and the real-time power of the air conditioner and / or the real-time power of the equipment in the first computer room decreases, or the temperature in the first computer room does not rise and the real-time power of the air conditioner and / or the real-time power of the equipment in the first computer room decreases. The first model is a mapping relationship between the expert experience action data of the first computer room, the expert experience environment data of the first computer room, and the reward function.

[0007] Optionally, the method for controlling the air conditioning in the computer room further includes: acquiring in real time the action data of controlling the air conditioning in the first computer room according to the first model, and acquiring the environmental data of the first computer room within a third preset time period before controlling the air conditioning in the first computer room according to the first model and the environmental data of the first computer room within a fourth preset time period after controlling the air conditioning in the first computer room according to the first model; storing the acquired action data of controlling the air conditioning in the first computer room according to the first model, the environmental data of the first computer room within a third preset time period before controlling the air conditioning in the first computer room according to the first model, and the environmental data of the first computer room within a fourth preset time period after controlling the air conditioning in the first computer room according to the first model into an experience playback buffer; and updating the first model based on the acquired action data of controlling the air conditioning in the first computer room according to the first model, the environmental data of the first computer room within a third preset time period before controlling the air conditioning in the first computer room according to the first model, and the environmental data of the first computer room within a fourth preset time period after controlling the air conditioning in the first computer room according to the first model.

[0008] Optionally, the historical control data of the air conditioner in the first computer room includes: real-time action data of controlling the air conditioner in the first computer room according to the first model, environmental data of the first computer room within a third preset time before controlling the air conditioner in the first computer room according to the first model, and environmental data of the first computer room within a fourth preset time after controlling the air conditioner in the first computer room according to the first model.

[0009] Optionally, the method for controlling the air conditioning in the computer room further includes: responding to the existence of historical control data of the first computer room air conditioning in the data in the experience playback buffer, obtaining a second model based on the historical control data of the first computer room air conditioning using a reinforcement learning algorithm; and controlling the air conditioning in the first computer room according to the second model.

[0010] Optionally, controlling the air conditioner in the first computer room according to the first model or the second model includes: setting an upper limit, a lower limit, a threshold for the number of start-up actions, and a threshold for the number of shutdown actions for the temperature setting data; and controlling the air conditioner in the first computer room using the first model or the second model based on the upper limit, lower limit, threshold for the number of start-up actions, and threshold for the number of shutdown actions for the temperature setting data. In a second aspect, this disclosure provides a control device for a computer room air conditioner, comprising a data acquisition module, a model acquisition module, and a control module. The data acquisition module is configured to acquire data from an experience playback buffer. The model acquisition module is configured to: in response to the absence of historical control data for the first computer room air conditioner in the experience playback buffer, acquire expert experience data for the first computer room, and obtain a first model based on the expert experience data using a reinforcement learning algorithm. The control module is configured to: control the air conditioner in the first computer room according to the first model.

[0011] Thirdly, this disclosure provides a computer device including a memory and a processor. The memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the aforementioned method for controlling the computer room air conditioning.

[0012] Fourthly, this disclosure provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the processor performs the aforementioned control method for the computer room air conditioning.

[0013] Fifthly, this disclosure provides a computer program product including computer instructions. When executed by a processor, the computer instructions enable the processor to perform the aforementioned control method for the computer room air conditioning.

[0014] Sixthly, this disclosure provides a computer program comprising computer instructions. When executed by a processor, the computer instructions enable the processor to perform the aforementioned control method for the computer room air conditioning. Attached Figure Description

[0015] Figure 1 is a flowchart of a control method for a computer room air conditioner according to an embodiment of this disclosure.

[0016] Figure 2 is a flowchart of another method for controlling a computer room air conditioner in an embodiment of this disclosure.

[0017] Figure 3 is a flowchart of another method for controlling a computer room air conditioner in an embodiment of this disclosure.

[0018] Figure 4 is a flowchart of another method for controlling a computer room air conditioner in an embodiment of this disclosure.

[0019] Figure 5 is a flowchart of another control method for a computer room air conditioner in this embodiment of the present disclosure.

[0020] Figure 6 is a schematic diagram of an example architecture of a control algorithm for a computer room air conditioner in an embodiment of this disclosure.

[0021] Figure 7 is an example flowchart of a control algorithm for a computer room air conditioner in an embodiment of this disclosure.

[0022] Figure 8 is a structural diagram of a control device for a computer room air conditioner according to an embodiment of this disclosure.

[0023] Figure 9 is a structural diagram of another control device for a computer room air conditioner in an embodiment of this disclosure.

[0024] Figure 10 is a structural diagram of a control device for a computer room air conditioner according to another embodiment of this disclosure.

[0025] Figure 11 is a structural diagram of a computer device according to an embodiment of this disclosure. Detailed Implementation

[0026] To enable those skilled in the art to better understand the technical solutions of this disclosure, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings.

[0027] It is understood that the specific embodiments and accompanying drawings described herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0028] It is understood that, without conflict, the various embodiments and features in the embodiments of this disclosure can be combined with each other.

[0029] It is understood that, for ease of description, only the parts relevant to this disclosure are shown in the accompanying drawings, while parts unrelated to this disclosure are not shown in the drawings.

[0030] It is understood that each unit or module involved in the embodiments of this disclosure may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0031] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this disclosure may occur in a different order than that marked in the accompanying drawings.

[0032] It is understood that the flowcharts and block diagrams of this disclosure illustrate the architecture, functions, and operations of systems, apparatuses, devices, and methods implemented according to various embodiments of this disclosure. Each block in a flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified function. Moreover, each block or combination of blocks in the block diagrams and flowcharts may be implemented using a hardware-based system to implement the specified function, or using a combination of hardware and computer instructions.

[0033] It is understood that the units and modules involved in the embodiments of this disclosure can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0034] Currently, the control methods for computer room air conditioning systems mostly rely on manual experience or preset static control rules, which easily leads to energy waste, low efficiency, lack of adaptability and flexibility, and cannot be applied to different computer room operating environments and load requirements.

[0035] Therefore, some embodiments of this disclosure provide a method for controlling a computer room air conditioner. As shown in Figure 1, the method for controlling a computer room air conditioner includes steps 101 to 103.

[0036] In step 101, the data in the experience playback buffer is obtained.

[0037] Understandably, the data in the experience replay buffer can include control data for the air conditioning in various computer rooms. Experience replay is a technique in reinforcement learning designed to improve the efficiency and stability of learning. Experiences gained through real-time interaction with the environment (states, actions, rewards, etc.) are typically used immediately to update the model and may not necessarily be stored. Experience replay stores these experiences in a data structure called an "experience replay buffer." During subsequent training, random samples can be taken from the experience replay buffer for model updates.

[0038] In step 102, in response to the absence of historical control data for the air conditioning in the first computer room in the data in the experience playback buffer, the expert experience data of the first computer room is obtained, and the first model is obtained by using a reinforcement learning algorithm based on the expert experience data of the first computer room.

[0039] Understandably, the first computer room can be any computer room equipped with an air conditioner to be controlled. The first computer room may contain one or multiple air conditioners. Some embodiments of this disclosure provide a method for controlling the air conditioner in a computer room that can control each air conditioner in the first computer room.

[0040] In some embodiments, the expert experience data for the first computer room includes expert experience action data and expert experience environmental data for the first computer room. The expert experience action data for the first computer room includes action data for each adjustment made by the expert to the air conditioning system in the first computer room during multiple adjustments. This action data includes temperature setting data and power on / off action data. The expert experience environmental data for the first computer room includes environmental data corresponding to each adjustment action made by the expert to the air conditioning system in the first computer room during multiple adjustments. This environmental data corresponding to each adjustment action made by the expert to the air conditioning system in the first computer room includes environmental data for the first computer room within a first preset time period before each adjustment action and environmental data for the first computer room within a second preset time period after each adjustment action. The environmental data for the first computer room includes the temperature in the first computer room, the return air temperature of the air conditioning system, the set temperature of the air conditioning system, the on / off status of the air conditioning system, the real-time power of the air conditioning system, and the real-time power of the equipment in the first computer room.

[0041] Understandably, the first preset time can be set to a relatively short period, such as 1 minute or 30 seconds, to more intuitively reflect the environmental data of the first computer room before each adjustment action during multiple adjustments made by experts to the air conditioning. The real-time power of the equipment in the first computer room can refer to the real-time power of the Internet Technology (IT) equipment in the first computer room.

[0042] Understandably, environmental data such as temperature in the first computer room will only change after a certain period of time following the adjustment of the air conditioner. Therefore, the second preset time can be set to a longer duration, such as 10 or 20 minutes. This allows the environmental data from the first computer room within the second preset timeframe after each adjustment made by the expert during multiple adjustments to the air conditioner to more accurately reflect the environmental data after each adjustment. The length of the second preset time can be determined based on the actual situation. For example, the length of the second preset time can be set according to the actual effect of the air conditioner in the first computer room. For instance, if the air conditioner in the first computer room can quickly change the temperature (e.g., it can change the temperature within 10 minutes), the second preset time can be set to 10 minutes.

[0043] Understandably, comparing environmental data before and after a regulatory action allows for the assessment of the impact of the current action on the environmental data. By comparing multiple sets of data, the optimal regulatory strategy (e.g., effective and energy-saving) can be obtained for a given set of environmental data. With a sufficient number of data samples, the optimal regulatory strategy for any set of environmental data can be obtained, enabling the algorithm to output the optimal regulatory strategy for any given set of environmental data.

[0044] For example, the environmental data of the first computer room can be periodically acquired by the environmental monitoring platform equipment.

[0045] In some embodiments, as shown in FIG2, the implementation method of obtaining the first model in step 102 based on the expert experience data of the first computer room and using a reinforcement learning algorithm includes steps 201 to 202.

[0046] In step 201, the state space, action space, and reward space are defined.

[0047] The state space includes expert experience environment data for the first computer room. The action space includes expert experience action data for the first computer room. The reward space includes reward functions. The reward functions include penalty terms and reward terms. The penalty term includes: the temperature in the first computer room exceeds the highest value of the preset temperature range, as shown in the expert experience environment data. The reward term includes: the temperature in the first computer room does not rise significantly and the real-time power of the air conditioner and / or the real-time power of the equipment in the first computer room decreases, or the temperature in the first computer room does not rise and the real-time power of the air conditioner and / or the real-time power of the equipment in the first computer room decreases.

[0048] Understandably, the temperature inside the first computer room can be obtained by temperature sensors inside the first computer room.

[0049] Understandably, "no significant temperature rise in the first computer room" means that within a preset time period, the temperature rise in the first computer room is less than or equal to a threshold value. This threshold value can be 1°C, 2°C, 3°C, or 5°C. The value of this threshold value can be set according to the temperature sensitivity of the first computer room. For example, if the first computer room has extremely high temperature requirements, the threshold value could be 1°C. In other words, if the temperature rise in the first computer room is less than or equal to 1°C within a preset time period, then the temperature in the first computer room is considered to have no significant rise.

[0050] Understandably, the relationship between the reward function, the penalty term, and the reward term can be shown in equation (1). f(x,y)=a×x+b×y (1)

[0051] In equation (1), f(x,y) is the reward function, x is the penalty term, a is the weight coefficient of the penalty term, y is the reward term, and b is the weight coefficient of the reward term. The values ​​of a and b can be set according to the actual application scenario.

[0052] In step 202, the first model is obtained based on the definitions of the state space, action space, and reward space.

[0053] The first model is the mapping relationship between the expert experience action data of the first computer room, the expert experience environment data of the first computer room, and the reward function.

[0054] In essence, reinforcement learning involves an agent learning through trial and error, using rewards gained from interacting with the environment to guide its behavior, with the goal of maximizing the agent's reward. Reinforcement learning differs from supervised learning in connectionist learning, primarily in the reinforcement signal. In reinforcement learning, the reinforcement signal provided by the environment evaluates the quality of the action (usually a scalar signal), rather than instructing the reinforcement learning system (RLS) on how to produce the correct action. Because the external environment provides limited information, RLS must learn through its own experience. In this way, RLS acquires knowledge in the action-evaluation environment and improves its action plans to adapt to the environment.

[0055] For example, the reinforcement learning algorithm in step 102 is the Deep Deterministic Policy Gradient (DDPG) algorithm. The DDPG algorithm relies on an "actor-critic" architecture. The actor is used to adjust the parameters of the policy function, that is, to determine the best action in a given state.

[0056] For example, as shown in Figure 3, DDPG's approach is based on an Actor-Critic network (corresponding to the Q network and its target network (target_Q), and the policy network and its target network (target_P) in Figure 3). The algorithm update primarily involves updating the parameters of these four networks. The policy network updates the θ parameter by maximizing the cumulative expected reward, and the Q network updates the w parameter by minimizing the error between the evaluated value and the target value.

[0057] As shown in Figure 3, the Q-network update process optimizes the w parameter, for example: target_P calculates the action a′ under state s′ (i.e., ), and use target_Q to calculate the Q_target values ​​of s′ and a′ (i.e., Next, the policy network is used to calculate the evaluation Q-values ​​of s and a (i.e., Qw(s,a)). Finally, gradient descent is used to calculate the MSE of the two Q-values ​​(i.e., ...). ), where w represents the weight, γ represents the discount factor, r represents the current reward, and μ represents the learning rate.

[0058] As shown in Figure 3, the policy network update process is to optimize the θ parameter. For example, the policy network calculates the action a under state s (i.e., a = μθ(S)), uses the Q network to calculate the evaluation Q value of s and a (i.e., Qw(s,a)), and finally uses the gradient descent algorithm to optimize Q to maximize the cumulative expected reward (i.e., Loss = -Qw(s,a)).

[0059] In step 103, the air conditioning in the first computer room is adjusted according to the first model.

[0060] Understandably, after obtaining the first model, environmental data of the first computer room can be collected in real time, and the collected environmental data of the first computer room can be input into the first model. The first model can obtain the optimal control strategy corresponding to the collected environmental data of the first computer room, and output the control strategy for the air conditioner in the first computer room. Thus, the air conditioner in the first computer room can be controlled according to the control strategy for the air conditioner in the first computer room output by the first model.

[0061] In some embodiments, as shown in FIG4, the control method for the computer room air conditioner further includes steps 401 to 403.

[0062] In step 401, the action data of adjusting the air conditioner in the first computer room according to the first model is acquired in real time, and the environmental data of the first computer room within a third preset time before adjusting the air conditioner in the first computer room according to the first model and the environmental data of the first computer room within a fourth preset time after adjusting the air conditioner in the first computer room according to the first model are acquired.

[0063] In step 402, the real-time acquired action data of adjusting the air conditioner in the first computer room according to the first model, the environmental data of the first computer room within a third preset time before adjusting the air conditioner in the first computer room according to the first model, and the environmental data of the first computer room within a fourth preset time after adjusting the air conditioner in the first computer room according to the first model are stored in the experience playback buffer.

[0064] In step 403, the first model is updated based on the real-time acquired action data of the air conditioner in the first computer room being controlled according to the first model, the environmental data of the first computer room within a third preset time before the control of the air conditioner in the first computer room is started according to the first model, and the environmental data of the first computer room within a fourth preset time after the control of the air conditioner in the first computer room is started according to the first model.

[0065] In some embodiments, the historical control data of the air conditioner in the first computer room includes real-time action data of controlling the air conditioner in the first computer room according to the first model, environmental data of the first computer room within a third preset time before controlling the air conditioner in the first computer room according to the first model, and environmental data of the first computer room within a fourth preset time after controlling the air conditioner in the first computer room according to the first model.

[0066] Understandably, the first model is based on expert experience data, which is limited in amount. Storing the action data of each adjustment and the environmental data before and after the adjustment into the experience playback buffer as historical data can enrich the amount of data in the experience playback buffer, thereby obtaining a more accurate model.

[0067] In some embodiments, as shown in FIG5, the control method for the computer room air conditioner further includes steps 501 to 502.

[0068] In step 501, in response to the presence of historical control data of the first computer room air conditioner in the data in the experience playback buffer, a second model is obtained by using a reinforcement learning algorithm based on the historical control data of the first computer room air conditioner.

[0069] Understandably, if the data in the experience replay buffer contains historical data on the control of the air conditioner in the first computer room, the model can be trained directly based on the historical data on the control of the air conditioner in the first computer room and the reinforcement learning algorithm to obtain the second model.

[0070] Understandably, the method of obtaining the second model based on the historical control data of the air conditioner in the first computer room and using reinforcement learning algorithm can refer to the implementation method of obtaining the first model according to steps 201 and 202, which will not be repeated here.

[0071] In some embodiments, the second model may also be obtained by updating the first model based on real-time acquired action data of adjusting the air conditioner in the first computer room according to the first model, environmental data of the first computer room within a third preset time before adjusting the air conditioner in the first computer room according to the first model, and environmental data of the first computer room within a fourth preset time after adjusting the air conditioner in the first computer room according to the first model.

[0072] For example, after obtaining the first model, the first model can be stored in the experience replay buffer, so that the first model can be directly updated according to the real-time acquired action data of adjusting the air conditioner in the first computer room according to the first model, the environmental data of the first computer room within a third preset time before the first model starts adjusting the air conditioner in the first computer room, and the environmental data of the first computer room within a fourth preset time after the first model starts adjusting the air conditioner in the first computer room, so as to obtain the second model.

[0073] In step 502, the air conditioning in the first computer room is adjusted according to the second model.

[0074] The implementation method of step 502 can be referred to the implementation method of step 103, and will not be repeated here.

[0075] In some embodiments, regulating the air conditioner in the first computer room according to the first model or the second model includes: setting an upper limit of temperature setting data, a lower limit of temperature setting data, a threshold for the number of times the unit is turned on, and a threshold for the number of times the unit is turned off; and regulating the air conditioner in the first computer room using the first model or the second model according to the upper limit of temperature setting data, the lower limit of temperature setting data, the threshold for the number of times the unit is turned on, and the threshold for the number of times the unit is turned off.

[0076] For example, when adjusting the air conditioner in the first computer room according to the first model or the second model, the upper limit of the temperature setting (i.e., the temperature that can be set according to the first model or the second model) is 26°C, the lower limit of the temperature setting is 20°C, and the maximum number of shutdowns is 5. In this way, when adjusting the air conditioner in the first computer room according to the first model or the second model, it can be ensured that the environment in the first computer room will not change excessively, or that even if the first model or the second model outputs an abnormal control strategy for unknown reasons (e.g., improper parameter adjustment or sudden environmental changes), such an abnormal control strategy cannot be executed. This prevents damage to the machines in the first computer room due to excessive environmental changes and ensures the normal operation of the machines in the first computer room.

[0077] The control method for computer room air conditioning provided in some embodiments of this disclosure achieves intelligent learning and adaptive adjustment of the air-cooled air conditioning system through intelligent learning and optimized control strategies. This enables the air conditioning system to autonomously adjust its control strategy according to different operating states, improving control flexibility and accuracy, minimizing energy consumption, and increasing energy efficiency. Furthermore, the control method for computer room air conditioning provided in some embodiments of this disclosure optimizes the training process of the DDPG algorithm by using different training methods on expert experience data and historical control data of the first computer room air conditioner. This results in more reasonable data sampling and processing methods, higher algorithm computational efficiency, and more efficient completion of energy-saving control tasks in practical applications. In addition, the control method for computer room air conditioning provided in some embodiments of this disclosure introduces a safety protection mechanism by setting upper and lower limits for temperature setting data, thresholds for the number of start-up actions, and thresholds for the number of shutdown actions. This allows for safe control even in abnormal situations, ensuring normal operation of the air conditioner and preventing anomalies caused by improper parameter control or sudden environmental changes.

[0078] The following example illustrates the control method for computer room air conditioning provided in some embodiments of this disclosure.

[0079] This example provides an energy-saving algorithm for data center air-cooled precision air conditioning based on DDPG. It collects data from the air conditioning, temperature sensor, and electricity meter periodically reported by the FSU (Field Supervision Unit) equipment at the bottom-end environmental monitoring platform, inputs the data into the algorithm model, generates an air conditioning control strategy, automatically maps the device ID (identity) and measurement point ID of the environmental monitoring platform, and issues control commands through an open interface. This achieves decoupling of the bottom-end, environmental monitoring platform, and algorithm, and allows for plug-in deployment and operation.

[0080] In this example, the energy-saving algorithm automatically detects the current status of the data center environment (temperature of the temperature sensor, return air temperature of the air conditioner, set temperature of the air conditioner, working status of the air conditioner, power of the air conditioner, power of IT equipment), inputs it into the strategy model, generates action commands, maps the action commands to the air conditioner control range, and sends them to the data center energy-saving interface of the environmental monitoring platform to control the air conditioner.

[0081] The entire algorithm process includes the following steps a) to f).

[0082] a) Data docking configuration: After starting the algorithm service, it is necessary to configure the data center equipment information issued by the energy-saving algorithm and request the algorithm measurement point configuration interface. The body carries device_id (device ID), mete_id (measurement point ID), mete_label ({1: air conditioner, 2: temperature sensor, 3: air conditioner meter, 4: IT meter), mete_code (monitoring quantity code).

[0083] b) Periodic Data Storage: Activate the algorithm's data storage service, selecting between offline and online training based on the actual situation. Offline Training: Manually controlled expert experience data serves as an offline experience replay buffer for network training. Online Training: The algorithm directly outputs its strategy, interacting with the underlying device in real time; the interaction data is placed in the experience replay buffer.

[0084] c) Reinforcement Learning Model Definition: The State space is defined to include the temperature sensed in the computer room, the return air temperature of the air conditioner, the set temperature, the operating status, the real-time power of the air conditioner, and the real-time power of the IT equipment. The Action space is defined to include the temperature setting and on / off control of the air-cooled precision air conditioner. The Reward space is defined to include a penalty term for the number of temperature senseds exceeding the algorithm threshold (-10 × number of temperature senseds exceeding the high temperature × coefficient) and a reward term for no significant temperature increase or no temperature increase but a decrease in power (-1 × negative power change × coefficient). The two are weighted and combined to form the reward function (weight × penalty term + weight × reward term).

[0085] d) Model parameter saving: After training, the training data will be stored in the model / replay_buffer.csv file according to the model input specification. The model parameters (actor_model.pth, critic_model.pth) of the AC (Actor-Critic) network and the model loss value (loss.csv) will be stored in the model directory.

[0086] e) Algorithm Execution Interaction: After starting the algorithm, if there is no parameter data, it will first train the parameter data. If parameter data exists, it will directly read the data in the model directory to load the model, and then, according to the interaction interval, directly obtain the telemetry and tele-signaling data of the devices connected to the environmental monitoring unit, perform model inference, and output the air conditioning control operation.

[0087] f) Safety Restrictions: Limit the upper and lower limits of the air-cooled air conditioner temperature adjustment output by the algorithm to between 20-26 degrees Celsius. Parameters are configurable; restrict the air-cooled air conditioner control on / off operations output by the algorithm, count the number of shutdowns, specify a total shutdown value, and prohibit further actions once the threshold is reached.

[0088] As shown in Figure 3, DDPG's approach is based on the Actor-Critic network (corresponding to the Q network and its target network target_Q, and the policy network and its target network target_P in Figure 3). The algorithm update primarily involves updating the parameters of these four networks. The policy network maximizes the cumulative... target_Q calculates the Q_target values ​​of s′ and a′ (i.e., Next, the policy network is used to calculate the evaluation Q-values ​​of s and a (i.e., Qw(s,a)). Finally, gradient descent is used to calculate the MSE of the two Q-values ​​(i.e., ...). As shown in Figure 3, the policy network update process optimizes the θ parameter. For example, the policy network calculates the action a in state s (i.e., a = μθ(S)), uses the Q network to calculate the evaluation Q value of s and a (i.e., Qw(s,a)), and finally uses the gradient descent algorithm to optimize -Q to maximize the cumulative expected reward (i.e., Loss = -Qw(s,a)).

[0089] As shown in Figure 6, in the actual application architecture of this example, the environmental monitoring device communicates with the connected hardware devices via RS-485. The environmental monitoring device, the environmental network management service, and the air-cooled precision air conditioning energy-saving algorithm are connected via HTTP (Hypertext Transfer Protocol). The algorithm service obtains real-time data from the device and issues remote adjustment and control commands through the network management platform, which then issues instructions to the environmental monitoring device.

[0090] Understandably, a computer room can contain multiple air conditioners and multiple corresponding sensors and meters. For example, in Figure 6, the first and second air-cooled precision air conditioners are both air conditioners in the first computer room. The first and second temperature sensors are both temperature sensors in the first computer room. The first and second smart meters are both smart meters in the first computer room.

[0091] As shown in Figure 7, the overall operation process of this example in actual application includes: a) Deploying data storage services and configuring parameters. Continuously acquiring data from the computer room air conditioning, temperature sensors, and electricity meters; b) Deploying the air-cooled precision air conditioning energy-saving algorithm service, configuring database parameters, algorithm training parameters, and the interface address and port with the environmental monitoring platform; c) Determining whether historical data for air conditioning control exists. If data exists, offline training is performed; otherwise, online training is performed; d) Calculating whether the network loss and reward value have converged. If not, the parameters will be adjusted and training will continue until the accuracy of the loss and the range of convergence are reached; e) Acquiring the latest data reported by the devices in real time, and determining whether a policy needs to be issued under the premise of safety and control.

[0092] The key points of this example are: a) Based on the DDPG algorithm, the Deep Deterministic Policy Gradient (DDPG) algorithm is introduced into the energy-saving control of the air-cooled precision air conditioning system in the computer room to achieve intelligent optimization and dynamic adjustment of the air conditioning system, thereby achieving the energy-saving goal; b) Deep neural network construction: The Actor-Critic network is constructed using a deep neural network to realize the mapping from state to action and the evaluation of action value function, thereby improving the system's learning and decision-making capabilities; c) Environmental interaction and training: Through the interaction and feedback between the system and the environment, the parameters of the Actor-Critic network are continuously optimized, enabling the algorithm to learn intelligently to adapt to different operating environments and load requirements.

[0093] The technical problems this example solves and the beneficial effects are as follows: a) Traditional air-cooled air conditioning systems for computer rooms often use fixed control strategies, which cannot dynamically adjust parameters according to the actual operating environment and load demand, resulting in uneconomical energy consumption and energy waste. This example, however, uses intelligent energy-saving control methods, through intelligent learning and optimization of control strategies, to achieve adaptive adjustment of the air-cooled air conditioning system, minimizing energy consumption and improving energy efficiency. It can optimize the energy consumption of the air conditioning system, reduce its operating energy consumption, and thus achieve the goal of energy conservation and emission reduction. b) Due to changes in the operating environment and load demand of the computer room, the air conditioning system needs to adjust parameters such as temperature setpoint and fan speed in real time. However, traditional static control methods lack adaptability and cannot meet dynamic operating needs. This example integrates deep learning and reinforcement learning technologies to achieve intelligent learning and adaptive adjustment of the air-cooled air conditioning system, enabling the system to autonomously adjust its control strategy according to different operating states, improving the flexibility and accuracy of control. It can achieve precise control of the air-cooled precision air conditioner in the computer room by learning changes in the environment in real time. The system dynamically adjusts the control strategy to maintain the ambient temperature within a normal range, improving the utilization of air conditioning resources. c) While introducing intelligent control algorithms, ensuring the stability and reliability of the algorithms is also crucial. The training and convergence process of the DDPG algorithm may face very complex environments, and the air conditioning system, as a critical device, requires stable and safe control. This example incorporates a safety protection mechanism within the algorithm implementation to ensure safe control and normal operation of the computer room air conditioning system even under abnormal conditions. It ensures the algorithm remains stable and safe under various operating conditions, avoiding anomalies caused by improper parameter control or sudden environmental changes, and preventing adverse effects and malfunctions. d) The DDPG algorithm requires a large amount of computational resources and data during training and optimization. Therefore, how to effectively utilize resources and improve the training efficiency of the algorithm is also a consideration. This example selects online or offline training based on data conditions, optimizes the training process of the DDPG algorithm, and adopts reasonable data sampling and processing methods to improve the computational efficiency of the algorithm, enabling it to more efficiently complete energy-saving control tasks in practical applications.

[0094] Some embodiments of this disclosure provide a control device for a computer room air conditioner. As shown in FIG8, the control device 800 for a computer room air conditioner includes a data acquisition module 801, a model acquisition module 802, and a control module 803.

[0095] The data acquisition module 801 is configured to acquire data from the experience playback buffer.

[0096] The model acquisition module 802 is configured to: in response to the absence of historical control data of the air conditioner in the first computer room in the data in the experience playback buffer, acquire the expert experience data of the first computer room, and obtain the first model based on the expert experience data of the first computer room using a reinforcement learning algorithm.

[0097] In some embodiments, the expert experience data for the first computer room includes: expert experience action data and expert experience environmental data for the first computer room. The expert experience action data for the first computer room includes action data for each adjustment made by the expert to the air conditioning system in the first computer room. Action data includes temperature setting data and power on / off action data. The expert experience environmental data for the first computer room includes environmental data corresponding to each adjustment action made by the expert to the air conditioning system in the first computer room. The environmental data corresponding to each adjustment action made by the expert to the air conditioning system in the first computer room includes environmental data for the first computer room within a first preset time period before each adjustment action and environmental data for the first computer room within a second preset time period after each adjustment action. The environmental data for the first computer room includes: the temperature sensed in the first computer room, the return air temperature of the air conditioning system, the set temperature of the air conditioning system, the on / off status of the air conditioning system, the real-time power of the air conditioning system, and the real-time power of the IT equipment in the first computer room.

[0098] In some embodiments, as shown in FIG9, the model acquisition module 802 includes a spatial definition unit 901 and a first model acquisition unit 902.

[0099] Space definition unit 901 is configured to define a state space, an action space, and a reward space. The state space includes expert experience environment data for the first computer room. The action space includes expert experience action data for the first computer room. The reward space includes a reward function. The reward function includes penalty terms and reward terms. Penalty terms include: the temperature in the first computer room exceeds the highest value of a preset temperature range, as shown in the expert experience environment data. Reward terms include: the temperature in the first computer room does not rise significantly and the real-time power of the air conditioner and / or the real-time power of the IT equipment in the first computer room decreases; or the temperature in the first computer room does not rise and the real-time power of the air conditioner and / or the real-time power of the IT equipment in the first computer room decreases.

[0100] The first model acquisition unit 902 is configured to obtain the first model based on the definitions of the state space, action space, and reward space. The first model is the mapping relationship between the expert experience action data of the first computer room, the expert experience environment data of the first computer room, and the reward function.

[0101] The control module 803 is configured to control the air conditioning in the first computer room according to the first model.

[0102] In some embodiments, as shown in FIG10, the control device 800 for the computer room air conditioner further includes an environmental data acquisition module 804, a data storage module 805, and a model update module 806.

[0103] The environmental data acquisition module 804 is configured to: acquire in real time the action data of the air conditioner in the first computer room that is controlled according to the first model, and acquire the environmental data of the first computer room within a third preset time before the air conditioner in the first computer room is controlled according to the first model, and the environmental data of the first computer room within a fourth preset time after the air conditioner in the first computer room is controlled according to the first model.

[0104] The data storage module 805 is configured to store the real-time action data of the air conditioner in the first computer room that is controlled according to the first model, the environmental data of the first computer room within a third preset time before the control of the air conditioner in the first computer room begins according to the first model, and the environmental data of the first computer room within a fourth preset time after the control of the air conditioner in the first computer room begins according to the first model into the experience playback buffer.

[0105] The model update module 806 is configured to update the first model based on the real-time acquired action data of the first model for controlling the air conditioner in the first computer room, the environmental data of the first computer room within a third preset time before the first model starts controlling the air conditioner in the first computer room, and the environmental data of the first computer room within a fourth preset time after the first model starts controlling the air conditioner in the first computer room.

[0106] In some embodiments, the historical control data of the air conditioner in the first computer room includes: real-time acquired action data of controlling the air conditioner in the first computer room according to the first model, environmental data of the first computer room within a third preset time before controlling the air conditioner in the first computer room according to the first model, and environmental data of the first computer room within a fourth preset time after controlling the air conditioner in the first computer room according to the first model.

[0107] In some embodiments, the model acquisition module 802 is further configured to: in response to the existence of historical control data of the first computer room air conditioner in the data in the experience playback buffer, obtain a second model using a reinforcement learning algorithm based on the historical control data of the first computer room air conditioner. The control module 803 is further configured to: control the air conditioner in the first computer room according to the second model.

[0108] In some embodiments, the control module 803 is further configured to: set an upper limit for temperature setting data, a lower limit for temperature setting data, a threshold for the number of times the machine is turned on, and a threshold for the number of times the machine is turned off; and control the air conditioner in the first computer room using a first model or a second model based on the upper limit for temperature setting data, the lower limit for temperature setting data, the threshold for the number of times the machine is turned on, and the threshold for the number of times the machine is turned off.

[0109] In some embodiments, the second model may also be obtained by updating the first model based on the real-time acquired action data of the air conditioner in the first computer room that is controlled according to the first model, the environmental data of the first computer room within a third preset time before the first model starts controlling the air conditioner in the first computer room, and within a fourth preset time after the first model starts controlling the air conditioner in the first computer room.

[0110] The scheme and beneficial effects of the control device 800 for the computer room air conditioner provided in some embodiments of this disclosure can be found in the relevant description of the control method for the computer room air conditioner provided in some embodiments of this disclosure, and will not be repeated here.

[0111] Some embodiments of this disclosure provide a computer device. As shown in FIG11, the computer device 1100 includes a memory 1101 and a processor 1102. The memory 1101 stores a computer program. When the processor 1102 runs the computer program stored in the memory 1101, the processor 1102 executes the above-described control method for the computer room air conditioning.

[0112] The solutions and beneficial effects of the computer equipment 1100 provided in some embodiments of this disclosure can be found in the relevant descriptions of the control methods for computer room air conditioning provided in some embodiments of this disclosure, which will not be repeated here.

[0113] Some embodiments of this disclosure provide a computer-readable storage medium (e.g., a non-transitory computer-readable storage medium) on which a computer program is stored. When the computer program is executed by a processor, the processor performs the above-described method for controlling the computer room air conditioning.

[0114] The computer-readable storage medium solutions and beneficial effects provided by some embodiments of this disclosure can be found in the relevant descriptions of the control methods for computer room air conditioning provided by some embodiments of this disclosure, which will not be repeated here.

[0115] The control scheme for the computer room air conditioning provided in this disclosure achieves intelligent learning and adaptive adjustment of the air-cooled air conditioning system through intelligent learning and optimized control strategies. This enables the air conditioning system to autonomously adjust its control strategy according to different operating states, improving control flexibility and accuracy, minimizing energy consumption, and increasing energy efficiency. This disclosure optimizes the training process of the DDPG algorithm by using different training methods on expert experience data and historical control data of the first computer room air conditioning system. The data sampling and processing methods are more reasonable, and the algorithm's computational efficiency is higher, enabling it to complete energy-saving control tasks more efficiently in practical applications. Furthermore, this disclosure introduces a safety protection mechanism by setting upper and lower limits for temperature setting data, thresholds for the number of start-up actions, and thresholds for the number of shutdown actions. This allows for safe control even in abnormal situations, ensuring normal operation of the air conditioning system and preventing anomalies caused by improper parameter control or sudden environmental changes.

[0116] It is understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of this disclosure, and this disclosure is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and substance of this disclosure, and these modifications and improvements are also considered to be within the scope of protection of this disclosure.

Claims

1. A method for controlling a computer room air conditioner, comprising: Retrieve data from the experience replay buffer; In response to the absence of historical control data for the air conditioning in the first computer room in the data in the experience replay buffer, expert experience data for the first computer room is obtained, and a first model is obtained using a reinforcement learning algorithm based on the expert experience data for the first computer room; and The air conditioning in the first computer room is adjusted according to the first model.

2. The control method for computer room air conditioning according to claim 1, wherein, The expert experience data in the first computer room includes expert experience action data and expert experience environment data of the first computer room. The expert experience action data for the first computer room includes action data for each adjustment made by the expert to the air conditioning in the first computer room; the action data includes temperature setting data and power on / off action data. The expert experience environmental data of the first computer room includes environmental data corresponding to each adjustment action performed by the expert on the air conditioning of the first computer room during multiple adjustments; the environmental data corresponding to each adjustment action performed by the expert on the air conditioning of the first computer room during multiple adjustments includes environmental data of the first computer room within a first preset time period before each adjustment action performed by the expert on the air conditioning of the first computer room during multiple adjustments, and environmental data of the first computer room within a second preset time period after each adjustment action performed by the expert on the air conditioning of the first computer room during multiple adjustments; the environmental data of the first computer room includes the temperature in the first computer room, the return air temperature of the air conditioning, the set temperature of the air conditioning, the on / off status of the air conditioning, the real-time power of the air conditioning, and the real-time power of the equipment in the first computer room.

3. The control method for computer room air conditioning according to claim 2, wherein, The step of obtaining the first model based on the expert experience data from the first computer room and using the reinforcement learning algorithm includes: Define a state space, an action space, and a reward space; the state space includes the expert experience environment data of the first computer room; the action space includes the expert experience action data of the first computer room; the reward space includes a reward function; the reward function includes a penalty term and a reward term; the penalty term includes: the temperature in the first computer room exceeds the highest value of a preset temperature range in the expert experience environment data of the first computer room; the reward term includes: the temperature in the first computer room does not rise significantly and the real-time power of the air conditioner and / or the real-time power of the equipment in the first computer room decreases, or the temperature in the first computer room does not rise and the real-time power of the air conditioner and / or the real-time power of the equipment in the first computer room decreases; and The first model is obtained based on the definitions of the state space, the action space, and the reward space; the first model includes the mapping relationship between the expert experience action data of the first computer room, the expert experience environment data of the first computer room, and the reward function.

4. The control method for a computer room air conditioner according to any one of claims 1 to 3, further comprising: Real-time acquisition of action data for adjusting the air conditioner in the first computer room according to the first model, and acquisition of environmental data of the first computer room within a third preset time before adjusting the air conditioner in the first computer room according to the first model and environmental data of the first computer room within a fourth preset time after adjusting the air conditioner in the first computer room according to the first model. The real-time acquired action data of adjusting the air conditioner in the first computer room according to the first model, the environmental data of the first computer room within the third preset time before adjusting the air conditioner in the first computer room according to the first model, and the environmental data of the first computer room within the fourth preset time after adjusting the air conditioner in the first computer room according to the first model are stored in the experience playback buffer. as well as The first model is updated based on the real-time acquired action data of the air conditioner in the first computer room that is controlled according to the first model, the environmental data of the first computer room within the third preset time before the control of the air conditioner in the first computer room begins according to the first model, and the environmental data of the first computer room within the fourth preset time after the control of the air conditioner in the first computer room begins according to the first model.

5. The control method for computer room air conditioning according to claim 4, wherein, The historical control data of the first computer room air conditioner includes the real-time acquired action data of controlling the air conditioner in the first computer room according to the first model, the environmental data of the first computer room within a third preset time before the control of the air conditioner in the first computer room begins according to the first model, and the environmental data of the first computer room within a fourth preset time after the control of the air conditioner in the first computer room begins according to the first model.

6. The method for controlling the air conditioning in a computer room according to any one of claims 1 to 5, further comprising: In response to the presence of historical control data of the first computer room air conditioner in the data in the experience playback buffer, a second model is obtained by using a reinforcement learning algorithm based on the historical control data of the first computer room air conditioner. as well as The air conditioning in the first computer room is controlled according to the second model.

7. The control method for computer room air conditioning according to claim 6, wherein, The step of controlling the air conditioning in the first computer room according to the first model or the step of controlling the air conditioning in the first computer room according to the second model includes: Set the upper limit of the temperature setting data, the lower limit of the temperature setting data, the threshold for the number of power-on actions, and the threshold for the number of power-off actions; and Based on the upper limit of the temperature setting data, the lower limit of the temperature setting data, the threshold for the number of times the machine is turned on, and the threshold for the number of times the machine is turned off, the air conditioner in the first computer room is controlled using the first model or the second model.

8. A control device for a computer room air conditioner, comprising: The data acquisition module is configured to acquire data from the experience replay buffer. The model acquisition module is configured to: in response to the absence of historical control data for the air conditioning in the first computer room in the data in the experience playback buffer; acquire expert experience data of the first computer room; and obtain a first model based on the expert experience data of the first computer room using a reinforcement learning algorithm. as well as The control module is configured to control the air conditioning in the first computer room according to the first model.

9. A computer device comprising a memory and a processor, wherein, The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the control method for the computer room air conditioner according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, the processor performs the control method for the computer room air conditioner according to any one of claims 1 to 7.

11. A computer program product comprising computer instructions, wherein when executed by a processor, the processor performs a control method for a computer room air conditioner according to any one of claims 1 to 7.

12. A computer program comprising computer instructions, wherein when executed by a processor, the processor performs a control method for a computer room air conditioner according to any one of claims 1 to 7.

Images