Multi-agent reward function construction method and system for data center cooling system

Abstract

The application discloses a kind of multi-agent reward function construction method and system of data center cooling system, it is related to artificial intelligence and reinforcement learning control field, including the sampling state pair extracted based on data center cooling system is determined preference label;Ego perspective state representation input state score neural network, obtain score result;Difference of score result is determined based on Bradley-Terry model, and the difference is mapped as state advantage prediction probability for quantifying the state of operation advantage and disadvantage, determine log-likelihood loss in combination with preference label, state score neural network is trained by log-likelihood loss;The ego perspective state representation of the agent to be measured is input into the state score neural network after training, and a set of state score networks is obtained;Determine training reward based on the set of state score networks, use training reward as the reward signal of agent strategy optimization, update the strategy parameters of each agent, to obtain corresponding control action set.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence and reinforcement learning control, and in particular to a method and system for constructing a multi-agent reward function for a data center cooling system. Background Technology

[0002] In complex industrial scenarios involving multiple coupled devices such as data center cooling systems (e.g., terminal air conditioners, chillers, cooling towers, and their water systems in computer rooms), multi-agent reinforcement learning (MARL) is an important means to achieve system-level energy efficiency optimization and collaborative control.

[0003] Currently, traditional methods directly use a single system-level overall reward for feedback. This makes it difficult to accurately attribute changes in the overall performance of each agent in the data center cooling system to the actions of specific devices. Consequently, the reward function designed in this way cannot accurately express the quality of each agent's actions during the cooling process, resulting in the inability of agents to form efficient collaborative strategies. Summary of the Invention

[0004] This invention provides a method and system for constructing a multi-agent reward function for a data center cooling system, to solve the aforementioned problems in the prior art, namely, how to design a reward function that can accurately express complex engineering preferences, thereby improving efficient collaboration among agents. This invention provides a method for constructing a multi-agent reward function for a data center cooling system, the method comprising: Historical trajectory data generated by the interaction between multiple intelligent agents of the data center cooling system and the simulation environment of the data center cooling system is obtained, and sampled state pairs used to characterize different operating states of each intelligent agent are extracted from the historical trajectory data; the intelligent agent is a device that realizes the cooling process of the data center cooling system. Preference comparisons are performed on sampled state pairs to determine preference labels for evaluating the quality of each agent's actions; for each agent, an ego-view state representation is constructed to characterize the local state information and global associated state information related to the agent's control decisions. Each agent's state scoring neural network is constructed based on a multilayer perceptron. Paired ego-perspective state representations are input into the state scoring neural network to score the current running state of each agent and obtain the corresponding scoring results. The difference between the scoring results is determined based on the Bradley-Terry model, and the difference is mapped to the state quality prediction probability used to quantify the running quality of the state. Based on the state quality prediction probability and preference label, the log-likelihood loss is determined. The state scoring neural network is trained by minimizing the log-likelihood loss to determine the trained state scoring neural network. The ego-perspective state representation of the agent under test is input into the trained state scoring neural network to obtain the state score of each agent. Based on the state score, the training reward of each agent is determined. The training reward is used as the reward signal for the corresponding agent's policy optimization to update the policy parameters of each agent and obtain the corresponding set of control actions.

[0005] Optionally, obtaining the preference label includes: sampling state pairs The input is fed into a Large Language Model (LLM), which uses a multi-factor criterion to compare preferences and determine the corresponding binary preference labels. ; The multi-factor criteria include whether the data center intake air temperature is within a safe range and the extent of any deviations, the total system power or the power of critical equipment, and the supply-demand imbalance between the cooling and chiller sides; among these factors... express Superior Otherwise, it means Superior .

[0006] Optionally, when training the state scoring neural network, log-likelihood loss is used to update the network parameters; wherein, obtaining the log-likelihood loss specifically includes: ; in, For log-likelihood loss, The binary preference labels output by the LLM model. This is to combine the Bradley-Terry model with the network score difference mapped by the Sigmoid function to predict the state quality probability.

[0007] Optionally, the intelligent agent includes an end intelligent agent for end heat exchange, a chiller intelligent agent for generating cold source, a water pump intelligent agent for driving cold source circulation, and a cooling tower intelligent agent for heat dissipation; the ego perspective state representation includes load-side demand, actual load-side quantity, refrigeration-side demand, actual refrigeration-side quantity, cooling-side demand, and cooling-side capacity.

[0008] Optionally, based on the real-time operating power of the representative terminal fans and chilled water pumps, the equivalent waste heat is determined, and the equivalent waste heat is superimposed with the acquired basic heat dissipation load on the air side of the computer room to obtain the demand on the chilled side.

[0009] Optionally, the actual effective cooling capacity of the refrigeration side, the real-time input power of the chiller unit, and the real-time operating power of the cooling water pump are superimposed to determine the cooling side demand.

[0010] Optionally, the cooling capacity can be obtained by outputting the heat dissipation capacity based on the cooling tower model under given conditions of inlet water temperature, outdoor wet-bulb temperature, cooling water flow rate and fan frequency conversion, combined with the calibration coefficient representing the compensation for on-site deviation.

[0011] Optionally, the heat transfer items of the obtained building envelope and outdoor environment, server heat dissipation items, and power distribution and UPS loss heat are summed to determine the load-side demand; the minimum value between the heat transfer capacity achievable on the air side and the cold transfer capacity achievable on the water side is taken as the actual load-side quantity.

[0012] Optionally, the air side can achieve the acquisition of heat transfer capacity and the water side can acquire cold transfer capacity, specifically including: The minimum value between the total heat load of the computer room and the maximum heat exchange capacity on the terminal air side is taken as the heat transfer capacity that can be realized on the air side, specifically including: ; Based on the obtained total cooling capacity on the chiller side, and the weighted power consumption of the terminal fans and chilled water pumps, the transferable cooling capacity on the water side is obtained in the following manner: ; in, Heat can be transferred to the air side. The total heat load of the computer room, This represents the maximum heat exchange capacity on the terminal air side. This refers to the total cooling capacity on the refrigeration side. , These are the power consumption of the terminal fan and the chilled water pump, respectively. This is the electro-thermal conversion factor.

[0013] This invention provides a multi-agent reward function construction system for a data center cooling system, implementing the aforementioned method for constructing a multi-agent reward function for a data center cooling system.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: The present invention provides a method for constructing a multi-agent reward function for a data center cooling system. This method constructs a state scoring network for each agent device that implements the cooling process of the data center cooling system, obtains the state scores of each agent, and generates a training reward corresponding to each agent based on the state scores, thereby reducing the instability and bias caused by traditional manual reward weight design. Then, by using the training reward as the reward signal for the corresponding agent's policy optimization, each agent can obtain feedback signals related to its actions, thereby achieving efficient collaboration among the various agent devices in the data center cooling system. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0016] Figure 1 A flowchart illustrating a method for constructing a multi-agent reward function for a data center cooling system, provided in an embodiment of the present invention; Figure 2 A technical roadmap for a method of constructing a multi-agent reward function for a data center cooling system provided in this embodiment of the invention; Figure 3 The closed-loop simulation and data recording timing diagram provided for embodiments of the present invention; Figure 4 This is a flowchart of the undercooling consistency write-back process provided in an embodiment of the present invention; Figure 5 This is a flowchart of preference data sampling and large language model referee annotation provided in an embodiment of the present invention; Figure 6 This is a multi-agent ego perspective scoring learning block diagram provided in an embodiment of the present invention; Figure 7 This invention provides a multi-agent reinforcement learning control framework. Figure 8 This is a comparison curve of the computer room intake air temperature and the total system power provided in an embodiment of the present invention; Figure 9 A comparison curve of cooling demand and capacity provided for embodiments of the present invention; Figure 10 A comparison chart of key evaluation indicators provided for embodiments of the present invention; Figure 11 This is a diagram illustrating the multi-agent control actions and operational stability provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0018] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of the present invention will now be described with reference to the accompanying drawings.

[0019] Example 1 Figure 1 This is a flowchart of a method for constructing a multi-agent reward function for a data center cooling system, as provided in an embodiment of the present invention. Figure 1 As shown in the figure, this embodiment illustrates a method for constructing a multi-agent reward function for a data center cooling system, including: S1: Obtain historical trajectory data generated by the interaction between multiple intelligent agents of the data center cooling system and the simulation environment of the data center cooling system, and extract the sampled state pairs in the historical trajectory data to represent the different operating states of each intelligent agent; the intelligent agent is a device that realizes the cooling process of the data center cooling system.

[0020] For example, intelligent agents may include end agents, chiller agents, water pump agents, and cooling tower agents.

[0021] S2: Compare the sampled state pairs to determine the preference labels used to evaluate the quality of each agent's actions; for each agent, construct an ego-view state representation to characterize the local state information and global associated state information related to the agent's control decisions.

[0022] Optionally, obtaining the preference label includes: sampling state pairs The input is fed into a Large Language Model (LLM), which uses a multi-factor criterion to compare preferences and determine the corresponding binary preference labels. ; The multi-factor criteria include whether the data center intake air temperature is within a safe range and the extent of exceeding the limit, the total system power or the power of key equipment, and the cooling side and... For example, trajectory data is generated by the interaction between a multi-agent reinforcement learning strategy and the simulation environment, and state pairs are sampled. The state pairs are then input into a large language model referee for comparison and preference labeling. The same state pair is adjudicated an odd number of times and the majority vote is aggregated to obtain stable preference labels.

[0023] For example, this embodiment abstracts the cooling process of the cooling system as a series three-terminal heat chain: the load side, the refrigeration side, and the cooling side. In each control step, six core heat variables are constructed: load-side demand, actual load-side demand, refrigeration side demand, actual refrigeration side demand, cooling side demand, and cooling side capacity. The actual load-side demand is defined as the minimum of the achievable heat transfer on the air side and the transferable cold on the water side, reflecting the joint constraint of the terminal air side and chilled water side on the effective heat transfer of the server room. The refrigeration side demand and the achievable heat transfer on the air side are aligned and superimposed with an equivalent heat term calculated using auxiliary power consumption. This is then input into the chiller model to solve for the actual refrigeration side demand and chiller power, ensuring that the cooling target is consistent with the achievable heat transfer at the terminal and meets the chiller performance constraints.

[0024] Based on the energy balance, the cooling-side demand is constructed, and the cooling-side capacity is constructed using the cooling tower model. The cooling-side capacity represents the maximum heat dissipation limit that the cooling tower can achieve under the current environment, inlet water temperature, and fan frequency conversion operation. This value is solved in real time by the cooling tower heat and mass transfer mechanism model. After comparing this parameter with the cooling-side demand, a sub-condensation state can be determined. The reference cooling tower fan frequency conversion ratio is obtained through numerical inverse solving to match the capacity and demand as closely as possible. Then, the residual is output by the strategy and trimmed to obtain the actual fan frequency conversion ratio, thereby improving control stability and reducing learning difficulty. When the cooling-side capacity is insufficient (sub-condensation), the actual cooling capacity supplied by the refrigeration side is written back to ensure consistency, and the chiller power and cooling-side demand are updated simultaneously to ensure energy conservation and physical consistency.

[0025] For example, to characterize the energy transfer and bottleneck effect of the cooling system under different equipment constraints, this embodiment abstracts the cooling process as a three-terminal heat chain in series: the load end (machine room side), the refrigeration side (refrigerator evaporator side), and the cooling side (refrigerator condenser / cooling tower side). In each control step... Above, the demand / reference and actual / capability for each end are constructed respectively, forming a set of six core heat variables. superscript These correspond to the load side, the freezing side (evaporation side), and the cooling side (condensation side), respectively. This six-variable system is used to measure the link heat matching deviation and its sources, and provides interpretable physical quantity inputs for the large language model judges. It also serves as an important component of the scoring network and reinforcement learning state. On the load side, the instantaneous cooling load on the data center side is obtained by aggregating the load model, representing the load demand. In one embodiment, the load model, which integrates heat transfer from the building envelope, server heat dissipation, and heat loss from power distribution / UPS, can be represented as follows:

[0026] in, This represents the heat transfer term between the building envelope and the outdoor environment. This indicates server heat dissipation (which may include airflow temperature efficiency correction). and These represent heat loss from power distribution and UPS, respectively. Actual cooling capacity removed from the load side. Simultaneously limited by the heat transfer capability of the air-side end agent CRAH and the cold transfer capacity of the water side, it is defined as the heat transfer capability achievable on the air side. Cooling can be transferred to the water side. Minimum value:

[0027] The preferred method for heat transfer on the air side is to adopt a "demand and capacity cutoff" approach, reflecting the upper limit constraint of end-point capacity. in, The results are given by the CRAH model under the current supply air temperature setting and upper and lower limits of air volume. To ensure that the CRAH action simultaneously incorporates both supply air temperature and air volume, this embodiment uses the fan frequency conversion ratio. Calculate terminal air volume For example, it can be expressed using a calibration function or a similarity law. and in Cut within the specified range. It can be further determined by factors such as air volume and supply air temperature difference, for example, it can be compared with... Proportional, or calculated by an empirical / data-driven model. Cooling can be transferred on the water side. Based on the actual cooling supply on the chiller side, and considering the equivalent heat reinjection of CRAH fan power and chilled water pump power:

[0028] in, This represents the actual cooling capacity supplied to the refrigeration side. For CRAH fan power, For chilled water pump power, This is the power consumption conversion factor. Based on the above definition, if the air side becomes the bottleneck, then even increasing the cooling capacity of the chiller will not improve efficiency. This avoids the ineffective energy consumption and training bias caused by "virtual cooling." On the freezing side, this embodiment specifies the freezing side demand... Alignment with the air side enables heat transfer, and the equivalent heat term calculated from auxiliary power consumption is added, ensuring that the actual extent of cooling demand coverage in the server room for effective removal and auxiliary equipment recharge is achieved.

[0029] Actual cooling capacity on the refrigeration side With chiller power Calculated from the chiller model. The chiller model input must include at least the chilled water outlet set temperature. Target return water temperature (or equivalent load command) and cooling side inlet conditions (Such as cooling water inlet temperature, cooling water flow rate, outdoor wet-bulb temperature, or their equivalents). The target return water temperature can be constructed based on energy relationships, for example:

[0030] in, This refers to the mass flow rate of chilled water. Specific heat. The chiller model outputs the following under rated capacity and partial load characteristics constraints:

[0031] It must also meet constraints such as the upper limit of rated cooling capacity, minimum operable load, and partial load performance curves. The chiller model can be a mechanistic model, a manufacturer's performance curve interpolation model, or a data-driven model. On the cooling side, the cooling demand... Given from the chiller's energy balance, this represents the amount of heat the chiller needs to dissipate into the environment. A preferred representation is:

[0032] in, For cooling water pump power, This is a power consumption conversion factor (which can be consistent with the above or set independently). Cooling capacity. Provided by the cooling tower model. Given the inlet water temperature. Outdoor wet-bulb temperature Cooling water flow rate Compared with the frequency conversion ratio of the fan The cooling tower model outputs heat dissipation. And define the cooling-side capability as:

[0033] in, These are calibration coefficients used to compensate for model errors or field deviations. To stabilize control and reduce learning difficulty, this embodiment first uses numerical inverse solving to obtain the reference wind turbine ratio. make (Binary search is preferred), and then the strategy outputs a small residual. And cut to obtain the actual :

[0034] When under-condensation occurs, i.e. The system cannot maintain the original cooling supply on the refrigeration side without violating the law of conservation of energy.

[0035] This embodiment treats the cooling-side capacity as a hard constraint and performs a consistent write-back of the actual cooling capacity supplied by the refrigeration side, ensuring that the written-back cooling capacity meets the heat dissipation capacity limit of the cooling side. Preferably, the write-back can be expressed as:

[0036] in, To ensure the chiller power supply is consistent with the cooling capacity after write-back, it can be obtained by resolving the chiller model under write-back conditions or through an equivalent consistency function. The cooling demand will then be recalculated after write-back. To maintain consistency in caliber and ensure final compliance Through this write-back mechanism, the cooling bottleneck can be transmitted upstream along the link, manifested as a decrease in cooling supply on the refrigeration side, a reduction in the transferable cooling capacity on the water side, and a decrease in the load-satisfaction level, thereby maintaining the true consistency of the system's operating logic and significantly improving the stability of reinforcement learning training.

[0037] For example, the large language model judges adopt a multi-factor comprehensive criterion, which includes at least one or more of the following: data center intake air temperature safety (whether it is within the safe range and the extent of exceeding the limit), energy consumption factors (total system power or power of key equipment), link matching factors (supply and demand deviation between the cooling and freezing sides, load-side satisfaction, etc.), action / equipment constraint feasibility (upper and lower limit triggering, frequent edge-triggered situations), undercooling write-back triggering, and operational stability indicators (state fluctuation amplitude, frequent switching, etc.). Under the premise of ensuring temperature safety, the judges prioritize states with lower system energy consumption, tighter supply and demand matching, stronger feasibility, and more stable operation. The judges output two types of preference labels. ,when Superior Time is recorded as Otherwise, record as To reduce the randomness of a single decision and ensure the label is determined and usable, an odd number of decisions are made for the same state pair, and the final label is obtained by aggregating the majority votes. Let the number of repetitions be 1. ( If the majority vote is odd, then the majority vote aggregation can be represented as:

[0038] in, Indicates the first The result of this ruling This is an indicator function. This mechanism avoids ties and improves the consistency and usability of the preference dataset.

[0039] S3: Construct a state scoring neural network for each agent based on a multilayer perceptron; input paired ego-perspective state representations into the state scoring neural network to score the current running state of each agent and obtain the corresponding scoring results; determine the difference in scoring results based on the Bradley-Terry model and map the difference to a state quality prediction probability used to quantify the running quality of the state; determine the log-likelihood loss based on the state quality prediction probability and preference label; train the state scoring neural network by minimizing the log-likelihood loss to determine the trained state scoring neural network.

[0040] For example, this embodiment constructs an interactive cooling system simulation environment, such as... Figure 2It is used for multi-agent policy training, trajectory acquisition, preference labeling, and scoring network training. The data center cooling system consists of the following subsystems: the terminal air-side subsystem, the chilled water / chiller evaporator subsystem, the cooling water / chiller condenser subsystem, and the cooling tower heat dissipation subsystem.

[0041] The terminal air-side subsystem of the computer room preferably includes a precision air conditioner / air handling unit (CRAH equipment), whose controlled variables include at least the supply air temperature setting and the fan frequency conversion ratio (or speed ratio), and the air volume is calculated through an air volume model. The chilled water side includes chilled water pumps, whose controlled variable is the pump speed ratio, which is mapped to the chilled water flow rate. The chiller subsystem includes an evaporator and a condenser, and the chiller's controlled variables include at least the chilled water outlet temperature setting (or equivalent load command), whose output is the actual cooling capacity supplied by the chiller side and the chiller's electrical power. The cooling water side includes cooling water pumps, whose controlled variable is the pump speed ratio, which is mapped to the cooling water flow rate; the cooling tower subsystem includes a cooling tower and its variable frequency fan, whose controlled variable is the fan frequency conversion ratio, and outputs the heat dissipation or the outlet water temperature, used to construct the cooling-side capacity. External disturbances include at least outdoor dry-bulb temperature, outdoor wet-bulb temperature, and IT load utilization (or IT power), and can be expanded to include meteorological quantities such as outdoor wind speed and relative humidity. Outdoor dry-bulb temperature refers to the actual outdoor air temperature measured directly with a conventional thermometer without obstruction. This value is unaffected by air moisture content and only reflects the sensible heat level of the air, serving as a fundamental environmental temperature disturbance parameter. Outdoor wet-bulb temperature refers to the temperature measured after the air has been continuously evaporating and cooling down to a stable temperature with a temperature probe wrapped in damp gauze. This value is coupled with air temperature and water vapor content, directly determining the maximum heat transfer limit of the cooling tower. Cooling towers rely on air evaporation for heat dissipation; the lower the outdoor wet-bulb temperature, the higher the upper limit of cooling capacity, making it a core environmental disturbance variable affecting the heat dissipation performance of cooling towers.

[0042] The simulation environment can be a physical mechanism simulation, a semi-physical, semi-data-driven simulation, or a digital twin model fitted from historical operating data. The environment operates in discrete control steps, with the control step size set according to the engineering control cycle (e.g., 1 to 15 minutes). Each control step executes a closed loop of "reading status, outputting actions, calculating equipment response, updating heat transfer paths, and recording data," as shown below. Figure 3 As shown.

[0043] For example, the six variables are not only used as input to the scoring network, but also to transform the algorithm evaluation results into operational diagnostics that can be understood on-site, as shown in Table 2. For instance, when the actual load is lower than the demand, priority is given to checking the air volume on the terminal air side, the supply air temperature setting, and the chilled water cooling capacity; when the actual load on the chiller side is higher than the demand, it indicates that there may be oversupply of chillers or excessive water pump distribution; when the demand on the cooling side is greater than the capacity, it indicates that the cooling tower is not dissipating enough heat at the current wet-bulb temperature, water flow rate, or fan frequency, and it is necessary to constrain the chiller output and adjust the cooling water pump or cooling tower fan operation in the digital twin or constrained control.

[0044] Table 2. Correspondence between six variable anomalies and actual equipment actions The actual load is less than the load demand. Insufficient terminal air volume, excessively high supply air temperature, or insufficient water-side cooling End-point intelligent agent, chiller intelligent agent, chilled water pump intelligent agent The heat in the server room was not adequately removed; ensuring temperature safety must be a priority. Air-side capacity is lower than the data center's requirements. CRAH / AHU fan frequency, supply air temperature, or heat exchange capacity are limited. End agent Continuing to increase the cooling capacity of the chiller may be ineffective; improvements should be made to the terminal heat exchangers first. The actual volume on the frozen side is significantly higher than the demand on the frozen side. Oversupply of chiller or excessive distribution of chilled water pump Refrigeration unit intelligent agent, chilled water pump intelligent agent There is ineffective cooling and unnecessary energy consumption. The cooling demand exceeds the cooling capacity. High humidity bulb, insufficient heat dissipation from the cooling tower, and insufficient cooling water flow. Cooling tower intelligent agent, cooling water pump intelligent agent, chiller intelligent agent There is a risk of undercooling; therefore, unattainable cooling output should be limited. Frequent movements that hit upper or lower limits or fluctuate excessively Unstable control strategy or continuous pressure on equipment capability boundaries All intelligent agents The preference score decreases, triggering security verification or reverting to the original policy. It should be noted that consistency write-back is preferred as a physical consistency constraint in the simulation environment, digital twin model, or policy training phase. This is used to prevent reinforcement learning from discovering non-physical states where the cooling tower cannot dissipate heat but the chiller continues to provide high cooling output. In actual field closed-loop systems, this write-back result can be transformed into control suggestions, actionable domains, or safety constraints, and is covered by the original chiller protection logic, building automation system, and manual takeover mechanisms, rather than directly bypassing the underlying equipment protection. Figure 4 As shown.

[0045] For example, since the cooling plant system consists of multiple devices working together to achieve the overall goal, the physical quantities that different agents can perceive or should pay attention to are not the same. Therefore, this embodiment does not use a single global scorer, but instead learns the state scoring function from the ego perspective of each agent. For each agent... Construct its exclusive state representation It can include a subset of global states (such as outdoor dry / wet bulb temperature, server room intake air temperature, IT load, total system power, etc.), and further include local variables directly related to the device (such as device actions, power, flow rate, setpoints, and relevant subsets of the six variables, as well as touch / write-back flags) to enhance the scorer's sensitivity to the device's marginal contribution. Based on preference labels, a preference dataset is constructed for each agent. And independently train the corresponding state scoring neural network. Output scalar score .

[0046] For example, an ego-perspective state representation can be constructed for each agent, and an agent-by-agent preference dataset can be built. A state rating neural network can then be trained on each dataset based on the Bradley-Terry comparison model. The rating learning employs the Bradley-Terry comparison model, which is used for any pair of states. In the Bradley-Terry model, the preference probability is defined as:

[0047] in, The sigmoid function. Binary preference labels obtained from large language model judging. The scoring network is trained using log-likelihood loss.

[0048] in, After training, the accuracy of preference judgments can be used to evaluate the fitting effect: when... predict Superior Conversely, if the statistics and labeling are consistent, the accuracy will be obtained.

[0049] For example, to avoid preference labeling remaining at the level of abstract criteria, the state pairs input to the large language model are preferably described using structured fields, retaining the engineering units, source, and constraint meaning of the fields. An example is given in Table 3. Both server room temperatures in both states are within a safe range, but state A has higher system power and cooling demand exceeds the cooling tower's capacity, resulting in undercooling write-back triggering. State B has a still safe temperature, lower power, and better cooling supply-demand matching; therefore, the judge should output that state B is superior to state A. Examples of preference-labeled state pairs are shown in Table 3.

[0050] Table 3 Examples of Preference Labeling Status Pairs Computer room intake air temperature 24.5 degrees Celsius 24.8 degrees Celsius Both meet temperature safety constraints. Total system power 620 kW 590 kW State B has lower energy consumption. Supply and demand imbalance on the frozen side Demand was 510 kW, actual output was 545 kW Demand was 505 kW, actual demand was 510 kW Status B: Less oversupply Cooling side supply and demand status Demand: 780 kW; Capacity: 760 kW Demand: 720 kW; Capacity: 735 kW State A indicates insufficient heat dissipation capacity; State B indicates a tighter fit. Action contact edge / stability The cooling tower fan is nearing its maximum capacity, resulting in significant control fluctuations. The fan did not touch the edge, and the movement was smooth. State B offers better feasibility and stability. Large language model output Non-preferred state Preferred state Under the premise of temperature safety, priority should be given to low energy consumption, fewer bottlenecks, and stable operation. The numerical values ​​in the above state pair examples are only for illustrating the field organization method. In actual implementation, state pairs can be directly generated from historical running segments, digital twin simulation segments, or shadow running segments. Temperature, power, six variables, edge flag, write-back flag, and fluctuation index are input into the large language model together. This ensures that the generated preference labels are consistent with the actual operating conditions, and the corresponding structure is as follows: Figure 5 As shown in Table 4, which is a description of the input fields for the large language model's adjudication.

[0051] Table 4: Explanation of Judge Input Fields for Major Language Models Computer room intake air temperature C Core security constraints, determining whether boundaries have been exceeded. Total system power kW Energy efficiency core indicators for judging energy saving Cooling side supply and demand imbalance kW Determine the heat dissipation pressure and whether a consistency write-back is triggered. System performance index kW / kW Assess the power consumption cost per unit of cooling capacity Action touch mark {0, 1} Assess whether the action has reached its limit to determine system stability. Actual removal of cooling capacity kW Assess the load-side satisfaction level. The per-entity state scoring network does not assign a single global score to all devices. Instead, it selects input fields based on the controllable objects and marginal contributions of each device. The terminal agent primarily focuses on the data center inlet air temperature, supply air temperature setting, fan frequency, terminal fan power, load demand, and actual load. The chiller agent primarily focuses on the chilled water outlet temperature setting, chiller power, chiller demand, actual chiller load, and cooling constraints. The pump agent primarily focuses on pump frequency, water flow rate, supply and return water temperature difference, pump power, and corresponding supply-demand deviation. The cooling tower agent primarily focuses on outdoor wet-bulb temperature, cooling tower fan frequency, cooling demand, cooling capacity, and capacity insufficiency triggering conditions, as shown in Table 5.

[0052] Table 5 Mapping between the entity-by-entity scoring network and actual equipment responsibilities End agent Data center intake air temperature, supply air temperature setting, fan frequency, terminal fan power, load demand / actual quantity Supply air temperature, fan frequency or air volume The score increases when the temperature recovers safely and there is no significant waste of fan energy; the score decreases when there is a temperature risk or excessive fan energy consumption. Refrigeration Intelligent Agent Chilled water outlet temperature setting, chiller power, chiller-side demand / actual quantity, and cooling-side constraint information. Chilled water outlet temperature, cooling capacity or load command The score increases when cooling supply matches terminal demand and chiller power is reduced; the score decreases when over-cooling or when constrained by cooling-side limitations. Water pump intelligent agent Pump frequency, water flow rate, supply and return water temperature difference, pump power, supply and demand deviation on the chilled or cooling side. Chilled water pump and cooling water pump frequency or flow rate The score increases when energy consumption for transmission and distribution is reduced while heat exchange remains stable; the score decreases when the flow rate is insufficient or there is an ineffective high flow rate. Cooling tower intelligent agent Outdoor wet-bulb temperature, cooling tower fan frequency, cooling-side demand, cooling-side capacity, and capacity deficiency indicators. Cooling tower fan frequency or heat dissipation capacity adjustment amount The score increases when the heat dissipation requirement is met with lower fan power consumption; the score decreases when heat dissipation is insufficient under high humidity conditions and frequent contact with the edge occurs. Based on the above division, the scalar score learned by the Bradley-Terry model has clear engineering implications: the score difference not only represents the superiority or inferiority of two states, but also corresponds to the control responsibilities and areas for improvement of a certain type of equipment, thereby alleviating the credit allocation problem in multi-agent reinforcement learning, such as... Figure 6 As shown, the relevant fields are listed in Table 6.

[0053] Table 6. List of ego state representation fields for each agent CRAH 4 Balancing terminal heat exchange efficiency and fan power consumption Chiller 4 Refrigeration unit COP optimization and precise cooling capacity tracking Pumps 4 Hydraulic balance and pump frequency conversion efficiency Cooling tower 4 Environmental heat exhaust utilization and fan energy saving S4: Input the ego-perspective state representation of the agent under test into the trained state scoring neural network to obtain the state score of each agent. Based on the state score, determine the training reward of each agent. Use the training reward as the reward signal for the corresponding agent's policy optimization to update the policy parameters of each agent to obtain the corresponding set of control actions.

[0054] For example, in this embodiment, after obtaining the body-by-body state scoring network set... Subsequently, during the reinforcement learning training phase, instead of constructing an overall (team) reward, the training reward for each agent is directly the rating output of the corresponding scoring network for the state or the potential difference between adjacent time steps. This alleviates the credit allocation problem and aligns the optimization objective with the engineering preferences reflected in the large language model. (Environmental state) This may include at least outdoor dry-bulb temperature, wet-bulb temperature, server room intake air temperature, IT load utilization, and system energy consumption information, wherein the energy consumption information preferably includes the total system power. Alternatively, the power set of key equipment (chimney power, CRAH fan power, chilled water pump power, cooling water pump power, cooling tower fan power, etc.) can be used to explicitly consider energy consumption factors in the strategy. System actions are output jointly by multiple agents, and at least include the CRAH supply air temperature setting and fan frequency conversion ratio (from which airflow is calculated), chilled water outlet temperature setting, chilled water pump speed, cooling water pump speed, and cooling tower fan frequency conversion ratio (or residual). For any agent... Its rating is In one implementation, a score is used as an immediate reward:

[0055] In a preferred embodiment, the score potential difference between adjacent time steps is used as the immediate reward, making the reward more directly related to the state change caused by the agent's action and reducing the impact of single-point score bias on training. Each agent uses its own reward sequence as a reinforcement learning reward update strategy and jointly forms the system control action. Since the scoring network is trained by a large language model using multi-factor comparison preference supervision on state pairs, the agent-by-agent reward naturally carries comprehensive preferences such as temperature safety, energy consumption, link matching, constraint feasibility and stability, thereby reducing the instability and subjective bias caused by the design of artificial reward weights. Under the premise of ensuring physical consistency through the undercooled condensation consistency write-back mechanism, more stable training and more reliable policy output are achieved. The parameter settings are shown in Table 7.

[0056] Table 7 Training Hyperparameters and Action / State Range Control step size 5-15 min Balancing physical system response speed and action frequency [cite: 74] Power consumption conversion factor 1.0 [cite: 17, 85] The electrical power consumption of the equipment is included in the equivalent reinjection heat [cite: 83, 89]. Number of times the referee made the same ruling 3 [cite: 41, 123] An odd number of rulings is used to ensure the uniqueness of the majority vote aggregation. [cite: 41] Learning Rate Optimization speed of scoring network and policy network Discount factor (Gamma) 0.99 Weighting for long-term energy efficiency returns Sampling scale 500 pairs Ensuring sample diversity during training of the individual scorer [cite: 120] For example, after completing the inference of the state scoring network ensemble, the real-time state scores specific to each agent are extracted and training rewards are constructed. These rewards are then used as the reward signal for iterative optimization of the policy network and fed into the reinforcement learning update process. First, the long-term cumulative reward is calculated based on the single-step immediate reward combined with a discount factor and time-series accumulation. A policy gradient loss function is constructed with the goal of maximizing the expected cumulative reward. The gradient of the policy network parameters is solved layer by layer using the backpropagation algorithm. The Adam gradient optimizer is used to update the weights and biases of four independent policy networks: the end-point, chiller, chilled water pump, and cooling water pump. After parameter iteration, each agent's policy network outputs controlled actions in parallel based on the current global thermal state. The outputs of all agents are integrated to obtain a complete set of cooling system control actions, which are then sent to the simulation environment. After the environment executes the control inputs, it generates the operating state for the next time step, and the above steps are repeated to form a complete reinforcement learning closed-loop training process.

[0057] For example, in actual engineering deployments, the actions output by the multi-agent strategy are preferably used as upper-level optimization suggestions or restricted closed-loop settings, and are then distributed to the corresponding control points via the building automation system or chiller group control system. Control points may include CRAH / AHU supply air temperature settings, terminal fan frequencies, chilled water outlet temperature settings, chilled water pump frequencies, cooling water pump frequencies, and cooling tower fan frequencies, etc. Each action is verified against the equipment's permissible range, rate of change, temperature safety boundaries, and protection status before being distributed; if the verification fails, the original control strategy is maintained or the default safety values ​​are reverted. The control framework is as follows: Figure 7 As shown.

[0058] Therefore, this invention does not replace the interlocking and fault protection of the field-level controller, but rather provides a collaborative optimization layer on top of the existing monitoring and control architecture. The per-entity reward is used to train and select better control strategies, and its actual execution is still constrained by field safety boundaries, manual takeover mechanisms, and equipment protection logic.

[0059] Example 2 For example, the present invention can be used to conduct simulation comparison experiments under different external wet-bulb temperatures and different IT load variations, comparing the method of the present invention with rule control or traditional reinforcement learning reward design methods. Evaluation indicators may include at least: the compliance rate and extent of exceeding the limit of the data center intake air temperature, the total power and peak power of the system, the supply and demand deviation between the cooling and freezing sides, the number of undercooling write-back triggers and the cumulative write-back amount, the matching degree of the six variables, and the feasibility of the strategy actions (number of touches, degree of frequent switching, etc.).

[0060] For example, temperature curves, power curves, and six-variable comparison curves can be plotted, and under-condensation trigger points can be marked to demonstrate the robustness and interpretability of the invention under difficult operating conditions such as high load or high wet-bulb conditions. Figure 8 As shown, under the same load disturbance and external weather conditions, both the initial strategy and the method of the present invention can maintain the air intake temperature of the computer room below the safe limit. However, the total system power corresponding to the method of the present invention is lower than that of the initial strategy, indicating that the per-agent scoring reward can guide the multi-agent strategy to reduce the energy consumption of the cooling system while ensuring temperature safety.

[0061] like Figure 9 As shown, during the high wet-bulb period, the initial strategy's cooling-side demand easily exceeds the cooling-side capacity, thus creating a trigger point for under-condensation. The method of this invention, through cooling-side capacity constraints, supply-demand matching preferences, and a consistency write-back mechanism, makes the cooling-side demand closer to the capacity boundary and reduces the occurrence of exceeding the capacity. This result demonstrates that the six-variable calculation is not an abstract mathematical quantity, but rather corresponds to the actual cooling tower heat dissipation capacity, condenser-side load, and engineering bottlenecks under high wet-bulb conditions.

[0062] To demonstrate the correlation between the calculation results and the actual scenario, the effect verification should not only use the team's average reward, but also provide engineering interpretable indicators, including the average system power and peak power reduction, the power consumption per unit cooling capacity of the cooling system, the room temperature compliance rate and the maximum out-of-bounds range, the supply and demand deviation between the refrigeration and cooling sides, the number of undercooling triggers and the cumulative write-back amount, the number of action touch-outs, the range of control quantity changes, and the stability under high wet-bulb, high load, low load and load change scenarios, as shown in Table 8.

[0063] Table 8 Engineering Validation and Phased Deployment Process Offline training phase Historical operating data, equipment curves, and digital twin models Preference label consistency, rating network accuracy, policy training convergence Temperature safety constraints are met, and rewards are consistent with engineering performance indicators. Simulation verification stage High wet bulb, high load, low load and load change scenarios System power, supply and demand deviation, under-condensation triggering, operating contact, and temperature exceedance. The relative baseline strategy reduces energy consumption and introduces no new security risks. Shadow running phase Real-time field data, only providing suggestions and not distributing them to equipment. Compare the proposed actions with current strategies to assess potential energy savings and risks. The suggested actions are within the permissible range and the risk indicators are controllable. Limited closed-loop stage Some devices or some control quantities are gradually connected. Safety boundaries, rate of change of motion, manual takeover and return mechanisms Expand the equipment range and operating conditions only after long-term stability has been achieved. Through the above verification methods, reward enhancement, improved supply-demand matching, and reduced energy consumption can all be expressed using engineering indicators, avoiding the need to explain the effects solely through algorithm training curves, thereby strengthening the correspondence between the computing component and real data center application scenarios. For example... Figure 10 As shown, while maintaining the temperature safety compliance rate, the method of this invention reduces average system power, decreases supply-demand matching error and the number of under-condensation triggers, and increases the average team reward while reducing the number of action edge triggers. These results demonstrate that the method of this invention can simultaneously improve multiple engineering evaluation indicators such as temperature safety, energy consumption, link matching, and control feasibility. Table 9 shows a comparison of different methods in terms of temperature compliance rate, average power, peak power, supply-demand deviation, and number of under-condensation triggers.

[0064] Table 9 Comparison of different methods in terms of temperature compliance rate, average power, peak power, supply-demand deviation, and number of undercondensation events. Team average reward 9.597 9.709 +1.17% Tower Gap (supply and demand matching error) 1.016 kW 0.964 kW Decrease of 5.11% Temperature safety compliance rate 100% 100% Balanced (high security control) Link bottleneck identification rate Low high Explicit positioning bottleneck like Figure 11 As shown, compared with the initial strategy, the frequency changes of CRAH fan, chilled water pump, and cooling tower fan are more stable under the method of this invention, the phenomenon of frequently hitting the upper and lower limits of action is significantly reduced, and the rate of action change is also lower. This result indicates that the per-entity scoring reward can not only improve energy consumption and supply-demand matching, but also suppress the common problems of action jitter and boundary saturation in strategy training, thereby improving the deployability and operational stability of the control strategy in actual data center cooling systems.

[0065] The above describes a method for constructing a multi-agent reward function for a data center cooling system, provided by one or more embodiments of this specification. Based on the same approach, this specification also provides a corresponding system for constructing a multi-agent reward function for a data center cooling system. Specific limitations of the system for constructing a multi-agent reward function for a data center cooling system can be found in the limitations of the method described above, and will not be repeated here. Each module in the above-described system for constructing a multi-agent reward function for a data center cooling system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this invention.

Claims

1. A method for constructing a multi-agent reward function for a data center cooling system, characterized in that, include: Historical trajectory data generated by the interaction between multiple intelligent agents of the data center cooling system and the simulation environment of the data center cooling system is obtained, and sampled state pairs used to characterize different operating states of each intelligent agent are extracted from the historical trajectory data; the intelligent agent is a device that realizes the cooling process of the data center cooling system. Preference comparisons are performed on sampled state pairs to determine preference labels for evaluating the quality of each agent's actions; for each agent, an ego-view state representation is constructed to characterize the local state information and global associated state information related to the agent's control decisions. Each agent's state scoring neural network is constructed based on a multilayer perceptron. Paired ego-perspective state representations are input into the state scoring neural network to score the current running state of each agent and obtain the corresponding scoring results. The difference between the scoring results is determined based on the Bradley-Terry model, and the difference is mapped to the state quality prediction probability used to quantify the running quality of the state. Based on the state quality prediction probability and preference label, the log-likelihood loss is determined. The state scoring neural network is trained by minimizing the log-likelihood loss to determine the trained state scoring neural network. The ego-perspective state representation of the agent under test is input into the trained state scoring neural network to obtain the state score of each agent. Based on the state score, the training reward of each agent is determined. The training reward is used as a reward signal for the corresponding agent's policy optimization to update the policy parameters of each agent in order to obtain the corresponding set of control actions.

2. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 1, characterized in that, The acquisition of the preference label includes: sampling state pairs The input is fed into a Large Language Model (LLM), which uses a multi-factor criterion to compare preferences and determine the corresponding binary preference labels. ; The multi-factor criteria include whether the data center intake air temperature is within a safe range and the extent of any deviations, the total system power or the power of critical equipment, and the supply-demand imbalance between the cooling and chiller sides; among these factors... express Superior Otherwise, it means Superior .

3. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 2, characterized in that, When training the state scoring neural network, log-likelihood loss is used to update the network parameters; wherein, obtaining the log-likelihood loss specifically includes: ； in, For log-likelihood loss, The binary preference labels output by the LLM model. This is to combine the Bradley-Terry model with the network score difference mapped by the Sigmoid function to predict the state quality probability.

4. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 1, characterized in that, The intelligent agents include terminal intelligent agents for end heat exchange, chiller intelligent agents for cold source generation, water pump intelligent agents for driving cold source circulation, and cooling tower intelligent agents for heat dissipation; the ego perspective state representation includes load-side demand, actual load-side quantity, refrigeration-side demand, actual refrigeration-side quantity, cooling-side demand, and cooling-side capacity.

5. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 4, characterized in that, Based on the real-time operating power of the representative terminal fans and chilled water pumps, the equivalent waste heat is determined. The equivalent waste heat is then superimposed with the acquired basic heat dissipation load on the air side of the computer room to obtain the demand on the chilled side.

6. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 4, characterized in that, The actual effective cooling capacity of the refrigeration side, the real-time input power of the chiller unit, and the real-time operating power of the cooling water pump are superimposed to determine the cooling side demand.

7. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 4, characterized in that, Based on the cooling tower model, the heat dissipation capacity is output under given conditions of inlet water temperature, outdoor wet-bulb temperature, cooling water flow rate and fan frequency conversion. Combined with the calibration coefficient representing on-site deviation compensation, the cooling capacity is obtained.

8. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 4, characterized in that, The heat transfer items of the building envelope and the outdoor environment, the heat dissipation items of the server, and the heat loss of the power distribution and UPS are summed to determine the load demand; the minimum value between the heat transfer capacity that can be realized on the air side and the cold transfer capacity that can be transferred on the water side is taken as the actual load quantity.

9. The method for constructing a multi-agent reward function for a data center cooling system as described in claim 8, characterized in that, The air side can achieve the acquisition of heat transfer and the water side can achieve the acquisition of cold transfer, specifically including: The minimum value between the total heat load of the computer room and the maximum heat exchange capacity on the terminal air side is taken as the heat transfer capacity that can be realized on the air side, specifically including: ； Based on the obtained total cooling capacity on the chiller side, and the weighted power consumption of the terminal fans and chilled water pumps, the transferable cooling capacity on the water side is obtained in the following manner: ； in, Heat can be transferred to the air side. The total heat load of the computer room, This represents the maximum heat exchange capacity on the terminal air side. This refers to the total cooling capacity on the refrigeration side. , These are the power consumption of the terminal fan and the chilled water pump, respectively. This is the electro-thermal conversion factor.

10. A multi-agent reward function construction system for a data center cooling system, characterized in that, A method for constructing a multi-agent reward function for a data center cooling system as described in any one of claims 1-9.

Images