A Multi-Objective Optimization Method and System for HVAC Systems Based on Reinforcement Learning

Abstract

This invention discloses a multi-objective optimization method and system for HVAC systems based on reinforcement learning, relating to the field of building energy system optimization technology. The method includes: constructing a state space based on collected environmental parameters, equipment state parameters, and historical operating data of the HVAC system; the state space consists of state vectors generated after normalization and discretization; constructing an action space based on the controllable equipment of the HVAC system; the action space is generated by discretizing the operating modes and control parameters of the controllable equipment; constructing a multi-objective reward function to evaluate the control effect based on the state vectors and action space; using the state vectors as input, iteratively training using the Double Q learning algorithm until a trained control model is obtained; inputting the currently collected state vectors into the trained control model and outputting the optimal action to generate control commands for the HVAC system.

Description

Technical Field

[0001] This application relates to the field of building energy system optimization technology, and more specifically to the field of multi-objective optimization of HVAC systems. In particular, it relates to a multi-objective optimization method and system for HVAC systems based on reinforcement learning. Background Technology

[0002] Building energy consumption accounts for a significant proportion of total social energy consumption. As a core component of building energy consumption, the operating efficiency of HVAC systems directly impacts the achievement of building energy conservation goals, the improvement of indoor thermal comfort, and the protection of the ecological environment. With the deepening promotion of green building and sustainable development concepts, the scientific optimization of HVAC systems to achieve synergy between energy efficiency and user needs has become an important research direction in the field of building energy, and has significant practical implications for reducing carbon emissions and alleviating the contradiction between energy supply and demand.

[0003] Existing optimization methods for HVAC systems still have certain limitations in practical applications. Some optimization schemes are difficult to achieve balanced optimization of multiple objectives, often focusing on a single performance indicator (such as energy consumption reduction) while ignoring key usage requirements such as indoor environmental comfort, resulting in an overall operating effect that does not match the actual application scenario.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a multi-objective optimization method and system for HVAC systems based on reinforcement learning to solve the above-mentioned technical problems.

[0006] This application provides a multi-objective optimization method for HVAC systems based on reinforcement learning, comprising: constructing a state space based on collected environmental parameters, equipment state parameters, and historical operating data of the HVAC system; the state space consists of state vectors generated after normalization and discretization processing; constructing an action space based on controllable equipment of the HVAC system; the action space is generated by discretizing the operating modes and control parameters of the controllable equipment; constructing a multi-objective reward function to evaluate the control effect based on the state vectors and the action space; using the state vectors as input, iteratively training using a Double Q learning algorithm until a trained control model is obtained; wherein, in each iteration, a first Q network and a second Q network respectively estimate the state-action value function, and by alternately selecting one of the first Q network and the second Q network to perform action selection, and calculating the objective Q value of the other, the parameters of the network performing action selection are updated based on the multi-objective reward function; wherein, during the iterative training process, an adaptive ε-greedy strategy is used to select actions from the action space; inputting the currently collected state vector into the trained control model, outputting the optimal action, and generating control commands for the HVAC system.

[0007] This application provides a multi-objective optimization system for HVAC systems based on reinforcement learning, comprising: a state acquisition module for constructing a state space based on acquired environmental parameters, equipment state parameters, and historical operating data of the HVAC system; the state space consists of state vectors generated after normalization and discretization processing; an action set construction module for constructing an action space based on controllable equipment of the HVAC system; the action space is generated by discretizing the operating modes and control parameters of the controllable equipment; a reward function construction module for constructing a multi-objective reward function to evaluate the control effect based on the state vectors and the action space; and a training module for iteratively training the state vectors using a Double Q learning algorithm until a trained control model is obtained; wherein, in each iteration, a first Q network and a second Q network estimate the state-action value function respectively, by alternately selecting the first Q network... The system selects an action from one of the network and the second Q network, calculates the Q-value of the other objective, and updates the parameters of the action selection network based on the multi-objective reward function; wherein, during the iterative training process, an adaptive ε-greedy strategy is used to select actions from the action space; the output module is used to input the currently acquired state vector into the trained control model and output the optimal action to generate control commands for the HVAC system.

[0008] Based on the embodiments provided in this application, a normalized and discretized state space is constructed based on the environmental parameters, equipment status parameters, and historical operating data of the HVAC system. This state space comprehensively and systematically represents the system's operating status, providing a reliable data foundation for subsequent optimization decisions. Simultaneously, a discretized action space is constructed based on controllable equipment, ensuring precise matching of action selection with actual equipment control requirements and guaranteeing the executability of control commands. By constructing a multi-objective reward function, multiple key control indicators such as energy consumption and comfort can be evaluated simultaneously, effectively overcoming the limitations of single-objective optimization in existing technologies and achieving multi-objective collaborative optimization, balancing building energy conservation and user experience.

[0009] Iterative training using the Double Q learning algorithm, with the first and second Q networks alternately performing action selection and target Q-value calculation, avoids the overestimation problem common in traditional Q-learning, improving the stability of the training process and making the final control model more reliable. Combined with an adaptive ε-greedy policy for action selection, it dynamically balances action exploration and experience utilization, ensuring the exploration of potential better control strategies while fully utilizing accumulated training experience, thus improving model training efficiency and optimization results. Inputting the currently acquired state vector into the trained control model allows it to output the optimal action and generate control commands, enabling rapid response to real-time operating condition changes. This enhances the adaptability of HVAC systems to dynamic environments and equipment state fluctuations, ensuring the timeliness and accuracy of the control strategy, and aligning with the practical application scenarios of continuous operation of HVAC systems. Attached Figure Description

[0010] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0011] Figure 1 A flowchart of an optional reinforcement learning-based multi-objective optimization method for HVAC systems according to an embodiment of this application;

[0012] Figure 2 This is a flowchart of another optional reinforcement learning-based multi-objective optimization method for HVAC systems according to an embodiment of this application;

[0013] Figure 3 This is a structural diagram of an optional reinforcement learning-based multi-objective optimization system for HVAC systems according to an embodiment of this application;

[0014] Figure 4 This is a schematic diagram of the structure of an optional electronic device according to an embodiment of this application.

[0015] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] According to one aspect of the embodiments of this application, such as Figure 1 As shown, this application provides a multi-objective optimization method for HVAC systems based on reinforcement learning, including:

[0018] S101, construct a state space based on the collected environmental parameters, equipment status parameters and historical operating data of the HVAC system; the state space consists of state vectors generated after normalization and discretization processing;

[0019] In some embodiments, environmental parameters include outdoor temperature, outdoor humidity, and IT load; equipment status parameters include chiller load, pump frequency, and fan speed; and historical operating data includes historical PUE, historical load, and historical temperature.

[0020] A state space is constructed based on the collected environmental parameters, equipment status parameters, and historical operating data of the HVAC system, including:

[0021] The Min-Max normalization method is used to normalize each item in the environmental parameters, equipment status parameters and historical operating data, mapping them to the [0,1] interval;

[0022] The normalized continuous parameters are discretized by binning. The outdoor temperature value, IT load value, PUE historical value, and chiller load value are each divided into 10 levels, and the hourly information is divided into 4 time period levels.

[0023] Calculate the linear trends of PUE, IT load, chiller load and temperature over the past 24 hours, normalize the trend values ​​to the [-1,1] interval, and then divide them into 10 levels.

[0024] The various levels after binning and discretization are combined and encoded to generate a multi-dimensional discrete state identifier, which serves as the state vector.

[0025] S102, constructing an action space based on controllable devices in a HVAC system; the action space is generated by discretizing the operating modes and control parameters of the controllable devices;

[0026] In some embodiments, the control parameters include equipment start / stop control parameters, water supply temperature setting parameters, pump frequency adjustment parameters, fan speed control parameters, air supply temperature setting parameters, and air wall speed adjustment parameters; the operating modes include mechanical cooling, partial free cooling, and pure free cooling, which are three energy-saving cooling strategies that the data center switches according to changes in outdoor temperature: when the outdoor temperature is high, it relies entirely on power-consuming mechanical cooling; when the temperature is moderate, mechanical cooling is supplemented by natural cold sources (partial free cooling); when the temperature is low enough, it fully utilizes free natural cold sources (pure free cooling), which has the lowest energy consumption.

[0027] The controllable equipment based on the HVAC system constructs an action space, including:

[0028] The equipment start-stop control parameters are discretized into a set of actions including stopping, starting at 50% load, starting at 75% load, and starting at 100% load;

[0029] The water supply temperature setting parameter is discretized into 7 setting values ​​with 1°C intervals in the range of 6°C to 12°C;

[0030] The pump frequency adjustment parameter is discretized into 5 set values ​​with 5 Hz intervals in the range of 30 Hz to 50 Hz.

[0031] The fan speed control parameters and the wall speed adjustment parameters are discretized into multiple percentage levels;

[0032] The air supply temperature parameter is set discretely into 7 settings in 1°C intervals within the range of 20°C to 26°C;

[0033] The operating mode is combined with the discretized control parameters to form a complete action in the action space.

[0034] S103, Construct a multi-objective reward function to evaluate the control effect based on the state vector and action space;

[0035] Among them, the multi-objective reward function is used to quantitatively evaluate the system's comprehensive performance in terms of energy saving, safety, stability and economy.

[0036] S104, take the state vector as input, use the Double Q learning algorithm to perform iterative training until the trained control model is obtained;

[0037] In each iteration, the state-action value function is estimated by the first Q network and the second Q network respectively. By alternately selecting one of the first Q network and the second Q network to perform action selection, the other to perform target Q value calculation, and updating the parameters of the network for performing action selection based on the multi-objective reward function;

[0038] In the iterative training process, an adaptive ε-greedy strategy is used to select actions from the action space;

[0039] The training of the control model is considered complete when one of the following training termination conditions is met:

[0040] The number of training iterations completed reached 10,000.

[0041] On a predefined set of validation states, the average reward value of the model increased by less than 1% over 100 consecutive iterations.

[0042] S105 inputs the currently acquired state vector into the trained control model and outputs the optimal action to generate control commands for the HVAC system. It should be noted that the state-action space of an HVAC system is enormous, and traditional Q-learning is prone to overestimating the Q-value due to maximizing operations, thus learning suboptimal or even unsafe policies. Double Q-learning, by using two Q-networks that alternately assume the roles of "action selection" and "value assessment," effectively decouples the selection of the maximum action from the calculation of the target Q-value, significantly mitigating the overestimation problem and improving the reliability and safety of the learning strategy.

[0043] The HVAC system environment is dynamically changing, and a fixed exploration rate cannot adapt to the needs of different learning stages. The adaptive ε-greedy strategy can maintain a high exploration rate in the early stages of learning to make extensive attempts, and actively increase exploration to escape local optima when the strategy performance improves slowly. This dynamic balancing mechanism is crucial for finding the globally optimal strategy in complex environments.

[0044] Furthermore, the multi-objective reward function integrates energy-saving, safety, stability, and economic objectives; among which, the economic objective is calculated based on operating cost savings. The multi-objective reward function is constructed as follows:

[0045] Initial weights were assigned to the energy-saving, safety, stability, and economic objectives, respectively.

[0046] Based on the IT load value and outdoor temperature value in the state vector, adjust the energy-saving target weight W_energy and the safety target weight W_safety; specifically, when the IT load value is higher than the first load threshold, increase W_safety and decrease W_energy; when the outdoor temperature value is lower than the first temperature threshold, increase W_energy and decrease W_safety; the stability target weight W_stability and the economic target weight W_economic remain fixed.

[0047] For example, the first load threshold is set to 80% of the rated IT load. This threshold is determined based on historical operating data analysis; when the load exceeds this level, the system cooling pressure increases significantly, and the safety risks rise. The first temperature threshold is set to an outdoor temperature of 10°C. This threshold is based on local climate data and free cooling efficiency analysis; when the outdoor temperature is below this value, using free cooling mode has significant energy-saving potential.

[0048] The initial weights are: W_energy=0.4, W_safety=0.3, W_stability=0.2, W_economic=0.1.

[0049] Under high load conditions (IT load = 90% of rated value, outdoor temperature = 25°C), if the IT load (90%) is greater than the first load threshold (80%), a weight adjustment is triggered, increasing W_safety and decreasing W_energy. The adjusted weights are: W_safety = 0.3 + 0.2 = 0.5, W_energy = 0.4 - 0.2 = 0.2, W_stability = 0.2, W_economic = 0.1.

[0050] The reward value R_energy for energy saving targets is calculated based on the degree of improvement in the PUE value of the HVAC system relative to the historical baseline after the action is performed;

[0051] The energy-saving target incentive encourages systems to improve energy efficiency, calculated based on the improvement in current PUE relative to a dynamic historical baseline. The historical baseline uses the average PUE over the past 24 hours to reflect recent normal operating levels. In one example,

[0052]

[0053] in, The energy-saving reward coefficient is a preset normal number used to convert the degree of improvement into a reward value. It is recommended to set the value to 50. This value is based on simulation experiments and can ensure that when the PUE improves by 1%, the reward value is about 0.5, so that the reward signal occupies a reasonable proportion in the total reward, which can effectively guide learning without overdoing it.

[0054] This represents the system's average energy efficiency over the past 24 hours. This represents the system's current instantaneous energy utilization efficiency value after the action is performed.

[0055] The reward value R_safety for the safety target is calculated using a piecewise function, including: when the operating parameters of the HVAC system reflected by the state vector are within the safety boundary, R_safety is a fixed positive number; when an operating parameter exceeds the safety boundary, R_safety is a negative number with an absolute value greater than the fixed positive number; where the operating parameters include chilled water temperature, cooling water temperature, and chiller load;

[0056] The safety target reward uses a piecewise function to establish clear safety boundaries and impose severe penalties for exceeding those boundaries. Its design is based on the safe operating range of key parameters of the HVAC system. In one example,

[0057]

[0058] in, A fixed reward is set for safe operation, which is a preset normal number. In this embodiment, the value is 10. This value provides a moderate positive incentive to encourage the agent to stay in the safe zone. The safety penalty coefficient is a factor much larger than... The normal value is used to ensure the severity of the punishment. In this embodiment, the value is 1000, which is much greater than 10. This ensures that even a slight deviation will result in a strong negative reward, thereby strictly prohibiting any unsafe behavior. The actual value of the i-th safety parameter (such as chilled water temperature, cooling water temperature, chiller load) has been normalized to dimensionless. Let i be the target or ideal value (dimensionless) of the i-th safety parameter.

[0059] The permissible deviation threshold (dimensionless) for the i-th safety parameter is set based on historical data or equipment specifications, and according to system safety procedures. For example, for chilled water supply temperature, the target value is... =7.0 (equivalent to 0.2 after normalization), allowable deviation =0.1 (corresponding to actual temperature fluctuation ±1.5°C). For chiller load rate, the target safe operating range is 30%-90%, target value =0.6 (after normalization), allowable deviation =0.2 (corresponding to actual load rate fluctuation ±20%).

[0060] The reward value R_stability for the stability objective is calculated based on the fluctuation range of key operating parameters reflected by the state vector, and additional penalties are imposed on actions involving changes in equipment start-up and shutdown states in the action space; among which, key operating parameters include system PUE and chilled water temperature;

[0061] Stability target rewards are used to mitigate system fluctuations and reduce equipment wear and tear, with a focus on fluctuations in PUE and supply water temperature, as well as equipment start-up and shutdown. In one example,

[0062]

[0063] in, The fluctuation penalty coefficient is a preset normal value used to penalize fluctuations in key parameters. In this embodiment, it is set to 5, which is sufficient to suppress normalized fluctuations in PUE and water supply temperature. For example, if both PUE and water supply temperature fluctuate by one historical standard deviation (i.e., both are 1), the penalty is -10, which is enough to offset the positive rewards of several safe operations and effectively encourage stability. This represents the difference in PUE between the current decision-making period and the previous period. This represents the historical standard deviation of PUE changes, calculated based on data from the past week. The difference in chilled water supply temperature between the current decision-making cycle and the previous cycle has been normalized to dimensionless (e.g., normalized by dividing by the design temperature range). The historical standard deviation of chilled water supply temperature variation is calculated based on data from the past week. This is the equipment action penalty coefficient, a preset normal value specifically used to penalize equipment start-up and shutdown operations. In this embodiment, it is set to 15. This value is set so that the penalty for a single equipment start-up and shutdown operation is higher than the short-term fluctuation penalty, but lower than the safety violation penalty. For example, the penalty for a single fan start-up and shutdown (-15) is equivalent to a penalty of 3 standard deviations of PUE fluctuation, which can effectively reduce unnecessary frequent equipment start-ups and shutdowns and extend equipment life. This is an indicator function. If the action involves a change in the start / stop status of the chiller, water pump, or fan, the value is 1; otherwise, it is 0.

[0064] The reward value of each objective is multiplied by its corresponding weight and then summed to output the total reward value R_total, which is used for updating the network parameters of the Double Q learning algorithm.

[0065] Based on the embodiments provided in this application, by establishing a dynamic correlation mechanism between weights and key operating parameters, the reward function can autonomously perceive real-time changes in the operating conditions of the HVAC system. When the IT load increases, the optimization focus automatically shifts towards safety objectives to prevent overload operation of critical equipment such as chillers; when the outdoor temperature is suitable, the focus shifts to energy efficiency optimization, guiding the system to make full use of natural cooling sources. This design, which transforms operational experience into the algorithm's internal logic, enables the agent to adaptively balance competitive objectives according to actual operational needs, significantly improving the operating condition adaptability and engineering practicality of the control strategy.

[0066] Furthermore, the state vector is used as input, and the Double Q learning algorithm is used for iterative training until a fully trained control model is obtained, including:

[0067] Initialize the experience replay pool, which is used to store experience tuples. Each experience tuple includes the current state, the action performed, the reward obtained, and the next state to enter after performing the action.

[0068] The experience replay pool employs a storage strategy that combines first-in-first-out (FIFO) with priority:

[0069] Set the maximum capacity of the experience replay pool to 50,000 experiences.

[0070] When the number of experience points exceeds the maximum capacity, the old experience points with the lowest priority score are removed first.

[0071] At the same time, some high-quality experiences are retained to ensure the diversity and representativeness of the experience sample.

[0072] Initialize the parameters of the first Q-network and the second Q-network;

[0073] Initialize the exploration rate ε;

[0074] Repeat the following steps until the training termination condition is met:

[0075] a. Observe the system's state vector s_t at the current time t;

[0076] b. Based on the current exploration rate ε, select a random action a_t with probability ε, or select an action a_t that maximizes the output value of the first Q-network with probability 1-ε; action a_t is an action in the action space.

[0077] c. Execute action a_t on the HVAC system, wait for the system to enter the next state s_{t+1}, and calculate the immediate reward r_t based on the multi-objective reward function;

[0078] d. Store the experience tuple (s_t, a_t, r_t, s_{t+1}) into the experience replay pool;

[0079] e. Sample a preset number of experience tuples from the experience replay pool; wherein the preset number includes, but is not limited to, 32, 64, etc.

[0080] In some embodiments, sampling a predetermined number of experience tuples from the experience replay pool includes:

[0081] For each empirical tuple in the empirical replay pool, a priority score is maintained. The priority score is determined by the absolute value of the timing difference error generated when calculating the target Q value of the empirical tuple plus a small constant.

[0082] The sampling probability is calculated based on the priority score of each empirical tuple. The higher the priority score, the greater the probability that the empirical tuple will be sampled.

[0083] A predetermined number of empirical tuples are extracted from the empirical replay pool according to the sampling probability for network training.

[0084] f. For each sampled empirical tuple, randomly select to update the first Q network or the second Q network;

[0085] h. Based on the system performance reflected by the newly acquired experience tuples, update the exploration rate ε, and use the updated exploration rate ε in step b of the next iteration.

[0086] In one embodiment, updating the exploration rate ε in the adaptive ε-greedy policy includes:

[0087] The initial value of the exploration rate ε is set to 1.0, and the minimum value is set to 0.01.

[0088] After every 100 training iterations, the average reward value obtained in the most recent 100 iterations is calculated as the current performance metric.

[0089] When the difference between the current performance metric and the previous performance metric is less than 0.01, it is determined that the performance has stagnated, and the exploration rate ε is reset to 0.5.

[0090] When no performance stagnation occurs, the exploration rate ε is gradually reduced with a decay coefficient of 0.995 until it reaches a minimum value of 0.01.

[0091] Based on the embodiments provided in this application, the problem of training instability caused by strong correlation of time-series data in HVAC systems is solved by establishing a complete training process that includes experience replay and adaptive exploration rate. The use of the experience replay pool breaks the sequence dependency between data, while the exploration rate, based on a feedback adjustment mechanism of learning performance, ensures a dynamic balance between sufficient exploration and efficient utilization during the training process. This design provides a reliable guarantee for the stable and efficient training of high-quality control strategies in complex HVAC environments.

[0092] Furthermore, f, for each sampled empirical tuple, randomly select to update either the first Q-network or the second Q-network, including:

[0093] If you choose to update the first Q network, then:

[0094] The output values ​​of the second Q network for all possible actions in the next state s_{t+1} are used as inputs to the first Q network;

[0095] The first Q-network selects an action from the action space that maximizes the output value, denoted as the optimal action a'_max;

[0096] The target Q value is obtained by adding the output value of the second Q network for action a'_max in the next state s_{t+1} to the reward r_t in the experience tuple.

[0097] The parameters of the first Q-network are updated by minimizing the difference between the predicted values ​​of the first Q-network for state s_t and action a_t and the target Q-value.

[0098] If the second Q-network is chosen to be updated, its update steps are symmetrical to the steps described above, i.e., the roles are reversed: the first Q-network selects the optimal action, and the second Q-network calculates and updates the target Q-value; specifically as follows.

[0099] If you choose to update the second Q network, then:

[0100] The output values ​​of the first Q-network for all possible actions in the next state s_{t+1} are used as inputs to the second Q-network;

[0101] The second Q-network selects an action from the action space that maximizes the output value, denoted as the optimal action a''_max;

[0102] The target Q value is obtained by adding the output value of the first Q network for action a''_max in the next state s_{t+1} to the reward r_t in the experience tuple.

[0103] The parameters of the second Q-network are updated by minimizing the difference between the predicted values ​​of the second Q-network for state s_t and action a_t and the target Q-value.

[0104] In some embodiments, the Double Q-learning algorithm is used, and the target Q value is calculated by the following formula:

[0105]

[0106] in, This represents the immediate reward that the agent receives after performing action a_t in state s_t. The future reward discount factor is a constant between 0 and 1 (e.g., 0.95) used to weigh the importance of current rewards against future rewards. This represents the next state (state vector) that the environment transitions to after an action is performed. , These represent the first Q network and the second Q network, respectively. This operation means that the first Q network is responsible for selecting the action that it believes will bring the maximum Q value in the next state s_{t+1} from all possible actions, denoted as a_selected. This part indicates that the second Q-network objectively evaluates the actual Q-value of the action a_selected selected by the first network in state s_{t+1}.

[0107] The core innovation of this formula lies in the separation of responsibilities between "action selection" and "value assessment." The first Q-network nominates what it considers the best action, while the second Q-network acts as a "calm reviewer," independently evaluating the true value of the nominated action. In the next update, these roles are reversed. This alternating mechanism avoids overestimation caused by network "self-evaluation," resulting in a more conservative and realistic Q-value estimate.

[0108] Based on the embodiments provided in this application, by specifically defining the alternating update rules of the two networks in Double Q learning, the value estimation bias problem of traditional Q learning in the high-dimensional action space of HVAC systems is specifically overcome. This mechanism separates the action selection and value evaluation stages, and uses mutual verification between the two networks to suppress the estimation error of a single network. In the optimization of HVAC systems involving multi-device collaborative control, this design can produce more accurate and reliable value assessments.

[0109] In one embodiment, S105, the currently acquired state vector is input into the trained control model, and the optimal action is output to generate control commands for the HVAC system, including:

[0110] After model training is complete, the system enters the deployment and application phase, where its core task is to transform the real-time acquired system state into executable control commands. First, the system preprocesses the acquired real-time operational data, including normalization and discretization, to generate a state vector identical in format to that used in the training phase. This state vector is then input into the trained control model. The model, through internal calculations, quickly makes decisions from a predefined action space and outputs an optimal action index. Next, based on this action index, the system queries a pre-stored action-command mapping table, decoding it into specific equipment control parameters, such as chiller load setpoints, pump frequency, or operating mode commands. Finally, before issuing the command, the system performs a final safety check, comparing the decoded control parameters with an independent safety rule base. If the check passes, the command is issued to the field equipment for execution via an industrial communication protocol; if the check fails, a preset safety backup strategy is activated and an alarm is recorded.

[0111] Furthermore, before step b, selecting action a_t based on the current exploration rate ε, the method also includes an online dynamic filtering step:

[0112] Based on the current state vector s_t, invalid actions that do not meet the safety and efficiency constraints are filtered out from the action space according to the predefined equipment physical operation rules and action cost library, forming a subset of candidate actions; wherein, in step b, the selection range of action a_t is limited to the subset of candidate actions;

[0113] The action cost library defines a base cost for each action based on the potential damage to the device's lifespan. When calculating the total reward value R_total in the multi-objective reward function, the base cost is added to the reward values ​​of other objectives as an independent, fixed negative reward item.

[0114] It's important to explain that the action cost library assigns a static base cost to each action, reflecting the potential wear and tear on equipment lifespan or the additional economic costs incurred from performing that action. This cost is set based on the equipment's characteristics and maintenance data. For example, the cost of a "equipment start-stop action" is set significantly higher than the cost of a "frequency regulation action." This is because frequent start-stop cycles of large equipment such as chillers and water pumps generate enormous starting currents, impacting the power grid and the equipment itself, significantly shortening its mechanical lifespan and increasing maintenance costs. In contrast, smoothly adjusting motor speed via a frequency converter is a gentler and more energy-efficient control method. The quantitative values ​​in the cost library can be estimated based on start-stop cycles, energy consumption data, and maintenance costs provided by the equipment manufacturer.

[0115] The action filtering mechanism and the multi-objective reward function constitute a collaborative two-layer optimization system. Action filtering, as a "hard constraint," directly filters out physically infeasible or high-risk options before decision-making, ensuring an absolute safety baseline for exploration. The action cost library, as a "soft constraint," influences the learning process through the reward function. It guides the agent to prioritize actions with lower equipment wear and better long-term economic benefits among all safe and feasible actions. For example, even if starting a chiller and adjusting the water pump frequency can both meet the cooling needs, the agent will be more inclined to learn to meet the needs by adjusting the frequency due to the higher start-up and shutdown costs. This further optimizes the system's economy and equipment lifespan while ensuring safety. Even if an action is filtered and cannot be executed, it is still defined in the action cost library. When the agent selects an action with a basic cost from the candidate set and executes it, this cost takes effect. It guides the agent to prioritize the "better" (lower cost) action among all "good" actions.

[0116] Based on the embodiments provided in this application, a domain-knowledge-based filtering and evaluation mechanism is implemented before the action decision-making stage, constructing a safety barrier for the application of reinforcement learning in HVAC systems. By transforming the physical operating constraints of equipment into real-time filtering rules in the action space, it is ensured that all candidate actions meet the basic requirements of safety and efficiency. Simultaneously, by defining basic cost terms related to equipment wear and tear, long-term operational economic considerations are explicitly incorporated into the optimization objective, guaranteeing the safety, stability, and equipment lifespan of the system from the decision-making source.

[0117] Furthermore, the predefined physical operating rules for the equipment include:

[0118] When the IT load value in the state vector is lower than the first load threshold for filtering, actions in the action space involving cold start-up to high load level are prohibited.

[0119] When the outdoor temperature value in the state vector is higher than the filter temperature threshold, actions set to pure free cooling operation mode in the action space are prohibited.

[0120] When the chiller load value in the state vector is lower than the second load threshold of the filter, actions involving increasing the water pump frequency beyond the preset level in the action space are prohibited.

[0121] Among them, the first load threshold, the filter temperature threshold, and the second load threshold are all obtained based on the historical operating data of the HVAC system.

[0122] It should be understood that the first load threshold, the filter temperature threshold, and the second load threshold are set based on the specific equipment performance curve and historical operating data analysis.

[0123] The following is an example of values ​​based on a typical data center scenario:

[0124] The first load threshold for filtering (for high-load start-up of chillers) is set to 30% of the system's rated total cooling load. When the IT load is below this threshold, it is prohibited to start any chiller at high load levels such as 75% or 100%. This threshold is determined based on the centrifugal chiller performance curve; when operating below 30% load, the chiller enters the inefficient zone and approaches the surge boundary.

[0125] The filtration temperature threshold (for pure free cooling mode) is set to 15°C for the outdoor wet-bulb temperature. When the outdoor wet-bulb temperature exceeds this value, the "pure free cooling mode" operation in the operating space is disabled. This threshold is calculated based on the cooling tower's heat exchange capacity and indoor design parameters. When the outdoor wet-bulb temperature exceeds 15°C, pure free cooling alone is insufficient to reduce the chilled water temperature to the required range.

[0126] The second load threshold for filtering (for pump frequency increase) is set to 40% of the design flow rate of a single chiller. When the chiller's load rate is below this threshold during operation, it is prohibited to increase the frequency of the associated chilled water pump to a higher level. This threshold is designed to prevent excessive pressure differential on the valves and cavitation caused by excessive pump head under low flow demand.

[0127] In some embodiments, safety constraints refer to physical boundaries that, if violated, would directly lead to immediate equipment damage or system malfunction. For example, a centrifugal chiller operating at a high compression ratio under low load conditions may experience periodic airflow oscillations, known as "surge," which is a safety constraint violation that must be eliminated. Efficiency constraints, on the other hand, refer to operating areas that, while not immediately causing failure, lead to a significant decrease in energy efficiency or accelerated equipment lifespan. For instance, forcing a purely free cooling mode when the outdoor wet-bulb temperature is too high will result in insufficient heat exchange driving force, requiring the cooling tower fan to operate at overclocked frequency without achieving the expected cooling effect, leading to energy waste; this is an efficiency constraint violation.

[0128] It should be noted that each physical operation rule for the equipment stems from a deep understanding of the characteristics and operating patterns of HVAC system equipment. For example, the rule "When the IT load value in the state vector is lower than the first load threshold of the filter, actions involving starting the chiller to a high load level are prohibited in the action space" is based on the principle of preventing "surge" in centrifugal chiller units. Surge is a severe vibration and noise caused by the inability of refrigerant gas to be properly compressed when the chiller is running at high load under low load conditions, which can seriously damage the equipment. By setting this rule, this dangerous operating condition is fundamentally avoided.

[0129] For example, the rule "When the outdoor temperature value in the state vector is higher than the filter temperature threshold, actions set to pure free cooling operation mode in the action space are prohibited" is based on the engineering principle that in high-temperature and high-humidity environments, the enthalpy of outdoor air is too high to effectively remove heat from the computer room. Forcing pure free cooling will not only fail to cool the system but may also cause it to overheat. These rules encode the experience of human experts into executable logic, defining a safe exploration range for decision-making.

[0130] Based on the embodiments provided in this application, precise and reliable technical basis is provided for action filtering by specifically defining equipment operation rules based on historical data statistics. Setting a first load threshold for filtering to prevent cold-run surge under low load conditions and setting a filtering temperature threshold to avoid ineffective free cooling under high-temperature conditions, these rule designs targeting typical operating problems make the action filtering mechanism both conform to the physical characteristics of the equipment and closely match actual operating needs, effectively improving the operational reliability of the system.

[0131] Furthermore, before iterative training using the Double Q learning algorithm, a phased training process is executed to initialize the parameters of the first Q network and the second Q network.

[0132] like Figure 2 As shown, the phased training process includes:

[0133] S201, perform cluster analysis on the collected historical operating data of the HVAC system, and identify and classify at least two typical operating conditions based on environmental parameters and load characteristics;

[0134] S202, create a set of corresponding Double Q learning network models for typical operating conditions, denoted as the first operating condition expert model, the second operating condition expert model, ..., the Mth operating condition expert model, where each operating condition expert model includes its own first Q network and second Q network.

[0135] S203, Use historical data segments corresponding to the first typical operating condition to train the expert model for the first operating condition until it converges;

[0136] S204, use the network parameters of the first working condition expert model as the initialization parameters of the second working condition expert model;

[0137] S205, using historical data segments corresponding to the second typical operating condition, train the second operating condition expert model that has completed parameter initialization until it converges.

[0138] It should be noted that the phased training mechanism of this invention simulates the human learning process of "starting with the easy and then moving to the difficult". By organizing training data and processes in a structured way, it solves the problem that a single model is difficult to achieve optimal performance under complex and variable working conditions.

[0139] In some embodiments, to identify typical operating conditions, the present invention uses the K-Means clustering algorithm to analyze historical operating data. The feature vectors used for clustering mainly include key parameters that characterize the system's external environment and internal load, such as outdoor temperature, outdoor humidity, and IT load rate.

[0140] The number of clusters (i.e., the number of operating conditions) is not arbitrarily determined, but rather evaluated using objective indicators such as the silhouette coefficient method to select the number that yields the best clustering results. The analysis results usually naturally divide the operating conditions into several clusters with clear physical meanings, such as "high-load summer operating conditions," "low-load winter operating conditions," and "spring and autumn transitional season operating conditions," which proves the consistency between data-driven division and physical laws.

[0141] Based on the embodiments provided in this application, a phased training process based on operating condition clustering is established to decompose the complex year-round operation optimization task into structured training stages. This scheme first identifies the typical operating modes of the system. For example, the spring and autumn transitional operating conditions with moderate load and suitable temperature are used as the basic training stage, while the summer cooling operating conditions with higher load and the winter operating conditions with different load characteristics are used as advanced training stages. By first training the basic operating condition expert model and then transferring its network parameters to the complex operating condition model as the initialization starting point, subsequent training can be built upon existing knowledge. This phased and progressive training strategy significantly improves the training efficiency of control strategies under complex operating conditions, while ensuring professional control performance can be obtained under each seasonal operating mode.

[0142] Furthermore, the first typical operating condition is when both the system load rate and the outdoor temperature are within the 25th to 75th percentile range of their historical data.

[0143] The second typical operating condition is when at least one of the system load rate and the outdoor temperature is in the 80% to 100% quantile range of its historical data.

[0144] It should be explained that the use of quantiles to define operating conditions is based on statistical analysis of historical data. The "25%-75% quantile range" is defined as the steady-state operating condition because this range covers the most frequently operating states of the system, with moderate load and temperature. The system dynamics are smooth, the control strategy is relatively simple, and it is an ideal starting point for learning.

[0145] The "80%-100% quantile range" is defined as an extreme operating condition because this range represents the tens to hundreds of hours of operation the system may encounter each year, close to its design limits. Handling these conditions requires more refined and robust strategies. This division ensures the scientific rigor and progressive nature of the training courses.

[0146] Based on the embodiments provided in this application, statistical methods are used to provide precise quantitative standards for the division of operating conditions, ensuring the scientific nature and repeatability of phased training. The normal operating conditions are clearly defined within the 25%-75% quantile range, accurately corresponding to the operating characteristics of transitional seasons such as spring and autumn, when both system load and outdoor temperature are within a moderate range.

[0147] Limiting extreme operating conditions to the 80%-100% quantile range accurately describes high-load operating conditions in winter and summer. This method of segmentation based on statistical analysis of historical operating data ensures that the datasets for each training phase truly reflect the operating characteristics of different seasons, providing a reliable data foundation for developing specialized operating condition control models.

[0148] This embodiment describes a complete training process in detail to address the core pain point that "a single model is difficult to take into account the vastly different working conditions throughout the year".

[0149] Operating condition identification and classification:

[0150] Three key parameters—outdoor dry-bulb temperature, outdoor relative humidity, and IT load rate—were selected to form the feature vector for cluster analysis.

[0151] The K-Means algorithm was applied to historical data from the past three years (one data point per hour). Based on silhouette coefficient analysis, the optimal number of clusters was determined to be 3.

[0152] Cluster A (Steady-state operating condition): Characterized by outdoor temperatures between 10°C and 22°C and IT load rates between 40% and 70%. This cluster corresponds to the "spring and autumn transitional operating condition," where the system operates most smoothly.

[0153] Cluster B (High Load Condition): Characterized by outdoor temperatures above 28°C or IT load rates above 80%. This cluster corresponds to "Summer High Load Conditions," which place the most stringent demands on the cooling system.

[0154] Cluster C (Low Load Condition): Characterized by outdoor temperatures below 5°C and IT load rate below 50%. This cluster corresponds to "winter low load conditions" and has significant free cooling potential.

[0155] Phased expert training:

[0156] Phase 1 (Expert Training): First, using all historical data fragments corresponding to cluster A (spring and autumn transitional operating conditions), the first Double Q-learning agent, i.e., the first operating condition expert model, is trained. Because the system dynamics are smooth and there are few interfering factors under this operating condition, the agent can quickly and stably learn a set of safe and efficient basic control strategies, such as how to coordinate the operation of the chiller and the cooling tower.

[0157] Phase Two (Transfer and Consolidation): After obtaining a stable first expert model, training a second expert model for the second operating condition (summer high-load condition) begins. A key step is to copy all network parameters from the first expert model as the initial parameters for the second expert model. Subsequently, the second expert model is trained primarily using historical data from cluster B. At this point, the network is not learning from scratch; it already possesses the basic control concepts learned in Phase One. The training focus shifts to adjusting strategies to adapt to extreme environments with high temperatures and high loads, such as learning to utilize the ice storage cooling system more aggressively while ensuring safety.

[0158] This phased training method does not use all historical data for training, but instead mimics the learning path of human experts from easy to difficult scenarios. By first becoming an expert in a stable and simple operating condition, a solid foundation in control is established, and then this knowledge is transferred to more complex scenarios for adaptive refinement. In practice, this method has proven to converge faster, and the resulting expert models for each operating condition significantly outperform the single general-purpose model that attempts to perform well across all conditions within their respective areas of expertise. This effectively solves the classic problem of balancing model generalization and specialized performance in complex system control.

[0159] Furthermore, in the phased training process, when training the expert model for the second working condition, ..., the expert model for the Mth working condition, a progressive strategy transfer method is adopted; it should be understood that M is a positive integer greater than 2.

[0160] Progressive policy transfer methods include:

[0161] During the first N rounds of training, the parameters in the first K layers of the network architecture inherited from the expert model of the previous scenario are frozen, and only the parameters of the remaining layers are updated.

[0162] Starting from the N+1th round, unfreeze all previously frozen network layer parameters and set the global learning rate to a fixed value lower than the initial learning rate, and continue training;

[0163] Where N and K are preset positive integers.

[0164] It's important to note that a gradual approach is crucial when transferring knowledge from one operating condition expert model to another. The settings of parameters N (number of freeze epochs) and K (number of frozen layers) have a clear engineering logic. Typically, K is set to half or two-thirds of the total number of network layers, aiming to freeze the lower-level networks responsible for extracting general features (such as temperature and pressure variation patterns), updating only the higher-level networks responsible for making decisions specific to the operating condition. This protects the learned general knowledge from being destroyed. Parameter N can be set to one-third of the total number of training epochs. After this stage, all network layers are unfrozen, and the learning rate is reduced to one-tenth of the initial value for full-network fine-tuning. This strategy of first fixing the foundation and then fully fine-tuning effectively prevents catastrophic forgetting and ensures the stable accumulation of new knowledge based on old knowledge.

[0165] This phased training and transfer learning approach has significant advantages over directly training a general model by mixing data from all different operating conditions. Direct mixing training can lead to interference between control strategies under different operating conditions, making it difficult for the model to converge or resulting in mediocre performance across various conditions. Our method, however, first allows the model to become an "expert" under relatively simple steady-state conditions, establishing a solid control foundation. Then, these core control concepts are transferred to more complex extreme operating conditions for adaptive adjustments. Ultimately, the system obtains a set of multiple "expert models" that perform exceptionally well within their respective operating condition ranges, thus achieving optimal and stable overall performance across the entire operating range.

[0166] Based on the embodiments provided in this application, a smooth transition of knowledge transfer across operating conditions is achieved by controlling the update range and intensity of network parameters in stages. In the early stages of training, the parameters of the basic network layers are frozen to protect the learned general control features. In the later stages of training, comprehensive fine-tuning is used to achieve precise adaptation to new operating conditions. This staged and controllable parameter update strategy effectively avoids the catastrophic forgetting problem in transfer learning, ensuring that the new model can rapidly adapt to seasonal changes while robustly maintaining its core control capabilities.

[0167] According to another aspect of the embodiments of this application, a multi-objective optimization system for HVAC systems based on reinforcement learning is also provided. For example... Figure 3 As shown, the system includes:

[0168] The status acquisition module 301 is used to construct a state space based on the acquired environmental parameters, equipment status parameters and historical operating data of the HVAC system; the state space consists of state vectors generated after normalization and discretization processing;

[0169] Action set construction module 302 is used to construct the action space based on the controllable equipment of the HVAC system; the action space is generated by discretizing the operating mode and control parameters of the controllable equipment.

[0170] The reward function construction module 303 is used to construct a multi-objective reward function to evaluate the control effect based on the state vector and action space;

[0171] Training module 304 is used to take the state vector as input and perform iterative training using the Double Q learning algorithm until the trained control model is obtained.

[0172] In each iteration, the state-action value function is estimated by the first Q network and the second Q network respectively. By alternately selecting one of the first Q network and the second Q network to perform action selection, the other to perform target Q value calculation, and updating the parameters of the network that performs action selection based on the multi-objective reward function; wherein, during the iterative training process, an adaptive ε-greedy strategy is used to select actions from the action space.

[0173] The output module 305 is used to input the currently acquired state vector into the trained control model and output the optimal action to generate control commands for the HVAC system.

[0174] refer to Figure 3 The reinforcement learning-based multi-objective optimization method for HVAC systems in this application relates to the field of multi-objective optimization of HVAC systems.

[0175] It should be noted that the embodiments implemented by the reinforcement learning-based HVAC system multi-objective optimization system side in this application can be referenced with the embodiments implemented by the reinforcement learning-based HVAC system multi-objective optimization method side, and will not be described in detail here.

[0176] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described reinforcement learning-based multi-objective optimization method for HVAC systems is also provided. This electronic device may be... Figure 4 The terminal device or server shown. This embodiment uses this electronic device as an example of a server. Figure 4 As shown, the electronic device includes a memory 402, a processor 404, and a transmission device 406. The memory 402 stores a computer program, and the processor 404 is configured to execute the steps in any of the above method embodiments through the computer program.

[0177] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0178] Optionally, the transmission device 406 is used to receive or send data via a network. Specific examples of the network described above may include wired and wireless networks. In one example, the transmission device 406 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 406 is a Radio Frequency (RF) module used to communicate with the Internet wirelessly. Furthermore, the electronic device also includes a display 408 and a connection bus 410, which connects the various module components within the electronic device.

[0179] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for multi-objective optimization of a heating and ventilation system based on reinforcement learning, characterized in that, The method comprises the following steps: constructing a state space based on collected environmental parameters, equipment state parameters and historical operation data of the heating and ventilation system; the state space is composed of state vectors generated after normalization and discretization processing; constructing an action space based on controllable equipment of the heating and ventilation system; the action space is generated by discretization processing of operation modes and control parameters of the controllable equipment; constructing a multi-objective reward function to evaluate the control effect based on the state vector and the action space; using the state vector as input, iteratively training by using a Double Q learning algorithm until a trained control model is obtained; wherein, in each iteration, the state-action value function is estimated by the first Q network and the second Q network respectively, the action selection is performed by alternately selecting one of the first Q network and the second Q network, the other performs target Q value calculation, and the parameters of the network performing action selection are updated based on the multi-objective reward function; wherein, in the process of iterative training, an adaptive ε-greedy strategy is used to select actions from the action space; inputting the current collected state vector into the trained control model to output the optimal action, so as to generate control instructions for the heating and ventilation system; the method of using the state vector as input, iteratively training by using a Double Q learning algorithm until a trained control model is obtained, comprises: initializing an experience replay pool, which is used to store experience tuples, the experience tuples including the current state, the executed action, the obtained reward and the next state after executing the action; initializing the parameters of the first Q network and the parameters of the second Q network; initializing the exploration rate ε; repeating the following steps until the training termination condition is met: a, observing the state vector s_t of the system at the current time t; b, based on the current exploration rate ε, selecting a random action a_t with a probability of ε, or selecting an action a_t that maximizes the output value of the first Q network with a probability of 1-ε; the action a_t is an action in the action space; c, executing the action a_t on the heating and ventilation system, waiting for the system to enter the next state s_{t+1}, and calculating the immediate reward r_t based on the multi-objective reward function; d, storing the experience tuple (s_t, a_t, r_t, s_{t+1}) in the experience replay pool; e, sampling a preset number of experience tuples from the experience replay pool; f, for each sampled experience tuple, randomly selecting to update the first Q network or the second Q network; h. updating the exploration rate ε according to the system performance reflected by the newly obtained experience tuple, and using the updated exploration rate ε for step b of the next iteration; the f, for each sampled experience tuple, randomly selecting to update the first Q network or the second Q network, comprises: if the first Q network is selected to be updated: input the output values of the second Q network for all possible actions of the next state s_{t+1} into the first Q network; selecting, by the first Q-network, an action from the action space that maximizes the output value, denoted as the optimal action a'_max; adding, by the second Q-network, the output value of the action a'_max at the next state s_{t+1} to the reward r_t in the experience tuple to obtain a target Q value; updating parameters of the first Q-network by minimizing the difference between the predicted value of the first Q-network for the state s_t and the action a_t and the target Q value; if the second Q-network is selected to be updated, then: providing, as input, the output values of the first Q-network for all possible actions at the next state s_{t+1} to the second Q-network; selecting, by the second Q-network, an action from the action space that maximizes the output value, denoted as the optimal action a''_max; adding, by the second Q-network, the output value of the action a''_max at the next state s_{t+1} to the reward r_t in the experience tuple to obtain a target Q value; updating parameters of the second Q-network by minimizing the difference between the predicted value of the second Q-network for the state s_t and the action a_t and the target Q value.

2. The reinforcement learning based multi-objective optimization method for heating, ventilation, and air conditioning systems according to claim 1, wherein, The multi-objective reward function f (s, a) integrates the energy saving target, the safety target, the stability target and the economic target; wherein the economic target is calculated based on the amount of running cost saved; the multi-objective reward function f (s, a) is constructed by: allocating initial weights to the energy saving target, the safety target, the stability target and the economic target, respectively; adjusting the energy saving target weight W_energy and the safety target weight W_safety according to the IT load value and the outdoor temperature value in the state vector; wherein when the IT load value is higher than a first load threshold, W_safety is increased and W_energy is decreased; when the outdoor temperature value is lower than a first temperature threshold, W_energy is increased and W_safety is decreased; the stability target weight W_stability and the economic target weight W_economic remain fixed; calculating the reward value R_energy of the energy saving target according to the improvement degree of the PUE value of the heating and ventilation system after the action is executed relative to the historical benchmark level; calculating the reward value R_safety of the safety target by using a piecewise function, including: when the operating parameters of the heating and ventilation system reflected by the state vector are within the safety boundary, R_safety is a fixed positive number; when there is an operating parameter exceeding the safety boundary, R_safety is a negative number with an absolute value greater than the fixed positive number; wherein the operating parameters include the chilled water temperature, the cooling water temperature and the chiller load; calculating the reward value R_stability of the stability target according to the fluctuation amplitude of the key operating parameters reflected by the state vector, and imposing an additional penalty on the actions in the action space that involve changes in the device start-stop state; wherein the key operating parameters include the system PUE and the chilled water temperature. The reward values of each target are multiplied by their corresponding weights and summed to output a total reward value R_total, which is used for network parameter updating of the Double Q learning algorithm.

3. The reinforcement learning based multi-objective optimization method for heating, ventilation, and air conditioning systems of claim 1, wherein, Before b, selecting an action a_t according to the current exploration rate ε, the method further comprises: Based on the current state vector s_t, according to the predefined device physical operation rules and the action cost library, filtering out invalid actions that do not meet the safety constraints and efficiency constraints from the action space to form a candidate action subset; In step b, the selection range of action a_t is limited to the candidate action subset; Wherein, the action cost library defines a basic cost for each action based on the potential damage of the action to the device life; when calculating the total reward value R_total of the multi-objective reward function, the basic cost is added as an independent and fixed negative reward item together with other target reward values.

4. The method of claim 3, wherein, The predefined device physical operation rules include: When the IT load value in the state vector is lower than the filtering first load threshold, actions in the action space involving cold machine startup to high load gear are prohibited; When the outdoor temperature value in the state vector is higher than the filtering temperature threshold, actions in the action space set to pure free cooling operation mode are prohibited; When the cold machine load value in the state vector is lower than the filtering second load threshold, actions in the action space involving water pump frequency up-regulation beyond a preset gear are prohibited; Wherein, the filtering first load threshold, the filtering temperature threshold and the filtering second load threshold are all obtained based on historical operation data statistics of the HVAC system.

5. The reinforcement learning based multi-objective optimization method for heating, ventilation, and air conditioning systems of claim 1, wherein, Before the iterative training using the Double Q learning algorithm, a phased training process is performed to initialize the parameters of the first Q network and the second Q network; The phased training process includes: Performing cluster analysis on the collected historical operation data of the HVAC system, and identifying and dividing at least two typical operating conditions according to environmental parameters and load characteristics; Creating a set of corresponding Double Q learning network models for the typical operating conditions, denoted as first condition expert model, second condition expert model, …, Mth condition expert model, wherein each condition expert model includes its own first Q network and second Q network; Using the historical data segment corresponding to the first typical operating condition to train the first condition expert model until it converges; Using the network parameters of the first condition expert model as the initialization parameters of the second condition expert model; Using the historical data segment corresponding to the second typical operating condition to train the second condition expert model which has completed parameter initialization until it converges.

6. The multi-objective optimization method for HVAC systems based on reinforcement learning according to claim 5, wherein The first typical operating condition is a condition in which both the system load rate and the outdoor temperature are in the 25% to 75% quantile interval of its historical data; The second typical operating condition is a condition in which at least one of the system load rate and the outdoor temperature is in the 80% to 100% quantile interval of its historical data.

7. The method of claim 5 or 6, wherein, In the staged training process, the gradual strategy migration method is used when training the second working condition expert model, the Mth working condition expert model, and the like. The gradual strategy migration method comprises: During the first N rounds of training, the parameters of the first K layers of the network architecture in the parameters inherited from the previous working condition expert model are frozen, and only the parameters of the remaining layers are updated; Starting from the N+1th round, all the previously frozen network layer parameters are unfrozen, the global learning rate is set to a fixed value lower than the initial learning rate, and the training is continued; wherein N and K are preset positive integers.

8. A reinforcement learning based multi-objective optimization system for HVAC system, the system implements the reinforcement learning based multi-objective optimization method for HVAC system as claimed in claim 1, characterized in that, Comprise: A state acquisition module configured to construct a state space based on acquired environmental parameters, equipment state parameters, and historical operation data of the HVAC system; The state space is composed of state vectors generated after normalization and discretization processing; An action set construction module configured to construct an action space based on controllable equipment of the HVAC system; The action space is generated by discretization processing of the operating modes and control parameters of the controllable equipment; A reward function construction module configured to construct a multi-objective reward function to evaluate the control effect based on the state vector and the action space; A training module configured to use the Double Q learning algorithm to iteratively train the state vector as input until a trained control model is obtained; wherein in each iteration, the state-action value function is estimated by the first Q network and the second Q network respectively, the action selection is performed by alternately selecting one of the first Q network and the second Q network, the target Q value calculation is performed by the other, and the parameters of the action selection network are updated based on the multi-objective reward function; wherein in the iterative training process, the adaptive ε-greedy strategy is used to select actions from the action space; An output module configured to input the currently acquired state vector into the trained control model and output the optimal action to generate control instructions for the HVAC system; The Double Q learning algorithm is used to iteratively train the state vector as input until a trained control model is obtained, comprising: Initialize an experience replay pool, which is used to store experience tuples, the experience tuples including the current state, the executed action, the obtained reward, and the next state entered after executing the action; Initialize the parameters of the first Q network and the parameters of the second Q network; Initialize the exploration rate ε; Repeat the following steps until the training termination condition is met: a. Observe the state vector s_t of the system at the current time t; b. Based on the current exploration rate ε, select a random action a_t with a probability of ε, or select an action a_t that maximizes the output value of the first Q network with a probability of 1-ε; the action a_t is an action in the action space; c. Execute the action a_t on the HVAC system, wait for the system to enter the next state s_{t+1}, and calculate the immediate reward r_t based on the multi-objective reward function; d. Store the experience tuple (s_t, a_t, r_t, s_{t+1}) in the experience replay pool; e. sampling a preset number of experience tuples from the experience replay pool; f. for each sampled experience tuple, randomly selecting to update the first Q network or the second Q network; h. updating the exploration rate ε according to the system performance reflected by the newly acquired experience tuple, and using the updated exploration rate ε for step b of the next iteration; The f, for each sampled experience tuple, randomly selecting to update the first Q network or the second Q network, comprises: If the first Q network is selected to be updated: providing the output values of the second Q network for all possible actions under the next state s_{t+1} as input to the first Q network; selecting an action from the action space by the first Q network that maximizes the output value, denoted as the optimal action a'_max; adding the output value of the second Q network for the action a'_max under the next state s_{t+1} to the reward r_t in the experience tuple to obtain a target Q value; updating the parameters of the first Q network by minimizing the difference between the predicted value of the first Q network for the state s_t and the action a_t and the target Q value; If the second Q network is selected to be updated: providing the output values of the first Q network for all possible actions under the next state s_{t+1} as input to the second Q network; selecting an action from the action space by the second Q network that maximizes the output value, denoted as the optimal action a''_max; adding the output value of the first Q network for the action a''_max under the next state s_{t+1} to the reward r_t in the experience tuple to obtain a target Q value; updating the parameters of the second Q network by minimizing the difference between the predicted value of the second Q network for the state s_t and the action a_t and the target Q value.

Images