Air conditioning system control method and system based on reinforcement learning and twinborn model strategy

Abstract

The invention discloses an air conditioning system control method and system based on reinforcement learning and a twinborn model strategy. The method comprises the steps that S1, a data-driven twinborn model of a central air conditioning system is constructed; s2, performing offline pre-training based on a virtual environment formed by a twin model; s3, performing online integration on a plurality of pre-trained reinforcement learning algorithms and the twin model to form a reinforcement learning strategy integration framework; and S4, the final control action is issued to execution equipment of the central air conditioning system. According to the invention, by constructing the serial data-driven twinborn model, the running state, energy consumption and cooling and heating load supply-demand relationship of the central air-conditioning system are accurately simulated, a high-fidelity virtual environment is provided for various reinforcement learning algorithms, offline pre-training is realized, an initial control strategy is generated, and risks caused by blind exploration in a real system are avoided. Continuous self-adaptive optimization is achieved, and the stability and the energy efficiency level of intelligent control of the air conditioning system are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent building energy-saving control technology, specifically to an air conditioning system control method and system based on reinforcement learning and twin model strategies. Background Technology

[0002] As building energy consumption accounts for an increasingly larger share of total social energy consumption, central air conditioning systems, as major energy-consuming equipment in public buildings and large commercial facilities, directly impact overall energy utilization efficiency. Traditional air conditioning system control relies heavily on rule-based PID control or fixed setpoint adjustment, which struggles to adapt to complex and changing environmental conditions and dynamic load demands, resulting in low energy efficiency and significant energy waste. In recent years, the development of intelligent control technology has provided new solutions for improving the operating efficiency of air conditioning systems. In particular, the integration of data-driven methods and advanced control algorithms is driving the evolution of HVAC systems towards intelligence and precision. Patent CN109695943B discloses a temperature and humidity control system for coating air conditioners based on big data deep learning. By utilizing big data deep learning technology and multiple sets of temperature and humidity sensors and a DBN model to optimize air conditioning control, it solves the problems of traditional methods relying on experience and long debugging cycles.

[0003] Digital twin technology, as a crucial means of enabling real-time interaction between physical systems and virtual models, has been widely applied in fields such as industrial manufacturing and energy management. By constructing high-precision system simulation models, digital twins can reproduce equipment operating states in virtual space, supporting fault prediction, performance evaluation, and control strategy optimization. In the HVAC field, data-driven twin models can effectively capture the coupling relationships between various system components, providing a reliable simulation foundation for the dynamic control of complex systems, significantly reducing trial-and-error costs and improving decision reliability.

[0004] Reinforcement learning, a machine learning method that does not require precise mathematical models and can learn optimal strategies through trial and error, has shown great potential in the field of intelligent control. By continuously interacting with the environment and optimizing the decision-making process based on reward signals, it is suitable for nonlinear and time-varying control systems. Applying reinforcement learning to air conditioning system control allows for the autonomous exploration of energy-saving operation strategies while meeting comfort constraints, achieving optimal long-term energy efficiency and demonstrating strong adaptability and promising engineering applications.

[0005] However, most existing reinforcement learning control methods directly explore and learn on real systems, resulting in problems such as long training cycles, susceptibility to unsafe control actions, and high initial energy consumption. Furthermore, single algorithms struggle to simultaneously address the control requirements of both discrete and continuous action spaces and lack effective safety verification mechanisms, leading to insufficient stability of the control strategy. In addition, most solutions do not incorporate high-fidelity simulation models for pre-training and fail to form a complete closed-loop architecture from simulation to execution and from decision-making to feedback, limiting the reliable deployment and widespread adoption of intelligent control technology in practical engineering scenarios. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an air conditioning system control method and system based on reinforcement learning and twin model strategy, so as to overcome the problems of high risk and poor adaptability of single algorithm in the prior art when reinforcement learning is directly explored in real system, and improve the stability of control.

[0007] The objective of this invention is achieved through the following technical solution: A control method for an air conditioning system based on reinforcement learning and twin model strategies, comprising: S1. Construct a data-driven twin model of the central air conditioning system. The twin model is formed by connecting multiple subsystem models trained based on historical operating data. It is used to simulate the operating status, energy consumption, and supply and demand relationship of the central air conditioning system under a given control action. S2. Based on the virtual environment constructed by the twin model, multiple reinforcement learning algorithms with different action space processing capabilities are pre-trained offline to generate an initial control strategy; the multiple reinforcement learning algorithms include a first algorithm and a second algorithm for processing discrete action spaces and a third algorithm for processing continuous action spaces. S3. The pre-trained multiple reinforcement learning algorithms are integrated online with the Siamese model to form a reinforcement learning strategy integration framework. In each control cycle, the framework performs the following: collecting real-time system status data, obtaining candidate control actions recommended by each algorithm and model in parallel, performing double safety checks on the candidate values, and selecting the final control action from the candidate values ​​that pass the checks. S4. Send the final control action to the execution device of the central air conditioning system, collect actual operation feedback data, and use this feedback data to update the strategies of each reinforcement learning algorithm online to achieve adaptive learning; S5. The reinforcement learning strategy integration framework is encapsulated in an integrated control cabinet, and direct data communication and control execution with the field equipment of the air conditioning system are realized through the control cabinet.

[0008] By constructing a data-driven twin model composed of multiple subsystem models linked together, a virtual simulation environment consistent with the dynamic characteristics of a real central air conditioning system is established. Based on this, historical operating data is used to pre-train various reinforcement learning algorithms with different action space processing capabilities offline, enabling them to learn preliminary control strategies without interfering with the actual system operation. In the online operation phase, the pre-trained algorithms and the twin model are integrated into a unified framework. System status is collected in real time during each control cycle, and multiple candidate values ​​for control actions are generated in parallel. Subsequently, a dual safety detection mechanism is used to verify the feasibility of the candidate values ​​from two dimensions: indoor comfort and system energy consumption-supply-demand balance, eliminating high-risk actions. In the safe candidate set, the system energy consumption corresponding to each action is further predicted based on the twin model, and the action with the lowest energy consumption is selected as the final control command for execution. The system continuously collects actual feedback data to update the algorithm strategies online, achieving closed-loop adaptive optimization. The entire control logic is integrated and deployed through an integrated control cabinet, completing the fully automated operation from perception and decision-making to execution.

[0009] As a preferred embodiment, in step S1, the subsystem model includes at least an air source heat pump energy consumption model, a water pump flow model, a water pump energy consumption model, an air conditioning terminal energy consumption model, and an air conditioning terminal heat exchange model; the construction of the twin model is specifically as follows: based on preprocessed historical data, each subsystem model is independently trained using a deep neural network, and then the models are connected in series according to the physical connection and energy flow relationship of the central air conditioning system to form a digital simulation system that can reflect the overall operating characteristics of the system.

[0010] As a preferred embodiment, the first algorithm is a Q-learning algorithm for processing discrete action spaces.

[0011] As a preferred approach, the second algorithm is a Deep Q-Network (DQN) algorithm suitable for discrete action spaces.

[0012] As a preferred approach, the third algorithm is the flexible actor-critic (SAC) algorithm for processing continuous action spaces.

[0013] As a preferred embodiment, the offline pre-training step in S2 includes: S201. Model the variable frequency control problem of the central air conditioning water pump as a Markov decision process, with the state space... ,in Outdoor dry-bulb temperature, For building cooling load, For chilled water supply temperature; operating space pump frequency Reward function Based on total system energy consumption It also includes penalties for excessive terminal energy consumption and excessive deviation between cooling capacity and load; S202. Input historical state data into each reinforcement learning algorithm, each algorithm outputs an action and the twin model calculates the next state and reward to form simulated experience data for policy update. S203. By monitoring the convergence of the cumulative reward curves of each algorithm, the pre-training is determined to be complete.

[0014] As a preferred embodiment, the "dual security detection" in S3 specifically includes: S301, Indoor temperature constraint detection: Input each control action candidate value into a pre-trained indoor temperature prediction model. If the predicted indoor temperature exceeds the comfort range, the candidate value is rejected. S302. System supply and demand balance and energy consumption constraint detection: For candidate values ​​that pass S301, the twin model is used to predict the end-point energy consumption after performing the action. and the absolute deviation between the system's cooling capacity and the building's cooling load. ,like or , , If the preset threshold is used, the candidate value will be rejected.

[0015] As a preferred embodiment, in step S302, the twin model is used to predict the system's cooling capacity. At this time, a temporal convolutional network model based on an attention mechanism is used, called the ATCN model. The single-step prediction process of this model is represented as follows: ; in, The input is historical time series data, with dimensions of [batch size, time step, number of features], and the features include water supply temperature, return water temperature, total flow rate, and outdoor dry-bulb temperature; The output is the predicted cooling capacity, with dimensions [batch size, 1]. The ATCN model first performs dilated causal convolution on the input sequence to extract features, and then calculates the weights at different time steps through a temporal attention module. The calculation process is as follows: ; Where IntOut is the intermediate feature tensor after dilated causal convolution, Conv1D is a one-dimensional convolution with a kernel size of 2, and Sigmoid is the activation function. The calculated temporal attention weights are then weighted and summed to obtain the output. .

[0016] As a preferred method, the specific method of "selecting the best final control action" in S3 is as follows: for all candidate control action values ​​that pass the dual safety detection, the overall system energy consumption simulation value generated by executing each candidate value in the current system state is calculated using the twin model, and the candidate value with the smallest energy consumption simulation value is selected as the final control action.

[0017] As a preferred approach, in step S4, when "updating the strategies of each reinforcement learning algorithm online", each algorithm is allowed to explore within a safe framework; wherein, the first algorithm (Q-learning) and the second algorithm (DQN) adopt an ε-greedy exploration strategy, randomly selecting actions with probability ε; the third algorithm (SAC) explores based on the maximum entropy principle.

[0018] As a preferred approach, the objective function of the third algorithm (SAC) is to maximize the weighted sum of the expected cumulative reward and the policy entropy. Represented as: ; in, This indicates a strategy that needs to be optimized. This represents the total number of time steps in a training epoch; Indicates about the strategy The resulting state-action trajectory distribution Seeking expectations; Indicates at time step The state; Indicates at time step The action; Indicates the state Execute action The instant reward obtained afterward; The temperature coefficient is an adjustable hyperparameter greater than 0, used to balance the importance of reward and entropy. Representation strategy In state Entropy is used to measure the randomness of a strategy.

[0019] An air conditioning system control system based on reinforcement learning and twin model strategies, the system comprising: The twin model construction and maintenance module is used to build, train, and update the data-driven twin model. A multi-algorithm reinforcement learning pre-training module is used to train the multiple reinforcement learning algorithms offline in the virtual environment provided by the twin model to generate an initial control policy. The online policy integration and decision-making module is used to deploy and run the reinforcement learning policy integration framework, and to perform real-time data acquisition, multi-policy action candidate generation, dual security detection, and final control action selection and issuance. The online adaptive learning module is used to update the policy network parameters of each reinforcement learning algorithm online based on feedback data collected from a real air conditioning system. The integrated hardware execution and communication module, namely the integrated control cabinet, is used to integrate the computing, communication and control hardware of the system and provide a physical interface with field devices.

[0020] The present invention has at least the following beneficial effects: This invention constructs a serial data-driven twin model to accurately simulate the operating status, energy consumption, and supply and demand relationship of a central air conditioning system. It provides a high-fidelity virtual environment for various reinforcement learning algorithms, enabling offline pre-training and generating initial control strategies, thus avoiding the risks of blindly exploring in real systems. During online operation, multiple algorithms handling different action spaces are integrated with the twin model to form a strategy integration framework. Candidate values ​​for control actions are generated in parallel, and safe actions are screened through a dual safety detection mechanism. The action with the lowest energy consumption is selected as the final instruction, ensuring both control safety and energy efficiency. The algorithm strategies are updated online based on actual feedback data, achieving continuous adaptive optimization and significantly improving the stability and energy efficiency of the intelligent control of the air conditioning system. Attached Figure Description

[0021] To reveal the technical details of the embodiments of the present invention, the accompanying drawings involved in the embodiments will be briefly described below. It should be emphasized that these drawings only present several embodiments of the present invention and should not be considered as defining the scope of the invention. For those skilled in the art, other related drawings can still be derived based on these drawings without inventive effort.

[0022] Figure 1 This is a schematic diagram of a control method for an air conditioning system based on reinforcement learning and twin model strategies. Detailed Implementation

[0023] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the scope of protection of the present invention is not limited to the following description.

[0024] In the following description, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. However, it should be understood that the present disclosure is not limited to the specific forms shown herein. Rather, it should be understood to encompass various variations, equivalents, and / or alternatives to the embodiments of the present disclosure. In illustrating the drawings, the same reference numerals will be used to denote similar components.

[0025] In this disclosure, terminology is used to describe specific embodiments and does not constitute a limitation thereof. In this context, the use of the singular form also encompasses the plural form, unless otherwise expressly stated herein. In the course of description, terms such as “comprising” or “having” are intended to indicate the presence of features, quantities, steps, operations, structural components, parts, or combinations thereof, and do not preclude the possibility or addition of one or more other features, quantities, steps, operations, structural components, parts, or combinations thereof.

[0026] It should be clarified that while the following description provides detailed specific information to aid in a comprehensive understanding of the exemplary embodiments, those skilled in the art will recognize that the exemplary embodiments can be implemented even without these specific details. For example, the system may be illustrated using block diagrams to avoid excessive detail that could obscure the clarity of the example. In other cases, to maintain the clarity of the example, unnecessary details of well-known processes, structures, and techniques may be omitted.

[0027] like Figure 1 As shown, an air conditioning system control method based on reinforcement learning and twin model strategy includes: S1. Construct a data-driven twin model of the central air conditioning system. The twin model is formed by connecting multiple subsystem models trained based on historical operating data. It is used to simulate the operating status, energy consumption, and supply and demand relationship of the central air conditioning system under a given control action. S2. Based on the virtual environment constructed by the twin model, multiple reinforcement learning algorithms with different action space processing capabilities are pre-trained offline to generate an initial control strategy; the multiple reinforcement learning algorithms include a first algorithm and a second algorithm for processing discrete action spaces and a third algorithm for processing continuous action spaces. S3. The pre-trained multiple reinforcement learning algorithms are integrated online with the Siamese model to form a reinforcement learning strategy integration framework. In each control cycle, the framework performs the following: collecting real-time system status data, obtaining candidate control actions recommended by each algorithm and model in parallel, performing double safety checks on the candidate values, and selecting the final control action from the candidate values ​​that pass the checks. S4. Send the final control action to the execution device of the central air conditioning system, collect actual operation feedback data, and use this feedback data to update the strategies of each reinforcement learning algorithm online to achieve adaptive learning; S5. The reinforcement learning strategy integration framework is encapsulated in an integrated control cabinet, and direct data communication and control execution with the field equipment of the air conditioning system are realized through the control cabinet.

[0028] This embodiment first uses historical data to train subsystem models of key components such as air-source heat pumps, water pumps, and terminals based on the physical structure and operating mechanism of the central air conditioning system. These models are then connected in series according to energy transfer relationships to construct a data-driven twin model that accurately reflects the overall dynamic characteristics of the system. This twin model serves as a virtual simulation environment for offline pre-training of various reinforcement learning algorithms. This allows algorithms handling discrete actions (such as start-stop control), continuous states (such as temperature prediction), and continuous actions (such as frequency regulation) to accumulate control experience and form initial strategies under risk-free conditions. During online operation, each pre-trained algorithm and the twin model are integrated into a real-time decision-making framework. In each control cycle, the algorithms synchronously receive the system status, output control action suggestions, and generate multiple candidate solutions in parallel. These candidate actions undergo a dual safety check: first, the indoor temperature prediction model determines whether comfort requirements are met; second, the twin model evaluates the energy consumption level and the matching degree of cooling and heating supply and demand after execution, eliminating substandard options. Among the remaining safe actions, the corresponding total system energy consumption is further simulated, and the action with the lowest energy consumption is selected as the optimal action for execution. The system continuously collects feedback from actual operation to dynamically update the parameters of each algorithm strategy, enabling continuous learning and performance optimization. The entire control process is integrated into a single control cabinet, achieving end-to-end automated closed-loop operation from data acquisition and intelligent decision-making to command execution.

[0029] By constructing a high-fidelity data-driven twin model, a safe and efficient virtual training environment is provided for reinforcement learning, avoiding the operational risks and energy losses caused by blind exploration in real systems. An integrated strategy of multi-algorithm collaboration is adopted, taking into account the adaptability to different types of control tasks and improving the robustness and generalization performance of the strategy. By introducing a dual safety detection mechanism, dangerous actions that may lead to excessive comfort or system imbalance are eliminated during the decision-making stage, significantly enhancing the reliability of intelligent control. Combined with an optimization mechanism based on energy consumption simulation, the selected actions are ensured to achieve optimal energy efficiency while meeting safety constraints. Simultaneously, continuous strategy updates through online feedback enable the system to adapt to changes in operating conditions. The overall solution achieves a deep integration of simulation pre-training and online optimization, forming a complete closed loop of learning, decision-making, execution, and feedback, effectively improving the intelligence level, operational stability, and energy efficiency of the central air conditioning system.

[0030] In a preferred embodiment, in step S1, the subsystem model includes at least an air source heat pump energy consumption model, a water pump flow model, a water pump energy consumption model, an air conditioning terminal energy consumption model, and an air conditioning terminal heat exchange model; the construction of the twin model is specifically as follows: based on preprocessed historical data, each subsystem model is independently trained using a deep neural network, and then the models are connected in series according to the physical connection and energy flow relationship of the central air conditioning system to form a digital simulation system that can reflect the overall operating characteristics of the system.

[0031] In this invention, historical data accumulated over long-term operation of a central air conditioning system is first used. After cleaning and preprocessing, key variables are extracted for modeling. For different functional components within the system, separate sub-models are established, including the energy consumption characteristics of the air source heat pump, the flow regulation law of the water pump, the energy consumption changes of the water pump itself, the energy consumption behavior of the air conditioning terminals, and the terminal heat exchange capacity. These sub-models are independently trained using deep neural networks, enabling them to accurately learn the input-output relationships of their respective components under different operating conditions. Since a central air conditioning system is essentially an energy transfer system composed of multiple devices connected along specific physical paths, after the training of each sub-model, these models are sequentially linked together according to the actual system's pipe connection sequence and energy flow direction to form a complete digital simulation chain. When a control action is given (such as adjusting the water pump frequency), the digital system can progressively deduce its impact on subsequent stages, such as how flow rate changes affect supply and return water temperatures, thereby affecting terminal cooling capacity and overall energy consumption levels. This achieves high-precision simulation of the real system's operating state, energy consumption performance, and cooling / heating supply and demand relationship, providing a reliable virtual experimental environment for subsequent intelligent control.

[0032] In a preferred embodiment, the first algorithm is a Q-learning algorithm for processing discrete action spaces; The second algorithm is a Deep Q-Network (DQN) algorithm suitable for high-dimensional state inputs and discrete action spaces. Although both the first algorithm (Q-learning) and the second algorithm (DQN) are oriented towards discrete action spaces, they are applicable to different state scenarios: Q-learning is suitable for simple control tasks with discretizable states and low dimensionality; DQN uses deep neural networks to process high-dimensional, continuous state inputs and is suitable for intelligent decision-making under complex working conditions. The two complement each other.

[0033] The third algorithm is the Flexible Actor-Critic (SAC) algorithm for processing continuous motion spaces.

[0034] This embodiment employs three different types of reinforcement learning algorithms working together to address the diverse control needs of air conditioning systems. The first algorithm, Q-learning, is primarily used for discrete control tasks such as equipment start-up and shutdown, and operating mode switching. It accumulates experience through trial and error, selecting the optimal option from several preset actions to achieve long-term energy savings. The second algorithm uses a Deep Q-Network (DQN), also oriented towards discrete action spaces. However, compared to traditional Q-learning, DQN leverages the powerful nonlinear fitting capabilities of deep neural networks to more effectively handle high-dimensional and complex state information, such as the system operating state composed of data from multiple sensors, thus making more accurate decisions. It is suitable for scenarios involving judging and responding to the overall system operating condition. The third algorithm uses the Flexible Actor-Critics (SAC) algorithm, specifically designed for handling continuous control problems requiring fine-tuning, such as the smooth adjustment of pump or fan frequencies. This algorithm does not rely on enumerating actions but directly outputs a specific control value. By maximizing the joint objective of reward and policy entropy, it strikes a balance between exploring new strategies and utilizing known optimal strategies, enabling it to continuously seek better energy-saving paths while ensuring system stability. After being pre-trained in a twin model, these three algorithms are integrated into an online decision-making framework, achieving comprehensive coverage of different types of control commands and complementary advantages.

[0035] In a preferred embodiment, the offline pre-training step in S2 includes: S201. Model the variable frequency control problem of the central air conditioning water pump as a Markov decision process, with the state space... ,in Outdoor dry-bulb temperature, For building cooling load, For chilled water supply temperature; operating space pump frequency Reward function Based on total system energy consumption It also includes penalties for excessive terminal energy consumption and excessive deviation between cooling capacity and load; S202. Input historical state data into each reinforcement learning algorithm, each algorithm outputs an action and the twin model calculates the next state and reward to form simulated experience data for policy update. S203. By monitoring the convergence of the cumulative reward curves of each algorithm, the pre-training is determined to be complete.

[0036] In a preferred embodiment, the "dual security detection" in S3 specifically includes: S301, Indoor temperature constraint detection: Input each control action candidate value into a pre-trained indoor temperature prediction model. If the predicted indoor temperature exceeds the comfort range, the candidate value is rejected. S302. System supply and demand balance and energy consumption constraint detection: For candidate values ​​that pass S301, the twin model is used to predict the end-point energy consumption after performing the action. and the absolute deviation between the system's cooling capacity and the building's cooling load. ,like or , , If a preset threshold is set, then the candidate value is rejected. Specifically, It is the safe upper limit threshold for terminal energy consumption, used to prevent abnormal increases in energy consumption caused by excessive operation of air conditioning terminal equipment (such as fan coil units, air handling units, etc.). It is the allowable deviation threshold for the matching accuracy of cooling and heating supply and demand, used to measure the degree of matching between the system's cooling capacity and the actual building cooling load.

[0037] In this embodiment, offline pre-training is a crucial step to ensure that reinforcement learning algorithms can be effectively applied to the control of actual air conditioning systems. First, we model the variable frequency control problem of the central air conditioning system's water pumps as a Markov Decision Process (MDP). The state space includes key parameters such as outdoor dry-bulb temperature, building cooling load, and chilled water supply temperature, while the action space is defined as the pump frequency adjustment value. The reward function is designed based on the total system energy consumption, with additional penalties for excessive terminal energy consumption and excessive deviation between cooling capacity and load, to encourage the algorithm to explore energy-saving and supply-demand balanced control strategies. During the pre-training phase, historical operating data is used as input. Each reinforcement learning algorithm outputs corresponding control actions based on the current state. The Siamese model simulates the system state changes and corresponding reward values ​​after executing these actions, forming a series of simulated empirical data for continuously updating and optimizing the algorithm's strategy. By continuously monitoring the trend of the cumulative rewards of each algorithm, when it stabilizes, it indicates that the algorithm has learned an effective control strategy, and pre-training is complete.

[0038] Once in the online operation phase, to ensure the safety and effectiveness of the selected control actions, this invention introduces a dual safety detection mechanism. The first step is indoor temperature constraint detection: candidate control actions are input into a pre-trained indoor temperature prediction model to assess whether executing the action will cause the indoor temperature to exceed the comfort range. If it does, the action is directly rejected. Next is system supply-demand balance and energy consumption constraint detection: for actions that pass the first round of screening, a twin model is used to predict the terminal energy consumption after execution and the matching between the system's cooling capacity and the building's cooling load. If the prediction results show excessively high terminal energy consumption or a large deviation between the cooling capacity and the cooling load, the action is also rejected. Only after passing these two rigorous safety checks can a feasible control command be issued to the air conditioning system for execution, thereby ensuring that the system achieves efficient and energy-saving operation while meeting comfort requirements. This dual-layer filtering mechanism not only improves the safety of control decisions but also enhances system stability.

[0039] In a preferred embodiment, in step S302, the twin model is used to predict the system's cooling capacity. At this time, a temporal convolutional network model based on an attention mechanism is used, called the ATCN model. The single-step prediction process of this model is represented as follows: ; in, The input is historical time series data, with dimensions of [batch size, time step, number of features], and the features include water supply temperature, return water temperature, total flow rate, and outdoor dry-bulb temperature; The output is the predicted cooling capacity, with dimensions [batch size, 1]. The ATCN model first performs dilated causal convolution on the input sequence to extract features, and then calculates the weights at different time steps through a temporal attention module. The calculation process is as follows: ; Where IntOut is the intermediate feature tensor after dilated causal convolution, Conv1D is a one-dimensional convolution with a kernel size of 2, and Sigmoid is the activation function. The calculated temporal attention weights are then weighted and summed to obtain the output. .

[0040] In the Temporal Convolutional Network (ATCN) model based on attention mechanisms, temporal attention weights The calculation of this is a core step in enabling the model to focus on key historical information. The specific process is as follows: First, the input historical time series data... After dilated causal convolution, an intermediate feature tensor IntOut containing high-level features is obtained. Subsequently, IntOut is processed by a one-dimensional convolutional layer (Conv1D) with a kernel size of 2 to generate inactive importance scores, which are then compressed to the (0, 1) interval using a sigmoid activation function to finally obtain the temporal attention weights. Its calculation formula is: The weight vector Each element Each time step corresponds to an importance score. Finally, The weighted sum is calculated with the intermediate feature IntOut, i.e. This allows for the fusion of information from all time steps, highlighting the contributions of key moments, to obtain the final output used to predict the system's cooling capacity. .

[0041] To more accurately predict the cooling capacity after an air conditioning system executes a specific control action, a Temporal Convolutional Network (ATCN) model with an incorporating attention mechanism is employed to improve the response accuracy to dynamic load changes. This model uses historical data of key parameters such as supply water temperature, return water temperature, total flow rate, and outdoor dry-bulb temperature as input, capturing the complex relationships between multiple variables changing over time during system operation. First, the model extracts local features from the time series layer by layer by expanding the causal convolutional structure, effectively expanding the receptive field without relying on future information, thereby capturing operating trends over a longer time range. The intermediate features obtained after convolution processing are fed into the temporal attention module, which automatically evaluates the impact of data from different historical moments on the current cooling capacity. For example, temperature differences or flow rate fluctuations at certain times may have a stronger indicative effect on the current cooling capacity, and the attention mechanism assigns higher weights to these key time points. Finally, the model comprehensively judges based on the weighted feature information and outputs an accurate predicted cooling capacity value. This design enables the system not only to identify stable operating patterns but also to respond sensitively to sudden load changes, providing a highly reliable predictive basis for subsequent safety verification and optimal decision-making.

[0042] To further improve the physical fidelity and predictive consistency of the digital twin model during long-term operation, this invention introduces a dynamic correction mechanism based on the law of energy conservation at the output of the twin model. A central air conditioning system should satisfy a basic energy balance relationship under steady-state or quasi-steady-state conditions, meaning that the total system energy consumption consists of the energy consumption of the chiller and the energy consumption of auxiliary equipment, and the chiller energy consumption and cooling capacity are correlated through the coefficient of performance (COP). Therefore, the correction formula for the total system energy consumption is defined as follows: ; in, This indicates the total energy consumption of the corrected system, expressed in kilowatts (kW). This is the system cooling capacity predicted by the ATCN model, in kilowatts (kW); CO2 The system's overall performance coefficient is defined as follows: ,in Energy consumption (kW) of the refrigeration unit predicted by the air source heat pump energy consumption model; The energy consumption of auxiliary equipment, including water pumps and fans, is predicted in kilowatts (kW) and is obtained by summing the outputs of the water pump energy consumption model and the end-point energy consumption model.

[0043] In each control cycle, if the total system energy consumption directly predicted by the twin model... With the above physical correction values The relative deviation exceeds the preset threshold (For example, 5%), then the system will automatically adopt The original predicted values ​​were used to replace the actual values ​​in subsequent energy consumption simulations and safety verifications. This ensured that the twin model not only possessed high data fitting capabilities but also strictly adhered to fundamental thermodynamic laws, significantly improving its long-term simulation reliability and control safety under complex operating conditions.

[0044] In a preferred embodiment, the specific method of "selecting the best final control action" in S3 is as follows: for all candidate control action values ​​that pass the dual safety detection, the simulated value of the overall system energy consumption generated by executing each candidate value in the current system state is calculated using the twin model, and the candidate value with the smallest simulated energy consumption value is selected as the final control action.

[0045] After completing dual safety checks, the system has selected a batch of candidate control actions that meet indoor comfort requirements, ensure a balance between heating and cooling supply and demand, and avoid abnormal energy consumption. Next, to further improve energy efficiency, the system needs to select the optimal action from these "safe actions." To this end, a twin model-based energy consumption simulation and evaluation mechanism is adopted to conduct a detailed analysis of each candidate action that passes the checks. Specifically, based on the current actual operating state of the system, a pre-constructed high-precision digital twin model is used to simulate the operation of the entire air conditioning system after executing each candidate action, and to predict the corresponding overall system energy consumption. Because the twin model can accurately reflect the energy consumption coupling relationship between various devices, the simulation results have high reliability. Among all simulation results, the action with the lowest predicted energy consumption is selected as the final control command for execution. This strategy of "safety filtering first, then energy-saving optimization" avoids operational hazards caused by high-risk controls while ensuring optimal energy efficiency within a safe range, achieving a balance between safety and economy.

[0046] To further ensure the smooth execution of control commands, extend the service life of key equipment such as water pumps and frequency converters, and avoid system oscillations or motor overloads caused by drastic fluctuations in control actions, this invention introduces a control action change rate constraint model before the final action is selected. Let the current control period be... The pump frequency executed in the previous cycle was (Unit: Hz), the frequency of candidate action suggestions is (Unit: Hz), then its rate of change Defined as: ; in, This indicates the rate of change of the pump frequency, expressed in Hertz per minute (Hz / min). This indicates the pump frequency setting value corresponding to the current candidate control action, in Hertz (Hz). This indicates the actual pump frequency executed in the previous control cycle, in Hertz (Hz). Indicates the duration of the control cycle, in minutes (min), for example... min.

[0047] The system sets the maximum allowable rate of change. (e.g., 2 Hz / min). When selecting the best candidate from the set of safe candidates, all those that meet the requirements are first eliminated. The candidate actions are selected. If the remaining set is not empty, the action with the smallest simulated energy consumption is selected as the final control action; if none of the candidate actions meet the smoothness requirement, the system initiates a smooth transition strategy and generates the final instruction according to the following formula: ; in, This indicates the final smoothed pump frequency, expressed in Hertz (Hz). Indicates the frequency corresponding to the candidate action with the lowest energy consumption; sign Represents a symbolic function; This indicates the maximum permissible rate of change, expressed in Hz / min. This indicates the control cycle, in minutes.

[0048] This embodiment ensures that the control actions gradually approach the optimal operating conditions in a "ramp-like" manner, which significantly improves the system's operational stability and equipment reliability while ensuring energy-saving effects.

[0049] In a preferred embodiment, during S4, when "updating the strategies of each reinforcement learning algorithm online", each algorithm is allowed to explore within a safe framework; wherein, the first algorithm (Q-learning) and the second algorithm (DQN) adopt an ε-greedy exploration strategy, randomly selecting actions with probability ε; the third algorithm (SAC) explores based on the maximum entropy principle.

[0050] During actual system operation, to adapt the control strategy to dynamic factors such as environmental changes and equipment aging, the strategies of each reinforcement learning algorithm are updated online using collected real-world operational feedback data after each control action, thereby continuously optimizing control performance. During the update process, the algorithm does not simply adopt the currently considered optimal action; instead, it allows for moderate exploration, under safe conditions, to discover potentially better control strategies. For value function-based algorithms like Q-learning and DQN, an ε-greedy strategy is used for exploration. This means that most of the time, the currently optimal action is chosen to ensure control stability, but other actions are randomly selected with a small probability, thus offering opportunities to try new control methods and avoiding getting trapped in local optima. For the SAC algorithm, a more advanced maximum entropy exploration mechanism is employed. This mechanism not only encourages the algorithm to find high-reward actions but also actively tends to maintain policy diversity, that is, maintaining a certain degree of randomness among multiple possible actions, thereby more comprehensively perceiving the system's response characteristics. This balance between exploration and utilization is confined within a "safety framework" constructed by dual safety checks, ensuring that even if new actions are attempted, it will not lead to system instability or a sudden increase in energy consumption. In this way, the system can operate stably and continuously learn and adapt, thereby constantly improving its long-term control performance.

[0051] In a preferred embodiment, the policy optimization objective function of the third algorithm (SAC) is to maximize the weighted sum of the expected cumulative reward and the policy entropy. Represented as: ; in, This indicates a strategy that needs to be optimized. This represents the total number of time steps in a training epoch; Indicates about the strategy The resulting state-action trajectory distribution Seeking expectations; Indicates at time step The state; Indicates at time step The action; Indicates the state Execute action The immediate reward obtained is a numerical value, which is calculated by the reward function based on factors such as system energy efficiency and comfort. The temperature coefficient is an adjustable hyperparameter greater than 0, used to balance the importance of reward and entropy. Representation strategy In state Entropy is used to measure the randomness of a strategy.

[0052] The Soft Actor-Critic (SAC) algorithm aims not only to achieve high rewards but also to emphasize the exploratory power and stability of control strategies. During the learning process, it pursues two goals simultaneously: first, maximizing the cumulative benefits of system operation, such as reducing energy consumption and maintaining comfort; and second, preserving the diversity of the strategies themselves, that is, retaining a certain degree of randomness in decision-making to avoid premature reliance on a single action pattern. This mechanism of "balancing performance and exploration" is called maximum entropy reinforcement learning. Specifically, when evaluating the effectiveness of a control strategy, the algorithm considers both its immediate energy-saving effect and the richness of its behavior in the current state—the more diverse the behavior, the higher the entropy value, and the more helpful it is in discovering potentially better control methods. To balance these two aspects, an adjustable weight coefficient is introduced. This is used to control the weight between "trying new actions" and "selecting known good actions." In this way, the SAC algorithm can continuously explore new energy-saving potential while ensuring stable system operation. It is especially suitable for scenarios with slowly changing operating conditions, such as air conditioning systems, enabling long-term adaptive optimization, avoiding strategy rigidity, and improving the robustness and energy efficiency of overall intelligent control.

[0053] In a preferred embodiment, in step S5, the integrated control cabinet has a built-in industrial computer, a PLC communication module, and a switch; the industrial computer carries and runs the core program of the reinforcement learning strategy integration framework; the control cabinet communicates with the upper-level PLC via the Modbus TCP protocol, and connects to field sensors, meters, and frequency converters via the PLC via the Modbus RTU protocol, forming a complete data acquisition and control system.

[0054] To achieve the stable deployment of intelligent control strategies in practical engineering, we integrate the entire reinforcement learning and twin model decision-making system into an integrated control cabinet, forming an independent and reliable edge computing unit. The control cabinet integrates core components such as industrial computers, PLC communication modules, and switches. Among them, the industrial computer serves as the operation center, responsible for running all core programs including data processing, multi-algorithm decision-making, security verification, and policy update, realizing a complete logical closed-loop from perception to decision-making. The control cabinet communicates with the existing upper-layer PLC on-site through the Modbus TCP protocol, obtaining the operation status data of the air-conditioning system in real time, such as temperature, flow rate, equipment start-stop signals, etc.; at the same time, control instructions are also sent to the PLC through the same channel, and the PLC uses the Modbus RTU protocol to connect various underlying devices, such as sensors, electric meters, and frequency converters, to complete the precise regulation of actuator parameters such as pump frequency and valve opening. This hierarchical communication architecture not only compatible with the existing industrial control system without large-scale modification of the original equipment, but also ensures that intelligent algorithms can be directly connected to the real system and achieve rapid response. The entire control cabinet has good environmental adaptability and anti-interference ability, can be directly installed in the air-conditioning machine room, realizing the seamless integration of the algorithm model and the physical system, and improving the integration, reliability, and engineering popularization of the system.

[0055] In a preferred embodiment, the method further includes a software layer fault tolerance and data processing mechanism配套 with the integrated control cabinet, which includes: S501. Data parsing and conversion: Concatenate high and low bytes, perform type conversion and magnification conversion on the hexadecimal data read from the PLC to obtain the real physical quantity value; when writing control instructions, convert the decimal set value into hexadecimal format; S502. Fault and exception handling: Read the fault bits of each data point in the PLC in real time. If the fault bit is set, skip the current control cycle; perform rationality verification on the read value. If the relative error between the current value and the average value of the previous three historical values exceeds 90%, it is determined as an abnormal value, and the historical average value is used for replacement; S503. System special condition handling: When detecting special situations such as the system just starting, the host not running, or data not updated for a long time, suspend the algorithm control and maintain the system safe operation mode.

[0056] It should be noted that there is an inaccuracy in the original text. "配套" is not the correct English expression here. It should be something like "matched with" or "associated with". I have made the correction in the translation of item .To ensure the long-term stable operation of the intelligent control system in complex industrial environments, a comprehensive software-level fault tolerance and data processing mechanism is designed within the integrated control cabinet. First, the raw data read from the PLC is parsed and converted. Since industrial communication often transmits data in hexadecimal format, the system automatically performs high and low byte concatenation, data type conversion, and rate conversion, restoring the raw code values ​​to actual physical quantities such as temperature, flow rate, and power. Simultaneously, when issuing control commands, the decimal setpoints are accurately converted to hexadecimal format recognizable by the PLC, ensuring accurate data exchange. During operation, the system monitors the fault flags of each data point in real time. If an equipment alarm or communication anomaly is detected, the current control cycle is proactively skipped to avoid misjudgments based on erroneous information. For numerical data, the system also performs rationality checks. If a parameter's sudden change exceeds 90% of its recent historical average, it is considered abnormal or abrupt, and the historical average is automatically used to replace it, preventing sensor malfunctions or interference signals from misleading control decisions. Furthermore, in special operating conditions such as system startup, host shutdown, or prolonged data inactivity, the intelligent algorithm will automatically pause its intervention and switch to a safe operating mode, with the PLC maintaining basic control logic to ensure the system remains under control at all times. This series of fault-tolerant mechanisms significantly improves the robustness and engineering applicability of the entire intelligent control system, enabling it to operate reliably in real-world, complex environments for extended periods.

[0057] An air conditioning system control system based on reinforcement learning and twin model strategies, the system comprising: The twin model construction and maintenance module is used to build, train, and update the data-driven twin model. A multi-algorithm reinforcement learning pre-training module is used to train the multiple reinforcement learning algorithms offline in the virtual environment provided by the twin model to generate an initial control policy. The online policy integration and decision-making module is used to deploy and run the reinforcement learning policy integration framework, and to perform real-time data acquisition, multi-policy action candidate generation, dual security detection, and final control action selection and issuance. The online adaptive learning module is used to update the policy network parameters of each reinforcement learning algorithm online based on feedback data collected from the real air conditioning system. Driven by the online adaptive learning module, the three algorithms Q-learning, DQN and SAC continuously learn and update their parameters using the operating data of the real system. This enables the decision-making ability of the entire control system to evolve over time, effectively adapting to long-term dynamic factors such as seasonal changes and equipment aging, and achieving sustained energy saving and stable operation.

[0058] The integrated hardware execution and communication module, namely the integrated control cabinet, is used to integrate the computing, communication and control hardware of the system and provide a physical interface with field devices.

[0059] The air conditioning system control system comprises five functional modules working collaboratively to form a complete closed loop from modeling and training to real-time decision-making and execution. First, the twin model construction and maintenance module, based on historical operating data, trains multiple subsystem models and connects them according to physical relationships to establish a digital twin capable of accurately simulating the operating characteristics of the air conditioning system. It continuously updates model parameters based on new data to ensure simulation accuracy. Next, the multi-algorithm reinforcement learning pre-training module uses this twin model to construct a virtual environment and performs offline training on various reinforcement learning algorithms such as Q-learning, DQN, and SAC. This allows the algorithms to grasp basic control laws without interfering with the real system and generate reliable initial strategies. In the online operation phase, the online strategy integration and decision-making module is responsible for real-time acquisition of system status, calling various pre-trained algorithms to generate candidate control actions in parallel, and using a dual safety detection mechanism to screen out safe actions that meet comfort and system balance requirements. Then, combined with the twin model's energy consumption prediction, the most energy-efficient solution is selected as the final command for issuance. Simultaneously, the online adaptive learning module uses actual operating feedback data to continuously optimize the strategy parameters of each algorithm, enabling the system to possess long-term self-learning and environmental adaptability. The above logic is carried out by an integrated hardware execution and communication module. This module uses an integrated control cabinet as a carrier to integrate industrial control computer, communication module and other hardware, and directly interconnects with field PLC and sensors, frequency converter and other equipment to realize end-to-end automated operation of data acquisition, intelligent decision and control execution. The overall system has both safety, energy saving and engineering practicality.

[0060] In a preferred embodiment, the dual security detection unit in the online policy integration and decision-making module includes: The indoor temperature prediction constraint subunit performs comfort compliance checks on candidate action values ​​based on the indoor temperature prediction model. The system twin model security verification subunit performs energy consumption and supply-demand balance feasibility checks on the candidate action values ​​based on the twin model.

[0061] In the online strategy integration and decision-making module, the dual safety detection unit is a crucial link in ensuring the reliable execution of control actions, and it is completed collaboratively by two sub-units. First is the indoor temperature prediction constraint sub-unit. This unit has a built-in, fully trained temperature prediction model that can quickly predict the indoor temperature change trend over a future period based on the current system status and candidate control actions. If the prediction shows that the temperature will exceed the set comfort range (e.g., too cold or too hot), it indicates that the action may affect occupant comfort and is immediately judged as non-compliant and eliminated. Second is the system twin model safety verification sub-unit. This unit uses a high-precision digital twin model to further evaluate the candidate actions that have passed the first hurdle, simulating their impact on the entire system. It focuses on checking whether execution will lead to an abnormal increase in terminal energy consumption or an excessive deviation between the cooling capacity and the actual cooling load of the building. If such risks exist, they are also considered unsafe actions and rejected. These two sub-units, with the former focusing on user experience and the latter on system operating efficiency and stability, together form a "comfort first, balance later" dual-layer safety defense line, ensuring that the final control commands are both safe and reliable, and meet the basic operational requirements of energy saving and comfort.

[0062] In a preferred embodiment, the integrated hardware execution and communication module includes: The industrial control computer, as the core computing unit, carries the online strategy integration and decision-making module and the online adaptive learning module; The PLC communication module, as a data acquisition and command issuance unit, interacts with the industrial control computer via the Modbus TCP protocol and connects to field devices via the Modbus RTU protocol. Switches are used to build local area networks within control cabinets; An independent air conditioning room management platform serves as a human-machine interface, providing system status monitoring, energy consumption analysis, historical data query, and remote manual control functions.

[0063] The integrated hardware execution and communication module forms the physical foundation for the entire intelligent control system, consisting of multiple collaborative hardware units. The industrial control computer (ICC), as the core computing device, is responsible for running key software modules such as online strategy integration and decision-making, and adaptive learning. It completes all intelligent computing tasks, from data processing and multi-algorithm fusion to security verification and optimal action generation, essentially acting as the system's "brain." The PLC communication module acts as the "nerve transmission" mechanism. On one hand, it communicates efficiently with the ICC via the Modbus TCP protocol, receiving control commands and uploading field data. On the other hand, it connects to underlying devices in the air conditioning system, such as sensors, meters, and frequency converters, via the Modbus RTU protocol, enabling real-time acquisition of parameters such as temperature, flow rate, and energy consumption, as well as precise control of actuators such as pumps and fans, ensuring accurate issuance and execution of intelligent decisions. An in-cabinet switch creates a stable local area network environment, ensuring high-speed and reliable data interaction between the ICC, communication module, and other components, preventing communication delays or packet loss from affecting control real-time performance. In addition, the system is equipped with an independent air conditioning room management platform, serving as a human-machine interface. Maintenance personnel can use this platform to intuitively view the system's operating status, energy consumption trends, and historical operation records, and perform remote monitoring or manual intervention when necessary, thus improving the system's transparency and operability. The entire hardware architecture is highly integrated and easy to deploy, ensuring both the efficient operation of intelligent algorithms and compatibility with existing industrial facilities, achieving a deep integration of intelligence and engineering practicality.

[0064] In a preferred embodiment, the system further includes a data preprocessing and quality assurance module, which is integrated into the industrial control computer software and is used to perform data parsing, fault diagnosis, outlier identification and correction logic before the data is used for control decisions.

[0065] The data preprocessing and quality assurance module, acting as the "data gatekeeper" for stable system operation, is integrated into the industrial control computer's software system. It performs comprehensive cleaning and verification of all data before it enters the control decision-making process. First, the module parses the raw data acquired from the PLC, correctly restoring the hexadecimal code stream in the communication protocol to actual physical quantities such as temperature, flow rate, and power, ensuring the accuracy of the data meaning. Then, the module reads the equipment status flags in real time for fault diagnosis. If an alarm signal is detected from a sensor or actuator, the module automatically marks the relevant data as unreliable, preventing the system from making incorrect judgments based on fault information. For numerical data, the module also continuously performs rationality analysis. If a parameter changes drastically, for example, if the current reading deviates from the average of several past samples by more than a set threshold, it is determined to be an abnormal jump, possibly caused by momentary sensor interference or communication errors. In this case, the system will replace it with a historical stable value to prevent malfunctions. Through this series of data analysis, diagnosis, identification and correction mechanisms, the adverse effects of common industrial site noise, packet loss, code errors and equipment failures are effectively filtered out, ensuring the accuracy and reliability of input data, providing a solid data foundation for subsequent intelligent decision-making, and significantly improving the robustness and operational safety of the entire control system.

[0066] In a preferred embodiment, the multi-algorithm reinforcement learning pre-training module supports an experience replay pool sharing mechanism, enabling the multiple reinforcement learning algorithms to learn from the same batch of historical interaction data generated by virtual environments or real systems, thereby improving data utilization efficiency and promoting policy complementarity among algorithms.

[0067] In the multi-algorithm reinforcement learning pre-training module of this invention, an experience replay pool sharing mechanism is introduced to improve learning efficiency and policy synergy. The core of this mechanism lies in establishing a unified data storage area to centrally store a large amount of interaction data accumulated by the system in the virtual environment of the twin model or during real-world operation. This data records the state changes and feedback rewards obtained after the system takes a certain control action in a specific state. Traditionally, each reinforcement learning algorithm needs to independently collect and use data for learning, which easily leads to data waste and low learning efficiency. However, by sharing the experience replay pool, multiple algorithms such as Q-learning, DQN, and SAC can jointly learn by randomly sampling from this batch of high-quality historical data, avoiding duplicate sampling and significantly improving data utilization efficiency. More importantly, since different algorithms have different learning characteristics and policy preferences, sharing the same data source allows them to learn from the same decision-making scenarios, thereby developing diverse coping strategies. This collaborative learning process not only accelerates the convergence of each algorithm but also helps to achieve policy complementarity in the subsequent online decision-making stage, improving the robustness and adaptability of the overall control system.

[0068] This invention proposes an intelligent control method and system for air conditioning systems based on the deep integration of reinforcement learning and digital twin technology. It constructs a closed-loop control architecture integrating high-precision simulation, multi-algorithm collaborative decision-making, safety constraint assurance, and online adaptive optimization. By using historical operating data of key components of the central air conditioning system to train subsystem models and connecting them in series according to physical connections to form a data-driven twin model, high-fidelity simulation of the dynamic characteristics of the real system is achieved, providing a safe and efficient virtual training environment for intelligent algorithms. Based on this, various reinforcement learning algorithms such as Q-learning, DQN, and SAC are used to handle discrete and continuous control tasks respectively. Initial strategies are accumulated through offline pre-training, and an experience replay pool sharing mechanism is introduced to improve learning efficiency and strategy diversity. During online operation, the system generates candidate actions in parallel using multiple algorithms, and combines a dual safety detection mechanism to eliminate high-risk actions from both comfort and system balance dimensions. Then, based on the energy consumption prediction of the twin model, the most energy-efficient scheme is selected for execution, achieving a balance between safety and economy. Simultaneously, the system continuously uses real feedback data for online updates and strategy evolution, possessing long-term adaptive capabilities. The entire control logic is integrated into a single control cabinet, compatible with existing industrial communication protocols, and equipped with comprehensive data preprocessing and fault-tolerance mechanisms to ensure stable and reliable operation of the system in complex field environments. This invention effectively solves the problems of low energy efficiency in traditional control methods and high risk of directly training existing reinforcement learning on real systems, significantly improving the intelligence level, operational stability, and energy-saving effect of air conditioning systems, and has good engineering applicability and promotional value.

[0069] It should be noted that the above embodiments are merely illustrative examples of the technical solutions of the present invention and are not intended to limit the present invention. Any obvious modifications, equivalent substitutions, or improvements made by those skilled in the art based on the content disclosed in this invention without departing from the core ideas and basic principles of the present invention, such as adjustments to the model structure, algorithm type, parameter settings, hardware configuration, or control flow, should be considered to fall within the protection scope of this invention. The protection scope of this invention should be determined by the appended claims, and not limited to the specific forms described in the specific embodiments.

Claims

1. A control method for an air conditioning system based on reinforcement learning and twin model strategies, characterized in that, include: S1. Construct a data-driven twin model of the central air conditioning system. The twin model is formed by connecting multiple subsystem models trained based on historical operating data. It is used to simulate the operating status, energy consumption, and supply and demand relationship of the central air conditioning system under a given control action. S2. Based on the virtual environment constructed by the twin model, multiple reinforcement learning algorithms with different action space processing capabilities are pre-trained offline to generate an initial control strategy; the multiple reinforcement learning algorithms include a first algorithm and a second algorithm for processing discrete action spaces and a third algorithm for processing continuous action spaces. S3. The pre-trained multiple reinforcement learning algorithms are integrated online with the Siamese model to form a reinforcement learning strategy integration framework. In each control cycle, the framework performs the following: collecting real-time system status data, obtaining candidate control actions recommended by each algorithm and model in parallel, performing double safety checks on the candidate values, and selecting the final control action from the candidate values ​​that pass the checks. S4. Send the final control action to the execution device of the central air conditioning system, collect actual operation feedback data, and use this feedback data to update the strategies of each reinforcement learning algorithm online to achieve adaptive learning; S5. The reinforcement learning strategy integration framework is encapsulated in an integrated control cabinet, and direct data communication and control execution with the field equipment of the air conditioning system are realized through the control cabinet.

2. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 1, characterized in that, In S1, the subsystem model includes at least an air source heat pump energy consumption model, a water pump flow model, a water pump energy consumption model, an air conditioning terminal energy consumption model, and an air conditioning terminal heat exchange model; the construction of the twin model is specifically as follows: based on preprocessed historical data, each subsystem model is independently trained using a deep neural network, and then the models are connected in series according to the physical connection and energy flow relationship of the central air conditioning system to form a digital simulation system that can reflect the overall operating characteristics of the system.

3. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 1, characterized in that, The first algorithm is a Q-learning algorithm for processing discrete action spaces; the second algorithm is a deep Q-network algorithm suitable for discrete action spaces; and the third algorithm is a flexible actor-critic algorithm for processing continuous action spaces.

4. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 1, characterized in that, In S2, the offline pre-training step includes: S201. Model the variable frequency control problem of the central air conditioning water pump as a Markov decision process, with the state space... ,in Outdoor dry-bulb temperature, For building cooling load, For chilled water supply temperature; operating space pump frequency Reward function Based on total system energy consumption It also includes penalties for excessive terminal energy consumption and excessive deviation between cooling capacity and load; S202. Input historical state data into each reinforcement learning algorithm, each algorithm outputs an action and the twin model calculates the next state and reward to form simulated experience data for policy update. S203. By monitoring the convergence of the cumulative reward curves of each algorithm, the pre-training is determined to be complete.

5. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 1, characterized in that, In S3, "dual security detection" specifically includes: S301, Indoor temperature constraint detection: Input each control action candidate value into a pre-trained indoor temperature prediction model. If the predicted indoor temperature exceeds the comfort range, the candidate value is rejected. S302. System supply and demand balance and energy consumption constraint detection: For candidate values ​​that pass S301, the twin model is used to predict the end-point energy consumption after performing the action. and the absolute deviation between the system's cooling capacity and the building's cooling load. ,like or , , If the preset threshold is used, the candidate value will be rejected.

6. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 5, characterized in that, In step S302, the twin model is used to predict the system's cooling capacity. At this time, a temporal convolutional network model based on an attention mechanism is used, called the ATCN model. The single-step prediction process of this model is represented as follows: ； in, The input is historical time series data, with dimensions of [batch size, time step, number of features], and the features include water supply temperature, return water temperature, total flow rate, and outdoor dry-bulb temperature; The output is the predicted cooling capacity, with dimensions [batch size, 1]. The ATCN model first performs dilated causal convolution on the input sequence to extract features, and then calculates the weights at different time steps through a temporal attention module. The calculation process is as follows: ； Where IntOut is the intermediate feature tensor after dilated causal convolution, Conv1D is a one-dimensional convolution with a kernel size of 2, and Sigmoid is the activation function. The calculated temporal attention weights are then weighted and summed to obtain the output. .

7. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 1, characterized in that, In S3, the specific method of "selecting the best final control action" is as follows: for all candidate control actions that pass the dual safety detection, the overall system energy consumption simulation value generated by executing each candidate value in the current system state is calculated using the twin model, and the candidate value with the smallest energy consumption simulation value is selected as the final control action.

8. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 1, characterized in that, In S4, when "updating the strategies of each reinforcement learning algorithm online", each algorithm is allowed to explore within a safe framework; wherein, the first algorithm and the second algorithm adopt an ε-greedy exploration strategy, randomly selecting actions with probability ε; the third algorithm explores based on the maximum entropy principle.

9. The air conditioning system control method based on reinforcement learning and twin model strategy according to claim 8, characterized in that, The objective function of the third algorithm is to maximize the weighted sum of the expected cumulative reward and the policy entropy. Represented as: ； in, This indicates a strategy that needs to be optimized. This represents the total number of time steps in a training epoch. Indicates about the strategy The resulting state-action trajectory distribution Seeking expectations; Indicates at time step The state; Indicates at time step The action; Indicates the state Execute action The instant reward obtained afterward; The temperature coefficient is an adjustable hyperparameter greater than 0, used to balance the importance of reward and entropy. Representation strategy In state Entropy is used to measure the randomness of a strategy.

10. An air conditioning system control system based on reinforcement learning and twin model strategy, used to implement the air conditioning system control method based on reinforcement learning and twin model strategy as described in any one of claims 1-9, characterized in that, The system includes: The twin model construction and maintenance module is used to build, train, and update the data-driven twin model. A multi-algorithm reinforcement learning pre-training module is used to train the multiple reinforcement learning algorithms offline in the virtual environment provided by the twin model to generate an initial control policy. The online policy integration and decision-making module is used to deploy and run the reinforcement learning policy integration framework, and to perform real-time data acquisition, multi-policy action candidate generation, dual security detection, and final control action selection and issuance. The online adaptive learning module is used to update the policy network parameters of each reinforcement learning algorithm online based on feedback data collected from a real air conditioning system. The integrated hardware execution and communication module, namely the integrated control cabinet, is used to integrate the computing, communication and control hardware of the system and provide a physical interface with field devices.

Images