Air conditioning control method, device, terminal and medium based on model reinforcement learning

Abstract

The present invention provides a model-based reinforcement learning-based air conditioning control method, device, terminal, and medium, relating to the field of artificial intelligence technology. The method comprises: constructing an RC model that reflects the thermal behavior of a target building as a training environment model; building a reinforcement learning agent, training the agent in a reinforcement learning simulation environment generated based on the RC model, and obtaining a Q-table with an optimal air conditioning control strategy; deploying the Q-table in the air conditioning system installed in the target building, and controlling the operation of the air conditioning system using the Q-table with the optimal air conditioning control strategy through the agent. By combining the advantages of RC models and reinforcement learning, and leveraging information from the interaction between reinforcement learning and the environment, the present invention reduces the need for historical data and can train efficient and accurate control strategies in a short period of time. This addresses the problems of traditional reinforcement learning control methods, such as high data requirements and long training times, which make it difficult to quickly and efficiently meet the practical application requirements of demand-response control.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an air conditioning control method, device, terminal and medium based on model reinforcement learning. Background Art

[0002] Nowadays, low-carbon development has become a global consensus. The construction industry is an important part of global energy use and greenhouse gas emissions. Among them, building air-conditioning systems account for a large proportion of energy consumption and have huge energy-saving potential. However, with the increase of volatile new energy and the widening of load peak-to-valley differences, the stable and safe operation of the power grid has been unprecedentedly impacted. Therefore, as an important part of the total energy consumption of buildings, the operation optimization and control of building air-conditioning systems is particularly important for reducing the energy consumption of the entire building air-conditioning system and even the total energy consumption of the building. It has a high load regulation capability and is an important resource for achieving demand response. Demand response technology (DR) provides a solution for reducing building energy consumption and alleviating grid pressure.

[0003] However, many traditional reinforcement learning control methods for building air-conditioning systems currently have shortcomings such as high data requirements and long training time, making it difficult to quickly and efficiently meet the actual application needs of demand response control. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide an air-conditioning control method, device, terminal and medium based on model reinforcement learning in response to the above-mentioned defects of the prior art, aiming to solve the problems that traditional reinforcement learning control methods have high data requirements, long training time, and are difficult to quickly and efficiently meet the actual application needs of demand response control.

[0005] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0006] An air conditioning control method based on model reinforcement learning, wherein the method comprises:

[0007] Build an RC model that reflects the thermal behavior of the target building as a training environment model;

[0008] Constructing a reinforcement learning agent and training the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy;

[0009] The Q table with the optimal air-conditioning control strategy is deployed to the air-conditioning system installed on the target building, and the operation of the air-conditioning system is controlled by the reinforcement learning agent based on the Q table with the optimal air-conditioning control strategy.

[0010] In one implementation, the RC model is composed of thermal resistance and thermal capacitance, and the expression of the RC model is:

[0011] ;

[0012] ;

[0013] ;

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] in, represents the heat transfer between indoor air and the air conditioning system, represents the heat transfer between the exterior envelope and the indoor air, represents the heat transfer between the outdoor air and the external envelope, Indicates the indoor temperature. Indicates the air conditioning system temperature. represents the temperature of the external protective structure, Indicates the outdoor temperature, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, represents the thermal resistance of the air conditioning system, represents the indoor air thermal resistance, represents the thermal resistance of the building envelope, represents the indoor heat capacity, represents the heat capacity of the building, Indicates the cooling power of the air conditioning system. Indicates indoor heat, ONOFF indicates the operating status of the compressor, t indicates time in seconds, and h indicates hours. Indicates the mapping relationship between indoor heat and hour h, Indicates the indoor heat corresponding to each hour.

[0019] In one implementation, constructing a reinforcement learning agent includes:

[0020] Determining agent basic parameters and environment parameters, and constructing a corresponding reinforcement learning agent based on the agent basic parameters and the environment parameters;

[0021] The basic parameters of the agent include state space, action space, reward function and ε-greedy strategy setting value, the ε-greedy strategy setting value is a constant between 0 and 1, and the environmental parameters include outdoor temperature, indoor temperature, air conditioning operation status, electricity price and time information;

[0022] In addition, the Q-value update formula of the reinforcement learning agent is:

[0023] ;

[0024] Wherein, s represents the state space, which includes indoor temperature, outdoor temperature and time information. represents the next state space after executing action a in state s, a represents the action space, and the action space is a binary discrete action set {0,1}, element 0 represents the air conditioner is turned off, and element 1 represents the air conditioner is turned on. Indicates that in the next state space The corresponding action space, r represents the reward function, which consists of temperature comfort penalty, electricity cost penalty, low temperature reward, high temperature reward and pre-cooling reward. represents the learning rate, Represents the discount factor.

[0025] In one implementation, the training of the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table having an optimal air conditioning control strategy includes:

[0026] Collecting historical outdoor temperature data and historical electricity price data of the target building;

[0027] The historical outdoor temperature data and the historical electricity price data are used to drive the reinforcement learning agent to interact with a reinforcement learning simulation environment generated based on the RC model, so as to train the reinforcement learning agent to select actions based on the state obtained from the RC model, receive rewards from the environment, and update the Q-table until the training converges, thereby obtaining a Q-table with an optimal air conditioning control strategy.

[0028] In one implementation, before deploying the Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed in the target building and controlling the operation of the air-conditioning system based on the Q-table with the optimal air-conditioning control strategy by the reinforcement learning agent, the method further includes:

[0029] Deploying the Q-table with the optimal air conditioning control strategy in the reinforcement learning simulation environment generated based on the RC model and having parameters consistent with the training environment to verify the air conditioning control performance of the Q-table and obtain corresponding verification results;

[0030] Optimizing the Q table based on the verification result to obtain an optimized Q table;

[0031] The step of deploying the Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed in the target building, and controlling the operation of the air-conditioning system based on the Q-table with the optimal air-conditioning control strategy by the reinforcement learning agent, includes:

[0032] The optimized Q table with the optimal air-conditioning control strategy is deployed to the air-conditioning system installed on the target building, and the operation of the air-conditioning system is controlled by the reinforcement learning agent based on the optimized Q table with the optimal air-conditioning control strategy.

[0033] In one implementation, controlling the operation of the air-conditioning system based on the Q table having the optimal air-conditioning control strategy by the reinforcement learning agent includes:

[0034] Acquire current real-time environmental data, and analyze the current real-time environmental data by the reinforcement learning agent based on the Q table with the optimal air conditioning control strategy to determine an optimal air conditioning control action, and generate an air conditioning control instruction corresponding to the optimal air conditioning control action;

[0035] The air conditioning control instruction is sent to the infrared controller based on the Modbus protocol, so as to control the operation of the air conditioning system through the infrared controller.

[0036] In one implementation, the process of controlling the operation of the air-conditioning system by the reinforcement learning agent based on the Q table having the optimal air-conditioning control strategy further includes:

[0037] A full-dimensional operation log including the current time, the current real-time environmental data, the air-conditioning control action and the energy consumption of the air-conditioning system is recorded in real time, so as to dynamically adjust the Q table based on the full-dimensional operation log.

[0038] The present invention also discloses an air conditioning control device based on model reinforcement learning, wherein the device comprises:

[0039] RC model building module, used to build an RC model that reflects the thermal behavior of the target building as a training environment model;

[0040] Agent building blocks for building reinforcement learning agents;

[0041] An agent training module, configured to train the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy;

[0042] A deployment module, configured to deploy the Q-table having the optimal air-conditioning control strategy to the air-conditioning system installed in the target building;

[0043] A control module is used to control the operation of the air-conditioning system based on the Q table with the optimal air-conditioning control strategy through the reinforcement learning agent.

[0044] The present invention also discloses a terminal, which includes: a memory, a processor, and an air-conditioning control program based on model reinforcement learning stored in the memory and runnable on the processor. When the air-conditioning control program based on model reinforcement learning is executed by the processor, the steps of the air-conditioning control method based on model reinforcement learning as described above are implemented.

[0045] The present invention also discloses a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed to implement the steps of the air-conditioning control method based on model reinforcement learning as described above.

[0046] The present invention provides a model-based reinforcement learning air conditioning control method, device, terminal, and medium. The model-based reinforcement learning air conditioning control method includes: building an RC model that reflects the thermal behavior of a target building as a training environment model; constructing a reinforcement learning agent and training the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy; deploying the Q table with the optimal air conditioning control strategy to the air conditioning system installed on the target building, and controlling the operation of the air conditioning system by the reinforcement learning agent based on the Q table with the optimal air conditioning control strategy. As can be seen, the present invention combines the advantages of RC models and reinforcement learning, fully utilizes the information of reinforcement learning interaction with the environment, reduces the demand for historical data, and can train efficient and accurate control strategies in a short period of time. In other words, it can reduce the dependence on high-precision building physical models or massive historical data, thereby achieving lightweight model construction and convenient deployment. The Q table with the optimal air conditioning control strategy deployed in the air conditioning system dynamically adjusts the air conditioning operating state, maximally balancing user comfort and demand response goals, and achieving efficient, stable, and highly robust air conditioning demand response management. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a preferred embodiment of the air conditioning control method based on model reinforcement learning in the present invention;

[0048] Figure 2 This is a schematic diagram of a specific reward function disclosed in the present invention changing with the number of iterations;

[0049] Figure 3 This is a schematic diagram of changes in indoor temperature and power consumption in a specific deployment application disclosed by the present invention;

[0050] Figure 4 is a schematic diagram of a reinforcement learning application disclosed in the present invention;

[0051] Figure 5 This is a logic schematic block diagram of a preferred embodiment of the air conditioning control method based on model reinforcement learning in the present invention;

[0052] Figure 6 This is a specific schematic diagram of verification results of indoor temperature and air conditioning start-stop strategy in virtual verification disclosed by the present invention;

[0053] Figure 7 This is a schematic diagram of the principle framework of an air conditioning control method based on model reinforcement learning disclosed in the present invention;

[0054] Figure 8 This is a schematic diagram comparing indoor temperature change results in a specific virtual verification and deployment application disclosed in the present invention;

[0055] Figure 9 This is a functional principle block diagram of a preferred embodiment of an air conditioning control device based on model reinforcement learning in the present invention;

[0056] Figure 10 It is a functional principle block diagram of a preferred embodiment of the terminal in the present invention. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear and distinct, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0058] Currently, several control strategies have been proposed for the control of air-conditioning systems during demand response periods, including rule-based control methods, model-based control methods, and reinforcement learning control methods. The specific strategies are as follows:

[0059] (1) Rule-based control method

[0060] Rule-based control relies on pre-defined rules or logic to adjust air conditioning loads. It typically doesn't require complex modeling or calculations and is suitable for scenarios with high real-time response requirements and simple control systems. However, due to its lack of flexibility, limited scope of application, and degree of optimization, rule-based control struggles to cope with complex environments and cannot guarantee efficient system operation.

[0061] (2) Model-based control method

[0062] Model-based control methods refer to achieving refined energy-saving control by establishing mathematical models of air-conditioning systems, such as thermodynamic models and energy consumption models, combined with optimization algorithms. They usually include model predictive control (MPC), optimization control, and multi-objective optimization control. However, model-based methods require a large amount of historical data and sensor information to establish accurate air-conditioning models, and usually lack good robustness. At the same time, they are not suitable for old buildings that lack historical data and sensors, and there are problems with high development costs and complex deployment.

[0063] (3) Control methods for reinforcement learning

[0064] Reinforcement learning control methods automatically adjust control strategies to adapt to dynamic environmental changes by learning from historical data and providing real-time feedback. As a feedback-based learning method, it has been widely used in demand-response control of air conditioning systems. As a data-driven control method, reinforcement learning continuously optimizes control strategies through interactive learning with the environment, demonstrating superior adaptability in highly complex and dynamically changing systems.

[0065] However, traditional reinforcement learning control methods have problems such as high data requirements, long training time, insufficient adaptability and long-term optimization capabilities, and complex model construction and deployment, making it difficult to quickly and efficiently meet the actual application needs of demand response control. To this end, this application provides an air conditioning control solution based on model reinforcement learning, which can reduce the dependence on high-precision building physical models or massive historical data, achieve lightweight model construction and convenient deployment, and dynamically adjust the air conditioning operating status to maximize the balance between user comfort and demand response goals, thereby achieving efficient, stable and highly robust air conditioning demand response management.

[0066] See Figure 1 , Figure 1 This is a flow chart of the air conditioning control method based on model reinforcement learning in the present invention. Figure 1 As shown, the air conditioning control method based on model reinforcement learning according to the embodiment of the present invention includes:

[0067] Step S11: Build an RC model that reflects the thermal behavior of the target building as a training environment model.

[0068] In this embodiment, for a target building, an RC (Resistance-Capacitance) model reflecting the indoor thermal behavior is established for the target building, that is, an RC model reflecting the thermal behavior of the target building is constructed as a training environment model.

[0069] The RC model is composed of thermal resistance and thermal capacitance. The three thermal resistance values ​​and three thermal capacitance values ​​are determined by RC model training. The expression of the RC model is:

[0070] ;

[0071] ;

[0072] ;

[0073] ;

[0074] ;

[0075] ;

[0076] ;

[0077] in, represents the heat transfer between indoor air and the air conditioning system, represents the heat transfer between the exterior envelope and the indoor air, represents the heat transfer between the outdoor air and the external envelope, Indicates the indoor temperature. Indicates the temperature of the air conditioning system. represents the temperature of the external envelope structure, Indicates the outdoor temperature, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, represents the thermal resistance of the air conditioning system, represents the indoor air thermal resistance, represents the thermal resistance of the building envelope, represents the indoor heat capacity, represents the heat capacity of the building, Indicates the cooling power of the air conditioning system. Indicates indoor heat, ONOFF indicates the operating status of the compressor, t indicates time in seconds, and h indicates hours. Indicates the mapping relationship between indoor heat and hour h, Indicates the unique indoor heat corresponding to each hour.

[0078] In addition, the operating status of the compressor is represented by 1 and 0, 1 represents on, and 0 represents off.

[0079] It is important to note that the target building (indoor) thermal behavior refers to the dynamic behavior of a specific building in a thermal environment, including the processes of heat absorption, storage, transfer, and release, as well as the impact of these processes on indoor temperature, energy consumption, and comfort. The core goal is to optimize the building's thermal performance to achieve energy conservation, environmental protection, and occupant comfort. Replacing high-precision physical models with RC models allows the building's thermal characteristics to be constructed using only a few parameters, such as thermal resistance and heat capacity. Utilizing the RC model's physical properties as the interaction environment for the reinforcement learning agent reduces the reliance on high-precision building physical models or massive amounts of historical data.

[0080] Step S12: construct a reinforcement learning agent, and train the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy.

[0081] In this embodiment, an RC model reflecting the thermal behavior of the target building is established as the training environment model. A reinforcement learning agent is also constructed, and relevant parameters are set, including environmental parameters and basic agent parameters (i.e., Q-learning parameters). After the RC model and the reinforcement learning agent are constructed, the reinforcement learning agent is trained in a reinforcement learning simulation environment generated based on the RC model to obtain a Q-table with the optimal air conditioning control strategy. The Q-table records the value of different actions in each state for query and update by the agent. The agent represents a decision-making entity, responsible for interacting with the environment to learn the strategy and ultimately update the Q-table. It is understood that during the model training phase, after the RC model and reinforcement learning agent are constructed, historical data is used to drive the agent to interact with the RC environment, train the agent to learn the optimal air conditioning control strategy, and output the corresponding Q-table. This air conditioning control strategy can improve the energy efficiency of the air conditioning system during demand response events while ensuring indoor thermal comfort, providing an efficient and adaptive control solution for building air conditioning demand response.

[0082] In this embodiment, constructing a reinforcement learning agent may specifically include: determining basic agent parameters and environmental parameters, and constructing a corresponding reinforcement learning agent based on the basic agent parameters and environmental parameters; wherein the basic agent parameters include a state space, an action space, a reward function, and an ε-greedy strategy setting value, the ε-greedy strategy setting value is a constant between 0 and 1, and the environmental parameters include outdoor temperature, indoor temperature, air conditioner operating status, electricity price, and time information;

[0083] Moreover, the Q-value update formula of the reinforcement learning agent is:

[0084] ;

[0085] Among them, s represents the state space, which includes indoor temperature, outdoor temperature and time information. It represents the next state space after executing action a in state s. a represents the action space. The action space is a binary discrete action set {0,1}. Element 0 represents the air conditioner off, and element 1 represents the air conditioner on. Indicates that in the next state space The corresponding action space, r represents the reward function, which consists of temperature comfort penalty, electricity cost penalty, low temperature reward, high temperature reward and pre-cooling reward. represents the learning rate, Represents the discount factor.

[0086] It should be noted that the ε-greedy strategy selects actions based on Q-values. This strategy sets a constant ε between 0 and 1, selects the action with the largest Q-value as the current action with a probability of 1-ε, and randomly selects an action with a probability of ε. Furthermore, the reward function r is set to optimize energy costs while ensuring indoor thermal comfort. The reward function r consists of a temperature comfort penalty (A), an electricity cost penalty (B), a low temperature reward (C), a high temperature reward (D), and a pre-cooling reward (E), namely:

[0087] ;

[0088] The Temperature Comfort Penalty (A) applies when the indoor temperature exceeds the set comfort range, such as falling below the lower comfort limit or exceeding the upper comfort limit. During the non-precooling period, a higher penalty is applied if the temperature exceeds the specified comfort range. However, during transitional periods, such as the 30 minutes before 12:00 PM and the 30 minutes before 5:00 PM, the penalty is relaxed to avoid excessive penalties due to rapid temperature fluctuations. The Electricity Cost Penalty (B) applies when the air conditioner is on (action = 1), a penalty is calculated based on the current time-of-use electricity price and a preset cost multiplier. This is intended to discourage frequent starts and stops and reduce electricity costs. The Low Temperature Reward (C) applies when the air conditioner is off (action = 0), providing the indoor temperature does not exceed the upper comfort limit for the current period, encouraging energy reduction while still meeting comfort requirements. The High Temperature Reward (D) provides an additional reward during peak hours if the indoor temperature remains between 26.5°C and 27°C. This helps fully utilize the device's control capabilities near the thermal comfort boundary, thereby reducing energy waste caused by frequent starts and stops. Pre-cooling incentive (E): To manage load before peak hours, a pre-cooling period is established, such as after 10:45 or 14:45. During this period, positive incentives are given if the indoor temperature remains between 23.5°C and 24.5°C. Conversely, penalties are imposed if the temperature falls short of the target, prompting the air conditioning system to achieve the target temperature as quickly as possible, creating favorable conditions for load regulation during peak hours.

[0089] In this embodiment, a reinforcement learning agent is trained in a reinforcement learning simulation environment generated based on an RC model to obtain a Q-table with an optimal air conditioning control strategy. Specifically, the training process may include: collecting historical outdoor temperature data and historical electricity price data of a target building; utilizing the historical outdoor temperature data and historical electricity price data to drive the reinforcement learning agent to interact with the reinforcement learning simulation environment generated based on the RC model, thereby training the reinforcement learning agent to select actions based on states obtained from the RC model, receive rewards from environmental feedback, and update the Q-table until the training converges, thereby obtaining a Q-table with an optimal air conditioning control strategy.

[0090] For example, in the training phase of the reinforcement learning agent, the agent training is based on the RC model, with outdoor temperature, indoor temperature, air conditioning operating status (ON, OFF), electricity price, time and other data as input. First, the agent's observation values ​​and environmental parameters are determined, the RC model parameters are loaded, and the collected multi-day historical outdoor temperature data and historical electricity price data are obtained to simulate the real environmental conditions (i.e., the reinforcement learning simulation environment). Secondly, the values ​​of the Q-learning related hyperparameters are set. Finally, the changes in the indoor temperature of the building are transmitted through the RC model. The agent obtains the current state quantity and updates the value function based on the current state quantity, and finally obtains the optimal action-value function (Q table with the optimal air conditioning control strategy).

[0091] For example, the training period can be set to 8:00 to 18:00 every day. The agent makes an action decision every 1 minute, that is, it performs 600 actions every day, and generates 600 state transitions (a four-tuple process containing state, action, reward, and new state). The complete interaction process of the agent from 8:00 to 18:00 every day is called an episode. After each episode, the system records the actual reward of the episode and calculates the average reward of the last n episodes to intuitively reflect the training process, where n can be set to 600. During the training process, the reward function changes with the number of iterations, see Figure 2 As shown in the figure, operating hours and temperature ranges are clearly defined: Peak hours are defined as 11:00–12:00 and 15:00–17:00, totaling three hours. Due to higher electricity costs during these periods, the system allows for a wider temperature range, such as 23°C to 27°C, to allow for flexible control while still meeting basic comfort requirements. During off-peak hours, to ensure tighter control over indoor temperature fluctuations, the comfortable temperature range is set at 24°C to 26°C.

[0092] Step S13: deploying the Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed in the target building, and controlling the operation of the air-conditioning system based on the Q-table with the optimal air-conditioning control strategy by the reinforcement learning agent.

[0093] In this embodiment, after the intelligent agent completes training and outputs a Q-table with the optimal air conditioning control strategy, it can be deployed and applied. That is, the Q-table is deployed to the air conditioning system installed in the target building, and the operation of the air conditioning system is controlled by the reinforcement learning intelligent agent based on the Q-table with the optimal air conditioning control strategy. It can be understood that the control strategy is applied to the actual operation of the building air conditioning system to perform real-time indoor and outdoor temperature monitoring. The intelligent agent analyzes the indoor and outdoor temperature output control actions based on the control strategy. The indoor temperature and power consumption changes are shown in Figure 2. Figure 3 shown.

[0094] Specifically, the system acquires current real-time environmental data and uses a reinforcement learning agent to analyze it based on a Q-table with an optimal air conditioning control strategy to determine the optimal air conditioning control action and generate corresponding air conditioning control instructions. These instructions are then sent to the infrared controller via the Modbus protocol, which then controls the operation of the air conditioning system. It is important to note that using the Modbus protocol to ensure real-time data transmission at a high time granularity allows for rapid response and dynamic adjustments to address sudden weather changes and equipment performance variations, ensuring adaptability to the actual building environment and achieving minute-by-minute control.

[0095] In this embodiment, in the process of controlling the operation of the air-conditioning system through the reinforcement learning intelligent agent based on the Q table with the optimal air-conditioning control strategy, it also includes: real-time recording of a full-dimensional operation log containing the current time, current real-time environmental data, air-conditioning control actions and energy consumption of the air-conditioning system, so as to dynamically adjust the Q table based on the full-dimensional operation log.

[0096] For example, see Figure 4 As shown in the figure, first, at the hardware level, the start and stop of the air-conditioning system is controlled by an infrared controller. The infrared controller and the Python console are connected through the Modbus protocol, which can achieve fast and reliable data interaction, including sending start and stop commands to the air-conditioning and obtaining operating status feedback from the device side. At the same time, the current real-time environmental data obtained, such as temperature and humidity sensors, indoor and outdoor temperature data from outdoor weather stations, and external time-varying electricity price information, are used as inputs of the intelligent agent to constitute the main source of environmental status. Specifically, the Python console can obtain the above environmental data in real time through a wireless network (Wi-Fi or 4G) and refresh the intelligent agent's perception of the environment in short time intervals; secondly, at the software level, the intelligent agent and the Python console together constitute the core of decision-making and execution. Based on the trained Q-table, the intelligent agent infers the environmental state collected each time and selects the optimal action, that is, whether to turn the air conditioner on or off. Finally, the Python console sends the action instruction to the infrared controller in the form of Modbus protocol, which triggers the air conditioner operation and realizes direct interaction with the physical device. In order to realize online or offline updates of reinforcement learning, the corresponding immediate reward can be calculated after each action is executed, and the Q-table can be adjusted online when necessary to maintain the adaptability between the strategy and the dynamic changes of the environment.

[0097] Specifically, the system is first deployed at the hardware interaction layer, enabling two-way communication between the infrared controller and the Python console via the Modbus protocol, issuing air conditioning control commands in real time and obtaining device status feedback. Secondly, at the software decision-making layer, the intelligent agent infers real-time environmental data (temperature and electricity price) based on the Q-table to generate air conditioning control action commands. Finally, a data closed loop and policy adjustment are established. During the actual air conditioning system control process, a full-dimensional operation log can be established, including the current time, current real-time environmental data, air conditioning control actions, and the energy consumption of the air conditioning system. By analyzing environmental changes offline and automatically adjusting policy parameters, a continuous balance between comfort and energy efficiency is achieved. Specifically, a reward value is calculated after each action is executed, and the Q-table is adjusted in real time when necessary, maximizing the balance between user comfort and demand response goals, achieving efficient, stable, and highly robust air conditioning demand response management.

[0098] It can be seen that in the embodiments of the present invention, by combining the advantages of the RC model and reinforcement learning, the information of the interaction between reinforcement learning and the environment is fully utilized, the demand for historical data is reduced, and an efficient and accurate control strategy can be trained in a short period of time. In other words, the dependence on high-precision building physical models or massive historical data can be reduced, thereby achieving lightweight model construction and convenient deployment. The air-conditioning operating status is dynamically adjusted by deploying a Q table with an optimal air-conditioning control strategy in the air-conditioning system, maximally balancing user comfort and demand response goals, and achieving efficient, stable, and highly robust air-conditioning demand response management.

[0099] In other words, the air conditioning control method based on model reinforcement learning in this application, by combining the advantages of RC models and reinforcement learning, uses RC models to replace high-precision physical models. Only a small number of parameters such as thermal resistance and heat capacity are required to construct the thermal characteristics of the building. The physical characteristics of the RC model are used as the interactive environment of the reinforcement learning agent, reducing the dependence on high-precision building physical models or massive historical data. Through real-time interaction with the RC model through the Q-learning algorithm, the construction of the RC model can be completed in just one day, and the training of the reinforcement learning Q table can be completed within 10 minutes. In other words, the air conditioning control method based on model reinforcement learning in this application is a lightweight and efficient air conditioning control solution with low data requirements, fast training speed, and strong adaptability. It has long-term optimization capabilities and hourly deployment capabilities. It can overcome the problems of high data requirements, long training time, insufficient adaptability and long-term optimization capabilities, and complex model construction and deployment in traditional air conditioning system control methods, thereby improving the robustness and optimization capabilities of the system. Compared with the problems existing in traditional air-conditioning control methods, the air-conditioning control method based on model reinforcement learning in this application has strong scalability and strong adaptability of control strategies. It effectively solves the problem that traditional methods are difficult to quickly and efficiently meet the actual application needs of demand response control, and provides an efficient and stable solution for demand response control.

[0100] See also Figure 5 As shown, the embodiment of the present invention discloses a specific air-conditioning control method based on model reinforcement learning. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution.

[0101] Step S21: Build an RC model that reflects the thermal behavior of the target building as a training environment model.

[0102] Step S22: construct a reinforcement learning agent, and train the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy.

[0103] Step S23: deploying the Q-table with the optimal air-conditioning control strategy in the reinforcement learning simulation environment generated based on the RC model and consistent with the training environment parameters to verify the air-conditioning control performance of the Q-table and obtain corresponding verification results.

[0104] In this embodiment, the air conditioning control strategy can be verified before deployment. The Q-table output by the agent, representing the optimal air conditioning control strategy, is deployed in a reinforcement learning simulation environment generated based on the RC model, consistent with the training environment parameters. This allows for actual control performance verification within a virtual interactive environment with the target building, yielding corresponding verification results. It is understood that the effectiveness and stability of the model strategy are verified on the RC model, and the Q-table output by the agent is deployed and verified on the RC model to evaluate the control strategy performance.

[0105] For example, during the verification process, the Q-table and RC model parameters for the optimal air conditioning control strategy generated during the training phase are first loaded to ensure consistent environmental settings used during training. New outdoor temperature data is also collected to simulate actual temperature fluctuations. Next, the agent selects the optimal action based on the current state (indoor and outdoor temperatures, time) and the Q-table. The state transition function calculates the changes in indoor, system, and wall temperatures every minute, based on environmental changes. Action selection and temperature changes are recorded in real time to simulate the performance of the control strategy in a real environment. New outdoor temperature data is then fed into the agent, which selects the optimal action based on the current state and the optimal action-value function. The RC model calculates the changes in indoor, system, and wall temperatures. Action selection and temperature changes are recorded in real time, along with instantaneous rewards. State changes, control actions, and reward values ​​are recorded at each time point. Detailed temperature curves, air conditioning start / stop status, and reward curves are plotted based on the real-time log data to comprehensively evaluate the overall effectiveness and performance stability of the strategy. Finally, the verification results are visualized using a time series graph, including indoor temperature, outdoor temperature, air conditioning action, and immediate reward curve. By comparing the actual temperature changes with the control strategy performance, we can deeply analyze the performance of the Q-learning model in different scenarios and further improve the effectiveness and robustness of the strategy.

[0106] Among them, in the verification stage, three sets of typical weather data with significant temperature differences can be selected: the average temperature of the first day is 28.8℃ (mild working condition), the average temperature of the second day is 33.4℃ (high temperature working condition), and the average temperature of the third day is 32.4℃ (fluctuating working condition). The environmental adaptability of the Q meter can be examined by comparing the test system under multiple working conditions. The verification results of one day can be found in Figure 6 The analysis of the verification results is as follows:

[0107] First, in terms of temperature control, the system successfully maintained the room temperature within the 26°C threshold during non-demand response periods and effectively controlled it below the 27°C limit during demand response periods. Before the DR events on the three verification days (11:00-12:00 and 15:00-17:00) were triggered, the intelligent agent activated the pre-cooling strategy in advance, resulting in a pre-cooling of 1.2-1.5°C.

[0108] Second, regarding air conditioning operation control, the first DR period saw a 92.3% reduction in air conditioning runtime compared to baseline operating conditions, from the typical 54.5 minutes to 4.2 minutes. The second DR period saw reductions of 79.7%, 75.5%, and 75.7%, respectively. All three groups saw electricity consumption reductions exceeding 75%.

[0109] Step S24: Optimize the Q table based on the verification result to obtain an optimized Q table.

[0110] In this embodiment, after verifying the effectiveness and stability of the model strategy on the RC model, the Q-table can be optimized based on the verification results to obtain an optimized Q-table. It is understood that if the effectiveness and stability of the strategy are found to fail to meet requirements during the verification phase, the Q-table can be further optimized to enable efficient and accurate control in subsequent deployment applications.

[0111] Step S25: deploy the optimized Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed on the target building, and control the operation of the air-conditioning system based on the optimized Q-table with the optimal air-conditioning control strategy through the reinforcement learning agent.

[0112] For the specific contents of the above steps S21 to S22 and step S25, reference may be made to the corresponding contents disclosed in the above embodiments, which will not be repeated here.

[0113] It can be seen that in the embodiment of the present invention, by combining the advantages of the RC model and reinforcement learning, the information of the interaction between reinforcement learning and the environment is fully utilized, the demand for historical data is reduced, and an efficient and accurate control strategy can be trained in a short period of time, that is, the dependence on high-precision building physical models or massive historical data can be reduced, thereby achieving lightweight model construction and convenient deployment, and dynamically adjusting the air-conditioning operating state through the Q table with the optimal air-conditioning control strategy deployed in the air-conditioning system, taking into account user comfort and demand response goals to the greatest extent, and achieving efficient, stable and highly robust air-conditioning demand response management. The technical solution of this application forms an adaptive closed-loop control system from the early stage simulation and verification to the later stage field deployment. It can dynamically adjust the air-conditioning operating state, taking into account user comfort and demand response goals to the greatest extent, and achieving efficient, stable and highly robust air-conditioning demand response management.

[0114] For example, see Figure 7As shown, this application combines the advantages of RC models and reinforcement learning to propose a three-step strategy of "model training + virtual verification + deployment application" to achieve a lightweight and efficient reinforcement learning control method. Specifically, in model training, an RC model reflecting the indoor thermal behavior of the target building is established, and a reinforcement learning agent is constructed. Relevant parameters are set, such as the basic agent parameters including the state space, action space, reward function, and ε-greedy strategy setting value, as well as environmental parameters including outdoor temperature, indoor temperature, air conditioning operating status, electricity price, and time information. Historical data is used to drive the agent to interact with the RC environment and output a Q-table after training. In virtual verification, the effectiveness and stability of the model strategy are verified on the RC model. The Q-table output by the agent is deployed on the RC model for verification to evaluate the control strategy performance. In deployment application, the control strategy is applied to the actual operation of the building air conditioning system to perform real-time indoor and outdoor temperature monitoring. The agent analyzes the indoor and outdoor temperature output control actions based on the control strategy. This solution of the present application can solve the problems of complex modeling, poor strategy flexibility, high deployment cost and complex deployment in traditional control methods. By dynamically adjusting the air conditioning control strategy, dual optimization of building thermal comfort and energy efficiency is achieved. In addition, the comparison of indoor temperature variation results in virtual verification and deployment applications can be found in Figure 8 As shown in Figure 2, outdoor temperature data from the experimental day was imported into the RC model, resulting in a "virtual verification - indoor temperature" curve based on virtual simulation. The two curves exhibit some discrepancies at specific time periods, primarily due to differences in thermodynamic properties and external interference factors between the experimental and virtual simulation environments. However, overall, the two exhibit highly consistent fluctuation patterns, with a correlation coefficient of 0.87 throughout the day, demonstrating that virtual simulation effectively simulates key features of the experimental process at a macro level.

[0115] In one embodiment, if Figure 9 As shown, based on the above-mentioned air conditioning control method based on model reinforcement learning, the present invention also provides an air conditioning control device based on model reinforcement learning, including:

[0116] The RC model building module 11 is used to build an RC model reflecting the thermal behavior of the target building as a training environment model;

[0117] An agent building module 12, used to build a reinforcement learning agent;

[0118] An agent training module 13 is used to train the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy;

[0119] A deployment module 14 is configured to deploy the Q table having the optimal air conditioning control strategy to the air conditioning system installed in the target building;

[0120] The control module 15 is configured to control the operation of the air-conditioning system based on the Q table having the optimal air-conditioning control strategy through the reinforcement learning agent.

[0121] Figure 10 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may include:

[0122] Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0123] When the processor 502 executes the program, the air-conditioning control method based on model reinforcement learning provided in the above embodiment is implemented.

[0124] Furthermore, the terminal further includes:

[0125] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0126] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0127] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0128] If memory 501, processor 502, and communication interface 503 are implemented independently, communication interface 503, memory 501, and processor 502 can be interconnected via a bus to facilitate communication. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be categorized as address buses, data buses, and control buses. For ease of illustration, the figure uses only one line, but this does not imply that there is only one bus or only one type of bus.

[0129] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0130] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0131] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned air-conditioning control method based on model reinforcement learning.

[0132] Other embodiments of the present invention will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses, or adaptations of the present invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the invention being indicated by the claims.

[0133] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0134] The logic and / or steps represented in the flowchart or otherwise described herein may be considered, for example, as a sequenced list of executable instructions for implementing the logical functions, and may be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other system that can read and execute instructions from an instruction execution system, apparatus, or device).

[0135] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having logic gate circuits for implementing logical functions on data signals, an application-specific integrated circuit having suitable combinational logic gate circuits, a programmable gate array (PGA), a field-programmable gate array (FPGA), etc.

[0136] It should be understood that the application of the present invention is not limited to the above examples. For those skilled in the art, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An air conditioning control method based on model reinforcement learning, characterized in that: The method comprises: Build an RC model that reflects the thermal behavior of the target building as a training environment model; Constructing a reinforcement learning agent and training the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy; Deploying the Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed in the target building, and controlling the operation of the air-conditioning system based on the Q-table with the optimal air-conditioning control strategy by the reinforcement learning agent; The RC model is composed of thermal resistance and thermal capacitance, and the expression of the RC model is: ； ； ； ； ； ； ； in, represents the heat transfer between indoor air and the air conditioning system, represents the heat transfer between the exterior envelope and the indoor air, represents the heat transfer between the outdoor air and the external envelope, Indicates the indoor temperature. Indicates the air conditioning system temperature. represents the temperature of the external protective structure, Indicates the outdoor temperature, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, represents the thermal resistance of the air conditioning system, represents the indoor air thermal resistance, represents the thermal resistance of the building envelope, represents the heat capacity of the air conditioning system, represents the indoor heat capacity, represents the heat capacity of the building, Indicates the cooling power of the air conditioning system. Indicates indoor heat, ONOFF indicates the operating status of the compressor, t indicates time in seconds, and h indicates hours. Indicates the mapping relationship between indoor heat and hour h, Indicates the indoor heat corresponding to each hour.

2. The air conditioning control method based on model reinforcement learning according to claim 1, characterized in that: The construction of the reinforcement learning agent includes: Determining agent basic parameters and environment parameters, and constructing a corresponding reinforcement learning agent based on the agent basic parameters and the environment parameters; The basic parameters of the agent include state space, action space, reward function and ε-greedy strategy setting value, the ε-greedy strategy setting value is a constant between 0 and 1, and the environmental parameters include outdoor temperature, indoor temperature, air conditioning operation status, electricity price and time information; In addition, the Q-value update formula of the reinforcement learning agent is: ； Wherein, s represents the state space, which includes indoor temperature, outdoor temperature and time information. represents the next state space after executing action a in state s, a represents the action space, and the action space is a binary discrete action set {0,1}, element 0 represents the air conditioner is turned off, and element 1 represents the air conditioner is turned on. Indicates that in the next state space The corresponding action space, r represents the reward function, which consists of temperature comfort penalty, electricity cost penalty, low temperature reward, high temperature reward and pre-cooling reward. represents the learning rate, represents the discount factor; Indicates that the status All possible actions The maximum value of the corresponding Q value.

3. The air conditioning control method based on model reinforcement learning according to claim 1, characterized in that: The step of training the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table having an optimal air conditioning control strategy includes: Collecting historical outdoor temperature data and historical electricity price data of the target building; The historical outdoor temperature data and the historical electricity price data are used to drive the reinforcement learning agent to interact with a reinforcement learning simulation environment generated based on the RC model, so as to train the reinforcement learning agent to select actions based on the state obtained from the RC model, receive rewards from the environment, and update the Q-table until the training converges, thereby obtaining a Q-table with an optimal air conditioning control strategy.

4. The air conditioning control method based on model reinforcement learning according to claim 1, characterized in that: Before deploying the Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed in the target building and controlling the operation of the air-conditioning system based on the Q-table with the optimal air-conditioning control strategy by the reinforcement learning agent, the method further includes: Deploying the Q-table with the optimal air conditioning control strategy in the reinforcement learning simulation environment generated based on the RC model and having parameters consistent with the training environment to verify the air conditioning control performance of the Q-table and obtain corresponding verification results; Optimizing the Q table based on the verification result to obtain an optimized Q table; The step of deploying the Q-table with the optimal air-conditioning control strategy to the air-conditioning system installed in the target building, and controlling the operation of the air-conditioning system based on the Q-table with the optimal air-conditioning control strategy by the reinforcement learning agent, includes: The optimized Q table with the optimal air-conditioning control strategy is deployed to the air-conditioning system installed on the target building, and the operation of the air-conditioning system is controlled by the reinforcement learning agent based on the optimized Q table with the optimal air-conditioning control strategy.

5. The air conditioning control method based on model reinforcement learning according to any one of claims 1 to 4, characterized in that: The step of controlling the operation of the air-conditioning system based on the Q table having the optimal air-conditioning control strategy by the reinforcement learning agent includes: Acquire current real-time environmental data, and analyze the current real-time environmental data by the reinforcement learning agent based on the Q table with the optimal air conditioning control strategy to determine an optimal air conditioning control action, and generate an air conditioning control instruction corresponding to the optimal air conditioning control action; The air conditioning control instruction is sent to the infrared controller based on the Modbus protocol, so as to control the operation of the air conditioning system through the infrared controller.

6. The air conditioning control method based on model reinforcement learning according to claim 5, characterized in that: In the process of controlling the operation of the air-conditioning system by the reinforcement learning agent based on the Q table with the optimal air-conditioning control strategy, the method further includes: A full-dimensional operation log including the current time, the current real-time environmental data, the air-conditioning control action and the energy consumption of the air-conditioning system is recorded in real time, so as to dynamically adjust the Q table based on the full-dimensional operation log.

7. An air conditioning control device based on model reinforcement learning, characterized in that: The device comprises: RC model building module, used to build an RC model that reflects the thermal behavior of the target building as a training environment model; Agent building blocks for building reinforcement learning agents; An agent training module, configured to train the reinforcement learning agent in a reinforcement learning simulation environment generated based on the RC model to obtain a Q table with an optimal air conditioning control strategy; A deployment module, configured to deploy the Q-table having the optimal air-conditioning control strategy to the air-conditioning system installed in the target building; a control module, configured to control the operation of the air-conditioning system based on the Q table having the optimal air-conditioning control strategy through the reinforcement learning agent; The RC model is composed of thermal resistance and thermal capacitance, and the expression of the RC model is: ； ； ； ； ； ； ； in, represents the heat transfer between indoor air and the air conditioning system, represents the heat transfer between the exterior envelope and the indoor air, represents the heat transfer between the outdoor air and the external envelope, Indicates the indoor temperature. Indicates the air conditioning system temperature. represents the temperature of the external protective structure, Indicates the outdoor temperature, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The temperature of the air conditioning system at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The indoor temperature at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, Indicates the The temperature of the external envelope structure at a certain moment, represents the thermal resistance of the air conditioning system, represents the indoor air thermal resistance, represents the thermal resistance of the building envelope, represents the heat capacity of the air conditioning system, represents the indoor heat capacity, represents the heat capacity of the building, Indicates the cooling power of the air conditioning system. Indicates indoor heat, ONOFF indicates the operating status of the compressor, t indicates time in seconds, and h indicates hours. Indicates the mapping relationship between indoor heat and hour h, Indicates the indoor heat corresponding to each hour.

8. A terminal, characterized in that: include: A memory, a processor, and an air-conditioning control program based on model reinforcement learning stored in the memory and executable on the processor, wherein the air-conditioning control program based on model reinforcement learning, when executed by the processor, implements the steps of the air-conditioning control method based on model reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which can be executed to implement the steps of the air-conditioning control method based on model reinforcement learning as described in any one of claims 1 to 6.

Images