Intelligent regulation and control method for room temperature of heat supply building

Abstract

An intelligent room temperature regulation and control method for a heat supply building belongs to the field of building heat supply, and comprises the following steps: constructing a heat supply building pipe network hydraulic and thermal model, and forming a hydraulic and thermal coupling model; the hydraulic and thermal coupling model comprises a pipe network hydraulic model, a regulating valve model, a heat exchanger model and a building thermal network model; constructing a multi-agent deep reinforcement learning model by using the hydraulic and thermal coupling model, and defining that each hot area of the heat supply building corresponds to one agent; and the multi-agent deep reinforcement learning model is used for solving the valve opening degree of each hot area, intelligent valve regulation and control are carried out according to the indoor temperature set value of the user, and the indoor temperature regulation and control of the heat user of the heat supply building are achieved. According to the method, based on the idea of reinforcement learning, the heat supply operation data of the building is fully utilized, the environment model adaptive to reinforcement learning is constructed based on the simulation model, and technical support is provided for development of intelligent heat supply.

Description

Technical Field

[0001] The invention relates to a temperature control method, in particular to an intelligent control method for room temperature of a heating building, and belongs to the field of building heating. Background Art

[0002] In recent years, the area of ​​centralized heating in the north has increased, and carbon emissions from building operations have gradually increased. The current centralized heating system has problems with low heating efficiency and inflexible adjustment, resulting in energy waste and additional carbon emissions. Current research still has gaps in the precise control of user demand-side response based on complex thermally coupled heating systems. It is impossible to accurately control the user's indoor temperature, and there are problems of energy waste and user discomfort caused by overheating or overcooling.

[0003] In view of the above, how to respond to users' demand side and accurately control users' indoor temperature to prevent energy waste and user discomfort caused by overheating or overcooling of indoor temperature needs to be solved urgently. Summary of the invention

[0004] In order to overcome the shortcomings of the existing technology, this application provides a method for intelligent control of room temperature in heating buildings. The method is based on a multi-agent deep reinforcement learning algorithm, and accurately controls the user's indoor temperature based on the user's demand-side response. Reinforcement learning, as a data-driven artificial intelligence optimization technology, is essentially interactive learning, that is, through an intelligent agent with autonomous decision-making ability to interact with its environment, and choose a response action based on the perceived external environmental state. Therefore, reinforcement learning has a high degree of structural fit with the problem of heating system control. By optimizing energy scheduling and accurately controlling the demand side, the energy supply cost is reduced, and energy waste and user discomfort caused by overheating or overcooling of indoor temperature are prevented.

[0005] A method for intelligently controlling room temperature of a heating building comprises the following steps:

[0006] S1. Construct hydraulic and thermal models of the heating building pipe network and form a hydraulic and thermal coupling model;

[0007] The hydraulic and thermal coupling model includes a pipe network hydraulic model, a regulating valve model, a heat exchanger model and a building thermal network model;

[0008] S2. Use the hydraulic-thermal coupling model to build a multi-agent deep reinforcement learning model, and define one agent for each thermal zone of the heating building;

[0009] S3. Utilize the multi-agent deep reinforcement learning model to solve the valve opening of each thermal zone, and intelligently adjust the valve according to the user's indoor temperature setting value to achieve room temperature control for heat users in the heating building.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] 1. The intelligent control technology of room temperature in heating buildings in this application can meet the personalized needs of users.

[0012] 2. This application conducts hydraulic and thermal coupling simulation modeling, and the proposed thermal simulation framework for multi-thermal zone heating buildings can fully and accurately reflect the hydraulic characteristics of the heating network and the thermal characteristics of the heating buildings.

[0013] 3. This application is based on the concept of reinforcement learning, makes full use of the heating operation data of the building, constructs a hydraulic and thermal coupling simulation model of the heating building, and constructs an environmental model adapted to reinforcement learning based on the simulation model, providing technical support for the development of smart heating.

[0014] The present application will be further described below in conjunction with the accompanying drawings and implementation methods: BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a flow chart of the method for intelligently controlling room temperature of a heating building of the present application;

[0016] Figure 2 It is the heat network model diagram of two adjacent heat zones;

[0017] Figure 3 Construct a flow chart for the coupled building hydraulic and thermal model;

[0018] Figure 4 This is a diagram of the principle of reinforcement learning;

[0019] Figure 5 This is a schematic diagram of the training control of the proximal strategy optimization algorithm used in this application;

[0020] Figure 6 Schematic diagram of the multi-agent interaction process;

[0021] Figure 7 This is a schematic diagram of the distribution of heat zones in a heating building in the embodiment;

[0022] Figure 8 It is a graph showing the measurement data of outdoor temperature and ground surface horizontal radiation in the embodiment;

[0023] Fig. 9 The reward function curve diagram of the embodiment in which the temperature difference between the actual indoor temperature and the set value is between [-3, 3];

[0024] Fig.10 It is a curve diagram of outdoor temperature and horizontal solar radiation in the embodiment;

[0025] Fig.11 It is a reward convergence curve diagram in the embodiment;

[0026] FIG. 12( a ) is a diagram showing the indoor temperature control result of room number 1 in the embodiment;

[0027] FIG. 12( b ) is a diagram showing the indoor temperature control result of room number 2 in the embodiment;

[0028] FIG. 12( c ) is a diagram showing the indoor temperature control result of room number 3 in the embodiment;

[0029] FIG. 12( d ) is a diagram showing the indoor temperature control result of room number 4 in the embodiment;

[0030] FIG. 12( e ) is a diagram showing the indoor temperature control result of room number 5 in the embodiment;

[0031] FIG. 12( f ) is a diagram showing the indoor temperature control result of room number 6 in the embodiment;

[0032] FIG12( g ) is a diagram showing the indoor temperature control result of room number 7 in the embodiment;

[0033] FIG12( h ) is a diagram showing the indoor temperature control result of room number 8 in the embodiment;

[0034] FIG. 12( i ) is a diagram showing the indoor temperature control result of room number 9 in the embodiment;

[0035] FIG. 12( j ) is a diagram showing the indoor temperature control results of room number 10 in the embodiment. DETAILED DESCRIPTION

[0036] The embodiments of the technical solution of the present application will be described in detail below in conjunction with the accompanying drawings. Unless otherwise specified, the technical terms or scientific terms used in the present application are generally understood by those skilled in the art.

[0037] Researching and improving the heating system in northern cities will not only help improve heating efficiency and reduce energy waste, but also significantly reduce carbon emissions from building operations, which is of great significance. Current research still has a gap in the precise control of user demand response based on complex thermal coupling heating systems.

[0038] See also Figure 1 As shown, a method for intelligently controlling room temperature of a heating building in this embodiment comprises the following steps:

[0039] S1. Construct hydraulic and thermal models of the heating building pipe network and form a hydraulic and thermal coupling model;

[0040] The hydraulic and thermal coupling model includes a pipe network hydraulic model, a regulating valve model, a heat exchanger model and a building thermal network model;

[0041] S2. Use the hydraulic-thermal coupling model to build a multi-agent deep reinforcement learning model, and define one agent for each thermal zone of the heating building;

[0042] S3. Utilize the multi-agent deep reinforcement learning model to solve the valve opening of each thermal zone, and intelligently adjust the valve according to the user's indoor temperature setting value to achieve room temperature control for heat users in the heating building.

[0043] As a data-driven artificial intelligence optimization technology, reinforcement learning is essentially interactive learning, that is, an intelligent agent with autonomous decision-making ability interacts with its environment and chooses response actions based on the perceived state of the external environment. Therefore, reinforcement learning has a high structural fit with the heating system regulation problem. By optimizing energy scheduling and precise regulation on the demand side, the energy supply cost can be reduced, and energy waste and user discomfort caused by overheating or overcooling of indoor temperatures can be prevented.

[0044] Based on the above concept, a method for intelligent control of room temperature in heating buildings based on multi-agent deep reinforcement learning algorithm is proposed, which includes the following contents:

[0045] First, obtain the topological structure data of the heating building and build a hydraulic and thermal coupling simulation model of the building, such as Figure 3 The process shown is as follows:

[0046] The construction process of the heating network hydraulic model is as follows:

[0047] Using graph theory methods, the target building hydraulic network is topologically transformed into a directed weighted graph. Heat users are abstracted as pipe segments. It is assumed that there are n+1 nodes and b pipes in the network.

[0048] The flow continuity equation of the pipe network is shown in formula (1):

[0049] A n×b G b×1 =q n×1 (1)

[0050] Where: A is the basic correlation matrix of the pipe network;

[0051] G is the flow vector of the pipe section;

[0052] q is the traffic of each node;

[0053] The pipe network loop equation is shown in formula (2):

[0054] B (b-n)×b H b×1 =0 (2)

[0055] Where: B is the basic loop matrix of the pipe network;

[0056] H is the pressure difference vector of the pipe section;

[0057] The head loss equation is shown in formula (3):

[0058] H b×1 =HW b×1 +WZ b×1 -DH b×1 (3)

[0059] Where: HW is the pipeline network pressure loss vector;

[0060] WZ is the water head difference vector at both ends of the pipe section;

[0061] DH is the pump head vector;

[0062] in:

[0063] HW b×1 =S b×b |G b×1 |G b×1 (4)

[0064] DH b×1 =a 1 |G b×1 |G b×1 +a 2 G b×1 +a 3 (5)

[0065] Where: S b×b is the pipe resistance coefficient matrix;

[0066] a 1 ,a 2 ,a 3 is the water pump characteristic parameter matrix;

[0067] The pipeline hydraulic model established according to formulas (1)-(5) is solved using the basic loop method.

[0068] The pipeline equivalent impedance is calculated according to the characteristic parameters such as valve opening, pipe diameter and pipe length. The equivalent impedance of the heating system pipeline and the characteristic parameters of the water pump are used as the input of the pipeline hydraulic model, and the flow rate of each node of the heating system is used as the output.

[0069] Secondly, build a regulating valve model:

[0070] According to the pressure difference and flow rate before and after the valve, the resistance characteristics of the regulating valve are calculated; as shown in formula (6);

[0071]

[0072] Where: ΔP v is the pressure difference before and after the regulating valve, Pa;

[0073] s v is the resistance coefficient of the regulating valve at a certain opening, Pa / (m3 / h) 2 ;

[0074] V v is the volume flow through the regulating valve, m 3 / h.

[0075] The ideal flow characteristics of the control valve are divided into four types: linear flow characteristics, equal percentage flow characteristics, quick opening flow characteristics and parabolic flow characteristics. According to different valve characteristic curves, the relationship between valve flow and valve opening can be constructed.

[0076] The thermal model is constructed below: The thermal model includes the heat exchanger model and the heating network model of the heating building;

[0077] The heat exchanger model building process in step S1 is:

[0078] According to the law of conservation of energy, the heat transfer process of the radiator is expressed by equation (7):

[0079]

[0080] Where: m t is the heat flow rate in the radiator at time t, kg / s;

[0081] c is the specific heat capacity of water, J / (kg·℃);

[0082] T g t is the water supply temperature at time t, °C;

[0083] T h t is the return water temperature at time t, °C;

[0084] T in t is the indoor temperature at time t, ℃;

[0085] F is the heat transfer area of ​​the radiator, m 2 ;

[0086] k is the heat transfer coefficient of the radiator, W / (m 2 ·℃).

[0087] According to the measured supply water temperature, return water temperature, heat flow rate and indoor room temperature, the kF parameter is identified using formula (7). The heat supply of the radiator is calculated based on the identified parameters.

[0088] The building thermal network model construction process in step S1 is:

[0089] First, the heating building is divided into building thermal zones, each of which has a control device (for example, an area less than 100m 2 ).

[0090] Establish a multi-thermal zone thermal network model. The multi-thermal zone thermal network model of the building is dominated by the dynamic heat transfer of the envelope structure and reaches a dynamic balance under the influence of internal heat gain, solar radiation and heat transfer between households. Figure 2 The heat network model diagram of two adjacent areas is shown;

[0091] A single hot zone at node T n The heat balance equation is:

[0092] Q n =Q h,n +Q i,n +Q s,n (8)

[0093] Where: Q n is the heat load of the hot zone, W;

[0094] Q h,n is the heat load of the heating system, W;

[0095] Q s,n is the solar radiation heat gain in the hot zone, W;

[0096] Q i,n Heat gained by lighting for personnel and equipment inside the hot zone, W;

[0097] Where: Q s,n =α n q s (9)

[0098] Where: α n is the solar radiation coefficient of the hot zone, m 2 ;

[0099] q s is the solar radiation per square meter of horizontal surface, W / m 2 ;

[0100] Then: Node T of hot zone n n The heat transfer differential equation is shown in formula (10):

[0101]

[0102] Where: n, m are the hot zone numbers;

[0103] T n is the indoor temperature of hot zone n, °C;

[0104] T m is the indoor temperature of hot zone m, °C;

[0105] T n,e is the temperature of the enclosure structure in hot zone n, °C;

[0106] T ext is the outdoor temperature, °C;

[0107] C n is the heat capacity of hot zone n, J / K;

[0108] R n,m is the thermal resistance of adjacent hot zones, K / W;

[0109] R n,e is the heat transfer resistance between the hot zone n and the surface of the enclosure structure, K / W;

[0110] R n,n is the thermal resistance of the external window in hot zone n, K / W;

[0111] Then: Node T of hot zone n n,e The heat transfer differential equation is shown in formula (11):

[0112]

[0113] Where: C n,e is the heat capacity of the enclosure structure in hot zone n, J / K;

[0114] R n,n,e is the thermal resistance between the heat zone n enclosure and the external environment, K / W;

[0115] Q n,e is the solar radiation heat gain of the enclosure structure in hot zone n, W;

[0116] Where: Q n,e =α n,e q s (12)

[0117] Where: α n,e is the solar radiation coefficient of the enclosure structure in hot zone n, m 2 .

[0118] Similarly, the relevant parameters of hot zone m are calculated in the same way as the above hot zone n, and will not be repeated here.

[0119] The multi-thermal zone thermal network model of the building is solved according to formulas (8)-(12), and the solar radiation coefficient, heat capacity and thermal resistance of the building are solved by genetic algorithm. The input parameters are the hourly measured temperature of each thermal zone, outdoor temperature, heat supply of the thermal zone and solar radiation. The solved solar radiation coefficient, heat capacity and thermal resistance are used to establish a complete multi-thermal zone thermal network model. The input parameters are the heat supply of the thermal zone, outdoor temperature and heat supply of the thermal zone. The output parameter is the indoor temperature of each thermal zone.

[0120] After the construction of the heating network hydraulic model, regulating valve model, heat exchanger model and building thermal network model, the building hydraulic and thermal coupling simulation model is constructed based on the above models. The initial temperature of the hot zone, outdoor temperature, solar radiation, valve opening and water supply temperature are used as inputs, the hydraulic model and regulating valve model are called, the water supply flow of the hot zone is calculated, the water supply flow is input into the heat exchanger model, the heat supply of the hot zone is calculated, the heat supply of the hot zone, outdoor temperature and solar radiation are input into the building thermal network model, and the indoor temperature of the hot zone is obtained.

[0121] Secondly, build a multi-agent deep reinforcement learning environment model, such as Figure 4 The reinforcement learning principle diagram shown in the figure obtains the optimal solution for intelligent control of indoor temperature of heat users through multi-agent deep reinforcement algorithm;

[0122] (1) Constructing a Markov game process for intelligent control of indoor temperature in multiple thermal zones

[0123] The intelligent control problem of building heating system is described as a Markov game process from a mathematical optimization model. In this problem, each independent thermal zone of the building is defined as an intelligent agent, the measurable information is the state space S, and the dispatching instruction set of the valve control device is the action space A, thus converting the original intelligent control problem of building heating system into a Markov game process. The specific game process is constructed as follows.

[0124] ① State space construction

[0125] Define the state observation of agent n at time t as s n,t ∈S, which includes weather data, indoor air temperature and indoor air set temperature, as shown in Table 1. Denoted as s n,t =[T in ,T set ] n The state set of N agents at the tth moment is denoted as (s 1,t ,s 2,t ,…,s n,t ,...,s N,t ).

[0126] Table 1 Status parameters

[0127]

[0128] ②Action space construction

[0129] Define the action of agent n at time t as a n,t ∈A, where the action space includes the heating valve openings of all hot zones. The action set of N agents at time t is denoted as (a 1,t ,a 2,t ,...,a n,t ,…,aN,t ). The valve opening range is [0,1].

[0130] ③Reward function design

[0131] According to the established multi-thermal zone thermal-hydraulic coupling model, the goal of intelligent control is to take into account the principle of user autonomous control, to control the indoor temperature while meeting user needs, without considering the operating cost of the system, and only considering a single goal. Therefore, when designing the reward function of the intelligent agent, it is still necessary to meet the user's room temperature needs as the goal, and use the reward function of reinforcement learning to guide the action preference of the intelligent agent. The reward functions of all intelligent agents are consistent. At time t, when the actual indoor temperature and the set temperature difference are between [-3,3], the reward function is set according to the Gaussian distribution function, with the expectation set to 0 and the variance set to 1. On this basis, the Gaussian distribution is scaled so that its maximum value is 2 and its minimum value is -1, such as Figure 5 When the difference between the actual indoor temperature and the set temperature exceeds the range of [-3,3], the reward function is -5, and the reward function is shown in formula (13):

[0132]

[0133] That is, all intelligent agents are in a cooperative relationship, the interests of all intelligent agents are consistent, and they have the same reward function.

[0134] (2) Constructing a multi-agent deep reinforcement learning training control framework

[0135] The Proximal Policy Optimization (PPO) algorithm is used, and the Actor-Critic dual network architecture is adopted. The Actor network (policy network) is used to guide the agent to take actions, represented by π, and the parameter θ is randomly initialized; the Critic network (value network) is used to guide the update of the policy network, represented by V, and its parameter is ω. In each environmental interaction, the agent observes the current state s t , select an action a from the current agent network's policy π t , then apply the action and immediately get the reward R t and transition to the next state s t+1 The reward is maximized by training the policy network and the value network. The updating process of the policy network and the value network involves calculating their respective loss functions; L CLIP (θ) is the strategy loss function, and L(ω) is the value loss function. The PPO algorithm process is as follows Figure 5 As shown (the batch size in the figure is the number of samples selected for one training);

[0136] The communication architecture of multi-agent deep reinforcement learning refers to how agents in a multi-agent system exchange information and collaborate to achieve a common goal or task. This invention adopts a communication framework with centralized training and decentralized execution. Figure 6 As shown in the figure, during the training phase, all agents can share each other's state, action and reward information, and can access the global environment state. During the execution phase, each agent makes decisions independently based on its own local observation information and cannot access the observation and action information of other agents. A multi-agent deep reinforcement learning algorithm is used to solve the problem of intelligent control of indoor temperature in heating buildings.

[0137] Build and train the policy network and value network of a complete multi-agent deep reinforcement learning algorithm. The number of hidden layers in the policy network and the value network is 1, and the number of neurons in the hidden layer is 64. The input layer and the output layer depend on the dimensions of the state space and the action space.

[0138] Through the multi-agent deep reinforcement learning algorithm, the room temperature of the heating building with thermal-hydraulic coupling in multiple thermal zones is intelligently controlled, the valve opening of each thermal zone is solved, and the valve is intelligently controlled according to the user's indoor temperature setting value to meet the user's personalized needs.

[0139] Here are some examples:

[0140] Taking an actual heating building in Qinhuangdao City as an example, the specific implementation method of this method is as follows:

[0141] The specific implementation process can be divided into four steps: constructing a hydraulic and thermal coupling simulation model of the heating building, constructing a multi-agent deep reinforcement learning environment based on the Markov game process, constructing a multi-agent deep reinforcement learning training and control framework, and performing intelligent control of the room temperature of the heating building.

[0142] Step 1: Construct a hydraulic and thermal coupling simulation model for heating buildings

[0143] A typical heating building in Qinhuangdao City was selected as the research object. The building covers an area of ​​749.25m 2 , with a construction area of ​​5244.75m 2 The building has 7 floors, the height of the 1st to 7th floors is 3.0 meters, and the building height is 22.5 meters. The thermal zones of each single floor of the building are hydraulically coupled, and the layers are hydraulically decoupled. The room with hydraulic coupling on the 7th floor is taken as the research object.

[0144] Using graph theory methods, the target building hydraulic network is topologically transformed into a directed weighted graph. Heat users are abstracted as pipe segments. It is assumed that there are 44 nodes and 57 pipes in the network.

[0145] Based on the above description, a hydraulic model of the heating network is constructed to accurately reflect the dynamic hydraulic characteristics of the heating network. The room is divided into thermal zones, such as Figure 7 As shown (room means room, and one room corresponds to one hot zone).

[0146] According to the heating network model of the heating building, the measured and calculated data of the heating building for a total of 7 days from January 18, 2024 to January 24, 2024, including the hourly indoor temperature, outdoor temperature, room heating and solar radiation of rooms 1-10 are used as the input of the thermal network model. The outdoor temperature and surface horizontal radiation are as follows Figure 8 As shown. The heat capacity, thermal resistance and solar radiation coefficient of the building thermal network model are identified, where U is the inverse of the thermal resistance, as shown in Table 2. The building thermal network model is constructed based on the heat capacity, thermal resistance and solar radiation coefficient, and the indoor temperature of the room is calculated based on the outdoor temperature, room heating and solar radiation.

[0147] Table 2 Identification results of thermal capacity, thermal resistance and solar radiation coefficient

[0148] zone 1 2 3 4 5 6 7 8 9 10 U1,n 0.001 0.011 U2,n 0.000054 0.011 U3,n 0.011 0.00087 0.011 U4,n 0.011 0.0008 0.011 U5,n 0.011 0.0016 0.011 U6,n 0.011 0.0005 0.011 U7,n 0.011 0.0026 0.011 U8,n 0.011 0.00011 0.011 U9,n 0.011 0.0038 U10,n 0.011 0.0002 Un,e 0.067 0.048 0.071 0.077 0.066 0.088 0.061 0.08 0.038 0.11 Un,n,e 0.3 0.24 0.07 0.24 0.21 0.24 0.39 0.24 0.41 0.25 Cn 0.84 0.25 0.33 0.54 0.33 0.92 0.44 1.2 0.5 0.45 Cn,e 50 20 50.4 30 41.5 40.21 49.6 20 48.5 40 an 2 0.0002 0.1 - 0.1 - 0.1 0.0001 0.1 - an,e 0.1 0.0001 0.8 - 1.2 - 20 0.0001 20 -

[0149] The mean absolute error (MAE) and root mean square error (RMSE) are calculated based on the calculated indoor temperature and the measured indoor temperature of the building hydraulic and thermal coupling simulation model. The MAE and RMSE of the model prediction results of all rooms are further calculated as shown in Table 3. The calculation results of MAE are between 0.08 and 0.31, and the calculation results of RMSE are between 0.11 and 0.36. It can be seen that the overall calculation accuracy of the model is high and the error is small. The model can predict the indoor temperature well.

[0150] Table 3 MAE and RMSE of calculation results of all room models

[0151] Room Number MAE RMSE room1 0.13 0.16 room2 0.18 0.21 room3 0.09 0.11 room4 0.13 0.19 room5 0.16 0.19 room6 0.08 0.11 room7 0.31 0.36 room8 0.14 0.16 room9 0.24 0.30 room10 0.19 0.26

[0152] Step 2: Build a multi-agent deep reinforcement learning environment based on the Markov game process

[0153] ① State space construction

[0154] Define the state observation of agent n at time t as s n,t ∈S, which includes weather data, indoor air temperature and indoor air set temperature, as shown in Table 4. Denoted as s n,t =[T in ,T set ] n The state set of N agents at the tth moment is denoted as (s 1,t ,s 2,t ,...,s n,t ,…,sN,t ).

[0155] Table 4 Status parameters

[0156]

[0157] ②Action space construction

[0158] Define the action of agent n at time t as a n,t ∈A, where the action space includes the heating valve openings of all hot zones. The action set of N agents at time t is denoted as (a 1,t ,a 2,t ,...,a n,t ,...,a N,t ). The valve opening range is [0,1].

[0159] ③Reward function design

[0160] According to the established multi-thermal zone thermal-hydraulic coupling model, the goal of intelligent control is to take into account the principle of user autonomous control, to control the indoor temperature while meeting user needs, without considering the operating cost of the system, and only considering a single goal. Therefore, when designing the reward function of the intelligent agent, it is still necessary to meet the user's room temperature needs as the goal, and use the reward function of reinforcement learning to guide the action preference of the intelligent agent. The reward functions of all intelligent agents are consistent. At time t, when the actual indoor temperature and the set temperature difference are between [-3,3], the reward function is set according to the Gaussian distribution function, with the expectation set to 0 and the variance set to 1. On this basis, the Gaussian distribution is scaled so that its maximum value is 2 and its minimum value is -1, such as Fig. 9 When the actual indoor temperature and the set temperature difference exceeds the range of [-3,3], the reward function is -5, and the reward function is shown in the following formula:

[0161]

[0162] That is, all intelligent agents are in a cooperative relationship, the interests of all intelligent agents are consistent, and they have the same reward function.

[0163] Step 3: Build a multi-agent deep reinforcement learning training control framework

[0164] This paper uses pytorch to build a multi-agent deep reinforcement learning algorithm. The PPO algorithm is adopted. According to the agent deep reinforcement learning training control framework, the multi-agent deep reinforcement learning algorithm training and control process is constructed. The parameter settings are shown in Table 5.

[0165] Table 5 Multi-agent deep reinforcement learning hyperparameters

[0166] parameter value Total rounds 4000 Learning Rate 5e-4 Discount Factor 0.99 Generalized odds estimate 0.95 Minimum batch size 1 Update iteration round 15 Shear proxy coefficient 0.2 Clipping loss of value function True Entropy coefficient 0.01 Value function coefficients 1 Maximum norm for gradient clipping 10

[0167] Step 4: Intelligently control the room temperature of the heating building.

[0168] The MAPPO algorithm is used to control the heating system. The environmental parameters include outdoor temperature, water supply temperature, and solar radiation parameters. During the control process, the water supply temperature is set to 45°C. The outdoor temperature and solar horizontal radiation of the control target are as follows: Fig.10 As shown, the horizontal solar radiation has the same meaning as the horizontal surface radiation. In the MAPPO algorithm, ten agents are used to correspond to the inlet valve openings of ten rooms (thermal zones), and the upper limit of valve action per day is 24 times. The hourly valve openings for the next 24 hours are obtained through the MAPPO algorithm based on the demand-side response.

[0169] The reward value convergence curve of the MAPPO algorithm is as follows: Fig.11 As shown in the figure (Episodes in the horizontal axis represents the number of training times, Reward in the vertical axis represents the reward value, and Agent represents the agent), when the model is trained for 800 times, the convergence speed slows down. When the model is trained for about 2000 times, the reward value of all agents converges and stabilizes at about 40. The algorithm converges within 1 hour.

[0170] like Figure 12(a)-Figure 12(j) Shown are the control results of ten rooms (thermal zones) (the red line in the figure indicates the control temperature, and the green line indicates the set temperature). Different thermal zones have different indoor set temperatures. There is also hydraulic coupling and thermal coupling in different rooms. Under complex control conditions, the algorithm gives the valve opening of each room per hour for the next 24 hours of the control target. It can be seen from the indoor temperature change curve that the control results obtained by the multi-agent deep reinforcement learning algorithm can make the indoor temperature of the room fluctuate within the range of ±1°C of the set value. The reinforcement learning algorithm has learned a reasonable control strategy.

[0171] The present application has been disclosed as above with preferred implementation cases, but it is not intended to limit the present application. Any technician familiar with the profession can make slight changes or modifications to the above-disclosed structures and technical contents without departing from the scope of the technical solution of the present application, and the equivalent implementation cases with equivalent changes are still within the scope of the technical solution of the present application.

Claims

1. A method for intelligently controlling room temperature of a heating building, characterized in that: The following steps are involved: S1. Construct hydraulic and thermal models of the heating building pipe network and form a hydraulic and thermal coupling model; The hydraulic and thermal coupling model includes a pipe network hydraulic model, a regulating valve model, a heat exchanger model and a building thermal network model; S2. Use the hydraulic-thermal coupling model to build a multi-agent deep reinforcement learning model, and define one agent for each thermal zone of the heating building; S3. Utilize the multi-agent deep reinforcement learning model to solve the valve opening of each thermal zone, and intelligently adjust the valve according to the user's indoor temperature setting value to achieve room temperature control for heat users in the heating building.

2. According to claim 1, a method for intelligently controlling room temperature of a heating building is characterized in that: The hydraulic model construction process in step S1 is: First, the hydraulic model of the pipe network is constructed: each heat user is abstracted as a pipe segment, and it is assumed that there are n+1 nodes and b pipes in the pipe network; the pipe network flow continuity equation is shown in formula (1): A n×b G b×1 =q n×1 (1) Where: A is the basic correlation matrix of the pipe network; G is the flow vector of the pipe section; q is the traffic of each node; The pipe network loop equation is shown in formula (2): B (b-n)×b H b×1 =0 (2) Where: B is the basic loop matrix of the pipe network; H is the pressure difference vector of the pipe section; The head loss equation is shown in formula (3): H b×1 =HW b×1 +WZ b×1 -DH b×1 (3) Where: HW is the pipeline network pressure loss vector; WZ is the water head difference vector at both ends of the pipe section; DH is the pump head vector; in: HW b×1 =S b×b |G b×1 |G b×1 (4) DH b×1 =a1|G b×1 |G b×1 +a2G b×1 +a3 (5) Where: S b×b is the pipe resistance coefficient matrix; a1, a2, a3 are the pump characteristic parameter matrices; Secondly, construct a control valve model: calculate the control valve resistance characteristics according to the pressure difference and flow before and after the valve, as shown in formula (6); Where: ΔP v is the pressure difference before and after the regulating valve, Pa; s v is the resistance coefficient of the regulating valve at a certain opening, Pa / (m 3 / h) 2 ; V v is the volume flow through the regulating valve, m 3 / h.

3. According to claim 1, a method for intelligently controlling room temperature of a heating building is characterized in that: The heat exchanger model building process in step S1 is: According to the law of conservation of energy, the heat transfer process of the radiator is expressed by equation (7): Where: m t is the heat flow rate in the radiator at time t, kg / s; c is the specific heat capacity of water, J / (kg·℃); T g t is the water supply temperature at time t, °C; T h t is the return water temperature at time t, °C; T in t is the indoor temperature at time t, ℃; F is the heat transfer area of ​​the radiator, m 2 ; k is the heat transfer coefficient of the radiator, W / (m 2 ·℃).

4. According to claim 1, a method for intelligently controlling room temperature of a heating building is characterized in that: The building thermal network model construction process in step S1 is: First, the heating building is divided into building thermal zones. A single hot zone at node T n The heat balance equation is: Q n =Q h,n +Q i,n +Q s,n (8) Where: Q n is the heat load of the hot zone, W; Q h,n is the heat load of the heating system, W; Q s,n is the solar radiation heat gain in the hot zone, W; Q i,n Heat gained by lighting for personnel and equipment inside the hot zone, W; Where: Q s,n =α n q s (9) Where: α n is the solar radiation coefficient of the hot zone, m 2 ; q s is the solar radiation per square meter of horizontal surface, W / m 2 ; Then: Node T of hot zone n n The heat transfer differential equation is shown in formula (10): Where: n, m are the hot zone numbers; T n is the indoor temperature of hot zone n, °C; T m is the indoor temperature of hot zone m, °C; T n,e is the temperature of the enclosure structure in hot zone n, °C; T ext is the outdoor temperature, °C; C n is the heat capacity of hot zone n, J / K; R n,m is the thermal resistance of adjacent hot zones, K / W; R n,e is the heat transfer resistance between the hot zone n and the surface of the enclosure structure, K / W; R n,n is the thermal resistance of the external window in hot zone n, K / W; Then: Node T of hot zone n n,e The heat transfer differential equation is shown in formula (11): Where: C n,e is the heat capacity of the enclosure structure in hot zone n, J / K; R n,n,e is the thermal resistance between the heat zone n enclosure and the external environment, K / W; Q n,e is the solar radiation heat gain of the enclosure structure in hot zone n, W; Where: Q n,e =α n,e q s (12) Where: α n,e is the solar radiation coefficient of the enclosure structure in hot zone n, m 2 .

5. According to claim 1, a method for intelligently controlling room temperature of a heating building is characterized in that: The construction process of the multi-agent deep reinforcement learning model in step S2 is as follows: S21. Construct a Markov game process for intelligent control of indoor temperature in multiple thermal zones; Contains state space construction, action space construction and reward function design; S22. Construct a multi-agent deep reinforcement learning training and control framework to realize information exchange and collaboration between agents. During the training phase, all agents can share each other's status, action and reward information, and can access the global environment status. During the execution phase, each agent makes decisions independently based on its own local observation information and cannot access the observation and action information of other agents.

6. According to claim 5, a method for intelligently controlling room temperature of a heating building is characterized in that: The state space construction process is as follows: define the state observation of the agent at time t as s n,t ∈S, which includes weather data, indoor air temperature and indoor air set temperature, denoted as s n,t =[T in ,T set ] n , the state set of N agents at the tth moment is recorded as (s 1,t ,s 2,t ,...,s n,t ,…,s N,t ); The process of constructing the action space is as follows: define the action of the agent at time t as a n,t ∈A, where the action space includes the heating valve openings of all hot zones, and the action set of N agents at the tth moment is recorded as (a 1,t ,a 2,t ,...,a n,t ,...,a N,t ), where the valve opening range is [0,1]; The reward function design process is as follows: the reward function of reinforcement learning is used to guide the action preference of the agent. The reward functions of all agents are consistent. At time t, the temperature difference between the actual indoor temperature and the set value is set between [-3, 3]. The reward function is set according to the Gaussian distribution function, with the expectation set to 0 and the variance set to 1. On this basis, the Gaussian distribution is scaled so that its maximum value is 2 and its minimum value is -1. When the temperature difference between the actual indoor temperature and the set value exceeds the range of [-3, 3], the reward function is -5. The reward function is shown in formula (13):

7. According to claim 6, a method for intelligently controlling room temperature of a heating building, characterized in that: In step S22, the PPO algorithm and the Actor-Critic dual network architecture are used. The Actor network is a policy network used to guide the agent to take actions, represented by π, and the parameter θ is randomly initialized; the Critic network is a value network used to guide the update of the policy network, represented by V, and its parameter is ω. In each environmental interaction, the agent observes the current state s t , select an action a from the policy network π of the current agent network t , then apply the action and immediately get the reward R t and transition to the next state s t+1 , by training the policy network and the value network to maximize the reward.

8. The method for intelligently controlling room temperature of a heating building according to claim 7, characterized in that: In step S22, a complete policy network and value network of the multi-agent deep reinforcement learning algorithm are built and trained. The number of hidden layers of the policy network and the value network is 1, and the number of neurons in the hidden layer is 64. The input layer and the output layer depend on the dimensions of the state space and the action space.

Images