Central air conditioner load low-carbon optimization method based on deep reinforcement learning and game theory

Abstract

The invention provides a central air-conditioning load and low-carbon optimization method based on deep reinforcement learning and a game theory. The method comprises the following steps of: firstly, modeling multiple uncertainties such as unit parameters, environment temperature and source load of each part in the central air conditioning system by adopting methods such as interval number and the like; secondly, according to the carbon emission, the power load and the mechanism of the system, the operation mode of the central air conditioning system is optimized by taking the lowest comprehensive operation cost and low carbon as targets. According to the invention, a novel multi-objective optimization algorithm of ''offline training'' and ''online guidance'' is carried out through a strategy of combining a deep Q network of deep reinforcement learning and an interval multi-objective particle swarm optimization algorithm, and the low-carbon low-energy-consumption optimization scheduling problem of the central air conditioner under multiple uncertain factors is efficiently solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of energy management and relates to a central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory. Background Art

[0003] In various large buildings, the energy consumed by central air conditioning accounts for a large proportion, especially in seasons with high or low temperatures. The increase in the frequency of central air conditioning use will further increase the proportion of energy consumption. The central air conditioning system is more complex than other building equipment. It is not a simple single device or a simple connection of equipment, but multiple devices are connected to each other based on the cold and hot principles of the central air conditioning system. Therefore, it is very difficult to transform this system structure and achieve air conditioning energy saving, and the effect may not be satisfactory. To truly achieve comprehensive energy saving and low carbonization of the air conditioning system, it is necessary to consider the energy consumption composition of each part of the central air conditioning, the randomness and volatility of indoor and outdoor temperatures, and adopt superior control methods. Comprehensively consider the various components such as the air conditioning host, cooling water pump, and chilled water pump to reasonably control the operation of the central air conditioning system. Improving energy utilization efficiency and reducing carbon emissions are the key to promoting the central air conditioning system industry to develop in a more sustainable and low-carbon direction.

[0004] At present, many scholars at home and abroad have carried out research on improving the control methods of central air-conditioning systems under the uncertainty of multiple factors. The research contents can be mainly divided into the following two categories:

[0005] (1) The first type of research focuses on uncertainty research methods, including stochastic programming, capacity scheduling, scenario analysis, and robust optimization.

[0006] (2) The second type of research focuses on reinforcement learning methods, which integrate perception, learning, and decision-making into intelligent agents.

[0007] Among the control methods of the central air-conditioning system mentioned above, although the capacity scheduling of the first type of method is to better adapt to the system demand, it may bring high costs, including hardware upgrades, equipment purchases and maintenance costs. This may become a burden in some economically limited scenarios, especially when the capacity needs to be adjusted frequently. Stochastic programming and scenario analysis methods usually rely on probabilistic models or scenario settings when dealing with uncertainty. However, due to the complexity of the real world, these methods may not accurately capture all possible sources of uncertainty, resulting in challenges to the accuracy of the results. Robust optimization uses the worst-case scenario to optimize system operation, which may make the results too conservative, even if the worst case may not be encountered in actual operation. This may not fully utilize the system in terms of performance and resource utilization. Interval optimization expresses uncertainty as an interval number, but when the interval is too large, it may lead to a decrease in the accuracy of the results. At the same time, in some cases, the algorithm may be difficult to converge due to the uncertainty of the interval, affecting the optimization effect.

[0008] The second type of method uses a reinforcement learning algorithm, which does not require precise mathematical formulas and codes to characterize the physical model of the central air conditioner for analysis. Therefore, in the construction of the central air conditioning system and its dynamic characteristics, solving some high-dimensional complex problems is better than traditional model-based algorithms. However, further research is needed in terms of parameter optimization to improve algorithm performance. In view of this, the present invention proposes a central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory. Summary of the invention

[0009] The technical solution adopted by the present invention is:

[0010] The purpose of the present invention is to develop a central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory under the influence of multiple uncertainties. Under the influence of uncertain factors such as ambient temperature, equipment parameters, and response degree, the model is modeled using methods such as interval numbers. With the goal of optimizing energy loss and carbon emission costs, an interval multi-objective game theory algorithm based on deep reinforcement learning is proposed to solve it. Through deep reinforcement learning training, the interval multi-objective is guided to perform Pareto principle optimization, improve the algorithm's optimization efficiency and convergence, and achieve optimal operation of the system. The specific scheme of the present invention is as follows:

[0011] A central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory, the method comprising the following steps:

[0012] (1) Constructing the mathematical model of the core link of central air conditioning (1.1) The fan uncertainty model is constructed as follows:

[0013] Q w,t =P w,t η wΔt (1)

[0014]

[0015]

[0016] Where: Q w,t P is the fan output energy consumption at time t; w,t is the electric power consumed by the fan at time t; η w is the fan output efficiency; a1~e1 are the fan fitting coefficients; β w Fan part load factor; is the actual fan air volume range; m w-N is the rated flow rate of the fan;

[0017] (1.2) The uncertainty model of the chilled water pump is constructed as follows:

[0018] Q co,t =η co P co,t Δt (4)

[0019]

[0020] Where: Q co,t P is the output energy consumption of the chilled water pump at time t; co,t is the electric power consumed by the chilled water pump at time t; η co is the chilled water pump output efficiency; a2~d2 are the chilled water pump fitting coefficients; is the chilled water flow range; m co-N is the rated flow rate of the chilled water pump;

[0021] (1.3) The uncertainty model of the refrigerator is constructed as follows:

[0022]

[0023] In the formula, p r is the refrigerator power; C is the range of available cooling capacity of the refrigerator; cop-N is the rated energy efficiency ratio of the refrigerator; η r2 ~η r3 It is the relationship curve between temperature and load;

[0024] (1.4) Construction of cooling water pump and cooling tower model

[0025] p c =p c-N (8)

[0026] p ct =p ct-N (9)

[0027] In the formula, p c is the cooling water pump power; p c-N is the rated power of the cooling water pump; p ct is the cooling tower power; p ct-N is the rated power of the cooling tower;

[0028] (2) Construction of a system model considering carbon trading and multi-objective optimization scheduling of energy consumption ranges

[0029] Considering the overall uncertainty of the central air-conditioning system, multiple uncertainties and multivariate scheduling objectives are calculated for energy consumption and carbon emission factors, and an interval multi-objective optimization scheduling target model is constructed. The specific description is as follows:

[0030] Central air conditioning system response model:

[0031]

[0032] In the formula, p i,t is the power of each link of the central air conditioner that does not respond to the load; p i,n The opening time of each link of central air conditioning start,i ,t end,i The actual power between sum It is the sum of the actual power of central air conditioners.

[0033] The carbon emissions of the central air-conditioning system are shown in formula (12):

[0034]

[0035] In the formula, E c is the total carbon emissions of the central air-conditioning system; q is the carbon emission coefficient corresponding to equipment q in the central air-conditioning system; p q, t is the energy consumption of device q at time t, and T is the operating time of the system equipment.

[0036] Preferably, the method further comprises the steps of:

[0037] (3) Solution of central air conditioning model based on deep reinforcement learning and game theory

[0038] By coupling deep reinforcement learning with the IMOPSO algorithm, we learn the relationship between the state, action, and reward of the central air-conditioning model, guide the interval population, and achieve the approximation and convergence of the Pareto frontier. The specific solution steps are as follows: (3.1) The state function St of the agent:

[0039] S t =f(x,v,P,G) (13)

[0040] In the formula, the position of the particle is x, the speed is v, the historical optimal position of the individual particle is P, and the optimal position of the particle group is G;

[0041] (3.2) Agent action function At:

[0042] A t ={w i ,a i} (14)

[0043] Where: Wi is the initial network weight; ai is the action under weight Wi;

[0044] (3.3) The reward function R(S, A) is set as the particle swarm MOPSO particle fitness and its interval length:

[0045]

[0046] Where R(S, A) is the reward function; λR is the R-th class coefficient; R belongs to {D, L}, D is the fitness and L is the interval length; l is the current reward of the R-th class; l is the maximum optimized reward of the R-th class; T is the fitness reward value; T1 is the interval length reward value; Fn is the n-th sub-goal interval function; ρ n is the weight of the target n; D is the interval median function; L is the interval length function;

[0047] (3.4) Reinforcement learning agent, which makes action decisions based on the current state through action strategy and decision network, where the action strategy and decision network are:

[0048] Q(s,a)=R(s,a)+γ∑ s' P(s′|s,a)∑ a' π(a′|s′)Q(s′,a′) (17)

[0049] Where: s is the current state; a is the action selected in state s; R(s,a) is the immediate reward obtained; γ is a discount factor, indicating the importance of future rewards. p(s′|s,a) is the probability of transitioning to the next state s'; π(a′|s′) is the strategy of selecting action a' in s';

[0050] (3.5) Model training

[0051] During the training process, IMOPSO transmits state information to the agent and makes corresponding decision actions and performs reward calculation and network training optimization on the current state. The network training process includes experience replay mechanism and parameter freezing mechanism to reduce the correlation between samples and improve the stability of training.

[0052] (3.6) Guidance

[0053] During the guidance process, the trained network is matched one by one with the initialized population, and the network is called repeatedly to guide the movement of individual particles until the iterative optimization is completed and convergence is completed; at the same time, the trained parameters have a certain universality in different scenarios.

[0054] The present invention has the following beneficial technical effects:

[0055] The present invention provides a central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory, which realizes the optimal scheduling of the central air conditioning system under the condition of multiple uncertain influences. The method has the advantages of mutual cooperation among various links of central air conditioning, high stability, good dynamic performance, strong adaptability, etc. It has good application prospects in improving the power consumption of central air conditioning systems and reducing the carbon emissions of central air conditioning systems. At the same time, it will have certain applicability in establishing an uncertain energy management system model and considering various uncertain factors. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The present invention is further described below with reference to the accompanying drawings and examples:

[0057] Figure 1 Central air conditioning model.

[0058] Figure 2 A central air conditioning model solution framework based on deep reinforcement learning and game theory.

[0059] Figure 3 Deep reinforcement learning and game theory algorithms. DETAILED DESCRIPTION

[0060] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The present invention comprises the following steps:

[0061] Design uncertainty models for each link of central air conditioning: The central air conditioning system mainly includes several components such as air conditioning host, cooling water pump, and refrigeration water pump. The fan energy consumption is related to the air supply volume, the operating power of the water pump is related to the water flow rate, and the refrigerator operates according to the cooling capacity required by the system.

[0062] Construct a carbon trading and energy consumption interval multi-objective optimization scheduling model: Taking into account the multiple uncertainties in the central air-conditioning system as a whole, the energy consumption and carbon emission factors are calculated, multiple uncertainties and multiple scheduling objectives are calculated, and an interval multi-objective optimization scheduling model is constructed.

[0063] Deep reinforcement learning interval multi-objective optimization: Deep reinforcement learning and interval multi-objective particle swarm optimization (IMOPSO) algorithm are used to learn the relationship between the state, action, reward, etc. of the central air-conditioning model, guide the interval population, and achieve the approximation and convergence of the Pareto frontier.

[0064] Training process: During the training process, IMOPSO passes the state to the agent to obtain the decision action Q(S,A) and calculates the reward and optimizes the network training for the current state. The network training process is based on the experience replay mechanism, using the agent to replay the (S t ,A t ,R t ,S t+1 )The array is stored in the memory pool, which is kept updated in real time through continuous storage and updating.

[0065] Guidance process: In the guidance process, the trained network is matched with the initialized population one by one, and the reward function R(S,A) is set as the population IMOPSO particle fitness and interval length, and the network is repeatedly called to guide the movement of individual particles. The Pareto strategy is used until the iterative optimization converges.

[0066] The specific implementation steps of the present invention are:

[0067] 1. Construction of mathematical model of the core link of central air conditioning: The central air conditioning system consists of various components such as air conditioning fan, cooling water pump, and refrigeration water pump. The indoor unit cools the indoor air through the evaporator, and the cooled air is distributed to each room through the air duct system. The evaporative condenser forms a closed refrigeration circuit through the compressor, evaporator, condenser and expansion valve. The compressor is mainly used to compress the refrigerant, so that it is heated and pressurized to flow to the condenser. Figure 1 The uncertainty model of the mutual cooperation of each link in the central air-conditioning working mode is obtained and described as follows.

[0068] 1.1 Construction of fan uncertainty model: The energy consumption of the fan in the central air-conditioning system is related to the air supply volume. Considering the uncertainty of the actual air volume of the fan, the fan model is shown in formula (1-3).

[0069] Q w,t =P w,t η w Δt (18)

[0070]

[0071] Where: Q w,t P is the fan output energy consumption at time t;w,t is the electric power consumed by the fan at time t; η w is the fan output efficiency; a1~e1 are the fan fitting coefficients; β w Fan part load factor; is the actual fan air volume range; m w-N is the rated flow rate of the fan.

[0072] 1.2 Construction of uncertainty model of chilled water pump: The operating power of the chilled water pump in the central air-conditioning system is related to the chilled water flow rate. Considering the uncertainty of the chilled water flow rate, its energy consumption model is shown in Equations (4)-(6).

[0073] Q co,t =η co P co,t Δt (21)

[0074]

[0075] Where: Q co,t P is the output energy consumption of the chilled water pump at time t; co,t is the electric power consumed by the chilled water pump at time t; η co is the chilled water pump output efficiency; a2~d2 are the chilled water pump fitting coefficients; is the chilled water flow range; m co-N Rated flow rate of the chilled water pump.

[0076] 1.3 Construction of uncertainty model of refrigeration machine: The refrigeration machine provides the cooling capacity required by the central air-conditioning system. Considering the uncertainty of available cooling capacity, its model is shown in (7):

[0077]

[0078] In the formula, p r is the refrigerator power; C is the range of available cooling capacity of the refrigerator; cop-N is the rated energy efficiency ratio of the refrigerator; η r2 ~η r3 This is the relationship curve between temperature and load.

[0079] 1.4 Construction of cooling water pump and cooling tower model: In the central air conditioning system, the cooling water pump adopts a fixed frequency water pump. In actual situations, the energy consumption of the cooling tower and cooling water pump is relatively small. In order to reduce the complexity of the model, the model is simplified and the rated power is taken. The model is shown in (8) and (9):

[0080] p c =p c-N (25)

[0081] p ct =pct-N (26)

[0082] In the formula, p c is the cooling water pump power; p c-N is the rated power of the cooling water pump; p ct is the cooling tower power; p ct-N is the rated power of the cooling tower.

[0083] Surface cooler model construction: The main function of the surface cooler is to connect the air circulation and the chilled water circulation, in which there is no power consumption.

[0084] 2. Construction of a system model considering carbon trading and multi-objective optimization scheduling of energy consumption intervals: Considering the overall multiple uncertainties in the central air-conditioning system, multiple uncertainties and multi-dimensional scheduling objectives are calculated for energy consumption and carbon emission factors, and an interval multi-objective optimization scheduling target model is constructed, which is described as follows:

[0085] The response model of each link of the central air-conditioning system is shown in (10) and (11):

[0086]

[0087] In the formula, p i,t is the power of each link of the central air conditioner that does not respond to the load; p i,n The opening time of each link of central air conditioning start,i ,t end,i The actual power between sum It is the sum of the actual power of central air conditioners.

[0088] The carbon emissions of the central air-conditioning system are shown in formula (12):

[0089]

[0090] In the formula, E c is the total carbon emissions of the central air-conditioning system; q is the carbon emission coefficient corresponding to equipment q in the central air-conditioning system; p q,t is the energy consumption of device q at time t, and T is the operating time of the system equipment.

[0091] 3. Central air conditioning model solution framework based on deep reinforcement learning and game theory: Deep reinforcement learning is coupled with the IMOPSO algorithm to learn the relationship between the state, action, and reward of the central air conditioning model, guide the interval population, and achieve the approximation and convergence of the Pareto frontier; as shown in the attached Figure 2 shown.

[0092] 3.1 The state function St of the agent:

[0093] S t=f(x,v,P,G) (30)

[0094] In the formula, the position of the particle is x, the velocity is v, the historical optimal position of the individual particle is P, and the optimal position of the particle group is G.

[0095] 3.2 Agent action function At:

[0096] A t ={w i ,a i} (31)

[0097] Where: Wi is the initial network weight; ai is the action under weight Wi.

[0098] 3.3 Reward function R(S, A), set as the particle swarm MOPSO particle fitness and its interval length:

[0099]

[0100] Where R(S, A) is the reward function; λR is the R-th class coefficient; R belongs to {D, L}, D is the fitness and L is the interval length; l is the current reward of the R-th class; l is the maximum optimized reward of the R-th class; T is the fitness reward value; T1 is the interval length reward value; Fn is the n-th sub-goal interval function; ρ n is the weight of the target n; D is the interval median function; L is the interval length function.

[0101] 3.4 Reinforcement learning agent, the agent mainly makes action decisions based on the current state through action strategy and decision network, where the action strategy and decision network are:

[0102] Q(s,a)=R(s,a)+γ∑ s' P(s′|s,a)∑ a' π(a′|s′)Q(s′,a′) (34)

[0103] Where: s is the current state; a is the action selected in state s; R(s,a) is the immediate reward obtained; γ is a discount factor, indicating the importance of future rewards. p(s′|s,a) is the probability of transitioning to the next state s'; π(a′|s′) is the strategy for selecting action a' in s'.

[0104] 3.5 Training process: During the training process, IMOPSO transmits state information to the agent and makes corresponding decision actions and performs reward calculation and network training optimization on the current state. The network training process includes experience replay mechanism and parameter freezing mechanism to reduce the correlation between samples and improve the stability of training.

[0105] 3.6 Guidance process: In the guidance process, the trained network is matched with the initialization population one by one, and the network is repeatedly called to guide the movement of individual particles until the iterative optimization is completed; as shown in the attached Figure 3 As shown, the parameters trained simultaneously have certain universality in different scenarios.

Claims

1. A central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory, characterized in that: The method comprises the following steps: (1) Constructing a mathematical model for the core links of central air conditioning (1.1) The wind turbine uncertainty model is constructed as follows: Q w,t =P w,t the w Δt(1) Where: Q w,t P is the fan output energy consumption at time t; w,t is the electric power consumed by the fan at time t; η w is the fan output efficiency; a1~e1 are the fan fitting coefficients; β w Fan part load factor; is the actual fan air volume range; m w-N is the rated flow rate of the fan; (1.2) The uncertainty model of the chilled water pump is constructed as follows: Q co,t =η co P co,t Δt(4) Where: Q co,t P is the output energy consumption of the chilled water pump at time t; co,t is the electric power consumed by the chilled water pump at time t; η co is the chilled water pump output efficiency; a2~d2 are the chilled water pump fitting coefficients; is the chilled water flow range; m co- N is the rated flow rate of the chilled water pump; (1.3) The uncertainty model of the refrigerator is constructed as follows: In the formula, p r is the refrigerator power; C is the range of available cooling capacity of the refrigerator; cop-N is the rated energy efficiency ratio of the refrigerator; η r2 ~η r3 It is the relationship curve between temperature and load; (1.4) Construction of cooling water pump and cooling tower model p c =p c-N (8) p ct =p ct-N (9) In the formula, p c is the cooling water pump power; p c-N is the rated power of the cooling water pump; p ct is the cooling tower power; p ct-N is the rated power of the cooling tower; (2) Construction of a system model considering carbon trading and multi-objective optimization scheduling of energy consumption ranges Considering the overall uncertainty of the central air-conditioning system, multiple uncertainties and multivariate scheduling objectives are calculated for energy consumption and carbon emission factors, and an interval multi-objective optimization scheduling target model is constructed. The specific description is as follows: Central air-conditioning system response model: In the formula, p i,t is the power of each link of the central air conditioner that does not respond to the load; p i,n The opening time of each link of central air conditioning start,i ,t end,i The actual power between sum It is the sum of the actual power of central air conditioners. The carbon emissions of the central air conditioning system are shown in formula (12): In the formula, E c is the total carbon emissions of the central air-conditioning system; q is the carbon emission coefficient corresponding to equipment q in the central air-conditioning system; p q,t is the energy consumption of device q at time t, and T is the operating time of the system equipment.

2. A central air conditioning load and low-carbon optimization method based on deep reinforcement learning and game theory as claimed in claim 1, characterized in that: The method further comprises the steps of: (3) Solution of central air conditioning model based on deep reinforcement learning and game theory By coupling deep reinforcement learning with the IMOPSO algorithm, we learn the relationship between the state, action, and reward of the central air-conditioning model, guide the interval population, and achieve the approximation and convergence of the Pareto frontier. The specific solution steps are as follows: (3.1) The state function St of the agent: S t =f(x,v,P,G)(13) In the formula, the position of the particle is x, the speed is v, the historical optimal position of the individual particle is P, and the optimal position of the particle group is G; (3.2) Agent action function At: A t ={w i ,a i }(14) Where: Wi is the initial network weight; ai is the action under weight Wi; (3.3) The reward function R(S, A) is set as the particle swarm MOPSO particle fitness and its interval length: Where R(S, A) is the reward function; λR is the R-th class coefficient; R belongs to {D, L}, D is the fitness and L is the interval length; l is the current reward of the R-th class; l is the maximum optimized reward of the R-th class; T is the fitness reward value; T1 is the interval length reward value; Fn is the n-th sub-goal interval function; ρ n is the weight of the target n; D is the interval median function; L is the interval length function; (3.4) Reinforcement learning agent, which makes action decisions based on the current state through action strategy and decision network, where the action strategy and decision network are: Q(s,a)=R(s,a)+γ∑ s' P(s′|s,a)∑ a' π(a′|s′)Q(s′,a′) (17) Where: s is the current state; a is the action selected in state s; R(s,a) is the immediate reward obtained; γ is a discount factor, which indicates the importance of future rewards. p(s′|s,a) is the probability of transitioning to the next state s′; π(a′|s′) is the strategy of choosing action a′ under s′; (3.5) Model training During the training process, IMOPSO transmits state information to the agent and makes corresponding decision actions and performs reward calculation and network training optimization on the current state. The network training process includes experience replay mechanism and parameter freezing mechanism to reduce the correlation between samples and improve the stability of training. (3.6) Guidance During the guidance process, the trained network is matched one by one with the initialized population, and the network is called repeatedly to guide the movement of individual particles until the iterative optimization is completed and convergence is completed; at the same time, the trained parameters have a certain universality in different scenarios.

Images