An optimization scheduling method for river water source heat pump energy supply system based on A3C algorithm

Abstract

The application provides a kind of river water source heat pump energy supply system optimization scheduling method based on A3C algorithm, by obtaining the renewable energy output, carbon dioxide emissions, water pump efficiency, electrical load and cold and heat load and other indexes of each time period in river water source heat pump energy supply system, and dynamic weighting is carried out for each type index, the A3C algorithm of reinforcement learning is used to solve the optimization scheduling model of energy supply system, and the optimization scheduling method strategy of energy supply system in each time period is obtained, that is, the optimization scheduling of river water source heat pump energy supply system can be realized.The application integrates social comprehensive information such as user electricity consumption behavior, based on the characteristics of fast convergence and easy parameter adjustment of reinforcement learning A3C algorithm, proposes a new optimization scheduling method for river water source heat pump energy supply system, which can effectively solve the problem of large peak-valley difference of power grid in load-intensive area, and realize high energy efficiency, high reliability and low emission comprehensive energy supply.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optimization and scheduling of energy supply and demand interaction in the context of comprehensive energy, and particularly relates to a method for optimizing and scheduling a river water source heat pump energy supply system based on an A3C algorithm. BACKGROUND

[0002] With the increasing energy consumption, developing renewable energy has become an important trend to promote the development concept of "carbon peak and carbon neutral". At present, the electrification level of the user side is rapidly rising, and the load concentration effect of large cities is continuously enhanced. The electricity consumption of urban dense load areas such as residential areas continues to rise, resulting in a continuously expanding peak-valley difference. The abnormal expansion of the peak-valley difference has a significant impact on the production, quality, safety and economic operation of electric energy, as well as electric power enterprises and users. During the peak or valley period, if the power grid is unable to adjust the peak, the frequency will be low during the peak, which will reduce the quality of electric energy and affect the reliability of electricity consumption. A large peak-valley difference in load will cause the daily peak-valley of the power grid to increase, resulting in deterioration of the voltage condition.

[0003] During the peak period of power load, due to the small adjustment margin of the operation mode, the power grid is under great pressure, and the transmission and transformation equipment is heavily loaded and overloaded. In addition, the power supply reliability requirement of load-intensive areas is high, and the line corridor is tight, so the difficulty of power grid construction and transformation increases year by year, and therefore, a new solution is needed to solve the problem of the contradiction between power supply and demand in urban load-intensive areas.

[0004] As a kind of renewable heat source, the energy supply system of river water source has the advantages of high efficiency, energy saving, green environmental protection, reliable operation and the like. Compared with clean energy such as wind energy and solar energy, the river water source is less affected by seasons and sunshine, and has a broad application prospect in regional energy supply systems. The river water source heat pump technology uses the characteristics that the water temperature of river water below 13-16 meters of river surface is higher than room temperature in winter and lower than room temperature in summer, to use the river water as a cold and heat source for air conditioning systems. The river water source heat pump has high energy coefficient, strong stability and small land occupation, and is particularly suitable for load-intensive areas such as residential areas close to rivers and with large refrigeration and heating load and space shortage. However, the output power of the river water source heat pump system is still affected by natural conditions such as water quality, water temperature and weather conditions, and has certain volatility and randomness, and therefore, a correct and complete optimization and scheduling strategy needs to be formed. At present, the research on the river water source heat pump energy supply system mainly focuses on how to improve the efficiency of each device in the system, construct an evaluation index system of the river water source heat pump, and optimize the dynamic economic scheduling with the lowest system operation cost as the target.

[0005] Through the search of the prior art field, it is found that Chinese patent application No. 202111218782.2, application publication No. CN114091728A, and patent name: A power supply system optimization scheduling method and system based on river water source heat pump, the patent obtains the time-of-use electricity price, new energy output, electrical load and thermal load in the microgrid containing the river water source heat pump, and brings them into the pre-established model, and uses the improved wolf swarm algorithm to solve the coordinated optimization scheduling model of the power supply system, and obtains the scheduling strategy of the microgrid power supply system in each time period, but the patent does not consider the influence of social comprehensive information such as user behavior on the scheduling method strategy, and ignores the interaction between supply and demand. SUMMARY

[0006] The problem to be solved by the present application is to provide a river water source heat pump power supply system optimization scheduling method, which can be used for load-intensive area cold and heat load demand, and can integrate user electricity behavior to form a complete power supply system optimization scheduling method.

[0007] In order to solve the above technical problems, the present application adopts the following technical solutions:

[0008] A river water source heat pump power supply system optimization scheduling method based on A3C algorithm, comprising the following steps:

[0009] Step 1, obtaining the environmental protection index, technical index, economic index and user electricity behavior index of the river water source heat pump power supply system in each time period;

[0010] Step 2: dynamically weighting the environmental protection index, technical index, economic index and user electricity behavior index;

[0011] Step 3: modeling the river water source heat pump power supply system optimization scheduling method by relying on Markov decision process;

[0012] Step 4: based on the multi-thread asynchronous training of the reinforcement learning A3C algorithm, introducing the Bayesian optimization algorithm to learn the hyperparameters, and combining the time difference algorithm to accelerate the convergence of the algorithm, and obtaining the power supply system optimization scheduling method in each time period.

[0013] Further improvement of the technical scheme of the present application, in step 1, the environmental protection index includes renewable energy output and total renewable energy installed capacity of the power supply system, the technical index includes energy load and maximum energy load, and the economic index includes system total amount and system total load maximum value.

[0014] Further improvement of the technical scheme of the present application, in step 2, the dynamic weighting function β1 of the environmental protection index is:

[0015]

[0016] In formula (1), Pre (t) is the renewable energy output of the river water source heat pump energy supply system at time t, P re_max is the total installed capacity of renewable energy of the river water source heat pump energy supply system;

[0017] The dynamic weighting function β2 of the technical index is:

[0018]

[0019] In formula (2), P L_k (t) is the energy load of the kth energy at time t; P L_kmax represents the maximum load of the kth energy;

[0020] The dynamic weighting function β3 of the economic index is:

[0021]

[0022] In formula (3), P L (t) is the total load of the river water source heat pump energy supply system at time t; P L_max is the maximum value of the total load of the river water source heat pump energy supply system;

[0023] The dynamic weighting function β4 of the user electricity consumption behavior index is:

[0024]

[0025] In formula (4), P max is the maximum load power in each time period, P min is the minimum load power in each time period.

[0026] Further improvement of the technical scheme of the application, step 3 modeling relies on Markov decision process, define four tuples <S, A, R, f>, S represents state, A represents action strategy, R represents reward function, f represents state transition function, f is represented by deep neural network in reinforcement learning; The state S is the combination of the device state in the energy supply system and the external natural environment;

[0027] The calculation formula of the reward function R is:

[0028] R=β3C HP +β2V EQ +β1M CO2 +β4E pur (5)

[0029] In formula (5), C HP is the cost of the energy supply system, V EQ is the efficiency of the energy supply system pump, M CO2 is the total carbon dioxide emission of the system in each time period, Epur The total amount of electricity purchased by users in each time period;

[0030] The thermal balance constraint calculation formula for the model is:

[0031]

[0032] In formula (6), is the heating power of the jth river water source heat pump at time t, is the total heat load of the system at time t, N HP The number of heat pump units in the river water source heat pump energy supply system;

[0033] The calculation formula for the heating power constraint of the heat pump unit is:

[0034]

[0035] In formula (7), Q min is the lower limit of the heating power of a single heat pump, Q max is the upper limit of the heating power of a single heat pump. is the start and stop flag of the jth heat pump at time t, is the heating power of the jth heat pump at time t.

[0036] As a further improvement to the technical solution of the present invention, in step 4, the advantage function calculation formula in the reinforcement learning A3C algorithm is:

[0037] A(S,t)=R t +γR t+1 +…γ n-1 R t+n-1 +γ n v(S′)-v(S) (8)

[0038] In formula (8), A(S,t) is the advantage function representing the value of the current state S, γ is the decay factor, n represents the state feature dimension; t represents the time series, and v(S) represents the input state value function;

[0039] Based on the single-threaded training of the A3C algorithm, the Bayesian optimization algorithm is introduced to learn the hyperparameters, and the update formula of the Actor network parameter θ is obtained as follows:

[0040]

[0041] In formula (9), α is the adjustment factor, π(s t ,a t ) is the action strategy in the current environment, c is the network parameter;

[0042] The Critic network updates the network parameters omega by calculating the time difference error value delta using mean square error as the loss function:

[0043] Delta = R + gamma * V (S') - V (S) (10)

[0044] Loss = Sum (R + gamma * V (S') - V (S, omega)) 2 (11)

[0045] In formula (10) and formula (11), V (S') represents the output state value function;

[0046] Based on the A3C algorithm multi-thread asynchronous training, the multi-thread respectively interacts with the external natural environment during the training process, and updates the parameters V (S) and pi (S), so as to obtain the optimization scheduling method of the energy supply system. The A3C algorithm in the reinforcement learning can effectively utilize the computing resources and improve the training efficiency, and the Bayesian optimization algorithm is set to learn the hyperparameters, so that the convergence of the operation process is improved.

[0047] Compared with the prior art, the beneficial effects of the present application are: by introducing the A3C algorithm, the social information data analysis optimization control model is combined, the user electricity consumption behavior is integrated in the system control strategy, the user participation degree is improved, the supply and demand side interaction is established, the river water source heat pump energy supply system optimization strategy is formed, so as to more effectively reduce the peak-valley difference of power grid in load-intensive areas such as residential areas and central business districts. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 The method flowchart of the present application.

[0049] Figure 2 The working schematic diagram of the river water source heat pump energy supply system.

[0050] Figure 3 The modeling and A3C algorithm single-thread training framework diagram of the present application.

[0051] Figure 4 The A3C algorithm multi-thread asynchronous training framework diagram of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0053] As Figure 1As shown, it is a working schematic diagram of a river water source heat pump energy supply system, and the water source heat pump in the river water source heat pump energy supply system bears the air conditioning cooling load in summer, the heating load in winter and the annual domestic hot water load.

[0054] As shown, a river water source heat pump energy supply system optimization scheduling method based on A3C algorithm includes the following steps: Figure 2

[0055] Step 1: Obtain the environmental protection index, technical index, economic index and user electricity behavior index of each time period in the river water source heat pump energy supply system;

[0056] Step 2: Dynamically weight the environmental protection index, technical index, economic index and user electricity behavior index;

[0057] Step 3: Model the river water source heat pump energy supply system optimization scheduling method by relying on Markov decision process;

[0058] Step 4: Based on the multi-thread asynchronous training of A3C algorithm of reinforcement learning, introduce the Bayesian optimization algorithm to learn the hyperparameters, and combine the time difference algorithm to accelerate the convergence of the algorithm, and obtain the energy supply system optimization scheduling method of each time period.

[0059] In step 1 of this embodiment, the environmental protection index includes renewable energy output of the energy supply system and total renewable energy installed capacity, the technical index includes energy load and maximum energy load, and the economic index includes system total capacity and system total load maximum value.

[0060] In step 2 of this embodiment, the dynamic weighting function β1 of the environmental protection index is:

[0061]

[0062] In formula (1), P re (t) is the renewable energy output of the river water source heat pump energy supply system at t time, and P re_max is the total renewable energy installed capacity of the river water source heat pump energy supply system;

[0063] The dynamic weighting function β2 of the technical index is:

[0064]

[0065] In formula (2), P L_k (t) is the energy load of the kth energy at t time; and P L_kmax represents the maximum load of the kth energy;

[0066] The dynamic weighting function β3 of the economic index is:

[0067] ​

[0068] In formula (3), P L (t) is the total load of the river water source heat pump energy supply system at time t; P L_max is the maximum value of the total load of the river water source heat pump energy supply system;

[0069] The dynamic weighting function β4 of the user electricity consumption behavior index is:

[0070]

[0071] In formula (4), P max is the maximum load power in each time period, P min is the minimum load power in each time period.

[0072] As Figure 3 shown, the modeling in step 3 of the embodiment relies on a Markov decision process, and a four-tuple <S, A, R, f> is defined, S represents a state, A represents an action policy, R represents a reward function, and f represents a state transition function, which is represented by a deep neural network in reinforcement learning; the state S is a combination of the device state in the energy supply system and the external natural environment; wherein the agent is defined as the river water source heat pump energy supply system;

[0073] The calculation formula of the reward function R is:

[0074] R = β3C HP + β2V EQ + β1M CO2 + β4E pur (5)

[0075] In formula (5), C HP is the cost of the energy supply system, V EQ is the efficiency of the pump of the energy supply system, M CO2 is the total carbon dioxide emission of the system in each time period, and E pur is the total electricity purchased by the user in each time period;

[0076] The calculation formula of the heat balance constraint of the modeling is:

[0077]

[0078] In formula (6), P is the heating power of the jth river water source heat pump at time t, Q is the total heat load of the system at time t, N HP is the number of heat pump units in the river water source heat pump energy supply system;

[0079] The calculation formula of the heat pump unit heating power constraint is:

[0080]

[0081] In formula (7), Q min is the lower limit of the heating power of a single heat pump, Q max is the upper limit of the heating power of a single heat pump. is the start and stop flag of the jth heat pump at time t, is the heating power of the jth heat pump at time t.

[0082] In step 4 of this embodiment, the advantage function calculation formula in the reinforcement learning A3C algorithm is:

[0083] A(S,t)=R t +γR t+1 +…γ n-1 R t+n-1 +γ n v(S′)-v(S) (8)

[0084] In formula (8), A(S,t) is the advantage function representing the value of the current state S, γ is the decay factor, n represents the state feature dimension; t represents the time series, and v(S) represents the input state value function;

[0085] like Figure 3 As shown in the figure, based on the single-threaded training of the A3C algorithm, the Bayesian optimization algorithm is introduced to learn the hyperparameters, and the update formula of the Actor network parameter θ is obtained as follows:

[0086]

[0087] In formula (9), α is the adjustment factor, π(s t ,a t ) is the action strategy in the current environment, c is the network parameter;

[0088] The critic network calculates the error value δ of the temporal difference method and uses the mean square error as the loss function to update its own network parameters ω:

[0089] δ=R+γv(S′)-v(S)(10)

[0090] loss=∑(R+γv(S′)-v(S,ω)) 2 (11)

[0091] In formula (10) and formula (11), v(S′) represents the output state value function;

[0092] like Figure 4 As shown in the figure, based on the multi-threaded asynchronous training of the A3C algorithm, during the training process, multiple threads interact with the external natural environment and update the parameters v(S) and π(S) to obtain the optimal scheduling method of the energy supply system.

[0093] In this embodiment, the Hyperopt tool is used to realize the Bayesian optimization algorithm.

[0094] In this embodiment, the environmental modeling parameter items of the river water source heat pump energy supply system are shown in Table 1:

[0095] Table 1

[0096]

[0097] As an agent, the river water source heat pump energy supply system needs to control the running state of the motor in the system in real time according to the actual situation to obtain the optimal scheduling scheme, so in the reinforcement learning, the action strategy A of the system is defined as <fast, hold, slow>, fast, hold and slow respectively represent the operation of adjusting the motor rotor speed in the energy supply system to be fast, unchanged and slow.

Claims

1. A method for optimizing and scheduling a river water source heat pump energy supply system based on the A3C algorithm, characterized in that: The following steps are involved: Step 1: Obtain environmental protection indicators, technical indicators, economic indicators, and user electricity consumption behavior indicators for each time period in the river water source heat pump energy supply system; Step 2: Dynamically assign weights to environmental indicators, technical indicators, economic indicators, and user electricity behavior indicators; The dynamic weighting function β1 of environmental protection indicators is: In formula (1), P re (t) is the renewable energy output of the river water source heat pump energy supply system at time t, P re_max Total installed renewable energy capacity for the river water source heat pump energy system; The dynamic weighting function β2 of the technical indicator is: In formula (2), P L_k (t) is the energy load of the kth energy at time t; P L_kmax represents the maximum load of the kth energy source; The dynamic weighting function β3 of economic indicators is: In formula (3), P L (t) is the total load of the river water source heat pump energy supply system at time t; P L_max The maximum value of the total load of the river water source heat pump energy supply system; The dynamic weighting function β4 of the user's electricity consumption behavior index is: In formula (4), P max is the maximum load power in each time period, P min is the minimum load power in each time period; Step 3: Model the optimal scheduling method of the river water source heat pump energy supply system based on the Markov decision process; The modeling relies on the Markov decision process and defines the four-tuple <S, A, R, f>, where S represents the state, A represents the action strategy, R represents the reward function, and f represents the state transition function. f is represented by a deep neural network in reinforcement learning. The state S is a combination of the device state in the energy supply system and the external natural environment. The reward function R is calculated as: R=β3C HP +β2V EQ +β1M CO2 +β4E pur (5) In formula (5), C HP is the energy supply system cost, V EQ is the efficiency of the water pump in the energy supply system, M CO2 is the total carbon dioxide emissions of the system in each time period, E pur The total amount of electricity purchased by users in each time period; The thermal balance constraint calculation formula for the model is: In formula (6), is the heating power of the jth river water source heat pump at time t, is the total heat load of the system at time t, N HP The number of heat pump units in the river water source heat pump energy supply system; The calculation formula for the heating power constraint of the heat pump unit is: In formula (7), Q min is the lower limit of the heating power of a single heat pump, Q max is the upper limit of the heating power of a single heat pump. is the start and stop flag of the jth heat pump at time t, is the heating power of the jth heat pump at time t; Step 4: Based on the multi-threaded asynchronous training of the reinforcement learning A3C algorithm, the Bayesian optimization algorithm is introduced to learn the hyperparameters, and the temporal difference algorithm is combined to accelerate the convergence of the algorithm to obtain the optimal scheduling method of the energy supply system in each time period.

2. The method for optimizing and scheduling a river water source heat pump energy supply system based on the A3C algorithm according to claim 1 is characterized in that: In step 1, the environmental indicators include the renewable energy output of the energy supply system and the total installed capacity of renewable energy, the technical indicators include the energy load and the maximum energy load, and the economic indicators include the total system load and the maximum total system load.

3. The method for optimizing and scheduling a river water source heat pump energy supply system based on the A3C algorithm according to claim 1 is characterized in that: In step 4, the advantage function calculation formula in the reinforcement learning A3C algorithm is: A(S,t)=R t +γR t+1 +…γ n-1 R t+n-1 +γ n v(S′)-v(S)(8) In formula (8), A(S,t) is the advantage function representing the value of the current state S, γ is the decay factor, n represents the state feature dimension; t represents the time series, and v(S) represents the input state value function; Based on the single-threaded training of the A3C algorithm, the Bayesian optimization algorithm is introduced to learn the hyperparameters, and the update formula of the Actor network parameter θ is obtained as follows: In formula (9), α is the adjustment factor, π(s t ,a t ) is the action strategy in the current environment, s t represents the state at time t, a t represents the action at time t, and c is the network parameter; The critic network calculates the error value δ of the temporal difference method and uses the mean square error as the loss function to update its own network parameters ω: δ=R+γv(S′)-v(S)(10) loss=∑(R+γv(S′)-v(S,ω)) 2 (11) In formula (10) and formula (11), v(S′) represents the output state value function; Based on the multi-threaded asynchronous training of the A3C algorithm, during the training process, multiple threads interact with the external natural environment and update the state and action parameters v(S) and π(S) to obtain the optimal scheduling method of the energy supply system.

Images