Multi-microgrid coordination control method based on deep learning

Abstract

The invention discloses a multi-microgrid coordination control method based on deep learning, which relates to the technical field of multi-microgrid coordination control and comprises three parts of multi-microgrid model construction, decision mathematical model construction and deep reinforcement learning-based model construction. Wherein the multi-micro-grid model is a novel micro-grid model composed of a distributed power supply for wind power generation, a public energy storage system, a group of constant-temperature control loads such as a park constant-temperature system, a heating system and a refrigerating system, and a group of park loads; wherein the decision mathematical model adopts a Markov decision process; wherein an AC algorithm is adopted for construction based on the deep reinforcement learning model. According to the multi-microgrid coordination control method based on deep learning disclosed by the invention, the minimum economic operation cost of the multi-microgrid is taken as an optimization target, the economic dispatching optimization of the microgrid is realized, the algorithm training time is shorter, the economic benefit of a dispatching strategy given by an algorithm model is higher, and the optimization efficiency is higher. And the influence of wind power volatility on safe and economical operation of the micro-grid can be effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-microgrid coordinated control, and in particular to a multi-microgrid coordinated control method based on deep learning. Background Art

[0002] Microgrid, also translated as microgrid, refers to a small power generation and distribution system composed of distributed power sources, energy storage devices, energy conversion devices, loads, monitoring and protection devices, etc. The purpose of proposing microgrids is to realize the flexible and efficient application of distributed power sources and solve the problem of grid connection of a large number of distributed power sources in various forms. The development and extension of microgrids can fully promote the large-scale access of distributed power sources and renewable energy, and realize the high-reliability supply of various energy forms to loads. It is an effective way to realize active distribution networks and make the transition from traditional power grids to smart grids. Microgrid is an autonomous system that can realize self-control, protection and management. As a complete power system, it relies on its own control and management to supply energy to realize power balance control, system operation optimization, fault detection and protection, power quality management and other functions.

[0003] Among them, a multi-microgrid is a system composed of multiple microgrids that can operate independently or be interconnected to achieve greater flexibility and reliability. Each microgrid is usually composed of distributed energy resources (DER), energy storage systems, power loads, metering, protection and energy management systems, etc., which can achieve local energy production and consumption balance. These microgrids are usually composed of the following components: Distributed power sources: including renewable energy power generation equipment such as solar energy, wind energy, and biomass energy. Energy storage systems: such as lithium-ion batteries, sodium batteries, etc., used to store and regulate energy. Power loads: including building loads and household loads. Power electronic equipment: such as smart power meters, flexible DC transmission equipment, etc. Energy management system: responsible for the dispatching and management of overall energy.

[0004] The application scenarios of multi-microgrids are very wide, and can be applied to industrial parks, commercial complexes, residential buildings, public buildings and other scenarios. Its advantages include: Improve energy efficiency: Reduce energy waste by optimizing energy management and distribution. Enhance system reliability: When the main power grid fails, multi-microgrids can operate independently to ensure power supply. Promote the use of renewable energy: Promote the use of clean energy through distributed power sources. Reduce carbon emissions: Reduce environmental pollution by using renewable energy.

[0005] However, a large number of distributed renewable energy sources such as wind power generation and photovoltaic power generation are connected to the distribution network, which increases the complexity of the distribution network structure and the difficulty of management and control, affecting the safety and reliability of the power grid operation. The concept of multi-microgrid system for distribution network can alleviate the above problems to a certain extent and provide a new idea for the coordinated operation of complex power grids. The energy management and optimal scheduling of multi-microgrid system is a key point in the study of multi-microgrid system, while economic scheduling is the focus of power grid operation and a hot spot for users. Therefore, it is necessary to adopt reasonable and effective scheduling methods to enable multi-microgrid system to coordinate economic operation and improve the coordination and safety of multi-microgrid system operation under distribution network. Summary of the invention

[0006] The present invention discloses a multi-microgrid coordinated control method based on deep learning, aiming to solve the technical problems in the background technology.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A multi-microgrid coordinated control method based on deep learning, including three parts: multi-microgrid model construction, decision mathematical model construction and deep reinforcement learning model construction;

[0009] Among them, the multi-microgrid model is a new type of microgrid model composed of a distributed power source of wind power generation, a public energy storage system, a group of constant temperature control loads such as the park constant temperature system, heating system and cooling system, and a group of park loads. At the same time, the multi-microgrid is connected to the main power grid, and energy can be bought and sold in the power system;

[0010] Among them, the decision mathematical model adopts Markov decision process;

[0011] Among them, the AC algorithm is used to build a deep reinforcement learning model, and multiple intelligent agents are used to jointly explore and realize asynchronous training, which accelerates the convergence of the model and reduces the correlation of samples.

[0012] 3. In a preferred solution, the construction of a multi-microgrid model includes the following steps:

[0013] S1. Multi-microgrid optimization scheduling controls the energy storage device and its interaction with the main grid according to the energy status, and controls the constant temperature control load and the park load at the same time. The objective function can be expressed as:

[0014]

[0015] In the above formula, R T represents the income from selling electricity to users and the main grid in the microgrid at time T, C Trepresents the cost of power generation of the microgrid at time T, and the cost of purchasing and transmitting power from the main grid; in the above formula, P T It represents the price of electricity sold by the microgrid to users; represents the power consumed by the i-th group of park loads; C gen is the power generation cost; u b,t (j) represents the switch action determined by the jth group of TCL backup controllers; w AC,T represents the total number of thermostatically controlled loads in the microgrid; It means a reduction in the price at which electricity is sold to the main grid; Represents the amount of electricity sold to the main grid; in the above formula, G T and They are the electricity generated by wind power and the electricity purchased from the main power grid; Indicates an increase in price, that is, the price of purchasing electricity from the main grid; C timp and They represent the transmission costs of electricity imported from and delivered to the external grid respectively;

[0016] S2. Satisfy the constraints. The optimal dispatching of multi-microgrids needs to satisfy the following constraints:

[0017] 1) Power balance constraints:

[0018] 2) Energy storage system operation constraints:

[0019] 3) Interaction constraints with the main power grid:

[0020] 4) Constant temperature control load constraints.

[0021] In a preferred solution, the Markov decision process includes a state space S, an action space A and a reward function R().

[0022] In a preferred solution, the state space S is generated by the interaction of the intelligent agent with the microgrid environment at each time T; the action space A is based on the microgrid state ST at time T, and the system schedules the power generated by the wind and the interaction with the main grid to realize the energy distribution of the microgrid system; the reward function R() defines the total operating cost of the microgrid at time T as the reward value.

[0023] In a preferred solution, the AC algorithm combines the value function network and the policy network. The Policy Gradient (PG) can perform actions in a continuous action space, but it only trains parameters after each round and cannot be updated in a single step. The learning efficiency is relatively slow, while Q Learning can perform single-step updates, but the actions can only be limited to discrete spaces. The two complement each other and are combined to form the AC algorithm.

[0024] In a preferred solution, the AC algorithm consists of two neural networks, π(a, s; θ) is the policy network Actor, θ is the network parameter; q(s, a; w) is the value network Critic, ω is the network parameter. When interacting with the environment, the Actor policy network outputs actions according to the environmental state, and the Critic value network needs to input the state and action. The role of this network is to judge the actions selected by the Actor network; the goal of updating the Actor network parameters is to obtain a larger Critic network output value to continuously optimize the strategy; the goal of updating the Critic network parameters is to optimize its own scoring system through the reward value given by the environment, so that the scoring is more standardized, thereby continuously optimizing the actions of the policy network.

[0025] As can be seen from the above, the present invention provides a multi-microgrid coordinated control method based on deep learning, with the minimization of the economic operation cost of multiple microgrids as the optimization goal. Based on the framework of the AC algorithm and based on the Markov decision process, the microgrid optimization scheduling problem is modeled and analyzed, and a large amount of scheduling experience is obtained through continuous interaction and trial and error with the microgrid environment to train the neural network. The trained neural network is used to generate decisions to control the action of the energy storage device and its power interaction with the main power grid, so as to achieve the economic scheduling optimization of the microgrid. In addition, the algorithm training time is shorter, and the scheduling strategy given by the algorithm model has higher economic benefits, which can effectively reduce the impact of wind power volatility on the safe and economic operation of the microgrid. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a structural diagram of a multi-microgrid model of a multi-microgrid coordinated control method based on deep learning proposed in the present invention.

[0027] Figure 2 This is an AC algorithm framework diagram of a multi-microgrid coordinated control method based on deep learning proposed in the present invention. DETAILED DESCRIPTION

[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0029] Reference Figure 1 and Figure 2 , a multi-microgrid coordinated control method based on deep learning, including three parts: multi-microgrid model construction, decision mathematical model construction and deep reinforcement learning model construction;

[0030] Among them, the multi-microgrid model is a new type of microgrid model composed of a distributed power source of wind power generation, a public energy storage system, a group of constant temperature control loads such as the park constant temperature system, heating system and cooling system, and a group of park loads. At the same time, the multi-microgrid is connected to the main power grid, and energy can be bought and sold in the power system;

[0031] Among them, the decision mathematical model adopts Markov decision process;

[0032] Among them, the AC algorithm is used to build a deep reinforcement learning model, and multiple intelligent agents are used to jointly explore and realize asynchronous training, which accelerates the convergence of the model and reduces the correlation of samples.

[0033] In a preferred embodiment, the multi-microgrid model construction includes the following steps:

[0034] S1. Multi-microgrid optimization scheduling controls the energy storage device and its interaction with the main grid according to the energy status, and controls the constant temperature control load and the park load at the same time. The objective function can be expressed as:

[0035]

[0036] In the above formula, R T represents the income from selling electricity to users and the main grid in the microgrid at time T, C T represents the cost of power generation of the microgrid at time T, and the cost of purchasing and transmitting power from the main grid; in the above formula, P T It represents the price of electricity sold by the microgrid to users; represents the power consumed by the i-th group of park loads; C gen is the power generation cost; u b,t (j) represents the switch action determined by the jth group of TCL backup controllers; w AC,T represents the total number of thermostatically controlled loads in the microgrid; It means a reduction in the price at which electricity is sold to the main grid; Represents the amount of electricity sold to the main grid; in the above formula, G T and They are the electricity generated by wind power and the electricity purchased from the main power grid; Indicates an increase in price, that is, the price of purchasing electricity from the main grid; C timp and They represent the transmission costs of electricity imported from and delivered to the external grid respectively;

[0037] S2. Satisfy the constraints. The optimal dispatching of multi-microgrids needs to satisfy the following constraints:

[0038] 1) Power balance constraints:

[0039]

[0040] In the above formula, C T is the active power output by the fan, is the active power purchased from the main grid, DT is the active power generated by the energy storage device, is the active power consumed by all park loads in the microgrid, is the active power consumed by all TCLS in the microgrid, is the active power sold to the main grid, C T is the active power charged into the energy storage system;

[0041] 2) Energy storage system operation constraints:

[0042] ω SOC,min ≤w SOC,T ≤w SOC,T.max.

[0043]

[0044] 0≤C T ≤C max ·

[0045] 0≤D T ≤D max ·

[0046] In the above formula, w SOC.T represents the charge state of the energy storage system at time T, ω SOC,min 、w SOC,T.max. are its upper and lower limit constraints; η c , η d is the charge and discharge coefficient of the energy storage system, C T , D T is the charging and discharging power of the energy storage system; C max , D max It is the maximum charging and discharging power of the energy storage system obtained based on the charging and discharging rate controlled by the microgrid dispatching.

[0047] 3) Interaction constraints with the main power grid:

[0048] -G T ≤P ex ≤P ex,max.

[0049] In the above formula, P ex is the power exchanged with the main grid, -G T The maximum power that can be sold to the main grid at time T is the power generated by wind power, P ex,max. Indicates the maximum power that can be purchased from the mains.

[0050] 4) Constant temperature control load constraints.

[0051] Among them, the operating parameters of the equipment in the multi-microgrid are shown in the following table:

[0052]

[0053] Reference Figure 1 and Figure 2 In a preferred embodiment, the Markov decision process includes a state space S, an action space A and a reward function R().

[0054] The state space S is generated by the interaction between the intelligent agent and the microgrid environment at each time T; the action space A is based on the microgrid state ST at time T, and the system schedules the amount of wind power generated and the interaction with the main grid to achieve energy distribution of the microgrid system; the reward function R() defines the total operating cost of the microgrid at time T as the reward value.

[0055] The Markov decision

[0056] P[S T+1 |S T ]=P[S T+1 |S1,...,S T ]

[0057] The right side of the above formula shows the next moment state S T+1 is the same as all the historical states S1…S T related.

[0058] The reward obtained by the agent in a specific iteration can be expressed as:

[0059]

[0060] In the above formula, R T Represents the reward value obtained by the agent in a certain iteration; R T+1 Indicates the transition to the next state S T+1 The reward value obtained; R T+2 Indicated by state S T+1 Convert to S T+2 The reward value obtained.

[0061] Reference Figure 1 and Figure 2In a preferred embodiment, the AC algorithm combines the value function network and the policy network. Policy Gradient (PG) can perform actions in a continuous action space, but it only trains parameters after each round and cannot be updated in a single step. The learning efficiency is relatively slow, while Q Learning can perform single-step updates, but the actions can only be limited to discrete spaces. The advantages of the two complement each other, and the two are combined to form the AC algorithm.

[0062] The AC algorithm consists of two neural networks, π(a, s; θ) is the policy network Actor, θ is the network parameter; q(s, a; w) is the value network Critic, ω is the network parameter. When interacting with the environment, the Actor policy network outputs actions according to the environmental state. The Critic value network needs to input the state and action. The role of this network is to judge the actions selected by the Actor network; the goal of updating the Actor network parameters is to obtain a larger Critic network output value so that the strategy is continuously optimized; the goal of updating the Critic network parameters is to optimize its own scoring system through the reward value given by the environment, so that the scoring is more standardized, thereby continuously optimizing the actions of the policy network.

[0063] Among them, the control training process of multi-microgrid coordination using AC algorithm is as follows:

[0064] 1) Initialize the environment, update the time series t=1, and reset the gradient update amount ds,d of the public neural network to 0.

[0065] 2) The threads synchronize the public neural network to obtain parameters θ'=0, w1=w; initialize the environment s,,tam=t.

[0066] 3) Select action a based on the policy network π(a,s;θ).

[0067] 4) Interact with the environment, input action a,, get reward R() and next state s1+10

[0068] 5) Update time t=t+1.

[0069] 6) If the training enters the termination state or reaches the maximum training step after the interaction, go to step 7), otherwise return to step 2) and loop.

[0070] 7) Calculate Q(s,t) of the last time series s:

[0071] Q(s,t)=V(s,w').

[0072] 8) Update network parameters based on TD Error sampling.

[0073] a) Calculate Q(s,i) at each moment:

[0074] fori∈(t-1,t-2,…,tran),Q(s,i)=r,+yQ(s,i+1).

[0075] b) Accumulate the local gradient of the Actor to update parameter 0:

[0076] do=do+Vslogπ(a,s;0')(Q(s,i)-V(s,w3))+cV,H(π(s;0')).

[0077] c) Accumulate the local gradient update w of the Critic:

[0078] 9) Update the model parameters of the global neural network:

[0079] θ=θ-adg,w=w-ad.

[0080] 10) If the maximum training time is reached, the output of the common neural network parameters 0 and w is terminated, otherwise, the process goes to step 2).

[0081] By adopting AC algorithm for multi-microgrid coordinated control training, a more effective dispatching strategy can be given, and the impact of wind power fluctuations on microgrids can be effectively reduced, ensuring that microgrids can operate safely and economically.

[0082] In summary, the present invention provides a multi-microgrid coordinated control method based on deep learning with the optimization goal of minimizing the economic operation cost of multiple microgrids. On the framework based on the AC algorithm and based on the Markov decision process, the microgrid optimization scheduling problem is modeled and analyzed. A large amount of scheduling experience is obtained by continuously interacting with the microgrid environment through trial and error to train the neural network. The trained neural network is used to generate decisions to control the action of the energy storage device and its power interaction with the main power grid, thereby realizing the economic scheduling optimization of the microgrid. In addition, the algorithm training time is shorter, and the scheduling strategy given by the algorithm model has higher economic benefits, which can effectively reduce the impact of wind power volatility on the safe and economic operation of the microgrid.

[0083] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A multi-microgrid coordinated control method based on deep learning, characterized in that: It includes three parts: multi-microgrid model construction, decision-making mathematical model construction and deep reinforcement learning model construction; Among them, the multi-microgrid model is a new type of microgrid model composed of a distributed power source of wind power generation, a public energy storage system, a group of constant temperature control loads such as the park constant temperature system, heating system and cooling system, and a group of park loads. At the same time, the multi-microgrid is connected to the main power grid, and energy can be bought and sold in the power system; Among them, the decision mathematical model adopts Markov decision process; Among them, the AC algorithm is used to build a deep reinforcement learning model, and multiple intelligent agents are used to jointly explore and realize asynchronous training, which accelerates the convergence of the model and reduces the correlation of samples.

2. The multi-microgrid coordinated control method based on deep learning according to claim 1 is characterized in that: The multi-microgrid model construction includes the following steps: S1. Multi-microgrid optimization scheduling controls the energy storage device and its interaction with the main grid according to the energy status, and controls the constant temperature control load and the park load at the same time. The objective function can be expressed as: In the above formula, R T represents the income from selling electricity to users and the main grid in the microgrid at time T, C T represents the cost of power generation of the microgrid at time T, and the cost of purchasing and transmitting power from the main grid; in the above formula, P T It represents the price of electricity sold by the microgrid to users; represents the power consumed by the i-th group of park loads; C gen is the cost of electricity generation; u b，t (j) represents the switch action determined by the jth group of TCL backup controllers; w AC，T represents the total number of thermostatically controlled loads in the microgrid; It means a reduction in the price at which electricity is sold to the main grid; Represents the amount of electricity sold to the main grid; in the above formula, G T and They are the electricity generated by wind power and the electricity purchased from the main power grid; Indicates an increase in price, that is, the price of purchasing electricity from the main grid; C timp and They represent the transmission costs of electricity imported from and delivered to the external grid respectively; S2. Satisfy the constraints. The optimal dispatching of multi-microgrids needs to satisfy the following constraints: 1) Power balance constraints: 2) Energy storage system operation constraints: 3) Interaction constraints with the main power grid: 4) Constant temperature control load constraints.

3. The multi-microgrid coordinated control method based on deep learning according to claim 1 is characterized in that: The Markov decision process includes a state space S, an action space A and a reward function R().

4. The multi-microgrid coordinated control method based on deep learning according to claim 3 is characterized in that: The state space S is generated by the interaction between the intelligent agent and the microgrid environment at each time T; the action space A is based on the microgrid state ST at time T, and the system schedules the amount of wind power generated and the interaction with the main grid to achieve energy distribution of the microgrid system; the reward function R() defines the total operating cost of the microgrid at time T as the reward value.

5. The multi-microgrid coordinated control method based on deep learning according to claim 1 is characterized in that: The AC algorithm combines the value function network and the policy network. Policy Gradient (PG) can perform actions in a continuous action space, but it only trains parameters after each round and cannot be updated in a single step. The learning efficiency is relatively slow. Q Learning can perform single-step updates, but the actions can only be limited to discrete spaces. The two complement each other and are combined to form the AC algorithm.

6. The multi-microgrid coordinated control method based on deep learning according to claim 5 is characterized in that: The AC algorithm consists of two neural networks, π(a, s; θ) is the policy network Actor, θ is the network parameter; q(s, a; w) is the value network Critic, ω is the network parameter. When interacting with the environment, the Actor policy network outputs actions according to the environmental state. The Critic value network needs to input the state and action. The role of this network is to judge the actions selected by the Actor network; the goal of updating the Actor network parameters is to obtain a larger Critic network output value so that the strategy is continuously optimized; the goal of updating the Critic network parameters is to optimize its own scoring system through the reward value given by the environment, so that the scoring is more standardized, thereby continuously optimizing the actions of the policy network.

Images