A method, system and related device for optimizing the operation of distribution network and microgrid group

Abstract

The present invention discloses a method, system and related devices for optimizing the operation of distribution networks and microgrid groups, and belongs to the technical field of power grid optimization. The method comprises: deploying a pre-trained multi-agent optimization model in a power grid environment and acting on the power grid environment, inputting the state space and reward fed back by the power grid environment into the pre-trained multi-agent optimization model, outputting the action space of the corresponding agent, and using the action space as the optimization scheduling result of the operation of the distribution network and the microgrid group; the method realizes accelerated training of agents for distribution networks and microgrid groups and optimizes operation decisions. By utilizing the adaptive ability, transferability and decision-making ability of multi-agent reinforcement learning, the consumption of renewable energy is effectively promoted, and the system operation efficiency is improved while suppressing voltage over-limit.

Description

Technical Field

[0001] The present invention proposes a method, system and related devices for optimizing the operation of a distribution network and a microgrid group, and belongs to the technical field of power grid optimization. Background Art

[0002] With the continuous access of a high proportion of renewable energy to the power grid, the phenomenon of wind and solar power abandonment often occurs due to the limitation of power grid capacity and the mismatch of energy demand. In order to improve the economic efficiency of system operation, microgrid technology has emerged. As an effective means to promote the consumption of renewable energy and improve the reliability of power systems, microgrids have been developing rapidly in recent years. They are usually coupled with multiple energy sources such as new energy power generation equipment, energy storage, loads and cogeneration, which can achieve comprehensive regulation and complementary utilization of energy. A large number of distributed energy sources are connected to the distribution network in the form of microgrids, so that the distribution network flow is no longer a single flow from the head end to the end, but changes with the source and load conditions, which may cause power to flow in reverse, resulting in voltage fluctuations or even over-limit problems. Therefore, it is crucial to give full play to the various flexible control resources in the multi-microgrid-distribution network and realize the coordinated optimization of active and reactive power for the economic and reliable operation of the system.

[0003] As more and more microgrids are connected to the distribution network, their dispatch optimization also faces many challenges. The active and reactive coordination optimization problem of the distribution network containing multiple microgrids is essentially a large-scale mixed integer nonlinear programming problem, which faces challenges such as non-convex model, high dimension, and complex solution. Traditional centralized optimization methods rely on accurate distributed system modeling and complete information interaction. However, the distribution network and microgrid belong to different stakeholders, and it is difficult to achieve complete information sharing between them and implement centralized control. The distribution network containing multiple microgrids presents the characteristics of multiple regions and multiple stakeholders, which is consistent with the mechanism of distributed optimization. However, due to the existence of discrete variable devices, traditional distributed methods are difficult to effectively converge for non-convex models. In addition, as the scale of the problem increases, the number of autonomous operating entities increases significantly, the solution time is long, it is difficult to apply online, and the algorithm convergence is poor. Therefore, how to improve the real-time and optimality of the distributed optimization calculation of the distribution network containing multiple microgrids has become a hot issue that needs to be solved urgently.

[0004] In recent years, some scholars have begun to try to solve the above problems by relying on model-free data-driven methods. The data-driven multi-agent deep method can be used to solve problems such as incomplete information interaction among multiple agents and non-convex models. It has good adaptive ability to solve the problem of renewable energy fluctuations, and uses measurable data to train multi-agent models to achieve online optimal decision-making. However, due to the large scale of the main body and the complex equipment in the microgrid, the distribution network containing multiple microgrids has many variables. The input dimension of the agent will increase exponentially with the actions, observations and number of agents of each agent, resulting in slow convergence of the multi-agent algorithm during training or even no convergence. How to accelerate the convergence of agents in the case of high-dimensional input of agents is a problem worth studying at present.

[0005] In addition, for active-reactive coordination optimization, network reconstruction and switching equipment are usually used to improve the power flow distribution of the system, which will make the operating conditions of the distribution network no longer static, but in dynamic change. However, the generalization ability of neural networks is limited, and it is impossible to train only one multi-agent model to meet the decision-making needs under different distribution network operating conditions. Retraining the agent model through a trial and error mechanism is very time-consuming, and it is difficult for the agent to make the best decision during training. Therefore, how to quickly train a model suitable for new scenarios based on the existing multi-agent model and use it for real-time decision-making is a direction worth considering.

[0006] In summary, the research on multi-microgrid operation optimization involves the difficulties in solving the non-convex mixed integer programming model, the poor convergence of the algorithm under the scenario of large-scale intelligent agents, and the poor generalization of changes in operating conditions, which makes the efficient solution of the model a huge challenge. Summary of the invention

[0007] In order to solve the problems existing in the prior art, the present invention proposes a method, system and related devices for optimizing the operation of distribution networks and microgrid groups. The method realizes accelerated training of intelligent agents and optimized operation decisions for distribution networks and microgrid groups.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0009] In a first aspect, the present invention provides a method for optimizing the operation of a distribution network and a microgrid group, comprising:

[0010] The pre-trained multi-agent optimization model is deployed in the power grid environment and acts on the power grid environment. The state space and reward of the power grid environment feedback are input into the pre-trained multi-agent optimization model, and the action space of the corresponding agent is output. The action space is used as the optimization scheduling result of the distribution network and the microgrid group operation. The training process of the pre-trained multi-agent model includes:

[0011] The microgrid agent and the distribution network agent perform random actions according to the state space through a trial-and-error mechanism, and act on the power grid environment to obtain the corresponding rewards of the power grid environment through the calculation of power flow feedback. They are trained based on the multi-agent model reinforcement learning method to obtain a multi-agent model based on policy reinforcement and output the current state.

[0012] Taking the current state and action as input, the deep neural network is used as the basic model for training to obtain the intelligent agent model based on the model algorithm, and the power grid operation state and reward action at the next moment are output;

[0013] The grid operation state and reward at the next moment are input into the policy-enhanced multi-agent model to guide the policy-enhanced multi-agent model to update its parameters; the training is repeated until the policy-enhanced multi-agent model converges, and the policy-enhanced multi-agent model is finally obtained as the pre-trained multi-agent optimization model.

[0014] As a further improvement of the present invention, the microgrid intelligent body includes new energy equipment, a gas turbine, a gas boiler, a power-to-gas device and electric energy storage;

[0015] The state space of the microgrid agent is used to characterize the state information of the environment, and contains the information required for the agent's decision-making. The state information includes the current renewable energy power generation, active load, reactive load, thermal load, time-of-use electricity purchase price, time-of-use electricity sales price, gas price, common coupling node voltage, active power, reactive power and thermal power output by the gas turbine, thermal power output by the gas boiler, natural gas energy produced by power-to-gas, energy storage capacity of electric energy storage, and equipment operation status in the current period;

[0016] The actions of the microgrid intelligent body include: the purchase and sale of electricity and gas from the upper distribution network, the output of active power, reactive power and thermal power of the gas turbine, the thermal power output of the gas boiler, the natural gas energy produced by power-to-gas, the charging and discharging power of the electric energy storage, and the reactive power output of new energy.

[0017] As a further improvement of the present invention, the optimization goal of the microgrid intelligent body is that each microgrid controls the purchase and sale of electricity from the upper power grid and the actions of internal equipment to maximize its own interests, so that the microgrid has the lowest operating cost while meeting the internal load demand;

[0018] The constraints of the optimization target of the microgrid intelligent body include: micro gas turbine constraints, gas boiler constraints, power-to-gas equipment constraints, electric energy storage equipment constraints, renewable energy constraints, power balance constraints, and node voltage constraints.

[0019] As a further improvement of the present invention, the distribution network intelligent agent includes: a static VAR compensation intelligent agent and a capacitor intelligent agent, and the state space includes an observable voltage condition of the entire network and an equipment operating condition;

[0020] The action space of the distribution network intelligent agent includes the reactive power output by the static reactive compensation intelligent agent, and the number of switched capacitor banks by the capacitor intelligent agent.

[0021] As a further improvement of the present invention, the optimization target of the distribution network intelligent body is modeled with the goal of minimizing the operating cost of reactive equipment and the network loss of the distribution network.

[0022] The constraints of the optimization target of the distribution network intelligent body include: static VAR compensator constraints, capacitor constraints; node power balance constraints; line power constraints, and voltage constraints.

[0023] As a further improvement of the present invention, the cyclic reciprocating training until the multi-agent model of guiding strategy reinforcement converges specifically includes:

[0024] S101, construct the actor and critic neural network, and initialize the neural network parameters: construct the distribution network and microgrid group collaborative optimization multi-agent; initialize the state space, action space and reward function, and initialize the state of the distribution network system containing multiple microgrids from the state space;

[0025] S102, the training cycle starts, each agent gives a scheduling action for the current time section based on the current state, and sends it to the distribution network environment;

[0026] S103, storing each round of experience data into an experience playback unit;

[0027] S104, batch extract multiple groups of samples from the experience playback unit for training, the samples are subjected to deep learning to construct the power grid environment model in the power grid environment module, directly output the next moment state and corresponding reward, and then combined with the multi-agent reinforcement learning algorithm, update the parameters of the valuation critic network and the valuation actor network respectively, and the target critic network and the target actor network are both soft-updated to regularly copy parameters from the valuation network for training;

[0028] S105, judging whether the multi-agent model guiding the strategy reinforcement converges, if not, returning to S102; if converged, the multi-agent model based on strategy reinforcement is finally obtained as the pre-trained multi-agent optimization model.

[0029] As a further improvement of the present invention, the deployment of the pre-trained multi-agent optimization model in the power grid environment also includes using a transfer learning method to migrate the parameters of the pre-trained multi-agent optimization model when the operating conditions of the power grid change, and performing autonomous random training after the migration is completed.

[0030] As a further improvement of the present invention, the parameters of the pre-trained multi-agent optimization model are transferred using a transfer learning method, and after the transfer is completed, autonomous random training is performed, including:

[0031] The topological connection relationship of the network is represented by the node admittance matrix, and the similarity of the node admittance matrix is ​​used to determine whether to migrate;

[0032] If the similarity meets the preset conditions, the next step is executed; otherwise, no migration is performed and training is performed directly in a random manner;

[0033] Using transfer learning methods, we transfer the parameters of the multi-agent model so that the strategy distribution of the agents under the new operating conditions will become closer and closer to the source agent.

[0034] The migration is completed and training is performed in a random manner.

[0035] As a further improvement of the present invention, the topological connection relationship of the network is characterized according to the node admittance matrix, and whether to perform migration is determined by using the similarity of the node admittance matrix, which specifically includes:

[0036] Generate two node admittance matrices according to source network parameters and target network parameters;

[0037] Calculate the eigenvalues ​​of the admittance matrix of two nodes;

[0038] Cosine similarity is used as a criterion to determine whether to perform strategy migration: the closer the cosine value is to 1 and the closer the angle is to 0 degrees, the higher the similarity between the working condition changes of the source network and the target network.

[0039] As a further improvement of the present invention, after determining whether to perform policy migration, the following step is further included:

[0040] The difference between the initial reward value of the randomly trained agent and the initial reward value after migration is used as a quantitative indicator, combined with cosine similarity to make a double judgment on whether the migration is successful.

[0041] As a further improvement of the present invention, the microgrid intelligent body includes new energy equipment, a gas turbine, a gas boiler, a power-to-gas device and electric energy storage;

[0042] The state space of the microgrid agent is used to characterize the state information of the environment, and contains the information required for the agent's decision-making. The state information includes the current renewable energy power generation, active load, reactive load, thermal load, time-of-use electricity purchase price, time-of-use electricity sales price, gas price, common coupling node voltage, active power, reactive power and thermal power output by the gas turbine, thermal power output by the gas boiler, natural gas energy produced by power-to-gas, energy storage capacity of electric energy storage, and equipment operation status in the current period;

[0043] The actions of the microgrid intelligent body include: the purchase and sale of electricity and gas from the upper distribution network, the output of active power, reactive power and thermal power of the gas turbine, the thermal power output of the gas boiler, the natural gas energy produced by power-to-gas, the charging and discharging power of the electric energy storage, and the reactive power output of new energy.

[0044] As a further improvement of the present invention, the optimization goal of the microgrid intelligent body is that each microgrid controls the purchase and sale of electricity from the upper power grid and the operation of internal equipment in order to maximize its own interests, so that the microgrid has the lowest operating cost while meeting the internal load demand; the constraints of the optimization goal include: micro gas turbine constraints, gas boiler constraints, power-to-gas equipment constraints, electric energy storage equipment constraints, renewable energy constraints, power balance constraints, and node voltage constraints.

[0045] As a further improvement of the present invention, the distribution network intelligent agent includes: a static VAR compensation intelligent agent and a capacitor intelligent agent, and the state space includes an observable voltage condition of the entire network and an equipment operating condition;

[0046] The action space of the distribution network intelligent agent includes the reactive power output by the static reactive compensation intelligent agent, and the number of switched capacitor banks by the capacitor intelligent agent.

[0047] As a further improvement of the present invention, the optimization objective of the distribution network intelligent body is modeled with the goal of minimizing the operating cost of reactive equipment and the network loss of the distribution network, and the constraints include: static VAR compensator constraints, capacitor constraints; node power balance constraints; line power constraints, and voltage constraints.

[0048] As a further improvement of the present invention, the cyclic reciprocating training until the multi-agent model of guiding strategy reinforcement converges specifically includes:

[0049] S101, construct the actor and critic neural network, and initialize the neural network parameters: construct the distribution network and microgrid group collaborative optimization multi-agent; initialize the state space, action space and reward function, and initialize the state of the distribution network system containing multiple microgrids from the state space;

[0050] S102, the training cycle starts, each agent gives a scheduling action for the current time section based on the current state, and sends it to the distribution network environment;

[0051] S103, storing each round of experience data into an experience playback unit;

[0052] S104, batch extract multiple groups of samples from the experience playback unit for training, the samples are subjected to deep learning to construct the power grid environment model in the power grid environment module, directly output the next moment state and corresponding reward, and then combined with the multi-agent reinforcement learning algorithm, update the parameters of the valuation critic network and the valuation actor network respectively, and the target critic network and the target actor network are both soft-updated to regularly copy parameters from the valuation network for training;

[0053] S105, judging whether the multi-agent model guiding the strategy reinforcement converges, if not, returning to S102; if converged, the multi-agent model based on strategy reinforcement is finally obtained as the pre-trained multi-agent optimization model.

[0054] As a further improvement of the present invention, the deployment of the pre-trained multi-agent optimization model in the power grid environment also includes using a transfer learning method to migrate the parameters of the pre-trained multi-agent optimization model when the operating conditions of the power grid change, and performing autonomous random training after the migration is completed.

[0055] As a further improvement of the present invention, the parameters of the pre-trained multi-agent optimization model are transferred using a transfer learning method, and after the transfer is completed, autonomous random training is performed, including:

[0056] The topological connection relationship of the network is represented by the node admittance matrix, and the similarity of the node admittance matrix is ​​used to determine whether to migrate;

[0057] If the similarity is high, proceed to the next step; otherwise, do not migrate and directly use random training;

[0058] Using transfer learning methods, we transfer the parameters of the multi-agent model so that the strategy distribution of the agents under the new operating conditions will become closer and closer to the source agent.

[0059] The migration is completed and training is performed in a random manner.

[0060] As a further improvement of the present invention, the node admittance matrix can characterize the topological connection relationship of the network, and the similarity of the node admittance matrix is ​​used to determine whether to migrate, which specifically includes:

[0061] Generate two node admittance matrices according to source network parameters and target network parameters;

[0062] Calculate the eigenvalues ​​of the admittance matrix of two nodes;

[0063] Cosine similarity is used as a criterion to determine whether to perform strategy migration: the closer the cosine value is to 1 and the closer the angle is to 0 degrees, the higher the similarity between the working condition changes of the source network and the target network.

[0064] As a further improvement of the present invention, after determining whether to perform policy migration, the following step is further included:

[0065] The difference between the initial reward value of the randomly trained agent and the initial reward value after migration is used as a quantitative indicator, combined with cosine similarity to make a double judgment on whether the migration is successful.

[0066] In a second aspect, the present invention provides a distribution network and microgrid operation optimization system, comprising:

[0067] A deployment scheduling module is used to deploy the pre-trained multi-agent optimization model in the power grid environment and act on the power grid environment, input the state space and reward fed back by the power grid environment into the pre-trained multi-agent optimization model, output the action space of the corresponding agent, and use the action space as the optimization scheduling result of the distribution network and microgrid group operation;

[0068] A multi-agent training module is used to train the pre-trained multi-agent model. The training process includes:

[0069] The microgrid agent and the distribution network agent perform random actions according to the state space through a trial-and-error mechanism, and act on the power grid environment to obtain the corresponding rewards of the power grid environment through the calculation of power flow feedback. They are trained based on the multi-agent model reinforcement learning method to obtain a multi-agent model based on policy reinforcement and output the current state.

[0070] Taking the current state and action as input, the deep neural network is used as the basic model for training to obtain the intelligent agent model based on the model algorithm, and the power grid operation state and reward action at the next moment are output;

[0071] The grid operation state and reward at the next moment are input into the policy-enhanced multi-agent model to guide the policy-enhanced multi-agent model to update its parameters; the training is repeated until the policy-enhanced multi-agent model converges, and the policy-enhanced multi-agent model is finally obtained as the pre-trained multi-agent optimization model.

[0072] In a third aspect, the present invention provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method for optimizing the operation of distribution networks and microgrids when executing the computer program.

[0073] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for optimizing the operation of distribution networks and microgrid groups is implemented.

[0074] In a fifth aspect, the present invention provides a computer program product, wherein the computer program product includes computer instructions, and the computer instructions instruct a computer to execute the method for optimizing the operation of distribution networks and microgrid groups.

[0075] The beneficial effects of the present invention compared with the prior art are as follows:

[0076] The present invention proposes a deep reinforcement learning acceleration method for multi-microgrid group operation optimization, which gives full play to the advantages of machine learning artificial intelligence algorithms in multi-agent optimization operation, and realizes the accelerated training of intelligent agents and online optimal decision-making under large-scale and variable working conditions. A distribution network active and reactive collaborative optimization acceleration method containing multiple microgrids based on multi-agent deep reinforcement learning is proposed. The introduction of model learning in the multi-agent deep reinforcement learning framework effectively alleviates the problem of model difficulty in convergence due to dimensionality explosion in high-dimensional state-action scenarios, and accelerates the training process of intelligent agents. A distribution network and microgrid group variable working condition acceleration training method based on transfer learning method is proposed. For scenarios with high similarity between source network and target network, the multi-agent model based on offline training accelerates the convergence process of multi-agent by adopting strategy migration. The present invention can be widely used in distribution network systems containing multiple microgrids, and utilizes the adaptive ability, transferability and decision-making ability of multi-agent reinforcement learning to effectively promote the consumption of renewable energy, and improve the system operation efficiency while suppressing voltage over-limit. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the embodiments of the present invention or the drawings of related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0078] Figure 1 A flow chart of a method for optimizing the operation of a distribution network and a microgrid group provided by the present invention;

[0079] Figure 2 A schematic diagram of a method for optimizing the operation of a distribution network and a microgrid group provided by an embodiment of the present invention;

[0080] Figure 3 Schematic diagram of the optimal strategy learned by the multi-agent deep reinforcement learning algorithm provided by the embodiment of the present invention;

[0081] Figure 4 A schematic diagram of a multi-agent model reinforcement learning algorithm flow chart provided by an embodiment of the present invention;

[0082] Figure 5 A distribution network and microgrid operation optimization device provided by the present invention;

[0083] Figure 6 A schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0084] The embodiments of the present invention are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limitations of the present invention. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0085] In the description of the present invention, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meanings of the above terms in the present invention based on the specific content of the technical solution.

[0086] Terminology explanation:

[0087] Micro Turbine (MT) is a small, efficient gas turbine commonly used in distributed power generation and combined heat and power systems. They can generate electricity and heat using natural gas, diesel or other fuels as power sources.

[0088] Gas Boiler: Gas Boiler (GB), a gas boiler is a device that uses natural gas or other combustible gases as fuel, generates heat energy through combustion, and then heats water or other media for heating, hot water supply or industrial use.

[0089] Power to Gas (P2G) is a technology or device that converts electrical energy into gaseous fuels such as hydrogen or methane. It usually produces hydrogen by electrolyzing water, and then may convert hydrogen into methane or other gaseous fuels for storage, transportation or as an energy supply.

[0090] Electrical Storage (ES) is a device used to store electrical energy, including various types of batteries (such as lithium-ion batteries, lead-acid batteries, etc.), supercapacitors, pumped storage power stations, etc. They can store electrical energy when electricity demand is low and release electrical energy when demand is high, thereby balancing the supply and demand of the power grid.

[0091] Static Var Compensator (SVC) is a device used for reactive power compensation in power systems. It can quickly respond to reactive power changes in the power grid by adjusting its internal capacitor or inductor components, thereby improving the voltage quality and stability of the power grid.

[0092] Capacitor: A capacitor is an electronic component that can store electric charge. It consists of two conductors (called plates) that are close but not touching and an insulating medium between them. Capacitors are commonly used in circuits for applications such as filtering, energy storage, coupling and decoupling.

[0093] As an effective means to promote the consumption of renewable energy and improve the reliability of power systems, it usually has multiple energy couplings such as new energy power generation equipment, energy storage, loads and cogeneration, which can achieve comprehensive regulation and complementary utilization of energy. Therefore, by studying the multi-agent acceleration method for multi-microgrid group operation optimization, it is beneficial to accelerate the training process of the agent and then achieve online optimal decision-making, which is of great significance to improving the economy of microgrid operation and the safety of distribution network operation.

[0094] Since multi-microgrid groups present the characteristics of multi-region and multi-stakeholders, it is urgent to break through key technologies such as the training speed of intelligent agents and the optimization decision-making ability under large-scale and variable working conditions, and realize the optimized operation of multi-microgrid systems based on multi-agent deep reinforcement learning. In order to give full play to the advantages of reinforcement learning methods in multi-agent collaborative optimization under large-scale, non-convex nonlinear and variable working conditions, the present invention proposes a deep reinforcement learning acceleration method for multi-microgrid group operation optimization, which provides support for the safe and optimized operation of distribution networks containing multiple microgrids.

[0095] like Figure 1 As shown, the first object of the present invention is to provide a method for optimizing the operation of a distribution network and a microgrid group, comprising the following steps:

[0096] S1, deploy the pre-trained multi-agent optimization model in the power grid environment and act on the power grid environment;

[0097] S2, input the state space and reward of the power grid environment feedback into the pre-trained multi-agent optimization model, output the action space of the corresponding agent, and use the action space as the optimization scheduling result of the distribution network and microgrid group operation;

[0098] The training process of the pre-trained multi-agent model includes:

[0099] S21, the microgrid agent and the distribution network agent perform random actions according to the state space through a trial-and-error mechanism, and act on the power grid environment to obtain the corresponding rewards of the power grid environment through the calculation of power flow feedback, and train based on the multi-agent model reinforcement learning method to obtain a multi-agent model based on strategy reinforcement, and output the current state;

[0100] S22, using the current state and action as input, trains the deep neural network as the basic model to obtain an intelligent agent model based on the model algorithm, and outputs the grid operation state and reward action at the next moment;

[0101] S23, inputting the grid operation status and reward at the next moment into the policy-enhanced multi-agent model to guide the policy-enhanced multi-agent model to update parameters; training is repeated until the policy-enhanced multi-agent model converges, and the policy-enhanced multi-agent model is finally obtained as the pre-trained multi-agent optimization model.

[0102] The present invention gives full play to the advantages of machine learning artificial intelligence algorithms in multi-agent optimization operation, realizes accelerated training of intelligent agents under large-scale and variable working conditions, and online optimal decision-making under non-convex model conditions. The present invention can be widely used in distribution network systems containing multiple microgrids, and utilizes the adaptive ability, transferability and decision-making ability of multi-agent reinforcement learning to effectively promote the consumption of renewable energy, and improve the system operation efficiency while suppressing voltage over-limit.

[0103] The principle of the method for optimizing the operation of distribution networks and microgrids proposed in the present invention is as follows: first, a pre-trained multi-agent optimization model is deployed in a power grid environment. This model consists of multiple agents, each of which represents an entity in a distribution network or a microgrid. When the model acts on the power grid environment, it receives the state space fed back by the power grid environment, which contains the current operating state of the power grid, such as voltage, current, power, etc. Based on the received state space, the multi-agent optimization model outputs the action space of the corresponding agent. This action space represents a series of actions that the agent can take to optimize the operation of the power grid. These actions may include adjusting the output power of the generator, changing the topology of the power grid, etc.

[0104] Furthermore, during the training phase, the microgrid agent and the distribution network agent perform random actions through a trial-and-error mechanism and observe the impact of these actions on the grid environment. The grid environment feeds back corresponding rewards by calculating the power flow, which represents the quality of the grid operation status. Based on these rewards and the policy-based reinforcement learning method, the agent can learn how to choose the best action based on the current state. In order to further improve the learning ability of the agent, a deep neural network is used as the basic model for training. This model takes the current state and action as input and the grid operation state and reward at the next moment as output. Through a large amount of training data, the deep neural network can learn the complex relationship between the grid state and action. The grid operation state and reward at the next moment are input into the policy-based reinforcement multi-agent model to guide the model to update parameters. This process is repeated until the model converges, that is, the agent's strategy no longer changes significantly. At this time, the model obtained is the pre-trained multi-agent optimization model.

[0105] Based on the above scheme, the present invention can quickly output the operation optimization scheduling results of the distribution network and the microgrid group through a pre-trained multi-agent optimization model, greatly improving the efficiency of power grid scheduling. Due to the use of reinforcement learning methods, the agent can adaptively adjust the strategy according to the changes in the power grid environment, so as to adapt to different power grid operation states and needs. The multi-agent model can be easily expanded to more agents to cope with larger-scale power grid optimization problems. At the same time, the introduction of deep neural networks also improves the generalization ability of the model. Through trial and error mechanisms and reinforcement learning, the agent can learn how to make optimal decisions in different situations, thereby improving the robustness of power grid operation.

[0106] The purpose of the present invention is to construct an active-reactive optimization model based on multiple microgrids and adopt a data-driven multi-agent deep reinforcement learning method to solve the problems of multi-agents, strong randomness, non-convex model, slow convergence, etc. of multi-microgrid-distribution networks under large-scale, nonlinear and variable operating conditions, so as to realize accelerated training of intelligent agents for distribution networks and microgrids and optimize operation decisions.

[0107] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0108] This embodiment adopts a multi-agent deep reinforcement learning algorithm, and uses a model reinforcement learning method for training and solving. This technology uses the collected data to learn a model to approximate the system dynamics, and then optimizes the strategy based on the model. It is worth noting that since the learned model does not depend on the specific task, it also has the ability to optimize the strategy for new tasks. In contrast, the model-free method relies on the agent's continuous exploration, trial and error, and learning of the environment, which often leads to low data utilization efficiency. The model-based method can make full use of existing models, efficiently use data, significantly improve sampling efficiency and convergence, and thus accelerate the training process. On this basis, by constructing continuous variable multi-agents and discrete variable multi-agents, continuous and discrete control equipment are coordinated and optimized, which significantly improves the feasibility and computational efficiency of large-scale non-convex model distributed optimization solution.

[0109] In addition, the present invention introduces transfer learning based on an offline trained multi-agent model for scenarios where the similarity between the original network and the target network in a multi-agent deep reinforcement learning algorithm is high, and accelerates the convergence process of the multi-agent by adopting strategy migration.

[0110] In order to achieve the above objectives, the present invention proposes a deep reinforcement learning acceleration method for optimizing the operation of distribution networks and microgrid groups. Figure 2 shown.

[0111] The present invention mainly includes four modules: power grid environment module, microgrid intelligent body decision module, distribution network intelligent body decision module, and strategy migration module, which are described in detail below:

[0112] (1) Grid environment module

[0113] The multi-agent deep reinforcement learning model in the present invention adopts the multi-agent "centralized training-distributed execution" architecture. In the centralized training stage, the agent performs random actions according to the state space through a trial-and-error mechanism and acts on the power grid environment. The power grid environment feeds back corresponding rewards to the agent by calculating the power flow, guides the agent to update the parameters, and repeats the cycle until the agent converges, thereby learning the optimal strategy. Then the model reinforcement learning method is used for training and solving. The trained current state and action are used as input, and the power grid operation state and reward at the next moment are used as output. The power grid environment model is trained through a deep neural network, and finally a deep learning model for predicting the power grid operation state and action reward is established to replace the power flow calculation environment of the power grid environment itself. At the same time, the multi-agent deep reinforcement learning algorithm is combined to learn the optimal strategy, such as Figure 3 In the distributed execution phase, the trained multi-agent model is deployed in the actual power grid, and the agent can give the optimal scheduling result in milliseconds according to the power grid environment.

[0114] (2) Microgrid agent decision module

[0115] The microgrid has internal autonomous decision-making power. It can adjust the equipment output according to the internal load conditions and improve the economic benefits of system operation. The internal equipment of the microgrid mainly includes new energy equipment, gas turbines, gas boilers, power-to-gas equipment, and electric energy storage. The parameters of the microgrid intelligent body decision module are designed as follows:

[0116] 1) State Space

[0117] The state space of the agent is used to characterize the state information of the environment, which contains the information required for the agent's decision-making. The observable state information of the microgrid agent includes the current renewable energy power generation (including wind power or photovoltaic), active load, reactive load, thermal load, time-of-use electricity purchase price, time-of-use electricity sales price, gas price, common coupling node voltage, active power, reactive power and thermal power output of the gas turbine, thermal power output of the gas boiler, natural gas energy produced by power-to-gas, energy storage capacity of electric energy storage, and because the equipment in the microgrid is affected by its upper and lower limit constraints, it is necessary to pay attention to the equipment operation status of the equipment in the current period, that is:

[0118] (1)

[0119] in:

[0120] is the state space of the microgrid agent; t For time; , are wind power active power and reactive power respectively; , are photovoltaic active power and reactive power respectively; , They are active load and reactive load respectively; is the heat load; , They are time-of-use electricity purchase price and time-of-use electricity sale price respectively; For gas price; is the common coupling node voltage; , , are respectively the active power, reactive power and thermal power output of the gas turbine; The thermal power output of the gas boiler; The energy of natural gas produced by power-to-gas conversion; The energy storage capacity of electrical energy storage.

[0121] 2) Action Space

[0122] The agent takes corresponding actions based on the current state and interacts with the environment. The actions of the microgrid agent include: the purchase and sale of electricity and gas from the upper distribution network, the thermal power output of the gas turbine, the thermal power output of the gas boiler, the natural gas energy produced by power-to-gas, the charging and discharging power of the electric energy storage, and the reactive power output of the new energy.

[0123] (2)

[0124] Where:

[0125] is the action space of the microgrid agent; For the purchase and sale of electricity from the upper distribution network, is the gas purchase volume, , , are the thermal power output of the gas turbine respectively; The thermal power output of the gas boiler; The energy of natural gas produced by power-to-gas conversion; is the charging and discharging power of the electric energy storage; The reactive power output of wind power, is the reactive power output by photovoltaic power.

[0126] 3) Optimization goals

[0127] Each microgrid and distribution network belong to different interest entities. In order to maximize its own interests, each microgrid reasonably controls the purchase and sale of electricity from the upper power grid and the operation of internal equipment, so that the microgrid has the lowest operating cost while meeting the internal load demand. The objective function is constructed as follows:

[0128] (3)

[0129] (4)

[0130] (5)

[0131] (6)

[0132] In formula (1), is the cost of the intelligent agent purchasing electricity from the upper power grid, is the cost of gas consumed by the gas turbine, is the gas cost consumed by the gas boiler. The objective function of the microgrid considers the energy interaction cost between the microgrid and other stakeholders, that is, the cost of purchasing and selling electricity and the cost of purchasing gas. It is divided into the purchase price of electricity from the distribution network and the sales price of electricity, both of which adopt time-of-use electricity price, while the purchase price of gas Use fixed value. is the scheduling period; For the period The purchase and sale price of electricity; , The time periods Active and reactive power of the interaction between the intragrid and the microgrid. and The time periods Gas consumption of gas turbines and gas boilers.

[0133] The above equipment should also meet the following constraints while pursuing the lowest cost:

[0134] (1) Microturbine (MT) constraints

[0135] (7)

[0136] (8)

[0137] (9)

[0138] (10)

[0139] (11)

[0140] In formula (7), for The output active power of the gas turbine during the period; for Natural gas consumption of gas turbines during the period; is the calorific value of natural gas; is the power generation efficiency of the gas turbine. In formula (8), for The thermal power output of the gas turbine during the period, is the heat generation efficiency of the gas turbine. In formula (9), and For gas turbines The upper and lower limits of the power generation during the time period. In formula (10), for The output reactive power of the gas turbine during the period, is the maximum apparent power of the gas turbine. In formula (11), and For gas turbines The upper and lower limits of thermal power output during a period.

[0141] (2) Gas boiler (GB) constraints

[0142] (12)

[0143] (13)

[0144] In formula (12), for Output thermal power of gas boiler during the period; for Natural gas consumption of gas boilers during the period; is the calorific value of natural gas; is the heating efficiency of the gas boiler. In formula (13), and For gas boilers The upper and lower limits of thermal power output during a period.

[0145] (3) Power-to-gas (P2G) equipment constraints

[0146] (14)

[0147] (15)

[0148] In formula (14), for The energy of natural gas output by P2G equipment during the period; for The electrical power input by the P2G device during the time period; is the conversion efficiency of the P2G device. In formula (15), is the rated power of the P2G device.

[0149] (4) Constraints on electric energy storage equipment (ES)

[0150] (16)

[0151] (17)

[0152] (18)

[0153] (19)

[0154] (20)

[0155] In formula (16), is the energy storage capacity in period t+1; is the energy storage capacity in period t; , are the charging and discharging efficiency of the electric energy storage, respectively; , are the charging and discharging power of energy storage respectively; is the time interval. In formula (17), , are the maximum and minimum storage capacity respectively. In formula (18) and formula (19), , are the maximum discharge and charge powers of the energy storage, respectively. In formula (20), To represent the energy storage The battery cannot be charged and discharged at the same time.

[0156] (5) Renewable energy constraints

[0157] (twenty one)

[0158] In the formula, and Represents wind power and photovoltaic power in Active and reactive power output during the time period, is the maximum apparent power of wind power and photovoltaic power.

[0159] (6) Power balance constraints

[0160] (twenty two)

[0161] (twenty three)

[0162] (twenty four)

[0163] In formulas (22) and (23), , They represent the active power and reactive power interacting between the microgrid and the upper distribution network respectively; , Respectively represent the active load and reactive load inside the microgrid. In formula (24), is the heat load inside the microgrid.

[0164] (7) Node voltage constraints

[0165] (25)

[0166] In formula (25), Represents microgrid The voltage of the connected distribution network node; and Indicates the upper and lower limits of the node voltage.

[0167] Therefore, the reward function of the microgrid agent is finally designed as follows. Since the agent can effectively control its action value, its action amount can be constrained to meet the constraints. That is, once the agent makes a decision (per-unit value), the agent will calculate the nominal value corresponding to the action and determine whether it exceeds the limit. If the action exceeds the limit, the upper and lower limits close to the limit are forced to be adopted. The corrected action value is converted into the per-unit value as the new action of the agent, and the power flow calculation is performed to calculate the voltage of each node. If the voltage exceeds the limit, the voltage penalty value is added to the reward function. Since the agent explores with the goal of maximizing the reward, the opposite of the objective function is taken:

[0168] (26)

[0169] (27)

[0170] Where: is the reward function of the microgrid; is the minimum cost objective function of the microgrid, see formula (3) for details; is the voltage over-limit penalty value; is the voltage over-limit penalty coefficient; is the node voltage value.

[0171] The current state in the state space and the corresponding action in the action space in the microgrid intelligent agent decision module are used as the data input for establishing the power grid environment model in the power grid environment module. The next moment state in the state space and the corresponding reward value in the microgrid intelligent agent decision module are used as the data output for establishing the power grid environment model in the power grid environment module for deep learning training.

[0172] (3) Distribution network intelligent agent decision module

[0173] 1) State Space

[0174] Static VAR compensation agents and capacitor agents are distribution network devices that can observe the voltage status of the entire network. And equipment conditions , ,Right now

[0175] (28)

[0176] Where: It is the state space of the static reactive power compensation agent; is the voltage value of each node in the distribution network; It is the reactive power value output by the capacitor.

[0177] (29)

[0178] Where: is the state space of the switching quantity of the capacitor agent; is the number of switched capacitor banks.

[0179] 2) Action Space

[0180] The action of the static reactive power compensation agent is to output reactive power ;

[0181] (30)

[0182] Where: The action space of the static VAR compensation agent

[0183] The number of capacitor banks that the capacitor agent acts as

[0184] (31)

[0185] Where: The action space of the capacitor agent switching quantity

[0186] 3) Optimization goals

[0187] This application aims to model the operation cost of reactive equipment and minimize the loss of distribution network:

[0188] (32)

[0189] (33)

[0190] (34)

[0191] (35)

[0192] In the formula, Optimize the objective function for the distribution network; Compensate the intelligent agent operating cost for static reactive power; is the switching cost of the capacitor intelligent agent; is the cost coefficient of unit network loss; is the power loss of the distribution network; for SVC unit cost coefficient for the period; for Reactive power output by SVC during the period; for The cost of switching a single group of capacitors during a certain period of time; for The number of capacitor banks switched during the time period; For line Resistance; for Time Route The magnitude of the current on.

[0193] The above-mentioned reactive equipment should also meet the following constraints while pursuing the lowest operating cost and the lowest distribution network loss:

[0194] (1) Static VAR Compensator (SVC) Constraints

[0195] (36)

[0196] In the formula, for Reactive power output by SVC during the period. and For SVC The upper and lower limits of reactive power output during a period.

[0197] (2) Capacitor (CB) constraint

[0198] (37)

[0199] (38)

[0200] (39)

[0201] (40)

[0202] In formula (37), for The number of capacitor banks put into use during the period, for The maximum number of capacitors put into use during the period; in formula (38), for The number of capacitor groups that operate in the time period; in formula (39), is the sum of the number of capacitor groups that operate within the dispatch period T; in formula (40), It is the maximum allowed number of capacitor action groups within the scheduling period T.

[0203] (3) Node power balance constraints

[0204] (41)

[0205] In the formula, and They are i Node and j Active power injection at nodes; Line Node i To Node jThe active power transmitted; For Node i active power consumption. and They are i Node and j Reactive power injection at nodes; Line Node i To Node j Reactive power transmitted; For Node i reactive power consumption.

[0206] (4) Line power constraints

[0207] (42)

[0208] In the formula, Line Node To Node The transmitted active power, Line Node To Node The upper limit of the transmitted active power.

[0209] (5) Voltage constraint:

[0210] (43)

[0211] In the formula, Indicates the network distribution node The voltage amplitude, and For Node The upper and lower limits of the voltage amplitude, is the number of nodes in the power distribution network.

[0212] Since the agent explores with the goal of maximizing rewards, we take the opposite of the objective function:

[0213] (44)

[0214] (twenty three)

[0215] Where: is the reward function of the microgrid; The objective function is to minimize the operating cost of reactive equipment in the distribution network and the network loss of the distribution network, see formula (32) for details; is the voltage over-limit penalty value; is the voltage over-limit penalty coefficient; is the node voltage value.

[0216] The current state in the state space and the corresponding action in the action space in the distribution network intelligent agent decision module are used as the data input for establishing the power grid environment model in the power grid environment module, and the next moment state in the state space and the corresponding reward value in the distribution network intelligent agent decision module are used as the data output for establishing the power grid environment model in the power grid environment module for deep learning training.

[0217] (4) Strategy Migration Module

[0218] When the operating conditions of the power grid change, the similarity between the source task and the target task is the key to determining whether transfer learning is effective. The node admittance matrix can characterize the topological connection relationship of the network, and the similarity of the node admittance matrix is ​​used to determine whether to migrate, as follows:

[0219] 1) Generate node admittance matrix based on source network parameters and target network parameters and ;

[0220] 2) Calculate the eigenvalues ​​of the two and , where and are the eigenvalues ​​of the node admittance matrices of the source network parameters and the target network parameters, and are the eigenvectors of the node admittance matrices of the source network parameters and the target network parameters respectively;

[0221] 3) The similarity meets the preset conditions: cosine similarity is used as a criterion to determine whether to perform strategy migration. The closer the cosine value is to 1, the closer the angle is to 0 degrees, indicating that the similarity between the source network and the target network is higher. The specific formula is as follows:

[0222] (45)

[0223] 4) The difference between the initial reward value of the randomly trained agent and the initial reward value after migration is used as a quantitative indicator, combined with 3) to double-judge whether to migrate.

[0224] The closer the cosine value of the eigenvector is to 1, the higher the similarity is and the better the migration effect is. If the source network and the target network can be migrated after similarity judgment, the transfer learning method is used to migrate the parameters of the multi-agent model. Once the migration is completed, the multi-agent will then conduct autonomous random training. Although the similarity before and after the working conditions are high, there is still a slight gap between the two. Therefore, the parameters after migration are only near the optimal value, so the autonomous training mode is used to further train the agent parameters, thereby accelerating the convergence process of the agent.

[0225] (5) Model reinforcement learning for distribution network and microgrid operation optimization to accelerate the training process, such as Figure 4As shown, the following steps are included:

[0226] Step 1: Construct the actor and critic neural network and initialize the network parameters. According to the environmental objects, construct the distribution network and microgrid group collaborative optimization multi-agent. Define the state space, action space and reward function of the microgrid, capacitor and static VAR compensation agent, and initialize the state of the distribution network system containing multiple microgrids from the state space;

[0227] Step 2: The training cycle starts, and each agent gives the scheduling action for the current time section based on the current state and sends it to the distribution network environment;

[0228] Step 3: Store each round of experience data in the experience playback unit; if the experience sample size is greater than the set value Execute step 4; if not, return to step 2.

[0229] Step 4: Multiple groups of samples are extracted in batches from the experience playback unit for training. The samples are used to construct the power grid environment model in the power grid environment module through deep learning, replacing the original power grid simulation environment, omitting the power flow calculation process, and directly outputting the next moment state and corresponding reward, thereby achieving the purpose of dimensionality reduction and acceleration. Then, combined with the multi-agent reinforcement learning algorithm, the parameters of the valuation critic network and the valuation actor network are updated respectively. The target critic network and the target actor network both adopt a soft update method to regularly copy parameters from the valuation network, train the agent, improve the reward return, and encourage it to learn better learning strategies.

[0230] Step 5: Output the results of this round of training and determine whether the model has converged. If so, save the neural network model. Otherwise, repeat steps 2 to 4.

[0231] (6) Transfer reinforcement learning under changing working conditions

[0232] Step 1: The node admittance matrix can characterize the topological connection relationship of the network. The node admittance matrix eigenvalue cosine similarity is used to determine whether to migrate. If the similarity is high (i.e., the similarity before and after the working condition changes), transfer learning is performed for migration; otherwise, random training is performed using the method in (5);

[0233] Step 2: Use the transfer learning method to transfer the parameters of the multi-agent model so that the agent strategy distribution under the new operating conditions will become closer and closer to the source agent, thereby achieving the purpose of quickly migrating the agent parameters.

[0234] Step 3: Once the transfer is completed, the multi-agents are trained autonomously and randomly using the method in (5) until the agents converge.

[0235] like Figure 5As shown, the second object of the present invention is to provide a distribution network and microgrid operation optimization system, comprising:

[0236] The deployment scheduling module 100 is used to deploy the pre-trained multi-agent optimization model in the power grid environment and act on the power grid environment, input the state space and reward fed back by the power grid environment into the pre-trained multi-agent optimization model, output the action space of the corresponding agent, and use the action space as the optimization scheduling result of the distribution network and microgrid group operation;

[0237] The multi-agent training module 200 is used to train the pre-trained multi-agent model. The training process includes:

[0238] The microgrid agent and the distribution network agent perform random actions according to the state space through a trial-and-error mechanism, and act on the power grid environment to obtain the corresponding rewards of the power grid environment through the calculation of power flow feedback. They are trained based on the multi-agent model reinforcement learning method to obtain a multi-agent model based on policy reinforcement and output the current state.

[0239] Taking the current state and action as input, the deep neural network is used as the basic model for training to obtain the intelligent agent model based on the model algorithm, and the power grid operation state and reward action at the next moment are output;

[0240] The grid operation state and reward at the next moment are input into the policy-enhanced multi-agent model to guide the policy-enhanced multi-agent model to update its parameters; the training is repeated until the policy-enhanced multi-agent model converges, and the policy-enhanced multi-agent model is finally obtained as the pre-trained multi-agent optimization model.

[0241] The system is based on the above-mentioned operation optimization method for distribution networks and microgrid groups.

[0242] like Figure 6 As shown, the third object of the embodiment of the present invention is to provide an electronic device, including a memory 701, a processor 702, and a computer program stored in the memory 701 and executable on the processor, wherein the processor implements the method for optimizing the operation of the distribution network and microgrid group when executing the computer program. It also includes a communication interface 703 and a bus 704.

[0243] A fourth objective of an embodiment of the present invention is to provide a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for optimizing the operation of a distribution network and a microgrid group is implemented.

[0244] A fifth objective of an embodiment of the present invention is to provide a computer program product, wherein the computer program product includes computer instructions, and the computer instructions instruct a computer to execute the method for optimizing the operation of distribution networks and microgrid groups.

[0245] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0246] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0247] The present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, readable storage media, optical storage, etc.) containing computer-usable program codes.

[0248] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0249] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0250] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for optimizing the operation of distribution networks and microgrids, characterized in that: include: The pre-trained multi-agent optimization model is deployed in the power grid environment and acts on the power grid environment. The state space and reward feedback from the power grid environment are input into the pre-trained multi-agent optimization model, and the action space of the corresponding agent is output. The action space is used as the optimization scheduling result of the distribution network and microgrid group operation; The training process of the pre-trained multi-agent model includes: The microgrid agent and the distribution network agent perform random actions according to the state space through a trial-and-error mechanism, and act on the power grid environment to obtain the corresponding rewards of the power grid environment through the calculation of power flow feedback. They are trained based on the multi-agent model reinforcement learning method to obtain a multi-agent model based on policy reinforcement and output the current state. Taking the current state and action as input, the deep neural network is used as the basic model for training to obtain an intelligent agent model based on the model algorithm, and output the grid operation state and reward at the next moment; Inputting the grid operation state and reward at the next moment into the multi-agent model based on policy reinforcement to guide the multi-agent model based on policy reinforcement to update parameters; training repeatedly until the multi-agent model guiding policy reinforcement converges, and finally obtaining the multi-agent model based on policy reinforcement as the pre-trained multi-agent optimization model; Among them, the reward represents the quality of the grid operation status; The state space of the microgrid intelligent body includes the state information of new energy equipment, gas turbines, gas boilers, power-to-gas equipment and electric energy storage; The actions of the microgrid intelligent body include: the purchase and sale of electricity and gas from the upper distribution network, the active power, reactive power and thermal power output of the gas turbine, the thermal power output of the gas boiler, the natural gas energy produced by power-to-gas, the charging and discharging power of the electric energy storage, and the reactive power output of the new energy; The distribution network intelligent agent includes: a static reactive power compensation intelligent agent and a capacitor intelligent agent, and the state space includes the voltage condition and equipment working condition of the entire network that can be observed; The action space of the distribution network intelligent body includes: the action of the static reactive compensation intelligent body is the output reactive power, and the action of the capacitor intelligent body is the number of switched capacitor banks.

2. The method for optimizing the operation of distribution network and microgrid group according to claim 1, characterized in that: The state space of the microgrid intelligent body is used to characterize the state information of the environment, and contains the information required for the intelligent body to make decisions. The state information includes the current renewable energy power generation power, active load, reactive load, thermal load, time-of-use electricity purchase price, time-of-use electricity sales price, gas price, common coupling node voltage, active power, reactive power and thermal power output of the gas turbine, thermal power output of the gas boiler, natural gas energy produced by power-to-gas, energy storage capacity of electric energy storage, and equipment operation status in the current period.

3. The method for optimizing the operation of distribution network and microgrid group according to claim 1, characterized in that: The optimization goal of the microgrid intelligent body is to maximize the interests of each microgrid, control the purchase and sale of electricity from the upper power grid and the actions of internal equipment, so that the microgrid can achieve the lowest operating cost while meeting the internal load demand; The constraints of the optimization target of the microgrid intelligent body include: micro gas turbine constraints, gas boiler constraints, power-to-gas equipment constraints, electric energy storage equipment constraints, renewable energy constraints, power balance constraints, and node voltage constraints.

4. The method for optimizing the operation of distribution network and microgrid group according to claim 1, characterized in that: The optimization objective of the distribution network intelligent agent is to minimize the operating cost of reactive equipment and the loss of the distribution network. The constraints of the optimization target of the distribution network intelligent body include: static VAR compensator constraints, capacitor constraints; node power balance constraints; line power constraints, and voltage constraints.

5. The method for optimizing the operation of distribution network and microgrid group according to claim 1, characterized in that: The training is repeated until the multi-agent model with enhanced guidance strategy converges, specifically including: S101, construct the actor and critic neural network, and initialize the neural network parameters: construct the distribution network and microgrid group collaborative optimization multi-agent; initialize the state space, action space and reward function, and initialize the state of the distribution network system containing multiple microgrids from the state space; S102, the training cycle starts, each agent gives a scheduling action for the current time section based on the current state, and sends it to the distribution network environment; S103, storing each round of experience data into an experience playback unit; S104, batch extract multiple groups of samples from the experience playback unit for training, the samples are subjected to deep learning to construct the power grid environment model in the power grid environment module, directly output the next moment state and corresponding reward, and then combined with the multi-agent reinforcement learning algorithm, update the parameters of the valuation critic network and the valuation actor network respectively, and the target critic network and the target actor network are both soft-updated to regularly copy parameters from the valuation network for training; S105, judging whether the multi-agent model guiding the strategy reinforcement converges, if not, returning to S102; if converged, the multi-agent model based on strategy reinforcement is finally obtained as the pre-trained multi-agent optimization model.

6. The method for optimizing the operation of distribution network and microgrid group according to claim 1, characterized in that: The deployment of the pre-trained multi-agent optimization model in the power grid environment also includes using a transfer learning method to migrate the parameters of the pre-trained multi-agent optimization model when the operating conditions of the power grid change, and performing autonomous random training after the migration is completed.

7. The method for optimizing the operation of distribution network and microgrid group according to claim 6, characterized in that: The method of using the transfer learning method to transfer the parameters of the pre-trained multi-agent optimization model, after the transfer is completed, performs autonomous random training, including: The topological connection relationship of the network is represented by the node admittance matrix, and the similarity of the node admittance matrix is ​​used to determine whether to migrate; If the similarity meets the preset conditions, the next step is executed; otherwise, no migration is performed and training is performed directly in a random manner; Using transfer learning methods, we transfer the parameters of the multi-agent model so that the strategy distribution of the agents under the new operating conditions will become closer and closer to the source agent. The migration is completed and training is performed in a random manner.

8. The method for optimizing the operation of distribution network and microgrid group according to claim 7, characterized in that: The method of characterizing the topological connection relationship of the network according to the node admittance matrix and determining whether to perform migration according to the similarity of the node admittance matrix specifically includes: Generate two node admittance matrices according to source network parameters and target network parameters; Calculate the eigenvalues ​​of the admittance matrix of two nodes; Cosine similarity is used as a criterion to determine whether to perform strategy migration: the closer the cosine value is to 1 and the closer the angle is to 0 degrees, the higher the similarity between the working condition changes of the source network and the target network.

9. The method for optimizing the operation of distribution network and microgrid group according to claim 8, characterized in that: After determining whether to perform policy migration, the following steps are also required: The difference between the initial reward value of the randomly trained agent and the initial reward value after migration is used as a quantitative indicator, combined with cosine similarity to make a double judgment on whether the migration is successful.

10. A distribution network and microgrid operation optimization system, based on the distribution network and microgrid operation optimization method according to any one of claims 1 to 9, characterized in that: include: A deployment scheduling module is used to deploy the pre-trained multi-agent optimization model in the power grid environment and act on the power grid environment, input the state space and reward fed back by the power grid environment into the pre-trained multi-agent optimization model, output the action space of the corresponding agent, and use the action space as the optimization scheduling result of the distribution network and microgrid group operation; A multi-agent training module is used to train the pre-trained multi-agent model. The training process includes: The microgrid agent and the distribution network agent perform random actions according to the state space through a trial-and-error mechanism, and act on the power grid environment to obtain the corresponding rewards of the power grid environment through the calculation of power flow feedback. They are trained based on the multi-agent model reinforcement learning method to obtain a multi-agent model based on policy reinforcement and output the current state. Taking the current state and action as input, the deep neural network is used as the basic model for training to obtain an intelligent agent model based on the model algorithm, and output the grid operation state and reward at the next moment; The grid operation state and reward at the next moment are input into the policy-enhanced multi-agent model to guide the policy-enhanced multi-agent model to update its parameters; the training is repeated until the policy-enhanced multi-agent model converges, and the policy-enhanced multi-agent model is finally obtained as the pre-trained multi-agent optimization model.

11. An electronic device, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the method for optimizing the operation of distribution networks and microgrids as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for optimizing the operation of a distribution network and a microgrid group according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising computer instructions, characterized in that: The computer instructions instruct the computer to execute the method for optimizing the operation of distribution networks and microgrids as described in any one of claims 1-8.

Images