Distributed energy coordination control method based on multi-agent reinforcement learning

Abstract

The invention discloses a distributed energy coordination control method based on multi-agent reinforcement learning, and the method comprises the following steps: S1, constructing a distributed energy coordination control architecture, taking each distributed energy as an agent, and defining a state space and an action space of the agent; s2, economics, environmental protection and reliability indexes in the distributed energy coordination control process are defined, and comprehensive indexes and constraint conditions are given; and S3, constructing a reinforcement learning network, carrying out training, integrating local information and a global target of each agent through a consistency coordination algorithm, ensuring that decisions of all agents can be coordinated and consistent, and obtaining a consistency coordination control strategy of the distributed energy. According to the invention, a deep reinforcement learning algorithm is utilized to realize adaptive coordination control of distributed energy, and the overall operation efficiency and reliability of the system are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart grid and energy management, and in particular relates to a distributed energy coordination control method based on multi-agent reinforcement learning. Background Art

[0002] With the transformation of the global energy structure and the improvement of environmental awareness, the proportion of renewable energy and distributed energy in the power system is increasing. This transformation has brought unprecedented challenges to the traditional centralized power system. As a new energy management model, virtual power plants provide a possibility to solve these challenges by integrating various decentralized energy resources. However, in practical applications, the coordinated control of distributed energy still faces many technical difficulties.

[0003] A major feature of distributed energy systems is their heterogeneity. The system contains many types of resources, such as photovoltaic power generation, wind power generation, small hydropower, biomass energy, fuel cells, energy storage systems, and controllable loads. These resources have significant differences in capacity, response characteristics, cost structure, and operating constraints. For example, the output of photovoltaic power generation is affected by the intensity of sunlight and has obvious daily cycle characteristics; while wind power generation exhibits random fluctuations. The energy storage system can be adjusted in both directions, but is limited by capacity and charge and discharge rate. This heterogeneity makes it extremely difficult to adopt a unified control strategy.

[0004] Another key challenge is the uncertainty of renewable energy output. Take wind power as an example. According to research by the National Renewable Energy Laboratory (NREL), even for short-term (within 4 hours) wind power forecasts, the average absolute error can still reach 10-15% of the installed capacity. The forecast error of photovoltaic power generation may be relatively small in clear weather, but the error may increase dramatically in cloudy or rainy weather conditions. This uncertainty poses a huge challenge to the real-time balance and economic dispatch of the power system.

[0005] In large-scale distributed systems, communication limitations are also a serious problem. If traditional centralized control methods are used, there will be serious communication bottlenecks. According to a research report by IEEE, in a system containing thousands of distributed energy units, the communication delay required for centralized control may reach seconds, far exceeding the millisecond level required for real-time control of power systems. In addition, the transmission of large amounts of data also puts tremendous pressure on communication bandwidth.

[0006] With the continuous access of distributed energy, the scale of the system is becoming larger and larger, and scalability has become a key issue. The computational complexity of traditional control methods is usually exponentially related to the scale of the system. Taking the classic single-unit optimization algorithm as an example, when the number of control variables increases from 100 to 1000, the solution time may increase from milliseconds to hours, which cannot meet the needs of real-time control.

[0007] The operating environment and market conditions of the power system are constantly changing, requiring the control strategy to be highly adaptive. For example, the price of electricity in the market may fluctuate dramatically in a very short period of time; user behavior patterns may change with seasons, weather or social events; new energy policies may significantly affect the operation of the system in the short term. The control strategy needs to be able to quickly learn and adapt to these changes. Summary of the invention

[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a distributed energy coordinated control method based on multi-agent reinforcement learning, which utilizes a deep reinforcement learning algorithm to achieve adaptive coordinated control of distributed energy and improve the overall operating efficiency and reliability of the system.

[0009] The objective of the present invention is achieved through the following technical solution: A distributed energy coordinated control method based on multi-agent reinforcement learning, comprising the following steps:

[0010] S1. Build a coordinated control architecture for distributed energy resources, treat each distributed energy resource as an intelligent agent, and define the state space and action space of the intelligent agent;

[0011] S2. Define the economic, environmental and reliability indicators in the coordinated control process of distributed energy, and give comprehensive indicators and constraints;

[0012] S3. Build a reinforcement learning network and train it. Then, through the consistency coordination algorithm, integrate the local information and global goals of each agent to ensure that the decisions of all agents can be coordinated and consistent, and obtain a consistent coordination control strategy for distributed energy.

[0013] The beneficial effects of the present invention are: the present invention realizes adaptive coordinated control of distributed energy without the need for an accurate system model; it has good scalability and is suitable for large-scale distributed systems; it can effectively handle the uncertainty and dynamic changes of the system; and it reduces communication requirements and computational complexity through distributed learning and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a flow chart of the method of the present invention;

[0015] Figure 2 Schematic diagram of the principle of distributed energy coordinated control architecture. DETAILED DESCRIPTION

[0016] The technical solution of the present invention is further described in detail below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0017] like Figure 1As shown, a distributed energy coordinated control method based on multi-agent reinforcement learning includes the following steps:

[0018] S1. Build a coordinated control architecture for distributed energy resources, treat each distributed energy resource as an intelligent agent, and define the state space and action space of the intelligent agent;

[0019] like Figure 2 As shown, the distributed energy coordinated control architecture constructed in step S1 includes: a power grid, a virtual power plant, and a power trading market;

[0020] The electricity trading market is used to provide real-time electricity price information;

[0021] The virtual electric field includes a central coordinator and multiple distributed energy units; the distributed energy units are divided into power generation units, power consumption units and energy storage systems; the central coordination controller is used to coordinate and control multiple distributed energy units; the power generation units of the entire virtual electric field are used to provide power to the power consumption units and the energy storage system, and when there is excess power, they input electric energy into the power grid, and when the power generation source and the energy storage system are insufficient to supply power to the power consumption units, they obtain electric energy from the power grid to ensure the supply and demand balance of the virtual electric field.

[0022] Wherein, the power generation unit includes a combination of one or more of a photovoltaic power station or a wind power station;

[0023] There are one or more power generation units, power consumption units and energy storage systems.

[0024] In step S1, each distributed energy source is regarded as an intelligent agent, and the state space and action space of the intelligent agent are defined, including:

[0025] Define the state space of the agent, which contains the states of all agents:

[0026] For the i-th agent, the state si,t at the current time t is [Pi(t), SOCi(t),t, Li(t), pricei(t)], where Pi(t) is the power generation, SOCi(t) is the state of charge of the energy storage system, Li(t) is the load demand, and pricei(t) is the electricity price information;

[0027] For the power generation unit, SOCi(t) = SOCref, Li(t) is 0;

[0028] For the energy storage system, Pi(t) and Li(t) are both 0;

[0029] For controllable power load, SOCi(t) = SOCref, Pi(t);

[0030] Define the action space of the agent, which contains the actions of all agents:

[0031] For the i-th agent, the action ai,t at the current time t is [ΔPi(t), Pbat,i(t), ΔLi(t)], where ΔPi(t) is the power generation regulation, Pbat,i(t) is the energy storage charging and discharging power, and ΔLi(t) is the controllable load regulation;

[0032] Among them, for the power generation unit Pbat,i(t), ΔLi(t) is 0;

[0033] For the energy storage system, ΔPi(t), ΔLi(t) are 0, Pbat,i(t) has positive and negative values, positive represents discharge, negative represents charge, and 0 represents no action;

[0034] Controllable power load, ΔPi(t), Pbat,i(t) is always 0;

[0035] Where i = 1, 2, ..., N, N represents the number of agents;

[0036] The distributed energy coordinated control method needs to achieve joint optimization of economic benefits, environmental protection and reliability through distributed energy coordinated control.

[0037] S2. Define the economic, environmental and reliability indicators in the coordinated control process of distributed energy, and give comprehensive indicators and constraints;

[0038] The step S2 comprises:

[0039] S201. Define the economic indicator Jecon:

[0040]

[0041] Among them, Cgen,i(Pi(t),ΔPi(t))=ai*(Pi(t)+ΔPi(t)) 2 +bi*(Pi(t)+ΔPi(t)), represents the power generation cost;

[0042] Cbat,i(SOCi(t),Pbat,i(t))=ci*|Pbat,i(t)|+di*(SOCi(t)-SOCref) 2 , represents the energy storage cost;

[0043] Cload,i(Li(t),ΔLi(t))=ei*|ΔLi(t)|*Li(t), represents the load regulation cost;

[0044] Rmarket,i(Pi(t),ΔPi(t),pricei(t))=pricei(t)*(Pi(t)+ΔPi(t)), represents the market return;

[0045] T represents the dispatching period of coordinated control; ai, bi are the coefficients of the power generation cost function; ci is the charging and discharging cost coefficient of the energy storage system; di is the penalty coefficient for the deviation of the energy storage system from the ideal state; ei is the load regulation cost coefficient; ai, bi, ci, di, ei are coefficients determined by the characteristics of the distributed energy itself and are used as known constants in this application.

[0046] S202. Define the environmental performance indicator Jenv:

[0047]

[0048] in:

[0049] Ei(Pi(t),ΔPi(t))=fi*(Pi(t)+ΔPi(t))

[0050] Wherein, fi represents the emission coefficient; fi is a coefficient determined by the characteristics of distributed energy itself and is used as a known constant in this application;

[0051] S203. Define reliability target Jrel:

[0052]

[0053] in,

[0054] Vi(t) represents the local voltage of agent i at time t, fi(t) represents the local frequency of agent i at time t, SOCi(t) represents the state of charge of the energy storage system of agent i at time t, Pi(t) represents the basic power generation of agent i at time t, ΔPi(t) represents the power generation adjustment of agent i at time t, Li(t) represents the load demand of agent i at time t, ΔLi(t) represents the controllable load adjustment of agent i at time t, Vref represents the voltage reference value, fref represents the frequency reference value, SOCref represents the ideal state of charge of the energy storage system, αi, βi, γi, δi are the reliability weight coefficients of agent i;

[0055] S204. Define comprehensive index function:

[0056] J = Jecon + Jenv + Jrel;

[0057] (1) Power balance constraints:

[0058]

[0059] Where P1(t) represents the power input from the power grid at time t. If P1(t) is positive, it means that the power grid inputs power to the virtual electric field; if P1(t) is negative, it means that the virtual electric field outputs power to the power grid.

[0060] (2) Power generation restrictions:

[0061] (3) Energy storage restrictions:

[0062]

[0063] (4) Load regulation limitation:

[0064] (5) Voltage and frequency limitation:

[0065] In an embodiment of the present application, the virtual power plant trades electricity with the power grid through real-time electricity price information. The market profit item Rmarket,i in the economic indicator Jecon can measure the market profit value. The reliability target indicator Jrel takes into account the voltage and frequency deviations to ensure the synchronous operation quality of the virtual power plant and the power grid. The power balance constraint ensures the internal supply and demand balance of the virtual power plant, and excess or insufficient electricity is exchanged with the main power grid. At the same time, voltage and frequency limitations ensure stable interconnection with the power grid.

[0066] S3. Build a reinforcement learning network and train it. Then, through the consistency coordination algorithm, integrate the local information and global goals of each agent to ensure that the decisions of all agents can be coordinated and consistent, and obtain a consistent coordination control strategy for distributed energy.

[0067] The step S3 comprises:

[0068] S301. Building a reinforcement learning network:

[0069] The reinforcement learning network includes a strategy network and a value network;

[0070] The policy network adopts a multi-layer perceptron network structure, uses a ReLU activation function and a softmax output layer, and is used to receive the state si,t of the agent i at time t and generate a probability distribution πi(ai,t|si,t) in the action space:

[0071] The value network structure also adopts a multi-layer perceptron, uses a ReLU activation function and a linear output layer, is used to receive the state si,t of the agent i at time t, and output the value estimate Vi(si,t) of the state;

[0072] S302. Use MAPPO algorithm for reinforcement learning:

[0073] (1) At each time t, agent i observes the current state si,t and feeds it into the policy network to generate a probability distribution πi(ai,t|si,t) in the action space. Then, it samples an action ai,t=[ΔP i(t) ,P bat,i(t) ,ΔL i(t) ];

[0074] (2) Execute the action and obtain the new state si,t+1;

[0075] (3) Calculate the reward ri,t based on si,t,ai,t,si,t+1;

[0076] (4) Use si,t,ai,t,ri,t,si,t+1 to update the strategy network and value network:

[0077] For the policy network, input si,t,ai,t,ri,t,Vi(si,t),si,t+1 is input, and the policy network is updated using the clipping objective function using PPO (Proximal Policy Optimization). PPO ensures the stability of the learning process by limiting the amplitude of policy updates;

[0078] For the value network, input si,t,ri,t and use the Adam optimizer to update to minimize the mean square error between the predicted value and the actual return Realize the update of value network;

[0079] (5) Update the time step, that is, increase the value of t by 1, and then return to step (1) to complete the learning of the next time.

[0080] The reward ri,t = -(k1*Cecon,i,t+k2*Cenv,i,t+k3*Crel,i,t);

[0081] in:

[0082] Cecon,i,t=Cgen,i(Pi(t),ΔPi(t))+Cbat,i(SOCi(t),Pbat,i(t))+Cload,i(Li(t),ΔLi(t))-Rmarket,i(Pi(t),ΔPi(t),pricei(t))

[0083] Cenv,i,t=Ei(Pi(t),ΔPi(t))

[0084] Crel,i,t=αi(Vi(t)-Vref) 2 +βi(fi(t)-fref)2 +γi(SOCi(t)-SOCref) 2 +δi(Pi(t)+ΔPi(t)-Li(t)-ΔLi(t)) 2

[0085] Where k1, k2, k3 are weight coefficients used to balance economic, environmental and reliability goals, usually set to k1 = 0.6, k2 = 0.2, k3 = 0.2; Cgen,i, Cbat,i, Cload,i represent the cost functions of power generation, energy storage and load regulation respectively; Rmarket,i is the market revenue function;

[0086] Ei is the emission cost function; Vi(t), fi(t) are voltage and frequency respectively; Vref, fref are reference voltage and reference frequency respectively; SOCref is the ideal energy storage state; αi, βi, γi, δi are reliability-related weight coefficients.

[0087] When calculating the reward ri,t, constraints need to be processed:

[0088] Calculate the constraint penalty term penalty_i,t:

[0089] penalty_i,t=λ1*max(0,|Pi(t)+ΔPi(t)|-Pi,max) 2 +

[0090] λ2*max(0,|SOCi(t)-SOCref|-SOCmax) 2 +

[0091] λ3*max(0,|Pbat,i(t)|-Pbat,i,max) 2 +

[0092] λ4*max(0,|ΔLi(t)|-ΔLi,max) 2

[0093] λ1,λ2,λ3,λ4 are penalty coefficients used to adjust the penalty intensity of constraint violation, Pi,max is the maximum power generation capacity, SOCmax is the maximum allowed SOC deviation, Pbat,i,max is the maximum charge and discharge power, and ΔLi,max is the maximum load regulation amount.

[0094] Final reward = ri,t-penalty_i,t

[0095] The step S3 also includes integrating the local information and global goals of each intelligent agent through a consistency coordination algorithm to ensure that the decisions of all intelligent agents can be coordinated and consistent, and obtain a consistent coordination control strategy for distributed energy:

[0096] a. Information Collection

[0097] The central coordinator collects local strategies πi(ai,t|si,t) and states si,t from all N agents;

[0098] Construct the global state s_global,t={s1,t,s2,t,...,sN,t};

[0099] b. Strategy aggregation

[0100] Calculate the weight wi of each agent:

[0101] (1) Define the contribution of each agent Ci: Ci = |Ji,econ| + |Ji,env| + |Ji,rel|, where Ji,econ, Ji,env and Ji,rel are the local index values ​​of agent i in terms of economy, environmental protection and reliability respectively;

[0102] (2) Calculate the weight wi: Where N is the total number of agents, based on their contribution to the global goal,

[0103] Combine local strategies using a weighted average:

[0104]

[0105] c. Consistency optimization

[0106] Construct the objective function:

[0107]

[0108] The ADMM (Alternating Direction Method of Multipliers) algorithm is used to optimize the local policy πi(ai,t|si,t) of each agent. This process is achieved by minimizing the difference between πi(ai,t|si,t) and the global policy π_global(a_t|s_global,t). The ADMM algorithm introduces auxiliary variables zi and Lagrange multipliers, and by iteratively updating these variables, global consistency is achieved while maintaining partial autonomy of the agent. The optimization process continues until convergence or the preset maximum number of iterations is reached.

[0109] Optimize and obtain the final local strategy πi_final(ai,t|si,t) of each agent under t, i=1,2,…,N;

[0110] d. Strategy distribution

[0111] Distribute the final local strategy πi_final(ai,t|si,t) of each coordinated agent to the corresponding agent;

[0112] Each agent executes the final local strategy πi_final(ai,t|si,t) to complete the coordinated control of distributed energy.

[0113] This consensus algorithm overall coordination layer ensures that the decisions of all agents are coordinated and consistent, while taking into account the local information and global goals of each agent. In this way, the virtual power plant can achieve a balance between distributed control and global optimization, so that each distributed energy unit can achieve the overall optimization goal of the system while maintaining a certain degree of autonomy.

[0114] This design fully considers the characteristics of virtual power plants, including the coordination of distributed energy units, the balance of economy, environmental protection and reliability, and the handling of system constraints. Through the MAPPO algorithm and the consistency coordination layer, we can achieve an efficient and stable virtual power plant control strategy. The settings of each parameter can be adjusted according to the specific virtual power plant scale and characteristics to obtain the best performance.

[0115] In the embodiments of the present application, the distributed energy coordinated control strategy based on multi-agent reinforcement learning proposed by the present invention is centered on realizing adaptive coordinated control of distributed energy through deep reinforcement learning algorithm. In a typical embodiment, we consider a virtual power plant containing 50 distributed energy units, including 20 photovoltaic power stations, 15 wind turbines, 10 energy storage systems and 5 controllable loads.

[0116] In the multi-agent system architecture, the present invention models each distributed energy unit as an agent. The state space of each agent includes local power generation or load demand, the state of charge (SOC) of the energy storage system, the current time (hours), the current electricity price, and weather forecast information (such as temperature, sunshine intensity, wind speed, etc.). The action space of the agent includes the output regulation of the power generation unit, the charging and discharging power of the energy storage system, and the regulation amount of the controllable load. The agent interacts with neighboring units through limited communication. The communication topology adopts the small-world network model, and each agent is connected to 5-8 neighbors on average.

[0117] In the implementation of the deep reinforcement learning algorithm, the present invention adopts the proximal policy optimization (PPO) algorithm based on the Actor-Critic architecture. Each agent contains an Actor network and a Critic network. The Actor network consists of an input layer, two hidden layers and an output layer, where the hidden layers have 64 and 32 neurons respectively, and use the ReLU activation function. The number of neurons in the output layer is equal to the dimension of the action space, and the tanh activation function is used. The structure of the Critic network is similar, but the output layer has only one neuron and uses a linear activation function.

[0118] The present invention designs a two-layer control structure: the agent layer and the coordination layer. In the agent layer, each agent generates a control action every 5 minutes based on local information and limited neighbor information using the PPO algorithm. The coordination layer performs global coordination once an hour, and its main tasks include collecting the state information of all agents, updating the weight of the reward function, and performing long-term planning to provide reference trajectories for the agent layer.

[0119] In order to achieve collaborative decision-making among agents, the present invention uses a distributed optimization method based on a consensus algorithm. In this process, each agent first initializes local variables and Lagrange multipliers, then iteratively solves local sub-problems, performs consensus averaging and updates multipliers, and finally reaches global consensus.

[0120] In terms of adaptive learning strategies, the present invention introduces a meta-learning mechanism, specifically using the model-independent meta-learning algorithm (MAML). By constructing different operating scenarios (such as peak periods, trough periods, extreme weather, etc.), the algorithm can quickly adapt to new tasks. In addition, the present invention also uses priority experience replay technology, which effectively improves learning efficiency by setting an appropriate replay pool size and sampling probability.

[0121] In order to ensure the safe operation of the system, the present invention designs a safety constraint reinforcement learning framework based on Lyapunov function. By adding safety constraint terms in the objective function of the PPO algorithm, the stability of voltage and frequency is effectively guaranteed.

[0122] In terms of multi-objective optimization, the present invention constructs a multi-objective reward function that takes into account economy, environmental protection, and reliability. By using a multi-objective reinforcement learning algorithm based on the Pareto frontier, the weights of different objectives are dynamically adjusted to achieve a balance between multiple objectives.

[0123] Through testing on a simulation platform, the coordinated control strategy proposed in this paper significantly reduces system operating costs, improves renewable energy utilization, and reduces voltage and frequency deviations compared to traditional centralized control methods. More importantly, this method exhibits excellent scalability. When the system scale is expanded tenfold, the control performance only decreases slightly, and the increase in computing time is also within an acceptable range.

[0124] It should be understood that the above embodiments are only used to illustrate the principles of the present invention and should not be regarded as limiting the scope of protection of the present invention. Under the guidance of the present invention, those skilled in the art can make various modifications and improvements without departing from the scope of protection of the claims of the present invention, and these modifications and improvements should also be regarded as the scope of protection of the present invention.

Claims

1. A distributed energy coordination control method based on multi-agent reinforcement learning, characterized in that: The following steps are involved: S1. Build a coordinated control architecture for distributed energy resources, treat each distributed energy resource as an intelligent agent, and define the state space and action space of the intelligent agent; S2. Define the economic, environmental and reliability indicators in the coordinated control process of distributed energy, and give comprehensive indicators and constraints; S3. Build a reinforcement learning network and train it. Then, through the consistency coordination algorithm, integrate the local information and global goals of each agent to ensure that the decisions of all agents can be coordinated and consistent, and obtain a consistent coordination control strategy for distributed energy.

2. The distributed energy coordinated control method based on multi-agent reinforcement learning according to claim 1 is characterized in that: The distributed energy coordinated control architecture constructed in step S1 includes: a power grid, a virtual power plant, and a power trading market; The electricity trading market is used to provide real-time electricity price information; The virtual electric field includes a central coordinator and multiple distributed energy units; the distributed energy units are divided into power generation units, power consumption units and energy storage systems; the central coordination controller is used to coordinate and control multiple distributed energy units; the power generation units of the entire virtual electric field are used to provide power to the power consumption units and the energy storage system, and when there is excess power, they input electric energy into the power grid, and when the power generation source and the energy storage system are insufficient to supply power to the power consumption units, they obtain electric energy from the power grid to ensure the supply and demand balance of the virtual electric field.

3. The distributed energy coordinated control method based on multi-agent reinforcement learning according to claim 2 is characterized in that: The power generation unit includes a combination of one or more of a photovoltaic power station or a wind power station; There are one or more power generation units, power consumption units and energy storage systems.

4. The distributed energy coordinated control method based on multi-agent reinforcement learning according to claim 2 is characterized by: In step S1, each distributed energy source is regarded as an intelligent agent, and the state space and action space of the intelligent agent are defined, including: Define the state space of the agent, which contains the states of all agents: For the i-th agent, the state si,t at the current time t is [Pi(t), SOCi(t),t, Li(t), pricei(t)], where Pi(t) is the power generation, SOCi(t) is the state of charge of the energy storage system, Li(t) is the load demand, and pricei(t) is the electricity price information; For the power generation unit, SOCi(t) = SOCref, Li(t) is 0; For the energy storage system, Pi(t) and Li(t) are both 0; For controllable power load, SOCi(t) = SOCref, Pi(t); Define the action space of the agent, which contains the actions of all agents: For the i-th agent, the action ai,t at the current time t is [ΔPi(t), Pbat,i(t), ΔLi(t)], where ΔPi(t) is the power generation regulation, Pbat,i(t) is the energy storage charging and discharging power, and ΔLi(t) is the controllable load regulation; Among them, for the power generation unit Pbat,i(t), ΔLi(t) is 0; For the energy storage system, ΔPi(t), ΔLi(t) are 0, Pbat,i(t) has positive and negative values, positive represents discharge, negative represents charge, and 0 represents no action; Controllable power load, ΔPi(t), Pbat,i(t) is always 0; Where i = 1, 2, ..., N, N represents the number of agents; The distributed energy coordinated control method needs to achieve joint optimization of economic benefits, environmental protection and reliability through distributed energy coordinated control.

5. The method for coordinated control of distributed energy based on multi-agent reinforcement learning according to claim 2 is characterized in that: The step S2 comprises: S201. Define the economic indicator Jecon: Among them, Cgen,i(Pi(t),ΔPi(t))=ai*(Pi(t)+ΔPi(t)) 2 +bi*(Pi(t)+ΔPi(t)), represents the power generation cost; Cbat,i(SOCi(t),Pbat,i(t))=ci*|Pbat,i(t)|+di*(SOCi(t)-SOCref) 2 , represents the energy storage cost; Cload,i(Li(t),ΔLi(t))=ei*|ΔLi(t)|*Li(t), represents the load regulation cost; Rmarket,i(Pi(t),ΔPi(t),pricei(t))=pricei(t)*(Pi(t)+ΔPi(t)), represents the market return; T represents the dispatching period of coordinated control; ai, bi are the coefficients of the power generation cost function; ci is the charging and discharging cost coefficient of the energy storage system; di is the penalty coefficient for the energy storage system to deviate from the ideal state; ei is the load regulation cost coefficient; S202. Define the environmental performance indicator Jenv: in: Ei(Pi(t),ΔPi(t))=fi*(Pi(t)+ΔPi(t)) Where, fi represents the emission factor; S203. Define reliability target Jrel: in, Vi(t) represents the local voltage of agent i at time t, fi(t) represents the local frequency of agent i at time t, SOCi(t) represents the state of charge of the energy storage system of agent i at time t, Pi(t) represents the basic power generation of agent i at time t, ΔPi(t) represents the power generation adjustment of agent i at time t, Li(t) represents the load demand of agent i at time t, ΔLi(t) represents the controllable load adjustment of agent i at time t, Vref represents the voltage reference value, fref represents the frequency reference value, SOCref represents the ideal state of charge of the energy storage system, αi, βi, γi, δi are the reliability weight coefficients of agent i, which are constants between 0 and 1, and their sum is equal to 1; S204. Define comprehensive index function: J = Jecon + Jenv + Jrel; S205. Given constraints: (1) Power balance constraints: Where P1(t) represents the power input from the power grid at time t. If P1(t) is positive, it means that the power grid inputs power to the virtual electric field; if P1(t) is negative, it means that the virtual electric field outputs power to the power grid. (2) Power generation restrictions: (3) Energy storage restrictions: (4) Load regulation limitation: (5) Voltage and frequency limitation:

6. The method for coordinated control of distributed energy based on multi-agent reinforcement learning according to claim 2 is characterized in that: The step S3 comprises: S301. Building a reinforcement learning network: The reinforcement learning network includes a strategy network and a value network; The policy network adopts a multi-layer perceptron network structure, uses a ReLU activation function and a softmax output layer, and is used to receive the state si,t of the agent i at time t and generate a probability distribution πi(ai,t|si,t) in the action space: The value network structure also adopts a multi-layer perceptron, uses a ReLU activation function and a linear output layer, is used to receive the state si,t of the agent i at time t, and output the value estimate Vi(si,t) of the state; S302. Use MAPPO algorithm for reinforcement learning: (1) At each time t, agent i observes the current state si,t and feeds it into the policy network to generate a probability distribution πi(ai,t|si,t) in the action space. Then, it samples an action ai,t=[ΔP i(t) ,P bat,i(t) ,ΔL i(t) ]; (2) Execute the action and obtain the new state si,t+1; (3) Calculate the reward ri,t based on si,t,ai,t,si,t+1; (4) Use si,t,ai,t,ri,t,si,t+1 to update the strategy network and value network: For the policy network, input si,t,ai,t,ri,t,Vi(si,t),si,t+1, and use the clipping objective function using PPO to update the policy network. PPO ensures the stability of the learning process by limiting the amplitude of the policy update; For the value network, input si,t,ri,t and use the Adam optimizer to update to minimize the mean square error between the predicted value and the actual return Realize the update of value network; (5) Update the time step, that is, increase the value of t by 1, and then return to step (1) to complete the learning of the next time.

7. The method for coordinated control of distributed energy based on multi-agent reinforcement learning according to claim 6 is characterized in that: The reward ri,t = -(k1*Cecon,i,t+k2*Cenv,i,t+k3*Crel,i,t); in: Cecon,i,t=Cgen,i(Pi(t),ΔPi(t))+Cbat,i(SOCi(t),Pbat,i(t))+Cload,i(Li(t),ΔLi(t))-Rmarket,i(Pi(t),ΔPi(t),pricei(t)) Cenv,i,t=Ei(Pi(t),ΔPi(t)) Crel,i,t=αi(Vi(t)-Vref) 2 +βi(fi(t)-fref) 2 +γi(SOCi(t)-SOCref) 2 +δi(Pi(t)+ΔPi(t)-Li(t)-ΔLi(t)) 2 Where k1, k2, k3 are weight coefficients, which are constants between 0 and 1 and add up to 1, and are used to balance the goals of economy, environmental protection, and reliability; Cgen,i, Cbat,i, Cload,i represent the cost functions of power generation, energy storage, and load regulation, respectively; Rmarket,i is the market revenue function; Ei is the emission cost function; Vi(t), fi(t) are voltage and frequency respectively; Vref, fref are reference voltage and reference frequency respectively; SOCref is the ideal energy storage state; αi, βi, γi, δi are reliability-related weight coefficients.

8. The method for coordinated control of distributed energy based on multi-agent reinforcement learning according to claim 7 is characterized in that: When calculating the reward ri,t, constraints need to be processed: Calculate the constraint penalty term penalty_i,t: penalty_i,t=λ1*max(0,|Pi(t)+ΔPi(t)|-Pi,max) 2 + λ2*max(0,|SOCi(t)-SOCref|-SOCmax) 2 + λ3*max(0,|Pbat,i(t)|-Pbat,i,max) 2 + λ4*max(0,|ΔLi(t)|-ΔLi,max) 2 λ1,λ2,λ3,λ4 are penalty coefficients, which are constants between 0 and 1 and are used to adjust the penalty intensity of constraint violation. Pi,max is the maximum power generation capacity, SOCmax is the maximum allowed SOC deviation, Pbat,i,max is the maximum charge and discharge power, and ΔLi,max is the maximum load regulation amount; Final reward = ri,t-penalty_i,t.

9. The method for coordinated control of distributed energy based on multi-agent reinforcement learning according to claim 6 is characterized in that: In step S3, the local information and global goal of each intelligent agent are integrated to ensure that the decisions of all intelligent agents can be coordinated and consistent, and a consistent coordinated control strategy of distributed energy is obtained, including: a. Information Collection The central coordinator collects local strategies πi(ai,t|si,t) and states si,t from all N agents; Construct the global state s_global,t={s1,t,s2,t,...,sN,t}; b. Strategy aggregation Calculate the weight wi of each agent: (1) Define the contribution of each agent Ci: Ci = |Ji,econ| + |Ji,env| + |Ji,rel|, where Ji,econ, Ji,env and Ji,rel are the local index values ​​of agent i in terms of economy, environmental protection and reliability respectively; (2) Calculate the weight wi: Where N is the total number of agents, based on their contribution to the global goal, Combine local strategies using a weighted average: c. Consistency optimization Construct the objective function: The optimization of this objective problem uses the ADMM algorithm to optimize the local strategy πi(ai,t|si,t) of each agent, so that the difference between the local strategy of each agent and the global strategy π_global(a_t|s_global,t) is minimized; Optimize and obtain the final local strategy πi_final(ai,t|si,t) of each agent under t, i=1,2,…,N; d. Strategy distribution Distribute the final local strategy πi_final(ai,t|si,t) of each coordinated agent to the corresponding agent; Each agent executes the final local strategy πi_final(ai,t|si,t) to complete the coordinated control of distributed energy.

Images