Distributed frequency control method and device based on security policy learning

Abstract

The invention discloses a distributed frequency control method and device based on security policy learning, relates to the technical field of optimized operation of a power system, and mainly aims to solve the problem that the uncertainty and load fluctuation of renewable energy power generation cannot be effectively handled by the existing frequency control of a networked micro-grid system. The method comprises the following steps: performing state design and action design on each micro-grid in the system based on historical data to obtain state characteristics and control action characteristics corresponding to each micro-grid; configuring a corresponding control agent for each micro-grid in the system, and performing centralized strategy learning processing on each control agent in an offline stage based on the corresponding state feature and the control action feature to obtain an optimal control agent meeting an offline learning requirement; and on-line distributed application is carried out based on the optimization control intelligent agent, so that each micro-grid carries out frequency autonomous adjustment processing.

Description

Technical Field

[0001] The present invention relates to the technical field of power system optimization operation, and in particular to a distributed frequency control method and device based on security strategy learning. Background Art

[0002] In recent years, with the rapid development of renewable energy such as wind and solar energy, networked microgrid systems have attracted widespread attention as a flexible and distributed energy management method. However, due to the intermittent and random nature of renewable energy, the frequency control problem of these systems has become particularly complex.

[0003] Existing frequency control methods include centralized control methods, distributed control methods, and decentralized control methods. Among them, although the centralized control method can provide more precise control, it relies on point-to-point communication links and is easily affected by single point failures, resulting in high system vulnerability. Distributed control methods improve the flexibility and reliability of the system through sparse communication networks, but still need to rely on communication networks for real-time information exchange, which may reduce the control effect when the communication network fails. Distributed control methods reduce the dependence on communication through non-communication schemes, but most methods are model-based, which will face the problems of modeling complexity and parameter uncertainty, affecting the accuracy and stability of control. Therefore, the existing frequency control methods are still difficult to effectively deal with the uncertainty of renewable energy generation and load fluctuations, especially when the load demand and power generation fluctuate greatly, the adaptability of the existing control methods is poor, and it is difficult to ensure the stable operation of the system. In addition, the existing frequency control methods often consider frequency control and economic dispatch separately and adopt a hierarchical design. When the forecast error of renewable energy causes the actual power generation to be inconsistent with the forecast, the system will deviate from the optimal operating point, thereby affecting the balance between frequency control and economic dispatch, resulting in overall inefficiency. Summary of the invention

[0004] In view of this, the present invention provides a distributed frequency control method and device based on security strategy learning, the main purpose of which is to solve the problem that the existing frequency control of the networked microgrid system cannot effectively cope with the uncertainty of renewable energy power generation and load fluctuations, thereby resulting in poor stability of the networked microgrid system.

[0005] According to one aspect of the present invention, a distributed frequency control method based on security policy learning is provided, comprising:

[0006] Based on historical data, state design and action design are performed on each microgrid in the networked microgrid system to obtain state characteristics and control action characteristics corresponding to each microgrid;

[0007] Configure corresponding control agents for each microgrid in the networked microgrid system, and perform centralized policy learning processing on each of the control agents in the offline stage based on the corresponding state characteristics and control action characteristics to obtain optimized control agents that meet the requirements of offline learning;

[0008] Perform online distributed applications based on the optimized control agents, so that each of the microgrids performs frequency autonomous regulation processing.

[0009] Further, the centralized policy learning processing of each of the control agents in the offline stage based on the corresponding state characteristics and control action characteristics to obtain optimized control agents that meet the requirements of offline learning includes:

[0010] Each of the control agents initializes a corresponding actor network, and generates a current action corresponding to the current state characteristics based on the actor network, the state characteristics, and the control action characteristics;

[0011] Execute the current actions corresponding to each of the control agents in the networked microgrid system to obtain transfer state characteristics corresponding to each of the control agents; and evaluate the overall performance of the current system through a global reward function to obtain the global reward of the current system;

[0012] Integrate the current state characteristics, the current actions, the transfer state characteristics, and the global reward of the current system into state transition information; and store the state transition information in a replay buffer corresponding to each microgrid;

[0013] Train a critic network corresponding to each control agent based on the state transition information in the replay buffer until the global reward of the system meets the threshold requirements to obtain the optimized control agents that meet the requirements of offline learning.

[0014] Further, the global reward function is:

[0015]

[0016] where α ω and α DG are the weight coefficients of the global reward function; is the reward related to the frequency deviation; is the reward related to the power generation cost of the synchronous generator; ω ref is the frequency reference value; is the actual frequency of the i-th microgrid at time t; N is the total number of microgrids; Δt is the time interval of the control action; a i ,b i ,c iThey are the coefficients of the power generation cost of the synchronous generator respectively; is the power generation power of the distributed generation unit i at time t.

[0017] Furthermore, the centralized policy learning process for each of the control agents in the offline stage based on the corresponding state features further includes:

[0018] Configure a safety model between each of the microgrids and the control agents, and the safety model is used to monitor the safety of actions and guide the control agents to generate safe actions;

[0019] Train the safety model based on the state transition information stored in the replay buffer corresponding to each of the microgrids to obtain an optimized safety model that meets the application requirements.

[0020] Furthermore, the safety model includes a safety evaluation network;

[0021] The objective of the safety evaluation network is determined by the transfer state features and is defined as follows:

[0022] C tar (s t+1 |s t , a t ) = (ω t+1 - ω ref ) 2

[0023] where S t+1 is the transfer state feature, S t is the current state feature, a t is the current action, C tar (s t+1 |s t , a t ) is the target safety cost, ω t+1 is the actual frequency in the transfer state; ω ref is the frequency reference value;

[0024] The loss function of the safety evaluation network is used to measure the gap between the network prediction value and the target safety cost and is defined as follows:

[0025]

[0026] where C(s t , a t |ψ) is the safety evaluation network prediction value, S t is the current state feature, and ψ is the network parameter of the safety evaluation network.

[0027] Furthermore, the safety model includes an action guidance network;

[0028] The action guidance network extracts the mapping relationship between actions and corresponding costs; and determines whether to generate a guidance action based on the mapping relationship;

[0029] The loss function of the action guidance network is defined as follows:

[0030]

[0031] where G(s t , c|ζ) is the generated guidance action, ζ represents network parameters, and c represents the cost value of the action.

[0032] Furthermore, the security model configures operating constraints;

[0033] The operating constraints include the action constraints of distributed generation units in the microgrid corresponding to the security model, the operation constraints of energy storage devices, and the microgrid frequency constraints;

[0034] During the centralized policy learning process, the security model conducts action exploration within the operating constraints.

[0035] According to another aspect of the present invention, a distributed frequency control device based on security policy learning is provided, including:

[0036] A state and action design module, configured to perform state design and action design on each microgrid in the networked microgrid system based on historical data, and obtain state features and control action features corresponding to each microgrid;

[0037] An offline learning module, configured to configure corresponding control agents for each microgrid in the networked microgrid system, and perform centralized policy learning on each control agent based on the corresponding state features and control action features in the offline stage to obtain optimized control agents that meet the requirements of offline learning;

[0038] An online application module, configured to perform online distributed applications based on the optimized control agents, so that each microgrid performs frequency autonomous regulation processing.

[0039] Furthermore, the offline learning module includes:

[0040] An actor network unit, configured to initialize a corresponding actor network for each control agent, and generate a current action corresponding to the current state feature based on the actor network, the state feature, and the control action feature;

[0041] A state transition and evaluation unit, which is used to execute the current actions corresponding to each of the control agents in the networked microgrid system to obtain the transition state features corresponding to each of the control agents; and evaluate the overall performance of the current system through a global reward function to obtain the global reward of the current system.

[0042] An information integration unit, which is used to integrate the current state features, the current actions, the transition state features, and the global reward of the current system into state transition information; and store the state transition information in the replay buffer corresponding to each microgrid.

[0043] A critic network unit, which is used to train the critic networks corresponding to each control agent based on the state transition information in the replay buffer until the global reward of the system meets the threshold requirements, and obtain the optimized control agents that meet the requirements of offline learning.

[0044] Further, the global reward function is:

[0045]

[0046]

[0047] where α ω and α DC are the weight coefficients of the global reward function; is the reward related to the frequency deviation; is the reward related to the power generation cost of the synchronous generator; ω ref is the frequency reference value; is the actual frequency of the i-th microgrid at time t; N is the total number of microgrids; Δt is the time interval of the control action; a i , b i , c i are the coefficients of the power generation cost of the synchronous generator respectively; is the power generation power of the distributed generation unit i at time t.

[0048] Further, the offline learning module further includes:

[0049] A security configuration unit, which is used to configure a security model between each of the microgrids and the control agents, and the security model is used to monitor the security of the actions and guide the control agents to generate secure actions.

[0050] A configuration optimization unit, which is used to train the security model based on the state transition information stored in the replay buffer corresponding to each microgrid to obtain an optimized security model that meets the application requirements.

[0051] Furthermore, the security model in the security configuration unit includes a security evaluation network;

[0052] The objective of the security evaluation network is determined by the transfer state feature and is defined as follows:

[0053] C tar (s t+1 |s t ,a t )=(ω t+1 -ω ref ) 2

[0054] where S t+1 is the transfer state feature, S t is the current state feature, a t is the current action, C tar (s t+1 |s t ,a t ) is the target security cost, ω t+1 is the actual frequency in the transfer state; ω ref is the frequency reference value;

[0055] The loss function of the security evaluation network is used to measure the gap between the network prediction value and the target security cost and is defined as follows:

[0056]

[0057] where C(s t ,a t |ψ) is the security evaluation network prediction value, S t is the current state feature, and ψ is the network parameter of the security evaluation network.

[0058] Furthermore, the security model in the security configuration unit includes an action guidance network;

[0059] The action guidance network extracts the mapping relationship between actions and corresponding costs; and determines whether to generate a guidance action based on the mapping relationship;

[0060] The loss function of the action guidance network is defined as follows:

[0061]

[0062] where G(s t ,c|ζ) is the generated guidance action, ζ represents the network parameter, and c represents the cost value of the action.

[0063] Furthermore, the security model in the security configuration unit configures operating constraints;

[0064] The operating constraint conditions include the action constraints of distributed generation units in the microgrid corresponding to the security model, the operation constraints of energy storage devices, and the microgrid frequency constraints;

[0065] During the centralized policy learning process, the security model conducts action exploration within the operating constraint conditions.

[0066] According to another aspect of the present invention, there is provided a storage medium storing at least one executable instruction, and the executable instruction causes a processor to execute operations corresponding to the distributed frequency control method based on security policy learning as described above.

[0067] According to another aspect of the present invention, there is provided a device including a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus;

[0068] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to execute operations corresponding to the distributed frequency control method based on security policy learning as described above.

[0069] By means of the above technical solutions, the technical solutions provided by the embodiments of the present invention have at least the following advantages:

[0070] The present invention provides a distributed frequency control method and device based on security policy learning. Compared with the prior art, the present invention forms a multi-agent DRL (MA-DRL) framework by configuring corresponding control agents for each individual microgrid in a networked microgrid system; and performs centralized policy learning processing on each control agent in the offline stage based on the corresponding state feature and control action feature, that is, the control agent observes the state from the networked microgrid system and uses it as input. Subsequently, the control agent will explore in the action domain to find the optimal policy, realizing centralized frequency optimization control of the entire networked microgrid system. In the online application stage, each trained control agent can operate dispersedly in the corresponding microgrid, and only needs the local observation information of the corresponding microgrid to generate control actions according to the learned policy, enabling each distributed energy source to autonomously perform frequency regulation while achieving the economy of the system. The microgrid distributed frequency control method proposed by the present invention can adapt to different load disturbances and network topology conditions, and has stronger robustness and adaptability compared with the existing frequency control.

[0071] The above description is only an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention, it can be implemented according to the content of the specification. And in order to make the above and other purposes, features and advantages of the present invention more obvious and understandable, the specific embodiments of the present invention are specifically given below. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Moreover, throughout the drawings, the same reference numerals are used to represent the same components. In the drawings:

[0073] Figure 1 shows a schematic flow chart of a distributed frequency control method based on security policy learning provided by an embodiment of the present invention;

[0074] Figure 2 shows an overall framework diagram of a distributed frequency control method based on security policy learning provided by an embodiment of the present invention;

[0075] Figure 3 shows a schematic flow chart of another distributed frequency control method based on security policy learning provided by an embodiment of the present invention;

[0076] Figure 4 shows a schematic diagram of the execution of a security model provided by an embodiment of the present invention;

[0077] Figure 5 shows a schematic structural diagram of a distributed frequency control device based on security policy learning provided by an embodiment of the present invention;

[0078] Figure 6 shows a schematic structural diagram of a device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0079] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully conveyed to those skilled in the art.

[0080] The application scenario of a distributed frequency control method based on security policy learning in the present invention is a networked microgrid system. Each microgrid in the system is a small-scale power system, including multiple distributed energy resources (DERs), such as distributed generators (DGs), energy storage systems (ESSs), and flexible loads. A microgrid can operate in coordination with the main grid as a controllable entity, or operate independently to ensure its own stability when the main grid fails, experiences a power outage, or is affected by natural disasters. However, due to its limited power generation capacity, low system inertia, and geographical boundaries, a single microgrid may not be able to ensure reliable operation in some extreme cases. To further improve the resilience of the system, individual microgrids can be interconnected into a networked microgrid (NMG) system. Compared with a single microgrid, frequency control in the networked microgrid NMG is more challenging due to its more complex network structure and interactions between microgrids.

[0081] An embodiment of the present invention provides a distributed frequency control method based on security policy learning, as Figure 1 shown, the method includes:

[0082] 101. Perform state design and action design on each microgrid in the networked microgrid system based on historical data to obtain state features and control action features corresponding to each of the microgrids;

[0083] In an embodiment of the present invention, in order to achieve frequency control and adapt to the randomness of load and renewable energy generation, the current execution end includes photovoltaic (PV) generation information and load information in the state design. Since the load information and photovoltaic power generation data at the current moment cannot be obtained in real time, the current execution end obtains the corresponding historical data from each microgrid and uses the historical data to define the state features of each microgrid. For example, the state feature of the i-th microgrid can be expressed by the following formula:

[0084]

[0085] where, is the historical value of the frequency deviation of the i-th microgrid, are the historical values of the load power and photovoltaic power generation respectively, and t D represents the time delay step of the data.

[0086] It should be noted that since the load information and photovoltaic power generation data contain historical information, the state design of each microgrid actually integrates the prediction of load and photovoltaic into the proposed method.

[0087] In this embodiment, the control action feature designed to adjust the output of the distributed generation unit (DG) and the energy storage device (ESS) can be expressed by the following formula:

[0088]

[0089] Among them, represents the action of the i-th microgrid at time t, represents the change in the control command of the i-th microgrid at time t, is the power reference value of the i-th microgrid at time t. Among them, the time interval of the control action can be set to 30 seconds.

[0090] 102. Configure corresponding control agents for each microgrid in the networked microgrid system, and perform centralized policy learning processing on each of the control agents based on the corresponding state characteristics and control action characteristics in the offline stage to obtain optimized control agents that meet the requirements of offline learning;

[0091] In the embodiment of the present invention, as Figure 2 shown, each microgrid in the networked microgrid (NMG) system is equipped with an independent control agent to execute the policy π. This policy π uses the observed state s as input and generates the corresponding action a, which can be expressed as a = π(s|θ), where θ is the parameter of the policy π.

[0092] In the embodiment of the present invention, for the overall stable operation of the entire networked microgrid (NMG) system, centralized policy learning processing is performed on the corresponding control agents based on the state characteristics and control action characteristics of each microgrid in the system in the offline stage, so as to obtain optimized control agents that meet the requirements of offline learning. Among them, centralized policy learning processing means performing centralized evaluation and optimization processing based on the actions of each microgrid at the same moment, so that the entire networked microgrid (NMG) system is in a globally optimal learning processing mode. In the embodiment of the present invention, the economic dispatch problem and the frequency restoration two control objectives can be integrated, and the integrated objective is set as the objective of the centralized policy learning processing, etc., and the embodiment of the present invention does not make specific limitations.

[0093] 103. Perform online distributed application based on the optimized control agents so that each of the microgrids performs frequency autonomous regulation processing.

[0094] In the embodiment of the present invention, after obtaining the optimized control agents that meet the requirements of offline learning in step 102, the optimized control agents can be operated in a decentralized manner and directly applied online and distributedly in each microgrid, as Figure 2 shown. Specifically, in the online application stage, each optimized control agent only needs the local observation information of the corresponding microgrid to generate control actions according to the learned policy, realizing the frequency autonomous regulation processing of each microgrid.

[0095] Furthermore, as a refinement and extension of the specific implementation of the above embodiment, in order to enable each control agent to meet the overall stability of the system in terms of frequency control and improve the overall performance of the networked microgrid (NMG) system, another distributed frequency control method based on security policy learning is provided. As Figure 3 shown, the steps perform centralized policy learning processing on each of the control agents in the offline stage based on the corresponding state features and control action features, and obtain optimized control agents that meet the requirements of offline learning, including:

[0096] 201. Each of the control agents initializes the corresponding actor network, and generates a current action corresponding to the current state feature based on the actor network, the state feature, and the control action feature;

[0097] In the embodiment of the present invention, the control agents corresponding to each microgrid in the networked microgrid (NMG) system preferably implement the deep reinforcement learning soft actor-critic algorithm (SAC). Specifically, each control agent first initializes the corresponding actor network. For the i-th microgrid, an actor network is initialized as the control agent to generate an action where, represents the state feature of the i-th microgrid at time t, and θ i represents the network parameters of the i-th microgrid, and π i represents the policy of the i-th microgrid.

[0098] 202. Execute the current actions corresponding to each of the control agents in the networked microgrid system to obtain the transfer state features corresponding to each of the control agents; and evaluate the overall performance of the current system through the global reward function to obtain the current system global reward;

[0099] In the embodiment of the present invention, after obtaining the current actions of each microgrid, that is, obtaining the action of the i-th (i = 1, 2, 3,..., N, N is the number of microgrids) microgrid after that, execute all the actions in the environment (networked microgrid system) to obtain the transfer state features corresponding to each of the control agents, denoted as In addition, the overall performance of the current system is evaluated through the global reward function to obtain the current system global reward r t . Among them, the global reward function can be expressed by the formula:

[0100]

[0101] where, α ω and α DG are the weight coefficients of the global reward function; is a reward related to frequency deviation; is a reward related to the power generation cost of the synchronous generator; ω ref is the frequency reference value; is the actual frequency of the i-th microgrid at time t; N is the total number of microgrids; Δt is the time interval of the control action; a i , b i , c i are the coefficients of the power generation cost of the synchronous generator respectively; is the power generation power of the distributed generation unit i at time t.

[0102] It should be noted that the global reward provided by the embodiments of the present invention achieves the control objective of minimizing the frequency deviation and the total power generation cost of the synchronous generator (SG) by maximizing the reward value. In the global reward function, and are both negative values. If the frequency deviation is larger and the cost is higher, the reward value is smaller.

[0103] 203. Integrate the current state feature, the current action, the transition state feature, and the current system global reward into state transition information; and store the state transition information in the replay buffer corresponding to each microgrid;

[0104] In the embodiments of the present invention, the current state feature, the current action, the transition state feature, and the current system global reward of each microgrid are integrated into state transition information, denoted as (i = 1, 2, 3,..., N, N is the number of microgrids). Then, the integrated state transition information is stored in the replay buffer corresponding to each microgrid.

[0105] It should be noted that after multiple iterations, all states, actions, and rewards (i.e., state transition information) will be stored in the experience replay buffer (Replay Buffer) corresponding to each microgrid. At the same time, as the number of scenarios in offline learning increases, the data collected in each replay buffer will also increase.

[0106] 204. Train the critic network corresponding to each control agent based on the state transition information in the replay buffer until the system global reward meets the threshold requirement, and obtain the optimized control agent that meets the offline learning requirement.

[0107] In the embodiments of the present invention, based on the state transition information stored in the replay buffer corresponding to each microgrid, it is also necessary to train the critic network corresponding to each control agent, denoted as where Q idenotes the critic network corresponding to the \(i\)-th control agent, \(\varphi\) i denotes the network parameters of the critic network corresponding to the \(i\)-th control agent. During specific training, a loss function can be used to approximate the state-action value function. Then, based on the loss function, the parameters \(\theta\) of the corresponding actor network are updated i to maximize the state-action value and the policy entropy. Optionally, the above-mentioned critic network can also be implemented using a dual critic network.

[0108] It should be noted that the parameters \(\theta\) of the actor network i generate new actions after being updated, and the global reward function in step 202 is used to evaluate the comprehensive performance of the frequency deviation and the total power generation cost. After multiple iterations, until the system global reward \(r\) t meets the threshold requirements, the optimized control agent that meets the requirements of offline learning is obtained.

[0109] Furthermore, as a refinement and extension of the specific implementation manner of the above embodiment, in order to ensure that action exploration is carried out within the operating constraints, thereby reducing the learning cost, another distributed frequency control method based on safe policy learning is provided. The step of performing centralized policy learning processing on each of the control agents in the offline stage based on the corresponding state features further includes:

[0110] Configure a safety model between each of the microgrids and the control agents, and the safety model is used to monitor the safety of actions and guide the control agents to generate safe actions;

[0111] In the embodiment of the present invention, as Figure 2 and Figure 4 shown, a safety model is configured between each of the microgrids and the control agents, and the safety model includes a safety evaluation network and an action guidance network. Among them, the safety evaluation network is used to monitor the safety of actions. Among them, the action guidance network is used to guide the control agents to generate safe actions.

[0112] In this embodiment, the objective of the safety evaluation network is determined by the transfer state feature, and is defined as follows:

[0113] C tar (s t+1 |s t , a t ) = (\(\omega\) t+1 - \(\omega\) ref ) 2

[0114] where \(S\) t+1 is the transfer state feature, \(S\) t is the current state feature, \(a\) t is the current action, \(C\)tar (s t+1 |s t ,a t ) is the target safety cost, ω t+1 is the actual frequency in the transfer state; ω ref is the frequency reference value;

[0115] The loss function of the safety evaluation network is used to measure the gap between the network prediction value and the target safety cost, and is defined as follows:

[0116]

[0117] where C(s t ,a t |ψ) is the prediction value of the safety evaluation network, S t is the current state feature, and ψ is the network parameter of the safety evaluation network.

[0118] In this embodiment, the action guidance network extracts the mapping relationship between the action and the corresponding cost; and determines whether to generate a guidance action based on the mapping relationship;

[0119] The loss function of the action guidance network is defined as follows:

[0120]

[0121] where G(s t ,c|ζ) is the generated guidance action, ζ represents the network parameter, and c represents the cost value of the action.

[0122] Based on the state transition information stored in the replay buffer corresponding to each microgrid, the safety model is trained to obtain an optimized safety model that meets the application requirements.

[0123] In the embodiment of the present invention, after configuring the safety model, it is also necessary to train the safety model based on the state transition information stored in the replay buffer corresponding to each microgrid. During specific training, the safety evaluation network and the action guidance network are trained according to their respective corresponding loss functions. Specifically, as Figure 4 shown, after training the safety evaluation network, the cost value can be predicted based on (s t ,a t ) before the action is executed. Compare the predicted cost value with a preset threshold. If the predicted cost value is lower than the threshold C th , that is, C(s t ,a t |ψ) ≤ C th, the actions generated by the control agent will be regarded as safe actions and directly delivered to the environment (networked microgrid system) for execution. On the other hand, if the predicted cost value exceeds the threshold, i.e., C(s t ,a t |ψ)≥C th , then the actions generated by the control agent are regarded as unsafe actions. In this case, the action guidance network G(s t ,c|ζ) will generate a new guidance action.

[0124] In this embodiment, in order to generate the guidance action, the input cost value c can be randomly selected within the interval [0,C th . For example, multiple cost values are selected from the interval [0,C th , and a set of guidance actions are generated. To achieve relatively stable performance, the finally selected guidance action can be the action that is closest to the action a t-1 at the previous moment as the execution action.

[0125] It should be noted that the initial parameters of the safety evaluation network and the action guidance network are random values. During the offline learning process, the safety model is updated together with the actor network and the critic network. However, in the initial stage of learning, due to insufficient historical data, the performance of the safety model may be poor. Therefore, the safety model designed by the present invention only starts to take effect after a certain number of training rounds. As the number of scenarios in the offline learning increases, the data collected in the replay buffer also increases, so that the safety model can be comprehensively trained. In the offline learning stage, the safety model can ensure that the action exploration is carried out within the operation constraints, greatly reducing the learning cost. By introducing various scenarios into the detailed model, the training can cover most of the operating conditions. Once the offline training is comprehensive enough, the reliability of the safety model solution in the online stage can be guaranteed.

[0126] Furthermore, as a refinement and extension of the specific implementation manner of the above embodiment, in order to meet the operation constraints such as frequency limitation and output power limitation, another distributed frequency control method based on safety policy learning is provided, and the safety model in this method configures the operation constraint conditions;

[0127] The operation constraint conditions include the action constraints of the distributed generation units in the microgrid corresponding to the safety model, the operation constraints of the energy storage devices, and the microgrid frequency constraints;

[0128] In the embodiment of the present invention, the action constraints of the distributed generation units can be expressed by the following formula:

[0129]

[0130] Among them, represents the maximum control power of the distributed generation unit in the i-th microgrid. represents the control power of the i-th microgrid at time t. is the power reference value of the i-th microgrid at time t.

[0131] In this embodiment, the operation constraints of the energy storage device include constraints on the state of charge, the state of health of the battery, and the maximum charge and discharge power, which are expressed by the following formulas:

[0132]

[0133] Among them, are the minimum and maximum values of the state of charge SOC of the energy storage device, respectively;

[0134] are the minimum and maximum values of the state of health SOH of the energy storage device battery, respectively; are the maximum charge power and discharge power of the ESS, respectively.

[0135] To ensure the overall safety of the microgrid, the microgrid frequency constraint in this embodiment is defined as follows:

[0136]

[0137] Among them, ω - and ω + represent the lower and upper limits of the microgrid frequency, respectively.

[0138] During the centralized policy learning process, the security model conducts action exploration within the above operating constraint conditions.

[0139] The embodiment of the present invention provides a distributed frequency control method based on security policy learning. Compared with the prior art, the present invention forms a multi-agent DRL (MA-DRL) framework by configuring corresponding control agents for each individual microgrid in a networked microgrid system; and conducts centralized policy learning for each control agent in the offline stage based on the corresponding state features, that is, the control agent observes the state from the networked microgrid system and uses it as input. Subsequently, the control agent will explore in the action domain to find the optimal policy, achieving centralized frequency optimization control of the entire networked microgrid system. In the online application stage, each trained control agent can operate dispersedly in the corresponding microgrid and only needs the local observation information of the corresponding microgrid to generate control actions according to the learned policy, enabling each distributed energy to autonomously adjust the frequency and realizing the economy of the system at the same time. The microgrid distributed frequency control method proposed by the present invention can adapt to different load disturbances and network topology conditions, and has stronger robustness and adaptability compared with the existing frequency control.

[0140] As an implementation of the method described above Figure 1 the embodiment of the present invention provides a distributed frequency control device based on security policy learning, as Figure 5 shown, the device includes:

[0141] A state and action design module 51, configured to perform state design and action design on each microgrid in the networked microgrid system based on historical data, so as to obtain state characteristics and control action characteristics corresponding to each microgrid;

[0142] An offline learning module 52, configured to configure corresponding control agents for each microgrid in the networked microgrid system, and perform centralized policy learning processing on each control agent based on the corresponding state characteristics and control action characteristics in the offline phase, so as to obtain optimized control agents that meet the requirements of offline learning;

[0143] An online application module 53, configured to perform online distributed applications based on the optimized control agents, so that each microgrid performs frequency autonomous regulation processing.

[0144] Further, the offline learning module 52 includes:

[0145] An actor network unit, configured to initialize a corresponding actor network for each control agent, and generate a current action corresponding to the current state characteristic based on the actor network, the state characteristic, and the control action characteristic;

[0146] A state transition and evaluation unit, configured to execute the current action corresponding to each control agent in the networked microgrid system to obtain transition state characteristics corresponding to each control agent; and evaluate the overall performance of the current system through a global reward function to obtain the global reward of the current system;

[0147] An information integration unit, configured to integrate the current state characteristic, the current action, the transition state characteristic, and the global reward of the current system into state transition information; and store the state transition information in a replay buffer corresponding to each microgrid;

[0148] A critic network unit, configured to train a critic network corresponding to each control agent based on the state transition information in the replay buffer until the global reward of the system meets the threshold requirement, so as to obtain the optimized control agent that meets the requirements of offline learning.

[0149] Further, the global reward function is:

[0150]

[0151] where αω and α DG are the weight coefficients of the global reward function; is the reward related to the frequency deviation; is the reward related to the power generation cost of the synchronous generator; ω ref is the frequency reference value; is the actual frequency of the i-th microgrid at time t; N is the total number of microgrids; Δt is the time interval of the control action; a i , b i , c i are the coefficients of the power generation cost of the synchronous generator, respectively; is the power generation power of the distributed generation unit i at time t.

[0152] Furthermore, the offline learning module 52 further includes:

[0153] A security configuration unit, configured to configure a security model between each of the microgrids and the control agent, where the security model is used to monitor the security of actions and guide the control agent to generate secure actions;

[0154] A configuration optimization unit, configured to train the security model based on the state transition information stored in the replay buffer corresponding to each of the microgrids to obtain an optimized security model that meets the application requirements.

[0155] Furthermore, the security model in the security configuration unit includes a security evaluation network;

[0156] The objective of the security evaluation network is determined by the transition state features, and is defined as follows:

[0157] C tar (s t+1 |s t ,a t )=(ω t+1 -ω ref ) 2

[0158] where S t+1 is the transition state feature, S t is the current state feature, a t is the current action, C tar (s t+1 |s t ,a t ) is the target security cost, ω t+1 is the actual frequency in the transition state; ω ref is the frequency reference value;

[0159] The loss function of the security evaluation network is used to measure the gap between the network prediction value and the target security cost, and is defined as follows:

[0160]

[0161] Among them, C(s t ,a t |ψ) is the prediction value of the security evaluation network, S t is the current state feature, and ψ is the network parameter of the security evaluation network.

[0162] Furthermore, the security model in the security configuration unit includes an action guidance network;

[0163] The action guidance network extracts the mapping relationship between actions and corresponding costs; and determines whether to generate a guidance action based on the mapping relationship;

[0164] The loss function of the action guidance network is defined as follows:

[0165]

[0166] Among them, G(s t , c|ζ) is the generated guidance action, ζ represents the network parameter, and c represents the cost value of the action.

[0167] Furthermore, the security model in the security configuration unit configures operating constraints;

[0168] The operating constraints include the action constraints of distributed generation units in the microgrid corresponding to the security model, the operation constraints of energy storage devices, and the microgrid frequency constraints;

[0169] During the centralized policy learning process, the security model performs action exploration within the operating constraints.

[0170] An embodiment of the present invention provides a microgrid distributed frequency control device. Compared with the prior art, the present invention forms a multi-agent DRL (MA-DRL) framework by configuring corresponding control agents for each single microgrid in a networked microgrid system; and performs centralized policy learning processing on each control agent in the offline stage based on the corresponding state characteristics and control action characteristics, that is, the control agent observes the state from the networked microgrid system and uses it as input. Subsequently, the control agent will explore in the action domain to find the optimal policy, realizing the centralized frequency optimization control of the entire networked microgrid system. In the online application stage, each trained control agent can operate dispersedly in the corresponding microgrid, and only needs the local observation information of the corresponding microgrid to generate control actions according to the learned policy, enabling each distributed energy source to autonomously perform frequency regulation and realizing the economy of the system at the same time. The microgrid distributed frequency control method proposed by the present invention can adapt to different load disturbances and network topology conditions, and has stronger robustness and adaptability compared with the existing frequency control.

[0171] According to an embodiment of the present invention, a storage medium is provided, and the storage medium stores at least one executable instruction, and the computer executable instruction can execute the distributed frequency control method based on security policy learning in any of the above method embodiments.

[0172] Figure 6 The structure diagram of a device provided according to an embodiment of the present invention is shown. The specific implementation of the device is not limited in the specific embodiments of the present invention.

[0173] As Figure 6 shown, the device may include: a processor 602, a communication interface 604, a memory 606, and a communication bus 608.

[0174] Among them: the processor 602, the communication interface 604, and the memory 606 complete mutual communication through the communication bus 608.

[0175] The communication interface 604 is used to communicate with network elements of other devices such as clients or other servers.

[0176] The processor 602 is used to execute the program 610, and specifically can execute the relevant steps of the above-mentioned distributed frequency control method based on security policy learning.

[0177] Specifically, the program 610 may include program code, and the program code includes computer operation instructions.

[0178] The processor 602 may be a central processing unit (CPU), or a specific application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention. One or more processors included in the device may be of the same type of processor, such as one or more CPUs; or they may be of different types of processors, such as one or more CPUs and one or more ASICs.

[0179] A memory 606 for storing a program 610. The memory 606 may include high-speed RAM memory and may also include non-volatile memory, such as at least one disk memory.

[0180] The program 610 may specifically be used to cause the processor 602 to perform the following operations:

[0181] Based on historical data, perform state design and action design for each microgrid in the networked microgrid system to obtain state characteristics and control action characteristics corresponding to each of the microgrids;

[0182] Configure corresponding control agents for each microgrid in the networked microgrid system, and perform centralized policy learning processing on each of the control agents based on the corresponding state characteristics and control action characteristics in the offline stage to obtain optimized control agents that meet the requirements of offline learning;

[0183] Perform online distributed applications based on the optimized control agents so that each of the microgrids performs frequency autonomous regulation processing.

[0184] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present invention can be implemented by a general-purpose computing device. They can be concentrated on a single computing device or distributed on a network composed of multiple computing devices. Optionally, they can be implemented by program codes executable by the computing device, so that they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order than here, or they can be separately fabricated into individual integrated circuit modules, or multiple modules or steps among them can be fabricated into a single integrated circuit module to implement. Thus, the present invention is not limited to any specific combination of hardware and software.

[0185] The above are only the preferred embodiments of the present invention and are not used to limit the present invention. For those skilled in the art, the present invention can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A distributed frequency control method based on security policy learning, characterized in that Including: Based on historical data, perform state design and action design on each microgrid in the networked microgrid system to obtain state characteristics and control action characteristics corresponding to each of the microgrids; Configure corresponding control agents for each microgrid in the networked microgrid system, and perform centralized policy learning processing on each of the control agents based on the corresponding state characteristics and control action characteristics in the offline stage to obtain optimized control agents that meet the requirements of offline learning; Perform online distributed applications based on the optimized control agents, so that each of the microgrids performs frequency autonomous regulation processing.

2. The method according to claim 1, wherein The centralized policy learning processing of each of the control agents based on the corresponding state characteristics and control action characteristics in the offline stage to obtain optimized control agents that meet the requirements of offline learning includes: Each of the control agents initializes a corresponding actor network, and generates a current action corresponding to the current state characteristics based on the actor network, the state characteristics, and the control action characteristics; Execute the current actions corresponding to each of the control agents in the networked microgrid system to obtain transfer state characteristics corresponding to each of the control agents; and evaluate the overall performance of the current system through a global reward function to obtain the current system global reward; Integrate the current state characteristics, the current actions, the transfer state characteristics, and the current system global reward into state transition information; and store the state transition information in a replay buffer corresponding to each microgrid; Train a critic network corresponding to each control agent based on the state transition information in the replay buffer until the system global reward meets the threshold requirement to obtain the optimized control agent that meets the requirements of offline learning.

3. The method according to claim 2, characterized in that, The global reward function is: Among them, α ω and α DG are the weight coefficients of the global reward function; is the reward related to the frequency deviation; is the reward related to the power generation cost of the synchronous generator; ω ref is the frequency reference value; is the actual frequency of the i-th microgrid at time t; N is the total number of microgrids; Δt is the time interval of the control action; a i , b i , c i are the coefficients of the power generation cost of the synchronous generator respectively; is the power generation power of the distributed generation unit i at time t.

4. The method according to claim 2, wherein The centralized policy learning processing of each of the control agents based on the corresponding state characteristics and control action characteristics in the offline stage further includes: Configure a safety model between each of the microgrids and the control agents, and the safety model is used to monitor the safety of actions and guide the control agents to generate safe actions; Train the safety model based on the state transition information stored in the replay buffer corresponding to each microgrid to obtain an optimized safety model that meets the application requirements.

5. The method according to claim 4, wherein The safety model includes a safety evaluation network; The target of the safety evaluation network is determined by the transfer state characteristics, and is defined as follows: C tar (s t+1 |s t ,a t )=(ω t+1 -ω ref ) 2 Among them, S t+1 is the transfer state feature, S t is the current state feature, a t is the current action, C tar (s t+1 |s t ,a t ) is the target safety cost, ω t+1 is the actual frequency in the transfer state; ω ref is the frequency reference value; The loss function of the safety evaluation network is used to measure the gap between the network prediction value and the target safety cost, and is defined as follows: Among them, C(s t , a t |ψ) is the predicted value of the safety assessment network, S t is the current state feature, and ψ is the network parameter of the safety assessment network.

6. The method according to claim 4, characterized in that, The safety model includes an action guidance network; The action guidance network extracts the mapping relationship between actions and corresponding costs; and determines whether to generate a guidance action based on the mapping relationship; The loss function of the action guidance network is defined as follows: Among them, G(s t , c|ζ) is the generated guiding action, ζ represents the network parameter, and c represents the cost value of the action.

7. The method according to any one of claims 4 to 6, characterized in that The safety model configures operating constraint conditions; The operating constraint conditions include action constraints of distributed generation units, operation constraints of energy storage devices, and microgrid frequency constraints in the microgrid corresponding to the safety model; During the centralized policy learning process, the security model explores actions within the operating constraints.

8. A distributed frequency control device based on security policy learning, characterized in that, Including: A state design module for designing the states of each microgrid in the networked microgrid system based on historical data to obtain state characteristics corresponding to each microgrid. An offline learning module for configuring corresponding control agents for each microgrid in the networked microgrid system and performing centralized policy learning on each control agent based on the corresponding state characteristics in the offline stage to obtain optimized control agents that meet the requirements of offline learning. An online application module for performing online distributed applications based on the optimized control agents so that each microgrid performs frequency autonomous regulation processing.

9. A storage medium storing at least one executable instruction, and the executable instruction performs operations corresponding to the distributed frequency control method based on security policy learning according to any one of claims 1-7.

10. A device including a processor, a memory, a communication interface, and a communication bus, and the processor, the memory, and the communication interface complete communication with each other through the communication bus. The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the distributed frequency control method based on security policy learning according to any one of claims 1-7.

Images