Frequency control method and system based on multi-agent deep reinforcement learning

Abstract

The invention discloses a frequency control method and system based on multi-agent deep reinforcement learning, and relates to the technical field of power system frequency control, and the method comprises the steps: constructing a power system frequency response model; a training framework of centralized training and distributed decision making is introduced into a depth deterministic policy gradient (MADDPG) algorithm; a power system environment is modeled as a Markov decision process based on a depth deterministic policy gradient (MADDPG) algorithm. The invention provides a multi-region power system frequency cooperative control method based on an MADDPG algorithm. The defects of a traditional control method in a multi-region complex environment are overcome. A centralized training and decentralized execution framework is adopted, so that an intelligent agent can realize cooperative work in a multi-region power system, and the flexibility and adaptability of a control strategy are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system frequency control, and in particular to a frequency control method and system based on multi-agent deep reinforcement learning. Background Art

[0002] In the power system, frequency control is the core link to ensure the stable operation of the system. With the increasing penetration of renewable energy, traditional frequency control methods face huge challenges. This is because renewable energy such as wind power and solar power has a high degree of uncertainty and intermittency, which leads to increased frequency fluctuations in the power system. In addition, the power system is gradually developing in a distributed and intelligent direction, and traditional control methods are difficult to cope with the complex and changing power demand and supply environment. Therefore, the development of smarter and more adaptable control strategies has become a current research hotspot.

[0003] In power systems, frequency control is generally achieved through automatic generation control (AGC). The current AGC control methods can be divided into two types: traditional analytical methods and machine learning. Among the traditional analytical methods, the most common is PID control. PID calculates the control signal of the unit through frequency deviation or area control error (ACE), but the performance of PID controller is limited when dealing with complex and nonlinear systems. In order to overcome the limitations of PID controllers, advanced methods such as linear quadratic regulator (LQR), model predictive control (MPC), and robust control have been introduced into power system frequency control. In the literature, the application of LQR controller in multi-area power systems is studied, and the response characteristics of the system are optimized by Kalman filter. The prior art proposes an MPC controller and uses particle swarm optimization algorithm to adjust the controller parameters, thereby enhancing the frequency control capability of the system. Another one explores the application of robust control in systems containing wind power, and reduces frequency fluctuations and tie line power fluctuations by designing robust controllers. Although traditional analytical methods have a wide range of application foundations and theoretical support in power system frequency control, they generally have the problem of being highly dependent on system models and are unable to cope with the complexity and dynamic changes of modern power systems. In addition, in multi-regional power systems, traditional methods usually fail to fully consider inter-regional coordinated control.

[0004] In recent years, machine learning has developed rapidly, and its methods can reduce the dependence on system models in a data-driven way. Reinforcement learning (RL) does not require an accurate system model and autonomously learns the optimal strategy through interaction with the environment. It shows strong adaptability and optimization capabilities in complex dynamic systems. However, single-agent reinforcement learning only considers the impact of its own behavior on the environment, while ignoring the impact of other agents and not considering the coordination between agents. Therefore, this method is difficult to use in multi-region systems. With the development of multi-agent reinforcement learning, some people have proposed to apply multi-agent reinforcement learning (MARL) to multi-region AGC. The existing technology combines multi-agent Q learning with game theory, which promotes information sharing and strategy exchange between regions and improves the overall long-term performance of the control area. However, traditional reinforcement learning such as multi-agent Q learning can only handle discrete action domains, which limits its control performance. Therefore, deep reinforcement learning is proposed to solve the problem of continuous action domains. After the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm was proposed, some scholars introduced the MADDPG algorithm into the multi-region power system, effectively reducing the frequency deviation caused by disturbances in each region. However, the above only considered reducing the frequency deviation and the power deviation of the tie line, ignoring the economic indicators of the unit frequency regulation. Summary of the invention

[0005] In view of the above-mentioned problems, the present invention is proposed.

[0006] Therefore, the technical problem solved by the present invention is: how to effectively deal with the problem of power system frequency fluctuation and regional coordinated control in a complex environment with high penetration of renewable energy and distributed smart grid.

[0007] To solve the above technical problems, the present invention provides the following technical solutions: a frequency control method based on multi-agent deep reinforcement learning, comprising: constructing a power system frequency response model; introducing a centralized training and distributed decision-making training framework into a deep deterministic policy gradient MADDPG algorithm; and modeling the power system environment as a Markov decision process based on the deep deterministic policy gradient MADDPG algorithm.

[0008] As a preferred solution of the frequency control method based on multi-agent deep reinforcement learning described in the present invention, wherein: the power system frequency response model includes a thermal power unit model, a generator-load model, a tie line power and a regional control error model;

[0009] The dynamic model of the speed governor in the thermal power unit model is expressed as:

[0010]

[0011] The dynamic model of the steam turbine in the thermal power unit model is expressed as:

[0012]

[0013] Among them, T sgi is the time constant of the thermal power unit speed governor, Δu ti is the control signal of the thermal power unit in region i, R i is the primary frequency regulation coefficient of thermal power units in region i, Δf i is the system frequency deviation in region i, ΔP tgi is the power deviation output by the speed regulator of the thermal power unit in area i, T ti , ΔP mi They are respectively the time constant and output power deviation of the steam turbine of the thermal power unit in area i.

[0014] As a preferred solution of the frequency control method based on multi-agent deep reinforcement learning described in the present invention, the relationship between the power difference of the generator-load model and the frequency fluctuation is expressed as:

[0015]

[0016] Where Δf i represents the system frequency deviation of region i, H i represents the system inertia time constant, D i represents the system damping factor, ΔP mi and ΔP Li They represent the unit output power and load disturbance in area i, ΔP tie is the tie line power deviation between areas 1 and 2.

[0017] As a preferred solution of the frequency control method based on multi-agent deep reinforcement learning described in the present invention, the dynamic model of the tie line power between areas 1 and 2 is expressed as:

[0018]

[0019] The regional control error model is expressed as:

[0020] ACE i =ΔP tie +B i Δf i

[0021] Among them, T 12represents the synchronization coefficient between regions 1 and 2, Δf1 and Δf2 are the system frequency deviations of regions 1 and 2 respectively, and B i is the frequency response coefficient, which is determined by the equivalent regulation coefficient of the region and the frequency regulation effect coefficient of the load, and is expressed as:

[0022] As a preferred solution of the frequency control method based on multi-agent deep reinforcement learning described in the present invention, the formula for calculating the Q value of the deep deterministic policy gradient MADDPG algorithm after introducing the centralized training and distributed decision-making training framework is:

[0023]

[0024] As the Q-value calculation formula changes, the objective function and gradient calculation formula of the MADDPG critic network and actor network update parameters also change. The changed formula is expressed as:

[0025]

[0026] in, is the action of all agents at time i, m is the number of agents, x i is the set of observations of all agents, that is θ j ,φ j They represent the critic network parameters and actor network parameters of the j-th agent, respectively. Represents the observation of the j-th agent at time i.

[0027] As a preferred solution of the frequency control method based on multi-agent deep reinforcement learning described in the present invention, wherein: the modeling is a Moore payment process including state and action space design, reward function design;

[0028] The state and action space design performs proportional, integral, and differential operations on the observable quantities of the system, and the resulting state space is:

[0029] s=[Δf,Δf / s,dΔf / dt,ΔP tie ,ΔP tie / s,dΔP tie / dt,ΔACE,ΔACE / s,dΔACE / dt]

[0030] In the frequency control of the power system, the command output by the controller is the control signal of the unit. There is only one unit in each area, so the action output of the intelligent agent is set to the control signal of the thermal power unit:

[0031] a=[Δu t ].

[0032] As a preferred solution of the frequency control method based on multi-agent deep reinforcement learning described in the present invention, the reward function is designed so that the area control error ACE can be quickly reduced to 0 after the system is subjected to load disturbance, which is expressed as:

[0033]

[0034] Among them, λ1 and λ2 are weight coefficients.

[0035] In the second aspect, another object of the present invention is to provide a frequency control system based on multi-agent deep reinforcement learning, including: a power system frequency response model module, a deep deterministic policy gradient MADDPG algorithm module and a Markov decision process modeling module; the power system frequency response model module is used to construct a mathematical model of the power system frequency response, describing the dynamic changes of the frequency of the power system under different load and disturbance conditions; the deep deterministic policy gradient MADDPG algorithm module is used to introduce a training framework of centralized training and distributed decision-making to improve the coordination and response efficiency of multiple agents in the power system; the Markov decision process modeling module is used to model the power system environment as a Markov decision process, so as to facilitate the use of deep learning algorithms for optimization decisions.

[0036] In a third aspect, a computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the frequency control method based on multi-agent deep reinforcement learning as described above are implemented.

[0037] In a fourth aspect, a computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of the frequency control method based on multi-agent deep reinforcement learning as described above.

[0038] Beneficial effects of the present invention: The frequency control method and system based on multi-agent deep reinforcement learning provided by the present invention are divided into two stages: centralized training and decentralized execution. In the centralized training stage, each agent can share global observation and action information and use global information to optimize the strategy; in the decentralized execution stage, each agent generates control instructions based on the local observation information of its own area. Through simulation verification, this method can significantly improve the performance of frequency control and reduce system frequency deviation.

[0039] A multi-regional power system frequency coordinated control method based on the MADDPG algorithm is proposed, which overcomes the shortcomings of traditional control methods in multi-regional complex environments. A centralized training and decentralized execution architecture is adopted to enable intelligent agents to work together in multi-regional power systems, improving the flexibility and adaptability of the control strategy. The action space, state space, reward function and neural network structure of MADDPG are designed according to the two-region power system model, and the economic indicators of frequency regulation are considered in the reward function. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0041] Figure 1 An overall flow chart of a frequency control method based on multi-agent deep reinforcement learning provided by one embodiment of the present invention;

[0042] Figure 2 A frequency response model framework diagram of a two-region power system based on a frequency control method based on multi-agent deep reinforcement learning provided by an embodiment of the present invention;

[0043] Figure 3 A schematic diagram of random load disturbance of a frequency control method based on multi-agent deep reinforcement learning provided by one embodiment of the present invention;

[0044] Figure 4 A frequency change curve of a two-region power system of a frequency control method based on multi-agent deep reinforcement learning provided by an embodiment of the present invention;

[0045] Figure 5 A power variation curve of a tie line of a two-region power system provided by a frequency control method based on multi-agent deep reinforcement learning according to an embodiment of the present invention;

[0046] Figure 6 A schematic diagram of reward changes during the training process of a frequency control method based on multi-agent deep reinforcement learning provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0048] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0049] Example 1

[0050] Reference Figure 1-2 , is an embodiment of the present invention, and provides a frequency control method based on multi-agent deep reinforcement learning, comprising:

[0051] S1: Construct power system frequency response model;

[0052] Furthermore, the present invention adopts a non-reheat thermal power unit, and its frequency response model is usually composed of a governor model and a steam turbine model. The governor dynamic model can be represented by the differential equation shown in formula (1):

[0053]

[0054] Where, T sgi is the time constant of the thermal power unit speed governor, Δu ti is the control signal of the thermal power unit in region i, R i is the primary frequency regulation coefficient of thermal power units in region i, Δf i is the system frequency deviation in region i, ΔP tgi is the power deviation output by the speed regulator of the thermal power unit in area i.

[0055] The steam turbine adjusts the output mechanical power by changing the valve position. Since the change of mechanical power is usually slower than the change of valve opening, the first-order inertia link is often used to represent it. The dynamic differential equation of the steam turbine is:

[0056]

[0057] Where, T ti , ΔP mi They are respectively the time constant and output power deviation of the steam turbine of the thermal power unit in area i.

[0058] Generator-load model:

[0059] The relationship between the generator-load power difference and the frequency fluctuation can be expressed by the following differential equation:

[0060]

[0061] Where Δf i represents the system frequency deviation of region i, H i represents the system inertia time constant, D i represents the system damping factor, ΔP mi and ΔP Li They represent the unit output power and load disturbance in area i, ΔP tie is the tie line power deviation between areas 1 and 2.

[0062] Tie-line power and regional control error model:

[0063] The dynamic model of the tie line power between areas 1 and 2 can be expressed by formula (4):

[0064]

[0065] Where, T 12 It represents the synchronization coefficient between areas 1 and 2, and Δf1 and Δf2 are the system frequency deviations of areas 1 and 2 respectively.

[0066] The interconnected power system of the present invention adopts the tie line power and frequency deviation control (TBC) mode, which requires simultaneous detection of ΔP tie And Δf, the calculation formula of ACE is:

[0067] ACE i =ΔP tie +B i Δf i (5)

[0068] In the formula, B i is the frequency response coefficient, which is determined by the equivalent regulation coefficient of the region and the frequency regulation effect coefficient of the load and can be calculated by formula (6):

[0069]

[0070] Frequency response model of two-region power system:

[0071] In order to construct the corresponding frequency control strategy, the present invention builds the following Figure 2 The frequency response model of the two-region power system is shown in Figure 1. There is only one thermal power unit in each region. In the figure, Δu1 and Δu2 are the control signals of the thermal power units in regions 1 and 2, respectively. ΔPm1 , ΔP m2 are the output power of thermal power units in regions 1 and 2, ΔP L1 , ΔP L2 These are the load disturbances in areas 1 and 2 respectively.

[0072] S2: Introducing a centralized training and distributed decision-making training framework into the deep deterministic policy gradient MADDPG algorithm;

[0073] Furthermore, Deep Deterministic Policy Gradient (DDPG) is a deep reinforcement learning algorithm that combines policy gradient and Q learning. It can handle the problem of multi-dimensional continuous action control and greatly expand the application scope of reinforcement learning. The policy function and Q function in the DDPG algorithm are both neural networks, namely the actor network and the critic network, which update the parameters of the neural network through continuous interaction with the environment. In order to improve the stability of training, DDPG divides the neural network into a main network and a target network. The main network calculates the action or Q value at the current moment, and the target network calculates the action and Q value at the next moment. The parameters of the target network are slowly approached to the parameters of the main network through soft update. Its parameter update formula is calculated by formula (7):

[0074] θ′=τθ+(1-τ)θ′

[0075] φ′=τφ+(1-τ)φ′ (7)

[0076] Where θ′, φ′ are the parameters of the target Critic network and the target Actor network respectively; θ, φ are the parameters of the main Critic network and the main Actor network respectively.

[0077] The output of the main critic network is the Q value of the state-action. Its parameter update goal is to minimize the loss function, and the parameters are updated by the gradient descent method. The loss function of the main critic network is calculated by equation (8), and its gradient can be calculated by equation (9):

[0078]

[0079] Where N is the minimum batch size for sampling, s i 、a i are the current state and action respectively, θ represents the gradient of the parameter θ, Qθ(si,ai) is the Q value of the main network at the current moment, y i is the Q value estimated by the Bellman equation, which can be calculated by formula (10):

[0080] y i = r(si ,a i )+γQ θ' (s i+1 ,μ φ' (s i+1 )) (10)

[0081] In the formula, r(s i ,a i ) is the reward at the current moment, γ is the discount factor; Q θ' (s i+1 ,μ φ' (s i+1 )),s i+1 , μ φ' (s i+1 ), which are the target network Q value, state and action at the next moment respectively.

[0082] The main Actor network is the policy network of the intelligent agent. It outputs action instructions based on the observations of the environment. The optimization goal of its neural network parameters is to maximize the Q value, and the parameters are updated by the gradient ascent method. The objective function of the main Actor network can be calculated by formula (11), and its gradient is calculated by formula (12):

[0083]

[0084] In the formula, μ φ (s i ) is the action value output by the main Actor network according to the current state, ▽ φ represents the gradient of the parameter φ.

[0085] It should be noted that MADDPG extends the actor-critic framework in DDPG to a multi-agent environment. In a multi-agent environment, the behavior of each agent depends not only on the state of the environment, but also on the strategies of other agents. This interdependent and interactive nature makes the learning and optimization of multi-agent systems complex and challenging. In order to effectively deal with this complexity, MADDPG introduces a "centralized training, distributed decision-making" training framework. During the training process, MADDPG adopts a centralized training method. The critic network of each agent not only depends on its own state and action, but also takes into account the joint state and action of all other agents. Through this centralized critic network, the agent can better capture the complex interaction relationships in the multi-agent system, thereby providing more accurate value estimates.

[0086] MADDPG achieves distributed decision-making by designing an independent Actor-Critic architecture for each agent. Each agent has its own actor network and critic network, enabling it to make decisions based on local observations. This decentralized execution method ensures that each agent only relies on its own observation information during execution, making the system more flexible and efficient in practical applications.

[0087] Furthermore, the parameter update formula of each agent neural network in MADDPG is basically the same as that of DDPG. However, since it uses a centralized critic network, it not only considers its own state and action, but also the state and action of other agents. Therefore, the formula for calculating Q value becomes:

[0088]

[0089] In the formula, is the action of all agents at time i, m is the number of agents, x i is the set of observations of all agents, that is

[0090] Therefore, as the Q value calculation formula changes, the objective function and gradient calculation formula of the MADDPG critic network and actor network update parameters also change. The changed formula is shown in formula (14):

[0091]

[0092] In the formula, θ j ,φ j They represent the critic network parameters and actor network parameters of the j-th agent, respectively. Represents the observation of the j-th agent at time i.

[0093] S3: Modeling the power system environment as a Markov decision process based on the deep deterministic policy gradient MADDPG algorithm.

[0094] Furthermore, in reinforcement learning, the agent takes actions based on the current state information of the environment and returns it to the environment. The environment that receives the action will return the next state and reward value. This interactive process is called Markov decision process (MDP), which is the basic framework of reinforcement learning. To use MADDPG to solve the frequency control problem of the power system, it is necessary to model the power system environment as a Markov decision process. The state space, action space and reward function are the core components of the Markov decision process and need to be reasonably designed according to the power system environment.

[0095] State and action space design:

[0096] In the two-area power system constructed by the present invention, the observable state quantities are frequency deviation, tie line power, and regional control error ACE. In order to allow the intelligent agent to receive more state feature information, the present invention performs proportional, integral, and differential operations on the observable quantities of the system, and the obtained state space is:

[0097] s=[Δf,Δf / s,dΔf / dt,ΔP tie ,ΔP tie / s,dΔP tie / dt,ΔACE,ΔACE / s,dΔACE / dt] (15)

[0098] In the frequency control of the power system, the command output by the controller is the control signal of the unit. In the system of the present invention, each area has only one unit, so the action output of the intelligent agent is set to be the control signal of the thermal power unit:

[0099] a=[Δu t ] (16)

[0100] Reward function design:

[0101] During the training process, the agent will optimize the strategy in the direction of maximizing the reward function value based on the trial-and-error exploration mechanism. Therefore, the reward function plays a role in guiding the agent's learning and decision-making in reinforcement learning. By rationally designing the reward function, the agent can learn a better strategy. The control goal of the system of the present invention is to enable the system to quickly reduce ACE to 0 after being disturbed by the load, and considering the frequency regulation cost, it is hoped that the output power fluctuation of the thermal power unit is as small as possible. Therefore, the reward function designed by the present invention is:

[0102]

[0103] In the formula, λ1 and λ2 are weight coefficients,

[0104] Neural network structure:

[0105] In the design of AGC controller based on MADDPG, the neural network structure is crucial for deep reinforcement learning. The neural network can capture complex environmental features and behavior patterns through multi-level nonlinear transformations, thereby enhancing the decision-making ability of the intelligent agent. The neural network structure set by the present invention is as follows Figure 3As shown in the figure, a fully connected layer and a relu activation function are used. The advantage of the fully connected layer is its versatility and powerful representation ability. It can effectively learn the high-dimensional mapping relationship between input data and output results. At the same time, the introduction of the relu activation function can avoid the gradient vanishing problem, improve training efficiency, and make it easier for the network to converge to the global optimal solution. This combination improves training stability and performance while ensuring network complexity.

[0106] Example 2

[0107] In one embodiment of the present invention, a mixed environmentally friendly gas proportioning system is provided, comprising:

[0108] Power system frequency response model module, deep deterministic policy gradient MADDPG algorithm module and Markov decision process modeling module;

[0109] The power system frequency response model module is used to construct a mathematical model of the power system frequency response and describe the dynamic frequency changes of the power system under different load and disturbance conditions;

[0110] The deep deterministic policy gradient MADDPG algorithm module is used to introduce a centralized training and distributed decision-making training framework to improve the coordination and response efficiency of multiple agents in the power system;

[0111] The Markov decision process modeling module is used to model the power system environment as a Markov decision process to facilitate the use of deep learning algorithms for optimization decisions.

[0112] Example 3

[0113] An embodiment of the present invention is different from the first two embodiments in that:

[0114] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0115] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in conjunction with these instruction execution systems, apparatuses or devices. For the purpose of this specification,

[0116] A "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0117] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0118] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] Example 4

[0120] See also Figure 3 to Figure 6 , which is an embodiment of the present invention, provides a frequency control method based on multi-agent deep reinforcement learning. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0121] Build in MATLAB / Simulink Figure 1The frequency response model of the power system in the two regions is shown in Table 1. The sampling time of the MADDPG algorithm is set to 1s, and the Actor network learning rate, Critic network learning rate, experience buffer size and discount factor are set to 0.0001, 0.001, 1e6 and 0.99 respectively.

[0122] Table 1 Model parameters

[0123]

[0124] In this paper, it is assumed that the initial state of the system is stable. A random load disturbance is imposed on area 1 of the system. The disturbance changes as follows: Figure 3 In order to analyze the control effect of MADDPG controller, this paper will compare it with the traditional PID controller. The simulation results are shown in Figure 4 and Figure 5 shown.

[0125] Figure 4 The frequency changes of regions 1 and 2 after the system is subjected to load disturbance are shown. It can be seen from the figure that compared with the PID controller, the controller based on the MADDPG algorithm can reduce the frequency fluctuation caused by load disturbance and can quickly reduce the frequency deviation to 0. The frequency fluctuation range of Figure (b) is obviously smaller than that of Figure (a). The reason is that this paper only applies disturbance to region 1, so the impact on region 1 is also the greatest. Figure 6 The power change of the system's interconnection line is shown. It can be seen from the figure that after being disturbed, the method proposed in this paper and the PID controller can restore the interconnection line power to 0, but the controller based on the MADDPG algorithm has a faster recovery speed.

[0126] Figure 6 This is the change in the reward value of the MADDPG algorithm during the training process. When the average reward value reaches a stable state, the algorithm can be considered to have converged. Whether the algorithm converges during training is related to the design of the reward function and the structure of the neural network. As can be seen from the figure, when the training reaches about 60 rounds, the reward value has stabilized and the algorithm converges quickly, indicating that the reward function and neural network structure designed in this paper are reasonable.

[0127] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A frequency control method based on multi-agent deep reinforcement learning, characterized in that: include: Construct power system frequency response model; Introducing a centralized training and distributed decision-making training framework into the deep deterministic policy gradient MADDPG algorithm; The power system environment is modeled as a Markov decision process based on the deep deterministic policy gradient (MADDPG) algorithm.

2. The frequency control method based on multi-agent deep reinforcement learning as claimed in claim 1, characterized in that: The power system frequency response model includes a thermal power unit model, a generator-load model, a tie line power and a regional control error model; The dynamic model of the speed governor in the thermal power unit model is expressed as: The dynamic model of the steam turbine in the thermal power unit model is expressed as: Among them, T sgi is the time constant of the thermal power unit speed governor, Δu ti is the control signal of the thermal power unit in region i, R i is the primary frequency regulation coefficient of thermal power units in region i, Δf i is the system frequency deviation in region i, ΔP tgi is the power deviation output by the speed regulator of the thermal power unit in area i, T ti , ΔP mi They are respectively the time constant and output power deviation of the steam turbine of the thermal power unit in area i.

3. The frequency control method based on multi-agent deep reinforcement learning as claimed in claim 2, characterized in that: The relationship between the generator-load model power difference and frequency fluctuation is expressed as: Where Δf i represents the system frequency deviation of region i, H i represents the system inertia time constant, D i represents the system damping factor, ΔP mi and ΔP Li They represent the unit output power and load disturbance in area i, ΔP tie is the tie line power deviation between areas 1 and 2.

4. The frequency control method based on multi-agent deep reinforcement learning as claimed in claim 3, characterized in that: The dynamic model of the tie line power between areas 1 and 2 is expressed as: The regional control error model is expressed as: ACE i =ΔP tie +B i Δf i Among them, T 12 represents the synchronization coefficient between regions 1 and 2, Δf1 and Δf2 are the system frequency deviations of regions 1 and 2 respectively, and B i is the frequency response coefficient, which is determined by the equivalent regulation coefficient of the region and the frequency regulation effect coefficient of the load, and is expressed as:

5. The frequency control method based on multi-agent deep reinforcement learning as claimed in claim 4, characterized in that: After introducing the centralized training and distributed decision-making training framework, the formula for calculating the Q value of the deep deterministic policy gradient MADDPG algorithm is: As the Q-value calculation formula changes, the objective function and gradient calculation formula of the MADDPG critic network and actor network update parameters also change. The changed formula is expressed as: in, is the action of all agents at time i, m is the number of agents, x i is the set of observations of all agents, that is θ j ,φ j They represent the critic network parameters and actor network parameters of the j-th agent, respectively. Represents the observation of the j-th agent at time i.

6. The frequency control method based on multi-agent deep reinforcement learning as claimed in claim 5, characterized in that: The modeling is a Moore's Law process including state and action space design, and reward function design; The state and action space design performs proportional, integral, and differential operations on the observable quantities of the system, and the resulting state space is: s=[Δf,Δf / s,dΔf / dt,ΔP tie ,ΔP tie / s,dΔP tie / dt,ΔACE,ΔACE / s,dΔACE / dt] In the frequency control of the power system, the command output by the controller is the control signal of the unit. There is only one unit in each area, so the action output of the intelligent agent is set to the control signal of the thermal power unit: a=[Δu t ]。 7. The frequency control method based on multi-agent deep reinforcement learning according to claim 6, characterized in that: The reward function is designed so that the area control error ACE can be quickly reduced to 0 after the system is disturbed by the load, which is expressed as: Among them, λ1 and λ2 are weight coefficients.

8. A system using the frequency control method based on multi-agent deep reinforcement learning as described in any one of claims 1 to 7, characterized in that: include: Power system frequency response model module, deep deterministic policy gradient MADDPG algorithm module and Markov decision process modeling module; The power system frequency response model module is used to construct a mathematical model of the power system frequency response, describing the dynamic frequency changes of the power system under different load and disturbance conditions; The deep deterministic policy gradient MADDPG algorithm module is used to introduce a centralized training and distributed decision-making training framework to improve the coordination and response efficiency of multiple agents in the power system; The Markov decision process modeling module is used to model the power system environment as a Markov decision process to facilitate optimization decision making using a deep learning algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the frequency control method based on multi-agent deep reinforcement learning described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the frequency control method based on multi-agent deep reinforcement learning described in any one of claims 1 to 7 are implemented.

Images