Optimal primary frequency control method and system based on reinforcement learning

Abstract

The invention provides an optimal primary frequency control method based on reinforcement learning. The method comprises the following steps: establishing a dynamic model of a power system; designing a frequency control strategy based on a deep reinforcement learning algorithm; the system frequency deviation is used as a reward signal, and a control strategy is trained through interaction with a power system; lyapunov function stability constraints are introduced into the reinforcement learning algorithm; a neural network is adopted to carry out parameterization design on the controller; discretizing the frequency dynamic equation, and proposing a multi-node collaborative optimization target; and deploying the trained control strategy to an actual power system, and adjusting control parameters in real time through an online optimization technology to adapt to the dynamic change of the system operation state. By means of the method, the frequency control accuracy and stability of the power system are improved, and the global optimal performance of frequency control of the whole power grid is achieved in actual deployment.

Description

Technical Field

[0001] This invention relates to the field of power system frequency control technology, specifically to an optimal primary frequency control method and system based on reinforcement learning. Background Technology

[0002] In power systems, frequency control is a crucial element in ensuring stable operation. Traditional frequency control methods primarily rely on linear droop controllers. However, with the increasing penetration of inverter infrastructure in power systems, these resources offer greater flexibility and enable more complex control functions. Nevertheless, designing a nonlinear optimal controller is no easy task, as it requires optimization in an infinite-dimensional function space, and the controller must guarantee system stability.

[0003] With the rapid development of renewable energy, wind power and photovoltaic power generation have become indispensable components of the power system. However, due to their intermittency and instability, they face some challenges in terms of smooth operation and frequency control compared to traditional power sources. To address these issues, Simulink is a powerful tool that can perform primary frequency regulation on the power system to control the frequency and improve system stability.

[0004] Lyapunov stability theory is a theory used to determine the stability of dynamic systems at their equilibrium points. It describes how the system's state changes when subjected to small disturbances from its equilibrium state. Lyapunov stability can be categorized into Lyapunov stability, asymptotic stability, and global asymptotic stability. In power system frequency control, Lyapunov stability theory is widely used in the analysis and design of controllers to ensure system stability.

[0005] Neural network parameterized controllers are a method that uses neural networks to parameterize controllers, exhibiting excellent approximation and generalization capabilities. CMAC (Cerebellar Model Articulated Controller) is a neural network that mimics the function of the cerebellum, used to map the input space to the output space, and also possesses excellent approximation and generalization abilities. In power system frequency control, neural network parameterized controllers can be used to design nonlinear controllers to achieve better control performance.

[0006] Recurrent Neural Networks (RNNs) are deep learning models used to process sequential data. They possess internal states, enabling them to process sequential data and capture long-term dependencies within the sequence. RNNs have achieved significant success in fields such as natural language processing and time series prediction. In power system frequency control, RNNs can be used to handle the time-coupled state dynamics of the power system, achieving more precise frequency control.

[0007] In summary, traditional power system frequency control methods mainly rely on linear droop controllers, but with the increasing penetration of inverter infrastructure in power systems, these methods can no longer meet the needs of modern power systems. Summary of the Invention

[0008] The technical problem to be solved by this invention is to improve the accuracy and stability of frequency control in power systems and to achieve the global optimal performance of frequency control for the entire power grid in actual deployment.

[0009] The present invention solves the above-mentioned technical problems through the following technical means:

[0010] This invention provides an optimal first-order frequency control method based on reinforcement learning, comprising:

[0011] S1. Establish a dynamic model of the power system, including the inertia and damping characteristic model of traditional generators, the dynamic behavior model of inverter basic resources, and the impact model of load changes on system frequency.

[0012] S2. Based on deep reinforcement learning algorithm, design frequency control strategy; and use system frequency deviation as reward signal to train control strategy through interaction with power system;

[0013] S3. Introduce Lyapunov function stability constraints into reinforcement learning algorithms;

[0014] S4. Use neural networks to parametrically design the controller;

[0015] S5. Discretize the frequency dynamic equation and propose a multi-node collaborative optimization objective; deploy the trained control strategy to the actual power system, and adjust the control parameters in real time through online optimization technology to adapt to the dynamic changes in the system's operating state.

[0016] Further, S1 includes the following steps:

[0017] S11. Construct the inertia and damping characteristic model of a traditional generator as follows:

[0018]

[0019] Where Δf is the frequency deviation, M is the system's inertial constant, and ΔP m It is the increase in the mechanical power of the generator, ΔP e It is the electrical power increment of the system, and D is the damping coefficient;

[0020] S12. Construct the dynamic behavior model of the inverter's basic resources as follows:

[0021]

[0022] Where τ is the time constant of the inverter, and u is the control input signal generated by the frequency controller;

[0023] S13. Construct the model of the impact of load changes on system frequency as follows:

[0024]

[0025] Where, ΔP L It is the change in load power.

[0026] Further, S2 includes the following steps:

[0027] S21. Model the reinforcement learning problem as a Markov decision process, whose elements are: states s constructed from system frequency deviation and inverter power increment. t =[Δf t ,ΔP e Action a constructed from control input signals t =u; the reward function constructed from frequency deviation and control cost. Where α is a weighting factor used to balance frequency deviation and control power cost; by optimizing the strategy π(a t |s t Maximize cumulative discount rewards:

[0028]

[0029] Where γ∈[0,1] is the discount factor;

[0030] S22. A recurrent neural network (RNN) is used to model the time coupling characteristics and capture the dynamic time series characteristics of frequency changes; the RNN hidden state update equation is:

[0031] h t =f(h t-1 ,s t ;θ)

[0032] Among them, h t Let f be the current hidden state, f be the activation function of the RNN, and θ be the RNN parameters;

[0033] S23. The output layer of a recurrent neural network (RNN) generates a control signal u as follows:

[0034] u = W0h t +b0

[0035] Where W0 and b0 are the weights and biases of the output layer, respectively, and u is the control input signal of the frequency controller.

[0036] Furthermore, S3 specifically refers to:

[0037] Introducing the Lyapunov function V(s) t ), defined as follows:

[0038]

[0039] The stability condition for Lyapunov functions is as follows:

[0040]

[0041] Combining the inertia and damping characteristic model of a traditional generator, stability constraints are obtained:

[0042]

[0043] Furthermore, S4 specifically includes:

[0044] The controller uses a neural network parameterization form, as shown in the following equation:

[0045] u = π θ (s t )

[0046] Where, π θ It is a policy network that optimizes parameters θ through deep learning; the input to the neural network is the current state s. t The output is a control signal u;

[0047] Define the loss function L(θ) that minimizes the training objective of the neural network as:

[0048]

[0049] Here, λ is a penalty factor used to balance performance optimization and stability constraints.

[0050] Further, S5 includes the following steps:

[0051] S51. The continuous-time form of the frequency dynamic equation is as follows:

[0052]

[0053] Where, ω i Let f(ω) be the frequency deviation of the i-th generator. i ,u i This describes the dynamic change of frequency, u i It is the control input of the i-th generator;

[0054] Discretizing the frequency dynamic equation yields:

[0055] ωi (k)=ω i (k-1)+Δt·f(ω i (k-1),u i (k-1))

[0056] Where, ω i (k) represents the frequency deviation of the i-th generator, and Δt is the discrete time step;

[0057] S52. In discrete time, the objective of multi-node collaborative optimization is as follows:

[0058]

[0059] Where, ||ω i || ∞ The maximum value of the frequency deviation is used to measure the frequency control performance, ∑ k u i (k) 2 The sum of squares of the control signals is used to measure the overall control energy consumption of the system.

[0060] S53. Deploy the trained controller to the power system and further enhance its performance using real-time optimization techniques:

[0061] u t =π θ (s t )+Δu

[0062] Where Δu is the fine-tuning amount for real-time optimization, calculated using the fast gradient descent method, as shown in the following equation:

[0063]

[0064] Where η is the learning rate.

[0065] This invention also provides an optimal first-order frequency control system based on reinforcement learning. The system operates using the aforementioned method and includes the following modules:

[0066] The dynamic model building module is used to build dynamic models of power systems, including inertia and damping characteristic models of traditional generators, dynamic behavior models of inverter basic resources, and models of the impact of load changes on system frequency.

[0067] The control strategy design module is used to design frequency control strategies based on deep reinforcement learning algorithms; and uses the system frequency deviation as a reward signal to train the control strategy through interaction with the power system.

[0068] The stability constraint module is used to introduce stability constraints for the Lyapunov function in reinforcement learning algorithms.

[0069] The parameter design module is used to perform parameter design of the controller using neural networks;

[0070] The collaborative optimization module is used to discretize the frequency dynamic equation and propose a multi-node collaborative optimization objective. The trained control strategy is deployed to the actual power system, and the control parameters are adjusted in real time through online optimization technology to adapt to the dynamic changes in the system's operating state.

[0071] Furthermore, the dynamic model construction module includes the following units:

[0072] The inertia and damping characteristic model building unit is used to construct the inertia and damping characteristic model of a traditional generator, as shown in the following equation:

[0073]

[0074] Where Δf is the frequency deviation, M is the system's inertial constant, and ΔP m It is the increase in the mechanical power of the generator, ΔP e It is the electrical power increment of the system, and D is the damping coefficient;

[0075] The inverter dynamic behavior model building unit is used to construct the dynamic behavior model of the inverter's basic resources, as shown in the following equation:

[0076]

[0077] Where τ is the time constant of the inverter, and u is the control input signal generated by the frequency controller;

[0078] The load impact model building unit is used to construct the model of the impact of load changes on system frequency, as shown in the following equation:

[0079]

[0080] Where, ΔP L It is the change in load power.

[0081] Furthermore, the control strategy design module includes the following units:

[0082] Markov decision building blocks are used to model reinforcement learning problems as Markov decision processes, whose elements are: states s constructed from system frequency deviation and inverter power increments. t =[Δf t ,ΔP e Action a constructed from control input signals t =u; the reward function constructed from frequency deviation and control cost. Where α is a weighting factor used to balance frequency deviation and control power cost; by optimizing the strategy π(a t |st Maximize cumulative discount rewards:

[0083]

[0084] Where γ∈[0,1] is the discount factor;

[0085] The neural network modeling unit is used to model the temporal coupling characteristics of recurrent neural networks (RNNs) and capture the dynamic time series characteristics; the RNN hidden state update equation is:

[0086] h t =f(h t-1 ,s t ;θ)

[0087] Among them, h t Let f be the current hidden state, f be the activation function of the RNN, and θ be the RNN parameters;

[0088] The control signal output unit is used to generate the control signal u at the output layer of the recurrent neural network (RNN) as follows:

[0089] u = W0h t +b0

[0090] Where W0 and b0 are the weights and biases of the output layer, respectively, and u is the control input signal of the frequency controller.

[0091] Furthermore, the stability constraint module operates as follows:

[0092] Introducing the Lyapunov function V(s) t ), defined as follows:

[0093]

[0094] The stability condition for Lyapunov functions is as follows:

[0095]

[0096] Combining the inertia and damping characteristic model of a traditional generator, stability constraints are obtained:

[0097]

[0098] The advantages of this invention are:

[0099] (1) This invention achieves adaptive optimization of the control strategy by combining a reinforcement learning framework with the dynamic characteristics of a recurrent neural network (RNN) modeling system, without relying on the precise modeling of system parameters using traditional control methods. The reinforcement learning strategy can continuously improve during training, adapting to complex scenarios with different load conditions and high-proportion renewable energy penetration.

[0100] (2) By incorporating the design of RNN, this invention can effectively capture the time-series characteristics of dynamic frequency changes in power systems, thus improving the ability to model complex dynamic behaviors. Compared with traditional methods, RNN can handle long-term dependencies, significantly improving the control effect on nonlinear systems.

[0101] (3) By introducing Lyapunov function stability constraints into the control strategy design, the Lyapunov function and stability constraints are constructed to ensure that the system maintains frequency stability under any control strategy, thus avoiding the instability that may occur in traditional optimization methods. The introduction of stability constraints makes the control strategy more reliable and suitable for application in practical complex power systems.

[0102] (4) The design of the reinforcement learning reward function balances the frequency deviation and control energy consumption, enabling the controller to effectively reduce control energy consumption while quickly restoring the frequency, thus achieving global optimization.

[0103] (5) This invention is specifically designed for high-penetration renewable energy access scenarios, proposing a multi-point collaborative optimization objective that comprehensively considers the maximum frequency deviation of the entire network and control energy consumption to achieve global optimization rather than local optimization. It can adapt to the volatility and randomness of new energy sources, effectively improving the robustness and flexibility of the power system. In scenarios where wind power, photovoltaic, and other new energy generation experiences frequent fluctuations, this method can significantly improve the stability and control efficiency of the system.

[0104] (6) Controller Design and Computational Efficiency Optimization: Neural network parameterized controllers have powerful nonlinear approximation capabilities, simplifying the design process of complex controllers. Using RNNs as the core architecture of the controller not only improves the ability to model sequence data, but also enhances training and prediction efficiency through parallel computation.

[0105] (7) It has good practical application value. The design of this method fully considers the complexity of modern power systems and can be seamlessly connected to existing power dispatch and control systems. Attached Figure Description

[0106] Figure 1 This is a flowchart illustrating the optimal first-order frequency control method based on reinforcement learning according to the present invention. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0108] Example 1

[0109] This embodiment provides an optimal first-order frequency control method based on reinforcement learning, such as... Figure 1 As shown, the specific implementation steps include:

[0110] S1. Establish a dynamic model of the power system, including the inertia and damping characteristic model of traditional generators, the dynamic behavior model of inverter foundation resources, and the model of the impact of load changes on system frequency. The specific implementation process is as follows:

[0111] S11. Construct the inertia and damping characteristic model of a traditional generator as follows:

[0112]

[0113] Where Δf is the frequency deviation, M is the system's inertial constant, and ΔP m It is the increase in the mechanical power of the generator, ΔP e It is the electrical power increment of the system, and D is the damping coefficient;

[0114] S12. Construct the dynamic behavior model of the inverter's basic resources as follows:

[0115]

[0116] Where τ is the time constant of the inverter, and u is the control input signal generated by the frequency controller;

[0117] S13. Construct the model of the impact of load changes on system frequency as follows:

[0118]

[0119] In power systems, load changes directly affect system frequency. When the load increases, the system's electrical power demand rises. If generation power is not balanced in time, the system frequency will decrease; conversely, if the load decreases, the system frequency will increase. The impact of load changes on system frequency can be described by the following formula:

[0120]

[0121] Where, ΔP L It is the change in load power.

[0122] In practical systems, load variations are usually correlated with the system frequency. This can be expressed using the load frequency characteristic coefficient D. L To describe the load's response to frequency changes:

[0123] ΔP L =D L Δf (2)

[0124] Among them, D L Δf is the load frequency characteristic coefficient, representing the load's sensitivity to frequency changes; Δf is the frequency deviation. Substituting equation (2) into equation (1) and rearranging, we get...

[0125]

[0126] This indicates that the system's equivalent damping coefficient is determined by D and D0. L Together, they determine the load frequency characteristic D L The larger the value, the stronger the system's ability to adapt to load changes and the smaller the frequency deviation.

[0127] S2. Based on deep reinforcement learning algorithms, a frequency control strategy is designed; and the system frequency deviation is used as a reward signal to train the control strategy through interaction with the power system. The specific implementation process is as follows:

[0128] S21. Model the reinforcement learning problem as a Markov decision process, whose elements are: states s constructed from system frequency deviation and inverter power increment. t =[Δf t ,ΔP e Action a constructed from control input signals t =u; the reward function constructed from frequency deviation and control cost. Where α is a weighting factor used to balance frequency deviation and control power cost; by optimizing the strategy π(a t |s t Maximize cumulative discount rewards:

[0129]

[0130] Where γ∈[0,1] is the discount factor;

[0131] S22. A recurrent neural network (RNN) is used to model the time coupling characteristics and capture the dynamic time series characteristics of frequency changes; the RNN hidden state update equation is:

[0132] h t =f(h t-1 ,s t ;θ)

[0133] Among them, ht Let f be the current hidden state, f be the activation function of the RNN, and θ be the RNN parameters;

[0134] S23. The output layer of a recurrent neural network (RNN) generates a control signal u as follows:

[0135] u = W0h t +b0

[0136] Where W0 and b0 are the weights and biases of the output layer, respectively, and u is the control input signal of the frequency controller.

[0137] S3. Introduce Lyapunov function stability constraints into the reinforcement learning algorithm. This ensures that the learned frequency control strategy not only optimizes performance metrics but also satisfies system stability requirements. The specific implementation process is as follows:

[0138] Introducing the Lyapunov function V(s) t ), defined as follows:

[0139]

[0140] The stability condition for Lyapunov functions is as follows:

[0141]

[0142] Combining the inertia and damping characteristic model of a traditional generator, stability constraints are obtained:

[0143]

[0144] The above conditions are introduced into reinforcement learning training as constraints to ensure the stability of the frequency control strategy.

[0145] S4. A neural network is used for parameterized controller design, leveraging its nonlinear fitting capability to improve response accuracy to complex frequency dynamics. The specific implementation process is as follows:

[0146] The controller uses a neural network parameterization form, as shown in the following equation:

[0147] u = π θ (s t )

[0148] Where, π θ It is a policy network that optimizes parameters θ through deep learning; the input to the neural network is the current state s. t The output is a control signal u;

[0149] Define the loss function L(θ) that minimizes the training objective of the neural network as:

[0150]

[0151] Here, λ is a penalty factor used to balance performance optimization and stability constraints.

[0152] S5. Discretize the frequency dynamic equations and propose a multi-node collaborative optimization objective; deploy the trained control strategy to the actual power system, and adjust the control parameters in real time through online optimization technology to adapt to the dynamic changes in the system's operating state. The specific implementation process is as follows:

[0153] S51. The continuous-time form of the frequency dynamic equation is as follows:

[0154]

[0155] Where, ω i Let f(ω) be the frequency deviation of the i-th generator. i ,u i This describes the dynamic change of frequency, u i It is the control input of the i-th generator;

[0156] Discretizing the frequency dynamic equation yields:

[0157] ω i (k)=ω i (k-1)+Δt·f(ω i (k-1),u i (k-1))

[0158] Where, ω i (k) represents the frequency deviation of the i-th generator, and Δt is the discrete time step;

[0159] S52. In discrete time, the objective of multi-node collaborative optimization is as follows:

[0160]

[0161] Where, ||ω i || ∞ The maximum value of the frequency deviation is used to measure the frequency control performance, ∑ k u i (k) 2 The sum of squares of the control signals is used to measure the overall control energy consumption of the system.

[0162] S53. Deploy the trained controller to the power system and further enhance its performance using real-time optimization techniques:

[0163] u t =π θ (s t )+Δu

[0164] Where Δu is the fine-tuning amount for real-time optimization, calculated using the fast gradient descent method, as shown in the following equation:

[0165]

[0166] Where η is the learning rate.

[0167] Example 2

[0168] It should be further explained that, based on the same inventive concept, this embodiment also provides an optimal primary frequency control system based on reinforcement learning. The system operates using the method described in Embodiment 1, including the following modules:

[0169] The dynamic model building module is used to establish dynamic models of the power system, including inertia and damping characteristic models of traditional generators, dynamic behavior models of inverter basic resources, and models of the impact of load changes on system frequency. The dynamic model building module includes the following units:

[0170] The inertia and damping characteristic model building unit is used to construct the inertia and damping characteristic model of a traditional generator, as shown in the following equation:

[0171]

[0172] Where Δf is the frequency deviation, M is the system's inertial constant, and ΔP m It is the increase in the mechanical power of the generator, ΔP e It is the electrical power increment of the system, and D is the damping coefficient;

[0173] The inverter dynamic behavior model building unit is used to construct the dynamic behavior model of the inverter's basic resources, as shown in the following equation:

[0174]

[0175] Where τ is the time constant of the inverter, and u is the control input signal generated by the frequency controller;

[0176] The load impact model building unit is used to construct the model of the impact of load changes on system frequency, as shown in the following equation:

[0177]

[0178] Where, ΔP L It is the change in load power.

[0179] The control strategy design module is used to design a frequency control strategy based on a deep reinforcement learning algorithm; and uses the system frequency deviation as a reward signal to train the control strategy through interaction with the power system. The control strategy design module includes the following units:

[0180] Markov decision building blocks are used to model reinforcement learning problems as Markov decision processes, whose elements are: states s constructed from system frequency deviation and inverter power increments. t =[Δf t ,ΔP e Action a constructed from control input signals t =u; the reward function constructed from frequency deviation and control cost. Where α is a weighting factor used to balance frequency deviation and control power cost; by optimizing the strategy π(a t |s t Maximize cumulative discount rewards:

[0181]

[0182] Where γ∈[0,1] is the discount factor;

[0183] The neural network modeling unit is used to model the temporal coupling characteristics of recurrent neural networks (RNNs) and capture the dynamic time series characteristics; the RNN hidden state update equation is:

[0184] h t =f(h t-1 ,s t ;θ)

[0185] Among them, h t Let f be the current hidden state, f be the activation function of the RNN, and θ be the RNN parameters;

[0186] The control signal output unit is used to generate the control signal u at the output layer of the recurrent neural network (RNN) as follows:

[0187] u = W0h t +b0

[0188] Where W0 and b0 are the weights and biases of the output layer, respectively, and u is the control input signal of the frequency controller.

[0189] The stability constraint module is used to introduce Lyapunov function stability constraints into reinforcement learning algorithms. The specific operation of the stability constraint module is as follows:

[0190] Introducing the Lyapunov function V(s) t ), defined as follows:

[0191]

[0192] The stability condition for Lyapunov functions is as follows:

[0193]

[0194] Combining the inertia and damping characteristic model of a traditional generator, stability constraints are obtained:

[0195]

[0196] The parameter design module is used to perform parameter design of the controller using neural networks;

[0197] The collaborative optimization module is used to discretize the frequency dynamic equation and propose a multi-node collaborative optimization objective. The trained control strategy is deployed to the actual power system, and the control parameters are adjusted in real time through online optimization technology to adapt to the dynamic changes in the system's operating state.

[0198] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An optimal first-order frequency control method based on reinforcement learning, characterized in that, include: S1. Establish a dynamic model of the power system, including the inertia and damping characteristic model of traditional generators, the dynamic behavior model of inverter basic resources, and the impact model of load changes on system frequency. S2. Based on deep reinforcement learning algorithm, design frequency control strategy; and use system frequency deviation as reward signal to train control strategy through interaction with power system; S3. Introduce Lyapunov function stability constraints into reinforcement learning algorithms; S4. Use neural networks to parametrically design the controller; S5. Discretize the frequency dynamic equation and propose a multi-node collaborative optimization objective; The trained control strategy is deployed to the actual power system, and the control parameters are adjusted in real time through online optimization technology to adapt to the dynamic changes in the system's operating status.

2. The optimal first-order frequency control method based on reinforcement learning according to claim 1, characterized in that, S1 includes the following steps: S11. Construct the inertia and damping characteristic model of a traditional generator as follows: Where Δf is the frequency deviation, M is the system's inertial constant, and ΔP m It is the increase in the mechanical power of the generator, ΔP e It is the electrical power increment of the system, and D is the damping coefficient; S12. Construct the dynamic behavior model of the inverter's basic resources as follows: Where τ is the time constant of the inverter, and u is the control input signal generated by the frequency controller; S13. Construct the model of the impact of load changes on system frequency as follows: Where, ΔP L It is the change in load power.

3. The optimal first-order frequency control method based on reinforcement learning according to claim 2, characterized in that, S2 includes the following steps: S21. Model the reinforcement learning problem as a Markov decision process, whose elements are: states s constructed from system frequency deviation and inverter power increment. t =[Δf t ,ΔP e Action a constructed from control input signals t =u; the reward function constructed from frequency deviation and control cost. Where α is a weighting factor used to balance frequency deviation and control power cost; by optimizing the strategy π(a t |s t Maximize cumulative discount rewards: Where γ∈[0,1] is the discount factor; S22. A recurrent neural network (RNN) is used to model the time coupling characteristics and capture the dynamic time series characteristics of frequency changes; the RNN hidden state update equation is: h t =f(h t-1 ,s t ;θ) Among them, h t Let f be the current hidden state, f be the activation function of the RNN, and θ be the RNN parameters; S23. The output layer of a recurrent neural network (RNN) generates a control signal u as follows: u=W0h t +b0 Where W0 and b0 are the weights and biases of the output layer, respectively, and u is the control input signal of the frequency controller.

4. The optimal first-order frequency control method based on reinforcement learning according to claim 3, characterized in that, Specifically, S3 is: Introducing the Lyapunov function V(s) t ), defined as follows: The stability condition for Lyapunov functions is as follows: Combining the inertia and damping characteristic model of a traditional generator, stability constraints are obtained:

5. The optimal first-order frequency control method based on reinforcement learning according to claim 4, characterized in that, Specifically, S4 is: The controller uses a neural network parameterization form, as shown in the following equation: u=π θ (s t ) Where, π θ It is a policy network that optimizes parameters θ through deep learning; the input to the neural network is the current state s. t The output is a control signal u; Define the loss function L(θ) that minimizes the training objective of the neural network as: Here, λ is a penalty factor used to balance performance optimization and stability constraints.

6. The optimal first-order frequency control method based on reinforcement learning according to claim 5, characterized in that, The S5 Includes the following steps: S51. The continuous-time form of the frequency dynamic equation is as follows: Where, ω i Let f(ω) be the frequency deviation of the i-th generator. i ,u i This describes the dynamic change of frequency, u i It is the control input of the i-th generator; Discretizing the frequency dynamic equation yields: oh i (k)=ω i (k-1)+Δt·f(ω i (k-1),u i (k-1)) Where, ω i (k) represents the frequency deviation of the i-th generator, and Δt is the discrete time step; S52. In discrete time, the objective of multi-node collaborative optimization is as follows: Where, ||ω i || ∞ The maximum value of the frequency deviation is used to measure the frequency control performance, ∑ k u i (k) 2 The sum of squares of the control signals is used to measure the overall control energy consumption of the system. S53. Deploy the trained controller to the power system and further enhance its performance using real-time optimization techniques: you t =π θ (s t )+Du Where Δu is the fine-tuning amount for real-time optimization, calculated using the fast gradient descent method, as shown in the following equation: Where η is the learning rate.

7. An optimal primary frequency control system based on reinforcement learning, characterized in that, Includes the following modules: The dynamic model building module is used to build dynamic models of power systems, including inertia and damping characteristic models of traditional generators, dynamic behavior models of inverter basic resources, and models of the impact of load changes on system frequency. The control strategy design module is used to design frequency control strategies based on deep reinforcement learning algorithms. The system frequency deviation is used as a reward signal, and the control strategy is trained through interaction with the power system. The stability constraint module is used to introduce stability constraints for the Lyapunov function in reinforcement learning algorithms. The parameter design module is used to perform parameter design of the controller using neural networks; The collaborative optimization module is used to discretize the frequency dynamic equation and propose a multi-node collaborative optimization objective. The trained control strategy is deployed to the actual power system, and the control parameters are adjusted in real time through online optimization technology to adapt to the dynamic changes in the system's operating status.

8. The optimal first-order frequency control system based on reinforcement learning according to claim 7, characterized in that, The dynamic model construction module includes the following units: The inertia and damping characteristic model building unit is used to construct the inertia and damping characteristic model of a traditional generator, as shown in the following equation: Where Δf is the frequency deviation, M is the system's inertial constant, and ΔP m It is the increase in the mechanical power of the generator, ΔP e It is the electrical power increment of the system, and D is the damping coefficient; The inverter dynamic behavior model building unit is used to construct the dynamic behavior model of the inverter's basic resources, as shown in the following equation: Where τ is the time constant of the inverter, and u is the control input signal generated by the frequency controller; The load impact model building unit is used to construct the model of the impact of load changes on system frequency, as shown in the following equation: Where, ΔP L It is the change in load power.

9. The optimal primary frequency control system based on reinforcement learning according to claim 8, characterized in that, The control strategy design module includes the following units: Markov decision building blocks are used to model reinforcement learning problems as Markov decision processes, whose elements are: states s constructed from system frequency deviation and inverter power increments. t =[Δf t ,ΔP e Action a constructed from control input signals t =u; the reward function constructed from frequency deviation and control cost. Where α is a weighting factor used to balance frequency deviation and control power cost; by optimizing the strategy π(a t |s t Maximize cumulative discount rewards: Where γ∈[0,1] is the discount factor; The neural network modeling unit is used to model the temporal coupling characteristics of recurrent neural networks (RNNs) and capture the dynamic time series characteristics; the RNN hidden state update equation is: h t =f(h t-1 ,s t ;θ) Among them, h t Let f be the current hidden state, f be the activation function of the RNN, and θ be the RNN parameters; The control signal output unit is used to generate the control signal u at the output layer of the recurrent neural network (RNN) as follows: u=W0h t +b0 Where W0 and b0 are the weights and biases of the output layer, respectively, and u is the control input signal of the frequency controller.

10. The optimal first-order frequency control system based on reinforcement learning according to claim 9, characterized in that, The stability constraint module operates as follows: Introducing the Lyapunov function V(s) t ), defined as follows: The stability condition for Lyapunov functions is as follows: Combining the inertia and damping characteristic model of a traditional generator, stability constraints are obtained:

Images