Safety reinforcement learning load frequency control method, system, equipment and medium

Abstract

The invention belongs to the field of energy storage system frequency control, and discloses a safety reinforcement learning load frequency control method, system and device and a medium, and the safety reinforcement learning load frequency control method comprises the steps: constructing a frequency control model; acquiring historical data of the power system to construct a labeled data set; constructing a cost prediction network according to the label data set, taking the state and action of the frequency control model as the input of the cost prediction network, and predicting the control cost at the current moment; constructing a deep reinforcement learning controller, and converting an inequality constraint of a target function in the deep reinforcement learning controller based on a Lagrange multiplier; and according to a conversion result, optimizing an actor network of the deep reinforcement learning controller by predicting the control cost at the current moment, and applying the optimized actor network to real-time load frequency control to realize control of the load frequency. Operation cost and frequency modulation performance of the power system are considered, and comprehensive benefits of the power system are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of frequency control of energy storage systems, and in particular to a method, system, device and medium for safe reinforcement learning load frequency control. Background Art

[0002] The frequency of the power system is a key indicator for measuring the quality of electric energy. The imbalance of active power will cause frequency fluctuations. Load frequency control (LFC) is an effective means to deal with the frequency deviation of the power system. It maintains the active power balance between power generation and load by adjusting the power output at the power generation end, thereby maintaining the stability of the system frequency. Energy storage systems have become an important technical means to achieve frequency control in modern power systems due to their fast response, precise control, and ability to achieve two-way regulation. As a power-type and physical energy storage device, the flywheel energy storage system has the advantages of environmental protection, fast response speed, and unlimited charging and discharging times. It has received widespread attention in load frequency control.

[0003] Traditional model-based load frequency control relies on accurate system models, which is often difficult to meet the requirements in complex power system scenarios, limiting its scope of application. In recent years, with the help of the fast computing power of neural networks, load frequency control methods based on reinforcement learning have gradually emerged, which can generate frequency regulation strategies for power systems by training neural networks. However, existing methods usually ignore the frequency regulation characteristics and power constraints of flywheel energy storage systems, and fail to give full play to the auxiliary frequency regulation capabilities of energy storage systems. In addition, when considering the operating costs and frequency regulation performance of the power system at the same time, existing methods often design reward functions by multiplying the values ​​of each objective function by weights, and the determination of weights often depends on simulation experiments, which may affect the accuracy of the optimization process.

[0004] In addition, some load frequency control methods based on safety reinforcement learning require a lot of computing resources when dealing with system constraints, which makes it difficult for the final control strategy to show good results in practical applications. Therefore, it is of great significance to further study the safety reinforcement learning load frequency control method that takes into account the characteristics of flywheel energy storage, power constraints, and the economic and frequency regulation performance of the power system. Summary of the invention

[0005] The embodiments of the present invention provide a safety reinforcement learning load frequency control method, system, device and medium to solve the problem that some load frequency control methods based on safety reinforcement learning in the prior art need to consume a large amount of computing resources when processing system constraints, resulting in the final control strategy being difficult to show good results in practical applications.

[0006] In order to have a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended to be a general review, nor is it intended to identify key / important components or to delineate the scope of protection of these embodiments. Its only purpose is to present some concepts in a simple form as a preface to the detailed description that follows.

[0007] According to a first aspect of an embodiment of the present invention, a method for controlling load frequency of safety reinforcement learning is provided.

[0008] In one embodiment, the safety reinforcement learning load frequency control method includes:

[0009] Combining the frequency regulation characteristics and power constraints of the flywheel energy storage system, a frequency control model is constructed based on the core elements of the constrained Markov decision process;

[0010] Acquire historical data of the power system, and construct a labeled data set based on the cost function of the core element; train a long short-term memory network according to the labeled data set, and construct a cost prediction network based on the trained long short-term memory network, and use the state and action of the frequency control model as input of the cost prediction network to predict the control cost at the current moment;

[0011] An upper limit value of the cost function is set, a deep reinforcement learning controller is constructed, and the inequality constraints of the objective function in the deep reinforcement learning controller are transformed based on Lagrange multipliers; according to the transformation result, the actor network of the deep reinforcement learning controller is optimized using the control cost predicted at the current moment, and the optimized actor network is applied to real-time load frequency control to achieve control of the load frequency.

[0012] In one embodiment, the frequency control model is constructed based on the core elements of the constrained Markov decision process and the frequency modulation characteristics and power constraints of the flywheel energy storage system, including:

[0013] Determine the power generation rate constraint and power increment constraint of the diesel generator;

[0014] Based on the power generation rate constraint and power increment constraint of the diesel generator, combined with the regulation dead zone and power increment constraint of the flywheel energy storage system, the state space, action space, reward function, cost function and objective function of the constrained Markov decision process are defined;

[0015] A frequency control model is constructed based on the state space, action space, reward function, cost function and objective function of the constrained Markov decision process.

[0016] In one embodiment, the frequency control model is expressed as:

[0017]

[0018] Where, Δf is the frequency deviation of the power system; H is the inertia time constant of the power system; ΔP G is the output power deviation of the diesel generator; ΔP E is the output power deviation of the flywheel energy storage system; ΔP W is the wind power deviation; ΔP L is the load power deviation; D is the load damping coefficient of the power system; ΔX G is the speed regulator output signal deviation; T t is the speed regulator constant; Δu G is the diesel generator control signal; T f is the diesel generator time constant; R f is the diesel generator adjustment coefficient; Δu E is the control signal of the flywheel energy storage system; T E is the time constant of the flywheel energy storage system.

[0019] In one embodiment, the acquiring of power system historical data, constructing a labeled data set based on the cost function of the core element; training a long short-term memory network according to the labeled data set, constructing a cost prediction network based on the trained long short-term memory network, using the state and action of the frequency control model as input to the cost prediction network, and predicting the control cost at the current moment includes:

[0020] The labeled data set is constructed using historical data, and the load frequency control is realized through the PID controller. Based on the cost function of the constrained Markov decision process, the state and action of the frequency control model constitute the training data;

[0021] Use the constructed labeled dataset to train a pre-configured long short-term memory network model;

[0022] Using the dynamic response data of the flywheel energy storage system, construct and train the data pairs of the long short-term memory network model;

[0023] Based on the data pairs of the trained LSTM network, a cost prediction network is constructed;

[0024] Through the cost prediction network, the label of the training data is received at each moment, and the current state is updated through the hidden state and memory unit;

[0025] After obtaining the updated hidden state, the output of the cost prediction network is mapped to the predicted cost through a fully connected layer;

[0026] Through supervised learning, a mapping relationship from state, action to cost is established to complete the training of the cost prediction network.

[0027] In one embodiment, mapping the output of the cost prediction network to the predicted cost through a fully connected layer includes:

[0028] Q C (s t ,a t )=h (n) [h (n-1) [...h (1) (h t )]];

[0029] In the formula, Q C (s t ,a t ) is the output of the cost prediction network; h (n) is the activation function of the nth layer; h (n-1) is the activation function of the n-1th layer; h (1) is the activation function of the first layer; (h t ) is the hidden state output by the LSTM network.

[0030] In one embodiment, the setting of the upper limit value of the cost function, constructing a deep reinforcement learning controller, and transforming the inequality constraints of the objective function in the deep reinforcement learning controller based on Lagrange multipliers; according to the transformation result, optimizing the actor network of the deep reinforcement learning controller by using the control cost predicted at the current moment, applying the optimized actor network to real-time load frequency control, and realizing the control of load frequency includes:

[0031] Set the upper limit of the cost function;

[0032] Build a deep reinforcement learning controller consisting of an actor network, a reward critic network, an actor target network, and a reward critic target network;

[0033] Initialize the network parameters of the actor network and the reward critic network, and assign the initialized network parameters of the actor network and the reward critic network to the actor target network and the reward critic network respectively;

[0034] Initialize the interaction experience pool and dual variables of the Lagrange multiplier, obtain some interaction experiences between the actor network and the power system by predicting the control cost at the current moment, and build an interaction experience pool based on some interaction experiences;

[0035] Optimizing the network parameters of the actor network and reward critic network as well as the target actor network and target reward critic network according to the interaction experience pool;

[0036] Repeat the steps of constructing an interactive experience pool and optimizing using the interactive experience pool to a preset number of times, and output an optimized actor network;

[0037] The optimized actor network is applied to real-time load frequency control to achieve load frequency control.

[0038] In one embodiment, optimizing the network parameters of the actor network, the reward critic network, the target actor network, and the target reward critic network according to the interaction experience pool includes:

[0039] Sampling a preset number of interaction experiences from the interaction experience pool;

[0040] Calculate the reward critic target value based on the target actor network and the network parameters of the target reward critic;

[0041] Based on the reward critic target value, the network parameters of the reward critic network are updated by minimizing the loss function;

[0042] Calculate the action gradient and update the network parameters of the actor network using the gradient descent method based on the action gradient;

[0043] Optimize the reward critic network according to the chain rule combined with the control cost predicted at the current moment;

[0044] Calculate the dual variable gradient and update the dual variable by gradient descent based on the dual variable gradient;

[0045] Optimize the target actor network and the target reward critic network.

[0046] In one embodiment, the expression for calculating the dual variable gradient is:

[0047]

[0048] Where, d λ is the dual variable gradient; K is the number of experiences sampled from the interactive experience pool; k is the experience number; Q C For cost prediction network;s k is the state in experience k; μ(s k |θ μ ) is the actor network; is the parameter of the cost prediction network; d is the constraint limit value.

[0049] According to a second aspect of an embodiment of the present invention, a safety reinforcement learning load frequency control system is provided.

[0050] In one embodiment, the safety reinforcement learning load frequency control system includes:

[0051] A frequency control model building module, which is used to combine the frequency regulation characteristics and power constraints of the flywheel energy storage system and build a frequency control model based on the core elements of the constrained Markov decision process;

[0052] A cost prediction network construction module is used to obtain historical data of the power system and construct a labeled data set based on the cost function of the core element; train a long short-term memory network according to the labeled data set, and construct a cost prediction network based on the trained long short-term memory network, and use the state and action of the frequency control model as input of the cost prediction network to predict the control cost at the current moment;

[0053] A deep reinforcement learning controller optimization module is used to set the upper limit value of the cost function, construct a deep reinforcement learning controller, and transform the inequality constraints of the objective function in the deep reinforcement learning controller based on Lagrange multipliers; according to the transformation result, the actor network of the deep reinforcement learning controller is optimized using the control cost predicted at the current moment, and the optimized actor network is applied to real-time load frequency control to achieve control of the load frequency.

[0054] In one embodiment, the frequency control model building module includes: a diesel generator constraint determination module, a constrained Markov decision process element definition module and a control model building module;

[0055] A diesel generator constraint determination module, used to determine the diesel generator power generation rate constraint and power increment constraint;

[0056] Constrained Markov decision process element definition module, which is used to define the state space, action space, reward function, cost function and objective function of the constrained Markov decision process based on the power generation rate constraint and power increment constraint of the diesel generator, combined with the regulation dead zone and power increment constraint of the flywheel energy storage system;

[0057] The control model building module is used to build a frequency control model based on the state space, action space, reward function, cost function and objective function of the constrained Markov decision process.

[0058] In one embodiment, the cost prediction network construction module includes: a labeled data set module, a long short-term memory network model configuration module, a flywheel energy storage system data pair construction module, a prediction network construction module, an output mapping module, a cost prediction module and a supervised learning training module;

[0059] The labeled dataset module is used to construct a labeled dataset using historical data, realize load frequency control through a PID controller, and form training data from the states and actions of the frequency control model based on the cost function of the constrained Markov decision process;

[0060] The LSTM network model configuration module is used to train the pre-configured LSTM network model using the constructed label dataset;

[0061] A flywheel energy storage system data pair construction module, used to construct and train data pairs of a long short-term memory network model using dynamic response data of the flywheel energy storage system;

[0062] The prediction network building module is used to build a cost prediction network based on the data pairs of the trained long short-term memory network;

[0063] The output mapping module is used to receive the label of the training data at each moment through the cost prediction network and update the current state through the hidden state and memory unit;

[0064] The cost prediction module is used to map the output of the cost prediction network to the predicted cost through a fully connected layer after obtaining the updated hidden state;

[0065] The supervised learning training module is used to establish a mapping relationship from state, action to cost through supervised learning, and complete the training of the cost prediction network.

[0066] In one embodiment, the deep reinforcement learning controller optimization module includes: an upper limit value setting module of the cost function, a deep reinforcement learning controller construction module, a network parameter initialization module, an interactive experience pool construction module, a network parameter optimization module, an optimization process iteration module and a real-time load frequency control module;

[0067] The upper limit value setting module of the cost function is used to set the upper limit value of the cost function;

[0068] A deep reinforcement learning controller building module, which is used to build a deep reinforcement learning controller containing an actor network, a reward critic network, an actor target network, and a reward critic target network;

[0069] The network parameter initialization module is used to initialize the network parameters of the actor network and the reward critic network, and assign the initialized network parameters of the actor network and the reward critic network to the actor target network and the reward critic network respectively;

[0070] The interactive experience pool construction module is used to initialize the interactive experience pool and dual variables of the Lagrange multiplier, obtain several interactive experiences between the actor network and the power system by predicting the control cost at the current moment, and construct the interactive experience pool based on several interactive experiences;

[0071] A network parameter optimization module, used to optimize the network parameters of the actor network and reward critic network and the target actor network and target reward critic network according to the interaction experience pool;

[0072] An optimization process iteration module, used to repeatedly execute the steps of building an interactive experience pool and optimizing using the interactive experience pool to a preset number of times, and output an optimized actor network;

[0073] The real-time load frequency control module is used to apply the optimized actor network to real-time load frequency control to achieve load frequency control.

[0074] In one embodiment, the network parameter optimization module includes: an interactive experience acquisition module, a reward critic target value calculation module, a reward critic network network parameter update module, an action gradient and update actor network network parameter calculation module, a reward critic network optimization module, a dual variable update module, and a target actor network and target reward critic network optimization module;

[0075] An interactive experience collection module, used to sample a preset number of interactive experiences from an interactive experience pool;

[0076] A reward critic target value calculation module, used to calculate the reward critic target value based on the target actor network and the network parameters of the target reward critic;

[0077] A network parameter update module of the reward critic network, used to update the network parameters of the reward critic network by minimizing the loss function based on the reward critic target value;

[0078] The action gradient and update network parameter calculation module of the actor network is used to calculate the action gradient and update the network parameters of the actor network based on the action gradient using the gradient descent method;

[0079] The reward critic network optimization module is used to optimize the reward critic network according to the chain rule combined with the control cost predicted at the current moment;

[0080] The dual variable update module is used to calculate the dual variable gradient and update the dual variable by gradient descent method according to the dual variable gradient;

[0081] Target Actor Network and Target Reward Critic Network Optimization Module, used to optimize the target actor network and target reward critic network.

[0082] In one embodiment, the expression for calculating the dual variable gradient is:

[0083]

[0084] Where, d λ is the dual variable gradient; K is the number of experiences sampled from the interactive experience pool; k is the experience number; Q C For cost prediction network;s k is the state in experience k; μ(s k |θ μ ) is the actor network; is the parameter of the cost prediction network; d is the constraint limit value.

[0085] According to a third aspect of an embodiment of the present invention, a computer device is provided.

[0086] In one embodiment, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0087] According to a fourth aspect of embodiments of the present invention, a computer-readable storage medium is provided.

[0088] In one embodiment, the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0089] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:

[0090] The present invention is a safe reinforcement learning load frequency control method taking into account the flywheel energy storage system, which fully considers the frequency regulation characteristics and power constraints of the flywheel energy storage system, and constructs the state space, action space, reward function, cost function and objective function of the constrained Markov decision process; wherein, in order to realize the nonlinear mapping of state-action pairs to costs, based on the long short-term memory network, the state and action of the frequency control model are used as the input of the prediction cost network to predict the control cost at the current moment; this method can capture the complex temporal dependency between the state and action during the operation of the system, and effectively improve the accuracy of cost prediction; finally, the Lagrange multiplier is introduced to transform the inequality constraint problem of the objective function into an unconstrained problem, and the cost prediction neural network is used to improve the efficiency of solving the constrained Markov decision process of Primal Dual-DDPG, realize the update of the actor network, and use the optimized actor network as a real-time load frequency control strategy model; the present invention fully considers the auxiliary role of the flywheel energy storage system in the load frequency control process and its frequency regulation characteristics. Compared with the general Markov process, the present invention explicitly considers the constraints, while taking into account the operation cost and frequency regulation performance of the power system, and ensuring the comprehensive benefits of the power system.

[0091] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0093] Figure 1 is a flow chart of a method for controlling load frequency of safety reinforcement learning according to an exemplary embodiment;

[0094] Figure 2 is a principle block diagram of a safety reinforcement learning load frequency control system according to an exemplary embodiment;

[0095] Figure 3 is a structural schematic diagram of a computer device according to an exemplary embodiment;

[0096] Figure 4 is a principle block diagram of a frequency control model in a safety reinforcement learning load frequency control method according to an exemplary embodiment;

[0097] Figure 5 It is a schematic diagram of updating a reward critic network and an actor network in a load frequency control method for secure reinforcement learning according to an exemplary embodiment. DETAILED DESCRIPTION

[0098] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.

[0099] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0100] As used herein, the term "plurality" means two or more than two, unless otherwise specified.

[0101] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.

[0102] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.

[0103] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0104] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.

[0105] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0106] Figure 1An embodiment of a safety reinforcement learning load frequency control method of the present invention is shown.

[0107] In this optional embodiment, the safety reinforcement learning load frequency control method includes:

[0108] Step S101, combining the frequency regulation characteristics and power constraints of the flywheel energy storage system and building a frequency control model based on the core elements of the constrained Markov decision process;

[0109] Step S103, acquiring historical data of the power system, constructing a labeled data set based on the cost function of the core element; training a long short-term memory network according to the labeled data set, constructing a cost prediction network based on the trained long short-term memory network, taking the state and action of the frequency control model as input of the cost prediction network, and predicting the control cost at the current moment;

[0110] Step S105, setting the upper limit value of the cost function, constructing a deep reinforcement learning controller, and transforming the inequality constraints of the objective function in the deep reinforcement learning controller based on Lagrange multipliers; according to the transformation result, optimizing the actor network of the deep reinforcement learning controller using the control cost predicted at the current moment, applying the optimized actor network to real-time load frequency control to achieve control of the load frequency.

[0111] In this optional embodiment, the frequency control model is constructed based on the core elements of the constrained Markov decision process and the frequency modulation characteristics and power constraints of the flywheel energy storage system, including:

[0112] Determine the power generation rate constraint and power increment constraint of the diesel generator;

[0113] Based on the power generation rate constraint and power increment constraint of the diesel generator, combined with the regulation dead zone and power increment constraint of the flywheel energy storage system, the state space, action space, reward function, cost function and objective function of the constrained Markov decision process are defined;

[0114] In addition, it should be noted that the state space of the constrained Markov decision process is defined as Define the action space of the constrained Markov decision process as The reward function of the constrained Markov decision process is defined as:

[0115]

[0116] Define the cost function c of the constrained Markov decision process t for:

[0117]

[0118] Define the objective function π* of the constrained Markov decision process as

[0119]

[0120] Where: s t is the state of the agent at time t; Δf t is the frequency deviation of the power system at time t; Δf t 2 is the square of the frequency deviation of the power system at time t; c t is the cost function; To consider the control signal difference and reduce control loss; It is the diesel generator power generation rate exceeding limit function; is the power generation rate exceeding limit function of the flywheel energy storage system; is the excess function of the power generation increment change; is the diesel generator control signal at time t; is the diesel generator control signal at time t-1; is the control signal of the flywheel energy storage system at time t; is the control signal of the flywheel energy storage system at time t-1; is the output power of the diesel generator at time t; ΔP Gmax The upper limit of the diesel generator power increment; ΔP Gmin is the lower limit of the diesel generator power increment; ΔP Emin is the lower limit of the power increment of the flywheel energy storage system; is the output power of the flywheel energy storage system at time t; ΔP Emax The upper limit of the power increment of the flywheel energy storage system; is the incremental change of power generation; δ is the threshold of the power ramp rate of the diesel generator; π* is the optimal control trajectory; R(π) is the future discount reward; is the expected function; γ is the discount factor, γ≤1; r i is the reward obtained by the agent; a t is the action of the agent at time t; C(π) is the future discounted cost, c i is the cost function; d is the constraint limit value; α is the positive reward value; and are the penalty coefficients corresponding to different frequency deviation intervals; β is the penalty value. When the frequency deviation exceeds 0.2Hz, the agent will be severely punished. To consider the control signal difference and reduce control loss; and They are respectively the exceeding limit functions of the generation rate and the generation power increment of the diesel generator; It is the excess function of the power generation rate of the flywheel energy storage system.

[0121] A frequency control model is constructed based on the state space, action space, reward function, cost function and objective function of the constrained Markov decision process.

[0122] In this alternative embodiment, if Figure 4 As shown, the expression of the frequency control model is:

[0123]

[0124] Where, Δf is the frequency deviation of the power system; H is the inertia time constant of the power system; ΔP G is the output power deviation of the diesel generator; ΔP E is the output power deviation of the flywheel energy storage system; ΔP W is the wind power deviation; ΔP L is the load power deviation; D is the load damping coefficient of the power system; ΔX G is the speed regulator output signal deviation; T t is the speed regulator constant; Δu G is the diesel generator control signal; T f is the diesel generator time constant; R f is the diesel generator adjustment coefficient; Δu E is the control signal of the flywheel energy storage system; T E is the time constant of the flywheel energy storage system.

[0125] In this optional embodiment, the acquiring of power system historical data, constructing a labeled data set based on the cost function of the core element; training a long short-term memory network according to the labeled data set, constructing a cost prediction network based on the trained long short-term memory network, using the state and action of the frequency control model as input to the cost prediction network, and predicting the control cost at the current moment includes:

[0126] The labeled data set is constructed using historical data, and the load frequency control is realized through the PID controller. Based on the cost function of the constrained Markov decision process, the state and action of the frequency control model constitute the training data;

[0127] Use the constructed labeled dataset to train a pre-configured long short-term memory network model;

[0128] Using the dynamic response data of the flywheel energy storage system, construct and train the data pairs of the long short-term memory network model;

[0129] Based on the data pairs of the trained LSTM network, a cost prediction network is constructed;

[0130] Through the cost prediction network, the label of the training data is received at each moment, and the current state is updated through the hidden state and memory unit;

[0131] After obtaining the updated hidden state, the output of the cost prediction network is mapped to the predicted cost through a fully connected layer;

[0132] Through supervised learning, a mapping relationship from state, action to cost is established to complete the training of the cost prediction network.

[0133] In addition, it should be noted that by collecting real-time information of the power system, a labeled data set (x t ,y t ), data set x t Contains the input and output of the PID controller, namely At the same time, in the actual control process, t The cost calculated according to the cost function of the constrained Markov decision process is also used as the label of the training data.

[0134] Based on the long short-term memory network architecture, the cost prediction network is constructed as shown in the state update formula of the cost prediction network; the state update formula of the cost prediction network is as follows:

[0135]

[0136] After getting the updated hidden state h t Afterwards, the output of the cost prediction network is mapped to the predicted cost through a fully connected layer:

[0137] Q C (s t ,a t )=h (n) [h (n-1) [...h (1) (h t )]];

[0138] The cost prediction network is trained using supervised learning and is expressed as follows:

[0139]

[0140] In the formula, f t is the output of the forget gate; σ is the activation function; W f is the weight matrix of the forget gate; h t-1 is the hidden state at time t-1; x t is the input of the long short-term memory network; b f is the bias term of the forget gate; i t is the output of the input gate; W i is the weight matrix of the input gate; b i is the bias term of the input gate; W is the transition memory unit;c is the weight matrix of the memory unit; b c is the bias term of the memory unit; c t is the memory unit at time t; c t-1 is the memory unit at time t-1; o t is the output of the output gate; W o is the weight matrix of the output gate; b o is the bias term of the output gate; h t is the hidden state at time t; Q C (s t ,a t ) is the cost prediction function; h (n) is the activation function of the nth layer; h (n-1) is the activation function of the n-1th layer; h (1) is the activation function of the first layer; is the minimum value; y is the sample label for supervised learning; ||yQ C (s t ,a t )|| 2 is the loss value of the supervised learning process.

[0141] In this optional embodiment, mapping the output of the cost prediction network to the predicted cost through a fully connected layer includes:

[0142] Q C (s t ,a t )=h (n) [h (n-1) [...h (1) (h t )]];

[0143] In the formula, Q C (s t ,a t ) is the output of the cost prediction network; h (n) is the activation function of the nth layer; h (n-1) is the activation function of the n-1th layer; h (1) is the activation function of the first layer; (h t ) is the hidden state output by the LSTM network.

[0144] In this alternative embodiment, if Figure 5As shown, the upper limit value of the cost function is set, a deep reinforcement learning controller is constructed, and the inequality constraints of the objective function in the deep reinforcement learning controller are transformed based on the Lagrange multiplier; according to the transformation result, the actor network of the deep reinforcement learning controller is optimized by using the control cost predicted at the current moment, and the optimized actor network is applied to real-time load frequency control to realize the control of load frequency, which includes:

[0145] Set the upper limit of the cost function;

[0146] Build a deep reinforcement learning controller consisting of an actor network, a reward critic network, an actor target network, and a reward critic target network;

[0147] Initialize the network parameters of the actor network and the reward critic network, and assign the initialized network parameters of the actor network and the reward critic network to the actor target network and the reward critic network respectively;

[0148] Initialize the interaction experience pool and dual variables of the Lagrange multiplier, obtain some interaction experiences between the actor network and the power system by predicting the control cost at the current moment, and build an interaction experience pool based on some interaction experiences;

[0149] Optimizing the network parameters of the actor network and reward critic network as well as the target actor network and target reward critic network according to the interaction experience pool;

[0150] Repeat the steps of constructing an interactive experience pool and optimizing using the interactive experience pool to a preset number of times, and output an optimized actor network;

[0151] The optimized actor network is applied to real-time load frequency control to achieve load frequency control.

[0152] In addition, it should be noted that the Lagrange multiplier λ is introduced to transform the inequality constraint problem of the objective function into an unconstrained problem. Based on the original dual optimization method, the network parameters and dual variables are updated in turn in the iteration. The specific steps are as follows:

[0153] Construct a deep reinforcement learning controller, including the actor network μ(a|θ μ ), reward critic network Actor-goal network μ'(a|θ μ' ) and reward critic target network

[0154] Initialize the interaction experience pool and the dual variable λ, and obtain some interaction experiences (s t ,a t ,r t ,ct ,s t+1 ), and obtain the interactive experience pool;

[0155] Optimizing network parameters of an actor network and a reward critic network, as well as a target actor network and a target reward critic network according to the interaction experience pool;

[0156] Repeat the steps of obtaining the interactive experience pool and optimizing the network parameters of the actor network and the reward critic network, as well as the target actor network and the target reward critic network, until a preset number of times, and output the optimized actor network.

[0157] In this optional embodiment, the optimizing the network parameters of the actor network and the reward critic network and the target actor network and the target reward critic network according to the interaction experience pool includes:

[0158] Sampling a preset number of interaction experiences from the interaction experience pool;

[0159] Calculate the reward critic target value based on the target actor network and the network parameters of the target reward critic;

[0160] Based on the reward critic target value, the network parameters of the reward critic network are updated by minimizing the loss function;

[0161] Calculate the action gradient and update the network parameters of the actor network using the gradient descent method based on the action gradient;

[0162] Optimize the reward critic network according to the chain rule combined with the control cost predicted at the current moment;

[0163] Calculate the dual variable gradient and update the dual variable by gradient descent based on the dual variable gradient;

[0164] Optimize the target actor network and the target reward critic network.

[0165] In this optional embodiment, the expression for calculating the dual variable gradient is:

[0166]

[0167] Where, d λ is the dual variable gradient; K is the number of experiences sampled from the interactive experience pool; k is the experience number; Q C For cost prediction network;s k is the state in experience k; μ(s k |θ μ ) is the actor network; is the parameter of the cost prediction network; d is the constraint limit value. In addition, it should be noted that K interactive experience points are sampled from the interactive experience pool.

[0168] The reward critic target value y is calculated by k The expression is:

[0169]

[0170] Based on the target value y k , by minimizing the loss function L R The expression for updating the network parameters of the reward critic network is:

[0171]

[0172] The action gradient is calculated by the following formula, and according to the action gradient d a The expression for updating the network parameters of the actor network by gradient descent is:

[0173]

[0174] The expression obtained according to the chain rule is:

[0175]

[0176] The dual variable gradient is calculated by the following formula, and according to the dual variable gradient d λ The expression for updating the dual variable by gradient descent is:

[0177]

[0178] The expressions for optimizing the target actor network and the target reward critic network are:

[0179]

[0180] Where: τ is the preset soft update parameter; K is the number of experiences sampled from the interactive experience pool; k is the experience number; s k is the state in experience k; a k is the action in experience k; r k is the reward in experience k; c k is the cost in experience k; s k+1 is the state in experience k+1; y k is the target value of the reward critic; γ is the discount factor; Q' R is the target reward critic network; μ' is the target actor network policy; θ μ' are the parameters of the target actor network; is the parameter of the target reward critic network; L R is the reward critic network loss function; Q Ris the reward critic network; μ is the actor network strategy; θ μ are the actor network parameters; is the reward critic network parameter; d a is the action gradient; Q C is the cost prediction network; μ(s|θ μ ) is the output of the actor network; d is the constraint limit value; where, and They are obtained by the back-propagation algorithm respectively from the reward critic network, the cost prediction network and the actor network.

[0181] The present invention is a safe reinforcement learning load frequency control method taking into account the flywheel energy storage system, which fully considers the frequency modulation characteristics and power constraints of the flywheel energy storage system, and constructs the state space, action space, reward function, cost function and objective function of the constrained Markov decision process; wherein, in order to realize the nonlinear mapping of state-action pairs to costs, based on the long short-term memory network, the state and action of the frequency control model are used as the input of the prediction cost network to predict the control cost at the current moment; this method can capture the complex temporal dependency between the state and action during the operation of the system, and effectively improve the accuracy of cost prediction; finally, the Lagrange multiplier is introduced to transform the inequality constraint problem of the objective function into an unconstrained problem, and the cost prediction neural network is used to improve the primal dual deep deterministic policy gradient (Primal The efficiency of solving the constrained Markov decision process of Dual-DDPG) is improved, the updating of the actor network is realized, and the optimized actor network is used as a real-time load frequency control strategy model; the present invention fully considers the auxiliary role of the flywheel energy storage system in the load frequency control process and its frequency regulation characteristics. Compared with the general Markov process, the present invention explicitly considers the constraints, while taking into account the operation cost and frequency regulation performance of the power system, to ensure the comprehensive benefits of the power system.

[0182] Figure 2 An embodiment of the safety reinforcement learning load frequency control system of the present invention is shown.

[0183] In this optional embodiment, the safety reinforcement learning load frequency control system includes:

[0184] A frequency control model building module 201 is used to combine the frequency regulation characteristics and power constraints of the flywheel energy storage system and build a frequency control model based on the core elements of the constrained Markov decision process;

[0185] The cost prediction network construction module 203 is used to obtain historical data of the power system, and construct a labeled data set based on the cost function of the core element; train a long short-term memory network according to the labeled data set, and construct a cost prediction network based on the trained long short-term memory network, and use the state and action of the frequency control model as the input of the cost prediction network to predict the control cost at the current moment;

[0186] The deep reinforcement learning controller optimization module 205 is used to set the upper limit value of the cost function, construct a deep reinforcement learning controller, and transform the inequality constraints of the objective function in the deep reinforcement learning controller based on Lagrange multipliers; according to the transformation result, the actor network of the deep reinforcement learning controller is optimized using the control cost predicted at the current moment, and the optimized actor network is applied to real-time load frequency control to achieve control of the load frequency.

[0187] In this optional embodiment, the frequency control model building module includes: a diesel generator constraint determination module, a constrained Markov decision process element definition module and a control model building module;

[0188] A diesel generator constraint determination module, used to determine the diesel generator power generation rate constraint and power increment constraint;

[0189] Constrained Markov decision process element definition module, which is used to define the state space, action space, reward function, cost function and objective function of the constrained Markov decision process based on the power generation rate constraint and power increment constraint of the diesel generator, combined with the regulation dead zone and power increment constraint of the flywheel energy storage system;

[0190] The control model building module is used to build a frequency control model based on the state space, action space, reward function, cost function and objective function of the constrained Markov decision process.

[0191] In this optional embodiment, the cost prediction network construction module includes: a labeled data set module, a long short-term memory network model configuration module, a flywheel energy storage system data pair construction module, a prediction network construction module, an output mapping module, a cost prediction module and a supervised learning training module;

[0192] The labeled dataset module is used to construct a labeled dataset using historical data, realize load frequency control through a PID controller, and form training data from the states and actions of the frequency control model based on the cost function of the constrained Markov decision process;

[0193] The LSTM network model configuration module is used to train the pre-configured LSTM network model using the constructed label dataset;

[0194] A flywheel energy storage system data pair construction module, used to construct and train data pairs of a long short-term memory network model using dynamic response data of the flywheel energy storage system;

[0195] The prediction network building module is used to build a cost prediction network based on the data pairs of the trained long short-term memory network;

[0196] The output mapping module is used to receive the label of the training data at each moment through the cost prediction network and update the current state through the hidden state and memory unit;

[0197] The cost prediction module is used to map the output of the cost prediction network to the predicted cost through a fully connected layer after obtaining the updated hidden state;

[0198] The supervised learning training module is used to establish a mapping relationship from state, action to cost through supervised learning, and complete the training of the cost prediction network.

[0199] In this optional embodiment, the deep reinforcement learning controller optimization module includes: an upper limit value setting module of the cost function, a deep reinforcement learning controller construction module, a network parameter initialization module, an interactive experience pool construction module, a network parameter optimization module, an optimization process iteration module and a real-time load frequency control module;

[0200] The upper limit value setting module of the cost function is used to set the upper limit value of the cost function;

[0201] A deep reinforcement learning controller building module, which is used to build a deep reinforcement learning controller containing an actor network, a reward critic network, an actor target network, and a reward critic target network;

[0202] The network parameter initialization module is used to initialize the network parameters of the actor network and the reward critic network, and assign the initialized network parameters of the actor network and the reward critic network to the actor target network and the reward critic network respectively;

[0203] The interactive experience pool construction module is used to initialize the interactive experience pool and dual variables of the Lagrange multiplier, obtain several interactive experiences between the actor network and the power system by predicting the control cost at the current moment, and construct the interactive experience pool based on several interactive experiences;

[0204] A network parameter optimization module, used to optimize the network parameters of the actor network and reward critic network and the target actor network and target reward critic network according to the interaction experience pool;

[0205] An optimization process iteration module, used to repeatedly execute the steps of building an interactive experience pool and optimizing using the interactive experience pool to a preset number of times, and output an optimized actor network;

[0206] The real-time load frequency control module is used to apply the optimized actor network to real-time load frequency control to achieve load frequency control.

[0207] In this optional embodiment, the network parameter optimization module includes: an interactive experience acquisition module, a reward critic target value calculation module, a reward critic network network parameter update module, an action gradient and update actor network network parameter calculation module, a reward critic network optimization module, a dual variable update module, and a target actor network and target reward critic network optimization module;

[0208] An interactive experience collection module, used to sample a preset number of interactive experiences from an interactive experience pool;

[0209] A reward critic target value calculation module, used to calculate the reward critic target value based on the target actor network and the network parameters of the target reward critic;

[0210] A network parameter update module of the reward critic network, used to update the network parameters of the reward critic network by minimizing the loss function based on the reward critic target value;

[0211] The action gradient and update network parameter calculation module of the actor network is used to calculate the action gradient and update the network parameters of the actor network based on the action gradient using the gradient descent method;

[0212] The reward critic network optimization module is used to optimize the reward critic network according to the chain rule combined with the control cost predicted at the current moment;

[0213] The dual variable update module is used to calculate the dual variable gradient and update the dual variable by gradient descent method according to the dual variable gradient;

[0214] Target Actor Network and Target Reward Critic Network Optimization Module, used to optimize the target actor network and target reward critic network.

[0215] In this optional embodiment, the expression for calculating the dual variable gradient is:

[0216]

[0217] Where, d λ is the dual variable gradient; K is the number of experiences sampled from the interactive experience pool; k is the experience number; Q C For cost prediction network;s k is the state in experience k; μ(s k |θ μ ) is the actor network; is the parameter of the cost prediction network; d is the constraint limit value.

[0218] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.

[0219] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0220] In addition, the present invention also provides a computer device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.

[0221] In addition, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.

[0222] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0223] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A safety reinforcement learning load frequency control method, characterized in that: The safety reinforcement learning load frequency control method includes: Combining the frequency regulation characteristics and power constraints of the flywheel energy storage system, a frequency control model is constructed based on the core elements of the constrained Markov decision process; Acquire historical data of the power system, and construct a labeled data set based on the cost function of the core element; train a long short-term memory network according to the labeled data set, and construct a cost prediction network based on the trained long short-term memory network, and use the state and action of the frequency control model as input of the cost prediction network to predict the control cost at the current moment; An upper limit value of the cost function is set, a deep reinforcement learning controller is constructed, and the inequality constraints of the objective function in the deep reinforcement learning controller are transformed based on Lagrange multipliers; according to the transformation result, the actor network of the deep reinforcement learning controller is optimized using the control cost predicted at the current moment, and the optimized actor network is applied to real-time load frequency control to achieve control of the load frequency.

2. A safety reinforcement learning load frequency control method according to claim 1, characterized in that: The frequency control model constructed by combining the frequency regulation characteristics and power constraints of the flywheel energy storage system and based on the core elements of the constrained Markov decision process includes: Determine the power generation rate constraint and power increment constraint of the diesel generator; Based on the power generation rate constraint and power increment constraint of the diesel generator, combined with the regulation dead zone and power increment constraint of the flywheel energy storage system, the state space, action space, reward function, cost function and objective function of the constrained Markov decision process are defined; A frequency control model is constructed based on the state space, action space, reward function, cost function and objective function of the constrained Markov decision process.

3. A safety reinforcement learning load frequency control method according to claim 2, characterized in that: The expression of the frequency control model is: Where, Δf is the frequency deviation of the power system; H is the inertia time constant of the power system; ΔP G is the output power deviation of the diesel generator; ΔP E is the output power deviation of the flywheel energy storage system; ΔP W is the wind power deviation; ΔP L is the load power deviation; D is the load damping coefficient of the power system; ΔX G is the speed regulator output signal deviation; T t is the speed regulator constant; Δu G is the diesel generator control signal; T f is the diesel generator time constant; R f is the diesel generator adjustment coefficient; Δu E is the control signal of the flywheel energy storage system; T E is the time constant of the flywheel energy storage system.

4. A safety reinforcement learning load frequency control method according to claim 1, characterized in that: The method of acquiring historical data of the power system and constructing a labeled data set based on the cost function of the core element; training a long short-term memory network according to the labeled data set, constructing a cost prediction network based on the trained long short-term memory network, using the state and action of the frequency control model as input of the cost prediction network, and predicting the control cost at the current moment includes: The labeled data set is constructed using historical data, and the load frequency control is realized through the PID controller. Based on the cost function of the constrained Markov decision process, the state and action of the frequency control model constitute the training data; Use the constructed labeled dataset to train a pre-configured long short-term memory network model; Using the dynamic response data of the flywheel energy storage system, construct and train the data pairs of the long short-term memory network model; Based on the data pairs of the trained LSTM network, a cost prediction network is constructed; Through the cost prediction network, the label of the training data is received at each moment, and the current state is updated through the hidden state and memory unit; After obtaining the updated hidden state, the output of the cost prediction network is mapped to the predicted cost through a fully connected layer; Through supervised learning, a mapping relationship from state, action to cost is established to complete the training of the cost prediction network.

5. A safety reinforcement learning load frequency control method according to claim 4, characterized in that: Mapping the output of the cost prediction network to the predicted cost through a fully connected layer includes: Q C (s t ,a t )=h (n) [h (n-1) [...h (1) (h t )]]; In the formula, Q C (s t ,a t ) is the output of the cost prediction network; h (n) is the activation function of the nth layer; h (n-1) is the activation function of the n-1th layer; h (1) is the activation function of the first layer; (h t ) is the hidden state output by the LSTM network.

6. A safety reinforcement learning load frequency control method according to claim 1, characterized in that: The method further comprises setting an upper limit value of the cost function, constructing a deep reinforcement learning controller, and transforming the inequality constraints of the objective function in the deep reinforcement learning controller based on Lagrange multipliers; According to the conversion result, the actor network of the deep reinforcement learning controller is optimized by using the control cost predicted at the current moment, and the optimized actor network is applied to real-time load frequency control to realize the control of load frequency, including: Set the upper limit of the cost function; Build a deep reinforcement learning controller consisting of an actor network, a reward critic network, an actor target network, and a reward critic target network; Initialize the network parameters of the actor network and the reward critic network, and assign the initialized network parameters of the actor network and the reward critic network to the actor target network and the reward critic network respectively; Initialize the interaction experience pool and dual variables of the Lagrange multiplier, obtain some interaction experiences between the actor network and the power system by predicting the control cost at the current moment, and build an interaction experience pool based on some interaction experiences; Optimizing the network parameters of the actor network and reward critic network as well as the target actor network and target reward critic network according to the interaction experience pool; Repeat the steps of constructing an interactive experience pool and optimizing using the interactive experience pool to a preset number of times, and output an optimized actor network; The optimized actor network is applied to real-time load frequency control to achieve load frequency control.

7. A safety reinforcement learning load frequency control method according to claim 6, characterized in that: The method of optimizing the network parameters of the actor network, the reward critic network, the target actor network, and the target reward critic network according to the interactive experience pool includes: Sampling a preset number of interaction experiences from the interaction experience pool; Calculate the reward critic target value based on the target actor network and the network parameters of the target reward critic; Based on the reward critic target value, the network parameters of the reward critic network are updated by minimizing the loss function; Calculate the action gradient and update the network parameters of the actor network using the gradient descent method based on the action gradient; Optimize the reward critic network according to the chain rule combined with the control cost predicted at the current moment; Calculate the dual variable gradient and update the dual variable by gradient descent based on the dual variable gradient; Optimize the target actor network and the target reward critic network.

8. A safety reinforcement learning load frequency control method according to claim 7, characterized in that: The expression for calculating the dual variable gradient is: Where, d λ is the dual variable gradient; K is the number of experiences sampled from the interactive experience pool; k is the experience number; Q C predicting networks for costs; s k is the state in experience k; μ(s k |θ μ ) is the actor network; Predict the parameters of the network for the cost; d is the constraint limit value.

9. A safety reinforcement learning load frequency control system, characterized in that: include: A frequency control model building module, which is used to combine the frequency regulation characteristics and power constraints of the flywheel energy storage system and build a frequency control model based on the core elements of the constrained Markov decision process; A cost prediction network construction module is used to obtain historical data of the power system and construct a labeled data set based on the cost function of the core element; train a long short-term memory network according to the labeled data set, and construct a cost prediction network based on the trained long short-term memory network, and use the state and action of the frequency control model as input of the cost prediction network to predict the control cost at the current moment; A deep reinforcement learning controller optimization module, used to set an upper limit value of the cost function, construct a deep reinforcement learning controller, and transform the inequality constraints of the objective function in the deep reinforcement learning controller based on Lagrange multipliers; According to the conversion result, the control cost predicted at the current moment is utilized to optimize the actor network of the deep reinforcement learning controller, and the optimized actor network is applied to real-time load frequency control to realize the control of the load frequency.

10. A safety reinforcement learning load frequency control system according to claim 9, characterized in that: The frequency control model building module includes: a diesel generator constraint condition determination module, a constrained Markov decision process element definition module and a control model building module; A diesel generator constraint determination module, used to determine the diesel generator power generation rate constraint and power increment constraint; Constrained Markov decision process element definition module, which is used to define the state space, action space, reward function, cost function and objective function of the constrained Markov decision process based on the power generation rate constraint and power increment constraint of the diesel generator, combined with the regulation dead zone and power increment constraint of the flywheel energy storage system; The control model building module is used to build a frequency control model based on the state space, action space, reward function, cost function and objective function of the constrained Markov decision process.

11. A safety reinforcement learning load frequency control system according to claim 10, characterized in that: The cost prediction network construction module includes: a labeled data set module, a long short-term memory network model configuration module, a flywheel energy storage system data pair construction module, a prediction network construction module, an output mapping module, a cost prediction module and a supervised learning training module; The labeled dataset module is used to construct a labeled dataset using historical data, realize load frequency control through a PID controller, and form training data from the states and actions of the frequency control model based on the cost function of the constrained Markov decision process; The LSTM network model configuration module is used to train the pre-configured LSTM network model using the constructed label dataset; A flywheel energy storage system data pair construction module, used to construct and train data pairs of a long short-term memory network model using dynamic response data of the flywheel energy storage system; The prediction network building module is used to build a cost prediction network based on the data pairs of the trained long short-term memory network; The output mapping module is used to receive the label of the training data at each moment through the cost prediction network and update the current state through the hidden state and memory unit; The cost prediction module is used to map the output of the cost prediction network to the predicted cost through a fully connected layer after obtaining the updated hidden state; The supervised learning training module is used to establish a mapping relationship from state, action to cost through supervised learning, and complete the training of the cost prediction network.

12. A safety reinforcement learning load frequency control system according to claim 11, characterized in that: The deep reinforcement learning controller optimization module includes: an upper limit value setting module of the cost function, a deep reinforcement learning controller construction module, a network parameter initialization module, an interactive experience pool construction module, a network parameter optimization module, an optimization process iteration module and a real-time load frequency control module; The upper limit value setting module of the cost function is used to set the upper limit value of the cost function; A deep reinforcement learning controller building module, which is used to build a deep reinforcement learning controller containing an actor network, a reward critic network, an actor target network, and a reward critic target network; The network parameter initialization module is used to initialize the network parameters of the actor network and the reward critic network, and assign the initialized network parameters of the actor network and the reward critic network to the actor target network and the reward critic network respectively; The interactive experience pool construction module is used to initialize the interactive experience pool and dual variables of the Lagrange multiplier, obtain several interactive experiences between the actor network and the power system by predicting the control cost at the current moment, and construct the interactive experience pool based on several interactive experiences; A network parameter optimization module, used to optimize the network parameters of the actor network and reward critic network and the target actor network and target reward critic network according to the interaction experience pool; An optimization process iteration module, used to repeatedly execute the steps of building an interactive experience pool and optimizing using the interactive experience pool to a preset number of times, and output an optimized actor network; The real-time load frequency control module is used to apply the optimized actor network to real-time load frequency control to achieve load frequency control.

13. A safety reinforcement learning load frequency control system according to claim 12, characterized in that: The network parameter optimization module includes: an interactive experience acquisition module, a reward critic target value calculation module, a reward critic network network parameter update module, an action gradient and update actor network network parameter calculation module, a reward critic network optimization module, a dual variable update module, and a target actor network and target reward critic network optimization module; An interactive experience collection module, used to sample a preset number of interactive experiences from an interactive experience pool; A reward critic target value calculation module, used to calculate the reward critic target value based on the target actor network and the network parameters of the target reward critic; A network parameter update module of the reward critic network, used to update the network parameters of the reward critic network by minimizing the loss function based on the reward critic target value; The action gradient and update network parameter calculation module of the actor network is used to calculate the action gradient and update the network parameters of the actor network based on the action gradient using the gradient descent method; The reward critic network optimization module is used to optimize the reward critic network according to the chain rule combined with the control cost predicted at the current moment; The dual variable update module is used to calculate the dual variable gradient and update the dual variable by gradient descent method according to the dual variable gradient; Target Actor Network and Target Reward Critic Network Optimization Module, used to optimize the target actor network and target reward critic network.

14. A safety reinforcement learning load frequency control system according to claim 13, characterized in that: The expression for calculating the dual variable gradient is: Where, d λ is the dual variable gradient; K is the number of experiences sampled from the interactive experience pool; k is the experience number; Q C predicting networks for costs; s k is the state in experience k; μ(s k |θ μ ) is the actor network; Predict the parameters of the network for the cost; d is the constraint limit value.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

16. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images