Non-linear power system model-free load frequency control method based on deep reinforcement learning

Abstract

The invention relates to the field of power system frequency control, in particular to a non-linear power system model-free load frequency control method based on deep reinforcement learning. Constructing a dual-network architecture, and optimizing actuator network parameters according to the dual-network architecture; initializing the actuator network based on pre-trained PID network parameters, generating exploration noise through random disturbance to perform control space exploration, and iteratively optimizing the actuator network parameters to convergence based on an experience playback mechanism to obtain a trained actuator network; and generating a frequency control instruction. According to the method, the deep network gradient anomaly problem is solved through symmetric difference estimation, the gradient disappearance or explosion problem caused by a complex network structure in a traditional gradient descent-based method is avoided, and the stability and reliability of intelligent agent parameter updating are ensured; and finally, an empirical playback mechanism is adopted to iteratively optimize actuator network parameters, and model-free frequency control of the power system under nonlinear characteristics of a speed regulator dead zone, power generation rate constraint and the like is realized.

Description

Technical Field

[0001] The present invention relates to the field of power system frequency control, and in particular to a model-free load frequency control method for nonlinear power systems based on deep reinforcement learning. Background Art

[0002] As a core indicator of power quality, the stability of power system frequency is directly related to the safe and reliable operation of the entire power grid. Under ideal operating conditions, power generation and load demand should be balanced in real time, with the system frequency maintained at the rated value of 50Hz. However, in actual operation, due to random load fluctuations, intermittent renewable energy output, and various fault disturbances, transient imbalances between power generation and load are inevitable, resulting in frequency deviations. If these frequency deviations exceed the allowable range, they can at best affect the normal operation of electrical equipment, or at worst cause system frequency collapse, triggering widespread power outages. Therefore, load frequency control (LFC) has always been one of the most critical aspects of automatic power generation control in power systems. Its main task is to quickly eliminate frequency deviations and maintain the system frequency within the allowable range by adjusting generator output in real time.

[0003] Traditional LFC technology has been developed primarily based on classical and modern control theories. Among them, the proportional-integral-derivative controller (PID) has been widely used due to its simple structure and high reliability. With the advancement of control theory, advanced control methods such as the linear quadratic regulator (LQRC) and model predictive control (MPC) have also been introduced to the LFC field. While these methods can achieve good control results under specific conditions, they all face a common limitation: they rely on a precise mathematical model of the power system. Specifically, the design of these control algorithms typically requires prior knowledge of key parameters such as the generator transfer function model, system inertia constant, and load frequency characteristic coefficient. However, in actual power systems, these parameters are often difficult to accurately obtain and can change with varying operating conditions. In particular, with a high penetration of renewable energy, the random fluctuations of intermittent energy sources such as wind power and photovoltaics result in significant time-varying system inertia and damping characteristics, further complicating the application of traditional LFC methods.

[0004] The rapid development of artificial intelligence (AI) technology in recent years has provided new approaches to addressing these challenges. Deep reinforcement learning (DRL), in particular, demonstrates unique advantages in solving complex system control problems due to its unique characteristics of trial-and-error learning and environmental interaction. However, existing DRL-based LFC research still faces challenges. First, most studies assume a linear power system and use linearized models as training environments, which is clearly inconsistent with reality. Second, existing methods often rely on partially known system dynamic equations to design reward functions or state spaces, which somewhat undermines DRL's "model-free" advantages. Furthermore, in terms of DRL algorithm implementation, traditional critic network-based approaches are prone to vanishing or exploding gradient problems, resulting in unstable training and difficulty converging to the optimal policy. Summary of the Invention

[0005] In response to the problems mentioned in the prior art, the present invention proposes a model-free load frequency control method for nonlinear power systems based on deep reinforcement learning to overcome the problems existing in the background technology.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions: The present invention proposes a model-free load frequency control method for nonlinear power systems based on deep reinforcement learning, comprising the following steps: Construct a dual-network architecture, including an actuator network and a simulator network; the actuator network receives a state vector from the power system to output a control action; the simulator network generates a predicted frequency deviation value for the next moment based on the state vector and the control action, and constructs a loss function to train the simulator network; S2. Optimizing actuator network parameters based on the dual-network architecture, including constructing an action-value function, calculating a policy gradient using a zero-order optimization algorithm based on the action-value function, and updating actuator network parameters based on the policy gradient; S3. Initializing the actuator network based on pre-trained PID network parameters, generating exploration noise through random perturbations to explore the control space, and iteratively optimizing the actuator network parameters based on the experience replay mechanism until convergence, thereby obtaining a trained actuator network; S4. Input the acquired state vector of the power system into the trained actuator network to generate frequency control instructions.

[0007] As a further improvement of the present invention, the actuator network includes an input layer, a hidden layer, and an output layer, wherein the input layer receives a state vector, and the state vector expression is as follows:

[0008] Where: is the state vector; for time The integral value of for time The differential value of for Frequency deviation of the power system at the moment; The hidden layer includes two fully connected layers, each of which includes 256 neurons and is activated by the ReLU function; Output layer outputs control actions , the output range of the output layer is limited to the interval by the tanh activation function between.

[0009] As a further improvement of the present invention, the simulator network is shown as follows:

[0010] Where: The frequency deviation prediction value for the next moment; is the state vector-control action pair; For state-action pairs Nonlinear mapping of The simulator network is trained by constructing a loss function that minimizes the mean squared error:

[0011] Where: is the number of training samples, for , that is, the frequency deviation of the power system at the next moment.

[0012] As a further improvement of the present invention, the expression of the action-value function is as follows:

[0013] Where: for The power system frequency deviation at the moment; is the power system state vector; To control movement; is the current control strategy of the actuator network.

[0014] As a further improvement of the present invention, the process of calculating the policy gradient includes: Based on the current control strategy of the actuator network, a small perturbation is applied, the change in the control action value function is evaluated through the simulator network, and the gradient estimate is calculated using the symmetric difference method, as shown in the following formula:

[0015] Where: is a small disturbance, which is taken as 0.001; is a standard normal distribution; is the mathematical expectation of the symmetric difference quotient; It is obedience The random perturbation vector of , has the same dimension as the action space.

[0016] As a further improvement of the present invention, initializing the actuator network based on pre-trained PID network parameters includes: sampling a training set from a database established by the PID controller, constructing a deep neural network, pre-training the deep neural network parameters through supervised learning, and migrating the trained deep neural network parameters to the actuator network.

[0017] As a further improvement of the present invention, the optimization of actuator network parameters based on the experience replay mechanism includes: Randomly sample M transfer samples from the experience pool and calculate the policy gradient using the chain rule:

[0018] Where: is the policy gradient; is the mini-batch sampling size; represents the control strategy based on the current actuator network parameters; represents the gradient of the action-value function with respect to the control action; Output the gradient of the actuator network with respect to the network parameters; And the policy gradient is used to update the actuator network parameters. The update formula is:

[0019] Where: For the The parameters of the actuator network at the learning iteration; For the The parameters of the actuator network at the learning iteration; is the learning rate.

[0020] A model-free load frequency control system for nonlinear power systems based on deep reinforcement learning, including: A construction module is used to construct a dual-network architecture, including an actuator network and a simulator network; the actuator network receives a state vector from the power system to output a control action; the simulator network generates a frequency deviation prediction value at the next moment based on the state vector and the control action, and constructs a loss function to train the simulator network; An optimization module is used to optimize the actuator network parameters based on the dual-network architecture, including constructing an action-value function, calculating the policy gradient using a zero-order optimization algorithm based on the action-value function, and updating the actuator network parameters based on the policy gradient; A training module is used to initialize the actuator network based on pre-trained PID network parameters, generate exploration noise through random perturbations to explore the control space, and iteratively optimize the actuator network parameters based on the experience replay mechanism until convergence, thereby obtaining a trained actuator network; The output module is used to input the acquired state vector of the power system into the trained actuator network to generate frequency control instructions; A nonlinear power system model-free load frequency control device based on deep reinforcement learning includes a processor and a memory, wherein the processor implements the above-mentioned nonlinear power system model-free load frequency control method based on deep reinforcement learning when executing a computer program stored in the memory.

[0021] A computer-readable storage medium is used to store a computer program, wherein when the computer program is executed by a processor, it implements the above-mentioned nonlinear power system model-free load frequency control method based on deep reinforcement learning.

[0022] Compared with the prior art, the present invention has achieved the following technical effects: The present invention takes into account the nonlinear problem of load regulation in power systems. First, a DDPG framework consisting of an actuator network and a simulator network is constructed. The actuator network receives a state vector and outputs a control action. The simulator network predicts the frequency deviation at the next moment through historical data training and continuously optimizes by minimizing the prediction error. Secondly, an action value function is designed, and the policy gradient is calculated using the simulator network. A zero-order optimization method is introduced to solve the gradient anomaly problem of deep networks through symmetric difference estimation, avoiding the gradient vanishing or exploding problem caused by the complex network structure in traditional gradient descent-based methods, and ensuring the stability and reliability of the intelligent agent parameter update. Finally, an experience replay mechanism is used to iteratively optimize the actuator network parameters to realize model-free frequency control of the power system under nonlinear characteristics such as governor dead zone and power generation rate constraints.

[0023] The present invention can simulate the dynamic characteristics of the power system through a simulator network, breaking away from the dependence on the precise mathematical model of the system; the policy gradient designed by the present invention based on the action value function has a clear physical meaning. Compared with the policy gradient generated by the traditional network, it can provide a clearer parameter update direction for the actuator network and significantly improve the efficiency of reinforcement learning training; in practical applications, this method can autonomously adapt to the time-varying parameters such as inertia and damping of the power system, especially in scenarios with a high proportion of intermittent energy such as wind power and photovoltaics, and exhibits stronger anti-interference capabilities. By generating frequency control instructions in real time, the instantaneous power imbalance between power generation and load is quickly eliminated, and the accumulation of frequency deviations is effectively suppressed, thereby avoiding the risk of power system frequency collapse. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a flow chart of the load frequency control method based on deep reinforcement learning in an embodiment of the present invention.

[0025] Figure 2 This is a model block diagram of a load frequency control system according to an embodiment of the present invention.

[0026] Figure 3 Schematic diagram of executor network update according to an embodiment of the present invention.

[0027] Figure 4 Schematic diagram of policy gradient calculation combined with simulation network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0029] like Figure 1 As shown, the present invention proposes a model-free load frequency control method for a nonlinear power system based on deep reinforcement learning, comprising the following steps: Construct a dual-network architecture, including an actuator network and a simulator network; the actuator network receives a state vector from the power system to output a control action; the simulator network generates a predicted frequency deviation value for the next moment based on the state vector and the control action, and constructs a loss function to train the simulator network; S2. Optimizing actuator network parameters based on the dual-network architecture, including constructing an action-value function, calculating a policy gradient using a zero-order optimization algorithm based on the action-value function, and updating actuator network parameters based on the policy gradient; S3. Initializing the actuator network based on pre-trained PID network parameters, generating exploration noise through random perturbations to explore the control space, and iteratively optimizing the actuator network parameters based on the experience replay mechanism until convergence, thereby obtaining a trained actuator network; S4. Input the acquired state vector of the power system into the trained actuator network to generate frequency control instructions.

[0030] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments: like Figure 2 As shown in the figure, a model-free load frequency control method for nonlinear power systems based on deep reinforcement learning is implemented. The load frequency control model shown below is constructed. This model includes key components such as the generator, speed regulator, and steam turbine. Its dynamic characteristics can be described by the following differential equations: (1) Wherein, the parameters are defined as follows: is the system frequency deviation; Generator mechanical power output deviation; is the valve opening position deviation; To include normal load variations and renewable energy fluctuations The total disturbance of ; is the turbine time constant; is the speed regulator time constant; is the system inertia constant; is the generator damping coefficient; It is the generator regulation instruction output by the controller.

[0031] In actual power system operation, generators have two important nonlinear constraints: (1) Governor dead zone characteristic (GDB): The governor has an insensitive zone. Only when the control command exceeds the dead zone threshold This characteristic can be expressed as: (2) Wherein, the parameters are defined as follows: express and The maximum value in ; express and The minimum value in ; is the dead band bandwidth of the speed regulator.

[0032] (2) Generation Rate Constraint (GRC): Due to the physical characteristics of the unit, there is an upper limit on the rate of change of generator power. , its mathematical expression is: (3) Wherein, the parameters are defined as follows: is the maximum rate of change of generated power; is the rate of change of generated power. The dual network architecture of the present invention includes an actuator network and a simulator network.

[0033] The actuator network is implemented using a deep neural network architecture, which includes an input layer, a hidden layer, and an output layer. The input layer receives a state vector, which is expressed as follows: (4) Where: is the state vector; for time The integral value of for time The differential value of for Frequency deviation of the power system at the moment; The hidden layer includes two fully connected layers, each of which includes 256 neurons and is activated by the ReLU function; Output layer outputs control actions , the output range of the output layer is limited to the interval by the tanh activation function The amplitude can be scaled according to the actual power system requirements.

[0034] The simulator network consists of a three-layer fully connected neural network, and the input dimension matches the sum of the state space and action space dimensions, as shown in the following formula: (5) Where: The frequency deviation prediction value for the next moment; is the state vector-control action pair; The simulator network is trained by constructing a loss function that minimizes the mean squared error: (6) Where: is the number of training samples, for , that is, the frequency deviation of the power system at the next moment.

[0035] The simulator network of the present invention uses historical operation data to directly establish the mapping relationship between state-action pairs and next states, and its output is the predicted value of the system state rather than the actual value. value, and the network training uses the Adam optimizer.

[0036] like Figure 3 As shown, the environmental interaction mechanism of the present invention includes a state observation module, an action execution module and an experience replay pool. The state observation module collects system frequency deviation in real time. , generator output and load disturbances The action execution module converts the control instructions output by the actuator network into Acting on the power system environment. The experience replay pool uses a ring buffer structure to store transfer samples ,in represents the current state vector, Indicates a control action. Indicates immediate reward, represents the next state vector.

[0037] After completing the exploration and storage of state transition samples, the present invention needs to update the network parameters through the policy gradient Traditional reinforcement learning-based load frequency control methods usually use the following two policy gradient calculation methods: the critic network training method and the system model formula derivation method. However, these traditional methods have obvious limitations: the critic network training is time-consuming and easily converges to a suboptimal solution, and the model formula derivation depends on the accuracy of the system parameters and the action value function. The derivation process is complicated.

[0038] Therefore, the present invention adopts a method for calculating policy gradients using a simulator network, including: Predict the frequency deviation state at the next moment through the simulator network , and calculate its action The gradient of . The zero-order optimization algorithm is used to implement the symmetric difference gradient estimation: (7) Where: is a small disturbance, which is taken as 0.001; is a standard normal distribution; is the mathematical expectation of the symmetric difference quotient; It is obedience The random perturbation vector of , has the same dimension as the action space.

[0039] Based on the optimization objective of load frequency control (LFC), the present invention defines the following action value function , used to evaluate the control instructions generated by the policy agent based on the power system state: (8) Where: for The power system frequency deviation at the moment; is the power system state vector; To control movement; is the current control strategy of the actuator network.

[0040] Strategy optimization target: actuator network parameters The optimization objective function Defined as the action-value function Expectations: (9) Where, Represents the set of actuator network parameters, including the weight matrix and the bias vector ; represents the mathematical expectation operator.

[0041] By maximizing the objective function Minimize frequency deviation: (10) The actuator network update uses experience replay technology, which includes: Randomly select small batches of samples from the playback buffer and sample from the interaction experience pool Interactive Experience ; Calculate the policy gradient based on the chain rule: (11) Where: is the policy gradient; is the mini-batch sampling size; represents the control strategy based on the current actuator network parameters; represents the gradient of the action-value function with respect to the control action; Output the gradient of the actuator network with respect to the network parameters; For the gradient of the actuator network output with respect to the network parameters, we can combine Equation (7) and use the following formula to calculate: (12) At the same time, the gradient component can be obtained by BP back propagation based on the executor network .

[0042] After calculating the policy gradient through the above steps, the parameters of the agent will be iteratively updated according to the following formula: (13) Where, For the The parameters of the actuator network at the learning iteration; For the The parameters of the actuator network at the learning iteration; is the learning rate.

[0043] The present invention can gradually optimize the control performance through the parameter updating mechanism shown in equations (11)-(13).

[0044] It should be noted that the random initialization of the actuator network of the present invention may generate control instructions that deviate from the normal action space, resulting in abnormal system states, which significantly prolongs the training time of the intelligent agent. These abnormal experiences increase the complexity of the simulator network's evaluation of control actions, causing unnecessary oscillations during the actuator network training process. To address this problem and accelerate the training process, the present invention adopts a pre-training scheme based on the PID controller: first, a training set is sampled from the LFC database established by the PID controller to construct a deep neural network (DNN) called the PID network. The parameters of this pre-trained network are then used to initialize the actuator network, providing a more reasonable initial control strategy for the DDPG algorithm, achieving smoother and more efficient exploration of the control space.

[0045] The actuator network is initialized with pre-trained PID network parameters to accelerate training convergence; exploration noise is generated through the Ornstein-Uhlenbeck process to achieve full exploration of the control space.

[0046] The above-mentioned experience replay mechanism is used to optimize the parameters of the actuator network, and iterative training is performed until convergence to obtain a trained actuator network.

[0047] During runtime, the system state variables are collected in real time, and the optimal control instructions are generated through the trained actuator network.

[0048] Based on the same inventive concept, an embodiment of the present invention also provides a nonlinear power system model-free load frequency control system based on deep reinforcement learning. Since the principle of solving the problem by the nonlinear power system model-free load frequency control system based on deep reinforcement learning is similar to the aforementioned nonlinear power system model-free load frequency control method based on deep reinforcement learning, the implementation of the nonlinear power system model-free load frequency control system based on deep reinforcement learning can refer to the implementation of the nonlinear power system model-free load frequency control method based on deep reinforcement learning, and the repeated parts will not be repeated.

[0049] In specific implementation, the nonlinear power system model-free load frequency control system based on deep reinforcement learning provided by the embodiment of the present invention specifically includes: A construction module is used to construct a dual-network architecture, including an actuator network and a simulator network; the actuator network receives a state vector from the power system to output a control action; the simulator network generates a frequency deviation prediction value at the next moment based on the state vector and the control action, and constructs a loss function to train the simulator network; An optimization module is used to optimize the actuator network parameters based on the dual-network architecture, including constructing an action-value function, calculating the policy gradient using a zero-order optimization algorithm based on the action-value function, and updating the actuator network parameters based on the policy gradient; A training module initializes the actuator network based on pre-trained PID network parameters, generates exploration noise through random perturbations to explore the control space, and iteratively optimizes the actuator network parameters based on the experience replay mechanism until convergence, thereby obtaining a trained actuator network; The output module is used to input the acquired state vector of the power system into the trained actuator network to generate frequency control instructions.

[0050] Accordingly, an embodiment of the present invention also provides a nonlinear power system model-free load frequency control device based on deep reinforcement learning, comprising a processor and a memory, wherein when the processor executes the computer program stored in the memory, it implements the nonlinear power system model-free load frequency control method based on deep reinforcement learning as provided in an embodiment of the present invention.

[0051] For more specific details about the above method, please refer to the corresponding contents disclosed in the aforementioned embodiments, which will not be described again here.

[0052] Accordingly, an embodiment of the present invention also provides a computer-readable storage medium for storing a computer program, wherein, when the computer program is executed by a processor, it implements the above-mentioned deep reinforcement learning nonlinear power system model-free load frequency control method provided in an embodiment of the present invention.

[0053] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. References to the same or similar portions of the various embodiments will be sufficient. The systems, devices, and storage media disclosed in the embodiments are described briefly because they correspond to the methods disclosed in the embodiments. For relevant details, refer to the method descriptions.

[0054] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0055] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0056] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0057] The above is a detailed introduction to the deep reinforcement learning nonlinear power system model-free load frequency control method, system, device and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A model-free load frequency control method for nonlinear power systems based on deep reinforcement learning, characterized by: The following steps are involved: S1. Construct a dual-network architecture, including an actuator network and a simulator network; the actuator network receives a state vector from the power system to output a control action; the simulator network generates a frequency deviation prediction value at the next moment based on the state vector and the control action, and constructs a loss function to train the simulator network; S2. Optimizing actuator network parameters based on the dual-network architecture, including constructing an action-value function, calculating a policy gradient using a zero-order optimization algorithm based on the action-value function, and updating actuator network parameters based on the policy gradient; S3. Initializing the actuator network based on pre-trained PID network parameters, generating exploration noise through random perturbations to explore the control space, and iteratively optimizing the actuator network parameters based on the experience replay mechanism until convergence, thereby obtaining a trained actuator network; S4. Input the acquired state vector of the power system into the trained actuator network to generate frequency control instructions.

2. The method for model-free load frequency control of nonlinear power systems based on deep reinforcement learning according to claim 1, characterized in that: The actuator network includes an input layer, a hidden layer, and an output layer, wherein the input layer receives a state vector, and the state vector expression is as follows: Where: is the state vector; for time The integral value of for time The differential value of for Frequency deviation of the power system at the moment; The hidden layer includes two fully connected layers, each of which includes 256 neurons and is activated by the ReLU function; Output layer outputs control actions , the output range of the output layer is limited to the interval by the tanh activation function between.

3. The method for model-free load frequency control of nonlinear power systems based on deep reinforcement learning according to claim 1, characterized in that: The simulator network is shown below: Where: is the frequency deviation prediction value for the next moment; is the state vector-control action pair; For state-action pairs Nonlinear mapping of The simulator network is trained by constructing a loss function that minimizes the mean squared error: Where: is the number of training samples, for , that is, the frequency deviation of the power system at the next moment.

4. The method for model-free load frequency control of nonlinear power systems based on deep reinforcement learning according to claim 1, characterized in that: The expression of the action value function is as follows: Where: for The power system frequency deviation at the moment; is the power system state vector; To control movement; is the current control strategy of the actuator network.

5. The method for model-free load frequency control of nonlinear power systems based on deep reinforcement learning according to claim 1, characterized in that: The process of calculating the policy gradient includes: Based on the current control strategy of the actuator network, a small perturbation is applied, the change in the control action value function is evaluated through the simulator network, and the gradient estimate is calculated using the symmetric difference method, as shown in the following formula: Where: is a small disturbance, which is taken as 0.001; is a standard normal distribution; is the mathematical expectation of the symmetric difference quotient; It is obedience The random perturbation vector of , has the same dimension as the action space.

6. The method for model-free load frequency control of nonlinear power systems based on deep reinforcement learning according to claim 1, characterized in that: Initializing the actuator network based on pre-trained PID network parameters includes: sampling a training set from a database established by the PID controller, constructing a deep neural network, pre-training the deep neural network parameters through supervised learning, and migrating the trained deep neural network parameters to the actuator network.

7. The method for model-free load frequency control of nonlinear power systems based on deep reinforcement learning according to claim 1, characterized in that: Optimization of actuator network parameters based on the experience replay mechanism includes: Randomly sample M transfer samples from the experience pool and calculate the policy gradient using the chain rule: Where: is the policy gradient; is the mini-batch sampling size; represents the control strategy based on the current actuator network parameters; represents the gradient of the action-value function with respect to the control action; Output the gradient of the actuator network with respect to the network parameters; And the policy gradient is used to update the actuator network parameters. The update formula is: Where: For the The parameters of the actuator network at the learning iteration; For the The parameters of the actuator network at the learning iteration; is the learning rate.

8. A model-free load frequency control system for nonlinear power systems based on deep reinforcement learning, characterized by: include: A construction module is used to construct a dual-network architecture, including an actuator network and a simulator network; the actuator network receives a state vector from the power system to output a control action; The simulator network generates a frequency deviation prediction value at the next moment based on the state vector and the control action, and constructs a loss function to train the simulator network; An optimization module is used to optimize the actuator network parameters based on the dual-network architecture, including constructing an action-value function, calculating the policy gradient using a zero-order optimization algorithm based on the action-value function, and updating the actuator network parameters based on the policy gradient; A training module is used to initialize the actuator network based on pre-trained PID network parameters, generate exploration noise through random perturbations to explore the control space, and iteratively optimize the actuator network parameters based on the experience replay mechanism until convergence, thereby obtaining a trained actuator network; The output module is used to input the acquired state vector of the power system into the trained actuator network to generate frequency control instructions.

9. A model-free load frequency control device for nonlinear power systems based on deep reinforcement learning, characterized in that: The method comprises a processor and a memory, wherein when the processor executes the computer program stored in the memory, the method for model-free load frequency control of a nonlinear power system based on deep reinforcement learning is implemented.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, it implements the model-free load frequency control method for a nonlinear power system based on deep reinforcement learning as described in any one of claims 1 to 7.

Images