Load frequency control method and device based on deep reinforcement learning, and electronic equipment

Abstract

The invention relates to the technical field of power systems, and discloses a load frequency control method and device based on deep reinforcement learning, and electronic equipment. The method comprises the following steps: for any regional subsystem, determining a model by using a corresponding initial controller parameter, and processing current state data to obtain target action data; executing each target action data in the environment to obtain a reward value and current state data of each target area subsystem; if each model meets the preset termination condition, storing each model parameter to control the load frequency; if each model does not meet a preset termination condition, training any target model by using batch sample experience extracted from a corresponding playback buffer area, and taking the obtained trained model as a new initial controller parameter determination model; and resetting the target interconnected power system, and returning to the first step. According to the invention, the accurate PID controller parameter adjustment amount can be generated according to the state of the power system.

Description

Technical Field

[0001] The present disclosure relates to the technical field of power systems, and in particular to a load frequency control method, device and electronic equipment based on deep reinforcement learning. Background Art

[0002] Modern power systems are facing challenges of growing demand for renewable energy and complex operating environments. Automatic Load Frequency Control (ALFC) is an important control link in power systems, which aims to maintain load and generation balance by regulating the power flow of tie lines and frequency oscillations between interconnected areas. Therefore, under uncertain conditions, an effective load frequency control strategy is crucial to achieve a balance between system reliability and efficiency.

[0003] At present, the load frequency control (LFC) of power systems still uses the classic proportional integral differential (PID) controller, which has a simple structure, high reliability, and a good performance-cost ratio. However, for decades, the gain value of the PID controller has been mainly based on experience, and is adjusted through trial and error procedures and traditional adjustment methods (such as the Ziegler-Nicholas method). However, under random load changes and various operating environments, the traditional PID controller control strategy reveals its limitations and cannot ensure the stability of the grid frequency and the balance between load and power generation. Therefore, it is urgent to solve this technical problem. Summary of the invention

[0004] In view of the above situation, the embodiments of the present disclosure provide a load frequency control method, device and electronic device based on deep reinforcement learning, aiming to solve the above problems or at least partially solve the above problems.

[0005] In a first aspect, an embodiment of the present disclosure provides a load frequency control method based on deep reinforcement learning, the method comprising:

[0006] For any regional subsystem in the target interconnected power system, obtain corresponding current state data and an initial controller parameter determination model, wherein the current state data is regional control error data, and the initial controller parameter determination model is generated based on a deep reinforcement learning algorithm;

[0007] For any target area subsystem, at each time step, the corresponding initial controller parameter determination model is used to process the corresponding current state data to obtain target action data, which is a set of controller parameter adjustment quantities; each target action data is executed in the target interconnected power system as the environment to obtain feedback data of the environment, which includes a reward value and the current state data of each target area subsystem; the reward value, and the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in a playback buffer corresponding to the target area subsystem;

[0008] If it is determined that each of the initial controller parameter determination models satisfies a preset termination condition, each of the initial controller parameter determination models is used to generate a controller parameter adjustment amount set according to current state data of the target interconnected power system to control the load frequency of the target interconnected power system;

[0009] If it is determined that each of the initial controller parameter determination models does not meet the preset termination conditions, then for any target initial controller parameter determination model, batch sample experience is extracted from the corresponding playback buffer, and the target initial controller parameter determination model is trained using the batch sample experience, and the trained model is used as the new initial controller parameter determination model; the target interconnected power system is reset, and the step of returning to any regional subsystem in the target interconnected power system to obtain the corresponding current state data is performed.

[0010] In a second aspect, the present disclosure also provides a load frequency control device based on deep reinforcement learning, the device comprising:

[0011] An acquisition module, used for acquiring corresponding current state data and an initial controller parameter determination model for any regional subsystem in the target interconnected power system, wherein the current state data is regional control error data, and the initial controller parameter determination model is generated based on a deep reinforcement learning algorithm;

[0012] An experience collection module is used for, for any target area subsystem, at each time step, using the corresponding initial controller parameter determination model, processing the corresponding current state data to obtain target action data, wherein the action data is a set of controller parameter adjustment amounts; executing each of the target action data in the target interconnected power system as the environment to obtain feedback data of the environment, wherein the feedback data includes a reward value and the current state data of each target area subsystem; the reward value, and the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in a playback buffer corresponding to the target area subsystem;

[0013] A control module, configured to generate a controller parameter adjustment amount set according to current state data of the target interconnected power system by using each of the initial controller parameter determination models if it is determined that each of the initial controller parameter determination models satisfies a preset termination condition, so as to control the load frequency of the target interconnected power system;

[0014] A training module is used for extracting batch sample experience from the corresponding playback buffer for any target initial controller parameter determination model if it is determined that each of the initial controller parameter determination models does not meet the preset termination conditions, using the batch sample experience to train the target initial controller parameter determination model, and using the obtained trained model as the new initial controller parameter determination model; resetting the target interconnected power system, and returning to the step of obtaining the corresponding current state data for any regional subsystem in the target interconnected power system.

[0015] In a third aspect, an embodiment of the present disclosure further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the above-mentioned load frequency control method based on deep reinforcement learning.

[0016] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple applications, the electronic device performs the steps of the above-mentioned load frequency control method based on deep reinforcement learning.

[0017] By means of the above technical scheme, the load frequency control method, device and electronic device based on deep reinforcement learning provided by the embodiment of the present disclosure are compared with the existing method of setting PID controller parameters according to experience to control the load frequency of the power grid. The embodiment of the present disclosure proposes a method for automatic control of power grid load frequency based on deep reinforcement learning. Specifically, for multiple regional subsystems in the target interconnected power system, the corresponding initial controller parameter determination model (agent) is trained respectively, and the experience data is obtained from the dynamic target interconnected power system environment through the interactive trial and error method, and the existing experience is effectively used for autonomous learning and training, so each agent can autonomously adjust its behavior to adapt to the changing power system conditions and load requirements, and improve the robustness and performance of the system under various uncertain conditions. According to the current state of the target interconnected power system, the PID controller parameter adjustment amount is generated to adjust the parameters of the PID controller controlling the target interconnected power system. The solution provided by the embodiment of the present disclosure can effectively deal with the nonlinear problems caused by load-generation changes, has a wide range of application prospects in the power system, and can effectively solve the frequency stability and energy management problems in complex environments.

[0018] The above description is only an overview of the technical solution of the present disclosure. In order to more clearly understand the technical means of the present disclosure, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present disclosure more obvious and easy to understand, the specific implementation methods of the present disclosure are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0020] Figure 1 A schematic diagram of a flow chart of a load frequency control method based on deep reinforcement learning provided in an embodiment of the present disclosure is shown;

[0021] Figure 2 A schematic diagram of the structure of a controller parameter determination model provided by an embodiment of the present disclosure is shown;

[0022] Figure 3 A schematic diagram of the structure of a load frequency control device based on deep reinforcement learning provided by an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of the present disclosure clearer, the technical solutions of the present disclosure will be clearly and completely described below in combination with the specific embodiments of the present disclosure and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0025] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that such usage is interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "including" and its variants are to be interpreted as open-ended terms meaning "including but not limited to".

[0027] As mentioned above, at present, the load frequency control (Load Frequency Control, LFC) of the power system still uses the classic proportional integral differential (PID) controller, which has a simple structure, high reliability, and a good performance-cost ratio. However, for decades, the gain value of the PID controller has mainly relied on experience and has been adjusted through trial and error procedures and traditional adjustment methods (such as the Ziegler-Nicholas method). However, under random load changes and various operating environments, the traditional PID controller control strategy reveals limitations and cannot ensure the stability of the grid frequency and the balance between load and power generation. Based on this, the present invention proposes a load frequency control method, device and electronic device based on deep reinforcement learning. The present disclosure is described in detail below through specific embodiments.

[0028] Before introducing the specific embodiments, the professional terms involved in the embodiments of the present disclosure are explained first:

[0029] 1) Interconnected Power Systems refers to a system in which several independent power systems are connected through interconnection lines or other connecting equipment.

[0030] 2) Area Control Error (ACE): An indicator used to measure the power generation and load conditions in a certain area.

[0031] 3) Environment: In deep reinforcement learning, the environment is the external world where the agent (in this disclosure, the controller parameter determination model) is located. It can be the real physical world or a virtual simulated environment. The agent receives state information of the environment at each time step, selects an appropriate action, and then the environment gives feedback to the action, including rewards and new states. In the interconnected power system of the present disclosure, everything except the agent is called the environment. The interconnected power system adopts an integrated hybrid power system architecture, which may include wind power, photovoltaics, electric vehicles, hydropower and thermal power plants, and nonlinear factors such as generation dead zone (GDB) and generation rate constraint (GRC) should be considered.

[0032] 4) Observation: Observation is the information obtained by the agent from the environment, usually including a description of the state and other relevant information.

[0033] 5) Action: An action is one of the decisions that an agent can take in a given state to maximize the reward in a certain state. In this disclosure, an action is in the form of a control signal output by the agent to control the power plant.

[0034] 6) Reward: Reward is the feedback given by the environment after the agent performs an action. It is a scalar value, usually used to measure the quality of the action.

[0035] 7) Time step: A basic unit of interaction between the agent and the environment. After each time step, the agent will take an action based on the current state, and the environment will give a new state and reward.

[0036] To facilitate understanding of this embodiment, a load frequency control method based on deep reinforcement learning disclosed in the embodiment of the present disclosure is first introduced in detail. The execution subject of the load frequency control method based on deep reinforcement learning provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server or other processing device, and the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, etc. In some possible implementations, the load frequency control method based on deep reinforcement learning can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0037] Figure 1 The flowchart of the load frequency control method based on deep reinforcement learning provided by the embodiment of the present disclosure is shown. Figure 1 It can be seen that the embodiment of the present disclosure at least includes steps S101-S104:

[0038] S101: For any regional subsystem in the target interconnected power system, obtain corresponding current state data and an initial controller parameter determination model, wherein the current state data is regional control error data, and the initial controller parameter determination model is generated based on a deep reinforcement learning algorithm.

[0039] S102: For any target area subsystem, at each time step, the corresponding initial controller parameter determination model is used to process the corresponding current state data to obtain target action data, where the action data is a set of controller parameter adjustment quantities; each target action data is executed in the target interconnected power system as the environment to obtain feedback data of the environment, where the feedback data includes a reward value and the current state data of each target area subsystem; the reward value, as well as the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in a playback buffer corresponding to the target area subsystem.

[0040] S103: If it is determined that each of the initial controller parameter determination models meets a preset termination condition, each of the initial controller parameter determination models is used to generate a controller parameter adjustment amount set according to the current state data of the target interconnected power system to control the load frequency of the target interconnected power system.

[0041] S104: If it is determined that each of the initial controller parameter determination models does not satisfy the preset termination conditions, then for any target initial controller parameter determination model, batch sample experience is extracted from the corresponding playback buffer, and the target initial controller parameter determination model is trained using the batch sample experience, and the trained model is used as the new initial controller parameter determination model; the target interconnected power system is reset, and the step of returning to any regional subsystem in the target interconnected power system to obtain the corresponding current state data is performed.

[0042] from Figure 1 It can be seen from the method shown that, compared with the existing method of setting PID controller parameters based on experience to control the power grid load frequency, the embodiment of the present disclosure proposes a method for automatic control of power grid load frequency based on deep reinforcement learning. Specifically, for multiple regional subsystems in the target interconnected power system, the corresponding initial controller parameter determination model (agent) is trained respectively, and the experience data is obtained from the dynamic target interconnected power system environment through an interactive trial and error method, and the existing experience is effectively used for autonomous learning and training. Therefore, each agent can autonomously adjust its behavior to adapt to the changing power system conditions and load requirements, and improve the robustness and performance of the system under various uncertain conditions. According to the current state of the target interconnected power system, the PID controller parameter adjustment amount is generated to adjust the parameters of the PID controller that controls the target interconnected power system. The solution provided by the embodiment of the present disclosure can effectively deal with the nonlinear problems caused by load-generation changes, has a wide range of application prospects in the power system, and can effectively solve the frequency stability and energy management problems in complex environments.

[0043] The above S101-S104 are described in detail below.

[0044] Regarding S101 above:

[0045] Exemplarily, the target interconnected power system includes two regional subsystems: regional subsystem A and regional subsystem B. The regional control error data 1 of the system can be obtained from regional subsystem A as the current state data of system A, and the initial controller parameter determination model 1 can be obtained; the regional control error data 2 of the system can be obtained from regional subsystem B as the current state data of system B, and the initial controller parameter determination model 2 can be obtained.

[0046] In a possible implementation manner, the regional control error data includes at least one of the following: a regional control error, an integral of the regional control error, and a rate of change of the regional control error. Exemplarily, the regional control error data includes: a regional control error, an integral of the regional control error, and a rate of change of the regional control error. It should be noted that the examples cited here are only illustrative and do not limit the embodiments of the present disclosure.

[0047] In one possible implementation, the deep reinforcement learning algorithm is: a twin delayed deep deterministic policy gradient algorithm, a deep Q network, a dual deep Q network, or a competitive deep Q network.

[0048] Exemplarily, if the deep reinforcement learning algorithm is a twin delayed deep deterministic policy gradient algorithm, then for any regional subsystem in the target interconnected power system, a corresponding initial controller parameter determination model can be created by initializing the actor network and the critic network, and initializing the target actor network and the target critic network using the parameters of the main actor-critic network.

[0049] The output of the fully connected layer in the traditional twin delayed deep deterministic policy gradient algorithm model is usually an arbitrary real value, including negative values. However, it has been found that this network structure may cause the PID controller parameter adjustment amount of the network output to be negative during the gradient optimization process, thereby affecting the stability and effectiveness of the model. Based on this, in a possible implementation of the present disclosure, if the deep reinforcement learning algorithm is the twin delayed deep deterministic policy gradient algorithm; the main actor network and the target actor network in the initial controller parameter determination model include an input layer, a non-negative fully connected layer, and an output layer connected in sequence.

[0050] This embodiment improves the main actor network and the target actor network in the initial controller parameter determination model by introducing a new fully connected layer. Specifically, the function form of the non-negative fully connected layer is y=abs(weights)*x, where x is the input of the network and abs(weights) represents the absolute value of the network weight. In this embodiment, the main actor network and the target actor network in the initial controller parameter determination model include an input layer, a non-negative fully connected layer and an output layer connected in sequence. This design ensures that the network weights are always non-negative values, thereby avoiding the problem of negative controller parameter adjustment that may be caused during the gradient optimization process, thereby helping to improve the stability and convergence speed of training, ensure the effectiveness of the model, and significantly reduce computational complexity and save computing resources.

[0051] Regarding S102 above:

[0052] The target interconnected power system can be a real power system or a simulated power grid environment that integrates renewable energy and adds random disturbances. Exemplarily, at time step t=1, for the aforementioned regional subsystem A, the initial controller parameter determination model 1 can be used to process the regional control error data 1 to obtain the target action data 1, that is, the controller parameter adjustment amount set 1; for the aforementioned regional subsystem B, the initial controller parameter determination model 2 can be used to process the regional control error data 2 to obtain the target action data 2, that is, the controller parameter adjustment amount set 2. Then, the target action data 1 and the target action data 2 are executed in the target interconnected power system, the state of the target interconnected power system changes, and the current state data (regional control error data 1' corresponding to regional subsystem A, regional control error data 2' corresponding to regional subsystem B) and the reward value are fed back. The tuple (regional control error data 1, action data 1, reward value, regional control error data 1') can be stored in the playback buffer 1 corresponding to the initial controller parameter determination model 1; the tuple (regional control error data 2, action data 2, reward value, regional control error data 2') can be stored in the playback buffer 2 corresponding to the initial controller parameter determination model 2. Similarly, the corresponding tuples can be calculated and stored in each playback buffer at time steps t=2, ..., T.

[0053] In order to generate the best PID controller parameters, the reward function plays an important role in guiding the controller parameter determination model to take actions to solve the load frequency control problem. A reasonable reward function helps to converge quickly, reduce the amount of calculation, and improve performance. In a possible implementation of the present disclosure, the reward value is calculated according to the following reward function:

[0054]

[0055] Where T is the total number of regional subsystems in the target interconnected power system, B i is the frequency deviation coefficient corresponding to the ith regional subsystem, Δf i is the frequency deviation of the ith regional subsystem, ΔP tie is the interconnection line power.

[0056] For example, after executing target action data 1 and target action data 2 in the target interconnected power system, the state of the target interconnected power system changes, and the tie line power and the frequency deviation of each regional subsystem can be obtained from the target interconnected power system to calculate the reward value according to the above formula. Among them, the frequency deviation coefficient corresponding to each regional subsystem can be set according to actual needs, which is not limited in the embodiment of the present disclosure.

[0057] In order to achieve the goal of minimizing the interconnection line power and the frequency fluctuation in each area, this embodiment defines the reward function as the sum of the absolute values ​​of the frequency deviation and the interconnection line power. Through the design of the reward function, it can be ensured that the controller parameter determination model subsequently converges quickly to the optimal PID controller parameter setting, thereby greatly improving the efficiency and stability of load frequency control.

[0058] Regarding S103 above:

[0059] In this step, it can be determined whether each initial controller parameter determination model meets the preset termination conditions. If so, it indicates that the performance of each initial controller parameter determination model meets the requirements. The current state data of the target interconnected power system is input into each initial controller parameter determination model, and multiple controller parameter adjustment amount sets can be obtained. Through multiple controller parameter adjustment amount sets, the proportional integral differential gain of the PID controller that controls each regional subsystem is adjusted, thereby realizing the control of the load frequency of the target interconnected power system.

[0060] In a possible implementation, the preset termination condition includes: the reward function converges, or the number of training rounds of each of the initial controller parameters determining the model reaches a preset number of training rounds.

[0061] Among them, the preset number of training rounds can be set according to actual needs, and this embodiment of the present disclosure does not limit this.

[0062] Regarding S104 above:

[0063] If it is determined that each initial controller parameter determination model does not meet the preset termination condition, then for any target initial controller parameter determination model, batch sample experience is extracted from the corresponding playback buffer. In specific implementation, random sampling can be performed in the corresponding playback buffer to obtain multiple sample experiences. The batch sample experience can be used to update the parameters of the target initial controller parameter determination model to obtain a trained model, and the trained model is used as a new initial controller parameter determination model; the initial state of the target interconnected power system is reset, and the process returns to step S101, and the process is iterated continuously until each initial controller parameter determination model meets the preset termination condition, and each initial controller parameter determination model that can accurately generate a set of PID parameter adjustment amounts according to the current state is obtained.

[0064] In one possible implementation, the initial controller parameter determination model is generated based on the twin delayed deep deterministic policy gradient algorithm; the target initial controller parameter determination model is trained using the batch sample experience, including: calculating the target Q value for each experience in the batch sample experience; updating the parameters of the two main critic networks in the target initial controller parameter determination model based on a preset loss function and each target Q value; updating the parameters of the main actor network in the target initial controller parameter determination model through a policy gradient algorithm; and updating the parameters of the target actor network and two target critic networks in the target initial controller parameter determination model through a soft update rule.

[0065] Figure 2 FIG. 1 shows a schematic diagram of the structure of the controller parameter determination model provided by the embodiment of the present disclosure. Figure 2 It can be seen that the controller parameter determination model includes an actor network and a critic network; the actor network includes 1 main actor network and 1 target actor network; the critic network includes critic 1, critic 2 (used to estimate long-term rewards) and target critic 1 and target critic 2; for the actor network: the network structure includes an interconnected input layer, a non-negative fully connected layer and an output layer. For the critic network: the network structure includes 2 input layers (used to process state s and action a respectively), 2 fully connected layers, a concatenation layer (used to connect 2 inputs), a RELU layer, a fully connected layer, a RELU layer, and a Q value layer (output layer). The following is combined with Figure 2 This embodiment is described as an example.

[0066] In this embodiment, the target Q value may be calculated for each piece of experience data. Specifically, the target Q value may be calculated according to the following formula:

[0067]

[0068] Among them, y i is the target Q value of the i-th sample experience, r i is the instantaneous reward value obtained at time step t, γ is the discount rate, Q′ i (s′,a′) represents the Q-value estimate of the i-th critic network for action a′ in state s′, s t+1 Represents the state data at time step t+1. Pick up and The minimum value of the two.

[0069] Then, based on the preset loss function and each of the target Q values, the parameters φ1 and φ2 of the two main critic networks in the target initial controller parameter determination model are updated. During implementation, the preset loss function is determined according to the following formula:

[0070]

[0071] Where M is the total number of samples in the batch sample experience Q(s i ,a i ) is the Q value predicted by the critic network for the state-action pair of the i-th sample experience.

[0072] The policy gradient algorithm can also be used to update the target initial controller parameters to determine the parameters θ of the main actor network in the model. Specifically, the following update rule can be used to update the parameters θ of the main actor network:

[0073]

[0074] in, is the gradient of the objective function with respect to the actor network parameters, is the gradient of the Q value output by the critic network with respect to action a, is the gradient of the action a output by the actor network with respect to the parameter θ.

[0075] The parameters φ'1 and φ'2 of the target actor network and the two target critic networks in the target initial controller parameter determination model can be updated by the soft update rule, and finally the trained model is obtained. Specifically, the parameters of the target actor network and the target critic network can be updated according to the following formula:

[0076] θ μ′ =τθ μ +(1-τ)θ μ′

[0077] θ Q′ =τθ Q +(1-τ)θ Q′

[0078] Among them, θ μ′ is the network parameter of the target actor network, τ is the smoothing factor, θ μ are the network parameters of the main actor network, θ Q′ is the network parameter of the target critic network, θ Q are the network parameters of the main critic network.

[0079] When implemented, the Adam optimizer can be used to update the parameters of the actor and critic networks, and the Glorot initializer is used for weight initialization of the fully connected layers.

[0080] The present disclosure also provides a load frequency control method based on deep reinforcement learning, comprising the following steps:

[0081] Step S1: For any regional subsystem in the target interconnected power system, the corresponding current state data and the initial controller parameter determination model are obtained, the current state data includes the regional control error, the integral of the regional control error and the rate of change of the regional control error, and the initial controller parameter determination model is generated based on the twin delayed deep deterministic policy gradient algorithm. During implementation, the main actor network and the target actor network in the initial controller parameter determination model include an input layer, a non-negative fully connected layer and an output layer connected in sequence.

[0082] Step S2: For any target area subsystem, at each time step, the corresponding initial controller parameter determination model is used to process the corresponding current state data to obtain target action data, which is a set of controller parameter adjustment quantities; each target action data is executed in the target interconnected power system as the environment to obtain feedback data of the environment, which includes a reward value and the current state data of each target area subsystem; the reward value, as well as the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in the playback buffer corresponding to the target area subsystem.

[0083] When implemented, the reward value is calculated according to the following reward function:

[0084]

[0085] Where T is the total number of regional subsystems in the target interconnected power system, B i is the frequency deviation coefficient corresponding to the ith regional subsystem, Δf i is the frequency deviation of the ith regional subsystem, ΔP tie is the interconnection line power.

[0086] Step S3: If it is determined that each initial controller parameter determination model meets the preset termination condition, each initial controller parameter determination model is used to generate a controller parameter adjustment amount set according to the current state data of the target interconnected power system to control the load frequency of the target interconnected power system. During implementation, the preset termination condition includes: the reward function converges, or the number of training rounds of each initial controller parameter determination model reaches the preset number of training rounds.

[0087] Step S4: If it is determined that each initial controller parameter determination model does not meet the preset termination condition, then for any target initial controller parameter determination model, batch sample experience is extracted from the corresponding playback buffer, and the target initial controller parameter determination model is trained using the batch sample experience, and the trained model is used as the new initial controller parameter determination model; the target interconnected power system is reset, and the step of returning to any regional subsystem in the target interconnected power system to obtain the corresponding current state data is performed.

[0088] Specifically, the target initial controller parameter determination model is trained using the batch sample experience, including: calculating the target Q value for each experience in the batch sample experience;

[0089] Based on the preset loss function and each target Q value, the target initial controller parameters are updated to determine the parameters of the two main critic networks in the model; through the policy gradient algorithm, the target initial controller parameters are updated to determine the parameters of the main actor network in the model; through the soft update rule, the target initial controller parameters are updated to determine the parameters of the target actor network and the two target critic networks in the model.

[0090] Those skilled in the art will appreciate that, in the above method of the specific implementation manner, the writing order of the steps does not mean a strict execution order, but does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0091] It should be noted that, in practical applications, all possible implementations described above can be combined in any manner to form possible embodiments of the present disclosure, which will not be described one by one here.

[0092] Based on the same concept, the embodiment of the present disclosure also provides a key point detection device, Figure 3 A schematic diagram of the structure of a load frequency control device based on deep reinforcement learning provided by an embodiment of the present disclosure is shown. Figure 3 As shown, the load frequency control device 300 based on deep reinforcement learning provided by the embodiment of the present disclosure includes:

[0093] An acquisition module 301 is used to acquire corresponding current state data and an initial controller parameter determination model for any regional subsystem in the target interconnected power system, wherein the current state data is regional control error data, and the initial controller parameter determination model is generated based on a deep reinforcement learning algorithm;

[0094] The experience collection module 302 is used for processing the corresponding current state data of any target area subsystem at each time step using the corresponding initial controller parameter determination model to obtain target action data, wherein the action data is a set of controller parameter adjustment amounts; executing each target action data in the target interconnected power system as the environment to obtain feedback data of the environment, wherein the feedback data includes a reward value and the current state data of each target area subsystem; the reward value, and the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in the playback buffer corresponding to the target area subsystem;

[0095] The control module 303 is used to generate a controller parameter adjustment amount set according to the current state data of the target interconnected power system by using each of the initial controller parameter determination models if it is determined that each of the initial controller parameter determination models meets a preset termination condition, so as to control the load frequency of the target interconnected power system;

[0096] The training module 304 is used to extract batch sample experience from the corresponding playback buffer for any target initial controller parameter determination model if it is determined that each of the initial controller parameter determination models does not meet the preset termination conditions, and use the batch sample experience to train the target initial controller parameter determination model, and use the obtained trained model as the new initial controller parameter determination model; reset the target interconnected power system, and return to the step of obtaining the corresponding current state data for any regional subsystem in the target interconnected power system.

[0097] In a possible implementation manner, in the above device, the regional control error data includes at least one of the following: a regional control error, an integral of the regional control error, and a rate of change of the regional control error.

[0098] In a possible implementation, in the above-mentioned device, the deep reinforcement learning algorithm is: a twin delayed deep deterministic policy gradient algorithm, a deep Q network, a dual deep Q network or a competitive deep Q network.

[0099] In one possible implementation, in the above-mentioned device, if the deep reinforcement learning algorithm is the twin delayed deep deterministic policy gradient algorithm; the main actor network and the target actor network in the initial controller parameter determination model include an input layer, a non-negative fully connected layer and an output layer connected in sequence.

[0100] In a possible implementation manner, in the above device, the reward value is calculated according to the following reward function:

[0101]

[0102] Where T is the total number of regional subsystems in the target interconnected power system, B i is the frequency deviation coefficient corresponding to the ith regional subsystem, Δf i is the frequency deviation of the ith regional subsystem, ΔP tie is the interconnection line power.

[0103] In a possible implementation manner, in the above device, the preset termination condition includes:

[0104] The reward function converges, or the number of training rounds of each initial controller parameter determination model reaches a preset number of training rounds.

[0105] In a possible implementation, in the above-mentioned device, the initial controller parameter determination model is generated based on the twin delayed deep deterministic policy gradient algorithm; the training module 304 is used to: calculate the target Q value for each experience in the batch sample experience; update the parameters of the two main critic networks in the target initial controller parameter determination model based on a preset loss function and each of the target Q values; update the parameters of the main actor network in the target initial controller parameter determination model through a policy gradient algorithm; and update the parameters of the target actor network and the two target critic networks in the target initial controller parameter determination model through a soft update rule.

[0106] It should be noted that any of the above-mentioned load frequency control devices based on deep reinforcement learning can implement the above-mentioned load frequency control method based on deep reinforcement learning one-to-one, which will not be repeated here.

[0107] Figure 4 FIG. 1 shows a schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure. Figure 4 As shown, at the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. Among them, the memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage, etc. Of course, the electronic device may also include hardware required for other services.

[0108] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] The memory is used to store programs. Specifically, the program may include program code, and the program code includes computer operation instructions. The memory may include internal memory and non-volatile memory, and provides instructions and data to the processor.

[0110] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a load frequency control device based on deep reinforcement learning at the logical level. The processor executes the program stored in the memory and is specifically used to execute the above method.

[0111] The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0112] The electronic device can execute the load frequency control method based on deep reinforcement learning provided by multiple embodiments of the present disclosure, and realize a load frequency control device based on deep reinforcement learning. Figure 3 The functions of the illustrated embodiments will not be described in detail in the embodiments of the present disclosure.

[0113] The embodiments of the present disclosure also propose a computer-readable storage medium, which stores one or more programs, and the one or more programs include instructions. When the instructions are executed by an electronic device including multiple application programs, the electronic device can execute the load frequency control method based on deep reinforcement learning provided by multiple embodiments of the present disclosure.

[0114] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0116] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0117] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0118] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0119] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0120] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0121] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity or device including the element.

[0122] It will be appreciated by those skilled in the art that the embodiments of the present disclosure may be provided as methods, systems or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0123] The above are only embodiments of the present disclosure and are not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. A load frequency control method based on deep reinforcement learning, characterized in that: The method comprises: For any regional subsystem in the target interconnected power system, obtain corresponding current state data and an initial controller parameter determination model, wherein the current state data is regional control error data, and the initial controller parameter determination model is generated based on a deep reinforcement learning algorithm; For any target area subsystem, at each time step, the corresponding initial controller parameter determination model is used to process the corresponding current state data to obtain target action data, which is a set of controller parameter adjustment quantities; each target action data is executed in the target interconnected power system as the environment to obtain feedback data of the environment, which includes a reward value and the current state data of each target area subsystem; the reward value, and the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in a playback buffer corresponding to the target area subsystem; If it is determined that each of the initial controller parameter determination models satisfies a preset termination condition, each of the initial controller parameter determination models is used to generate a controller parameter adjustment amount set according to current state data of the target interconnected power system to control the load frequency of the target interconnected power system; If it is determined that each of the initial controller parameter determination models does not meet the preset termination conditions, then for any target initial controller parameter determination model, batch sample experience is extracted from the corresponding playback buffer, and the target initial controller parameter determination model is trained using the batch sample experience, and the trained model is used as the new initial controller parameter determination model; the target interconnected power system is reset, and the step of returning to any regional subsystem in the target interconnected power system to obtain the corresponding current state data is performed.

2. The method according to claim 1, characterized in that The regional control error data includes at least one of the following: a regional control error, an integral of the regional control error, and a rate of change of the regional control error.

3. The method according to claim 1, characterized in that The deep reinforcement learning algorithm is: a twin delayed deep deterministic policy gradient algorithm, a deep Q network, a dual deep Q network or a competitive deep Q network.

4. The method according to claim 3, characterized in that If the deep reinforcement learning algorithm is the twin delayed deep deterministic policy gradient algorithm; The main actor network and the target actor network in the initial controller parameter determination model include an input layer, a non-negative fully connected layer and an output layer connected in sequence.

5. The method according to claim 1, characterized in that The reward value is calculated according to the following reward function: Where T is the total number of regional subsystems in the target interconnected power system, B i is the frequency deviation coefficient corresponding to the ith regional subsystem, Δf i is the frequency deviation of the ith regional subsystem, ΔP tie is the interconnection line power.

6. The method according to claim 1, characterized in that The preset termination conditions include: The reward function converges, or the number of training rounds of each initial controller parameter determination model reaches a preset number of training rounds.

7. The method according to claim 3 or 4, characterized in that: The initial controller parameter determination model is generated based on the twin delay deep deterministic policy gradient algorithm; and the use of the batch sample experience to train the target initial controller parameter determination model includes: For each experience in the batch of sample experiences, calculate a target Q value; Based on a preset loss function and each of the target Q values, updating the parameters of the two main critic networks in the target initial controller parameter determination model; Updating the target initial controller parameters to determine the parameters of the main actor network in the model through a policy gradient algorithm; The parameters of the target actor network and two target critic networks in the target initial controller parameter determination model are updated through a soft update rule.

8. A load frequency control device based on deep reinforcement learning, characterized in that: The device comprises: An acquisition module, used for acquiring corresponding current state data and an initial controller parameter determination model for any regional subsystem in the target interconnected power system, wherein the current state data is regional control error data, and the initial controller parameter determination model is generated based on a deep reinforcement learning algorithm; An experience collection module is used for, for any target area subsystem, at each time step, using the corresponding initial controller parameter determination model, processing the corresponding current state data to obtain target action data, wherein the action data is a set of controller parameter adjustment amounts; executing each of the target action data in the target interconnected power system as the environment to obtain feedback data of the environment, wherein the feedback data includes a reward value and the current state data of each target area subsystem; the reward value, and the target action data, current state data and next current state data corresponding to the target area subsystem constitute an experience data, which is stored in a playback buffer corresponding to the target area subsystem; A control module, configured to generate a controller parameter adjustment amount set according to current state data of the target interconnected power system by using each of the initial controller parameter determination models if it is determined that each of the initial controller parameter determination models satisfies a preset termination condition, so as to control the load frequency of the target interconnected power system; A training module is used for extracting batch sample experience from the corresponding playback buffer for any target initial controller parameter determination model if it is determined that each of the initial controller parameter determination models does not meet the preset termination conditions, using the batch sample experience to train the target initial controller parameter determination model, and using the obtained trained model as the new initial controller parameter determination model; resetting the target interconnected power system, and returning to the step of obtaining the corresponding current state data for any regional subsystem in the target interconnected power system.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, characterized in that: When the one or more programs are executed by an electronic device including a plurality of application programs, the electronic device executes the steps of the method as claimed in any one of claims 1 to 7.

Images