An automatic generation control dynamic optimization method, system, device and medium based on imitation reinforcement learning

Abstract

The present application belongs to the technical field of automatic generation control, and relates to an automatic generation control dynamic optimization method, system, equipment and medium based on imitative reinforcement learning. The method comprises the following steps: defining operation constraints and control targets of AGC dynamic optimization based on an AGC dynamic optimization problem model; modeling the AGC dynamic optimization problem as a Markov decision process model based on the operation constraints and control targets of the AGC dynamic optimization; performing imitative training on the AGC controller based on offline expert experience data, completing strategy initialization, and obtaining an initialized AGC strategy; performing online interactive training on the AGC controller based on the initialized AGC strategy, solving the strategy of the Markov decision process model, realizing iterative updating of the AGC controller, and obtaining an optimal AGC strategy. The method solves the problem that key operation constraints of automatic generation control under a traditional artificial intelligence training framework cannot be satisfied.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of automatic generation control, and particularly relates to an automatic generation control dynamic optimization method, system, device and medium based on imitative reinforcement learning. BACKGROUND

[0002] Automatic Generation Control (AGC) is an important part of the power system. The regional power grid dispatching center needs to implement closed-loop correction control according to the monitored real-time regional control error (ACE) deviation, which is of great significance to realize system frequency stability and smooth tie-line power between interconnected power grids. At present, the research on traditional AGC strategies has achieved fruitful results, such as proportional-integral-derivative (PID) control, model predictive control, and learning-based intelligent control. However, the traditional AGC is a typical control with feedback delay, which may cause over-regulation or under-regulation when coordinating different AGC units (such as hydroelectric and thermal power units). In addition, with the increasing penetration of wind power, the centralized grid-connected wind power brings a large amount of minute-level power fluctuations. This further complicates the AGC regulation task and puts forward higher requirements for the real-time coordinated control of AGC.

[0003] In order to cope with the increasing power fluctuations and lag problems, in recent years, the concept of AGC dynamic optimization has been proposed. The core idea is to optimize the scheduling of AGC units in advance based on the prediction of ultra-short-term load and renewable energy (such as wind power). The main advantage of introducing AGC dynamic optimization is that it can effectively handle short-time (within 15 minutes) fluctuations caused by renewable energy, as it takes into account the information of future load and renewable energy. Therefore, AGC dynamic optimization is of great significance in power systems with random renewable energy.

[0004] Currently, the most common method to solve the AGC dynamic control problem is to use optimization programming based on the probabilistic model of wind power. However, the optimization-based method relies heavily on the accurate probabilistic model of renewable energy, which is difficult to obtain in practice. In addition, due to the uncertainty involved, the model of the stochastic process is usually non-convex, which is difficult to solve analytically and has a huge computational burden. Therefore, the fluctuations of future renewable energy are difficult to consider in advance in the AGC scheduling process. In recent years, deep reinforcement learning (RL) has become increasingly popular in dealing with AGC dynamic optimization problems by using neural networks for uncertainty prediction. Many scholars use proximal policy optimization RL algorithm, double-delay deep deterministic policy gradient method with multi-experience pool replay, or soft actor-critic (SAC) algorithm to solve the AGC dynamic optimization problem. However, these traditional RL algorithms must be trained through a large number of "trial and error" interactions with the real system to converge to become intelligent. This means that the controller may make some "wrong" decisions during the training process, resulting in serious system frequency violations. This is very unsafe and unacceptable for real systems.

[0005] In summary, the traditional automatic generation control (AGC) method only considers the economic scheduling every 15 minutes and the inertia response at the second level, and lacks optimization of AGC units within 15 minutes, so it is difficult to cope with the fluctuations in power system frequency caused by high-penetration new energy and the high uncertainty brought by real-time changing dynamic environment.

[0006] Traditional reinforcement learning either only considers training on a simulation system, but this method will result in poor application effect of the controller trained on the simulation system in the actual system due to the difference in data distribution between the simulation system and the real system; or directly trains on the real system, but direct training cannot guarantee the safety of system operation during the training process, such as the key operating constraints of automatic generation control cannot be met. This is because the initial strategy of the traditional reinforcement learning method is usually random, and the strategy needs to be iteratively updated in the continuous "trial and error" process, so it may cause catastrophic control results in the early stage of strategy training process, and therefore cannot theoretically guarantee the safety of key constraints such as system frequency and equipment limits. SUMMARY

[0007] The present application aims to provide an automatic generation control dynamic optimization method, system, device and medium based on imitation reinforcement learning, which solves the problem that the key operating constraints of automatic generation control cannot be met under the traditional artificial intelligence training framework.

[0008] The present application is achieved by the following technical solutions:

[0009] An automatic generation control dynamic optimization method based on imitation reinforcement learning, comprising:

[0010] Defining operation constraints and control targets of AGC dynamic optimization based on AGC dynamic optimization problem model;

[0011] Based on the operation constraints and control targets of AGC dynamic optimization, the AGC dynamic optimization problem is modeled as a Markov decision process model, the core elements of which include state space, action space and reward function;

[0012] Based on the offline expert experience data, the AGC controller is trained by imitation to complete policy initialization and obtain an initialized AGC policy;

[0013] Based on the initialized AGC policy, the AGC controller is trained by online interaction to solve the policy of the Markov decision process model, realize the iterative update of the AGC controller, and obtain an optimal AGC policy.

[0014] Further, the operation constraints of the AGC dynamic optimization include system power balance, AGC unit regulation characteristics, frequency deviation constraint and tie-line power deviation constraint;

[0015] The expression of system power balance is:

[0016] Among them, represents the operation power of the adjustable AGC unit i at time t; represents the operation power of the non-adjustable AGC unit i at time t; represents the grid-connected power of the wind farm at time t; represents the injection power of the demand side at time t; represents the power of the tie-line at time t; represents the line loss power at time t;

[0017] The expression of frequency deviation constraint is: Among them, Δf t represents the frequency deviation; Δ f represents the lower limit of the frequency deviation; represents the upper limit of the frequency deviation;

[0018] The expression of tie-line power deviation constraint is:

[0019] Among them, represents the tie-line power deviation; Δ P tie represents the lower limit of the tie-line power deviation; represents the upper limit of the tie-line power deviation.

[0020] Further, the control target is to schedule AGC units to meet the minimum economic cost of ancillary services and evaluation indexes of control performance indexes;

[0021] The minimum economic cost of ancillary services is expressed as:

[0022]

[0023] Wherein, min f1 represents the minimum economic cost of ancillary services, c i is the ancillary service cost coefficient of AGC unit i; is the ramping power of adjustable AGC unit i at time t; u i,t ∈{-1,0,1} represents the change direction of power output; represents the operating power of adjustable AGC unit i at time t; represents the initial time of adjustable AGC unit power generation;

[0024] The evaluation indexes of the control performance indexes include CPS1 and CPS2, wherein CPS1 is used to evaluate the correlation between the frequency deviation and the regional control deviation of the power system; CPS2 is defined as the average regional control deviation within 15 minutes, indicating that the regional control deviation remains within the tolerance range, and is used to ensure that the power exchange between regions does not exceed the specified limit value.

[0025] Further, the design of the state space considers the current power system operating state and the uncertain new energy generation information to be predicted;

[0026] The current power system operating state includes the operating power of the current adjustable AGC unit i at time t the frequency deviation Δf t , the tie-line power deviation and the regional control deviation

[0027] For the uncertain new energy generation information to be predicted, l historical wind power information is introduced into the state space to achieve better prediction, and the state is defined as follows:

[0028]

[0029] Wherein, represents the operating power of the current adjustable AGC unit i at time t, Δf t represents the frequency deviation, represents the tie-line power deviation, represents the regional control deviation; represents the historical power generation power of wind power;

[0030] In the design of the action space, considering that the control variable is the adjustment direction and the adjustment power of each AGC unit, the action is defined as the power adjustment amount of the AGC unit:

[0031] wherein I represents a set of adjustable AGC units; represents the power adjustment amount of the adjustable AGC unit; in S2, the expression of the reward function is:

[0032] r t =w1f1+w2K cps +w3r penalty ;

[0033] wherein f1 represents an operation cost index of the power system; K cps represents a performance evaluation index of AGC frequency modulation; r penalty represents a penalty index for violating the operation constraint of the power system; w1, w2, and w3 are weight factors, respectively, for balancing the trade-off between the three sub-reward items.

[0034] Further, the offline expert experience data is a set of state-action pairs, denoted as (s t ,a t );

[0035] The AGC controller is imitated and trained based on the offline expert experience data to complete the strategy initialization, and the specific process is as follows:

[0036] First, the offline expert experience data is collected, a fully connected neural network is constructed as a strategy network, the strategy network is trained by using the offline expert experience data, and the strategy network parameter φ is updated, the goal being to find a best matching imitation strategy π φ that provides a set of state-action pairs.

[0037] The update of the strategy network parameter φ uses maximum likelihood estimation, and the expression is:

[0038]

[0039] wherein φ * represents the optimal strategy network parameter; a i |s i represents the action probability under the state s i ; t represents the time, t = 0, 1, 2... T; and T represents the update step length.

[0040] Further, the update of the strategy network parameter φ is specifically as follows:

[0041] The Adam stochastic gradient descent method is used to iteratively update the strategy network parameter φ.

[0042] Further, the SAC algorithm is used to solve the strategy of the Markov decision process model, and the solution target is to maximize the reward expectation.

[0043] The expression of the reward expectation is:

[0044]

[0045] Wherein, r t represents a reward function, H (pi (·|s t )) is a balance entropy, and alpha is a hyperparameter of the balance entropy and reward feedback parameter;J π represents a reward expectation;s t represents a state at time t.

[0046] In the SAC algorithm, a double Q network is introduced, the policy network parameter phi is updated constantly through the Adam stochastic gradient descent method, converges to the reward expectation maximization, and then the optimal AGC strategy is obtained;The optimal AGC strategy is used for controlling the AGC unit, and the control instruction of each AGC unit is output, and the automatic generation control is completed.

[0047] The application also discloses an automatic generation control dynamic optimization system based on imitation expert experience combined with reinforcement learning, comprising:

[0048] An AGC dynamic optimization problem definition module is used for defining the operation constraint and control target of the AGC dynamic optimization based on the AGC dynamic optimization problem model;

[0049] A modeling module is used for modeling the AGC dynamic optimization problem as a Markov decision process model based on the operation constraint and control target of the AGC dynamic optimization, and the core elements of the Markov decision process model include a state space, an action space and a reward function;

[0050] An imitation training module is used for imitation training of the controller based on offline expert experience data, completing strategy initialization, and obtaining an initialized AGC strategy;

[0051] An online interactive training module is used for online interactive training of the AGC controller based on the initialized AGC strategy, solving the strategy of the Markov decision process model, realizing iterative update of the AGC controller, and obtaining an optimal AGC strategy.

[0052] The application also discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor realizes the steps of the automatic generation control dynamic optimization method based on the imitation reinforcement learning when executing the computer program.

[0053] The application further discloses a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement steps of the automatic generation control dynamic optimization method based on imitation reinforcement learning.

[0054] Compared with the prior art, the application has the following beneficial technical effects:

[0055] The application discloses an automatic generation control dynamic optimization method based on imitation reinforcement learning, to ensure that key operation constraints of a power system can be met in a training process. Considering that "trial and error" usually occurs in an initial stage of AGC controller training, because an initialization strategy is usually randomly generated. In order to avoid random exploration of reinforcement learning in an early stage, the application proposes to use imitation learning, to obtain an initialization strategy similar to expert experience based on offline training, to initialize the controller based on the imitation strategy, and to further train an optimal AGC strategy online by using a SAC algorithm. The imitation learning framework based on expert experience can improve the performance of the initialization strategy, and enable the early exploration strategy to imitate the historical experience of experts, thereby avoiding random exploration caused by a too poor initialization strategy, and avoiding serious constraint violations (such as a frequency not meeting upper and lower limits) in the training process. The application is based on a mode of combining online training and offline training in stages, which not only uses offline data to ensure the safety of system constraints, but also considers online training, to avoid the phenomenon of invalid controller caused by errors between a simulation system and an actual system. The imitation learning of giving expert experience is introduced into a SAC algorithm framework of traditional reinforcement learning, to effectively improve the performance of the initialization strategy, to avoid random and unsafe exploration caused by a poor strategy in an early stage, and to reduce the risk of system frequency violation.

[0056] Further, when constructing a Markov decision model, the AGC dynamic optimization process with a step of 1 minute in 15 minutes is considered, and a reinforcement learning algorithm is introduced to realize real-time dynamic prediction and advance response to high-proportion new energy fluctuations, without relying on an accurate system model, so that the power system has better self-adaptive control capability.

[0057] Further, compared with a traditional RL algorithm, SAC uses a random strategy to encourage exploration by adding entropy to the reward. Therefore, SAC is not easy to fall into a local optimum, and can better explore the action space.

[0058] In order to improve the stability and accuracy of value estimation, the SAC algorithm introduces a double Q network, uses the minimum value of two Q networks to evaluate the Q function, helps to reduce the overestimation bias that may occur in the Q learning process, and avoids the problem of possible overestimation in the evaluation of a single Q function. Two Q networks are used, and the policy network parameters φ are constantly updated by a gradient descent method. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 is a specific implementation flowchart of an automatic generation control dynamic optimization method based on imitation reinforcement learning of the present application;

[0060] Figure 2 is an AGC dynamic optimization problem model diagram of the present application;

[0061] Figure 3 is a safety reinforcement learning algorithm framework diagram combined with imitation learning of the present application;

[0062] Figure 4 is a core algorithm flowchart of an automatic generation control dynamic optimization method based on imitation reinforcement learning of the present application;

[0063] Figure 5 is a principle diagram of an automatic generation control dynamic optimization system based on imitation reinforcement learning of the present application. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present application clearer and more comprehensible, the following further describes in detail in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application, that is, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments.

[0065] The detailed description of the embodiments of the present application provided in the following accompanying drawings is not intended to limit the scope of the claimed present application, but only represents a selected embodiment of the present application. Based on the accompanying drawings and embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts fall within the protection scope of the present application.

[0066] The features and performances of the present application are further described in detail in combination with the embodiments below.

[0067] Embodiment 1

[0068] As shown in Figure 1 , the present application discloses an automatic generation control dynamic optimization method based on imitation reinforcement learning, comprising the following steps:

[0069] S1, defining the operation constraints and control targets of AGC dynamic optimization based on the AGC dynamic optimization problem model;

[0070] S2, modeling the AGC dynamic optimization problem as a Markov decision process model based on the operation constraints and control targets of AGC dynamic optimization, the core elements of the Markov decision process model including state space, action space and reward function;

[0071] S3, collect offline expert experience data as training data, imitate training of the AGC controller based on the offline expert experience data, complete strategy initialization, and obtain an initialized AGC strategy;

[0072] S4, based on the initialized AGC strategy, online interactive training of the AGC controller is performed, a strategy of a Markov decision process model is solved, iterative updating of the AGC controller is realized, and an optimal AGC strategy is obtained.

[0073] Embodiment 2

[0074] Based on the embodiment 1, the process of each step is introduced in detail.

[0075] As shown in Figure 4 , the application discloses an automatic generation control dynamic optimization method based on imitation reinforcement learning, including the following steps:

[0076] Step 1, defining the target and constraint of the AGC dynamic optimization model

[0077] First, the AGC dynamic optimization problem model graph is constructed, including an AGC controller, a saturation module, a rate limiter and a wind power grid connection unit. The AGC controller, the saturation module and the rate limiter are connected in sequence, and the decision variable of the AGC controller Figure 2 After the saturation module and the rate limiter, the final power adjustment amount is output, and then the wind power change amount of the wind power grid connection unit is added, and the system frequency is affected through the power grid system.

[0078] The fluctuation of the system frequency Δf will affect the tie-line power ΔP tie and the change of the conventional area control error |e ACE |; the AGC controller gives the regulation instruction of the adjustable AGC unit based on the monitoring of the conventional area control error |e ACE |; after the limitation of the saturation module and the ramp function, the final update is to the power system to stabilize the frequency of the power system.

[0079] Among them, the saturation module imposes upper and lower limit constraints on the output value. When the output value exceeds the upper limit, it is limited to the upper limit value output; when the output value is lower than the lower limit, it is limited to the lower limit value output; when it is between the upper and lower limits, the original output is maintained. The saturation module includes a default input port and an output port.

[0080] According to the dynamic process of the power system operation, then according to how the key variables are dynamically changed, the system dynamic differential equation of the key variables is derived. Among them, the key variables include the system frequency f, the tie-line power P​​tie and regional control error e ACE .

[0081] In the present application, the control objective is to schedule AGC units to meet the minimum economic cost of ancillary services and the evaluation index of control performance index.

[0082] Therefore, the minimum economic cost of ancillary services can be expressed as:

[0083]

[0084] where min f1represents the minimum economic cost of ancillary services, c i is the ancillary service cost coefficient of AGC unit i; is the ramping power of adjustable AGC unit i at time t; u i,t ∈{-1, 0, 1} represents the change direction of power output; represents the operating power of adjustable AGC unit i at time t; represents the operating power of adjustable AGC unit at the initial time.

[0085] The evaluation index of control performance index (CPS) includes CPS1 and CPS2, where CPS1 is used to evaluate the correlation between the frequency deviation of the power system and the area control error (ACE), and CPS2 is defined as the average ACE within 15 minutes, indicating that the ACE remains within the tolerance range to ensure that the power exchange between regions does not exceed the specified limit.

[0086] In addition, when constructing the AGC dynamic optimization problem model, the power system operation constraints need to be described by equations or inequalities.

[0087] The operation constraints of AGC dynamic optimization include system power balance, AGC unit regulation characteristics (such as control signal being limited by saturation function and ramp rate before execution), frequency deviation constraint, and tie-line power deviation constraint.

[0088] The expression of system power balance is:

[0089] The expression of frequency deviation constraint is: where Δf t represents the frequency deviation; Δ f represents the lower limit of frequency deviation; represents the upper limit of frequency deviation;

[0090] The expression of tie-line power deviation constraint is: where, represents the tie-line power deviation; Δ P tie represents the lower limit of the tie-line power deviation; represents the upper limit of the tie-line power deviation.

[0091] where, represents the operating power of the adjustable AGC unit i at time t; represents the operating power of the non-adjustable AGC unit i at time t; represents the grid-connected power of the wind farm at time t; represents the injection power of the demand side at time t; represents the power of the tie-line at time t; represents the line loss power at time t; Δf t represents the frequency deviation; represents the tie-line power deviation.

[0092] Step 2. Modeling as a Markov decision process

[0093] According to the AGC problem, the core elements of the Markov decision process (MDP) model corresponding to the control problem, such as the state space, the action space, and the reward function, are defined to obtain a sequential decision mathematical model equivalent to the original AGC dynamic optimization problem.

[0094] Specifically, 1) define the state space: the design of the state space needs to capture necessary information from two aspects: the current power system operating state and the uncertain new energy generation information to be predicted.

[0095] For the current power system operating state, the present invention considers four factors, including the operating power of the current adjustable AGC unit i at time t the frequency deviation Δf t the tie-line power deviation and the regional control deviation

[0096] For the uncertain new energy generation information to be predicted, the present invention only takes wind power prediction as an example, and introduces l historical wind power information into the state space to achieve better prediction. The state definition is as follows:

[0097]

[0098] where, represents the power generation power of the wind power at the historical time.

[0099] 2) Define the action space: in the AGC dynamic optimization problem, the control variable is the adjustment direction and adjustment power of each AGC unit. In order to simplify the action space with smaller action dimension, the action is defined as the power adjustment amount of the AGC unit: I represents a set of adjustable AGC units; represents the power adjustment amount of the adjustable AGC unit.

[0100] 3) Define the reward function: The reward design of MDP should consider the target and constraints. Here, the reward function r is designed from three aspects t =w1f1+w2K cps +w3r penalty , f1 represents the power system operation cost index; K cps represents the performance evaluation index of AGC frequency modulation; r penalty represents the penalty index for violating the power system operation constraints; here w1, w2, w3 are weight factors, used to balance the trade-off between the three sub-reward items. The third term r penalty is used to punish the total violation beyond the upper and lower limits, such as the output power, slope power, tie line power and frequency deviation in step 1.

[0101] Step 3. Offline training of AGC strategy based on imitation learning

[0102] As shown in Figure 3 , the imitation learning method can directly imitate the demonstrator (i.e. expert experience) without interacting with the real environment, and then use a classifier or regressor to copy the expert's strategy based on the pre-collected state-action pair training data. Therefore, the learning goal of the AGC controller in the offline training phase is to obtain an imitation strategy as the initial strategy π0. Given a set of pre-collected state-action pairs (s t ,a t ), the goal of the agent is to find an imitation strategy π φ that best matches the provided set of state-action pairs. The update of the policy network parameter φ uses maximum likelihood estimation:

[0103]

[0104] Since the action space considered in the design of the present application is continuous, it is assumed that the strategy of the present application follows a Gaussian distribution in each action dimension. The present application uses a fully connected neural network to approximate the strategy π, and then uses the Adam stochastic gradient descent method to update the neural network parameters to solve the optimal strategy.

[0105] Step 4. Online training combined with reinforcement learning SAC algorithm to obtain the optimal AGC strategy

[0106] In order to solve the optimal strategy π * , the present application uses the SAC algorithm, which can maximize the cumulative reward of the agent while satisfying the safety constraints.

[0107] The SAC algorithm is based on the Adam stochastic gradient descent method, which learns the strategy by maximizing the expected return. It uses two neural networks: the Actor network and the Critic network. The Actor network is used to generate actions, selecting the optimal action based on the current state and policy parameters; the Critic network is used to estimate the state value function and state-action value function.

[0108] In SAC, the policy is a probability distribution, and the agent collects data by sampling and uses these data to update the policy network parameters. SAC uses a Softmax policy to parameterize the policy network as a weighted sum of a set of basis functions, allowing the agent to explore the environment and obtain more information. Compared with traditional RL algorithms, SAC uses a random policy to encourage exploration by adding entropy to the reward. Therefore, SAC is less likely to fall into local optima and can better explore the action space.

[0109] Using the SAC algorithm to solve the policy of the Markov decision process model, the objective is to maximize the expected reward, and the optimal AGC strategy is obtained; the expression of the expected reward is:

[0110]

[0111] where r t represents the reward function, H(π(·|s t )) is the balance entropy, and α is the hyperparameter of the balance entropy and reward feedback parameter; J π represents the expected reward; s t represents the state at time t.

[0112] In addition, in order to improve the stability and accuracy of value estimation, the SAC algorithm introduces a double Q network, which uses the minimum value of two Q networks to evaluate the Q function, which helps to reduce the overestimation bias that may occur in the Q learning process, avoiding the problem of overestimation that may exist in single Q function evaluation. Using two Q networks, the policy network parameters φ are constantly updated by gradient descent method.

[0113] According to the real-time online interaction training results of the previous step, the AGC controller is constantly feedback corrected, and the AGC controller is automatically adjusted to balance between random exploration training and experience utilization optimization until the AGC controller converges to the maximum reward expectation, and the optimal AGC strategy is obtained.

[0114] The trained reinforcement learning AGC controller is deployed in the regional power grid system, which monitors the dynamic operating frequency of the power system in real time and outputs the control instructions of each AGC unit to complete the automatic generation control.

[0115] Embodiment 3

[0116] AsFigure 5 The application also discloses an automatic generation control dynamic optimization system based on imitation of expert experience and reinforcement learning, which comprises the following modules:

[0117] An AGC dynamic optimization problem definition module is configured to define operation constraints and control targets of AGC dynamic optimization based on an AGC dynamic optimization problem model.

[0118] A modeling module is configured to model the AGC dynamic optimization problem as a Markov decision process model based on the operation constraints and control targets of the AGC dynamic optimization, wherein core elements of the Markov decision process model include a state space, an action space and a reward function.

[0119] An imitation training module is configured to perform imitation training on the controller based on offline expert experience data, complete policy initialization and obtain an initialized AGC policy.

[0120] An online interactive training module is configured to perform online interactive training on the AGC controller based on the initialized AGC policy, solve the policy of the Markov decision process model, realize iterative update of the AGC controller and obtain an optimal AGC policy.

[0121] Embodiment 4

[0122] The application further discloses a computer device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements steps of the automatic generation control dynamic optimization method based on imitation reinforcement learning when executing the computer program, wherein the memory can comprise a memory, such as a high-speed random memory, and can also comprise a non-volatile memory, such as at least one disk memory; the processor, the network interface and the memory are connected with each other through an internal bus, which can be an industry standard architecture bus, a peripheral component interconnect standard bus, an extended industry standard structure bus or the like, and the bus can be divided into an address bus, a data bus and a control bus. The memory is used for storing programs, and specifically, the programs can comprise program codes, and the program codes comprise computer operation instructions. The memory can comprise a memory and a non-volatile memory, and provides instructions and data for the processor.

[0123] Embodiment 5

[0124] The application further discloses a computer readable storage medium storing a computer program, the computer program is executed by a processor to implement steps of the automatic generation control dynamic optimization method based on imitative reinforcement learning, specifically, the computer readable storage medium includes but is not limited to, for example, volatile memory and / or non-volatile memory. The volatile memory can include random access memory and / or cache memory, etc. The non-volatile memory can include read-only memory, hard disk, flash memory, optical disc, magnetic disc, etc.

[0125] Those skilled in the art will appreciate that embodiments of the application can be supplied as methods, systems, or computer program products. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk memory, optical memory, etc.) having computer usable program code embodied therein.

[0126] The application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions, which are executed via the processor of the computer or other programmable data processing apparatus, generate means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart

[0127] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including instruction means, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart

[0128] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for carrying out the functions specified in the flowchart

[0129] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application but not to limit it. Although the present application has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the specific embodiments of the present application can be modified or equivalently replaced without departing from the spirit and scope of the present application, and any modification or equivalent replacement should be covered in the protection scope of the claims of the present application.

Claims

1. An automatic generation control dynamic optimization method based on imitation reinforcement learning, characterized in that, The method comprises the following steps: Defining operation constraints and control targets of AGC dynamic optimization based on an AGC dynamic optimization problem model; Modeling the AGC dynamic optimization problem as a Markov decision process model based on the operation constraints and control targets of AGC dynamic optimization, wherein the core elements of the Markov decision process model include a state space, an action space and a reward function; Performing imitation training on the AGC controller based on offline expert experience data to complete policy initialization and obtain an initialized AGC policy; Performing online interactive training on the AGC controller based on the initialized AGC policy to solve the policy of the Markov decision process model, realize iterative updating of the AGC controller and obtain an optimal AGC policy; The operation constraints of the AGC dynamic optimization include system power balance, AGC unit regulation characteristics, frequency deviation constraints and tie-line power deviation constraints; The expression for system power balance is: wherein, representing the adjustable AGC unit at time of the operating power; representing the non-adjustable AGC unit at time of the operating power; representing the grid-connected power of the wind farm at time ; representing the injection power of the demand side at time ; representing the power of the tie-line at time ; representing the line loss power at time ; The frequency deviation constraint expression is: ; wherein, represents the frequency deviation; represents the lower limit of the frequency deviation; represents the upper limit of the frequency deviation; The tie-line power bias constraint expression is: ; wherein, represents a tie-line power bias; represents a lower limit of the tie-line power bias; represents an upper limit of the tie-line power bias; The design of the state space takes into account the current power system operation state and the uncertain new energy generation information to be predicted; Current power system operating state includes current adjustable AGC units At time of operation power , frequency deviation , tie-line power deviation and area control deviation ; For uncertain new energy power generation information that needs to be predicted, Historical wind power information is incorporated into a state space to achieve better predictions. The state is defined as follows: wherein, representing the current adjustable AGC unit at time operating power, representing the frequency deviation, representing the tie-line power deviation, representing the regional control deviation; representing the wind power at the historical time; In the design of action space, considering that the control variable is the adjustment direction and the adjustment power of each AGC unit, the action is defined as the power adjustment amount of the AGC unit: ; wherein, represents a set of adjustable AGC units; represents a power adjustment amount of the adjustable AGC unit; the expression of the reward function is: ； wherein, represent an index of power system operation cost; represent an index of performance evaluation of AGC frequency modulation; represent an index of penalty for violating power system operation constraints; are weight factors, respectively, for balancing the trade-off among the three sub-reward items.

2. The automatic generation control dynamic optimization method based on imitation reinforcement learning according to claim 1, wherein, The control target is to schedule AGC units to meet the minimum economic cost of auxiliary services and evaluation indexes of control performance indexes; The minimum economic cost of auxiliary services is expressed as: wherein, represents the minimum economic cost of auxiliary services, is the AGC unit auxiliary service cost coefficient; is the adjustable AGC unit ramp-up power at time ; represents the change direction of power output; represents the operating power of the adjustable AGC unit at time ; represents the initial time adjustable AGC unit power generation power; The evaluation indexes of the control performance indexes include CPS1 and CPS2, wherein CPS1 is used to evaluate the correlation between the frequency deviation of the power system and the regional control deviation, and CPS2 is defined as the average regional control deviation within 15 minutes, indicating that the regional control deviation remains within the tolerance range and is used to ensure that the power exchange between regions does not exceed the specified limit.

3. The automatic generation control dynamic optimization method based on imitation reinforcement learning according to claim 1, wherein, The offline expert experience data is a set of state-action pairs, denoted as ; The specific process of performing imitation training on the AGC controller based on offline expert experience data to complete policy initialization is as follows: First, collect offline expert experience data, construct a fully connected neural network as a policy network, and train the policy network using offline expert experience data to update the policy network parameters The goal is to find a best matching provided state-action pair set of the imitation policy ​ Policy network parameters The update of the policy network parameters using maximum likelihood estimation is expressed as: ； in, Represents the network parameters of the optimal policy; Representative at The probability of an action in a given state; t represents time, t=0, 1, 2...T; T represents the update step size.

4. The automatic generation control dynamic optimization method based on imitation reinforcement learning according to claim 3, wherein, The updated policy network parameters Specifically: Adam stochastic gradient descent method is used to update the parameters of the strategy network iteratively. update the parameters of the strategy network iteratively.

5. The automatic generation control dynamic optimization method based on imitation reinforcement learning according to claim 1, wherein, Solving the policy of the Markov decision process model by using the SAC algorithm, and the solving target is to maximize the reward expectation; The expression of the reward expectation is as follows: ； wherein, represents a reward function, is a balanced entropy, is a hyperparameter of the balanced entropy and reward feedback parameter; represents a reward expectation; represents a state at time t. In the SAC algorithm, a double Q network is introduced, and the policy network parameters are constantly updated by the Adam stochastic gradient descent method , and converge to the reward expectation maximization, and then the optimal AGC strategy is obtained; the optimal AGC strategy is used to control the AGC unit, output the control instructions of each AGC unit, and complete the automatic generation control.

6. A dynamic optimization system for automatic generation control based on imitative reinforcement learning of any one of claims 1 to 5, characterized in that, The method comprises the following steps: An AGC dynamic optimization problem definition module is configured to define operation constraints and control targets of AGC dynamic optimization based on an AGC dynamic optimization problem model; A modeling module is configured to model the AGC dynamic optimization problem as a Markov decision process model based on the operation constraints and control targets of AGC dynamic optimization, wherein the core elements of the Markov decision process model include a state space, an action space and a reward function; An imitation training module is configured to perform imitation training on the controller based on offline expert experience data to complete policy initialization and obtain an initialized AGC policy; An online interactive training module is configured to perform online interactive training on the AGC controller based on the initialized AGC policy to solve the policy of the Markov decision process model, realize iterative updating of the AGC controller and obtain an optimal AGC policy.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the automatic generation control dynamic optimization method based on imitation reinforcement learning in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to realize the steps of the automatic generation control dynamic optimization method based on imitation reinforcement learning in any one of claims 1 to 5.

Images