Frequency control method and device for water-wind-solar-storage system based on improved PPO algorithm

Abstract

A frequency control method and device for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm are disclosed. The method includes: constructing a frequency control model for the clean energy base; introducing an expert knowledge learning mechanism and a phased training mechanism based on the PPO algorithm to construct an improved PPO algorithm; the expert knowledge learning mechanism refers to fitting the mapping relationship between the state and action of an agent through supervised learning based on an expert dataset; the phased training mechanism refers to using a preset number of training rounds as a phase switching condition, automatically triggering parameter adjustment of the agent after reaching the preset number of training rounds, and triggering the expert knowledge learning mechanism when the training effect of the agent is poor; and intelligently controlling the frequency of the clean energy base based on the improved PPO algorithm. This invention employs a phased training strategy, enabling a single training process to cover multiple operating conditions, and combines expert guidance to meet diverse control needs, thereby improving convergence efficiency and strategy performance, and ultimately achieving intelligent frequency control of the clean energy base.

Description

Technical Field

[0001] This invention relates to the field of power system frequency control, and in particular to a frequency control method and apparatus for hydro-wind-solar-pressure-storage systems based on an improved PPO algorithm. Background Technology

[0002] By the end of 2023, Southwest China had built the regional power grid with the highest proportion of clean energy in the country, with installed capacity of clean energy sources such as hydropower, wind power, and solar power reaching 130 million kilowatts, accounting for nearly 80% of the total installed capacity. However, the connection of a large number of integrated hydropower, wind power, solar power, and energy storage bases has led to a significant decrease in system inertia and frequent frequency fluctuations. Against this backdrop, it is urgent to study frequency control strategies under multi-source coordinated operation to fully explore the frequency regulation potential of various resources within the bases.

[0003] Frequency restoration in power systems primarily relies on automatic generation control (AGC), which receives the total regulation power command from the control loop and then distributes power according to specific rules, issuing power commands to each generating unit. Some scholars have used online parameter tuning and ultra-short-term optimization methods to optimize the control and distribution loops of AGC, but their adaptive capabilities or solution efficiency remain unsatisfactory. Other scholars have made significant progress using deep reinforcement learning (DRL) to improve single-loop control and distribution, but the following shortcomings remain: 1) Research on optimizing the entire control and distribution loop using DRL is still relatively limited; 2) The applicability and performance of stochastic strategy algorithms in frequency control have not been fully studied and verified. Summary of the Invention

[0004] The purpose of this invention is to overcome the above-mentioned defects and problems in the prior art and provide a frequency control method and device for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm. On the basis of the near-end strategy optimization algorithm, an expert knowledge learning mechanism and a phased training mechanism are introduced to improve convergence efficiency and strategy performance.

[0005] To achieve the above objectives, the technical solution of the present invention is:

[0006] In a first aspect, the present invention provides a frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm, comprising:

[0007] A frequency control model is constructed for a clean energy base encompassing hydropower stations, wind farms, photovoltaic power stations, and compressed air energy storage. The frequency control model includes a generator and load model, a primary frequency regulation model, a frequency regulation model for conventional power sources, a frequency regulation model for new energy sources, a frequency regulation model for energy storage power stations, and an automatic generation control model.

[0008] Based on the proximal policy optimization algorithm, an expert knowledge learning mechanism and a phased training mechanism are introduced to construct an improved proximal policy optimization algorithm. The expert knowledge learning mechanism refers to fitting the mapping relationship between the agent's state and actions through supervised learning based on an expert dataset. The phased training mechanism refers to using a preset number of training rounds as the phase switching condition. When the preset number of training rounds is reached, the parameters of the agent are automatically adjusted. When the training effect of the agent is not good, the expert knowledge learning mechanism is triggered.

[0009] Intelligent control of frequency in clean energy bases is achieved based on an improved near-end policy optimization algorithm.

[0010] Preferably, the primary frequency regulation model includes: the primary frequency regulation process of the new energy power station and its supporting energy storage adopts a comprehensive inertial control strategy of droop control and virtual inertia control.

[0011] ;

[0012] In the formula, This is a frequency modulation command; and These are the control coefficients for virtual inertia and droop control coefficients, respectively. This represents the per-unit value of the system frequency variation.

[0013] Preferably, the frequency regulation model of the energy storage power station includes a frequency regulation model of the compressed air energy storage system and a frequency regulation model of the electrochemical energy storage power station; the frequency regulation model of the compressed air energy storage system is as follows:

[0014] ; ;

[0015] ;

[0016] In the formula, For the first Shaft power of the primary expander; The specific heat at constant pressure of air; The air mass flow rate of the expander; For the first The inlet gas temperature of the primary expander; For the first The outlet gas temperature of the primary expander; For the first The expansion ratio of a multistage expander; Specific heat ratio; It is isentropic efficiency; Power command for compressed air energy storage system; For the flow control proportional element parameters of the compressed air energy storage system; For the flow control integral parameters of the compressed air energy storage system; and These are two relevant parameters in the turbine gas emission time delay stage;

[0017] The frequency regulation model of the electrochemical energy storage power station is as follows:

[0018] ; ;

[0019] In the formula, The state of charge of energy storage; This represents the initial state of charge of the stored energy. and These represent the minimum and maximum states of charge for energy storage, respectively. This refers to the actual power of the energy storage. The charging and discharging efficiency of energy storage.

[0020] Preferably, the automatic power generation control model uses PI control to obtain the total regulated power and distributes the total regulated power to each unit according to the adjustable capacity ratio;

[0021] The input to the PI control is the regional control error, and the formula for calculating the regional control error is as follows:

[0022] ; ;

[0023] In the formula, This refers to the regional control error; This is the deviation factor for the control area; The current system frequency; The system reference frequency; These are the composite frequency response characteristic coefficients; The equivalent rate of decrease in the system; Let be the load damping constant; For the unit The droop control coefficient;

[0024] The adjustable capacity ratio is:

[0025] ;

[0026] In the formula, Adjustable capacity ratio; and The first one within the control period The unit's capacity is adjustable upwards and downwards.

[0027] Preferably, the process of the near-end strategy optimization algorithm is as follows:

[0028] The agent in the near-side policy optimization algorithm uses the current policy. Interact with the environment, collect multiple experiences, and then end the interaction; the environment is a clean energy base;

[0029] The generalized dominance estimate is calculated using the following formula:

[0030] ; ;

[0031] In the formula, For generalized advantage estimation; For discount parameters; As a compromise parameter; For time step The temporal difference error is the difference between the agent's value estimate of the current state and the value estimate of the next state based on the actual immediate reward and the next state. The time step at the end of the round; For time step The timing difference error; It is a value function; For time step Instant rewards; For time step The state space;

[0032] Update the Actor and Critic networks: The collected and calculated trajectory data is randomly divided into multiple small batches, and the Actor and Critic networks are updated in multiple rounds using Adam optimization.

[0033] The loss function used by the Actor network during the update process is shown in the following equation:

[0034] ;

[0035] ;

[0036] ;

[0037] In the formula, and It is a constant; This is the clipping function; Importance sampling ratio; For policy entropy; Let be the dimension of the action space; For A policy network with parameters; and They are time steps Actions and states; According to the strategy In state Select action The probability of; For the first Dimensional action;

[0038] The loss function used by the Critic network during the update process is shown in the following formula:

[0039] ; ;

[0040] In the formula, is the loss function of the Critic network; The state value estimated by the current Critic network; The target value for the state value; To estimate the value of the Critic network using the old Critic network before updating the Critic network. The state value under the given state;

[0041] Repeat the above steps until the near-end policy optimization algorithm converges.

[0042] Preferably, the time step Instant rewards for:

[0043] ;

[0044] In the formula, The reward function is related to the regional control error; Output a reward function related to the action for the agent; A reward function related to frequency modulation mileage; For time step Regional control error; and They are time steps and time step Control actions; For the first Each frequency modulation unit at time step Frequency modulation mileage; , and They are respectively , , The corresponding weighting factor;

[0045] The time step state space for: ;

[0046] In the formula, For time step The vector of output changes of each unit within the system; For time step The vector of power command changes for each unit within the system;

[0047] Action space for: ;

[0048] In the formula, Output power commands from the intelligent agent to the thermal power unit; The power command output by the intelligent agent to the compressed air energy storage system; The power command is output from the intelligent agent to the hydroelectric generator; Output power commands to the wind farm for the intelligent agent.

[0049] Preferably, the first Each frequency modulation unit at time step FM mileage for:

[0050] ;

[0051] ;

[0052] ; ;

[0053] ;

[0054] In the formula, The overall frequency modulation performance score; The price is the subsidy for FM mileage per unit. For the first Each frequency modulation unit at time step ; output power; For the first Each frequency modulation unit at time step ; output power; To adjust the weights corresponding to the delay; Weights corresponding to response delays; To adjust the weights corresponding to the precision; For the first The adjustment rate fraction of each frequency modulation unit; For the first The maximum ramp rate of each frequency modulation unit; To control the average ramp rate of all frequency modulation units within the control area; For the first The response delay score of each frequency modulation unit; For the first The response delay time of each frequency modulation unit; For the first The frequency modulation accuracy score of each frequency modulation unit; The number of samples; For the first At the start of the first control interval Power commands issued by each frequency modulation unit; For the first At the start of the first control interval The actual regulating power of each frequency modulation unit; For the first The allowable adjustment error of each frequency modulation unit.

[0055] Preferably, a multi-swarm particle swarm optimization algorithm is used to optimize the PI parameters, and the output actions of the PI controller are obtained as an expert dataset. The fitness function of the multi-swarm particle swarm optimization algorithm is:

[0056] ;

[0057] In the formula, For adaptive value functions; and It is a constant; The current system frequency; The system reference frequency; For time steps; To control the performance adjustment coefficient; This refers to the regional control error; This is the economic performance adjustment coefficient; This represents the simulation time.

[0058] Preferably, the supervised learning includes:

[0059] The training network is organized in a two-level structure of training rounds and mini-batch data, using the loss function shown in the following formula. In each training round, the mini-batch data is randomly divided into training set and test set. The Actor network is updated iteratively using the training set, and the loss in each round is recorded. Training is terminated early when the loss in 5 consecutive rounds does not exceed the improvement threshold.

[0060] ;

[0061] In the formula, The loss function; For small batch sizes; The input state for the Actor network is Output action at time; The output action for the expert dataset; The L2 regularization coefficient; These are the parameters of the policy network.

[0062] Secondly, the present invention provides a frequency control device for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm. This device is used to execute the method described above, and the device includes:

[0063] The frequency control model acquisition module is used to construct frequency control models covering clean energy bases such as hydropower stations, wind farms, photovoltaic power stations, and compressed air energy storage. The frequency control models include generator and load models, primary frequency regulation models, frequency regulation models of conventional power sources, frequency regulation models of new energy sources, frequency regulation models of energy storage power stations, and automatic generation control models.

[0064] An improved PPO algorithm acquisition module is used to construct an improved proximal policy optimization algorithm by introducing an expert knowledge learning mechanism and a phased training mechanism on the basis of the proximal policy optimization algorithm. The expert knowledge learning mechanism refers to fitting the mapping relationship between the agent's state and action through supervised learning based on an expert dataset. The phased training mechanism refers to using a preset number of training rounds as a phase switching condition. When the preset number of training rounds is reached, the parameters of the agent are automatically adjusted. When the training effect of the agent is not good, the expert knowledge learning mechanism is triggered.

[0065] The frequency control strategy acquisition module is used to intelligently control the frequency of clean energy bases based on an improved near-end strategy optimization algorithm.

[0066] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0067] This invention discloses a frequency control method and device for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm. The constructed frequency control model can take power disturbances as input and output frequency changes, serving as a training and verification environment for the intelligent agent. Furthermore, based on the proximal policy optimization algorithm, an expert knowledge learning mechanism is introduced, and a phased training mechanism is introduced according to the complexity of the task to construct an improved proximal policy optimization algorithm. The phased training strategy enables a single training process to cover multiple operating conditions, and combined with expert guidance, it meets diverse control requirements, thereby improving convergence efficiency and policy performance, and ultimately achieving intelligent control of the frequency of clean energy bases. Attached Figure Description

[0068] Figure 1 This is a flowchart of the frequency control method for a hydro-wind-solar-pressure storage system based on the improved PPO algorithm proposed in this invention.

[0069] Figure 2 This is a structural block diagram of the frequency control device for a hydro-wind-solar-pressure storage system based on the improved PPO algorithm proposed in this invention.

[0070] Figure 3This is a structural block diagram of the frequency control device for a hydro-wind-solar-pressure storage system based on the improved PPO algorithm proposed in this invention.

[0071] Figure 4 This is a training framework diagram of the improved PPO algorithm proposed in this invention.

[0072] Figure 5 This is the network structure diagram of the strategy proposed in this invention.

[0073] Figure 6 This is the evaluation network structure diagram proposed in this invention.

[0074] Figure 7 This is a schematic diagram of the PPO reward curve under a weak perturbation scenario in Embodiment 1 of the present invention.

[0075] Figure 8 This is a schematic diagram of the PPO reward curve under a strong disturbance scenario in Embodiment 1 of the present invention.

[0076] Figure 9 This is a schematic diagram of the phased parameter adjustment process of the improved PPO algorithm in Embodiment 1 of the present invention.

[0077] Figure 10 This is a schematic diagram of the training process of the improved PPO algorithm in Embodiment 1 of the present invention.

[0078] Figure 11 This is a schematic diagram of policy performance evaluation during the training process of the improved PPO algorithm in Embodiment 1 of the present invention.

[0079] Figure 12 This is a schematic diagram of a weak disturbance scenario in Embodiment 2 of the present invention.

[0080] Figure 13 This is a schematic diagram of a strong disturbance scenario in Embodiment 2 of the present invention.

[0081] Figure 14 This is a schematic diagram of the frequency deviation of the three strategies under weak perturbation in Embodiment 2 of the present invention.

[0082] Figure 15 This is a schematic diagram of the frequency deviation of the three strategies under strong disturbance in Embodiment 2 of the present invention.

[0083] Figure 16 This is a radar chart of the performance indicators of the three strategies under weak disturbances in Embodiment 2 of the present invention.

[0084] Figure 17 This is a radar chart of the performance indicators of three strategies under strong disturbances in Embodiment 2 of the present invention.

[0085] Figure 18 This is a schematic diagram of frequency modulation mileage under weak disturbance in Embodiment 3 of the present invention.

[0086] Figure 19 This is a schematic diagram of the frequency modulation mileage under strong disturbance in Embodiment 3 of the present invention. Detailed Implementation

[0087] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0088] See Figure 1 This invention provides a frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm, comprising:

[0089] S1. Construct a frequency control model for a clean energy base encompassing hydropower stations, wind farms, photovoltaic power stations, and compressed air energy storage. The frequency control model includes a generator and load model, a primary frequency regulation model, a frequency regulation model for conventional power sources, a frequency regulation model for new energy sources, a frequency regulation model for energy storage power stations, and an automatic generation control model.

[0090] S2. Based on the proximal policy optimization algorithm, an expert knowledge learning mechanism and a phased training mechanism are introduced to construct an improved proximal policy optimization algorithm. The expert knowledge learning mechanism refers to fitting the mapping relationship between the agent's state and actions through supervised learning based on an expert dataset. The phased training mechanism refers to using a preset number of training rounds as the phase switching condition. When the preset number of training rounds is reached, the parameters of the agent are automatically adjusted. When the training effect of the agent is not good, the expert knowledge learning mechanism is triggered.

[0091] S3. Intelligent control of frequency in clean energy bases based on improved near-end strategy optimization algorithm.

[0092] This invention relates to frequency regulation and intelligent control technology for power systems supported by clean energy bases with multiple resources. To fully consider the optimization of the entire process of Automatic Generation Control (AGC) and improve the exploratory nature and stability of DRL algorithm training, and to achieve intelligent frequency control of clean energy bases using hydro-wind-solar-pressure-storage systems, this invention constructs an intelligent frequency control strategy for hydro-wind-solar-pressure-storage systems that integrates a stepwise expert-teaching PPO algorithm (SETP). Specifically, based on the proximity policy optimization (PPO) algorithm, an expert knowledge learning mechanism is introduced, and a stepwise training mechanism is introduced according to the task complexity to construct an improved proximity policy optimization algorithm. This algorithm adopts a stepwise training strategy according to the task complexity, enabling a single training process to cover multiple operating conditions, and combines expert guidance to meet diverse control needs, thereby improving convergence efficiency and strategy performance, and ultimately achieving intelligent frequency control of the clean energy base.

[0093] Furthermore, the frequency control model (LFC) is as follows:

[0094] (1) Generator and load model

[0095] Since this invention deals with a small disturbance scenario, the first-order generator rotor motion equation and the first-order load model are considered:

[0096] ;

[0097] In the formula, The inertial time constant; This refers to the rotor's mechanical angular velocity; This refers to the minute deviation of the rotor's mechanical angular velocity; This refers to a small deviation in mechanical torque; This represents a small deviation in electromagnetic torque; Let be the load damping constant; For a small deviation in load; This represents the system frequency deviation.

[0098] Obtained by Laplace transform The domain model is: ;

[0099] In the formula, This represents the per-unit value of the system frequency variation; This is the per-unit value of the system frequency; This is the system frequency reference value; For the complex frequency variable in the Laplace transform; This represents the per-unit value of a small deviation in mechanical torque; This is the per-unit value of a small deviation in electromagnetic torque.

[0100] The simulation environment of this invention is a single-area clean energy base, in which thermal power units, hydropower units, and compressed air energy storage have mechanical inertia; the equivalent inertial time constant of this area. The calculation is shown in the following formula:

[0101] ;

[0102] In the formula, For the unit The inertial time constant; For the unit Rated capacity.

[0103] (2) Primary frequency modulation model

[0104] 1) Speed ​​controller with speed-decline characteristics

[0105] The simulation environment of this invention has multiple power supplies, therefore a speed controller with speed reduction characteristics is used. The domain model is:

[0106] ;

[0107] In the formula, Per-unit value for valve position; The rate of decline; The time constant of the speed controller, This indicates the valve position.

[0108] 2) Governor of the water turbine

[0109] Considering the effects of water hammer, the governor of the turbine has a transient sag compensation circuit. The domain model is:

[0110] ;

[0111] In the formula, This is the per-unit value for the guide vane position; This refers to the droop control coefficient for hydropower stations. , , For model parameters, the present invention uses typical values.

[0112] 3) Integrated inertial control strategy

[0113] The clean energy base studied in this invention includes new energy power plants and their associated energy storage facilities. These are connected to the grid via converters and are decoupled from the grid side, therefore they cannot actively provide inertial support to the grid. This invention considers a comprehensive inertial control strategy combining droop control and virtual inertial control for its primary frequency regulation process.

[0114] ;

[0115] In the formula, This is a frequency modulation command; and These are the control coefficients for virtual inertia and droop control coefficients, respectively. This represents the per-unit value of the system frequency variation.

[0116] (3) Frequency modulation model of conventional power supply

[0117] 1) Water turbine

[0118] water turbine Domain model: ;

[0119] In the formula, This refers to the per-unit value of the mechanical power of the water turbine. For model parameters, the present invention uses typical values.

[0120] 2) Steam turbine

[0121] This invention selects a reheat steam turbine, which The domain model is:

[0122] ;

[0123] In the formula, This refers to the per-unit value of the mechanical power of the steam turbine in a thermal power unit. , , For model parameters, the present invention uses typical values; This refers to the per-unit value for the valve position of the thermal power unit.

[0124] (4) Frequency modulation model of new energy

[0125] The converters in photovoltaic power plants and wind farms have relatively fast response speeds, and their power response characteristics are considered to be first-order inertial elements.

[0126] ;

[0127] In the formula, and These are the actual power outputs of wind farms and photovoltaic power plants, respectively. and These are power commands for wind farms and photovoltaic power plants, respectively. This is the converter control time constant.

[0128] In addition, considering the safe and stable operation of wind farms and photovoltaic power stations, and assuming that the power output of the stations is less than the maximum output of wind and solar power, the following safety constraints should be applied, assuming that the prediction error of wind and solar power output is 10%: ;

[0129] In the formula, and These represent the predicted power output of wind farms and photovoltaic power plants during the control period, respectively.

[0130] (5) Frequency regulation model of energy storage power station

[0131] 1) Frequency regulation model of compressed air energy storage system

[0132] This invention considers the compressed air energy storage system to participate in system frequency regulation under expansion power generation conditions. Since the control period is short, it is assumed that the pressure of the gas storage tank and the temperature of the heat storage tank do not change much. Therefore, the influence of the heat storage tank, heat exchange system and gas storage tank is not considered.

[0133] No. The output power and outlet temperature of the multistage expander are:

[0134] ; ;

[0135] In the formula, For the first The outlet gas temperature of the primary expander; For the first The inlet gas temperature of the primary expander; For the first The expansion ratio of a multistage expander; Specific heat ratio, also known as adiabatic index. and These are the specific heat at constant pressure and the specific heat at constant volume of air, respectively. The air mass flow rate of the expander; For the first Shaft power of the primary expander; It is isentropic efficiency.

[0136] Compressed air energy storage system through control The power of the compressor and expander is adjusted in response to power grid commands; its airflow control system... The domain model is: ;

[0137] In the formula, Power command for compressed air energy storage system; For the flow control proportional element parameters of the compressed air energy storage system; For the flow control integral parameters of the compressed air energy storage system; and These are two relevant parameters related to the turbine gas emission time delay stage.

[0138] 2) Frequency regulation model of electrochemical energy storage power station

[0139] The converter model of the electrochemical energy storage power station is similar to that of new energy sources. Its state of charge is calculated by the following formula and must meet the constraints shown in the following formula to avoid overcharging and over-discharging.

[0140] ; ;

[0141] In the formula, The state of charge of energy storage; This represents the initial state of charge of the stored energy. and These represent the minimum and maximum states of charge for energy storage, respectively. This refers to the actual power of the energy storage. The charging and discharging efficiency of energy storage.

[0142] (6) Automatic generation control model

[0143] AGC uses PI control to obtain the total regulating power. The input of PI control is the regional control error. Then, the total regulating power is allocated to each unit according to rules such as "according to the adjustable capacity ratio".

[0144] 1) Area control error

[0145] The simulation environment of this invention is a single area; therefore, the AGC control mode adopts constant frequency control, and the formula for calculating the area control error is as follows: ;

[0146] In the formula, For the area control error, when This indicates that the region is generating too much electricity. This is the deviation factor for the control area; The current system frequency; This is the system reference frequency.

[0147] Specifically, The value is usually determined based on the composite frequency response coefficient. Select:

[0148] ;

[0149] In the formula, The system's equivalent degradation rate is expressed in Hz / MW. Let be the load damping constant; For the unit The droop control coefficient.

[0150] 2) Power Allocation Rules

[0151] Regarding the power allocation rules, this invention allocates power according to a fixed ratio based on the adjustable capacity of each unit:

[0152] ;

[0153] In the formula, Adjustable capacity ratio; and The first one within the control period The unit's capacity is adjustable upwards and downwards.

[0154] The aforementioned frequency control model can take power disturbances as input and output frequency changes, serving as a training and verification environment for the intelligent agent. Specifically, the primary frequency regulation model characterizes the spontaneous power changes of each power source when the system frequency is disturbed; the individual power source frequency regulation model characterizes the power regulation characteristics of each power source upon receiving power commands; and the automatic generation control model is responsible for actively issuing power commands to each unit based on system frequency changes.

[0155] Furthermore, the DRL-based AGC strategy adopts an offline training and online application paradigm, utilizing neural networks to represent the control strategy, thereby improving the adaptability and generalization ability of frequency control. In this paradigm, the agent replaces the control and allocation links of traditional AGC, interacting with the environment to collect experience to complete offline learning and policy evaluation. This invention constructs a training framework based on the Proximal Policy Optimization (PPO) algorithm, and combines expert knowledge learning and a phased training mechanism to improve convergence efficiency and policy performance.

[0156] For details, see Figure 4 The SETP algorithm framework includes the PPO algorithm training process, reinforcement learning environment design, high-quality action dataset generation method, expert knowledge integration learning mechanism, and phased training mechanism, as detailed below:

[0157] (1) Training process of PPO algorithm. The main feature of PPO algorithm is that it introduces a pruning factor on the basis of policy gradient algorithm to limit the policy update magnitude. Its application process is as follows:

[0158] 1) The PPO agent uses the current policy. The system interacts with the environment, collects multiple experiences, and then terminates the interaction; the environment of this invention is a clean energy base.

[0159] 2) Calculate the generalized advantage estimate (GAE) according to the following formula: ;

[0160] In the formula, For generalized advantage estimation; For discount parameters; As a compromise parameter; For time step The temporal difference error is the difference between the agent's value estimate of the current state and the value estimate of the next state based on the actual immediate reward and the next state. This is the time step at the end of the round.

[0161] Time step Timing difference error for: ;

[0162] In the formula, It is a value function; For time step Instant rewards; For time step The state space.

[0163] 3) Update the Actor network and Critic network:

[0164] The trajectory data obtained from steps 1) and 2) are usually... The network is randomly divided into multiple mini-batches, and the Actor and Critic networks are updated multiple times using Adam optimization. The parameter name for the number of empirical batches is "mini-batch size," and the parameter name for the number of update epochs is "number of epochs." The loss function used by the Actor network during the update process is shown in the following equation:

[0165] ;

[0166] In the formula, and It is a constant; This is the clipping function; Importance sampling ratio; Let be the policy entropy.

[0167] ;

[0168] In the formula, Let be the dimension of the action space; For A policy network with parameters; and They are time steps Actions and states; The meaning is based on strategy In state Select action The probability of.

[0169] ;

[0170] In the formula, For the first Dimensional action.

[0171] The loss function used by the Critic network during the update process is shown in the following formula:

[0172] ; ;

[0173] In the formula, is the loss function of the Critic network; The state value estimated by the current Critic network; The target value for the state value; To estimate the value of the Critic network using the old Critic network before updating the Critic network. The state value under the given state.

[0174] 4) Repeat the above steps until the near-end policy optimization algorithm converges.

[0175] (2) Enhance the design of the learning environment

[0176] 1) Environment and reward function

[0177] The environment is primarily responsible for determining the next state based on the current state and actions; this effect can be achieved through a frequency control model for clean energy bases. Additionally, the environment should provide reward feedback to the agent based on a reward function. This invention primarily considers the control performance of the intelligent power generation control strategy, as well as its economic efficiency; therefore, the following reward function is adopted:

[0178] ;

[0179] In the formula, For time step Instant rewards; The reward function is related to the regional control error and is mainly used to adjust the control performance of the agent. Output a reward function related to the action for the agent; A reward function related to frequency modulation mileage; For time step Regional control error; and They are time steps and time step Control actions; For the first Each frequency modulation unit at time step Frequency modulation mileage; , and They are respectively , , The corresponding weighting factor.

[0180] The formula used for frequency modulation mileage is: ;

[0181] In the formula, The price is the subsidy for FM mileage per unit. For the first Each frequency modulation unit at time step ; output power; For the first Each frequency modulation unit at time step . output power.

[0182] Overall FM performance score Related to the dynamic adjustment performance of the AGC frequency modulation unit, a fast-response AGC frequency modulation unit has a higher overall frequency modulation performance score. Specifically, It is considered comprehensively based on the following three different indicators:

[0183] ①Climbing rate score: mainly considers the climbing rate of the frequency modulation unit.

[0184] ;

[0185] In the formula, For the first The adjustment rate fraction of each frequency modulation unit; For the first The maximum ramp rate of each AGC frequency modulation unit; To control the average ramp rate of all AGC frequency modulation units within the control area.

[0186] ② Response delay score: This mainly considers the response delay time of the frequency modulation unit.

[0187] ;

[0188] In the formula, For the first The response delay score of each AGC frequency modulation unit; For the first The response delay time of each AGC frequency modulation unit.

[0189] ③ Frequency modulation accuracy score: Frequency modulation accuracy is mainly measured by the deviation between the power command and the actual output power.

[0190] ;

[0191] In the formula, For the first Frequency modulation accuracy score of each AGC frequency modulation unit; The number of samples; For the first At the start of the first control interval Power commands issued by each AGC frequency modulation unit; For the first At the start of the first control interval The actual regulating power of each AGC frequency modulation unit; For the first The allowable adjustment error for each AGC frequency modulation unit is 1.5% of its rated power.

[0192] The weighted average of the three scores mentioned above is: : In the formula, To adjust the weights corresponding to the delay; Weights corresponding to response delays; The weights are adjusted to correspond to the precision.

[0193] 2) State Space

[0194] Traditional automatic power generation control has a control cycle of 4-6 seconds. In this invention, the intelligent agent collects state information from the environment every 5 seconds. The state space of the PPO or SAC intelligent agent... It is set to the current output status of each power source or energy storage, the state of charge of each energy storage, the area control error, and the system frequency deviation.

[0195] ;

[0196] In the formula, For time step The vector of output changes of each unit within the system; For time step The vector of power command changes for each unit within the system.

[0197] 3) Action Space

[0198] The action space of an intelligent agent is the power commands of each power source and energy storage.

[0199] ;

[0200] In the formula, Output power commands from the intelligent agent to the thermal power unit; The power command output by the intelligent agent to the compressed air energy storage system; The power command is output from the intelligent agent to the hydroelectric generator; Output power commands to the wind farm for the intelligent agent.

[0201] 4) Actor and Critic Networks

[0202] See Figure 5 The Actor network takes a 13-dimensional state vector as input and shares a common part consisting of two fully connected layers (256 neurons each) and a ReLU activation function. After shared feature extraction, the network splits into a mean branch and a standard deviation branch. The mean branch outputs the action mean through a fully connected layer, while the standard deviation branch outputs the action standard deviation through a fully connected layer, a Softplus activation function, and a Clipping Layer (used to control the size of the standard deviation), thus constructing a Gaussian policy distribution. The final sampled actions must pass through a safety layer, which is a linear mapping.

[0203] ;

[0204] In the formula, For the output of the security layer; This refers to the normalized action vector sampled from the Gaussian policy; and These are the lower and upper bounds of the action space at the element level, respectively.

[0205] See Figure 6 The Critic network structure is similar to the shared part of the Actor network. It also takes a 13-dimensional state vector as input, passes through two fully connected layers (256 neurons in each layer) and the ReLU activation function, and finally outputs the state value estimate through a fully connected layer.

[0206] (3) Methods for generating high-quality motion datasets

[0207] The selection of PI parameters has a significant impact on AGC performance. To collect expert datasets and enhance the persuasiveness of comparative examples, this invention employs a multi-swarm particle swarm optimization algorithm to optimize the PI parameters, obtaining the output actions of the PI controller as the expert dataset. This algorithm has stronger global search capabilities compared to particle swarm optimization, effectively preventing the algorithm from getting trapped in local optima. The fitness function in this algorithm considers the integral of the frequency offset over time during the AGC control process; and to avoid excessively large controller output commands that could cause unnecessary system oscillations, the fitness function also considers the integral of the controller output signal over time. The fitness function of the multi-swarm particle swarm optimization algorithm is as follows:

[0208] ;

[0209] In the formula, For adaptive value functions; and It is a constant; The current system frequency; The system reference frequency; For time steps; To control the performance adjustment coefficient; This refers to the regional control error; This is the economic performance adjustment coefficient; This represents the simulation time.

[0210] (4) Integrating expert knowledge learning mechanisms

[0211] Based on expert datasets, the mapping relationship between states and actions is fitted through supervised learning. The supervised learning process is organized in a two-level structure of training epochs and mini-batch, using the loss function shown in the following formula. In each training epoch, the mini-batch data is randomly divided into training and test sets. The Actor network is updated iteratively using the training set, and the loss in each epoch is recorded. To improve training efficiency and avoid overfitting, an early stopping mechanism is introduced: training is terminated early when the loss fails to improve significantly for 5 consecutive epochs (the improvement threshold is set to 10).

[0212] ;

[0213] In the formula, The loss function; For small batch sizes; The input state for the Actor network is Output action at time; The output action for the expert dataset; The L2 regularization coefficient; These are the parameters of the policy network.

[0214] Specifically, a multi-swarm particle swarm optimization algorithm is used to optimize the PI parameters in multiple scenarios to obtain the actions under the optimal PI controller, which serve as the expert dataset. When the agent training effect is poor, the mean squared error between the Actor output and the expert output under the same input conditions is used as the loss function to update the Actor network.

[0215] (5) Phased training mechanism

[0216] In the phased mechanism of this invention, a fixed number of training rounds is used as the phase switching condition. Once the preset number of rounds is reached, parameter adjustment and expert knowledge learning mechanisms are automatically triggered. Specifically, at the end of each phase, key parameters such as the perturbation standard deviation and learning rate are updated based on a preset strategy or expert experience. Simultaneously, supervised learning is conducted using expert datasets as needed, allowing the agent to continue training under new parameter settings and more complex perturbation environments. Through this cyclical mechanism of "fixed number of rounds - parameter adjustment - expert knowledge learning," the agent can gradually transition from simple to complex operating conditions, balancing convergence speed and generalization performance throughout the training process.

[0217] This invention addresses the frequency control needs of clean energy bases integrating water, wind, solar, and compressed air energy storage. It proposes an intelligent frequency control strategy for clean energy bases based on a phased PPO (Proximity-Oriented Optimization) approach incorporating expert knowledge. The strategy integrates the PPO algorithm (Proximity-Oriented Optimization algorithm), a phased training strategy, and an expert guidance mechanism into the SETP (Improved Proximity-Oriented Optimization) framework. By improving the PPO algorithm to replace automatic generation control, power commands are issued to each unit. The control performance and frequency regulation mileage economy were evaluated under both weak and strong disturbance scenarios, leading to the following conclusions:

[0218] 1) Compared with the PPO algorithm, SETP adopts a phased training strategy from easy to difficult, which enables a single training process to cover multiple operating conditions and meets diverse control requirements with expert guidance, while significantly improving training stability.

[0219] 2) In terms of control performance, SETP exhibits faster frequency recovery, smaller overshoot, and better steady-state performance under various disturbance conditions. In weak disturbance scenarios, the IAE is reduced by approximately 22% compared to fixed PI control; in strong disturbance scenarios, it is reduced by approximately 23%, and the average frequency recovery time and root mean square deviation are significantly better than the comparison method in both cases.

[0220] 3) In terms of frequency modulation mileage, SETP reduces the mileage by about 22% in weak disturbance scenarios and by about 22% in strong disturbance scenarios compared with traditional PI control, indicating that it has lower control action and better economy.

[0221] Example 1: Training Comparison between Standard PPO Algorithm and SETP Algorithm

[0222] Step A: Set the training scenario parameter range as shown in the table below.

[0223] Table 1 Training Scenario Parameter Settings

[0224]

[0225] Step B: Set the parameters for the PPO and SETP algorithms; design the strategy and evaluation network, such as... Figure 5 and Figure 6 As shown.

[0226] Step C: Comparison of the training processes of the PPO and SETP algorithms.

[0227] PPO was trained separately under weak and strong perturbation scenarios, and the corresponding learning curves are as follows: Figure 7 , Figure 8 As shown, where Figure 7 For scenarios with weak disturbances. Figure 8 This corresponds to scenarios with strong perturbations. In each graph, the round reward represents the total reward obtained in each round, and the average reward is its moving average, used to reflect the steady improvement of overall performance; the round action value characterizes the action value estimation, and the two curves highly overlap in the later stages, indicating that the policy training has basically converged. In both scenarios, the reward function only contains... and , omitted This is to avoid further increasing the training difficulty given the limited frequency adjustment performance. During training, It helps suppress frequent motion oscillations in the early stages, but inhibits them in the later stages. The improvement.

[0228] In contrast, the SETP proposed in this invention employs a phased training strategy, gradually transitioning the perturbation standard deviation from weak to strong perturbations, enabling a single training iteration to cover a wider range of perturbation conditions. The reward function coefficients are dynamically adjusted at each stage, taking into account various factors. , , Then, when the training converges, The coefficient drops to zero, such as Figure 9 As shown. The effectiveness of this strategy is reflected in the training curve: as... Figure 10 As shown, the expert-guided loss continued to decrease during training; although the reward curve changed little in the later stages, the strategy remained stable despite the continuous changes in environmental difficulty and reward coefficient, demonstrating strong adaptability. Figure 11 This further indicates that the frequency deviation integral continuously decreases throughout the training process, suggesting that the frequency deviation gradually decreases; while the frequency tuning mileage increases with the improvement of tuning performance and is further optimized after the performance stabilizes.

[0229] Example 2: Performance of the SETP control strategy under large and small disturbances

[0230] Step A: Set up specific strong and weak disturbance scenarios, such as Figure 12 , Figure 13 As shown.

[0231] Step B: Performance analysis of the SETP strategy.

[0232] In weak disturbance scenarios, the disturbance signal is as follows: Figure 12 As shown. By Figure 14 As can be seen, the proposed SETP can achieve rapid frequency recovery after each perturbation, with a smooth dynamic process and small overshoot. It can also enter and stabilize within the ±0.05Hz range in a short time, demonstrating superior dynamic response and steady-state performance. In contrast, although PPO has converged during training, its recovery speed after perturbation is slow, and it exhibits significant overshoot and undershoot, indicating insufficient stability in its dynamic adjustment process. The traditional fixed PI exhibits a larger steady-state deviation and continuous oscillation, with overall adjustment performance significantly inferior to SETP.

[0233] The quantitative results in Tables 2 and 3 further validate the advantages of SETP in terms of dynamic adjustment accuracy and steady-state stability, consistent with the results of the frequency deviation curve analysis. Specifically, SETP achieves the lowest values ​​in both IAE and ITAE indices, indicating the smallest cumulative frequency deviation throughout the entire range and within the weighted disturbance window; the lowest RMS deviation reflects the smallest overall fluctuation amplitude; the shortest average settling time indicates the fastest recovery speed; and the lowest steady-state standard deviation reflects optimal steady-state performance. Figure 16The radar chart shows that the envelope curve of SETP wraps around PPO and fixed PI for most indicators, intuitively demonstrating its advantages.

[0234] Table 2. Quantitative Results of Policy Performance Indicators in Weak Perturbation Scenarios

[0235]

[0236] In scenarios with strong disturbances, the disturbance signal is as follows: Figure 13 As shown. By Figure 15 As can be seen, SETP retains the aforementioned advantages, showing significant improvements over PPO and fixed PI in recovery speed, dynamic process smoothness, and steady-state bias. The following table shows the quantification results and... Figure 17 The radar charts all indicate that SETP remains robust under more demanding operating conditions.

[0237] Table 3. Quantitative Results of Policy Performance Indicators under Strong Perturbation Scenarios

[0238]

[0239] In Tables 2 and 3, SETP represents the improved PPO algorithm; PPO represents the standard PPO algorithm; Fixed PI represents the fixed PI parameter control strategy; IAE represents the absolute value integral of frequency deviation; ITAE represents the time-weighted integral of the absolute value of frequency deviation; RMS represents the root mean square of frequency deviation; Settle Time represents the settling time; and Steady Std represents the steady-state output standard deviation.

[0240] In summary, SETP achieves faster frequency recovery, smaller oscillation amplitude, and better steady-state stability under both weak and strong disturbance conditions, verifying its adaptability and effectiveness under different operating conditions.

[0241] Example 3: Performance Analysis of SETP under Multiple Control Requirements

[0242] Step A: Set scene parameters, same as in Example 2. Step B: Result analysis.

[0243] Depend on Figure 18 , Figure 19As can be seen, under both weak and strong disturbance scenarios, the cumulative frequency modulation mileage of SETP is consistently significantly lower than that of fixed PI control. Under weak disturbance conditions, SETP reduces the frequency modulation mileage by approximately 22% compared to fixed PI control, and the curve remains at a lower level throughout the entire process, indicating that it effectively reduces the frequency and amplitude of adjustment actions while achieving rapid frequency recovery and steady-state maintenance. Under strong disturbance conditions, even with a significant increase in overall adjustment demand, SETP still reduces the frequency modulation mileage by approximately 22%, demonstrating significant economic advantages even under more demanding operating conditions. In summary, SETP not only outperforms traditional strategies in dynamic and steady-state performance but also effectively reduces unnecessary adjustment actions, thereby significantly reducing operating costs in the long term.

[0244] See Figure 2 The present invention also provides a frequency control device for a hydro-wind-solar-pressure-storage system based on an improved PPO algorithm. This device is used to execute the frequency control method for a hydro-wind-solar-pressure-storage system based on the improved PPO algorithm as described above. The device includes:

[0245] The frequency control model acquisition module is used to construct frequency control models covering clean energy bases such as hydropower stations, wind farms, photovoltaic power stations, and compressed air energy storage. The frequency control models include generator and load models, primary frequency regulation models, frequency regulation models of conventional power sources, frequency regulation models of new energy sources, frequency regulation models of energy storage power stations, and automatic generation control models.

[0246] An improved PPO algorithm acquisition module is used to construct an improved proximal policy optimization algorithm by introducing an expert knowledge learning mechanism and a phased training mechanism on the basis of the proximal policy optimization algorithm. The expert knowledge learning mechanism refers to fitting the mapping relationship between the agent's state and action through supervised learning based on an expert dataset. The phased training mechanism refers to using a preset number of training rounds as a phase switching condition. When the preset number of training rounds is reached, the parameters of the agent are automatically adjusted. When the training effect of the agent is not good, the expert knowledge learning mechanism is triggered.

[0247] The frequency control strategy acquisition module is used to intelligently control the frequency of clean energy bases based on an improved near-end strategy optimization algorithm.

[0248] See Figure 3 The present invention also provides a frequency control device for a hydro-wind-solar-pressure-storage system based on an improved PPO algorithm, comprising a memory and a processor; the memory is used to store computer program code and transmit the computer program code to the processor; the processor is used to execute the frequency control method for a hydro-wind-solar-pressure-storage system based on the improved PPO algorithm as described above, according to the instructions in the computer program code.

[0249] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the frequency control method for a hydro-wind-solar-pressure storage system based on the improved PPO algorithm as described above.

[0250] For details regarding the aforementioned apparatus, equipment, and computer-readable storage media, please refer to the specific description of the frequency control method and beneficial effects of a hydro-wind-solar-pressure storage system based on the improved PPO algorithm, which will not be repeated here.

[0251] Although embodiments of the present invention have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm, characterized in that, include: A frequency control model is constructed for a clean energy base encompassing hydropower stations, wind farms, photovoltaic power stations, and compressed air energy storage. The frequency control model includes a generator and load model, a primary frequency regulation model, a frequency regulation model for conventional power sources, a frequency regulation model for new energy sources, a frequency regulation model for energy storage power stations, and an automatic generation control model. Based on the proximal policy optimization algorithm, an expert knowledge learning mechanism and a phased training mechanism are introduced to construct an improved proximal policy optimization algorithm. The expert knowledge learning mechanism refers to fitting the mapping relationship between the agent's state and actions through supervised learning based on an expert dataset. The phased training mechanism refers to using a preset number of training rounds as the phase switching condition. When the preset number of training rounds is reached, the parameters of the agent are automatically adjusted. When the training effect of the agent is not good, the expert knowledge learning mechanism is triggered. Intelligent control of frequency in clean energy bases is achieved based on an improved near-end policy optimization algorithm.

2. The frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 1, characterized in that, The primary frequency modulation model includes: The primary frequency regulation process of new energy power plants and their associated energy storage adopts a comprehensive inertial control strategy that combines droop control and virtual inertia control. ； In the formula, This is a frequency modulation command; and These are the control coefficients for virtual inertia and droop control coefficients, respectively. This represents the per-unit value of the system frequency variation.

3. The frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 1, characterized in that, The frequency regulation model of the energy storage power station includes the frequency regulation model of the compressed air energy storage system and the frequency regulation model of the electrochemical energy storage power station. The frequency regulation model of the compressed air energy storage system is as follows: ； ； ； In the formula, For the first Shaft power of the primary expander; The specific heat at constant pressure of air; The air mass flow rate of the expander; For the first The inlet gas temperature of the primary expander; For the first The outlet gas temperature of the primary expander; For the first The expansion ratio of a multistage expander; Specific heat ratio; It is isentropic efficiency; Power command for compressed air energy storage system; For the flow control proportional element parameters of the compressed air energy storage system; For the flow control integral parameters of the compressed air energy storage system; and These are two relevant parameters in the turbine gas emission time delay stage; The frequency regulation model of the electrochemical energy storage power station is as follows: ； ； In the formula, The state of charge of energy storage; This represents the initial state of charge of the stored energy. and These represent the minimum and maximum states of charge for energy storage, respectively. This refers to the actual power of the energy storage. The charging and discharging efficiency of energy storage.

4. The frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 1, characterized in that, The automatic power generation control model uses PI control to obtain the total regulated power and distributes the total regulated power to each unit according to the adjustable capacity ratio. The input to the PI control is the regional control error, and the formula for calculating the regional control error is as follows: ； ； In the formula, This refers to the regional control error; The deviation factor for the control area; The current system frequency; The system reference frequency; These are the composite frequency response characteristic coefficients; The equivalent rate of decrease in the system; Let be the load damping constant; For the unit The droop control coefficient; The adjustable capacity ratio is: ； In the formula, Adjustable capacity ratio; and The first one within the control period The unit's capacity is adjustable upwards and downwards.

5. The frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 1, characterized in that, The process of the near-end strategy optimization algorithm is as follows: The agent in the near-side policy optimization algorithm uses the current policy. Interact with the environment, collect multiple experiences, and then end the interaction; the environment is a clean energy base; The generalized dominance estimate is calculated using the following formula: ； ； In the formula, For generalized advantage estimation; For discount parameters; As a compromise parameter; For time steps The temporal difference error is the difference between the agent's value estimate of the current state and the value estimate of the next state based on the actual immediate reward and the next state. The time step at the end of the round; For time steps The timing difference error; Value function; For time steps Instant rewards; For time steps The state space; Update the Actor network and Critic network: The collected and calculated trajectory data is randomly divided into multiple small batches, and the Actor network and Critic network are updated in multiple rounds using Adam optimization. The loss function used by the Actor network during the update process is shown in the following equation: ； ； ； In the formula, and It is a constant; This is the clipping function; Importance sampling ratio; For policy entropy; Let be the dimension of the action space; For A policy network with parameters; and They are time steps Actions and states; According to the strategy In state Select action The probability of; For the first Dimensional action; The loss function used by the Critic network during the update process is shown in the following formula: ； ； In the formula, Let be the loss function of the Critic network; The state value estimated by the current Critic network; The target value for the state value; To estimate the value of the Critic network using the old Critic network before updating the Critic network. The state value under the given state; Repeat the above steps until the near-end policy optimization algorithm converges.

6. The frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 5, characterized in that, The time step Instant rewards for: ； In the formula, The reward function is related to the regional control error; Output a reward function related to the action for the agent; A reward function related to frequency modulation mileage; For time steps Regional control error; and They are time steps and time step Control actions; For the first Each frequency modulation unit at time step Frequency modulation mileage; , and They are respectively , , The corresponding weighting factor; The time step state space for: ； In the formula, For time steps The vector of output changes of each unit within the system; For time steps The vector of power command changes for each unit within the system; Action space for: ； In the formula, Output power commands from the intelligent agent to the thermal power unit; The power command output by the intelligent agent to the compressed air energy storage system; The power command is output from the intelligent agent to the hydroelectric generator; Output power commands to the wind farm for the intelligent agent.

7. A frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 6, characterized in that, The first Each frequency modulation unit at time step FM mileage for: ； ； ； ； ； In the formula, The overall frequency modulation performance score; The price is the subsidy for FM mileage per unit. For the first Each frequency modulation unit at time step ; output power; For the first Each frequency modulation unit at time step ; output power; To adjust the weights corresponding to the delay; Weights corresponding to response delays; To adjust the weights corresponding to the precision; For the first The adjustment rate fraction of each frequency modulation unit; For the first The maximum ramp rate of each frequency modulation unit; To control the average ramp rate of all frequency modulation units within the control area; For the first The response delay score of each frequency modulation unit; For the first The response delay time of each frequency modulation unit; For the first The frequency modulation accuracy score of each frequency modulation unit; The number of samples; For the first At the start of the first control interval Power commands issued by each frequency modulation unit; For the first At the start of the first control interval The actual regulating power of each frequency modulation unit; For the first The allowable adjustment error of each frequency modulation unit.

8. A frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 4, characterized in that, A multi-swarm particle swarm optimization algorithm is used to optimize the PI parameters, and the output actions of the PI controller are obtained as an expert dataset. The fitness function of the multi-swarm particle swarm optimization algorithm is: ； In the formula, For adaptive value functions; and It is a constant; The current system frequency; The system reference frequency; For time steps; To control the performance adjustment coefficient; This refers to the regional control error; This is the economic performance adjustment coefficient; This is the simulation time.

9. A frequency control method for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm according to claim 5, characterized in that, The supervised learning includes: The training network is organized in a two-level structure of training rounds and mini-batch data, using the loss function shown in the following formula. In each training round, the mini-batch data is randomly divided into training set and test set. The Actor network is updated iteratively using the training set, and the loss in each round is recorded. Training is terminated early when the loss in 5 consecutive rounds does not exceed the improvement threshold. ； In the formula, The loss function; For small batch sizes; The input state for the Actor network is Output action at time; The output action for the expert dataset; The L2 regularization coefficient; These are the parameters of the policy network.

10. A frequency control device for a hydro-wind-solar-pressure storage system based on an improved PPO algorithm, characterized in that, The apparatus is used to perform the method as described in any one of claims 1-9, the apparatus comprising: The frequency control model acquisition module is used to construct frequency control models covering clean energy bases such as hydropower stations, wind farms, photovoltaic power stations, and compressed air energy storage. The frequency control models include generator and load models, primary frequency regulation models, frequency regulation models of conventional power sources, frequency regulation models of new energy sources, frequency regulation models of energy storage power stations, and automatic generation control models. An improved PPO algorithm acquisition module is used to construct an improved proximal policy optimization algorithm by introducing an expert knowledge learning mechanism and a phased training mechanism on the basis of the proximal policy optimization algorithm. The expert knowledge learning mechanism refers to fitting the mapping relationship between the agent's state and action through supervised learning based on an expert dataset. The phased training mechanism refers to using a preset number of training rounds as a phase switching condition. When the preset number of training rounds is reached, the parameters of the agent are automatically adjusted. When the training effect of the agent is not good, the expert knowledge learning mechanism is triggered. The frequency control strategy acquisition module is used to intelligently control the frequency of clean energy bases based on an improved near-end strategy optimization algorithm.

Images