A new energy power generation system stability improvement method based on deep reinforcement learning
By using a deep reinforcement learning agent to adaptively adjust the control parameters of a new energy power generation system, the stability problem of the new energy power generation system during grid connection is solved, achieving rapid response and improved stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2025-02-18
- Publication Date
- 2026-06-19
AI Technical Summary
New energy power generation systems have stability problems such as subsynchronous oscillation or supersynchronous oscillation during grid connection. Existing control methods are difficult to adapt to complex and ever-changing operating conditions and large-scale time-varying oscillations. Moreover, they rely on oscillation component detection and mechanism understanding, resulting in slow response speed.
By employing a deep reinforcement learning-based approach, a reward function is designed based on real-time operational data. The deep reinforcement learning agent is then trained offline, and control parameters are adaptively adjusted to improve system stability.
It achieves fast-response adaptive control, improves the stability of new energy power generation systems, adapts to complex and ever-changing operating conditions, and avoids reliance on oscillation component detection and mechanism understanding.
Smart Images
Figure CN120127626B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of new energy power generation, and more specifically, to a method for improving the stability of new energy power generation systems based on deep reinforcement learning. Background Technology
[0002] With the increasing global demand for clean energy and the transformation of the energy structure, renewable energy power generation technology has developed rapidly. However, during the grid connection process of new energy power generation systems, the use of power electronic converters as grid connection interface devices can easily lead to stability problems such as subsynchronous or supersynchronous oscillations. These problems seriously affect the effective utilization of new energy and the safe and stable operation of the power grid. Currently, many scholars at home and abroad have proposed various stability control methods, such as active damping control, to address the grid connection stability problem of new energy power generation. However, most of these methods are based on fixed damping control structures and parameters, mainly targeting the suppression of oscillations under specific operating conditions and at specific frequency bands. Therefore, they are difficult to adapt to scenarios with complex and variable operating conditions and large-scale time-varying oscillation amplitude and frequency. In addition, in recent years, some scholars have proposed adaptive oscillation suppression strategies. These strategies suppress oscillations by adjusting damping parameters online, which are adjusted according to the detected oscillation components. However, this strategy is limited by the rapid detection time of the oscillation components after the oscillation occurs, which limits the effectiveness of oscillation suppression to some extent. Furthermore, the principle of online adjustment of damping parameters is based on existing mechanistic understanding, making it difficult to apply to "black / grey box" systems. Summary of the Invention
[0003] To address the shortcomings of existing technologies, the purpose of this invention is to provide a method for improving the stability of new energy power generation systems based on deep reinforcement learning.
[0004] According to one aspect of the present invention, a method for improving the stability of a new energy power generation system based on deep reinforcement learning is provided, comprising:
[0005] Obtain real-time operational data from new energy power generation systems;
[0006] Design a reward function, the input of which is the real-time operating data and stability evaluation criteria of the new energy power generation system, and the output is a reward value;
[0007] The real-time operating data and reward value of the new energy power generation system are used to train a deep reinforcement learning agent offline.
[0008] The real-time operating data is input into the trained deep reinforcement learning agent to obtain the control parameters of the new energy power generation system.
[0009] Using the control parameters of the new energy power generation system as a standard, the control parameters of the new energy power generation system are adjusted online adaptively.
[0010] Preferably, the new energy power generation system includes a photovoltaic power generation system, a doubly-fed wind power generation system, or a direct-drive wind power generation system.
[0011] Preferably, the real-time operating data of the new energy power generation system is data under multiple operating conditions, which refers to multiple short-circuit ratios and power changes; the real-time operating data includes DC bus voltage, three-phase voltage and current on the AC side of the converter, output power, and the errors between the above physical quantities and given values.
[0012] Preferably, the stability evaluation criterion refers to a parameter that can quantify the stability of the system using numerical indicators, and this parameter refers to the system phase margin;
[0013] The process for obtaining the system phase margin is as follows:
[0014] Establish a broadband impedance model for new energy grid-connected systems;
[0015] Based on the aforementioned broadband impedance model, the system phase margin under different operating conditions and controller parameters is calculated using impedance stability analysis methods. The operating conditions include the operating point and grid strength.
[0016] Preferably, the reward value includes two parts: instantaneous error and stability; wherein, the instantaneous error is calculated based on the operating data of the new energy power generation system, and the stability is calculated based on the quantitative index obtained from the stability criterion;
[0017] The deep reinforcement learning agent learns by maximizing the reward value.
[0018] Preferably, the deep reinforcement learning agent is a neural network that includes a Critic value network and an Actor policy network.
[0019] Preferably, the offline training of the deep reinforcement learning agent using the real-time operating data and reward value of the new energy power generation system includes:
[0020] The real-time operating data, including DC bus voltage, AC three-phase voltage and current of the converter, output active and reactive power and their error values, are acquired and input into the intelligent agent to be trained.
[0021] The real-time running data and the phase margin of the current system state are input into the reward function to calculate the corresponding reward value, and the reward value is input into the agent to be trained.
[0022] Training is conducted in an offline environment using simulation or actual systems, using real-time simulation environments and historical experience replays, and continuously correcting the neural network parameters of the agent to be trained through multiple rounds of simulation to maximize the expected reward value.
[0023] Train until the reward value reaches the set standard.
[0024] Preferably, the step of training in an offline environment using a simulation system or a real system, training using a real-time simulation environment and historical experience replay, and continuously correcting the neural network parameters of the agent to be trained through multiple rounds of simulation to maximize the expected reward value includes:
[0025] Define the loss at each step;
[0026] The neural network parameters are updated using gradient descent or gradient ascent after the loss is defined at each step, and the target network is soft-updated.
[0027]
[0028] θ′←τθ+(1-τ)θ′
[0029] In the formula, θ is the parameter of the main network of the value network, θ' is the parameter of the target network of the value network, α is the learning rate, τ is the soft update coefficient, and L(θ) is the loss function, which is defined using the Bellman error:
[0030]
[0031] In the formula, r is the reward, s is the state, a is the output action, γ is the discount factor, Q is the total reward value obtained by the network evaluation, which represents the cumulative reward that the agent can obtain in the future after taking a certain action a in a given state s, and s' and a represent the state and action at the next moment.
[0032] Preferably, the loss function may also include a regularization factor λ, specifically:
[0033]
[0034] Preferably, real-time running data is continuously collected in an online environment for fine-tuning and optimization of neural network parameters.
[0035] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:
[0036] The embodiments of the present invention can adaptively adjust controller parameters online according to changes in system operating conditions, thereby improving the stability of new energy power generation systems. This method does not require the detection of oscillation components, and the online adjustment of parameters does not depend on the understanding of mechanisms. It has the advantages of fast response speed and strong adaptability to changes in operating conditions. Attached Figure Description
[0037] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0038] Figure 1 This is a flowchart of a preferred embodiment of the present invention for a method to improve the stability of a new energy power generation system based on deep reinforcement learning.
[0039] Figure 2 This is a schematic diagram of the framework structure of a method for improving the stability of a new energy power generation system based on deep reinforcement learning in a preferred embodiment of the present invention.
[0040] Figure 3 This is a schematic diagram of a typical wind power generation system in a preferred embodiment of the present invention.
[0041] Figure 4 This is a flowchart of deep reinforcement learning in a preferred embodiment of the present invention.
[0042] Figure 5 This is a schematic diagram of the control structure of a new energy power generation system incorporating a deep reinforcement learning agent in a preferred embodiment of the present invention.
[0043] Figure 6 This is a simulation waveform diagram of a new energy power generation system under conventional control under a step change in active power in a preferred embodiment of the present invention.
[0044] Figure 7 This is a simulation waveform diagram of a new energy power generation system based on deep reinforcement learning under the condition of a step change in active power in a preferred embodiment of the present invention. Detailed Implementation
[0045] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.
[0046] In one embodiment of the present invention, a method for improving the stability of a new energy power generation system based on deep reinforcement learning is provided. (See also...) Figure 1 and Figure 2 The following steps can be taken:
[0047] Step 1: Obtain real-time operating data of the new energy power generation system;
[0048] Step 2: Design a reward function. The input to the reward function is the real-time operating data and stability evaluation criteria of the new energy power generation system, and the output is the reward value.
[0049] Step 3: Using the real-time operation data of the new energy power generation system obtained in Step 1 and the reward value obtained in Step 2, train the deep reinforcement learning agent offline.
[0050] Step 4: Input the real-time running data into the deep reinforcement learning agent trained in Step 3 to obtain the control parameters of the new energy power generation system;
[0051] Step 5: Using the control parameters of the new energy power generation system obtained in Step 4 as the standard, adjust the control parameters of the new energy power generation system online adaptively.
[0052] The above embodiments train a deep reinforcement learning agent to learn parameter adjustment strategies that can improve stability, especially in the case of weak grids and high power output, thereby improving the stability of the new energy grid-connected system.
[0053] In a preferred embodiment, the real-time operating data of the new energy power generation system is data under various operating conditions, namely, data under various short-circuit ratios and power variations. The real-time operating data includes DC bus voltage, three-phase voltage and current on the AC side of the converter, output power, and the errors between the above physical quantities and given values.
[0054] In a preferred embodiment, the stability evaluation criterion can be used as a numerical index to quantify the stability of the system. The specific process for obtaining this criterion is as follows:
[0055] Establish a broadband impedance model for new energy grid-connected systems;
[0056] The system phase margin under different operating conditions (including different operating points and grid strength) and controller parameters is calculated using impedance stability analysis.
[0057] The above embodiments use real-time data that is operational data under different short-circuit ratios and power changes, and use system phase margin under different operating conditions as the criterion, so that the trained agent can adapt to the usage requirements of different operating conditions and learn parameter adjustment strategies to improve stability under various conditions.
[0058] In a preferred embodiment, the designed reward function includes two parts: instantaneous error and stability. The instantaneous error is calculated based on relevant operating data of the new energy power generation system, while the stability is calculated based on quantitative indicators obtained from stability criteria. The overall objective is that the smaller the instantaneous error and the closer the phase margin is to a preset value, the higher the reward. The reward serves as an important basis during offline training, with the agent learning by maximizing the reward; however, reward input is no longer required for online applications.
[0059] Furthermore, in one specific embodiment, the reward function used is as follows:
[0060] r = r1 + r2
[0061] r1 represents the instantaneous error, and r2 represents the stability.
[0062]
[0063] r2=-ω5·|PM-30°|
[0064] +ω6·S(20° <PM<60°)
[0065] -ω6·S (PM<0° or PM>90°)
[0066] -2ω6·S (PM < -20° or PM > 110°)
[0067] in, e represents the instantaneous error of DC voltage. Q U represents the instantaneous error of reactive power. dc,ref P represents the DC voltage reference value. ref This represents the active power reference value. Each ω represents a weight, PM is the phase margin, and the S-function is defined as:
[0068]
[0069] To obtain a more accurate agent, in a preferred embodiment of the present invention, the training method of the agent is a deep reinforcement learning method, including DQN (Deep Q-Network), DDPG (deep deterministic policy gradient), TD3 (Twin Delayed Deep Deterministic policy gradient), PPO (Proximal Policy Optimization), etc.; the agent is composed of a neural network, and the training process involves adjusting the network parameters based on the training data.
[0070] Furthermore, in a preferred embodiment, the training process of the intelligent agent specifically includes:
[0071] 1) Acquire real-time operating data, including DC bus voltage, AC three-phase voltage and current of the converter, output active and reactive power and their error values, and input them into the agent to be trained. Real-time operating parameters need to be used as observations for the agent. By observing these values, the agent can evaluate the current operating conditions and make corresponding decisions to improve stability.
[0072] 2) The stability of the current system state is evaluated by a pre-designed reward function. The reward function is combined with stability evaluation criteria such as phase margin to calculate the corresponding reward value, which is then input into the agent to be trained.
[0073] 3) Training can be conducted offline using a simulation system or a real system. Real-time simulation and historical experience playback are used for training. The network parameters are continuously corrected through multiple rounds of simulation, thereby maximizing the expected reward.
[0074] 4) Train until the agent can adaptively adjust control parameters under different operating conditions to improve system stability. Specifically:
[0075] The reward function reflects the stability of the system; therefore, the goal of training is to maximize the reward function. During training, random initial operating conditions are given in each round. If the reward obtained in several consecutive training rounds reaches a certain threshold, it indicates that the agent's adaptive stability improvement under different operating conditions is good enough, and training can be stopped.
[0076] Data can then be collected in an online environment for fine-tuning and optimization.
[0077] Furthermore, in a preferred embodiment, step 3) is further explained in detail:
[0078] After defining the loss at each step, the network parameters are updated using gradient descent or gradient ascent, with soft updates applied to the target network, as shown in the following formula:
[0079]
[0080] θ′←τθ+(1-τ)θ′
[0081] In the formula, θ represents the parameters of the main network of the value network, θ' represents the parameters of the target network of the value network, α represents the learning rate, τ represents the soft update coefficient, and L(θ) represents the loss function. The loss function is generally defined using the Bellman error definition:
[0082]
[0083] In the formula, r is the reward, s is the state, a is the output action, and γ is the discount factor. The Q-value is the total reward value obtained by the network evaluation, similar to a reward function, representing the cumulative reward (weighted with the discount factor) that the agent can obtain in the future after taking an action a given state s. s' and a' represent the state and action at the next time step. That is, the DDPG method attempts to make the current Q-value (the last term Q) approach a more stable target value (the sum of the first two terms r + γQ) to achieve more stable training.
[0084] The loss function described above can also include an additional regularization factor λ, for example:
[0085]
[0086] The target network is a copy of the main network, used to assist in learning. The actor and critic networks each have their own main and target networks. The target network has the same structure as the main network, but its parameter update method differs: the main network's parameters are updated directly using gradient descent. The target network's parameters are updated from the main network at a slower rate, typically using a "soft update" method described in the documentation. Using the target network provides a relatively stable reference value for the main network's updates, preventing oscillations or divergence (gradient explosion) during learning, thus serving as an auxiliary training method.
[0087] In one embodiment of the invention, the collected operational data is input to a trained agent. The agent's Actor network employs a deep neural network architecture, containing multiple fully connected hidden layers and nonlinear activation functions (e.g., ReLU and tanh), designed to extract state features and generate appropriate control policies. The network's input is the real-time operational data of the current system. Linear mapping is performed on the original input through multiple fully connected layers, and feature extraction is performed using nonlinear activation functions to capture the complex relationships between state variables and ensure that the control policy can adapt to various operational states. After a series of linear and nonlinear mapping processes, the optimized control policy parameters are finally output through the output layer.
[0088] To verify the feasibility and effectiveness of the stability improvement method for new energy power generation systems in the above embodiments, a specific embodiment of the present invention uses a typical wind power generation system as an example for verification. Figure 3 As shown, the typical wind power generation system provided in this embodiment may include: a wind turbine, a generator, a converter, and a filter. The converter in the wind power generation system is a back-to-back converter, consisting of a turbine-side converter and a grid-side converter. The turbine-side converter is responsible for controlling the generator's speed or torque to achieve maximum wind energy capture; the grid-side converter is responsible for controlling the DC bus voltage and achieving synchronization with the power grid.
[0089] Figure 4 As shown, the deep reinforcement learning flowchart provided in this embodiment may include the following operations:
[0090] S1, Parameter Predefined: Initial parameter values are roughly set based on experience, specifically the main circuit parameters and controller parameters. Main circuit parameters include rated DC-side voltage, rated AC-side voltage, rated output power, DC bus capacitance, and filter inductance. Controller parameters include PI control parameters for the current loop, DC voltage loop, phase-locked loop, and reactive power loop.
[0091] S2, Configure the environment and associate the simulation model:
[0092] First, manually bind the `agent` variable to the RL Agent module to enable the training function to recognize the agent. Then, begin setting the dimensions and intervals for observations and actions (i.e., the agent's inputs and outputs).
[0093] Secondly, determine the dimensions of observation and action. To avoid overly unreasonable action output, define a range for the action.
[0094] Furthermore, a baseline value is determined. After selecting the baseline value, considering the need for exploration space for each parameter, the upper and lower bounds are set as a certain multiple of the baseline value. Taking into account the individual influence of the two parameters and their combined effect on the bandwidth of each control loop, the range of the proportional coefficient is set to 1 / 4 to 4 times, and the range of the integral coefficient is set to 1 / 8 to 8 times. Within this parameter selection range, it can be basically guaranteed that the bandwidth of each loop fluctuates around 1 / 4 to 4 times the baseline value, leaving sufficient exploration space while avoiding completely unreasonable action outputs.
[0095] In addition, the environment must include an initialization function that restores the environment to a random initial state before the start of each training round. In this function, to simulate changes in the operating point in reality, the short-circuit ratio is set to vary between 2 and 6, the mains voltage varies between 0.95 and 1.05 times the rated voltage, and the output power varies between 0.2 and 1.0 times the rated power.
[0096] In addition, setting multiple short-circuit ratios and power variations allows the intelligent agent to learn more different operating conditions.
[0097] S3, Create and configure the Critic and Actor networks: First, design the structure of the neural network. The network structure is generally determined by two aspects: the number of layers and the type and number of neurons in each layer. A simpler network may fail to effectively capture details in the information, resulting in poor fitting performance; an overly complex network may have excessive data analysis capabilities, fitting noise in the data and causing overfitting, leading to insufficient generalization. Therefore, the network structure needs to be continuously adjusted based on actual training results to achieve optimal performance. In this embodiment, the Critic network needs to receive information from both observation and action. The information from observation has higher dimensionality and richer details, so it is processed using a single ReLU layer, which is then merged with the information from action. After merging, two more ReLU layers and two fully connected layers with 64 neurons each are used for further processing. The Actor network also needs a certain level of complexity because it needs to analyze information from observation and derive a strategy. However, considering the final deployment in a real-world scenario, and taking into account computational load and real-time computation time, it should not be overly complex. Therefore, the Critic network structure in this embodiment is: Observation Input Layer - Fully Connected Layer - ReLU - Fully Connected Layer - Merging Layer (merged with input from the Actor network) - ReLU - Fully Connected - ReLU - Fully Connected Output Layer. The Actor network structure is: Input Layer - Fully Connected - ReLU - Fully Connected - ReLU - Fully Connected - tanh - Output Layer. Different network structures (such as different numbers of layers, the arrangement of different layers, the number of neurons, the choice of non-linear activation functions, etc.) only affect training effectiveness and actual performance, but all belong to the critic-actor framework.
[0098] S4, Set agent options and create the agent: Before training, to define the agent's behavior during training, certain hyperparameters need to be manually configured and continuously adjusted based on the training results. In this embodiment, the agent hyperparameters include agent sampling period, discount factor, experience pool length, experience pool sample size, network learning rate, learning gradient threshold, L2 regularization factor, training noise, minimum noise, noise decay rate, and target policy smoothing noise, etc. Among them: the discount factor makes the value function more likely to converge; the experience pool length needs to be large enough to achieve stable value network updates; the learning gradient threshold ensures that each learning step size is within an appropriate range, avoiding negative impacts such as gradient explosion; the exploration noise needs to be moderate to ensure that the agent learns an effective but not overly aggressive policy; the L2 regularization factor can limit the coefficients of high-order polynomials, thereby avoiding overfitting of the entire network.
[0099] S5, Set training options: Before training, it is also necessary to set the parameters related to the entire training process. In this embodiment, the training process parameters include the maximum number of rounds in a single training session, the number of steps in a single round, the average reward calculation window length, the criteria for stopping training, the criteria for saving the agent, whether to output the training results to the console, and whether to plot the training results.
[0100] S6, training begins.
[0101] like Figure 5 As shown, the control structure for a new energy power generation system combined with a deep reinforcement learning agent provided in this embodiment includes a deep reinforcement learning agent. This agent takes relevant operational data of the new energy power generation system as input, outputs control parameters of the new energy power generation system, and updates the control system of the new energy power generation system to improve system stability. The agent obtained through deep reinforcement learning observes the environmental state State S. i After measuring the DC bus voltage, converter AC three-phase voltage and current, output active and reactive power and their error values, etc., the agent needs to interact with the environment by selecting actions through a strategy. In this embodiment, the agent does this by changing the PI parameters (K) of the DC voltage loop, current loop, and phase-locked loop. pdc K idc K pi K ii K ppll K ipll This is achieved through a combination of methods, including: The Critic network takes inputs of current operating data and the control strategy output by the Actor network. Its role is to evaluate the quality of actions generated by the current Actor network in conjunction with the environmental state, and to calculate the Q-value as the basis for optimizing the Actor network's training. This network consists of multiple fully connected layers, with ReLU activation functions used for nonlinear transformation between the fully connected layers, ultimately outputting a scalar Q-value. The Actor network takes inputs of real-time acquired data and outputs control strategy parameters (including PI parameters for the DC voltage loop, current loop, and PLL). This network consists of multiple fully connected layers, containing ReLU and tanh activation functions to extract state features and generate reasonable control strategies. The final output is optimized control strategy parameters used to adjust the performance of the control system in real time.
[0102] like Figure 6As shown, the conventional control provided in this embodiment is a fixed-parameter reactive power / DC voltage outer loop and current inner loop control. During the simulation, the initial short-circuit ratio is 4, and the initial output power is 0.2 pu. Then, at 0.5 s, it switches to a weak grid with a short-circuit ratio of 2.7. Subsequently, at 1 s, 2 s, and 3 s, the active power output is increased to 0.6 pu, 0.8 pu, and 1.0 pu, respectively. As can be seen from the figure, with the increase of active power, the system gradually oscillates and eventually becomes divergent and unstable.
[0103] like Figure 7 As shown in the figure, during the simulation of the new energy power generation system based on deep reinforcement learning agents provided in this embodiment, the initial short-circuit ratio is 4, and the initial output power is 0.2 pu. Then, at 0.5 s, it switches to a weak grid with a short-circuit ratio of 2.7. Subsequently, at 1 s, 2 s, and 3 s, the active power output is increased to 0.6 pu, 0.8 pu, and 1.0 pu, respectively. As can be seen from the figure, the system remains stable under different output active power levels with small overshoot, demonstrating the advantages of the stability improvement method for new energy power generation systems based on deep reinforcement learning provided in this invention.
[0104] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention. The above preferred features can be used in any combination without conflict.
Claims
1. A method for improving stability of a new energy power generation system based on deep reinforcement learning, characterized in that, include: Obtain real-time operational data from new energy power generation systems; Design a reward function, the input of which is the real-time operating data and stability evaluation criteria of the new energy power generation system, and the output is a reward value; The real-time operating data and reward value of the new energy power generation system are used to train a deep reinforcement learning agent offline. The real-time operating data is input into the trained deep reinforcement learning agent to obtain the control parameters of the new energy power generation system, and then updated to the control system of the new energy power generation system. The reward value includes two parts: instantaneous error and stability. The instantaneous error is calculated based on the operating data of the new energy power generation system, while the stability is calculated based on a quantitative index obtained from a stability criterion. The quantitative index refers to the system phase margin. The deep reinforcement learning agent learns by maximizing the reward value.
2. The method of claim 1, wherein The new energy power generation system includes photovoltaic power generation system, doubly fed wind power generation system or direct-drive wind power generation system.
3. The method of claim 1, wherein The real-time operating data of the new energy power generation system is data under multiple operating conditions, which refers to multiple short-circuit ratios and power changes; the real-time operating data includes DC bus voltage, three-phase voltage and current on the AC side of the converter, output power, and the error between each physical quantity and the given value.
4. The method of claim 1, wherein The stability evaluation criteria refer to the parameters that quantify the stability of the system, which are the system phase margins. The process for obtaining the system phase margin is as follows: Establish a broadband impedance model for new energy grid-connected systems; Based on the aforementioned broadband impedance model, the system phase margin under different operating conditions and controller parameters is calculated using impedance stability analysis methods. The operating conditions include the operating point and grid strength.
5. The method for improving the stability of a new energy power generation system according to claim 1, characterized in that, The deep reinforcement learning agent is a neural network that includes a Critic value network and an Actor policy network.
6. The method of claim 5, wherein The offline training of a deep reinforcement learning agent using the real-time operating data and reward value of the new energy power generation system includes: The real-time operating data, including DC bus voltage, AC three-phase voltage and current of the converter, output active and reactive power and their error values, are acquired and input into the intelligent agent to be trained. The real-time running data and the phase margin of the current system state are input into the reward function to calculate the corresponding reward value, and the reward value is input into the agent to be trained. Training is conducted in an offline environment using simulation or actual systems, using real-time simulation environments and historical experience replays, and continuously correcting the neural network parameters of the agent to be trained through multiple rounds of simulation to maximize the expected reward value. Train until the reward value reaches the set standard.
7. The method for improving the stability of a new energy power generation system according to claim 6, characterized in that, The process of training in an offline environment using a simulation system or a real system, employing real-time simulation and historical experience replay, and continuously refining the neural network parameters of the agent under training through multiple rounds of simulation to maximize the expected reward value includes: Define the loss at each step; The neural network parameters are updated using gradient descent or gradient ascent after the loss is defined at each step, and the target network is soft-updated. ； ； In the formula, These are the parameters of the main network of the value network. ' represents the parameters of the target network in the value network, and α is the learning rate. It is a soft update coefficient. L ( ) is the loss function, which is defined using the Bellman error: ； In the formula, r It's a reward. s It's a state. a It is an output action. γ Q is the discount factor; Q is the total reward value obtained by the network evaluation, which represents the cumulative reward that the agent can obtain in the future after taking a certain action a in a given state s. s' and a' represent the state and action at the next moment.
8. The method of claim 7, wherein the method further comprises: The loss function further adds a regularization factor Specifically, 。 9. The method of claim 8, wherein the method further comprises: In an online environment, real-time running data is continuously collected for fine-tuning and optimization of neural network parameters.