A method and apparatus for quantizing and evaluating the inertia of a network-type inverter based on deep reinforcement learning.

By using a deep reinforcement learning-based method to collect and process power system data, build a series-parallel identification model and design a neural network, the problem of inertia assessment for new energy inverters is solved, and an accurate quantitative assessment of the inertia level of grid-connected inverters is achieved, supporting the stable operation of the power grid.

CN117150892BActive Publication Date: 2026-11-17ZHEJIANG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202311042377.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-08-16
Publication Date
2026-11-17
Estimated Expiration
2043-08-16

AI Technical Summary

Technical Problem

Existing technologies are difficult to effectively quantify and assess the inertia level of grid-connected inverters for new energy sources. In particular, the analysis of frequency measurement is too simplified and cannot be applied to the inertia assessment of grid-connected inverters.

Method used

A deep reinforcement learning-based approach is adopted to collect power system data, build a series-parallel identification model, design an action space and reward function, establish an action neural network and a reward neural network, and train them to quantify the inverter inertia level.

Benefits of technology

It realizes the quantitative evaluation of inertia considering frequency measurement delay and attenuation, which can accurately evaluate the frequency support capability of grid-connected inverters, help build an inertia evaluation system for grid-connected inverters of new energy, and guide the economic operation of the power grid and safe and reliable power supply.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117150892B_ABST
    Figure CN117150892B_ABST
Patent Text Reader

Abstract

This invention discloses a method and apparatus for quantifying and evaluating the inertia of a grid-type inverter based on deep reinforcement learning. The apparatus includes a data acquisition module for collecting data, a model building module for combining deep learning and reinforcement learning, and a neural network training and inertia evaluation module for training the model and outputting inertia evaluation results. Considering the delay and attenuation in the frequency measurement process, this invention achieves quantifying and evaluating the inertia of a grid-type inverter, thereby assessing the frequency support capability of the grid-type inverter.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of new energy grid-connected inverter technology, and discloses a method and device for quantitative evaluation of the inertia level of grid-connected inverters based on deep reinforcement learning. Background Technology

[0002] With the increasing penetration of new energy sources in modern power grids, the power electronic characteristics of power systems are becoming more pronounced. The large-scale integration of new energy grid-connected inverters is leading to a decrease in the power system's inertia level, posing a significant challenge to the safe and stable operation of the new power system. The introduction of grid-connected inverters has improved the inverter's support role in the power grid. Against this backdrop, a comprehensive evaluation of the actual inertia support effect of grid-connected inverters and a quantitative calculation of their control performance can quantify their support role in the power grid, providing an information foundation for the safe and stable operation of the power grid.

[0003] Current methods for assessing the inertia of power systems are not sufficiently in-depth and do not consider all factors comprehensively. In particular, the simplification in the analysis of frequency measurement is relatively ideal and difficult to apply to the inertia assessment of inverters. Therefore, a suitable method and device for quantitative assessment of the inertia of grid-connected inverters is needed. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and, in order to address the current situation where there is a lack of quantitative evaluation technology for the inertia level of grid-connected new energy inverters, proposes a method and device for quantitative evaluation of the inertia of grid-connected inverters based on deep reinforcement learning.

[0005] This invention is achieved through the following technical solution:

[0006] This invention first provides a method for quantizing and evaluating the inertia of a network-type inverter based on deep reinforcement learning, comprising the following steps:

[0007] Step 1: Collect the timing data of the three-phase voltage and three-phase current output from the grid-type inverter output side of the power system under normal / fault operation conditions, as well as the timing data of the bus frequency connected to the grid-type inverter obtained based on phase-locked loop measurement; and normalize the timing data of the three-phase voltage and three-phase current based on the rated capacity of the grid-type inverter, and normalize the timing data of the bus frequency connected to the grid-type inverter based on the rated frequency of the power system.

[0008] Step 2: Based on the structure of the grid-connected inverter system, a series-parallel identification model is built, and a simulation environment, action space, and reward function based on deep reinforcement learning algorithm are designed in the series-parallel identification model. The action space is discrete inertia control parameters, and the reward function is used to evaluate the discrete inertia control parameters and output the penalty for the frequency measurement data error between the simulation environment and the power system.

[0009] Step 3: Establish an agent including an action neural network and a reward neural network; the action neural network is used to acquire active power and frequency data measured by the power system and output discrete inertia control parameters, which serve as the action space of the serial-parallel identification model; the reward neural network is used to output the reward that can be obtained in the current state based on the frequency measurement data error between the simulation environment and the power system.

[0010] Step four: Combine the serial-parallel identification model with the intelligent agent, and train it using the time series data after normalization in step one to obtain the inertia quantization evaluation system. Use the output of the training as the inverter inertia quantization evaluation result.

[0011] The present invention also provides a grid-type inverter inertia quantification and evaluation device, comprising:

[0012] The data acquisition module includes a phase-locked loop (PLL) for acquiring the timing data of the three-phase voltage and three-phase current output from the output side of the grid-type inverter in the power system under normal / fault operation conditions, as well as the timing data of the frequency of the bus connected to the grid-type inverter obtained based on the PLL measurement, and normalizing the acquired timing data.

[0013] The model building module is used to build a series-parallel identification model based on the grid-type inverter system structure, as well as an intelligent agent including an action neural network and a reward neural network.

[0014] The neural network training and inertia assessment module is used to combine the serial-parallel identification model with the intelligent agent, and to train the combined model with time-series data collected and processed by the data acquisition module to obtain a model equivalent to the inertia environment of the power system, and output the inertia assessment results through the action neural network.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] This invention addresses the inertia level assessment of grid-connected inverters. Based on deep reinforcement learning algorithms, it proposes a method and apparatus for quantitative assessment of the inertia of grid-connected inverters. Considering the delay and attenuation in the frequency measurement process, it achieves quantitative assessment of the inertia of grid-connected inverters, thereby evaluating the frequency support capability of grid-connected inverters.

[0017] This invention can help build an evaluation system for the inertia of grid-connected inverters for new energy, comprehensively assess the inertia capability of installed grid-connected new energy equipment, thereby clearly understanding the overall inertia level of flexible interconnected power grids and guiding the economic operation, safe and reliable power supply, and future development planning of the power grid. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a phase-locked loop (PLL) structure.

[0019] Figure 2 This is a structural diagram of a grid-connected inverter system;

[0020] Figure 3 For identifying the structure diagram of series and parallel connections;

[0021] Figure 4 A diagram illustrating the framework of a deep reinforcement learning algorithm;

[0022] Figure 5 Here is a structural diagram of the inertia assessment device;

[0023] Figure 6 The image shows the experimental data from the RT-box hardware-in-the-loop simulation.

[0024] Figure 7 This is a diagram illustrating the convergence process of the method in this specific embodiment;

[0025] Figure 8 The figure shows the quantitative evaluation results based on hardware-in-the-loop simulation. Detailed Implementation

[0026] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0027] Existing inertia assessment methods are mainly applicable to single units (synchronous generators), power system regions, and the entire power system based on synchronous generators. It is difficult to establish an inertia assessment model for virtual inertia (grid-type inverters), and the delay and attenuation in the frequency measurement link are not considered. Therefore, they cannot be applied to new power systems containing virtual inertia.

[0028] Therefore, this invention designs a method and device for quantitative evaluation of the inertia of a grid-type inverter based on deep reinforcement learning, including establishing a transfer function model of the active-frequency control loop and synchronous machine rotor motion equations of the grid-type inverter; building a deep reinforcement learning training environment, action and reward neural network; and training the deep reinforcement learning algorithm based on the operating data to obtain the inertia level of the grid-type inverter.

[0029] In the research scenario, a transfer function model of the active-frequency control module and synchronous machine rotor motion equation of the grid-type inverter is established to prepare for the subsequent construction of a deep reinforcement learning environment. The construction of the deep reinforcement learning training environment, action and reward neural network lays the foundation for the evaluation of the inertia of the grid-type inverter. Based on the operating data, the deep reinforcement learning algorithm is trained to obtain the inertia level of the grid-type inverter, realizing the quantitative evaluation of the inertia level of the grid-type inverter.

[0030] A method for quantizing and evaluating the inertia of a network-type inverter based on deep reinforcement learning, the steps of which are as follows:

[0031] Step 1: Collect the three-phase voltage and three-phase current output from the grid-connected inverter under normal / fault operation conditions of the power system, as well as the timing data of the frequency of the connected bus obtained based on phase-locked loop measurements. For example... Figure 6 The voltage and current timing data are based on the RT-box hardware-in-the-loop simulation platform. The phase-locked loop structure is shown below. Figure 1 As shown, the system includes a phase detector, a loop filter, and a voltage-controlled oscillator (VCO). The system adjusts the frequency of the VCO to eliminate phase differences and tracks the frequency of a sinusoidal three-phase signal based on the VCO. Where u... abc The three-phase voltage signal to be measured; u d and u q These are the dq-axis components of the three-phase voltage signal, respectively; ω PLL The angular frequency is obtained from the loop filter. The phase-locked loop measurement frequency structure transfer function is:

[0032]

[0033] Where V1 is the voltage amplitude, k pPLL k is the scaling parameter of the loop filter. iPLL These are the integral parameters of the loop filter.

[0034] The time-series data were then preprocessed by normalization based on the unit's rated capacity and the system's rated frequency.

[0035] Step two, with Figure 3 The serial-parallel identification structure shown is a framework. A serial-parallel identification model is built, which includes the system to be identified based on a grid-type inverter system and a neural network based on a deep reinforcement learning algorithm. In the serial-parallel identification model, the action space and reward function of the deep reinforcement learning algorithm are designed. The action space is a discrete inertia control parameter, and the reward function is used to output the frequency measurement data error between the simulation environment and the original system.

[0036] In step two, the grid-connected inverter system structure is constructed. A commonly used grid-connected inverter system structure diagram is shown below. Figure 1As shown, it includes a main circuit section and a control circuit section. The main circuit section, U... d X is the DC bus voltage; f C f Form an LC filter; u is the capacitor branch voltage; i is the inverter output current; U g and Z g These represent the grid voltage and grid line impedance, respectively. The control circuit mainly includes a power calculation stage and an active-frequency control loop. The power calculation stage calculates the active power P output by the inverter at the grid connection point based on the grid connection point voltage and current. out and reactive power Q out The active-frequency control loop, which includes the inertia and damping elements, is expressed by the following formula:

[0037]

[0038] Where J is the inertia parameter of the inverter; Δω is the change in angular frequency; P m P represents the mechanical power of the inverter. e δ is the electromagnetic power of the inverter; D is the damping parameter of the inverter; δ is the virtual power angle of the inverter; ω is the angular velocity of the inverter; ω N This refers to the grid synchronization angular velocity.

[0039] Step 3: Establish an action neural network, acquire the active power and frequency data measured by the original system, and output inertia control parameters;

[0040] For action neural networks, the network weights are updated using J(θ).

[0041]

[0042]

[0043]

[0044] Where J(θ) is the loss function of the action neural network; E t (·) is the pair (s) under the π(·;θ') strategy. t ,a t The expectation of the state optimization loss function; r t (θ) is the importance sampling ratio at time t; A(a t |s t ) is the dominance function; It is the estimated value of the advantage function; θ is the weight of the action neural network; π(a t |s t ;θ) is the output value of the action neural network with parameter θ at time t; π(a t |s t;θ') is the output value of the action neural network with parameter θ' at time t; ε is a hyperparameter used to limit r t The range of variation of (θ).

[0045] The weights of the action neural network are updated using the gradient ascent algorithm. The update method is as follows:

[0046]

[0047] Where β is the learning rate of the action neural network; This is the gradient calculation function.

[0048] Step 4: Establish a reward neural network, obtain the frequency measurement data error between the simulation environment and the original system, and output the reward that can be obtained in the current state;

[0049] For reward neural networks, the network weights are updated by optimizing the loss function.

[0050] L(ω)=E[y t -V(s t ;ω)] 2

[0051] y t =R t +γV(s t+1 ;ω)

[0052] In the formula, L(ω) is the loss function of the reward neural network, and V(s) is the loss function of the reward neural network. t ;ω) is the output value of the reward neural network at time t, y t For time t+1, V(s) t The estimated value R made by ω) t Let ω be the instantaneous reward at time t, and ω be the weight of the reward neural network.

[0053] The weights of the reward neural network are updated using the gradient descent algorithm, and the update method is as follows:

[0054]

[0055] In the formula, α is the gradient calculation function; α is the learning rate of the reward neural network.

[0056] The advantage function A(a) of the action neural network t |s t ) is defined as

[0057] A(s t ,a t )=Q(s t ,a t )-V(s t )

[0058] Where Q(a) t |s t V(s) is the action value function; t ) is the state value function.

[0059] To improve training speed, Q(a) t |s t By performing the Monte Carlo approximation, the advantage function is transformed into:

[0060] A(s t ,a t )=y t -V(s t ;ω).

[0061] Step 5: In the constructed serial-parallel identification structure, a proximal policy optimization algorithm is used to compare the actual reward output by the agent's action neural network at each step with the reward predicted by the reward neural network. This is repeated iteratively until the maximum number of iterations is reached, and then the output of the trained action neural network is recorded. For example... Figure 7 The diagram illustrates a training convergence process of the method of this invention. The output obtained from the training is the inertia quantization evaluation result of the inverter. The near-end policy optimization algorithm uses a four-tuple of data—state space, action space, state transition probability, and reward function—as the basic data for neural network training. Then, a dynamic step size mechanism and importance sampling technique are incorporated into the action neural network-reward neural network architecture for network training. The distance between the old and new policies is limited by the pruning mechanism of the advantage function, and the optimal action policy is found under the guidance of the reward function using a gradient update mechanism. For example... Figure 8 The figure shown is a quantitative evaluation result of the method of the present invention. The results of the 8 experiments are close to the actual parameters.

[0062] like Figure 5 As shown, the present invention also provides an evaluation device for a deep reinforcement learning-based method for quantizing the inertia of a network inverter, the device comprising:

[0063] The data acquisition module includes a phase-locked loop (PLL) for acquiring the timing data of the three-phase voltage and three-phase current output from the output side of the grid-type inverter in the power system under normal / fault operation conditions, as well as the timing data of the frequency of the bus connected to the grid-type inverter obtained based on the PLL measurement, and normalizing the acquired timing data.

[0064] The model building module is used to build a series-parallel identification model based on the grid-type inverter system structure, as well as an intelligent agent including an action neural network and a reward neural network.

[0065] The neural network training and inertia assessment module is used to combine the serial-parallel identification model with the intelligent agent, and to train the combined model with time-series data collected and processed by the data acquisition module to obtain a model equivalent to the inertia environment of the power system, and output the inertia assessment results through the action neural network.

[0066] The above-described embodiments are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. Those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for quantitative evaluation of the inertia of a network-type inverter based on deep reinforcement learning, characterized in that, Includes the following steps: Step 1: Collect the timing data of the three-phase voltage and three-phase current output from the grid-type inverter output side of the power system under normal / fault operation conditions, as well as the timing data of the bus frequency connected to the grid-type inverter obtained based on phase-locked loop measurement; and normalize the timing data of the three-phase voltage and three-phase current based on the rated capacity of the grid-type inverter, and normalize the timing data of the bus frequency connected to the grid-type inverter based on the rated frequency of the power system. Step 2: Based on the structure of the grid-connected inverter system, a series-parallel identification model is built, and a simulation environment, action space, and reward function based on deep reinforcement learning algorithm are designed in the series-parallel identification model. The action space is discrete inertia control parameters, and the reward function is used to evaluate the discrete inertia control parameters and output the penalty for the frequency measurement data error between the simulation environment and the power system. Step 3: Establish an agent including an action neural network and a reward neural network; the action neural network is used to acquire active power and frequency data measured by the power system and output discrete inertia control parameters, which serve as the action space of the serial-parallel identification model; the reward neural network is used to output the reward that can be obtained in the current state based on the frequency measurement data error between the simulation environment and the power system. Step four: Combine the serial-parallel identification model with the intelligent agent, and train it using the time series data after normalization in step one to obtain the inertia quantization evaluation system. Use the output of the training as the inverter inertia quantization evaluation result.

2. The inertia quantification evaluation method for grid-type inverters according to claim 1, characterized in that, The simulation environment is a grid-type inverter and a phase-locked loop built in a series-parallel identification model; the power system is an actual grid-type inverter; the power system inertia environment is an actual grid-type inverter and a phase-locked loop with inertia control parameters.

3. The inertia quantification evaluation method for grid-type inverters according to claim 1, characterized in that, In step 1, the power system eliminates the phase difference by adjusting the frequency of the voltage-controlled oscillator, and tracks the frequency of the sinusoidal three-phase signal based on the voltage-controlled oscillator.

4. The method for quantitative evaluation of the inertia of a grid-type inverter according to claim 1, characterized in that, In step two, the specific structure of the grid-connected inverter system is as follows: the DC power supply is connected to the grid-connected inverter, and then connected to the main grid through filters and transmission lines; the control circuit of the grid-connected inverter system mainly includes a power calculation stage and an active-frequency control loop; the power calculation stage calculates the active power P output by the inverter at the grid connection point based on the voltage and current at the grid connection point. out and reactive power Q out The active-frequency control loop, which includes the inertia and damping elements, is formulated as follows: Where J is the inertia parameter of the inverter; Δω is the change in angular frequency; P m P represents the mechanical power of the inverter. e δ is the electromagnetic power of the inverter; D is the damping parameter of the inverter; δ is the virtual power angle of the inverter; ω is the angular velocity of the inverter; ω N This refers to the grid synchronization angular velocity.

5. The inertia quantification evaluation method for grid-type inverters according to claim 1, characterized in that, Step four specifically involves: in the constructed serial-parallel identification model, using a proximal policy optimization algorithm to compare the reward function value output by the action neural network at each step with the reward predicted by the reward neural network, and performing iterative training alternately until the number of iterations reaches the upper limit, thereby obtaining the inertia quantization evaluation system; and recording the output results of the trained action neural network. The output obtained from the training is the inertia quantization evaluation result of the inverter; the reward function value is the penalty for the frequency measurement data error between the simulation environment and the power system.

6. The inertia quantification evaluation method for grid-type inverters according to claim 5, characterized in that, In step four, for the action neural network, the network weights are updated using the loss function of the action neural network: The weights of the action neural network are updated using the gradient ascent algorithm. The update method is as follows: θ←θ-β▽ θ J(θ) Where J(θ) is the loss function of the action neural network; It is the pair (s) under the π(·;θ') strategy t ,a t The expectation of the state optimization loss function; r t (θ) is the importance sampling ratio at time t; s t Let a be the state of the simulation environment at time t; t Let A(a) be the action of the agent at time t; t |s t ) is the dominance function; It is the estimated value of the advantage function; θ is the weight of the action neural network; π(a t |s t ;θ) is the output value of the action neural network with parameter θ at time t; π(a t |s t ;θ') is the output value of the action neural network with parameter θ' at time t; ε is a hyperparameter used to limit r t The range of variation of (θ); β is the learning rate of the action neural network; ▽ ω This is the gradient calculation function.

7. The method for quantitative evaluation of the inertia of a grid-type inverter according to claim 6, characterized in that, Step four employs a proximal policy optimization algorithm to compare the actual reward output by the action neural network at each step with the reward predicted by the reward neural network, and iterates this process alternately. For reward-based neural networks, the network weights are updated by optimizing the loss function: L(ω)=E[y t -V(s t (oh)) 2 yes t =R t +γV(s t+1 ;ω) In the formula, L(ω) is the loss function of the reward neural network, and V(s) is the loss function of the reward neural network. t ;ω) is the output value of the reward neural network at time t, y t For time t+1, V(s) t The estimated value R made by ω) t Let γ be the instantaneous reward at time t, and γ be the weight of the reward neural network. The weights of the reward neural network are updated using the gradient descent algorithm, with the update method being ω←ω-α▽. ω L(ω) The advantage function A(a) in the loss function of an action neural network t |s t ) is defined as A(s t ,a t )=Q(s t ,a t )-V(s t ) Where Q(a) t |s t V(s) is the action value function; t ) is the state value function; To improve training speed, Q(a) t |s t By performing the Monte Carlo approximation, the advantage function is transformed into: A(s t ,a t )=y t -V(s t ;ω) Among them, ▽ ω Let α be the gradient calculation function, and α be the learning rate of the reward neural network.

8. An evaluation device for the inertia quantification evaluation method of a grid-type inverter as described in claim 1, characterized in that, include: The data acquisition module includes a phase-locked loop (PLL) for acquiring the timing data of the three-phase voltage and three-phase current output from the output side of the grid-type inverter in the power system under normal / fault operation conditions, as well as the timing data of the frequency of the bus connected to the grid-type inverter obtained based on the PLL measurement, and normalizing the acquired timing data. The model building module is used to build a series-parallel identification model based on the grid-type inverter system structure, as well as an intelligent agent including an action neural network and a reward neural network. The neural network training and inertia assessment module is used to combine the serial-parallel identification model with the intelligent agent, and to train the combined model with time-series data collected and processed by the data acquisition module to obtain a model equivalent to the inertia environment of the power system, and output the inertia assessment results through the action neural network.

Citation Information

Patent Citations

  • Inverter controller based on deep reinforcement learning

    CN112187074A

  • Deep learning multi-agent micro-grid cooperative control method based on double neural networks

    CN115333143A