A network construction type converter adaptive control method based on a trust region strategy optimization algorithm

Abstract

This invention discloses an adaptive control method for grid-type converters based on the Trust Region Policy Optimization (TRPO) algorithm. The method includes: establishing a Virtual Synchronous Generator (VSG) basic model and defining constraint boundaries for virtual inertia and damping coefficients based on the VSG basic model; constructing the adaptive control process of the VSG basic model as a Markov Decision Process (MDP) and designing the corresponding state space, action space, and reward function; training an agent using the TRPO algorithm under the guidance of the state space, action space, and reward function to obtain a trained agent; and applying the trained agent to the actual grid-type converter control, monitoring the system state in real time, and adaptively adjusting the virtual inertia and damping coefficients according to the agent's output actions. This invention effectively solves the problem of poor adaptability in traditional fixed-parameter control and improves the robustness of frequency control.

Description

Technical Field

[0001] This invention relates to the field of power system automation control technology, and in particular to an adaptive control method for grid-type converters based on a trust region strategy optimization algorithm. Background Technology

[0002] With the rapid development of renewable energy and the gradual expansion of power system scale, the frequency control problem of power grids has become increasingly complex, especially in the context of high penetration rates of fluctuating new energy sources such as wind and solar power. Most traditional frequency control methods rely on fixed virtual inertia and damping coefficients. While these fixed parameters are effective in controlling frequency oscillations, they may not be able to adequately adapt to the dynamic changes of the system when faced with complex and variable load fluctuations, system disturbances, and sudden events, leading to decreased system stability and even frequency exceeding limits or system collapse. Therefore, how to dynamically adjust virtual inertia and damping coefficients according to the real-time system state to maintain the frequency stability of the power system has become a key research direction in the field of modern power system frequency control.

[0003] For scenarios involving high-penetration renewable energy integration, frequent load fluctuations, and frequent extreme operating conditions, existing technologies suffer from multiple shortcomings. Currently, the design of virtual inertia and damping coefficients in grid-connected converters mainly relies on expert experience. Under extreme scenarios such as sudden increases or decreases in photovoltaic and wind power output, industrial load start-ups and shutdowns, black starts, or DC blocking, problems such as frequency overshoot, slow oscillation decay, or delayed recovery can easily occur, and even risks such as frequency exceeding limits and system collapse may arise. To avoid over-reliance on expert experience, some researchers use heuristic algorithms to optimize the parameters of virtual synchronous generators (VSGs). Generally, heuristic algorithms can not only achieve different optimization effects based on the objective function, but also flexibly handle various constraints, making them more widely applicable than methods based on expert experience. Nevertheless, heuristic algorithms still have their limitations: their iterative process requires a mathematical model of the system, but power systems typically have complex topologies and many uncertainties, making it a very challenging task to establish an accurate power system model; in addition, heuristic algorithms are resource-intensive, requiring a large amount of computation time and memory, and placing high demands on hardware capabilities, thus posing challenges in meeting the real-time requirements of power system control.

[0004] Reinforcement learning is a machine learning paradigm in which an agent iteratively interacts with its environment, making decisions based on feedback from past actions, with the ultimate goal of maximizing its cumulative reward over time. Based on this, this invention delves into the optimal control problem of parameter regulation in a model-free scenario for Virtual Synchronous Generators (VSGs) and employs the Trust Region Policy Optimization (TRPO) algorithm. The TRPO optimization process enables the agent to dynamically adjust its strategy based on environmental feedback, thereby coping with complex and dynamic environmental changes. In applications of power grid frequency control, TRPO can continuously learn and optimize its decision-making process to address constantly changing environmental and task requirements. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention proposes an adaptive control method for network converters based on the trust region policy optimization algorithm. By utilizing the constraint optimization characteristics and monotonic improvement mechanism of the TRPO algorithm, it overcomes the shortcomings of heuristic algorithms, such as reliance on system models and insufficient real-time performance, reduces policy optimization time, and significantly improves dynamic response speed and frequency stability accuracy under complex operating conditions.

[0006] To achieve the above objectives, this invention provides an adaptive control method for network-type converters based on a trust-region policy optimization algorithm, comprising: A basic model of a virtual synchronous generator (VSG) is established, and the constraint boundaries of virtual inertia and damping coefficient are defined based on the basic model of the VSG. The adaptive control process of the VSG basic model is constructed as a Markov decision process (MDP), and the corresponding state space, action space and reward function are designed. The Trust Region Policy Optimization (TRPO) algorithm is used to train the agent under the guidance of the state space, action space, and reward function, resulting in a well-trained agent. The trained agent is applied to the actual grid-type converter control to monitor the system status in real time and adaptively adjust the virtual inertia and damping coefficient according to the actions output by the agent.

[0007] Preferably, a basic model of a virtual synchronous generator (VSG) is established, and constraint boundaries for virtual inertia and damping coefficients are defined based on the VSG basic model, including: The rotor motion equation of VSG is established, and the frequency deviation dynamic equation including virtual inertia and damping coefficient is derived near the rated operating point through small signal analysis. By introducing the power angle feedback relationship between VSG and the power grid, the frequency deviation dynamic equation is extended into a standard second-order small-signal model, which quantitatively describes the coupling effect of virtual inertia and damping coefficient on the system oscillation frequency and damping ratio. Based on the second-order small-signal model, with system stability as the objective, the first constraint relationship between virtual inertia and damping coefficient is derived using the damping ratio constraint condition of the second-order system. Based on the grid connection standard's limit on the maximum allowable frequency deviation, and combined with the physical relationship between damping coefficient, frequency deviation, and power deviation, the second constraint relationship between virtual inertia and damping coefficient is derived. By combining the first and second constraint relationships, and taking into account the typical value of the synchronization power coefficient and the frequency change rate requirement in the early stage of the disturbance, the joint constraint boundary of the virtual inertia and damping coefficient is determined.

[0008] Preferably, the state space contains state vectors that can comprehensively describe the dynamic response of the system, and the state vectors are specifically: ; In the formula, Characterizes the degree to which the frequency deviates from the rated value; It characterizes the rate of frequency change and is used to reflect the strength of disturbances and the inertia support capability; , The virtual inertia and damping coefficient applied at the previous moment are respectively; It is a state vector; S It is a set of states.

[0009] Preferably, the action space is the change in virtual inertia and the change in damping coefficient, specifically: ; In the formula, and These represent the changes in virtual inertia and the changes in damping coefficient, respectively. This is the action space.

[0010] Preferably, the reward function is a comprehensive reward function, including: The frequency deviation reward function is used to penalize the degree to which the frequency deviates from the rated value; The frequency change rate reward function is used to constrain the transient frequency change rate of the system under power disturbances; Adjust the time reward function to penalize the duration for which the system frequency has not reached a steady state; A stable reward function is used to provide a positive constant reward value when the system frequency reaches a steady state.

[0011] Preferably, the reward function is as follows: ; In the formula, , and These represent the weighting coefficients for each item; This represents the constant value of the reward given when the system reaches steady state; The time to reach a steady state; R For the comprehensive reward function, t For training time.

[0012] Preferably, the trained agent is applied to the actual grid-type converter control to monitor the system status in real time and adaptively adjust the virtual inertia and damping coefficient according to the actions output by the agent, including: Using the VSG basic model as a simulation environment, the TRPO agent is trained through several interactive iterations with the simulation environment. In each iteration, the agent outputs an action based on the current observed system state and the current strategy. The action includes the change in virtual inertia and the change in damping coefficient. The simulation environment receives and executes corresponding actions, updates the virtual inertia and damping coefficient, transitions the system state to the new state, and calculates an instant reward based on the reward function to feed back to the agent. The agent updates its state, action, and reward data accumulated during the interaction process using a trust domain policy optimization algorithm to maximize the expected cumulative discount reward. Repeat the above process until the agent's policy converges, thus obtaining the optimal policy for adaptive control.

[0013] Compared with the prior art, the present invention has the following advantages and technical effects: This invention proposes an adaptive control method for grid-connected converters based on a trust-region policy optimization algorithm (TRPO), aiming to address the shortcomings of traditional methods in dynamic frequency control through reinforcement learning. The TRPO algorithm, through continuous exploration and learning in the environment by an agent, can autonomously adjust the control strategy based on real-time data to optimize system performance. Without prior modeling, the TRPO algorithm can dynamically and adaptively adjust the virtual inertia and damping coefficient based on real-time signals such as grid frequency deviation and rate of change. Compared to traditional control methods, the control strategy of this invention not only responds to frequency fluctuations more quickly but also stabilizes the system frequency under uncertain environments, making it particularly suitable for scenarios with drastic frequency changes and frequent load fluctuations in modern power systems. Attached Figure Description

[0014] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an adaptive control method for a network converter based on a trust region policy optimization algorithm according to an embodiment of the present invention. Figure 2 This is a reinforcement learning VSG control framework diagram of an adaptive control method for a network converter based on a trust region policy optimization algorithm according to an embodiment of the present invention. Figure 3 This is a schematic diagram illustrating the reinforcement learning principle of an embodiment of the present invention; Figure 4 This is a diagram of the actor-commentator structure of the TPRO algorithm in an embodiment of the present invention. Figure 5 This is a diagram showing the reinforcement learning training results of an embodiment of the present invention; Figure 6 The figure shows the simulation results of an embodiment of the present invention. Detailed Implementation

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0016] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0017] An adaptive control method for network converters based on trust region policy optimization algorithm, such as Figures 1-6 ,include: A basic model of a virtual synchronous generator (VSG) is established, and the constraint boundaries of virtual inertia and damping coefficient are defined based on the basic model of the VSG. The adaptive control process of the VSG basic model is constructed as a Markov decision process (MDP), and the corresponding state space, action space and reward function are designed. The Trust Region Policy Optimization (TRPO) algorithm is used to train the agent under the guidance of the state space, action space, and reward function, resulting in a well-trained agent. The trained agent is applied to the actual grid-type converter control to monitor the system status in real time and adaptively adjust the virtual inertia and damping coefficient according to the actions output by the agent.

[0018] Specifically, this embodiment transforms the optimal adaptive control problem of VSG into an RL task, thereby avoiding the need for complex mathematical models and expert experience. Subsequently, the TRPO algorithm is used to train the agent, enabling it to discover the optimal policy. Traditional VSG optimization objectives focus on mitigating frequency fluctuations while neglecting the optimization of system transient response time. In the reward function design, this embodiment introduces an adjustment time component to incentivize the agent to further refine the policy to enhance system performance.

[0019] Specifically, the following steps are included: Establishing the VSG basic model includes: Step 1: Construct the motion equations of the VSG rotor; The rotor motion equations of a synchronous generator are described as follows: ; In the formula, For mechanical torque, For electromagnetic torque, t For training time.

[0020] Because power and torque satisfy ,and ,Will , Substituting the values, we obtain the VSG rotor motion equations: ; In the formula, and These are the converter's reference active power and output active power, respectively. , These represent the rotor angular velocity and the rated angular velocity, respectively. , These are the virtual inertia coefficient and the damping coefficient, respectively.

[0021] Step 2: Construct the VSG small signal model; Without introducing droop control, the VSG can be equivalently represented by the rotor motion equations of a synchronous generator, letting... Output angular velocity for VSG, The rated angular velocity, For equivalent mechanical power, To output active power, the rotor motion equation can be written as: ; To facilitate small-signal analysis, a deviation is introduced near the rated operating point: ; Substituting this into the above equation and rearranging, we obtain the continuous-time dynamic equation for the frequency deviation: ; In the formula, For angular velocity deviation, This is a power disturbance.

[0022] The above relationship directly reflects the decisive role of virtual inertia and damping on frequency dynamics: power disturbance The angular velocity changes, and the damping term provides negative feedback to suppress the angular velocity deviation, thereby achieving the attenuation of the frequency deviation.

[0023] Step 3: Introduce power angle feedback to establish a second-order small-signal model; If only If considered as an external disturbance, the system is a first-order model, and its stability is characterized by single-pole decay. However, to analyze the electrical oscillation modes that VSG may exhibit under grid-connected / weak grid conditions, it is necessary to further characterize the "power angle-active power" feedback relationship on the electrical side.

[0024] First, define the power angle of the VSG relative to the power grid as follows: When the angular velocity of the power grid is approximately constant... At that time, the deviation of the work angle and the deviation of the angular velocity satisfy: ; In the formula, This represents the rate of change of the power angle deviation.

[0025] Secondly, for typical VSG equivalent potential E Through reactance X Connected grid voltage V The structure has the following active power requirements: ; At work After linearization, we get: ; In the formula, The synchronous power factor reflects the sensitivity of the power angle deviation to active power. For active power deviation, For the angle of work deviation, This is the initial work angle.

[0026] Since this embodiment does not introduce droop control, it can generally be considered that the small signal is below... ,therefore: ; In the formula, This refers to the mechanical power deviation.

[0027] Substituting it into the frequency dynamic equation: ; Combined This yields the standard second-order small-signal model: ; In the formula, It is the second derivative of the angle of work deviation.

[0028] Therefore, after considering the power angle feedback, the VSG system behaves as a typical second-order system, and its oscillation frequency and decay rate will be simultaneously affected. and and grid coupling parameters Joint influence.

[0029] Step 4: Derive the state-space expression and system eigenvalues; Take state variables: Then the small-signal state-space model can be expressed as: ; The corresponding characteristic equation is: ; Further comparison with standard second-order systems By comparison, the natural frequency and damping ratio can be obtained: ; In the formula, For the damping ratio, This is the rated angular frequency.

[0030] In summary, virtual inertia With damping coefficient The impact on system stability can be summarized as follows: The primary factors determining the frequency inertial support capacity and frequency change rate in the initial stage of the disturbance are ( )size, The larger the value, the smaller the rate of frequency change caused by power imbalance, and the "smoother" the frequency response, but at the same time, it will make the dynamic process of the system "slower". It mainly determines the oscillation decay and steady-state convergence speed. The larger the value, the stronger the negative feedback to frequency / angular velocity deviations, resulting in faster oscillation decay, smaller overshoot, and easier system stability. Therefore, both need to be tuned together to achieve a balance between frequency support, response speed, and damping attenuation.

[0031] Step 5: Construct a multi-objective optimization function; Therefore, this embodiment describes the oscillation suppression process of VSG as a multi-objective optimization problem. Based on the optimization objectives, the objective function can be described as: ; In the formula, N This represents the total number of steps. , and These represent the frequency deviation cost, the frequency change rate deviation cost, and the recovery time cost, respectively. , and These are the corresponding weighting coefficients; For time; Let be the total cost function. The frequency deviation at the k-th sampling time is... The deviation of the frequency change rate at the k-th sampling time.

[0032] Step 6: Establish constraints; Therefore, the active power loop of the VSG conforms to the characteristics of a typical second-order system. Its natural oscillation angular frequency is denoted as... Damping ratio, denoted as As shown in the following formula: ; To ensure system stability, the damping ratio of the second-order system must be limited. In the case of an underdamped second-order system, the overshoot... With damping ratio The relationship between them can be represented as: ; Based on the damping ratio range and The constraints can be determined as follows: ; The relationship between damping coefficient, frequency deviation, and power deviation is as follows: ; According to the EN50438 renewable energy grid connection standard, for every 1Hz change in grid frequency, the inverter's output active power should vary within 40% to 100% of its rated capacity. In this embodiment, the inverter's rated capacity is 100kW, therefore the maximum permissible frequency deviation is specified as 1Hz. From the above formula, the range of values ​​for the damping coefficient D is as follows: ; This constraint is used to ensure that the damping is not too small, otherwise the frequency deviation will decay slowly and the oscillation will be obvious, and it is not too large either, otherwise the equivalent control will be too "hard", which may lead to problems such as power sudden change, noise amplification or control saturation.

[0033] Virtual Inertia The derivation of the range is based on constraints, combined with the damping ratio constraint of the second-order system ( Damping coefficient range ( Synchronous power factor ( Typical values ​​are derived by modifying the damping ratio formula. The upper limit is limited by the minimum damping coefficient, which is derived from the correlation between the initial frequency change rate of the disturbance and the maximum power deviation. Therefore, the range of virtual inertia is taken as follows in this embodiment: ; This range satisfies the system stability requirements and adapts to the balance between dynamic response and inertial support, providing a clear constraint boundary for the adaptive parameter optimization of the TRPO algorithm.

[0034] Furthermore, VSG adaptive control is implemented based on the TRPO algorithm, including: Step 1: Construct a Markov decision model; The parameter adjustment process of VSG can be viewed as a sequential decision problem. Furthermore, it satisfies the Markov property, meaning the system's state at the next moment depends only on the current state and is independent of past states. Therefore, the adjustment of virtual inertia and damping coefficients can be expressed as a Markov Decision Process (MDP) and processed using a reinforcement learning algorithm. It consists of several key elements: ; In the formula, S Let be the set of states, representing all possible system states; A The action set represents the actions that the agent can choose in each state; P Transition probability This indicates that after taking action a from state s, the state transitions to state s. probability; R For reward function The immediate benefit of taking action a in state s; and the discount factor. The decay weight of future rewards is used to balance short-term and long-term returns.

[0035] Its goal is to find the optimal strategy. This maximizes the expected value of the cumulative discount reward, starting from any initial state: ; In the formula, To expect cumulative discount returns, For the expectation, For a moment t Instant rewards Discount factor t Power of 1.

[0036] Step 2: Design the state space; The state is obtained through environmental observation. The main characteristic of the dynamic response of the VSG active power loop is its frequency response. To comprehensively describe this response process, the state space is selected as follows: considering that the control quantity is "incremental regulation," in order to satisfy the Markov property and reflect the dynamic inertia of parameter changes, the virtual inertia and damping coefficient of the previous time step are introduced as state components to construct the state vector. : ; In the formula, Characterizes the degree to which the frequency deviates from the rated value; It characterizes the rate of frequency change and is used to reflect the strength of disturbances and the inertia support capability; , The virtual inertia and damping coefficients applied at the previous moment are used to facilitate adaptive updates of the strategy based on the current dynamics.

[0037] Step 3: Design the motion space; In this embodiment, the control variables are the virtual inertia and damping coefficient of the VSG. Therefore, the action space... Defined as the change in virtual inertia and damping coefficient, it can be expressed as: ; In the formula, and These represent the changes in virtual inertia and damping coefficient, respectively.

[0038] Therefore, actual virtual inertia and damping coefficient Available from: ; In the formula, and These represent the initial values ​​of the virtual inertia and the damping coefficient, respectively.

[0039] Step 4: Construct the reward function; The goal of reinforcement learning is to maximize the cumulative discounted reward, making the reward function a key factor in guiding the agent to learn the optimal policy. Existing VSG adaptive control methods mainly focus on optimizing frequency oscillations. However, suppressing oscillations may prolong the system's settling time and negatively impact system stability. Therefore, this embodiment incorporates frequency, rate of change of frequency, and time term into the reward function design to address these considerations. The design method is as follows: Frequency Deviation Reward Function In practical applications, the frequency of a power system needs to operate stably near its rated frequency, and the smaller the deviation, the better. Based on this, the frequency deviation reward function structure in this embodiment is as follows: ; In the formula, The weighting coefficients represent the frequency deviation reward function.

[0040] Rate of change of frequency ( This is used to constrain the rate of transient frequency change of the system under power disturbances, avoiding the problem of excessive transient impact caused by relying solely on damping to suppress frequency deviations. Therefore, the structure of the frequency change rate reward function in this embodiment is as follows: ; In the formula, The weighting coefficients represent the reward function for the rate of change of frequency.

[0041] Adjusting the time reward function To prevent excessively long adjustment times, this embodiment incorporates a time term as part of the reward function. Specifically, for each sampling time during which the system fails to reach steady state, a constant penalty is applied, as shown below: ; In the formula, This represents the penalty coefficient for the time term.

[0042] Stable reward function When the system frequency reaches its steady-state value, adjusting the virtual inertia and damping coefficient will not change the system's operating state. Therefore, this embodiment designs a stability reward function that provides a positive constant reward value when the system reaches a steady state. ; in, This represents the constant value of the reward given when the system reaches steady state. The time to reach a steady state, combined with the above formulas, yields the following reward function: .

[0043] Furthermore, the TRPO algorithm is used to train the agent and simulation verification is performed, including: Step 1: Based on the designed reward function, train the agent of the TRPO algorithm and optimize the control strategy through multiple simulations; Step 2: Input the trained agent into the Simulink simulation environment to verify the effectiveness of the proposed adaptive virtual inertia and damping coefficient control strategy. Step 3: Compare the simulation results of the traditional method with those of this embodiment, and analyze the performance indicators such as frequency fluctuation, recovery time and system stability, as shown in Table 1.

[0044] Table 1 This embodiment aims to address the shortcomings of traditional methods in dynamic frequency control through reinforcement learning technology. The TRPO algorithm, through continuous exploration and learning in the environment by an intelligent agent, can autonomously adjust the control strategy based on real-time data to optimize system performance. Without prior modeling, the TRPO algorithm can dynamically and adaptively adjust the virtual inertia and damping coefficient based on real-time signals such as grid frequency deviation and rate of change. Compared with traditional control methods, the control strategy of this embodiment can not only respond to frequency fluctuations more quickly, but also stabilize the system frequency in uncertain environments, making it particularly suitable for scenarios with drastic frequency changes and frequent load fluctuations in modern power systems.

[0045] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An adaptive control method for a networked converter based on a trust-region policy optimization algorithm, characterized in that, include: A basic model of a virtual synchronous generator (VSG) is established, and the constraint boundaries of virtual inertia and damping coefficient are defined based on the basic model of the VSG. The adaptive control process of the VSG basic model is constructed as a Markov decision process (MDP), and the corresponding state space, action space and reward function are designed. The Trust Region Policy Optimization (TRPO) algorithm is used to train the agent under the guidance of the state space, action space, and reward function, resulting in a well-trained agent. The trained agent is applied to the actual grid-type converter control to monitor the system status in real time and adaptively adjust the virtual inertia and damping coefficient according to the actions output by the agent.

2. The adaptive control method for network converters based on trust region policy optimization algorithm according to claim 1, characterized in that, A basic model of a virtual synchronous generator (VSG) is established, and the constraint boundaries for virtual inertia and damping coefficients are defined based on the VSG basic model, including: The rotor motion equation of VSG is established, and the frequency deviation dynamic equation including virtual inertia and damping coefficient is derived near the rated operating point through small signal analysis. By introducing the power angle feedback relationship between VSG and the power grid, the frequency deviation dynamic equation is extended into a standard second-order small-signal model, which quantitatively describes the coupling effect of virtual inertia and damping coefficient on the system oscillation frequency and damping ratio. Based on the second-order small-signal model, with system stability as the objective, the first constraint relationship between virtual inertia and damping coefficient is derived using the damping ratio constraint condition of the second-order system. Based on the grid connection standard's limit on the maximum allowable frequency deviation, and combined with the physical relationship between damping coefficient, frequency deviation, and power deviation, the second constraint relationship between virtual inertia and damping coefficient is derived. By combining the first and second constraint relationships, and taking into account the typical value of the synchronization power coefficient and the frequency change rate requirement in the early stage of the disturbance, the joint constraint boundary of the virtual inertia and damping coefficient is determined.

3. The adaptive control method for network converters based on trust region policy optimization algorithm according to claim 1, characterized in that, The state space contains state vectors that can comprehensively describe the dynamic response of the system. Specifically, the state vectors are: ； In the formula, Characterizes the degree to which the frequency deviates from the rated value; It characterizes the rate of frequency change and is used to reflect the strength of disturbances and the inertia support capability; , The virtual inertia and damping coefficient applied at the previous moment are respectively; It is a state vector; S It is a set of states.

4. The adaptive control method for network converters based on trust region strategy optimization algorithm according to claim 3, characterized in that, The action space refers to the change in virtual inertia and the change in damping coefficient, specifically: ； In the formula, and These represent the changes in virtual inertia and the changes in damping coefficient, respectively. This is the action space.

5. The adaptive control method for network converters based on trust region strategy optimization algorithm according to claim 4, characterized in that, The reward function is a comprehensive reward function, including: The frequency deviation reward function is used to penalize the degree to which the frequency deviates from the rated value; The frequency change rate reward function is used to constrain the transient frequency change rate of the system under power disturbances; Adjust the time reward function to penalize the duration for which the system frequency has not reached a steady state; A stable reward function is used to provide a positive constant reward value when the system frequency reaches a steady state.

6. The adaptive control method for network converters based on trust region policy optimization algorithm according to claim 5, characterized in that, The reward function is specifically as follows: ； In the formula, , and These represent the weighting coefficients for each item; This represents the constant value of the reward given when the system reaches steady state; The time to reach a steady state; R For the comprehensive reward function, t For training time.

7. The adaptive control method for network converters based on trust region strategy optimization algorithm according to claim 1, characterized in that, The trained agent is applied to the actual grid-type converter control to monitor the system status in real time and adaptively adjust the virtual inertia and damping coefficient based on the actions output by the agent, including: Using the VSG basic model as a simulation environment, the TRPO agent is trained through several interactive iterations with the simulation environment. In each iteration, the agent outputs an action based on the current observed system state and the current strategy. The action includes the change in virtual inertia and the change in damping coefficient. The simulation environment receives and executes corresponding actions, updates the virtual inertia and damping coefficient, transitions the system state to the new state, and calculates an instant reward based on the reward function to feed back to the agent. The agent updates its state, action, and reward data accumulated during the interaction process using a trust domain policy optimization algorithm to maximize the expected cumulative discount reward. Repeat the above process until the agent's policy converges, thus obtaining the optimal policy for adaptive control.

Images