New energy flexible direct output system oscillation suppression method based on double-delay depth deterministic strategy gradient algorithm

Abstract

The invention provides a new energy flexible direct output system oscillation suppression method based on a double-delay depth deterministic strategy gradient algorithm. The method comprises the steps of obtaining dynamic parameters and electrical quantity data of a new energy flexible direct output system, constructing a deep reinforcement learning model, and training the deep reinforcement learning model by using a TD3 algorithm to obtain a trained deep reinforcement learning model; current state space data of the new energy flexible direct output system are input into the trained deep reinforcement learning model, the trained deep reinforcement learning model outputs additional damping control parameters, the additional damping control parameters are applied to an additional damping controller, the parameters of the additional damping controller are adjusted online, and the new energy flexible direct output system is obtained. And an optimal control strategy of the new energy flexible direct delivery system is generated. The invention provides an oscillation suppression control method for a new energy flexible direct output system, and the method achieves the adaptive adjustment of additional damping control parameters through a TD3 algorithm, and achieves the adaptive suppression of system oscillation.

Description

Technical Field

[0001] The present invention relates to the field of power system control technology, and in particular to a method for suppressing oscillations in a new energy flexible direct current transmission system based on a TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm. Background Art

[0002] With the large-scale integration of renewable energy, flexible direct current (HVDC) transmission has become an important means of renewable energy transmission. However, dynamic interactions in the transmission system can lead to broadband oscillations, seriously threatening the safe and stable operation of the transmission system.

[0003] Currently, existing methods for suppressing oscillations in power transmission systems primarily include parameter optimization, nonlinear controller replacement, and additional damping control. However, these methods often employ a fixed set of parameters, making them difficult to adapt to changing system operating conditions. Therefore, there is an urgent need for an oscillation suppression method that can adaptively adjust control parameters. Summary of the Invention

[0004] An embodiment of the present invention provides a method for suppressing oscillations of a new energy system via flexible direct current transmission based on a TD3 algorithm, so as to effectively suppress oscillations of the new energy system via flexible direct current transmission.

[0005] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions.

[0006] A method for suppressing oscillations in a new energy flexible direct current transmission system based on a double-delayed deep deterministic policy gradient TD3 algorithm includes:

[0007] Acquire dynamic parameters and electrical quantity data of the flexible direct current transmission system for renewable energy, including DC side voltage and current and AC side D-axis voltage data;

[0008] A deep reinforcement learning model is constructed based on the dynamic parameters and electrical quantity data of the new energy flexible direct current transmission system, and a state space set, an action space set, and a reward function of the deep reinforcement learning model are defined; the deep reinforcement learning model is trained using the TD3 algorithm to obtain a trained deep reinforcement learning model;

[0009] The current state space data of the new energy direct current transmission system is input into the trained deep reinforcement learning model, and the trained deep reinforcement learning model outputs additional damping control parameters. The additional damping control parameters are applied to the additional damping controller, and the parameters of the additional damping controller are adjusted online to generate the optimal control strategy for the new energy direct current transmission system.

[0010] Preferably, the construction of a deep reinforcement learning model based on the dynamic parameters and electrical quantity data of the new energy flexible direct current transmission system includes:

[0011] Constructing a deep reinforcement learning model based on a deep neural network structure comprising an input layer, a hidden layer, and an output layer, the deep reinforcement learning model comprising a state space, an action space, and a reward function, wherein the state space comprises the system's voltage, current, and power electrical quantities, the action space comprises control parameters of an additional damping controller, and the reward function is designed based on system performance indicators;

[0012] The deep reinforcement learning model is used as an adaptive optimizer of the additional damping controller. The deep reinforcement learning model extracts system features through a multi-layer deep neural network structure and outputs additional damping control parameters.

[0013] Preferably, the state space set of the deep reinforcement learning model is defined as follows:

[0014] (1) For the VSC1 agent, its state space is defined as follows:

[0015]

[0016] in:

[0017] -v dc1 is the DC voltage of VSC1

[0018] -Δv dc1 (t) is v dc1 Deviation from its reference value

[0019] -i dc1 (t) is the input DC current of the new energy output

[0020] (2) For the VSC2 agent, its state space is defined as follows:

[0021] in:

[0022] -v Ld2 is the d-axis output voltage of VSC2

[0023] -Δv Ld2 v Ld2 Deviation from its reference value

[0024] - v Ld2 Reference value

[0025] The action space of the deep reinforcement learning model is defined as follows:

[0026] Action space of VSC1:

[0027] Action space of VSC2:

[0028] Where K is the gain, and T1 and T2 are time constants.

[0029] The reward function of the deep reinforcement learning model is defined as follows:

[0030] The damping ratio based on the dominant oscillation mode of the system is defined as follows:

[0031] r t VSC1 =min[ξ VSC1 (t)-ξ * ,0]

[0032] r t VSC2 =min[ξ VSC2 (t)-ξ * ,0]

[0033] where ξ VSC1 is the system damping ratio when VSC1 additional damping control is implemented, ξ * is the target damping ratio, and the reward function r of VSC2 t VSC2 The definition is similar.

[0034] Reward calculation based on the damping ratio of the system's dominant oscillation mode

[0035]

[0036] Among them, ξ(t) is the system damping ratio at the current moment, ξ target is the target damping ratio.

[0037] When the system damping ratio ξ VSC1 When it is greater than the target value, the reward is 0

[0038] When the system damping ratio ξ VSC1 When the value is less than the target, a negative reward is given.

[0039] Preferably, the use of the TD3 algorithm to train the deep reinforcement learning model to obtain a trained deep reinforcement learning model includes:

[0040] Initialize the actor network parameters, critic network parameters, target network parameters, and experience replay buffer of the TD3 algorithm. For each training round, select an action based on the current policy and add exploration noise, execute the action and observe the reward and next state, store the transition in the experience replay buffer, sample batch transitions from the buffer for training, update the critic network and actor network, and soft-update the target network.

[0041] The TD3 algorithm uses two critic networks to estimate Q values, uses the minimum Q value to form the target value, adds noise to the target policy action and performs pruning, delays updating the policy network and target network until the deep reinforcement learning model converges, and obtains a trained deep reinforcement learning model.

[0042] Preferably, the current state space data of the new energy direct current transmission system is input into the trained deep reinforcement learning model, the trained deep reinforcement learning model outputs additional damping control parameters, the additional damping control parameters are applied to the additional damping controller, the parameters of the additional damping controller are adjusted online, and the optimal control strategy of the new energy direct current transmission system is generated, including:

[0043] Obtain the current state space data s of the renewable energy flexible direct current transmission system, which includes the renewable energy equivalent direct current, the VSC direct current side voltage, and the AC side voltage;

[0044] The state space data s is input into the trained deep reinforcement learning model, and the trained deep reinforcement learning model calculates the additional damping control parameter a←π φ (s), the additional damping control parameter a includes a gain and a time constant;

[0045] The additional damping control parameter a is applied to the additional damping controller, the output signal of the additional damping controller is applied to the DC voltage control loop or the AC voltage control loop of the VSC, and the additional damping control parameter a of the additional damping controller is adjusted online. After the deep reinforcement learning model evaluates and makes decisions on the system status in real time, continuous adjustments are made through online optimization to achieve the optimal operating state of the system and generate the optimal control strategy for the new energy transmission system via flexible direct current.

[0046] It can be seen from the technical solutions provided by the above embodiments of the present invention that the present invention provides an oscillation suppression control method for a new energy flexible direct current transmission system, and an additional damping controller is designed through the TD3 algorithm to achieve adaptive suppression of system oscillations.

[0047] Additional aspects and advantages of the present invention will be set forth in part in the following description, will be obvious from the following description, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A system topology diagram of a new energy flexible direct current transmission system provided by an embodiment of the present invention;

[0050] Figure 2 A processing flow chart of a method for implementing adaptive adjustment of additional damping control parameters provided by an embodiment of the present invention;

[0051] Figure 3 A network structure diagram of a TD3 algorithm provided in an embodiment of the present invention;

[0052] Figure 4 A comparison diagram of oscillation suppression effects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0054] It will be understood by those skilled in the art that, unless expressly stated otherwise, the singular forms "a", "an", "said" and "the" used herein may also include the plural forms. It should be further understood that the term "comprising" used in the description of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, "connected" or "coupled" as used herein may include wireless connections or couplings. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0055] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art in the art to which the present invention pertains. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with their meanings in the context of the prior art and, unless defined as such herein, will not be interpreted in an idealized or overly formal sense.

[0056] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0057] The research object of this invention is a new energy flexible direct current transmission system. The structure of the new energy flexible direct current transmission system is as follows: Figure 1 As shown, it includes: a renewable energy converter VSC1, a VSC-HVDC (Voltage Source Converter-High Voltage Direct Current, voltage source converter type high voltage direct current transmission) transmitting end converter VSC2 and a receiving end converter VSC3. VSC1 adopts VdcIq (DC Voltage and Q-axis Current Control, DC voltage and q-axis current control) control, VSC2 adopts Vf (voltage-frequency control) control, and VSC3 adopts VdcIq (DC voltage and q-axis current control) control. The present invention installs additional damping controllers on VSC1 and VSC2. The adaptive additional damping control can adaptively adjust its own internal parameters according to real-time working conditions to achieve effective suppression of oscillations.

[0058] The embodiment of the present invention provides a method for suppressing oscillation of a new energy flexible direct current transmission system based on the TD3 algorithm. Figure 2 As shown, the processing steps include the following:

[0059] Step S10: Acquire dynamic parameters and electrical quantity data of the flexible direct current transmission system for renewable energy, wherein the electrical quantity data include DC side voltage and current and AC side d-axis voltage and other data.

[0060] Step S20: Build a deep reinforcement learning model based on the dynamic parameters and electrical quantity data of the new energy flexible direct current transmission system, and define the state space set, action space set and reward function of the deep reinforcement learning model.

[0061] Step S30: Use the TD3 algorithm to train the deep reinforcement learning model until convergence.

[0062] Step S40: Use the TD3 algorithm to design an adaptive additional damping controller, adjust the parameters of the additional damping controller online through the trained deep reinforcement learning model, and generate the optimal control strategy for the new energy direct current transmission system to achieve system oscillation suppression.

[0063] Step S10 includes the following: In practical applications, a PMU (phasor measurement unit) can be used to obtain dynamic parameters and electrical quantity data of the flexible DC transmission system for renewable energy. PMUs are widely used for data acquisition in flexible DC transmission systems for renewable energy and are typically installed at locations such as the AC busbar of renewable energy stations, the AC side of VSC converter stations, key system nodes, and grid access points. They can synchronously collect key electrical quantity data such as voltage / current phasors (including amplitude and phase angle), frequency and frequency change rate, active / reactive power, and sequence components. The sampling frequency is typically ≥50 times / second, and synchronization is ensured to microsecond-level synchronization accuracy through a satellite positioning system.

[0064] Step S20 includes the following: The deep reinforcement learning model is a deep neural network structure, consisting of three core elements: a state space, an action space, and a reward function. The state space includes key electrical quantities such as voltage, current, and power; the action space includes the control parameters of the additional damping controller; and the reward function is designed based on system performance indicators such as stability. The deep reinforcement learning model uses a deep neural network as a value function approximator or policy function, extracting system features and outputting a control strategy through a multi-layer network structure.

[0065] The deep reinforcement learning model in this embodiment of the present invention can serve as an adaptive optimizer for the additional damping controller. It continuously optimizes control parameters by learning the system's dynamic characteristics in real time, enabling the damping controller to adapt to changes in the system's operating state. The output of the deep reinforcement learning model can be used directly as the control signal for the damping controller or to dynamically adjust its parameters, such as the gain coefficient and time constant.

[0066] The process of building this deep reinforcement learning model involves first preprocessing the raw data collected by devices like PMUs, including cleaning, normalization, and time alignment. Key features (such as voltage, current, and power) are then extracted from the preprocessed data as state-space inputs. Based on this, a deep reinforcement learning model is designed, consisting of a deep neural network structure with input, hidden, and output layers. After the model is built, offline training is performed using historical operating data to establish an initial policy model. Once the model is deployed, the control policy is continuously optimized and updated using real-time data. Control effectiveness is continuously monitored, and model parameters are dynamically adjusted based on system response. This ultimately forms a closed-loop adaptive learning mechanism to continuously improve control performance.

[0067] The present invention adopts the TD3 algorithm to design an adaptive additional damping controller, and the system operating condition parameters in the state space include: new energy equivalent DC current, VSC DC side voltage and AC side voltage.

[0068] The state space set defining the deep reinforcement learning model is as follows:

[0069] (1) For the VSC1 agent, its state space is defined as follows:

[0070] s t VSC1 ={v dc1 (t),Δv dc1 (t),i dc1 (t)}

[0071] in:

[0072] -v dc1 is the DC voltage of VSC1

[0073] -Δv dc1 (t) is v dc1 Deviation from its reference value

[0074] -i dc1 (t) is the input DC current of the new energy output

[0075] (2) For the VSC2 agent, its state space is defined as follows:

[0076] in:

[0077] -v Ld2 is the d-axis output voltage of VSC2

[0078] -Δv Ld2 v Ld2 Deviation from its reference value

[0079] - v Ld2 Reference value

[0080] The action space of a deep reinforcement learning model is defined as follows:

[0081] Action space of VSC1:

[0082] Action space of VSC2:

[0083] Where K is the gain, and T1 and T2 are time constants.

[0084] The reward function of the deep reinforcement learning model is defined as follows:

[0085] The damping ratio based on the dominant oscillation mode of the system is defined as follows:

[0086] r t VSC1 =min[ξ VSC1 (t)-ξ * ,0]

[0087] r t VSC2 =min[ξ VSC2 (t)-ξ * ,0]

[0088] where ξ VSC1 is the system damping ratio when VSC1 additional damping control is implemented, ξ * is the target damping ratio. The reward function r of VSC2 t VSC2 The definition is similar.

[0089] Reward calculation based on the damping ratio of the system's dominant oscillation mode

[0090]

[0091] Among them, ξ(t) is the system damping ratio at the current moment, ξ target is the target damping ratio.

[0092] When the system damping ratio ξ VSC1 When it is greater than the target value, the reward is 0

[0093] When the system damping ratio ξ VSC1 When the value is less than the target, a negative reward is given.

[0094] The system eigenvalues ​​are calculated using the frequency domain impedance network modeling method and used to calculate the reward function.

[0095] The above step S30 includes:

[0096] Figure 3 This is a network structure diagram of a TD3 algorithm provided by an embodiment of the present invention. The implementation steps of the TD3 algorithm include: The key to the TD3 algorithm is to maintain two critic networks to estimate Q values ​​and use the minimum value of these two networks to form the target, thereby reducing overestimation bias. The specific steps are as follows:

[0097] (1) Initialization:

[0098] - Initialize the actor network parameters θ and the critic network parameters φ1 and φ2

[0099] - Initialize the target network parameters θ', φ1' and φ2'

[0100] - Initialize the experience replay buffer R

[0101] (2) For each training round: select an action based on the current policy and add exploration noise, execute the action and observe the reward and next state, store the transition in the experience replay buffer, sample batches of transitions from the buffer for training, update the critic network and actor network, and soft-update the target network.

[0102] a) Initialize the power system

[0103] b) For each time step t:

[0104] -Select an action with exploration noise: a←π φ (s)+ε,

[0105] -Perform an action and observe the reward r and next state s'

[0106] -Store a state transition process (s,a,r,s′) in R

[0107] - Sample N state transition processes (s i ,a i ,r i ,s i ′)

[0108] -Calculate the target Q value:

[0109]

[0110] where ε is the noise smoothed by the target policy

[0111] - Updated critic network:

[0112]

[0113] - Update the actor network every d steps:

[0114]

[0115] -Soft update target network:

[0116] φ′←τφ+(1-τ)φ′

[0117] θ i ′←τθ i +(1-τ)θ i ′for i=1,2

[0118] The process of training a deep reinforcement learning model using the TD3 algorithm involves establishing two critic networks to estimate Q values, using the minimum Q value to form the target value, reducing overestimation bias, adding noise to the target policy action and pruning it, and delaying updates to the policy and target networks until the deep reinforcement learning model converges, resulting in a trained deep reinforcement learning model. Data from different operating conditions and disturbance levels is used during the training phase to improve the controller's adaptability.

[0119] The convergence of deep reinforcement learning models is mainly judged by monitoring the changing trend of the reward function value, the loss function value of the policy network, and the key performance indicators of the system (such as transition time, overshoot, steady-state error, etc.); when the fluctuation range of these indicators is lower than the preset threshold for multiple consecutive training cycles and the dynamic response characteristics of the system meet the control requirements, the model can be considered to have reached convergence. At the same time, it is also necessary to verify the generalization ability of the model through cross-validation and testing under different working conditions to ensure that the model can maintain a stable control effect under various operating conditions. In practical applications, multiple convergence criteria can be set, and the model is considered to have fully converged only when all criteria are met.

[0120] The above step S40 includes:

[0121] After the deep reinforcement learning model is trained, the added damping controller can be run online based on real-time measurement data:

[0122] The input data for the trained deep reinforcement learning model is the previously defined state space data measured online, namely, the equivalent DC current of the renewable energy source, the DC side voltage of the VSC, and the AC side voltage. The output data is the previously defined action space data, namely, the control parameters of the additional damping controller, including the gain and time constant.

[0123] (1) Obtain the current state space data s of the system, which includes the equivalent DC current of the new energy source, the DC side voltage of the VSC, and the AC side voltage.

[0124] (2) The above state space data s is input into the trained deep reinforcement learning model, and the trained deep reinforcement learning model calculates the additional damping control parameter a←π φ (s), the additional damping control parameter a includes gain and time constant.

[0125] (3) Apply the additional damping control parameter a to the additional damping controller, apply the output signal of the additional damping controller to the DC voltage control loop or AC voltage control loop of the VSC, and adjust the additional damping control parameter a of the additional damping controller online. Through the real-time evaluation and decision-making of the system state by the deep reinforcement learning model, the dynamic characteristics, stability requirements and operation constraints of the system are comprehensively considered, and continuous adjustment is made through online optimization to achieve the optimal operating state of the system, and generate the optimal control strategy for the new energy transmission system through flexible direct current to achieve system oscillation suppression.

[0126] Verification of control effect, Figure 4 A comparison chart of oscillation suppression effects provided by an embodiment of the present invention. The method of the present invention was verified under three different working conditions:

[0127] Working condition 1:i dc1 =1000A,R Load =0.1Ω→0.05Ω,

[0128] Working condition 2:i dc1 =900A,R Load =0.1Ω→0.05Ω,

[0129] Working condition 3:i dc1 =1100A,R Load =0.1Ω→0.15Ω,

[0130] like Figure 4 As shown in the figure, simulation results show that the proposed method has better oscillation suppression effect than traditional fixed parameter methods and DQN (Deep Q-Network) methods, manifested in smaller overshoot and faster settling time. This verifies that the proposed method has good adaptability to changing working conditions.

[0131] The method of the present invention can be applied to multiple VSC converters in the system simultaneously to achieve coordinated control. The method of the present invention is suitable for suppressing low-frequency oscillation, subsynchronous oscillation and high-frequency oscillation.

[0132] In summary, the present invention provides an oscillation suppression control method for a new energy flexible direct current transmission system, which realizes adaptive adjustment of additional damping control parameters through the TD3 algorithm to achieve adaptive suppression of system oscillations.

[0133] The present invention adopts a dual-critic network structure to effectively solve the Q-value over-estimation problem and improve the training effect. The reward function design based on model driving fully utilizes the system damping characteristic information to accelerate the training process. The actual application results show that compared with the traditional fixed parameter method, the method of the present invention has better oscillation suppression effect.

[0134] The method proposed in this paper evaluates the system's damping ratio through the design of a model-based reward function, combined with the calculation of system eigenvalues. This effectively guides the agent to adjust controller parameters and achieve adaptive suppression of oscillations under different operating conditions. By employing a two-stage training mechanism, a single agent is initially trained in a model-driven manner, followed by coordinated multi-agent training using a data-driven reward function, improving overall control effectiveness. Compared with existing methods, this method offers significant advantages in oscillation suppression response speed, adaptability, and control accuracy.

[0135] Those skilled in the art will appreciate that the accompanying drawings are merely schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0136] From the above description of the embodiments, it can be seen that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus the necessary general-purpose hardware platform. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present invention or certain parts of the embodiments.

[0137] Each embodiment in this specification is described in a progressive manner. The same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, refer to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without making any creative efforts.

[0138] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for suppressing oscillation of a new energy flexible direct current transmission system based on a double-delayed deep deterministic policy gradient algorithm, characterized in that: include: Obtain dynamic parameters and electrical quantity data of the new energy flexible direct current transmission system, wherein the electrical quantity data includes the voltage and current on the direct current side and the d-axis voltage data on the alternating current side; Based on the dynamic parameters and electrical quantity data of the new energy flexible direct current transmission system, a deep reinforcement learning model is constructed, and the state space set, action space set and reward function of the deep reinforcement learning model are defined; the deep reinforcement learning model is trained using the TD3 algorithm to obtain a trained deep reinforcement learning model; The current state space data of the new energy via flexible direct current transmission system is input into the trained deep reinforcement learning model, the trained deep reinforcement learning model outputs additional damping control parameters, the additional damping control parameters are applied to the additional damping controller, the parameters of the additional damping controller are adjusted online, and the optimal control strategy of the new energy via flexible direct current transmission system is generated.

2. The method according to claim 1, characterized in that The deep reinforcement learning model constructed based on the dynamic parameters and electrical quantity data of the new energy flexible direct current transmission system includes: Constructing a deep reinforcement learning model based on a deep neural network structure including an input layer, a hidden layer, and an output layer, wherein the deep reinforcement learning model includes a state space, an action space, and a reward function, wherein the state space includes voltage, current, and power electrical quantities of the system, the action space includes control parameters of an additional damping controller, and the reward function is designed based on system performance indicators; The deep reinforcement learning model is used as an adaptive optimizer of the additional damping controller. The deep reinforcement learning model extracts system features through a multi-layer deep neural network structure and outputs additional damping control parameters.

3. The method according to claim 2, characterized in that: The state space set of the deep reinforcement learning model is defined as follows: (1) For the VSC1 agent, its state space is defined as follows: s t VSC1 ={v dc1 (t),Δv dc1 (t),i dc1 (t)} in: -v dc1 is the DC voltage of VSC1 -Δv dc1 (t) is v dc1 Deviation from its reference value -i dc1 (t) is the input DC current of the renewable energy output (2) For the VSC2 agent, its state space is defined as follows: in: -v Ld2 is the d-axis output voltage of VSC2 -Δv Ld2 v Ld2 Deviation from its reference value - v Ld2 Reference value The action space of the deep reinforcement learning model is defined as follows: Action space of VSC1: Action space of VSC2: Where K is the gain, T1 and T2 are time constants. The reward function of the deep reinforcement learning model is defined as follows: The damping ratio based on the dominant oscillation mode of the system is defined as follows: r t VSC1 =min[ξ VSC1 (t)-ξ * ,0] r t VSC2 =min[ξ VSC2 (t)-ξ * ,0] where ξ VSC1 is the system damping ratio when VSC1 additional damping control is implemented, ξ * is the target damping ratio, and the reward function r of VSC2 t VSC2 The definition is similar. Reward calculation based on the damping ratio of the dominant oscillation mode of the system Among them, ξ(t) is the system damping ratio at the current moment, ξ target is the target damping ratio. When the system damping ratio ξ VSC1 When it is greater than the target value, the reward is 0 When the system damping ratio ξ VSC1 When it is less than the target value, a negative reward is given.

4. The method according to claim 3, characterized in that The method of using the TD3 algorithm to train the deep reinforcement learning model to obtain a trained deep reinforcement learning model includes: Initialize the actor network parameters, critic network parameters, target network parameters and experience replay buffer of the TD3 algorithm. For each training round, select actions based on the current strategy and add exploration noise, execute actions and observe rewards and next states, store transfers in the experience replay buffer, sample batch transfers from the buffer for training, update the critic network and actor network, and soft-update the target network. The TD3 algorithm uses two critic networks to estimate Q values, uses the minimum Q value to form the target value, adds noise to the target policy action and prunes it, delays updating the policy network and the target network until the deep reinforcement learning model converges, and obtains a trained deep reinforcement learning model.

5. The method according to claim 4, characterized in that The method of inputting the current state space data of the new energy through flexible direct current transmission system into the trained deep reinforcement learning model, the trained deep reinforcement learning model outputting additional damping control parameters, applying the additional damping control parameters to the additional damping controller, adjusting the parameters of the additional damping controller online, and generating the optimal control strategy of the new energy through flexible direct current transmission system includes: Obtain the current state space data s of the new energy flexible direct current transmission system, where the state space data s includes the new energy equivalent direct current, the VSC direct current side voltage and the alternating current side voltage; The state space data s is input into the trained deep reinforcement learning model, and the trained deep reinforcement learning model calculates the additional damping control parameter a←π φ (s), the additional damping control parameter a includes a gain and a time constant; The additional damping control parameter a is applied to the additional damping controller, the output signal of the additional damping controller is applied to the DC voltage control loop or the AC voltage control loop of the VSC, the additional damping control parameter a of the additional damping controller is adjusted online, and the system state is evaluated and decided in real time by a deep reinforcement learning model. Through online optimization, continuous adjustments are made to achieve the optimal operating state of the system, and the optimal control strategy for the new energy flexible direct current transmission system is generated.

Images