New energy grid-connected system oscillation coordination suppression method based on MA-DDPG algorithm

Abstract

The invention provides a new energy grid-connected system oscillation coordination suppression method based on an MA-DDPG algorithm. The method comprises the following steps: acquiring system parameters of a new energy flexible direct delivery system; the method comprises the following steps: constructing a multi-agent deep reinforcement learning model based on system parameters of a new energy flexible direct delivery system, and defining a state space set, an action space set and a reward function of agents; training the multi-agent deep reinforcement learning model by using an MA-DDPG algorithm until convergence, and obtaining a trained multi-agent deep reinforcement learning model; and adjusting parameters of the additional damping controller on line through the trained multi-agent deep reinforcement learning model, generating an optimal control strategy of the new energy flexible direct delivery system, and performing system oscillation suppression by using the optimal control strategy. The flexible and direct energy delivery system can coordinate operation of a plurality of converters, has higher adaptive capacity and better oscillation suppression effect, and can keep stable operation of the system under various complex working conditions.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and in particular to a method for coordinated suppression of oscillation of a new energy grid-connected system based on a MA-DDPG algorithm. Background Art

[0002] With the growing demand for clean energy worldwide, the access ratio of renewable energy such as wind and solar energy is increasing year by year. Flexible direct current transmission technology has gradually become an important technology for transmitting renewable energy to load centers due to its advantages in long-distance and large-scale transmission. However, the interaction between multiple converters in the flexible direct current transmission system of new energy can easily cause broadband oscillations, which cover the low-frequency to high-frequency range and may seriously affect the operational stability of the system.

[0003] At present, the oscillation suppression methods in the prior art mainly include the following schemes:

[0004] 1. Single converter control: Most oscillation suppression methods are optimized for a single converter and cannot effectively deal with the dynamic interactions and oscillations between multiple converters.

[0005] 2. Fixed parameter control: Traditional supplementary damping control methods rely on pre-set fixed parameters, which cannot be automatically adjusted when the system operating conditions change, resulting in poor control effects.

[0006] 3. Lack of coordinated control between converters: Due to the dynamic interaction of multiple converters in the flexible direct current transmission system for renewable energy, adjusting the control parameters of a certain converter alone may cause new oscillations in other converters. It is difficult for existing control methods to coordinate the oscillation suppression strategies of multiple converters.

[0007] As the penetration rate of renewable energy in power systems increases, system operating conditions become increasingly complex, and existing single-agent and fixed-parameter control methods are difficult to meet the needs of dynamic changes. Therefore, an intelligent control method that can coordinate multiple converters is needed to ensure stable operation of the system under various operating conditions. Summary of the invention

[0008] An embodiment of the present invention provides a method for coordinated suppression of oscillations in a new energy grid-connected system based on a multi-agent deep deterministic policy gradient (MA-DDPG) algorithm.

[0009] In order to achieve the above object, the present invention adopts the following technical scheme.

[0010] A method for coordinated suppression of oscillation of a renewable energy grid-connected system based on a MA-DDPG algorithm, comprising:

[0011] Obtain system parameters of the new energy flexible direct current transmission system;

[0012] Based on the system parameters of the new energy flexible direct current transmission system, a multi-agent deep reinforcement learning model is constructed to define the state space set, action space set and reward function of the agent;

[0013] Use the MA-DDPG algorithm to train the multi-agent deep reinforcement learning model until convergence to obtain a trained multi-agent deep reinforcement learning model;

[0014] The parameters of the additional damping controller are adjusted online through the trained multi-agent deep reinforcement learning model to generate the optimal control strategy for the new energy flexible direct current transmission system, and the system oscillation is suppressed using the optimal control strategy.

[0015] Preferably, the method of obtaining system parameters of the new energy flexible direct current transmission system includes:

[0016] A new energy flexible direct current transmission system is provided, including: a new energy rectifier VSC1, a flexible direct current sending-end inverter VSC2 and a receiving-end converter VSC3. VSC1 adopts VdcIq control, VSC2 adopts Vf control, and VSC3 adopts VdcIq control. Additional damping controllers are installed on VSC1 and VSC2 to obtain system parameters of the new energy flexible direct current transmission system. The system parameters include dynamic parameters and electrical quantity data. The electrical quantity data include DC side voltage and current and AC side d-axis voltage data.

[0017] Preferably, the multi-agent deep reinforcement learning model is constructed based on the system parameters of the new energy flexible direct current transmission system, and the state space set, action space set and reward function of the agent are defined, including:

[0018] A multi-agent deep reinforcement learning model is constructed based on the system parameters of the new energy flexible direct current transmission system. The multi-agent deep reinforcement learning model includes two agents, Agent 1 and Agent 2. Agent 1 is connected to VSC 1 and serves as an additional damping controller of VSC 1. Agent 2 is connected to VSC 2 and serves as an additional damping controller of VSC 2.

[0019] The state space of each agent includes: control reference value, deviation and system operation status. The state space of agent Agent1 is defined as:

[0020]

[0021] where v dc1 is the DC voltage of VSC1, Δv dc1 (t) is v dc1 Deviation from its reference value, i dc1 (t) Input DC current for renewable energy output;

[0022] The state space of Agent2 is defined as:

[0023]

[0024] where v Ld2 is the d-axis output voltage of VSC2, Δv Ld2 v Ld2 Deviation from its reference value, v Ld2 Reference value of

[0025] The action space of each agent is the parameters of the additional damping controller;

[0026] The action space of agent VSC1 is:

[0027] The action space of agent VSC2 is:

[0028] Where K is the gain, T1 and T2 are time constants;

[0029] Design a data-driven reward function:

[0030] r t =-|Δv dc1 (t)|-|Δv Ld2 (t)|

[0031] Where Δv dc1 (t) is the deviation of the DC voltage of VSC1 from its reference value, Δv Ld2 (t) is the deviation of the d-axis output voltage of VSC2 from its reference value.

[0032] Preferably, the multi-agent deep reinforcement learning model is trained using the MA-DDPG algorithm until convergence to obtain a trained multi-agent deep reinforcement learning model, including:

[0033] In the collaborative training of the multi-agent deep reinforcement learning model, the MA-DDPG algorithm is used to achieve coordinated control between converters. The input of the training process includes the state space information of each agent: for Agent1 (VSC1), including the DC voltage v dc1 and its deviation Δv dc1 、Input DC current i dc1 and output active power P1; for Agent2 (VSC2), including d-axis output voltage v Ld2 and its deviation Δv Ld2and output active power P2. The output of the training is the action space of each agent, that is, the parameters of the additional damping controller, including the parameters K of the VSC1 controller VSC1 (t),T1 VSC1 (t),T2 VSC1 (t) and the parameter K of the VSC2 controller VSC2 (t),T1 VSC2 (t),T2 VSC2 (t);

[0034] The MA-DDPG algorithm adopts a centralized training and decentralized execution framework for training. In the centralized training phase, the two agents share the same experience replay pool to store interaction data samples (s t ,a t ,r t ,s t+1 ), and share a reward function r based on the electrical quantity measurement value. Each agent contains an Actor network and a Critic network, where the Actor network is responsible for outputting control actions based on the current state, and the Critic network is responsible for evaluating the value of the current state-action pair;

[0035] During the training process, the Actor and Critic networks of the two agents interact to update network parameters. The Actor network considers the influence of the other agent's actions when updating, and the Critic network takes the states and actions of the two agents as input when evaluating.

[0036] Coordinated training of multiple inverters using a shared data-driven reward function.

[0037] During the training of the multi-agent deep reinforcement learning model, the agents of VSC1 and VSC2 share the same data-driven reward function r t , the reward function is built based on real-time measurement data, and the calculation formula is:

[0038] r t =-|Δv dc1 (t)|-|Δv Ld2 (t)|

[0039] Where Δv dc1 (t) represents the DC voltage deviation of VSC1 at time t, Δv Ld2 (t) represents the d-axis voltage deviation of VSC2 at time t, and the negative sign indicates that the smaller the deviation, the greater the reward value;

[0040] The training termination of the multi-agent deep reinforcement learning model requires the following conditions to be met at the same time: first, the cumulative reward values ​​of the two agents have reached convergence, that is, the change in the cumulative reward values ​​of multiple consecutive training rounds is less than the preset threshold; second, the oscillation suppression effect of the system under different operating conditions meets the standard; third, the control parameters output by the two agents remain relatively stable in multiple consecutive training rounds, and the change in the parameters is less than the set threshold;

[0041] When the above conditions are met at the same time, the multi-agent deep reinforcement learning model training is completed, and a trained multi-agent deep reinforcement learning model is obtained. Preferably, the parameters of the additional damping controller are adjusted online through the trained multi-agent deep reinforcement learning model to generate the optimal control strategy of the new energy flexible direct current transmission system, and the optimal control strategy is used to suppress system oscillations, including:

[0042] The input data of the trained multi-agent deep reinforcement learning model includes the state space information of each agent: for Agent1 (VSC1), it includes the DC voltage v dc1 and its deviation Δv dc1 、Input DC current i dc1 and output active power P1; for Agent2 (VSC2), including d-axis output voltage v Ld2 and its deviation Δv Ld2 And output active power P2;

[0043] The multi-agent deep reinforcement learning model normalizes the input data to make it meet the numerical range set during training, and then inputs the input data into the Actor networks corresponding to VSC1 and VSC2 respectively. Based on the current state, the Actor networks obtain their respective optimal control strategies through forward calculation. The optimal control strategy includes the parameter K of the VSC1 controller. VSC1 ,T1 VSC1 ,T2 VSC1 And the parameter K of the VSC2 controller VSC2 ,T1 VSC2 ,T2 VSC2 .

[0044] It can be seen from the technical solutions provided by the above embodiments of the present invention that the energy through flexible direct current transmission system in the present invention can coordinate the operation of multiple converters to achieve global optimization control of the system. Compared with the existing single control method, the present invention has stronger adaptive ability and better oscillation suppression effect, and can maintain stable operation of the system under various complex working conditions.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0047] Figure 1 A processing flow chart of a method for implementing an adaptive adjustment of additional damping control parameters provided by an embodiment of the present invention;

[0048] Figure 2 A system topology diagram of a new energy flexible direct current transmission system provided in an embodiment of the present invention;

[0049] Figure 3 A network structure diagram of a TD3 algorithm provided by an embodiment of the present invention;

[0050] Figure 4 A comparison diagram of oscillation suppression effects provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0052] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0053] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0054] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0055] This paper proposes a coordinated control method for oscillation suppression of renewable energy grid-connected systems based on the MA-DDPG algorithm. The coordinated control method based on multi-agent deep reinforcement learning is used to suppress broadband oscillations during the transmission of renewable energy through a flexible direct current transmission system. The method adopts a centralized training / distributed execution framework to achieve coordinated control between multiple converters and improve the flexibility of oscillation suppression.

[0056] The processing flow of a method for coordinated suppression of oscillation of a new energy grid-connected system based on the MA-DDPG algorithm provided in an embodiment of the present invention is as follows: Figure 1 As shown, the processing steps include the following:

[0057] Step S10: Obtain system parameters of the new energy flexible direct current transmission system.

[0058] Step S20: Construct a multi-agent deep reinforcement learning model based on the system parameters of the new energy flexible direct current transmission system, and define the state space set, action space set and reward function of the agent.

[0059] Step S30: Use the MA-DDPG algorithm to train the multi-agent deep reinforcement learning model until convergence.

[0060] Step S40: The parameters of the additional damping controller are adjusted online through the trained multi-agent deep reinforcement learning model to generate the optimal control strategy for the new energy direct current transmission system, and the system oscillation is suppressed using the optimal control strategy.

[0061] The above step S10 includes: the structure of a new energy flexible direct current transmission system provided by the embodiment of the present invention is as follows Figure 2 As shown, the system includes: a new energy rectifier VSC1, a flexible DC sending-end inverter VSC2 and a receiving-end converter VSC3. VSC1 adopts VdcIq control, VSC2 adopts Vf control, and VSC3 adopts VdcIq control. Additional damping controllers are installed on VSC1 and VSC2 to suppress the wide-band oscillation of the system.

[0062] Obtain the system parameters of the above-mentioned new energy through flexible direct current transmission system, which include dynamic parameters and electrical quantity data. The above-mentioned electrical quantity data includes the state space information of each intelligent agent: for Agent1 (VSC1), including DC voltage v dc1 and its deviation Δv dc1 、Input DC current i dc1 and output active power P1; for Agent2 (VSC2), including d-axis output voltage v Ld2 and its deviation Δv Ld2 And output active power P2.

[0063] The above step S20 includes:

[0064] Based on the system parameters of the above-mentioned new energy flexible direct current transmission system, a multi-agent deep reinforcement learning model is constructed. The multi-agent deep reinforcement learning model includes two agents (Agent1 and Agent2). Agent1 is connected to VSC1 as an additional damping controller of VSC1. Agent2 is connected to VSC2 as an additional damping controller of VSC2. The two agents (Agent1 and Agent2) are trained using the same set of reward functions to achieve coordinated control.

[0065] The state space of each agent includes: control reference value, deviation and system operation status. The state space of agent Agent1 is defined as:

[0066] s t VSC1 = {v dc1 (t),Δv dc1 (t),i dc1 (t)}

[0067] where v dc1 is the DC voltage of VSC1, Δv dc1 (t) is v dc1 Deviation from its reference value, i dc1 (t) is the input DC current of the renewable energy output.

[0068] The state space of Agent2 is defined as:

[0069]

[0070] where v Ld2 is the d-axis output voltage of VSC2, Δv Ld2 v Ld2 Deviation from its reference value, v Ld2 reference value.

[0071] The action space of each agent is the parameter of the additional damping controller.

[0072] The action space of agent VSC1 is:

[0073] The action space of agent VSC2 is:

[0074] Where K is the gain, T1 and T2 are time constants.

[0075] In order to achieve coordinated control of the two agents, this paper designs a data-driven reward function:

[0076] r t =-|Δv dc1 (t)|-|Δv Ld2 (t)|

[0077] Where Δv dc1 (t) is the deviation of the DC voltage of VSC1 from its reference value, Δv Ld2 (t) is the deviation of the d-axis output voltage of VSC2 from its reference value. This reward function is based on measured data and is a data-driven reward function.

[0078] The above step S30 includes: the present invention uses a multi-agent deep deterministic policy gradient (MA-DDPG) algorithm to train a multi-agent deep reinforcement learning model. The above shared data-driven reward function is used to coordinate the training of multiple converters.

[0079] Before describing the algorithm steps, we first explain the mathematical symbols used:

[0080] - The actor network of the i-th agent maps states to actions;

[0081] - The critic network of the i-th agent evaluates the value of the state-action pair

[0082] - Parameters of the i-th agent actor network

[0083] - Parameters of the i-th agent-critic network

[0084] -N t : Exploration noise, usually generated using the Ornstein-Uhlenbeck process

[0085] The training process of the above multi-agent deep reinforcement learning model includes:

[0086] 1. Initialization:

[0087] - Initialize the actor network for each agent i and critic network

[0088] - Initialize the target network μ for each agent i i ′ and Q i '

[0089] - Initialize the experience playback buffer D

[0090] 2. For each round:

[0091] - Initialize the random process N for action exploration

[0092] -Receive the initial observation state s1

[0093] For each time step t:

[0094] -For each agent i, choose action a according to the current strategy i,t =μ i (s t )+N t

[0095] -Perform action a t =(a 1,t ,...,a N,t ), observe the reward rt and the new state st +1 s_{t+1}

[0096] - Convert (s t ,a t ,r t ,s t+1 ) is stored in D

[0097] - Randomly sample a batch of transformations (s,a,r,s') from D

[0098] -For each agent i:

[0099] - Calculate target value

[0100] where r i is the immediate reward, γ is the discount factor, Q i ′ is the target critic network, μ1′(s′),...,μ′ N (s′) is the target action of all agents

[0101] -Update the critic network to minimize the loss:

[0102] MSE (mean-square error) is used as the loss function, and the goal is to minimize the gap between the predicted Q value and the target Q value.

[0103] -Update the actor network, using policy gradients:

[0104]

[0105] -Soft update the target network for each agent:

[0106] The mixing parameter τ is a soft update parameter, which usually takes a smaller value.

[0107] 3. End the loop and get the trained multi-agent deep reinforcement learning model. The training termination of the agent deep reinforcement learning model requires the following conditions to be met at the same time: first, the cumulative reward values ​​of the two agents have reached convergence, that is, the change in the cumulative reward values ​​of multiple consecutive training rounds is less than the preset threshold; second, the oscillation suppression effect of the system under different operating conditions meets the standard, which is specifically manifested in the DC voltage v of VSC1. dc1 and the d-axis voltage v of VSC2 Ld2 The fluctuation amplitude is less than 2% of the rated value and can be restored to stability within 0.5 seconds; third, the control parameters output by the two agents remain relatively stable in multiple consecutive training rounds, and the parameter change amplitude is less than the set threshold. When the above conditions are met at the same time, the multi-agent deep reinforcement learning model can be considered to have completed training. At this time, the trained model parameters are saved for subsequent online control.

[0108] The above step S40 includes: online application

[0109] After training, the two agents can output adaptive additional damping control parameters for VSC1 and VSC2 respectively according to different operating conditions to achieve decentralized execution.

[0110] 7. Control effect

[0111] Through simulation verification, Figure 4 As shown, the multi-agent deep reinforcement learning method proposed in the present invention can more effectively suppress system oscillations under different frequencies and different operating conditions compared with the traditional fixed parameter method and the DQN method, thereby improving the stability and adaptability of the system.

[0112] In summary, the present invention realizes coordinated control based on multi-agent deep reinforcement learning, effectively suppresses broadband oscillations in the flexible direct current transmission system of new energy, and improves the stability of the system.

[0113] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0114] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0115] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0116] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for coordinated suppression of oscillation of a new energy grid-connected system based on the MA-DDPG algorithm, characterized in that: include: Obtain system parameters of the new energy flexible direct current transmission system; Based on the system parameters of the new energy flexible direct current transmission system, a multi-agent deep reinforcement learning model is constructed to define the state space set, action space set and reward function of the agent; Use the MA-DDPG algorithm to train the multi-agent deep reinforcement learning model until convergence to obtain a trained multi-agent deep reinforcement learning model; The parameters of the additional damping controller are adjusted online through the trained multi-agent deep reinforcement learning model to generate the optimal control strategy for the new energy flexible direct current transmission system, and the system oscillation is suppressed using the optimal control strategy.

2. The method according to claim 1, characterized in that The system parameters of the flexible direct current transmission system for new energy are obtained, including: A new energy flexible direct current transmission system is provided, including: a new energy rectifier VSC1, a flexible direct current sending-end inverter VSC2 and a receiving-end converter VSC3. VSC1 adopts VdcIq control, VSC2 adopts Vf control, and VSC3 adopts VdcIq control. Additional damping controllers are installed on VSC1 and VSC2 to obtain system parameters of the new energy flexible direct current transmission system. The system parameters include dynamic parameters and electrical quantity data. The electrical quantity data include DC side voltage and current and AC side d-axis voltage data.

3. The method according to claim 2, characterized in that The multi-agent deep reinforcement learning model is constructed based on the system parameters of the new energy flexible direct current transmission system, and the state space set, action space set and reward function of the agent are defined, including: A multi-agent deep reinforcement learning model is constructed based on the system parameters of the new energy flexible direct current transmission system. The multi-agent deep reinforcement learning model includes two agents, Agent 1 and Agent 2. Agent 1 is connected to VSC 1 and serves as an additional damping controller of VSC 1. Agent 2 is connected to VSC 2 and serves as an additional damping controller of VSC 2. The state space of each agent includes: control reference value, deviation and system operation status. The state space of agent Agent1 is defined as: s t VSC1 ={v dc1 (t),Δv dc1 (t),i dc1 (t)} where v dc1 is the DC voltage of VSC1, Δv dc1 (t) is v dc1 Deviation from its reference value, i dc1 (t) Input DC current for renewable energy output; The state space of Agent2 is defined as: where v Ld2 is the d-axis output voltage of VSC2, Δv Ld2 v Ld2 Deviation from its reference value, v Ld2 Reference value of The action space of each agent is the parameters of the additional damping controller; The action space of agent VSC1 is: The action space of agent VSC2 is: Where K is the gain, T1 and T2 are time constants; Design a data-driven reward function: r t =-|Δv dc1 (t)|-|Δv Ld2 (t)| Where Δv dc1 (t) is the deviation of the DC voltage of VSC1 from its reference value, Δv Ld2 (t) is the deviation of the d-axis output voltage of VSC2 from its reference value.

4. The method according to claim 3, characterized in that The multi-agent deep reinforcement learning model is trained using the MA-DDPG algorithm until convergence to obtain a trained multi-agent deep reinforcement learning model, including: In the collaborative training of the multi-agent deep reinforcement learning model, the MA-DDPG algorithm is used to achieve coordinated control between converters. The input of the training process includes the state space information of each agent: for Agent1 (VSC1), including the DC voltage v dc1 and its deviation Δv dc1 、Input DC current i dc1 and output active power P1; for Agent2 (VSC2), including d-axis output voltage v Ld2 and its deviation Δv Ld2 and output active power P2. The output of the training is the action space of each agent, that is, the parameters of the additional damping controller, including the parameters K of the VSC1 controller VSC1 (t),T1 VSC1 (t),T2 VSC1 (t) and the parameter K of the VSC2 controller VSC2 (t),T1 VSC2 (t),T2 VSC2 (t); The MA-DDPG algorithm adopts a centralized training and decentralized execution framework for training. In the centralized training phase, the two agents share the same experience replay pool to store interaction data samples (s t ,a t ,r t ,s t+1 ), and share a reward function r based on the electrical quantity measurement value. Each agent contains an Actor network and a Critic network, where the Actor network is responsible for outputting control actions based on the current state, and the Critic network is responsible for evaluating the value of the current state-action pair; During the training process, the Actor and Critic networks of the two agents interact to update network parameters. The Actor network considers the influence of the other agent's actions when updating, and the Critic network takes the states and actions of the two agents as input when evaluating. Coordinated training of multiple inverters using a shared data-driven reward function. During the training of the multi-agent deep reinforcement learning model, the agents of VSC1 and VSC2 share the same data-driven reward function r t , the reward function is built based on real-time measurement data, and the calculation formula is: r t =-|Δv dc1 (t)|-|Δv Ld2 (t)| Where Δv dc1 (t) represents the DC voltage deviation of VSC1 at time t, Δv Ld2 (t) represents the d-axis voltage deviation of VSC2 at time t, and the negative sign indicates that the smaller the deviation, the greater the reward value; The training termination of the multi-agent deep reinforcement learning model requires the following conditions to be met at the same time: first, the cumulative reward values ​​of the two agents have reached convergence, that is, the change in the cumulative reward values ​​of multiple consecutive training rounds is less than the preset threshold; second, the oscillation suppression effect of the system under different operating conditions meets the standard; third, the control parameters output by the two agents remain relatively stable in multiple consecutive training rounds, and the change in the parameters is less than the set threshold; When the above conditions are met at the same time, the training of the multi-agent deep reinforcement learning model is completed, and a trained multi-agent deep reinforcement learning model is obtained.

5. The method according to claim 4, characterized in that The method of adjusting the parameters of the additional damping controller online through the trained multi-agent deep reinforcement learning model to generate the optimal control strategy of the new energy flexible direct current transmission system, and using the optimal control strategy to suppress system oscillations, includes: The input data of the trained multi-agent deep reinforcement learning model includes the state space information of each agent: for Agent1 (VSC1), it includes the DC voltage v dc1 and its deviation Δv dc1 、Input DC current i dc1 and output active power P1; for Agent2 (VSC2), including d-axis output voltage v Ld2 and its deviation Δv Ld2 And output active power P2; The multi-agent deep reinforcement learning model normalizes the input data to make it meet the numerical range set during training, and then inputs the input data into the Actor networks corresponding to VSC1 and VSC2 respectively. Based on the current state, the Actor networks obtain their respective optimal control strategies through forward calculation. The optimal control strategy includes the parameter K of the VSC1 controller. VSC1 ,T1 VSC1 ,T2 VSC1 And the parameter K of the VSC2 controller VSC2 ,T1 VSC2 ,T2 VSC2 .

Images