Network construction type flexible direct current power transmission system control method based on deep reinforcement learning

Abstract

The invention relates to the technical field of flexible direct-current power transmission systems, in particular to a network construction type flexible direct-current power transmission system control method based on deep reinforcement learning, and the method comprises the steps: building a network construction type flexible direct-current power transmission system control model, building a Markov decision process based on the network construction type flexible direct-current power transmission system control model, a double-pool-double-delay depth deterministic strategy gradient algorithm is adopted to train an intelligent agent, and the intelligent agent provides an additional power input value for a P-f control loop in a virtual synchronous control model, so that the stability of the network construction type flexible direct-current power transmission system is improved. According to the method, the two-valued function network is introduced, and the target value is calculated by using a relatively small Q value in the two-valued function network, so that excessive estimation of some state-action pairs on the value can be avoided, and the overall stability of the VSG system is improved.

Description

Technical Field

[0001] The present application relates to the technical field of flexible direct current transmission systems, and in particular to a control method for a meshed flexible direct current transmission system based on deep reinforcement learning. Background Art

[0002] With the continuous progress and increasing intelligence of power systems, the application of virtual synchronous generator (VSG) technology in flexible DC transmission is becoming more and more extensive. The promotion of this technology is conducive to the frequency stability of the power grid and the improvement of power quality. However, VSG still faces many stability problems in practical applications, especially when the power grid is disturbed, its dynamic response and control stability become one of the key research challenges. In order to further improve the stability and performance of VSG, researchers are constantly exploring new control strategies. Among them, deep reinforcement learning (DRL), as an emerging artificial intelligence technology, has demonstrated its great potential in controlling complex systems in recent years. DRL has the ability to adapt and make optimal decisions, and can make reasonable control decisions in dynamic and nonlinear environments, which makes it a powerful tool to solve the stability problem of VSG.

[0003] However, traditional reinforcement learning methods face the problem of dimensionality curse when dealing with tasks involving continuous state variables, because the number of discretized states will increase exponentially with the increase of spatial dimensions. This makes effective learning impractical. At present, the Deep Deterministic Policy Gradient (DDPG) algorithm can maintain the continuity of action and state space, and can effectively avoid the problem of dimensionality curse. For example, in the patent with publication number CN115276093A, a virtual synchronous generator parameter adaptive control method based on the DDPG algorithm is provided, including first establishing a virtual synchronous generator control model, and then collecting the active power P1, reactive power Q1, system frequency f1 and voltage U1 output by the control model to form training data, and combining the DDPG algorithm to train the agent to obtain the optimal control variables of the VSG system under the corresponding state: moment of inertia J and damping coefficient D, and encapsulating the agent after training; finally, in actual operation, the output of the agent is adjusted according to the real-time environmental state, and the frequency response of the system is controlled, thereby improving the system operation performance.

[0004] However, the DDPG algorithm has the phenomenon of "overestimation" in practical applications. This phenomenon is due to the fact that the algorithm may over-estimate the value of certain state-action pairs during the training process, causing the learned strategy to be unstable in practical applications, thereby affecting the overall stability of the VSG system. Summary of the invention

[0005] Based on this, it is necessary to provide a control method, device, computer equipment, storage medium and computer program product for a grid-type flexible direct current transmission system based on deep reinforcement learning to address the problem that the DDPG algorithm may over-estimate the value of certain state-action pairs during the training process, resulting in unstable performance of the learned strategies in practical applications.

[0006] In a first aspect, the present application provides a control method for a meshed flexible direct current transmission system based on deep reinforcement learning. The method comprises:

[0007] Step S1, establishing a grid-forming flexible direct current transmission system control model; the grid-forming flexible direct current transmission system control model includes a voltage and current double closed-loop control model and a virtual synchronous control model;

[0008] Step S2, establishing a Markov decision process based on a grid-type flexible DC transmission system control model;

[0009] Step S3, adopting the dual-pool-dual-delay deep deterministic policy gradient algorithm to train the intelligent agent, the intelligent agent improves the stability of the grid-type flexible DC transmission system by providing additional power input value to the Pf control loop in the virtual synchronous control model; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the intelligent agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

[0010] In one embodiment, the Markov decision process includes a state value function, an action value function, an agent environment state, and an agent output action; wherein,

[0011] The state value function is:

[0012]

[0013] Among them, V π (x) is the state value function, which represents the total return expected from starting from state x and executing strategy π, r t To represent the return at time t, x t represents the state at time t, E represents the expected function, γ is the discount factor, the value range is [0,1], k is the time step, r t+k+1 represents the return at time t+k+1;

[0014] The action value function is:

[0015]

[0016] In the formula, Q π(x,a) is the action value function, which indicates the total return expected after the agent performs action a in the initial state x and continues to execute according to the strategy π. t represents the action performed by the agent at time t;

[0017] Agent environment state x t The expression is:

[0018] x t ={Δf,ΔP}

[0019] In the formula, Δf represents the difference between the real-time frequency and the reference frequency, and ΔP represents the difference between the active power and the reference active power;

[0020] Output action a of the agent t Defined as:

[0021] a t = {PΔ f}

[0022] Where P Δf Indicates the additional power input value of the Pf control loop in virtual synchronous control.

[0023] In one embodiment, the return r at time t t The calculation formula is:

[0024] r t =-(λ1|f ref -f t |+λ2|P ref -P t |)

[0025] In the formula, f ref is the reference frequency, f t is the actual output frequency of the virtual synchronous generator at time t, P ref is the reference active power, P t is the actual output power of the virtual synchronous generator at time t, λ1 is the frequency weight coefficient, and λ2 is the active power weight coefficient.

[0026] In one embodiment, step S3 comprises:

[0027] Step S31, obtain the experience tuple (x t , a t , r t , x t+1 ) and discount factor γ, according to the formula

[0028]

[0029] Calculate the target Q value; where y tis the target Q value at time t, represents the action value function, x t+1 represents the state at time t+1, represents the action at time t+1 generated by the target policy network, i=1,2;

[0030] Step S32, based on the target Q value, the parameters of the value network are updated by gradient descent with the goal of minimizing the mean square error loss function of the value network, that is,

[0031]

[0032] In the formula, α Q is the learning rate of the value network, is the gradient operator, K(w i ) is the mean square error loss function of the value network; where the expression of the mean square error loss function of the value network is:

[0033]

[0034] Where D is the experience replay buffer; w i Represents the value network Q i Parameters, i = 1, 2; Indicates that the parameter is w i Value Network The calculated Q value;

[0035] Step S33, update the parameters of the target value network based on the parameters of the value network, that is

[0036] w′ i ←τw i +(1-τ)w′ i

[0037] Where w' i is the parameter of the target value network, i=1,2; τ is the soft update coefficient, the value range is (0,1);

[0038] Step S34, with the goal of maximizing the expected cumulative reward function, the parameters θ of the policy network are updated by gradient descent, that is,

[0039]

[0040] In the formula, α π is the learning rate of the policy network, H(θ) is the expected cumulative reward function; the expression of the expected cumulative reward function is:

[0041]

[0042] In the formula, Q θ (xt ,π θ (x t )) means in state x t Next, execute action π θ (x t ) after the expected total return, π θ (x t ) indicates that the policy network with parameter θ is in state x t The following actions are generated.

[0043] Step S35, update the parameters of the target policy network based on the parameters of the policy network, that is

[0044] θ'←τθ+(1-τ)θ'

[0045] Where θ' is the parameter of the target policy network.

[0046] In one embodiment, the action at time t+1 generated by the target policy network is the action after adding Gaussian noise, that is,

[0047]

[0048] In the formula, π(x t+1 ) is the next state x t+1 When , N(0,σ) represents Gaussian noise with mean 0 and variance σ, and clip is a function that limits the Gaussian noise to the range of [-C,C].

[0049] In one embodiment, the step of dividing the successful experience pool and the failed experience pool includes:

[0050] Step S30, dividing the experience pool into a success experience pool and a failure experience pool based on a judgment function; the expression of the judgment function is:

[0051]

[0052] In the formula, t v is the total training time in seconds.

[0053] In one embodiment, step S1 comprises:

[0054] Step S11, obtaining a voltage loop equation of the main circuit of the grid-forming flexible DC transmission system; the voltage loop equation is:

[0055]

[0056] In the formula, R represents the resistance of the line, i sn represents the output current of the converter, L represents the reactance of the line, e n Indicates the voltage of the power grid, U norepresents the output voltage of the converter, n=A, B, C;

[0057] Step S12, after performing dq transformation and Laplace transformation on the voltage loop equation, a voltage-current dual closed-loop control model is obtained; the expression of the voltage-current dual closed-loop control model is:

[0058]

[0059] Where s is the Laplace operator, i d is the d-axis component of the three-phase current in the rotating coordinate system, i q is the q-axis component of the three-phase current in the rotating coordinate system, u d is the d-axis component of the three-phase voltage in the rotating coordinate system, u q is the q-axis component of the three-phase voltage in the rotating coordinate system, e d is the d-axis component of the grid voltage in the rotating coordinate system, e q is the q-axis component of the grid voltage in the rotating coordinate system, L is the line inductance, and ω is the mechanical angular velocity of the virtual synchronous generator;

[0060] The virtual synchronous control model includes the mechanical equations of the virtual synchronous generator and the virtual exciter control equations; step S1 also includes:

[0061] Step S13, establishing the mechanical equation of the virtual synchronous generator and the virtual exciter control equation; wherein the mechanical equation of the virtual synchronous generator is:

[0062]

[0063] Where J is the moment of inertia of the virtual synchronous generator, ω0 is the synchronous angular velocity of the power grid, and P m is the mechanical power of the virtual synchronous generator, P e is the electromagnetic power of the virtual synchronous generator, T m is the mechanical torque, T e is the electromagnetic torque, T d is the damping torque, D is the damping coefficient, and δ is the rotor angle;

[0064] The mathematical expression of the virtual exciter control loop is:

[0065]

[0066] In the formula, E is the virtual electromotive force, E0 is the set value of the no-load electromotive force, U is the actual calculated voltage value, and U n is the rated voltage value, Q ref is the reference value of reactive power, Q e is the actual reactive power; is the reactive power integration coefficient.

[0067] In a second aspect, the present application also provides a control device for a meshed flexible direct current transmission system based on deep reinforcement learning. The device comprises:

[0068] A system control model building module is used to build a grid-forming flexible DC transmission system control model; the grid-forming flexible DC transmission system control model includes a voltage and current double closed-loop control model and a virtual synchronous control model;

[0069] A Markov decision process establishment module is used to establish a Markov decision process based on a grid-type flexible DC transmission system control model;

[0070] An agent training module is used to train the agent using a dual-pool-dual-delay deep deterministic policy gradient algorithm; the agent provides additional power input values ​​to the Pf control loop in the virtual synchronous control to improve the stability of the grid-type flexible DC transmission system; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

[0071] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the method for controlling a grid-type flexible direct current transmission system described in the technical solution of the first aspect and any of its optional solutions are implemented.

[0072] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the method for controlling a grid-type flexible direct current transmission system described in the technical solution of the first aspect and any of its optional solutions.

[0073] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the control method of a grid-type flexible direct current transmission system described in the above-mentioned technical solution of the first aspect and any of its optional solutions.

[0074] The control method, device, computer equipment, storage medium and computer program product of the meshed flexible DC transmission system based on deep reinforcement learning in the present application firstly establish a meshed flexible DC transmission system control model, and establish a Markov decision process based on the meshed flexible DC transmission system control model, which can accurately reflect the dynamic characteristics and control requirements of the system. Secondly, by adopting a dual-pool-dual-delay deep deterministic policy gradient algorithm to train the intelligent agent, a dual-valued function network is introduced, and the target value is calculated using the smaller Q value in the dual-valued function network, which can avoid over-estimation of the value of certain state-action pairs and improve the overall stability of the VSG system. At the same time, by dividing the experience pool into a successful experience pool and a failed experience pool, and sampling data from the two experience pools to train the intelligent agent, the data distribution in the training process can be further balanced, the over-estimation problem caused by data deviation can be reduced, and the reliability of the additional power input value can be improved, thereby further improving the stability of the VSG system. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 It is a schematic flow chart of a control method of a meshed flexible direct current transmission system based on deep reinforcement learning in one embodiment;

[0076] Figure 2 A schematic diagram of a Markov decision process in one embodiment;

[0077] Figure 3 A schematic diagram of a dual-pool-dual-delay deep deterministic policy gradient algorithm for training an intelligent agent in one embodiment;

[0078] Figure 4 A block diagram of voltage and current dual closed-loop control in one embodiment;

[0079] Figure 5 is an overall control block diagram of a virtual synchronous generator in one embodiment;

[0080] Figure 6 It is an overall control block diagram of a control method for a meshed flexible direct current transmission system based on deep reinforcement learning in one embodiment;

[0081] Figure 7 A comparison diagram of the control performance of three control methods on frequency under different receiving-end power grid strengths in one embodiment;

[0082] Figure 8 A comparison diagram of control performance of active power by three control methods under different receiving-end power grid strengths in one embodiment;

[0083] Fig. 9 A comparison diagram of control performance of reactive power by three control methods under different receiving-end power grid strengths in one embodiment;

[0084] Fig.10 A comparison diagram of the control performance of three control methods on the grid connection point voltage under different receiving-end grid strengths in one embodiment;

[0085] Fig.11 It is a structural block diagram of a meshed flexible direct current transmission system control device based on deep reinforcement learning in one embodiment. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0087] In one embodiment, Figure 1 As shown, a control method for a meshed flexible DC transmission system based on deep reinforcement learning is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The following steps are included:

[0088] Step S1, establishing a grid-forming flexible direct current transmission system control model; the grid-forming flexible direct current transmission system control model includes a voltage and current double closed-loop control model and a virtual synchronous control model.

[0089] The virtual synchronous control model includes the Pf control loop and the QE control loop, which are responsible for calculating the electromagnetic power P provided by the power calculation module. e and actual reactive power Q e , dynamically generate virtual electromotive force E and mechanical angular velocity ω. The virtual electromotive force E is used as the input reference voltage amplitude of the voltage and current dual closed-loop control model, and the mechanical angular velocity ω is used as the input reference voltage phase angle of the voltage and current dual closed-loop control model. The voltage and current dual closed-loop control model includes voltage outer loop control and current inner loop control. The voltage and current dual closed-loop control model is based on the three-phase current component i in the rotating coordinate system. d 、i q And the three-phase voltage components u d 、u q , combined with the virtual electromotive force E and mechanical angular velocity ω output by the virtual synchronous control model, dual closed-loop control of the voltage outer loop and the current inner loop is performed to ensure stable operation of the system.

[0090] Step S2, establishing a Markov decision process based on the grid-type flexible DC transmission system control model.

[0091] Reinforcement learning is a self-learning method. The intelligent agent obtains the optimal strategy through trial and error in the process of interacting with the environment. The flowchart is as follows: Figure 2 As shown in Figure 1. The Markov Decision Process (MDP) is constructed based on a set of interacting objects, namely the agent and the environment, and its elements include the state x t 、Action a t , strategy π and reward r t In the MDP simulation, the agent perceives the current system state and takes actions on the environment according to the strategy, thereby changing the state of the environment and receiving rewards. The accumulation of rewards over time is called return.

[0092] Specifically, the Markov decision process is established based on the control model of the grid-type flexible DC transmission system, including defining the state value function, action value function, agent environment state (i.e. state x t ) and the agent output action (i.e. action a t ). In this embodiment, the agent environment state is set to:

[0093] x t ={Δf,ΔP} (1)

[0094] In the formula, Δf represents the real-time frequency f and the reference frequency f ref The difference between the real-time active power P and the reference active power P ref The difference between them; wherein, the real-time active power P can be calculated by the current and voltage in the power system.

[0095] The agent output action is set to:

[0096] a t = {P Δf} (2)

[0097] Where P Δf It represents the additional power input value of the Pf control loop in the virtual synchronous control model. In the continuous training iteration process, the agent gradually adjusts the action value output in each state by continuously optimizing its strategy π to ensure that the most appropriate action can be selected in each current state.

[0098] Specifically, Figure 6 As shown, the additional power input value output by the intelligent agent is added to the original electromagnetic power Pe, and the result is used as the updated electromagnetic power Pe', which is directly applied to the calculation of the pf control loop.

[0099] The value function can be used to evaluate the expected return of strategy π. The value function can include the state value function and the action value function. Among them, the state value function V π (x) represents the total return expected from starting from state x and executing strategy π, and its function expression is:

[0100]

[0101] Among them, r t To represent the reward at time t, it can be calculated by the reward function shown in formula (6), x t represents the state at time t, E represents the expected function, γ is the discount factor, the value range is [0,1], k is the time step, r t+k+1 represents the return at time t+k+1.

[0102] Action-value function Q π (x,a) represents the expected cumulative return after executing action a in the initial state x and continuing to execute according to strategy π. Its function expression is:

[0103]

[0104] In the formula, a t represents the action performed by the agent at time t, as shown in formula (2); r t+k+1 represents the reward at time t+k+1, which can also be calculated by the reward function shown in formula (6).

[0105] Specifically, Q π (x,a) evaluates the expected value of the cumulative reward that the agent can obtain in the future by following the policy π after taking an action a in a specific state x. This function not only considers the immediate reward of the current state and action, but also includes the discounted sum of the rewards of all possible future states and actions. Therefore, Q π (x,a) provides an indicator for the agent to measure the quality of its decision-making, helping it to choose an action strategy that can maximize long-term returns. In the process of reinforcement learning, by continuously updating and optimizing the Q function, the agent can gradually learn the optimal strategy and make the best decision in various complex environments. π (x) is the action-value function Q π (x,a) is the expected function of action a. Specifically, the state value function V π (x) evaluates the expected total reward when executing policy π in state x. It is calculated by taking a weighted average of all possible actions a, where the weight is the probability of policy π choosing action a in state x. Thus, V π (x) can be obtained by Q π (x,a) is expressed as:

[0106] V π (x) = E[r t |x t =x] = E a [Q π(x,a)] (5)

[0107] Q π (x,a) is the state value function V π (x) and action reward feedback value r t The sum of , E is the expected function.

[0108] In step S3, a dual-pool-dual-delay deep deterministic policy gradient algorithm is used to train the intelligent agent; the intelligent agent provides an additional power input value to the Pf control loop in the virtual synchronous control to improve the stability of the grid-type flexible direct current transmission system.

[0109] In view of the problem of overestimation of Q value in DDPG, this embodiment adopts TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm for agent training. The TD3 algorithm uses the Actor-Critic algorithm architecture and introduces a deep neural network (DNN). In the Actor-Critic algorithm architecture, Actor (policy network) and Critic (value network) cooperate with each other. Actor is responsible for generating actions, while Critic is responsible for evaluating the quality of these actions and providing feedback to guide Actor's learning. Under this framework, the network parameters of the policy network and the value network are usually continuously updated during the training process to optimize the ability of the policy network to generate actions. The core of the TD3 algorithm is that it uses two independent value networks to calculate the action value functions Q1 and Q2 respectively. When updating the value network, the smaller value of Q1 and Q2 is selected to calculate the target value, thereby alleviating the problem of excessive deviation in Q value estimation.

[0110] In addition, experience replay is a key component in the TD3 algorithm. It breaks the temporal correlation of data by storing past experience samples. When the policy network and the value network are trained, a mini-batch of experience is randomly sampled from the experience replay buffer. However, when there is only one experience pool, the agent is prone to fall into the problem of local optimality during training. Therefore, this embodiment proposes a dual experience pool training method, specifically, the experience replay buffer includes two experience pools: a failed experience pool and a successful experience pool. The successful experience pool stores successful data, that is, actions that lead to improved system stability. t , state x t and reward r t Data; the failure experience pool stores failed data, that is, actions that cause system stability to decrease or fail to achieve the expected results. t , state x t and reward r tThe agent randomly samples experience tuples (x t , a t , r t , x t+1 ) for training, so that we can learn from successful experiences and draw lessons from failures to avoid repeating mistakes.

[0111] Specifically, it can be based on a key indicator in the experience tuple, such as the state data x t or reward data t To determine whether the experience tuple belongs to the success experience pool or the failure experience pool. t For example, we first set a judgment condition, which can be a predefined threshold or a complex expression that is dynamically adjusted according to task requirements or training process. t When this judgment condition is met, the experience tuple is considered to represent successful experience, so it is divided into the successful experience pool. On the contrary, if the reward data r t If the judgment condition is not met, the experience tuple is considered to represent a failed experience and is divided into a failed experience pool. By dividing the experience pool in this way, it is possible to effectively distinguish valuable successful experiences from failed experiences that need to be improved from a large amount of experience data, thereby providing more targeted data support for the learning and optimization of the intelligent agent.

[0112] In the control method of the grid-type flexible direct current transmission system based on deep reinforcement learning in this embodiment, firstly, by establishing a grid-type flexible direct current transmission system control model, a Markov decision process is established based on the grid-type flexible direct current transmission system control model, which can accurately reflect the dynamic characteristics and control requirements of the system. Secondly, by adopting a dual-pool-dual-delay deep deterministic policy gradient algorithm to train the intelligent agent, a dual-value function network is introduced, and the target value is calculated using the smaller Q value in the dual-value function network, which can avoid over-estimation of the value of certain state-action pairs and improve the overall stability of the VSG system. At the same time, by dividing the experience pool into a successful experience pool and a failed experience pool, and sampling data from the two experience pools to train the intelligent agent, the data distribution in the training process can be further balanced, the over-estimation problem caused by data deviation can be reduced, and the reliability of the additional power input value can be improved, thereby further improving the stability of the VSG system.

[0113] The reward function is a key signal that drives the agent to explore and discover the best action strategy. In a grid-based flexible DC controller, frequency stability and power stability are two important factors to ensure the reliable operation of the system. Therefore, in order to optimize the performance of the controller and ensure the stable operation of the system, the setting of the reward function must fully consider these factors. Therefore, the reward function needs to be able to reflect the frequency fluctuations and power output deviations of the system in real time. Specifically, by incorporating frequency stability and power stability into the reward function, the agent can continuously adjust its action strategy during the learning process to maximize the stability of these key performance indicators. When the system frequency and power output reach the expected stable state, the agent will receive a higher reward, otherwise it will be punished. In this way, the agent will be guided to find the optimal control strategy that can maintain the stability of the system. In one embodiment, the agent's reward function r t Used to calculate the return r at time t t , whose expression is:

[0114] r t =-(λ1|f ref -f t |+λ2|P ref -P t |) (6)

[0115] In the formula, f ref is the reference frequency, f t is the actual output frequency of the virtual synchronous generator at time t, P ref is the reference active power, P t is the actual power output by the virtual synchronous generator at time t, λ1 is the frequency weight coefficient, λ2 is the active power weight coefficient, and λ1 and λ2 can be continuously adjusted and tested by technical personnel in this field based on the actual training process.

[0116] The reward function in this embodiment can instantly capture and quantify the dynamic changes of system frequency fluctuations and power output deviations, effectively guiding the intelligent agent to explore and execute optimal control strategies that can significantly improve the stability of the virtual synchronous generator (VSG), thereby achieving a significant enhancement in the stability of the VSG system.

[0117] In one embodiment, step S3 includes:

[0118] Step S31, obtain the experience tuple (x t , a t , r t , x t+1 ) and discount factor γ, according to the formula:

[0119]

[0120] Calculate the target Q value; where yt is the target Q value at time t, represents the action value function, x t+1 represents the state at time t+1, represents the action at time t+1 generated by the target policy network, i=1, 2. The discount factor γ determines the weight of future rewards in the Q value update, that is, the degree of attenuation of future rewards, and its value range is generally between 0 and 1. When the discount factor is close to 1, the agent will pay more attention to future rewards, which helps to optimize the long-term strategy. When the discount factor is close to 0, the agent will pay more attention to immediate rewards and pay more attention to short-term returns.

[0121] Specifically, Figure 3 As shown in FIG. 1 , the agent first randomly samples experience tuples (x ) from the success experience pool and the failure experience pool according to a predetermined ratio, such as 70% success experience and 30% failure experience, or a ratio adjusted according to specific needs. t , a t , r t , x t+1 ). After sampling is completed, the target policy network in the agent is based on the next state x after executing the action in the experience tuple. t+1 Generate the corresponding next action Target value network 1 is based on the next state x after executing the action t+1 and the corresponding next action Calculate the first Q value Right now Target value network 2 is based on the next state x after executing the action t+1 and the corresponding next action Calculate the second Q value Right now Then select the first Q value With the second Q value The smaller value in is used to calculate the target Q value at time t, corresponding to the smaller value in formula (6) part.

[0122] Step S32, based on the target Q value, the parameters of the value network are updated by gradient descent with the goal of minimizing the mean square error loss function of the value network, that is:

[0123]

[0124] In the formula, α Q is the learning rate of the value network, is the gradient operator, K(w i ) is the mean square error loss function of the value network; where the expression of the mean square error loss function of the value network is:

[0125]

[0126] Where D is the experience replay buffer, w i Represents the value network Q i Parameters, i = 1, 2, Indicates that the parameter is w i Value Network The calculated Q value.

[0127] Specifically, with the goal of minimizing the mean square error loss function of the value network as shown in formula (8), the parameters of each Q network, namely the value network Q1 and the value network Q2, are updated in the manner shown in formula (7).

[0128] In addition, in order to maintain the stability of the target value network, the parameters of the target value network are not updated directly according to the gradient descent method, but are soft-updated based on the parameters of the current value network. This soft update method is controlled by a soft update coefficient τ (ranging from 0 to 1), so that the parameter changes of the target value network are smoother, which helps to stabilize the learning process, as shown in step S33.

[0129] Step S33, updating the parameters of the target value network based on the parameters of the value network, namely:

[0130] w′ i ←τw i +(1-τ)w′ i (10)

[0131] Where w′ i is the parameter of the target value network, i = 1, 2; τ is the soft update coefficient, which is usually a very small constant with a value range of (0, 1). Adjusting the soft update coefficient can control the update rate of the target value network. Specifically, according to formula (9), the parameter w1' of the target value network 1 is updated based on the parameter w1 of the value network 1, and the parameter w2' of the target value network 2 is updated based on the parameter w2 of the value network 2.

[0132] The TD3 algorithm introduces a delayed policy update mechanism in policy update, that is, the Actor network (policy network) is updated every d steps. This reduces the oscillation of the policy network and increases the stability of learning. The goal of the Actor network is to maximize the value estimated by the Critic network, that is, to find the action that can maximize the expected cumulative reward. The specific steps for updating the network parameters of the policy network are shown in steps S34 and S35.

[0133] Step S34, with the goal of maximizing the expected cumulative reward function, the parameters θ of the policy network are updated by gradient descent, that is:

[0134]

[0135] In the formula, α π is the learning rate of the policy network, H(θ) is the expected cumulative reward function; the expression of the expected cumulative reward function is:

[0136]

[0137] In the formula, Q θ (x t ,π θ (x t )) means in state x t Next, execute action π θ (x t ) after the expected total return, π θ (x t ) indicates that the policy network with parameter θ is in state x t The following actions are generated.

[0138] Step S35, update the parameters of the target policy network based on the parameters of the policy network, that is

[0139] θ'←τθ+(1-τ)θ' (13)

[0140] Where θ' is the parameter of the target policy network. Similar to updating the target value network, the parameters of the target policy network are also soft-updated based on the current parameters of the policy network. This update method helps maintain the stability and convergence of the policy network.

[0141] This embodiment describes the update mechanism of network parameters under the deep reinforcement learning framework, specifically involving the parameter update process of the value network and its corresponding target value network, as well as the synchronous optimization of the parameters of the policy network and the target policy network, ensuring that each network can evolve co-evolved and gradually approach the best strategy for specific tasks. This process not only accelerates the learning speed of the intelligent agent, but also significantly improves its decision-making ability and adaptability in complex environments, thereby effectively enhancing the overall stability and operating efficiency of the VSG.

[0142] In one embodiment, in order to achieve target policy smoothing, the action at time t+1 generated by the policy network is the action after adding Gaussian noise, that is:

[0143]

[0144] In the formula, π(x t+1 ) means that in the next state x t+1 When , N(0,σ) represents Gaussian noise with mean 0 and variance σ, and clip is a function that limits the Gaussian noise to the range of [-C,C].

[0145] In one embodiment, the step of dividing the successful experience pool and the failed experience pool includes:

[0146] Step S30, dividing the experience pool into a success experience pool and a failure experience pool based on a judgment function; the expression of the judgment function is:

[0147]

[0148] In the formula, t v is the total training time in seconds.

[0149] Specifically, based on the instant reward r in each experience tuple in the experience pool t This indicator is used to determine whether the experience tuple is divided into the successful experience pool or the failed experience pool. For example, the experience tuple that satisfies formula (15) is divided into the successful experience pool, and the experience tuple that does not satisfy formula (15) is divided into the failed experience pool.

[0150] This embodiment constructs a judgment function as the basis for dividing the experience tuples into the success experience pool and the failure experience pool, which significantly improves the learning efficiency and strategy reliability of the intelligent agent during the training process, thereby effectively enhancing the stability of the VSG system.

[0151] In one embodiment, step S1 includes:

[0152] Step S11, obtaining a voltage loop equation of the main circuit of the grid-forming flexible DC transmission system; the voltage loop equation is:

[0153]

[0154] In the formula, R represents the resistance of the line, i sn represents the output current of the converter, L represents the reactance of the line, e n Indicates the voltage of the power grid, U no represents the output voltage of the converter, n = A, B, C. The mathematical model of the flexible DC transmission system is usually based on the dynamic behavior of its voltage source converter. Assume that the output voltage and current of the converter are U Ao , U Bo , U Co and I sA ,I sB ,I sB The voltage of the power grid is e A 、e B 、e C , the line impedance is Z = R + jX, and Kirchhoff's voltage law (KVL) is written for the main circuit as shown in equation (16).

[0155] Step S12, after performing dq transformation and Laplace transformation on the voltage loop equation, a voltage-current dual closed-loop control model is obtained; the expression of the voltage-current dual closed-loop control model is:

[0156]

[0157] Where s is the Laplace operator, i d is the d-axis component of the three-phase current in the rotating coordinate system, i q is the q-axis component of the three-phase current in the rotating coordinate system, u d is the d-axis component of the three-phase voltage in the rotating coordinate system, u q is the q-axis component of the three-phase voltage in the rotating coordinate system, e d is the d-axis component of the grid voltage in the rotating coordinate system, e q is the q-axis component of the grid voltage in the rotating coordinate system, L is the line inductance, and ω is the mechanical angular velocity of the virtual synchronous generator. From formula (17), we can get Figure 4 The voltage and current double closed-loop control block diagram shown in the figure, where i dref and i qref for i d 、i q The reference value, k p and k i are the proportional coefficient and integral coefficient of the PI controller respectively.

[0158] Virtual synchronous control is a power system control strategy that aims to make renewable energy generation systems have stability and control performance similar to traditional synchronous generators. By adjusting the control parameters, virtual synchronous control can make renewable energy generation systems respond quickly to changes in grid frequency and voltage, thereby improving grid stability and reliability. The virtual synchronous control model includes the mechanical equations of the virtual synchronous generator and the virtual exciter control loop.

[0159] Step S1 also includes:

[0160] Step S13, establishing the mechanical equation of the virtual synchronous generator and the virtual exciter control equation; wherein the mechanical equation of the virtual synchronous generator is:

[0161]

[0162] Where J is the moment of inertia of the virtual synchronous generator. When the pole pair number is 1, the mechanical angular velocity ω of the virtual synchronous generator is its electrical angular velocity, ω0 is the synchronous angular velocity of the power grid, and P is m is the mechanical power of the virtual synchronous generator, P e is the electromagnetic power of the virtual synchronous generator, T m is the mechanical torque, T eis the electromagnetic torque, T d is the damping torque, D is the damping coefficient, and δ is the rotor angle. From formula (18), we can get Figure 5 The pf control loop in the virtual synchronous generator is shown.

[0163] Based on the idea of ​​reactive power-voltage droop control, the design scheme of the virtual exciter control loop can be obtained, and the expression of the virtual exciter control equation is:

[0164]

[0165] In the formula, E is the virtual electromotive force, E0 is the set value of the no-load electromotive force, and U0 is the actual calculated voltage value. U0 is obtained by calculating the grid voltage u0 through the RMS module. The RMS module represents the effective value of the voltage. n is the rated voltage value, Q ref is the reference value of reactive power, Q e is the actual reactive power; is the reactive power integral coefficient. From formula (19), we can get Figure 5 The QE control loop in the virtual synchronous generator is shown.

[0166] In order to better understand the control method of the grid-type flexible DC transmission system based on deep reinforcement learning in the above embodiment, Figure 6 The overall control block diagram of the grid-type flexible direct current transmission system control based on deep reinforcement learning in this application is shown. Among them, the grid-type flexible direct current transmission system 101, as the core of power transmission, provides key electrical parameters to the virtual synchronous motor 102, including the d-axis current (i d ), q-axis current (i q ), d-axis load current (i Ld ), q-axis load current (i Lq )、d-axis voltage (u d ) and q-axis voltage (u q ). During the interaction process, the virtual synchronous motor 102 first uses its built-in power calculation module to calculate the power of the received u d 、u q 、i d and i q Parameters, accurately calculate the active power (P e ) and reactive power (Q e ). Then, the virtual synchronous motor 102 generates and sends status information x according to the operating status of the system. t and the reward signal r t To agent 103. Here, x t and r tare the environment state and immediate reward in reinforcement learning, providing a basis for learning and optimization for the agent 103. As the core of control decision-making, the agent 103 receives x t and r t After that, the reinforcement learning algorithm is used for strategy iteration and optimization. Based on this optimization process, the intelligent agent 103 outputs the key control instruction, namely the additional power input value, which is directly applied to the Pf (active power-frequency) control loop in the virtual synchronous control model. This additional power input value is dynamically adjusted according to system requirements to optimize the power distribution and frequency stability of the system. Finally, the virtual synchronous motor 102 implements precise control of the grid-type flexible direct current transmission system 101 according to the additional power input value output by the intelligent agent 103. This closed-loop control mechanism effectively improves the response speed and stability of the system, ensuring the reliability and efficiency of power transmission.

[0167] In addition, in order to verify the effectiveness of the control method of the grid-type flexible DC transmission system based on deep reinforcement learning, this embodiment uses traditional PI control, DDPG algorithm and the improved TD3 algorithm of this application to perform additional damping control on VSG. When t = 0 ~ 0.5s, the short circuit ratio (SCR) of the receiving end grid is 1.4, which is a weak grid; when t = 0.5s, the SCR of the receiving end grid increases to 4.1, and the grid is stronger. Figure 7-10 The control performance of different control methods when the receiving-end power grid strength changes is compared. Figure 7 The comparison of the frequency control performance of the three control methods under different receiving-end power grid strengths is shown. Figure 8 The control performance comparison of three control methods on active power under different receiving-end grid strengths is shown. Fig. 9 The control performance comparison of reactive power by three control methods under different receiving-end grid strengths is shown. Fig.10 The control performance comparison of the three control methods on the grid connection point voltage under different receiving-end grid strengths is shown.

[0168] from Figure 7-Figure 10 It can be directly observed that when the short-circuit ratio increases to 4.1, that is, when t=0.5s, the active power and reactive power change most smoothly under the TD3 additional damping control, followed by the DDPG algorithm. The system frequency and PCC point phase voltage (i.e., grid connection point voltage) under the TD3 additional damping control are least impacted, and the system recovers stability within 0.15s. The DDPG algorithm is second, and the system recovers stability within 0.25s. Under traditional PI control, the active power and reactive power amplitudes are large, the system frequency and PCC point phase voltage oscillate greatly, and it takes 1s for the system to recover stability. Therefore, when the strength of the receiving-end power grid changes, the additional damping control based on the improved TD3 algorithm proposed in this application can better improve the stability of the power grid.

[0169] Based on the same inventive concept, the embodiment of the present application also provides a deep reinforcement learning-based meshed flexible DC transmission system control device for implementing the deep reinforcement learning-based meshed flexible DC transmission system control method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the deep reinforcement learning-based meshed flexible DC transmission system control device provided below can refer to the above limitations on the deep reinforcement learning-based meshed flexible DC transmission system control method, which will not be repeated here.

[0170] In one embodiment, Fig.11 As shown, a grid-type flexible direct current transmission system control device 900 based on deep reinforcement learning is provided, comprising: a system control model establishment module 901, a Markov decision process establishment module 902 and an agent training module 903, wherein:

[0171] The system control model establishment module 901 is used to establish a grid-forming flexible DC transmission system control model; the grid-forming flexible DC transmission system control model includes a voltage and current double closed-loop control model and a virtual synchronous control model;

[0172] A Markov decision process establishing module 902 is used to establish a Markov decision process based on a grid-type flexible DC transmission system control model;

[0173] The intelligent agent training module 903 is used to train the intelligent agent using a dual-pool-dual-delay deep deterministic policy gradient algorithm; the intelligent agent provides an additional power input value to the Pf control loop in the virtual synchronous control to improve the stability of the grid-type flexible direct current transmission system; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the intelligent agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

[0174] In one embodiment, the agent training module 903 is also used to obtain the experience tuple (x t , a t , r t , r t+1 ) and discount factor γ, calculate the target Q value according to the formula shown in formula (7); according to the target Q value, with the goal of minimizing the mean square error loss function of the value network, perform gradient descent update on the parameters of the value network; update the parameters of the target value network based on the parameters of the value network; with the goal of maximizing the expected cumulative reward function, perform gradient descent update on the parameters of the policy network; update the parameters of the target policy network based on the parameters of the policy network.

[0175] In one embodiment, the system control model building module 901 is also used to obtain the voltage loop equation of the main circuit of the grid-type flexible direct current transmission system; obtain the voltage and current dual closed-loop control model after dq transformation and Laplace transformation of the voltage loop equation; and establish the mechanical equation of the virtual synchronous generator and the virtual exciter control equation.

[0176] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0178] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0179] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A control method for a grid-type flexible direct current transmission system based on deep reinforcement learning, characterized in that: The method comprises: Step S1, establishing a grid-forming flexible direct current transmission system control model; the grid-forming flexible direct current transmission system control model includes a voltage and current double closed-loop control model and a virtual synchronous control model; Step S2, establishing a Markov decision process based on the grid-type flexible direct current transmission system control model; Step S3, adopting a dual-pool-dual-delay deep deterministic policy gradient algorithm to train an intelligent agent, the intelligent agent improves the stability of the grid-type flexible direct current transmission system by providing an additional power input value to the Pf control loop in the virtual synchronous control model; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the intelligent agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

2. The control method of a meshed flexible direct current transmission system based on deep reinforcement learning according to claim 1, characterized in that: The Markov decision process includes a state value function, an action value function, an agent environment state, and an agent output action; wherein, The state value function is: Among them, V π (x) is the state value function, which represents the total return expected from starting from state x and executing strategy π, r t represents the return at time t, x t represents the state at time t, E represents the expected function, γ is the discount factor, the value range is [0,1], k is the time step, r t+k+1 represents the return at time t+k+1; The action value function is: In the formula, Q π (x,a) is the action value function, which indicates the total return expected after the agent performs action a in the initial state x and continues to execute according to the strategy π. t represents the action performed by the agent at time t; The agent environment state x t The expression is: x t ={Δf,ΔP} In the formula, Δf represents the difference between the real-time frequency and the reference frequency, and ΔP represents the difference between the active power and the reference active power; The agent outputs action a t Defined as: a t ={P Δf } Where P Δf Indicates the additional power input value of the Pf control loop in virtual synchronous control.

3. The control method of a meshed flexible direct current transmission system based on deep reinforcement learning according to claim 2 is characterized in that: The return r at time t t The calculation formula is: r t =-(λ1|f ref -f t |+λ2|P ref -P t |) In the formula, f ref is the reference frequency, f t is the actual output frequency of the virtual synchronous generator at time t, P ref is the reference active power, P t is the actual output power of the virtual synchronous generator at time t, λ1 is the frequency weight coefficient, and λ2 is the active power weight coefficient.

4. The control method of a grid-type flexible direct current transmission system based on deep reinforcement learning according to claim 1 is characterized in that: The step S3 comprises: Step S31, obtain the experience tuple (x t , a t , r t , x t+1 ) and discount factor γ, according to the formula Calculate the target Q value; where y t is the target Q value at time t, represents the action value function, x t+1 represents the state at time t+1, represents the action at time t+1 generated by the target policy network, i=1,2; Step S32, based on the target Q value, the parameters of the value network are updated by gradient descent with the goal of minimizing the mean square error loss function of the value network, that is, In the formula, α Q is the learning rate of the value network, is the gradient operator, K(w i ) is the mean square error loss function of the value network; wherein the expression of the mean square error loss function of the value network is: Where D is the experience replay buffer; w i Represents the value network Q i Parameters, i = 1, 2; Indicates that the parameter is w i Value Network The calculated Q value; Step S33, updating the parameters of the target value network based on the parameters of the value network, that is w i '←τw i +(1-τ)w i ' In the formula, w' i is the parameter of the target value network, i=1, 2; τ is the soft update coefficient, and its value range is (0, 1); Step S34, with the goal of maximizing the expected cumulative reward function, the parameters θ of the policy network are updated by gradient descent, that is, In the formula, α π is the learning rate of the policy network, H(θ) is the expected cumulative reward function; wherein the expression of the expected cumulative reward function is: In the formula, Q θ (x t ,π θ (x t )) means in state x t Next, execute action π θ (x t ) after the expected total return, π θ (x t ) indicates that the policy network with parameter θ is in state x t The following actions are generated. Step S35, updating the parameters of the target policy network based on the parameters of the policy network, that is θ'←τθ+(1-τ)θ' Where θ' is the parameter of the target policy network.

5. The control method of a meshed flexible direct current transmission system based on deep reinforcement learning according to claim 4 is characterized in that: The action at time t+1 generated by the target strategy network is the action after adding Gaussian noise, that is, In the formula, π(x t+1 ) is the next state x t+1 When , N(0,σ) represents Gaussian noise with mean 0 and variance σ, and clip is a function that limits the Gaussian noise to the range of [-C,C].

6. The control method of a meshed flexible direct current transmission system based on deep reinforcement learning according to claim 1, characterized in that: The steps of dividing the success experience pool and the failure experience pool include: Step S30, dividing the experience pool into the success experience pool and the failure experience pool based on a judgment function; the expression of the judgment function is: Where, t v is the total training time in seconds.

7. The control method of a meshed flexible direct current transmission system based on deep reinforcement learning according to claim 1, characterized in that: The step S1 comprises: Step S11, obtaining a voltage loop equation of a main circuit of a grid-forming flexible direct current transmission system; the voltage loop equation is: In the formula, R represents the resistance of the line, i sn represents the output current of the converter, L represents the reactance of the line, e n Indicates the voltage of the power grid, U no represents the output voltage of the converter, n=A, B, C; Step S12, performing dq transformation and Laplace transformation on the voltage loop equation to obtain the voltage-current dual closed-loop control model; the expression of the voltage-current dual closed-loop control model is: Where s is the Laplace operator, i d is the d-axis component of the three-phase current in the rotating coordinate system, i q is the q-axis component of the three-phase current in the rotating coordinate system, u d is the d-axis component of the three-phase voltage in the rotating coordinate system, u q is the q-axis component of the three-phase voltage in the rotating coordinate system, e d is the d-axis component of the grid voltage in the rotating coordinate system, e q is the q-axis component of the grid voltage in the rotating coordinate system, L is the line inductance, and ω is the mechanical angular velocity of the virtual synchronous generator; The virtual synchronous control model includes the mechanical equations of the virtual synchronous generator and the virtual exciter control equations; the step S1 also includes: Step S13, establishing the mechanical equation of the virtual synchronous generator and the virtual exciter control equation; wherein the mechanical equation of the virtual synchronous generator is: Where J is the moment of inertia of the virtual synchronous generator, ω0 is the synchronous angular velocity of the power grid, and P m is the mechanical power of the virtual synchronous generator, P e is the electromagnetic power of the virtual synchronous generator, T m is the mechanical torque, T e is the electromagnetic torque, T d is the damping torque, D is the damping coefficient, and δ is the rotor angle; The mathematical expression of the virtual exciter control loop is: In the formula, E is the virtual electromotive force, E0 is the set value of the no-load electromotive force, U is the actual calculated voltage value, and U n is the rated voltage value, Q ref is the reference value of reactive power, Q e is the actual reactive power; is the reactive power integral coefficient.

8. A control device for a grid-type flexible direct current transmission system based on deep reinforcement learning, characterized in that: The device comprises: A system control model building module is used to build a grid-type flexible direct current transmission system control model; the grid-type flexible direct current transmission system control model includes a voltage and current double closed-loop control model and a virtual synchronous control model; A Markov decision process establishment module, used for establishing a Markov decision process based on the grid-type flexible direct current transmission system control model; An agent training module is used to train an agent using a dual-pool-dual-delay deep deterministic policy gradient algorithm; the agent provides an additional power input value to the Pf control loop in the virtual synchronous control to improve the stability of the grid-type flexible direct current transmission system; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images