Network-formation type flexible HVDC system control method based on deep reinforcement learning

Abstract

The application relates to the technical field of flexible direct current power transmission systems, in particular to a network-constructing flexible direct current power transmission system control method based on deep reinforcement learning, which comprises the following steps: establishing a network-constructing flexible direct current power transmission system control model, establishing a Markov decision process based on the network-constructing flexible direct current power transmission system control model, training an intelligent agent by adopting a double-pool-double-delay deep deterministic policy gradient algorithm, and improving the stability of the network-constructing flexible direct current power transmission system by providing an additional power input value for a P-f control loop in a virtual synchronous control model by the intelligent agent. The method introduces a double-value function network, uses a smaller Q value in the double-value function network to calculate a target value, can avoid overestimation of the value of some state-action pairs, and improves the overall stability of the VSG system.

Description

Technical Field

[0001] The present application relates to the technical field of flexible direct current transmission systems, and in particular to a control method for a meshed flexible direct current transmission system based on deep reinforcement learning. Background Art

[0002] With the continuous advancement and increasing intelligence of power systems, virtual synchronous generator (VSG) technology is being used more and more widely in flexible direct current transmission. The promotion of this technology helps to stabilize the frequency of the power grid and improve the power quality. However, VSG still faces many stability issues in practical applications, especially when disturbances occur in the power grid, its dynamic response and control stability become one of the key research challenges. In order to further improve the stability and performance of VSG, researchers are constantly exploring new control strategies. Among them, deep reinforcement learning (DRL), as an emerging artificial intelligence technology, has demonstrated its great potential in controlling complex systems in recent years. DRL has adaptive and optimal decision-making capabilities, and can make reasonable control decisions in dynamic and nonlinear environments, which makes it a powerful tool for solving VSG stability problems.

[0003] However, traditional reinforcement learning methods face the curse of dimensionality when dealing with tasks involving continuous state variables, as the number of discretized states increases exponentially with the increase in spatial dimensions. This makes effective learning impractical. Currently, the Deep Deterministic Policy Gradient (DDPG) algorithm can maintain the continuity of action and state spaces, effectively circumventing the curse of dimensionality. For example, patent publication number CN115276093A provides a DDPG-based method for adaptive parameter control of virtual synchronous generators. This method involves first establishing a virtual synchronous generator control model, then collecting the active power P1, reactive power Q1, system frequency f1, and voltage U1 output by the control model to form training data. The DDPG algorithm is then used to train an intelligent agent to obtain the optimal control variables for the VSG system under the corresponding state: moment of inertia J and damping coefficient D. After training, the intelligent agent is packaged. Finally, in actual operation, the intelligent agent output is adjusted according to the real-time environmental state to control the system's frequency response, thereby improving system performance.

[0004] However, the DDPG algorithm suffers from the phenomenon of "overestimation" in practical applications. This phenomenon stems from the fact that the algorithm may overestimate the value of certain state-action pairs during training, causing the learned strategy to be unstable in practical applications, thereby affecting the overall stability of the VSG system. Summary of the Invention

[0005] Based on this, it is necessary to provide a grid-type flexible direct current transmission system control method, device, computer equipment, storage medium and computer program product based on deep reinforcement learning to address the problem that the DDPG algorithm may over-estimate the value of certain state-action pairs during training, resulting in unstable performance of the learned strategy in practical applications.

[0006] In a first aspect, the present application provides a control method for a grid-type flexible direct current transmission system based on deep reinforcement learning. The method comprises:

[0007] Step S1, establishing a grid-type flexible DC transmission system control model; the grid-type flexible DC transmission system control model includes a voltage and current dual closed-loop control model and a virtual synchronous control model;

[0008] Step S2, establishing a Markov decision process based on the grid-type flexible HVDC system control model;

[0009] Step S3, a dual-pool-dual-delay deep deterministic policy gradient algorithm is used to train the intelligent agent. The intelligent agent improves the stability of the grid-type flexible direct current transmission system by providing an additional power input value to the Pf control loop in the virtual synchronous control model. The dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool. The intelligent agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

[0010] In one embodiment, the Markov decision process includes a state-value function, an action-value function, an agent-environment state, and an agent-output action; wherein,

[0011] The state value function is:

[0012]

[0013] Among them, V π (x) is the state value function, which represents the total return expected from starting from state x and executing strategy π, r t To represent the return at time t, x t represents the state at time t, E represents the expected function, γ is the discount factor, the value range is [0,1], k is the time step, r t+k+1 represents the return at time t+k+1;

[0014] The action-value function is:

[0015]

[0016] Where Q π(x,a) is the action value function, which indicates the total return expected after the agent performs action a in the initial state x and continues to execute according to the strategy π. t represents the action performed by the agent at time t;

[0017] Agent environment state x t The expression is:

[0018] x t ={Δf,ΔP}

[0019] Where Δf represents the difference between the real-time frequency and the reference frequency, and ΔP represents the difference between the active power and the reference active power.

[0020] The agent outputs action a t Defined as:

[0021] a t ={PΔ f}

[0022] Where, P Δf Indicates the additional power input value of the Pf control loop in virtual synchronous control.

[0023] In one embodiment, the reward r at time t t The calculation formula is:

[0024] r t =-(λ1|f ref -f t |+λ2|P ref -P t |)

[0025] Where, f ref is the reference frequency, f t is the actual output frequency of the virtual synchronous generator at time t, P ref is the reference active power, P t is the actual output power of the virtual synchronous generator at time t, λ1 is the frequency weight coefficient, and λ2 is the active power weight coefficient.

[0026] In one embodiment, step S3 includes:

[0027] Step S31, obtain the experience tuple (x t , a t , r t , x t+1 ) and discount factor γ, according to the formula

[0028]

[0029] Calculate the target Q value; where y tis the target Q value at time t, represents the action value function, x t+1 represents the state at time t+1, represents the action at time t+1 generated by the target policy network, i=1,2;

[0030] Step S32, based on the target Q value, the parameters of the value network are updated by gradient descent with the goal of minimizing the mean square error loss function of the value network, that is,

[0031]

[0032] Where, α Q is the learning rate of the value network, is the gradient operator, K(w i ) is the mean square error loss function of the value network; where the expression of the mean square error loss function of the value network is:

[0033]

[0034] Where D is the experience replay buffer; w i Represents the value network Q i Parameters, i = 1, 2; Indicates that the parameter is w i Value Network The calculated Q value;

[0035] Step S33: Update the parameters of the target value network based on the parameters of the value network, i.e.

[0036] w′ i ←τw i +(1-τ)w′ i

[0037] Where w' i is the parameter of the target value network, i=1,2; τ is the soft update coefficient, the value range is (0,1);

[0038] Step S34, with the goal of maximizing the expected cumulative reward function, the parameters θ of the policy network are updated by gradient descent, that is,

[0039]

[0040] Where, α π is the learning rate of the policy network, H(θ) is the expected cumulative reward function; the expression of the expected cumulative reward function is:

[0041]

[0042] Where Q θ (xt ,π θ (x t )) means in state x t Next, perform action π θ (x t ) after the expected total return, π θ (x t ) represents the policy network with parameter θ in state x t The following actions are generated.

[0043] Step S35: Update the parameters of the target policy network based on the parameters of the policy network, i.e.

[0044] θ'←τθ+(1-τ)θ'

[0045] Where θ' is the parameter of the target policy network.

[0046] In one embodiment, the action at time t+1 generated by the target policy network is the action after adding Gaussian noise, that is,

[0047]

[0048] In the formula, π(x t+1 ) is the next state x t+1 When , N(0,σ) represents Gaussian noise with mean 0 and variance σ, and clip is a function that limits the Gaussian noise to the range of [-C,C].

[0049] In one embodiment, the step of dividing the successful experience pool and the failed experience pool includes:

[0050] Step S30: Divide the experience pool into a success experience pool and a failure experience pool based on a judgment function; the judgment function is expressed as:

[0051]

[0052] Where, t v is the total training time in seconds.

[0053] In one embodiment, step S1 includes:

[0054] Step S11, obtaining a voltage loop equation of the main circuit of the grid-type flexible DC transmission system; the voltage loop equation is:

[0055]

[0056] In the formula, R represents the resistance of the circuit, i sn represents the output current of the converter, L represents the reactance of the line, e n Indicates the voltage of the power grid, U noIndicates the output voltage of the converter, n = A, B, C;

[0057] Step S12: Performing dq transformation and Laplace transformation on the voltage loop equation to obtain a voltage-current dual closed-loop control model; the expression of the voltage-current dual closed-loop control model is:

[0058]

[0059] Where s is the Laplace operator, i d is the d-axis component of the three-phase current in the rotating coordinate system, i q is the q-axis component of the three-phase current in the rotating coordinate system, u d is the d-axis component of the three-phase voltage in the rotating coordinate system, u q is the q-axis component of the three-phase voltage in the rotating coordinate system, e d is the d-axis component of the grid voltage in the rotating coordinate system, e q is the q-axis component of the grid voltage in the rotating coordinate system, L is the line inductance, and ω is the mechanical angular velocity of the virtual synchronous generator;

[0060] The virtual synchronous control model includes the mechanical equations of the virtual synchronous generator and the virtual exciter control equations; step S1 also includes:

[0061] Step S13: Establishing the mechanical equation of the virtual synchronous generator and the virtual exciter control equation; wherein the mechanical equation of the virtual synchronous generator is:

[0062]

[0063] Where J is the moment of inertia of the virtual synchronous generator, ω0 is the synchronous angular velocity of the power grid, and P m is the mechanical power of the virtual synchronous generator, P e is the electromagnetic power of the virtual synchronous generator, T m is the mechanical torque, T e is the electromagnetic torque, T d is the damping torque, D is the damping coefficient, and δ is the rotor angle;

[0064] The mathematical expression of the virtual exciter control loop is:

[0065]

[0066] Where, E is the virtual electromotive force, E0 is the set value of the no-load electromotive force, U is the actual calculated voltage value, and U n is the rated voltage value, Q ref is the reference value of reactive power, Q e is the actual reactive power; is the reactive power integral coefficient.

[0067] In a second aspect, the present application also provides a control device for a grid-type flexible direct current transmission system based on deep reinforcement learning. The device comprises:

[0068] A system control model building module is used to establish a grid-type flexible DC transmission system control model; the grid-type flexible DC transmission system control model includes a voltage and current dual closed-loop control model and a virtual synchronous control model;

[0069] A Markov decision process establishment module is used to establish a Markov decision process based on the grid-type flexible DC transmission system control model;

[0070] An agent training module is used to train the agent using a dual-pool-dual-delay deep deterministic policy gradient algorithm; the agent improves the stability of the grid-type flexible direct current transmission system by providing additional power input values ​​to the Pf control loop in the virtual synchronous control; the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

[0071] In a third aspect, the present application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the steps in the method for controlling a grid-type flexible direct current transmission system described in the technical solution of the first aspect and any of its optional solutions.

[0072] In a fourth aspect, the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the grid-type flexible direct current transmission system control method described in the above-mentioned first aspect technical solution and any optional solution thereof are implemented.

[0073] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the grid-type flexible direct current transmission system control method described in the above-mentioned first aspect technical solution and any optional solution thereof.

[0074] The control method, device, computer equipment, storage medium and computer program product of the grid-type flexible direct current transmission system based on deep reinforcement learning in the present application first establish a grid-type flexible direct current transmission system control model and establish a Markov decision process based on the grid-type flexible direct current transmission system control model, which can accurately reflect the dynamic characteristics and control requirements of the system. Secondly, by adopting a dual-pool-dual-delay deep deterministic policy gradient algorithm to train the intelligent agent, a dual-valued function network is introduced, and the smaller Q value in the dual-valued function network is used to calculate the target value, which can avoid over-estimation of the value of certain state-action pairs and improve the overall stability of the VSG system. At the same time, by dividing the experience pool into a successful experience pool and a failed experience pool, and sampling data from the two experience pools to train the intelligent agent, the data distribution during the training process can be further balanced, the over-estimation problem caused by data bias can be reduced, and the reliability of the additional power input value is improved, thereby further improving the stability of the VSG system. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 1. A schematic flow chart of a control method for a grid-type flexible direct current transmission system based on deep reinforcement learning in one embodiment;

[0076] Figure 2 Schematic diagram of a Markov decision process in one embodiment;

[0077] Figure 3 A schematic diagram of a dual-pool-dual-delay deep deterministic policy gradient algorithm for training an intelligent agent in one embodiment;

[0078] Figure 4 This is a block diagram of voltage and current dual closed-loop control in one embodiment;

[0079] Figure 5 is an overall control block diagram of a virtual synchronous generator in one embodiment;

[0080] Figure 6 1 is an overall control block diagram of a control method for a grid-type flexible direct current transmission system based on deep reinforcement learning in one embodiment;

[0081] Figure 7 A comparison chart of frequency control performance of three control methods under different receiving-end power grid strengths in one embodiment;

[0082] Figure 8 1. A comparison chart of the control performance of active power of three control methods under different receiving-end grid strengths in one embodiment;

[0083] Figure 9 1. A graph comparing the control performance of reactive power by three control methods under different receiving-end grid strengths in one embodiment;

[0084] Figure 10 A comparison chart of the control performance of three control methods on the grid connection point voltage under different receiving end grid strengths in one embodiment;

[0085] Figure 11 The figure is a structural block diagram of a grid-type flexible direct current transmission system control device based on deep reinforcement learning in one embodiment. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0087] In one embodiment, Figure 1 As shown, a control method for a grid-type flexible DC transmission system based on deep reinforcement learning is provided. This embodiment uses the method applied to a terminal as an example for illustration. It is understandable that the method can also be applied to a server, or to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. The method includes the following steps:

[0088] Step S1: establishing a grid-type flexible DC transmission system control model; the grid-type flexible DC transmission system control model includes a voltage and current dual closed-loop control model and a virtual synchronous control model.

[0089] Among them, the virtual synchronous control model includes the Pf control loop and the QE control loop, which are responsible for the electromagnetic power P provided by the power calculation module. e and actual reactive power Q e , dynamically generate virtual electromotive force E and mechanical angular velocity ω. The virtual electromotive force E serves as the input reference voltage amplitude of the voltage and current dual closed-loop control model, and the mechanical angular velocity ω serves as the input reference voltage phase angle of the voltage and current dual closed-loop control model. The voltage and current dual closed-loop control model includes voltage outer loop control and current inner loop control. The voltage and current dual closed-loop control model is based on the three-phase current component i in the rotating coordinate system. d 、i q and three-phase voltage components u d 、u q , combined with the virtual electromotive force E and mechanical angular velocity ω output by the virtual synchronous control model, dual closed-loop control of the voltage outer loop and the current inner loop is performed to ensure stable operation of the system.

[0090] Step S2: establishing a Markov decision process based on the grid-type flexible HVDC system control model.

[0091] Reinforcement learning is a self-learning method in which the agent obtains the optimal strategy through trial and error in the process of interacting with the environment. The flow chart is as follows:Figure 2 As shown. The Markov Decision Process (MDP) is constructed based on a set of interacting objects, namely the agent and the environment, and its elements include the state x t 、Action a t , strategy π and reward r t In the MDP simulation, the agent perceives the current system state and takes actions on the environment according to the strategy, thereby changing the state of the environment and receiving rewards. The accumulation of rewards over time is called returns.

[0092] Specifically, the Markov decision process based on the grid-type flexible DC transmission system control model includes defining the state value function, action value function, agent environment state (i.e. state x t ) and the agent output action (i.e. action a t ). In this embodiment, the agent environment state is set to:

[0093] x t ={Δf,ΔP} (1)

[0094] Where Δf represents the difference between the real-time frequency f and the reference frequency f ref The difference between the real-time active power P and the reference active power P is ΔP. ref The real-time active power P can be calculated by the current and voltage in the power system.

[0095] The agent output action is set to:

[0096] a t ={P Δf} (2)

[0097] Where, P Δf represents the additional power input value of the Pf control loop in the virtual synchronous control model. During the continuous training iteration process, the agent continuously optimizes its strategy π and gradually adjusts the action value output in each state to ensure that the most appropriate action is selected in each current state.

[0098] Specifically, such as Figure 6 As shown in Figure 1, the additional power input value output by the agent is added to the original electromagnetic power Pe, and the result is used as the updated electromagnetic power Pe', which is directly applied to the calculation of the pf control loop.

[0099] The value function can be used to evaluate the expected return of the strategy π. The value function can include the state value function and the action value function. Among them, the state value function V π (x) represents the total return expected from starting from state x and executing strategy π, and its function expression is:

[0100]

[0101] where r t represents the reward at time t, which can be calculated by the reward function as shown in equation (6), x t represents the state at time t, E represents the expectation function, γ is the discount factor with a value range of [0, 1], k is the time step, and r t+k+1 represents the reward at time t+k+1.

[0102] The action value function Q π (x, a) represents the expected cumulative reward that will be obtained by continuing to follow the policy π after performing action a in the initial state x, and its functional expression is:

[0103]

[0104] where a t represents the action performed by the agent at time t, as shown in equation (2); and r t+k+1 represents the reward at time t+k+1, which can also be calculated by the reward function as shown in equation (6).

[0105] Specifically, Q π (x, a) evaluates the expected value of the cumulative reward that the agent can obtain in the future by following the policy π after making a certain action a in a specific state x. This function not only considers the immediate reward of the current state and action, but also includes the discounted sum of the rewards of all possible future states and actions. Therefore, Q π (x, a) provides an index for the agent to measure the quality of its decision-making, helping it to choose an action strategy that can maximize long-term rewards. In the process of reinforcement learning, by continuously updating and optimizing the Q function, the agent can gradually learn the optimal strategy and make the best decisions in various complex environments. The state value function V π (x) is the expectation function of the action value function Q π (x, a) with respect to action a. Specifically, the state value function V π (x) evaluates the expected total reward when the policy π is executed in state x. It calculates this by taking a weighted average of all possible actions a, where the weight is the probability of the policy π selecting action a in state x. Thus, V π (x) can be represented by Q π (x, a) as:

[0106] V π (x) = E[r t |x t =x] = E a [Q π(x, a) (5)

[0107] Q π (x, a) is a state value function V π (x) and action reward feedback value r t , and E is an expectation function.

[0108] Step S3, training the agent by using a double-pool-double-delay deep deterministic policy gradient algorithm; the agent provides an additional power input value to the P-f control loop in virtual synchronous control, so as to improve the stability of the meshed flexible HVDC power transmission system.

[0109] In view of the problem of overestimation of Q value in DDPG, the embodiment adopts a TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm to train the agent. The TD3 algorithm uses an Actor-Critic algorithm architecture and introduces a deep neural network (DNN). In the Actor-Critic algorithm architecture, the Actor (policy network) and the Critic (value network) cooperate with each other. The Actor is responsible for generating actions, while the Critic is responsible for evaluating the goodness of these actions and providing feedback to guide the learning of the Actor. Under this framework, the network parameters of the policy network and the value network are usually updated during the training process to optimize the ability of the policy network to generate actions. The core of the TD3 algorithm lies in that it uses two independent value networks to calculate action value functions Q1 and Q2. When updating the value network, the smaller value of Q1 and Q2 is selected to calculate the target value, so as to alleviate the problem of overestimation of Q value.

[0110] In addition, experience replay is a key component in the TD3 algorithm, which breaks the time correlation of data by storing past experience samples, and when the policy network and the value network are trained, a small batch of experience is randomly sampled from the experience replay buffer. However, when there is only one experience pool, the agent is prone to fall into a local optimum during the training process. Therefore, the embodiment proposes a double-experience-pool training method, specifically, the experience replay buffer includes two experience pools: a failure experience pool and a success experience pool. The success experience pool stores successful data, i.e. the action a t , state x t and reward r t data that leads to improvement of system stability; the failure experience pool stores failure data, i.e. the action a t , state x t and reward r tData. The agent randomly samples experience tuples (x t , a t , r t , x t+1 ) to conduct training, so that we can learn from successful experiences and draw lessons from failures to avoid repeating mistakes.

[0111] Specifically, it can be based on a key indicator in the experience tuple, such as the state data x t or reward data t To determine whether the experience tuple belongs to the successful experience pool or the failed experience pool. t For example, we first set a judgment condition, which can be a predefined threshold or a complex expression that is dynamically adjusted according to task requirements or training process. t When this judgment condition is met, the experience tuple is considered to represent successful experience, so it is divided into the successful experience pool. On the contrary, if the reward data r t If the judgment condition is not met, the experience tuple is considered to represent a failed experience and is divided into the failed experience pool. By dividing the experience pool in this way, it is possible to effectively distinguish valuable successful experiences from failed experiences that need improvement from a large amount of experience data, thereby providing more targeted data support for the learning and optimization of the intelligent agent.

[0112] The deep reinforcement learning-based control method for a grid-type flexible direct current transmission system in this embodiment first establishes a grid-type flexible direct current transmission system control model and then develops a Markov decision process based on the grid-type flexible direct current transmission system control model, which accurately reflects the system's dynamic characteristics and control requirements. Secondly, by training the intelligent agent using a dual-pool, dual-delay deep deterministic policy gradient algorithm, a dual-valued function network is introduced. The smaller Q value in the dual-valued function network is used to calculate the target value, which can avoid overestimation of the value of certain state-action pairs and improve the overall stability of the VSG system. Furthermore, by dividing the experience pool into a successful experience pool and a failed experience pool and sampling data from each experience pool to train the intelligent agent, the data distribution during training can be further balanced, reducing the problem of overestimation caused by data bias, improving the reliability of the additional power input value, and thus further improving the stability of the VSG system.

[0113] The reward function is the key signal that drives the agent to explore and discover the best action strategy. In the grid-type flexible DC controller, frequency stability and power stability are two important factors to ensure the reliable operation of the system. Therefore, in order to optimize the performance of the controller and ensure the stable operation of the system, the setting of the reward function must fully consider these factors. Therefore, the reward function needs to be able to reflect the frequency fluctuations and power output deviations of the system in real time. Specifically, by incorporating frequency stability and power stability into the reward function, the agent can continuously adjust its action strategy during the learning process to maximize the stability of these key performance indicators. When the system frequency and power output reach the expected stable state, the agent will receive a higher reward, otherwise it will be punished. In this way, the agent will be guided to find the optimal control strategy that can maintain the stability of the system. In one embodiment, the agent's reward function r t Used to calculate the return r at time t t , whose expression is:

[0114] r t =-(λ1|f ref -f t |+λ2|P ref -P t |) (6)

[0115] Where, f ref is the reference frequency, f t is the actual output frequency of the virtual synchronous generator at time t, P ref is the reference active power, P t is the actual power output of the virtual synchronous generator at time t, λ1 is the frequency weight coefficient, and λ2 is the active power weight coefficient. λ1 and λ2 can be continuously adjusted and tested by those skilled in the art based on the actual training process.

[0116] The reward function in this embodiment can instantly capture and quantify the dynamic changes of system frequency fluctuations and power output deviations, effectively guiding the intelligent agent to explore and execute optimal control strategies that can significantly improve the stability of the virtual synchronous generator (VSG), thereby significantly enhancing the stability of the VSG system.

[0117] In one embodiment, step S3 includes:

[0118] Step S31, obtain the experience tuple (x t , a t , r t , x t+1 ) and discount factor γ, according to the formula:

[0119]

[0120] Calculate the target Q value; where yt is the target Q value at time t, represents the action value function, x t+1 represents the state at time t+1, represents the action at time t+1 generated by the target policy network, where i = 1, 2. The discount factor γ determines the weight of future rewards in the Q-value update, that is, the degree of attenuation of future rewards. Its value range is generally between 0 and 1. When the discount factor is close to 1, the agent will pay more attention to future rewards, which helps optimize the long-term strategy. When the discount factor is close to 0, the agent will pay more attention to immediate rewards and prioritize short-term returns.

[0121] Specifically, such as Figure 3 As shown in the figure, the agent first randomly samples experience tuples (x from the success experience pool and the failure experience pool) according to a predetermined ratio, such as 70% success experience and 30% failure experience, or a ratio adjusted according to specific needs. t , a t , r t , x t+1 ). After the sampling is completed, the target policy network in the agent is based on the next state x after executing the action in the experience tuple. t+1 Generate the corresponding next action Target value network 1 is based on the next state x after executing the action t+1 and the corresponding next action Calculate the first Q value Right now Target value network 2 is based on the next state x after executing the action t+1 and the corresponding next action Calculate the second Q value Right now Then select the first Q value With the second Q value The smaller value of is used to calculate the target Q value at time t, corresponding to the smaller value in formula (6) part.

[0122] Step S32: Based on the target Q value, the parameters of the value network are updated by gradient descent with the goal of minimizing the mean square error loss function of the value network, that is:

[0123]

[0124] Where, α Q is the learning rate of the value network, is the gradient operator, K(w i ) is the mean square error loss function of the value network; where the expression of the mean square error loss function of the value network is:

[0125]

[0126] Where D is the experience replay buffer, w i Represents the value network Q i Parameters, i = 1, 2, Indicates that the parameter is w i Value Network The calculated Q value.

[0127] Specifically, with the goal of minimizing the mean square error loss function of the value network as shown in formula (8), the parameters of each Q network, namely the value network Q1 and the value network Q2, are updated in the manner shown in formula (7).

[0128] Furthermore, to maintain the stability of the target value network, the parameters of the target value network are not updated directly using gradient descent. Instead, they are soft-updated based on the parameters of the current value network. This soft-update method is controlled by a soft-update coefficient τ (ranging from 0 to 1), which makes the parameter changes of the target value network smoother and helps stabilize the learning process. For details, see step S33.

[0129] Step S33: Update the parameters of the target value network based on the parameters of the value network, namely:

[0130] w′ i ←τw i +(1-τ)w′ i (10)

[0131] Where w′ i is the parameter of the target value network, i = 1, 2; τ is the soft update coefficient, usually a small constant with a value range of (0, 1). Adjusting the soft update coefficient can control the update rate of the target value network. Specifically, according to Equation (9), the parameters w1' of target value network 1 are updated based on the parameters w1 of value network 1, and the parameters w2' of target value network 2 are updated based on the parameters w2 of value network 2.

[0132] The TD3 algorithm introduces a delayed policy update mechanism for policy updates, meaning that the actor network (policy network) is updated only every d steps. This reduces oscillations in the policy network and increases learning stability. The actor network's goal is to maximize the value estimated by the critic network, that is, to find the action that maximizes the expected cumulative reward. The specific steps for updating the policy network's network parameters are shown in steps S34 and S35.

[0133] Step S34, with the goal of maximizing the expected cumulative reward function, the parameters θ of the policy network are updated by gradient descent, that is:

[0134]

[0135] wherein α π is the learning rate of the policy network, and H(θ) is the expected cumulative reward function, wherein the expression of the expected cumulative reward function is:

[0136]

[0137] wherein Q θ (x t ,π θ (x t )) represents the expected total return after performing the action π t (x θ ) in the state x t , and π θ (x t ) represents the action generated by the policy network with parameters θ in the state x t .

[0138] In step S35, the parameters of the target policy network are updated based on the parameters of the policy network, i.e.

[0139] θ'←τθ+(1-τ)θ' (13)

[0140] wherein θ' is the parameter of the target policy network. Similar to the update of the target value network, the parameter of the target policy network is also soft-updated based on the current parameter of the policy network. This updating method helps to maintain the stability and convergence of the policy network.

[0141] The embodiment illustrates the updating mechanism of network parameters under the deep reinforcement learning framework, specifically involving the parameter updating process of the value network and its corresponding target value network, as well as the parameter synchronization optimization of the policy network and the target policy network, which ensures the collaborative evolution of each network and gradually approaches the optimal policy under a specific task. This process not only accelerates the learning speed of the agent, but also significantly improves its decision-making ability and adaptability in complex environments, thereby effectively enhancing the overall stability and running efficiency of the VSG.

[0142] In one embodiment, in order to realize target policy smoothing, the action generated by the policy network at time t+1 is the action after adding Gaussian noise, i.e.

[0143]

[0144] wherein π(x t+1 ) represents the policy function in the next state x t+1 , N(0,σ) represents Gaussian noise with mean 0 and variance σ, and clip is a function that limits the Gaussian noise within the range [-C,C].

[0145] In one embodiment, the steps of dividing the successful experience pool and the failed experience pool include:

[0146] Step S30: Divide the experience pool into a success experience pool and a failure experience pool based on a judgment function; the judgment function is expressed as:

[0147]

[0148] Where, t v is the total training time in seconds.

[0149] Specifically, based on the immediate reward r in each experience tuple in the experience pool t This indicator is used to determine whether the experience tuple is divided into the successful experience pool or the failed experience pool. For example, the experience tuple that meets formula (15) is divided into the successful experience pool, and the experience tuple that does not meet formula (15) is divided into the failed experience pool.

[0150] This embodiment significantly improves the learning efficiency and strategy reliability of the intelligent agent during the training process by constructing a judgment function as the basis for dividing the experience tuples into the success experience pool and the failure experience pool, thereby effectively enhancing the stability of the VSG system.

[0151] In one embodiment, step S1 includes:

[0152] Step S11, obtaining a voltage loop equation of the main circuit of the grid-type flexible DC transmission system; the voltage loop equation is:

[0153]

[0154] In the formula, R represents the resistance of the circuit, i sn represents the output current of the converter, L represents the reactance of the line, e n Indicates the voltage of the power grid, U no Represents the output voltage of the converter, n=A, B, C. The mathematical model of the flexible DC transmission system is usually based on the dynamic behavior of its voltage source converter. Assume that the output voltage and current of the converter are U Ao 、U Bo 、U Co and I sA , I sB , I sB The voltage of the grid is e A 、e B 、e C , the line impedance is Z = R + jX, and Kirchhoff's voltage law (KVL) is written for the main circuit as shown in equation (16).

[0155] Step S12: Performing dq transformation and Laplace transformation on the voltage loop equation to obtain a voltage-current dual closed-loop control model; the expression of the voltage-current dual closed-loop control model is:

[0156]

[0157] Where s is the Laplace operator, i d is the d-axis component of the three-phase current in the rotating coordinate system, i q is the q-axis component of the three-phase current in the rotating coordinate system, u d is the d-axis component of the three-phase voltage in the rotating coordinate system, u q is the q-axis component of the three-phase voltage in the rotating coordinate system, e d is the d-axis component of the grid voltage in the rotating coordinate system, e q is the q-axis component of the grid voltage in the rotating coordinate system, L is the line inductance, and ω is the mechanical angular velocity of the virtual synchronous generator. From formula (17), we can get Figure 4 The voltage and current dual closed-loop control block diagram shown in the figure is as follows: dref and i qref for i d 、i q Reference value, k p and k i are the proportional coefficient and integral coefficient of the PI controller respectively.

[0158] Virtual synchronous control is a power system control strategy designed to enable renewable energy generation systems to achieve stability and control performance similar to that of traditional synchronous generators. By adjusting control parameters, virtual synchronous control enables renewable energy generation systems to rapidly respond to changes in grid frequency and voltage, thereby improving grid stability and reliability. The virtual synchronous control model consists of the mechanical equations of the virtual synchronous generator and the virtual exciter control loop.

[0159] Step S1 further includes:

[0160] Step S13: Establishing the mechanical equation of the virtual synchronous generator and the virtual exciter control equation; wherein the mechanical equation of the virtual synchronous generator is:

[0161]

[0162] Where J is the moment of inertia of the virtual synchronous generator. When the pole pair number is 1, the mechanical angular velocity ω of the virtual synchronous generator is its electrical angular velocity, ω0 is the synchronous angular velocity of the power grid, and P is m is the mechanical power of the virtual synchronous generator, P e is the electromagnetic power of the virtual synchronous generator, T m is the mechanical torque, T eis the electromagnetic torque, T d is the damping torque, D is the damping coefficient, and δ is the rotor angle. From formula (18), we can get Figure 5 The pf control loop in the virtual synchronous generator is shown.

[0163] Based on the idea of ​​reactive power-voltage droop control, the design scheme of the virtual exciter control loop can be obtained. The expression of the virtual exciter control equation is:

[0164]

[0165] Where, E is the virtual electromotive force, E0 is the set value of the no-load electromotive force, and U0 is the actual calculated voltage value. U0 is obtained by calculating the grid voltage u0 through the RMS module. The RMS module represents the effective value of the voltage. n is the rated voltage value, Q ref is the reference value of reactive power, Q e is the actual reactive power; is the reactive power integral coefficient. From formula (19), we can get Figure 5 The QE control loop in the virtual synchronous generator is shown.

[0166] In order to better understand the control method of the grid-type flexible DC transmission system based on deep reinforcement learning in the above embodiment, Figure 6 The overall control block diagram of the grid-type flexible DC transmission system based on deep reinforcement learning in this application is shown. Among them, the grid-type flexible DC transmission system 101 serves as the core of power transmission and provides key electrical parameters to the virtual synchronous motor 102, including the d-axis current (i d ), q-axis current (i q ), d-axis load current (i Ld ), q-axis load current (i Lq ), d-axis voltage (u d ) and q-axis voltage (u q During the interaction, the virtual synchronous motor 102 first uses its built-in power calculation module to calculate the power of the virtual synchronous motor based on the received u d 、u q 、i d and i q Parameters, accurately calculate the active power (P e ) and reactive power (Q e ). Subsequently, the virtual synchronous motor 102 generates and sends status information x according to the operating status of the system. t and reward signal r t To agent 103. Here, x t and r tThey are the environment state and immediate reward in reinforcement learning, providing a basis for learning and optimization for the agent 103. As the core of control decision-making, the agent 103 receives x t and r t Afterwards, a reinforcement learning algorithm is used for policy iteration and optimization. Based on this optimization process, agent 103 outputs a key control instruction, namely an additional power input value, which is directly applied to the Pf (active power-frequency) control loop in the virtual synchronous control model. This additional power input value is dynamically adjusted according to system requirements to optimize the system's power distribution and frequency stability. Ultimately, the virtual synchronous motor 102 precisely controls the grid-connected flexible direct current transmission system 101 based on the additional power input value output by agent 103. This closed-loop control mechanism effectively improves the system's response speed and stability, ensuring the reliability and efficiency of power transmission.

[0167] Furthermore, to verify the effectiveness of the deep reinforcement learning-based control method for a grid-connected flexible DC transmission system, this embodiment uses traditional PI control, the DDPG algorithm, and the improved TD3 algorithm of this application to perform additional damping control on the VSG. From t = 0 to 0.5 seconds, the receiving-end grid's short-circuit ratio (SCR) is 1.4, indicating a weak grid. At t = 0.5 seconds, the receiving-end grid's SCR increases to 4.1, indicating a stronger grid. Figures 7-10 The control performance of different control methods when the receiving end power grid strength changes is compared. Figure 7 The comparison of frequency control performance of three control methods under different receiving-end grid strength is shown. Figure 8 The comparison of the control performance of three control methods on active power under different receiving-end grid strength is shown. Figure 9 The comparison of the control performance of three control methods on reactive power under different receiving-end grid strength is shown. Figure 10 The comparison of the control performance of the three control methods on the grid connection point voltage under different receiving end grid strengths is shown.

[0168] from Figures 7-10 It can be directly observed that when the short-circuit ratio increases to 4.1, that is, when t=0.5s, the active power and reactive power under the TD3 additional damping control change most smoothly, followed by the DDPG algorithm. The system frequency and PCC point phase voltage (i.e., grid connection point voltage) under the TD3 additional damping control are least impacted, and the system recovers stability within 0.15s. The DDPG algorithm is second, and the system recovers stability within 0.25s. Under traditional PI control, the active power and reactive power amplitudes are large, the system frequency and PCC point phase voltage oscillate greatly, and it takes 1s for the system to recover stability. Therefore, when the receiving-end power grid strength changes, the additional damping control based on the improved TD3 algorithm proposed in this application can better improve the stability of the power grid.

[0169] Based on the same inventive concept, embodiments of the present application also provide a deep reinforcement learning-based meshed flexible DC transmission system control device for implementing the deep reinforcement learning-based meshed flexible DC transmission system control method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more embodiments of the deep reinforcement learning-based meshed flexible DC transmission system control device provided below can be found in the above-mentioned limitations of the deep reinforcement learning-based meshed flexible DC transmission system control method, and will not be repeated here.

[0170] In one embodiment, Figure 11 As shown, a grid-type flexible direct current transmission system control device 900 based on deep reinforcement learning is provided, comprising: a system control model establishment module 901, a Markov decision process establishment module 902 and an intelligent agent training module 903, wherein:

[0171] The system control model establishment module 901 is used to establish a grid-type flexible DC transmission system control model; the grid-type flexible DC transmission system control model includes a voltage and current dual closed-loop control model and a virtual synchronous control model;

[0172] A Markov decision process establishing module 902 is used to establish a Markov decision process based on a grid-type flexible HVDC system control model;

[0173] The intelligent agent training module 903 is used to train the intelligent agent using a dual-pool-dual-delay deep deterministic policy gradient algorithm; the intelligent agent provides an additional power input value to the Pf control loop in the virtual synchronous control to improve the stability of the grid-type flexible direct current transmission system; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the intelligent agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

[0174] In one embodiment, the agent training module 903 is further configured to obtain the experience tuple (x t , a t , r t , r t+1 ) and discount factor γ, calculate the target Q value according to the formula shown in formula (7); based on the target Q value, with the goal of minimizing the mean square error loss function of the value network, perform gradient descent update on the parameters of the value network; update the parameters of the target value network based on the parameters of the value network; with the goal of maximizing the expected cumulative reward function, perform gradient descent update on the parameters of the policy network; update the parameters of the target policy network based on the parameters of the policy network.

[0175] In one embodiment, the system control model establishment module 901 is further used to obtain the voltage loop equation of the main circuit of the grid-type flexible direct current transmission system; obtain a voltage-current dual closed-loop control model after performing dq transformation and Laplace transformation on the voltage loop equation; and establish the mechanical equation of the virtual synchronous generator and the virtual exciter control equation.

[0176] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0177] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0178] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0179] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A control method for a grid-type flexible direct current transmission system based on deep reinforcement learning, characterized in that: The method comprises: Step S1, establishing a grid-type flexible direct current transmission system control model; the grid-type flexible direct current transmission system control model includes a voltage and current dual closed-loop control model and a virtual synchronous control model; The step S1 comprises: Step S11, obtaining a voltage loop equation of the main circuit of the grid-type flexible DC transmission system; the voltage loop equation is: In the formula, R represents the resistance of the circuit, i sn represents the output current of the converter, L represents the reactance of the line, e n Indicates the voltage of the power grid, U no Indicates the output voltage of the converter, n = A, B, C; Step S12, performing dq transformation and Laplace transformation on the voltage loop equation to obtain the voltage-current dual closed-loop control model; the expression of the voltage-current dual closed-loop control model is: Where s is the Laplace operator, i d is the d-axis component of the three-phase current in the rotating coordinate system, i q is the q-axis component of the three-phase current in the rotating coordinate system, u d is the d-axis component of the three-phase voltage in the rotating coordinate system, u q is the q-axis component of the three-phase voltage in the rotating coordinate system, e d is the d-axis component of the grid voltage in the rotating coordinate system, e q is the q-axis component of the grid voltage in the rotating coordinate system, L is the line inductance, and ω is the mechanical angular velocity of the virtual synchronous generator; The virtual synchronous control model includes the mechanical equations of the virtual synchronous generator and the virtual exciter control equations; the step S1 further includes: Step S13: Establishing the mechanical equation of the virtual synchronous generator and the virtual exciter control equation; wherein the mechanical equation of the virtual synchronous generator is: Where J is the moment of inertia of the virtual synchronous generator, ω0 is the synchronous angular velocity of the power grid, and P m is the mechanical power of the virtual synchronous generator, P e is the electromagnetic power of the virtual synchronous generator, T m is the mechanical torque, T e is the electromagnetic torque, T d is the damping torque, D is the damping coefficient, and δ is the rotor angle; The mathematical expression of the virtual exciter control equation is: Where, E is the virtual electromotive force, E0 is the set value of the no-load electromotive force, U is the actual calculated voltage value, and U n is the rated voltage value, Q ref is the reference value of reactive power, Q e is the actual reactive power; is the reactive power integral coefficient; Step S2, establishing a Markov decision process based on the grid-type flexible HVDC system control model; Step S3: Using a dual-pool-dual-delay deep deterministic policy gradient algorithm to train an intelligent agent, the intelligent agent improves the stability of the grid-type flexible direct current transmission system by providing an additional power input value to the Pf control loop in the virtual synchronous control model; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the intelligent agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training; The step S3 comprises: Step S31, obtain the experience tuple (x t , a t , r t , x t+1 ) and the discount factor γ, according to the formula Calculate the target Q value; where y t is the target Q value at time t, represents the action value function, x t+1 represents the state at time t+1, represents the action at time t+1 generated by the target policy network, i=1,2; Step S32: Based on the target Q value, the parameters of the value network are updated by gradient descent with the goal of minimizing the mean square error loss function of the value network, i.e. Where, α Q is the learning rate of the value network, is the gradient operator, K(w i ) is the value network mean square error loss function; wherein, the expression of the value network mean square error loss function is: Where D is the experience replay buffer; w i Represents the value network Q i Parameters, i = 1, 2; Indicates that the parameter is w i Value Network The calculated Q value; Step S33: update the parameters of the target value network based on the parameters of the value network, i.e. w′ i ←τw i +(1-τ)w′ i Where w′ i is the parameter of the target value network, i=1, 2; τ is the soft update coefficient, the value range is (0, 1); Step S34, with the goal of maximizing the expected cumulative reward function, the parameters θ of the policy network are updated by gradient descent, that is, Where, α π is the learning rate of the policy network, H(θ) is the expected cumulative reward function; wherein, the expression of the expected cumulative reward function is: Where Q θ (x t ,π θ (x t )) means in state x t Next, perform action π θ (x t ) after the expected total return, π θ (x t ) represents the policy network with parameter θ in state x t The generated action; Step S35: update the parameters of the target policy network based on the parameters of the policy network, i.e. θ'←τθ+(1-τ)θ' Where θ′ is the parameter of the target policy network.

2. The control method of a grid-type flexible direct current transmission system based on deep reinforcement learning according to claim 1, characterized in that: The Markov decision process includes a state value function, an action value function, an agent environment state, and an agent output action; wherein, The state value function is: Among them, V π (x) is the state value function, which represents the total return expected from starting from state x and executing strategy π, r t represents the return at time t, x t represents the state at time t, E represents the expected function, γ is the discount factor, the value range is [0,1], k is the time step, r t+k+1 represents the return at time t+k+1; The action value function is: Where Q π (x,a) is the action value function, which indicates the total return expected after the agent performs action a in the initial state x and continues to perform according to the strategy π. t represents the action performed by the agent at time t; The agent environment state x t The expression is: x t ={Δf,ΔP} Where Δf represents the difference between the real-time frequency and the reference frequency, and ΔP represents the difference between the active power and the reference active power. The agent outputs action a t Defined as: a t ={P Δf } Where, P Δf Indicates the additional power input value of the Pf control loop in virtual synchronous control.

3. The control method of a grid-type flexible direct current transmission system based on deep reinforcement learning according to claim 2, characterized in that: The return r at time t t The calculation formula is: r t =-(λ1|f ref -f t |+λ2|P ref -P t |) Where, f ref is the reference frequency, f t is the actual output frequency of the virtual synchronous generator at time t, P ref is the reference active power, P t is the actual output power of the virtual synchronous generator at time t, λ1 is the frequency weight coefficient, and λ2 is the active power weight coefficient.

4. The control method of a grid-type flexible direct current transmission system based on deep reinforcement learning according to claim 1, characterized in that: The action at time t+1 generated by the target strategy network is the action after adding Gaussian noise, that is, In the formula, π(x t+1 ) is the next state x t+1 When , N(0,σ) represents Gaussian noise with mean 0 and variance σ, and clip is a function that limits the Gaussian noise to the range of [-C,C].

5. The control method of a grid-type flexible direct current transmission system based on deep reinforcement learning according to claim 1, characterized in that: The steps of dividing the success experience pool and the failure experience pool include: Step S30: dividing the experience pool into the success experience pool and the failure experience pool based on a judgment function; the judgment function is expressed as: Where, t v is the total training time in seconds.

6. A device using the method according to any one of claims 1 to 5, characterized in that: The device comprises: A system control model establishment module is used to establish a grid-type flexible DC transmission system control model; the grid-type flexible DC transmission system control model includes a voltage and current dual closed-loop control model and a virtual synchronous control model; A Markov decision process establishment module, used for establishing a Markov decision process based on the grid-type flexible direct current transmission system control model; An agent training module is used to train an agent using a dual-pool-dual-delay deep deterministic policy gradient algorithm; the agent improves the stability of the grid-type flexible direct current transmission system by providing an additional power input value to the Pf control loop in the virtual synchronous control; wherein the dual-pool-dual-delay deep deterministic policy gradient algorithm includes a successful experience pool and a failed experience pool, and the agent randomly samples experience tuples from the successful experience pool and the failed experience pool according to a predetermined ratio for training.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images