An Adaptive Virtual Synchronous Machine Control Method Based on Deep Reinforcement Learning

Abstract

An adaptive virtual synchronous machine control method based on deep reinforcement learning belongs to the field of power grid operation control. It establishes a photovoltaic storage grid-connected model containing photovoltaic systems, battery energy storage and variable loads; uses virtual synchronous generator control to achieve complementary photovoltaic output power for the battery energy storage system; in order to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator; the double deep Q learning algorithm in deep reinforcement learning is used to control the virtual inertia and damping coefficient; a double-layer evaluation network is constructed, combined with a reward function, and after multiple trainings, the adaptive adjustment of the virtual inertia and damping coefficient is achieved. It uses deep reinforcement learning technology to adaptively control the virtual inertia and damping coefficient, suppress the frequency deviation, accelerate the recovery of the system operating frequency, and improve the stability of the system.

Description

Technical Field

[0001] The present invention belongs to the field of power grid operation control, and in particular relates to an adaptive virtual synchronous machine control method based on deep reinforcement learning. Background Art

[0002] With the rapid development of distributed power sources and their increasing proportion in energy applications, microgrids have been proposed as a way to connect distributed power sources to the grid. How to maintain the stability of microgrid frequency when the output of distributed power sources fluctuates and the load changes has become a research focus in the power field.

[0003] At the same time, artificial intelligence technology is also developing rapidly, and applying artificial intelligence technology to microgrid control is a good choice.

[0004] When the photovoltaic storage microgrid is connected to the grid, the energy storage system uses virtual synchronous generator control technology to complement the photovoltaic output power, while the virtual inertia and damping coefficient in traditional virtual synchronous generator technology remain unchanged.

[0005] The invention patent application with the application publication date of March 9, 2021 and the application publication number of CN 112467784 A discloses "a hybrid microgrid converter adaptive virtual synchronous machine control method", which adopts adaptive virtual moment of inertia J and adaptive virtual damping D in the VSG control system to suppress the power fluctuation of the hybrid microgrid AC / DC converter. In order to obtain the rated virtual parameter value range, a small signal model is established for the adaptive VSG control system, and the damping ratio and rated damping coefficient are obtained by analyzing its characteristic value and referring to the synchronous generator oscillation angular frequency value. The present invention adopts adaptive virtual moment of inertia J and adaptive virtual damping D in the VSG control system, and proposes a dual virtual parameter adaptive control scheme that increases J and appropriately reduces D in the angular frequency increase stage; and reduces J and appropriately increases D in the angular frequency reduction stage. This technical solution can ensure the system response speed and accelerate the power to enter a stable state. The dual adaptive virtual parameters can effectively suppress power fluctuations. However, it adopts dual adaptive virtual parameters, which invisibly increases the amount of calculation, slows down the response speed of the system, and is not conducive to improving the response speed of the system.

[0006] The invention patent application with the application publication date of March 22, 2022 and the application publication number of CN 114221394 A discloses "an adaptive virtual synchronous machine control method based on an energy storage system". It first uses a virtual synchronous machine control method to simulate the inertia and damping characteristics of the synchronous generator for the energy storage system to suppress the system frequency fluctuations; further, by adding a bandpass filter, a system enhancement model for model predictive control is established; finally, according to the system frequency deviation, the inertia coefficient and damping coefficient of the virtual synchronous machine are dynamically adjusted by model predictive control to achieve adaptive control. Under the premise of ensuring system reliability, this technical solution optimizes the virtual synchronous machine control parameters in real time according to the system operating status, gives full play to the frequency support capability of energy storage, and improves the system frequency response. This technical solution focuses on the frequency response problem of the energy storage system, does not involve how to deal with the microgrid when it is connected to the grid, and there is no clear discussion on the specific control mode of the virtual inertia and damping coefficient.

[0007] In the actual operation and control process of the power supply network of the power system, how to use deep reinforcement learning technology to control the virtual inertia and damping coefficient so that the virtual inertia and damping coefficient can change adaptively when the power in the microgrid changes to suppress the frequency deviation is a practical problem in the actual power grid operation monitoring. Summary of the invention

[0008] The technical problem to be solved by the present invention is to provide an adaptive virtual synchronous machine control method based on deep reinforcement learning. It applies artificial intelligence technology to microgrid control. When the photovoltaic storage microgrid is connected to the grid, the energy storage system adopts virtual synchronous generator control technology to achieve complementary photovoltaic output power; the virtual inertia and damping coefficient are controlled by deep reinforcement learning technology. When the power in the microgrid changes, the virtual inertia and damping coefficient can be adaptively changed to suppress the frequency deviation.

[0009] The technical solution of the present invention is to provide an adaptive virtual synchronous machine control method based on deep reinforcement learning, which is characterized by comprising the following steps:

[0010] Step 1: In order to reduce the impact of photovoltaic output fluctuations and load mutations on the photovoltaic storage grid-connected system, a photovoltaic storage grid-connected model containing photovoltaic systems, battery energy storage and variable loads is established;

[0011] Step 2: To ensure that the photovoltaic energy storage system provides stable power output to the external power grid when it is connected to the grid, a virtual synchronous generator control is used to complement the photovoltaic output power of the battery energy storage system. In order to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator.

[0012] Step 3: To solve the problem of adaptively changing the virtual inertia and damping coefficient when the power changes, the dual deep Q learning algorithm in deep reinforcement learning is used to control the virtual inertia and damping coefficient. A double-layer evaluation network is constructed, combined with a reward function, and adaptive adjustment of the virtual inertia and damping coefficient is achieved after multiple trainings.

[0013] The adaptive virtual synchronous machine control method applies artificial intelligence technology to microgrid control. When the photovoltaic storage microgrid is connected to the grid, the energy storage system adopts virtual synchronous generator control technology to achieve complementarity with photovoltaic output power. Deep reinforcement learning technology is used to control the virtual inertia and damping coefficient. When the power in the microgrid changes, the virtual inertia and damping coefficient can be adaptively changed to suppress the frequency deviation, accelerate the recovery of the system operating frequency, and improve the stability of the system.

[0014] Specifically, in step 1, a photovoltaic-storage grid-connected model containing a photovoltaic system, a battery energy storage system, a variable load and an external power grid is first established. Photovoltaic power generation and energy storage power generation are respectively connected to the power grid using different inverters; the photovoltaic system adopts maximum power tracking control and is connected to the power grid in the form of an equivalent current source; the grid-connected inverter of the battery energy storage system adopts virtual synchronous generator control to complement the output power of the photovoltaic system; the grid-connected model is called a microgrid, which is connected to the external power grid through a connecting switch; when the switch is closed, the microgrid is in a grid-connected operation state, and at this time the microgrid outputs power to the power grid at a fixed output to provide energy to the power grid.

[0015] Specifically, in step 2, in order to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator. According to the model built in step 1, the control target is frequency stability.

[0016] Furthermore, in step 2, according to the model built in step 1, the control target is frequency stability; the DDQN algorithm is selected as the control algorithm; the state variables provided by the selected external environment are the measured frequency f, the error e between the measured frequency and the reference frequency, the integral value ∫e of the frequency error and the external active power P out ;

[0017] s=(f,e,∫e,P out ) (1)

[0018] The control objects are selected as virtual inertia J and damping coefficient D, where:

[0019] J∈[J min ,J max ](2)

[0020] D∈[D min ,Dmax ](3)

[0021] Where: J min and J max is the minimum and maximum value of the virtual inertia, D min and D max are the minimum and maximum values ​​of the damping coefficient; the DDQN output action is a discrete quantity. If the virtual inertia step is ΔJ, there are p virtual inertia discrete action options; if the damping coefficient step is ΔD, there are q damping coefficient action options, and there are a total of g=pq combinations in the action space.

[0022] Furthermore, in step 2, the DDQN algorithm has two evaluation networks, namely the Q network and the target Q network;

[0023] Among them, the Q network parameter is θ, and the target Q network parameter is θ target ; The role of the evaluation network is to calculate the target value y t , the calculation formula is:

[0024]

[0025] Where: r t+1 is the return value at time t+1, γ is the discount coefficient, s t+1 is the state variable at time t+1, a is the action selected in the action space, Q(·|θ) is the output value of the Q network, Q(·|θ target ) is the output value of the target Q network;

[0026] The Q network parameters θ are updated according to the gradient back propagation of formula (6):

[0027]

[0028] Where: m is the number of samples, y is j 、s j 、a j They are the target value, state variable, and action variable corresponding to sample j.

[0029] Specifically, step 3 is to construct a dual evaluation network and action space, combined with a reward function, and realize adaptive adjustment of virtual inertia and damping coefficient after multiple trainings.

[0030] Furthermore, the two Q networks in step 3 have the same results, so a fully connected network with L layers is selected, where the last layer contains h nodes and each of the other connected layers contains n nodes;

[0031] When training a reinforcement learning agent, a reward function is designed based on the frequency error e;

[0032]

[0033] Where: |e| represents the absolute value of the frequency error, b1 and b2 are the frequency error segmentation points, λ1, λ2 and λ3 are weight coefficients. The smaller the frequency error, the larger the weight coefficient.

[0034] To improve training efficiency, if the agent enters the termination state during training, the current round of training will end and a new state sequence will be restarted to continue training. The termination function designed according to the measured frequency is:

[0035]

[0036] Where: c1, c2 are the upper and lower limits of the measured frequency.

[0037] Furthermore, in step 3, according to the selected neural network and the designed reward function and termination function, the Reinforcement Learning Toolbox in matlab2021a is used to train the model built in step 1. The algorithm used is the DDQN algorithm. The specific process is as follows:

[0038] 1. Set the maximum number of iterations to T, and the maximum number of iterations per round to T f , randomly initialize θ,θ target =θ; Clear the set D of experience replays and target Q network parameter update frequency C;

[0039] 2. From t=1 to T rounds of iteration;

[0040] a) Initialize s = (f, e, ∫e, P out ) is the first state of the current state sequence;

[0041] b) The Q network obtains the Q values ​​of all actions based on the state s and selects the appropriate action a using the ξ-greedy rule;

[0042] c) Execute action a, get the new state s′, reward r, and the terminal state information is_end;

[0043] d) Store the five-tuple {s, a, r, s', is_end} into the experience playback set D;

[0044] e) s = s';

[0045] f) Sample m samples {s j , a j , r j ,s j ', is_end j}, j = 1, 2,,, m, the current target Q value y is calculated by formula (5) i ;

[0046] g) According to formula (6), the parameters θ of the Q network are updated using gradient back propagation;

[0047] h) If T / C = 1, update the target Q network parameter θ target =θ;

[0048] j) If the training converges or reaches the training number, it ends, otherwise go to step k);

[0049] k) If s′ is in the terminal state or reaches the maximum training step length, go to step a), otherwise go to step b);

[0050] Furthermore, in order to achieve better convergence of the algorithm, the exploration rate ξ needs to become smaller as the iteration proceeds.

[0051] The adaptive virtual synchronous machine control method described in the present invention controls and adjusts the virtual inertia and damping coefficient of the battery energy storage virtual synchronous generator through deep reinforcement learning technology, thereby realizing the improvement of the virtual synchronous generator; the dual deep Q learning algorithm in the deep reinforcement learning algorithm is selected to construct two networks, namely the current evaluation network and the target evaluation network, and the powerful fitting ability of the neural network is used to reduce the dimensionality curse, thereby realizing the adaptive adjustment of the virtual synchronous generator technology.

[0052] The adaptive virtual synchronous machine control method described in the present invention enables the deep reinforcement learning controller to be fully trained by designing the state space, action space and reward function.

[0053] Compared with the prior art, the advantages of the present invention are:

[0054] 1. The technical solution of the present invention controls and adjusts the virtual inertia and damping coefficient of the battery energy storage virtual synchronous generator through deep reinforcement learning technology, thereby realizing the improvement of the virtual synchronous generator;

[0055] 2. The technical solution of the present invention selects the dual deep Q learning algorithm in the deep reinforcement learning algorithm, constructs two networks, the current evaluation network and the target evaluation network, and uses the powerful fitting ability of the neural network to reduce the dimensionality curse, thereby realizing the adaptive adjustment of the virtual synchronous generator technology;

[0056] 3. The technical solution of the present invention enables the deep reinforcement learning controller to be fully trained by designing the state space, action space and reward function;

[0057] 4. During grid-connected operation, the adaptive virtual synchronous generator adjusts the virtual inertia and damping coefficient according to the change in active power, thereby suppressing the deviation of the system frequency, accelerating the recovery of the system operating frequency, and improving the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of the method of the present invention;

[0059] Figure 2 It is a schematic diagram of the microgrid topology including photovoltaic, energy storage and load;

[0060] Figure 3 Schematic diagram of the motion equation structure of the virtual synchronous generator rotor;

[0061] Figure 4 Schematic diagram of the learning process of the DDQN algorithm;

[0062] Figure 5 This is a comparison chart of frequency control effects. DETAILED DESCRIPTION

[0063] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0064] Combine the following Figures 1 to 5 As shown, the embodiments of the present invention are described in detail, specifically:

[0065] Step 1: In order to reduce the impact of photovoltaic output fluctuations and load mutations on the photovoltaic-storage grid-connected system, a photovoltaic-storage grid-connected model containing photovoltaic systems, battery energy storage and variable loads was established;

[0066] Step 2: To ensure that the photovoltaic energy storage system provides stable power output to the external power grid when it is connected to the grid, a virtual synchronous generator control is used to complement the photovoltaic output power of the battery energy storage system. In order to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator.

[0067] Step 3: To solve the problem of adaptively changing the virtual inertia and damping coefficient when the power changes, the double deep Q-learning algorithm (Double Deep-Q-Network, DDQN) in deep reinforcement learning is used to control the virtual inertia and damping coefficient. A dual evaluation network is constructed and combined with the reward function. After multiple trainings, the adaptive adjustment of the virtual inertia and damping coefficient is achieved.

[0068] In step 1, a grid-connected model including the photovoltaic system, battery energy storage system, variable load and external power grid is first established, as follows:

[0069] In step 1, a grid-connected model containing a photovoltaic system, a battery energy storage system, a variable load and an external power grid is first established. Photovoltaic power generation and energy storage power generation are connected to the grid using different inverters. The photovoltaic system adopts maximum power tracking control, and its grid-connected inverter adopts PQ control, which does not support the grid. It is connected to the grid in the form of an equivalent current source and can be replaced by a controlled current source. The grid-connected inverter of the energy storage system adopts virtual synchronous generator control, which complements the photovoltaic output power and plays a certain supporting role for the grid. The microgrid is connected to the external power grid through a connecting switch. When the switch is closed, the microgrid is in a grid-connected operation state. At this time, the photovoltaic storage microgrid outputs power to the grid at a fixed output and provides green energy to the grid.

[0070] In step 2, to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator.

[0071] According to the model built in step 1, the control target is frequency stability.

[0072] Considering that the system frequency is a continuous quantity, the DDQN algorithm is selected as the control algorithm.

[0073] Reinforcement learning is based on the Markov process. The state variables provided by the selected external environment are the measured frequency f, the error e between the measured frequency and the reference frequency, the integral value ∫e of the frequency error and the external active power P out .

[0074] s=(f,e,∫e,P out ) (1)

[0075] The control objects are selected as virtual inertia J and damping coefficient D, where:

[0076] J∈[J min ,J max ](2)

[0077] D∈[D min ,D max ](3)

[0078] Where: J min and J max is the minimum and maximum value of the virtual inertia, D min and D max are the minimum and maximum values ​​of the damping coefficient.

[0079] Since the output action of DDQN is a discrete quantity, assuming the virtual inertia step size is ΔJ, there are p virtual inertia discrete action options; assuming the damping coefficient step size is ΔD, there are q damping coefficient action options, and there are a total of g=pq combinations in the action space.

[0080] Specifically, in step 2, the DDQN algorithm has two evaluation networks, namely the Q network and the target Q network. The Q network parameter is θ and the target Q network parameter is θ. target The role of the evaluation network is to calculate the target value y t , the calculation formula is:

[0081]

[0082] Where: r t+1 is the return value at time t+1, γ is the discount coefficient, s t+1 is the state variable at time t+1, a is the action selected in the action space, Q(·|θ) is the output value of the Q network, Q(·|θ target ) is the output value of the target Q network.

[0083] The Q network parameters θ are updated according to the gradient back propagation of equation (6).

[0084]

[0085] Where: m is the number of samples, y is j 、s j 、a j They are the target value, state variable, and action variable corresponding to sample j.

[0086] Step 3 is to construct a dual evaluation network, combined with a reward function, and achieve adaptive adjustment of the virtual inertia and damping coefficient after multiple trainings.

[0087] In step 3, the two Q networks have the same results. Select an L-layer fully connected layer network, where the last layer contains h nodes and each of the other connected layers contains n nodes.

[0088] The reward function is the key to training reinforcement learning agents, and the reward function is designed based on the frequency error e.

[0089]

[0090] Where: |e| represents the absolute value of the frequency error. b1 and b2 are the frequency error segmentation points, respectively. λ1, λ2, and λ3 are weight coefficients. The smaller the frequency error, the larger the weight coefficient.

[0091] To improve the training efficiency, if the agent enters the termination state during training, the current round of training will end and a new state sequence will be restarted to continue training. The termination function designed according to the measured frequency is:

[0092]

[0093] Where: c1, c2 are the upper and lower limits of the measured frequency.

[0094] In step 3, according to the selected neural network and the designed reward function and termination function, the Reinforcement Learning Toolbox in matlab2021a is used to train the model built in step 1. The algorithm used is the DDQN algorithm. The specific process is as follows:

[0095] 1. Set the maximum number of iterations to T, and the maximum number of iterations per round to T f , randomly initialize θ,θ target =θ; Clear the set D of experience replays and target Q network parameter update frequency C;

[0096] 2. From t=1 to T rounds of iteration;

[0097] a) Initialize s = (f, e, ∫e, P out ) is the first state of the current state sequence;

[0098] b) The Q network obtains the Q values ​​of all actions based on the state s and selects the appropriate action a using the ξ-greedy rule;

[0099] c) Execute action a, get the new state s′, reward r, and the terminal state information is_end;

[0100] d) Store the five-tuple {s, a, r, s', is_end} into the experience playback set D;

[0101] e) s = s';

[0102] f) Sample m samples {s j , a j , r j ,s j ', is_end j}, j = 1, 2,,, m, the current target Q value y is calculated by formula (5) i ;

[0103] g) According to formula (6), the parameters θ of the Q network are updated using gradient back propagation;

[0104] h) If T / C = 1, update the target Q network parameter θ target =θ;

[0105] j) If the training converges or reaches the training number, it ends, otherwise go to step k);

[0106] k) If s′ is in the terminal state or reaches the maximum training step length, go to step a), otherwise go to step b);

[0107] Furthermore, in order to achieve better convergence of the algorithm, the exploration rate ξ needs to become smaller as the iteration proceeds.

[0108] Simulation verification:

[0109] Simulation is performed in the matlab2021a / Simulink environment to compare technical effects.

[0110] The trained adaptive virtual synchronous generator control strategy is applied to the energy storage system in the photovoltaic storage microgrid. The load in the microgrid increases by P1 in 2 seconds, and the output power of photovoltaic power generation remains unchanged during this period. The frequency fluctuation of the period is compared with the frequency fluctuation of the photovoltaic storage microgrid using the traditional virtual synchronous generator control strategy. The comparison results are shown in the attached manual. Figure 5 As shown in the figure, it can be seen that after adopting the technical solution of the present invention, the system frequency offset becomes smaller and the frequency recovery speed becomes faster in the same time period, which can better improve the stability of the system and reflect the superiority of the adaptive virtual synchronous generator control strategy described in the technical solution of the present invention.

[0111] The technical solution of the present invention adopts virtual synchronous generator control to achieve complementarity with photovoltaic output power in the battery energy storage system, adopts deep reinforcement learning algorithm to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator, adopts double deep Q learning algorithm (Double Deep-Q-Network, DDQN) in deep reinforcement learning to control the virtual inertia and damping coefficient, constructs a dual evaluation network and action space, combines reward function, and realizes adaptive adjustment of virtual inertia and damping coefficient after multiple training. When the grid-connected power of photovoltaic storage changes, the virtual inertia and damping coefficient of the energy storage virtual synchronous generator can be adjusted to adapt to changes in the external environment, reduce frequency offset, and improve the stability of the system.

[0112] The present invention can be widely used in the fields of operation management of electric power supply network systems and operation control of microgrids.

Claims

1. An adaptive virtual synchronous machine control method based on deep reinforcement learning, characterized in that The following steps are involved: Step 1: In order to reduce the impact of photovoltaic output fluctuations and load mutations on the photovoltaic storage grid-connected system, a photovoltaic storage grid-connected model containing photovoltaic systems, battery energy storage and variable loads is established; Step 2: To ensure that the photovoltaic energy storage system provides stable power output to the external power grid when it is connected to the grid, a virtual synchronous generator control is used to complement the photovoltaic output power of the battery energy storage system. In order to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator. Step 3: To solve the problem of adaptively changing the virtual inertia and damping coefficient when the power changes, the dual deep Q learning algorithm in deep reinforcement learning is used to control the virtual inertia and damping coefficient. A double-layer evaluation network is constructed, combined with a reward function, and adaptive adjustment of the virtual inertia and damping coefficient is achieved after multiple trainings. The step 3 is to construct a dual evaluation network and action space, combine the reward function, and realize adaptive adjustment of the virtual inertia and damping coefficient after multiple trainings; In step 3, the two Q networks have the same results. Select an L-layer fully connected network, where the last layer contains h nodes and each of the other connected layers contains n nodes. When training the reinforcement learning agent, design the reward function based on the frequency error e. To improve the training efficiency, if the agent enters the termination state during training, the current round of training will be terminated and a new state sequence will be restarted to continue training. In step 3, according to the selected neural network and the designed reward function and termination function, the model built in step 1 is trained using the Reinforcement Learning Toolbox in matlab2021a, and the algorithm used is the DDQN algorithm; The adaptive virtual synchronous machine control method applies artificial intelligence technology to microgrid control. When the photovoltaic storage microgrid is connected to the grid, the energy storage system adopts virtual synchronous generator control technology to achieve complementarity with photovoltaic output power. Deep reinforcement learning technology is used to control the virtual inertia and damping coefficient. When the power in the microgrid changes, the virtual inertia and damping coefficient can be adaptively changed to suppress the frequency deviation, accelerate the recovery of the system operating frequency, and improve the stability of the system.

2. The adaptive virtual synchronous machine control method based on deep reinforcement learning according to claim 1 is characterized in that in step 1, a photovoltaic storage grid-connected model containing a photovoltaic system, a battery energy storage system, a variable load and an external power grid is first established, and photovoltaic power generation and energy storage power generation are respectively connected to the power grid using different inverters; The photovoltaic system adopts maximum power tracking control and is connected to the power grid in the form of an equivalent current source; The grid-connected inverter of the battery energy storage system adopts virtual synchronous generator control to complement the output power of the photovoltaic system; The grid-connected model is called a microgrid, which is connected to the external power grid through a connecting switch; When the switch is closed, the microgrid is in a grid-connected operation state. At this time, the microgrid outputs power to the grid at a fixed output and provides energy to the grid.

3. According to the adaptive virtual synchronous machine control method based on deep reinforcement learning as described in claim 1, it is characterized in that in step 2, in order to solve the problem of microgrid frequency deviation caused by microgrid load power changes, a deep reinforcement learning algorithm is used to control and adjust the virtual inertia and damping coefficient of the virtual synchronous generator, and according to the model constructed in step 1, the control target is frequency stability.

4. The adaptive virtual synchronous machine control method based on deep reinforcement learning according to claim 1 or 3 is characterized in that in step 2, according to the model built in step 1, the control target is frequency stability; the DDQN algorithm is selected as the control algorithm; the state variables provided by the selected external environment are the measured frequency f, the error e between the measured frequency and the reference frequency, the integral value ∫e of the frequency error and the external active power P out ; s=(f,e,∫e,P out ) (1) The control objects are selected as virtual inertia J and damping coefficient D, where: J∈[J min ,J max ](2)D∈[D min ,D max ](3) Where: J min and J max is the minimum and maximum value of the virtual inertia, D min and D max are the minimum and maximum values ​​of the damping coefficient; the DDQN output action is a discrete quantity. If the virtual inertia step is ΔJ, there are p virtual inertia discrete action options; if the damping coefficient step is ΔD, there are q damping coefficient action options, and there are a total of g=pq combinations in the action space.

5. The adaptive virtual synchronous machine control method based on deep reinforcement learning according to claim 4 is characterized in that In step 2, the DDQN algorithm has two evaluation networks, namely the Q network and the target Q network; Among them, the Q network parameter is θ, and the target Q network parameter is θ target ; The role of the evaluation network is to calculate the target value y t , the calculation formula is: Where: r t+1 is the return value at time t+1, γ is the discount coefficient, s t+1 is the state variable at time t+1, a is the action selected in the action space, Q(·|θ) is the output value of the Q network, Q(·|θ target ) is the output value of the target Q network; The Q network parameters θ are updated according to the gradient back propagation of formula (6): Where: m is the number of samples, y is j 、s j 、a j They are the target value, state variable, and action variable corresponding to sample j.

6. According to the adaptive virtual synchronous machine control method based on deep reinforcement learning according to claim 1, Its characteristic is that in step 3, when training the reinforcement learning agent, a reward function is designed according to the frequency error e; Where: |e| represents the absolute value of the frequency error, b1 and b2 are the frequency error segmentation points, λ1, λ2 and λ3 are weight coefficients. The smaller the frequency error, the larger the weight coefficient. In order to improve the training efficiency, the termination function designed according to the measured frequency during training is: Where: c1, c2 are the upper and lower limits of the measured frequency.

7. According to the adaptive virtual synchronous machine control method based on deep reinforcement learning according to claim 1, it is characterized in that in step 3, the specific process of the DDQN algorithm is: 7.

1. Let the maximum number of iterations be T, and the maximum number of iterations per round be T f , randomly initialize θ,θ target =θ; Clear the set D of experience replays and target Q network parameter update frequency C; 7.2, from t = 1 to T rounds of iteration; a) Initialize s = (f, e, ∫e, P out ) is the first state of the current state sequence; b) The Q network obtains the Q values ​​of all actions based on the state s and selects the appropriate action a using the ξ-greedy rule; c) Execute action a, get the new state s′, reward r, and the terminal state information is_end; d) Store the five-tuple {s, a, r, s', is_end} into the experience playback set D; e) s = s'; f) Sample m samples {s j , a j , r j ,s j ', is_end j }, j = 1, 2, ..., m, the current target Q value y is calculated by formula (5) i ; g) According to formula (6), the parameters θ of the Q network are updated using gradient back propagation; h) If T / C = 1, update the target Q network parameter θ target =θ; j) If the training converges or reaches the training number, it ends, otherwise go to step k); k) If s′ is in the terminal state or reaches the maximum training step length, go to step a), otherwise go to step b); Furthermore, in order to achieve better convergence of the algorithm, the exploration rate ξ needs to become smaller as the iteration proceeds.

8. The method for controlling an adaptive virtual synchronous machine based on deep reinforcement learning according to claim 1, characterized in that The adaptive virtual synchronous machine control method controls and adjusts the virtual inertia and damping coefficient of the battery energy storage virtual synchronous generator through deep reinforcement learning technology, thereby achieving improvement of the virtual synchronous generator; The adaptive virtual synchronous machine control method selects the dual deep Q learning algorithm in the deep reinforcement learning algorithm, constructs two networks, the current evaluation network and the target evaluation network, and uses the powerful fitting ability of the neural network to reduce the dimensionality curse, thereby realizing the adaptive adjustment of the virtual synchronous generator technology.

9. The adaptive virtual synchronous machine control method based on deep reinforcement learning according to claim 1 is characterized in that The adaptive virtual synchronous machine control method enables the deep reinforcement learning controller to be fully trained by designing the state space, action space and reward function.

Images