A data-driven multi-zone power grid energy storage adaptive virtual inertia control method and related device

Abstract

The application provides a kind of data-driven based multi-area power grid energy storage adaptive virtual inertia control method and related device, comprising: establishing at least two regional power grid models;Adopt the deep Q network improvement algorithm under the competition framework to adjust the inertia constant of the virtual inertia control unit of two regions, establish the action space required for Dueling DQN agent, state space and reward function establish target loss function, the neural network parameters of Dueling DQN agent are iterated, L1 parameter norm penalty term is introduced to improve the target loss function, improve the generalization ability of Dueling DQN agent.The inertia constant in the virtual inertia control unit of different regions can be adaptively adjusted, the reasonable compensation of inertia power in different scenarios and the cooperation between multi-area energy storage are realized, and the stability of multi-area power grid is improved.

Description

Technical Field

[0001] The present invention relates to the field of energy storage virtual inertia control, and in particular to a data-driven multi-regional power grid energy storage adaptive virtual inertia control method and related devices. Background Art

[0002] Renewable energy sources such as photovoltaics and wind power are typically connected to the grid through power electronic inverters. Unlike traditional synchronous generators, inverters lack rotor inertia. As the penetration of new energy sources like photovoltaics and wind power increases, the inertia of the power system decreases, leading to reduced grid stability and weakened interference resistance. Furthermore, the uncertainty and volatility of wind and solar power output negatively impacts grid frequency stability.

[0003] Energy storage, as an energy source, can use virtual inertia control to compensate for excess inertia power in the grid, thereby mitigating or eliminating the impact of large-scale wind and solar energy integration on the grid. However, traditional virtual inertia control uses a fixed inertia constant, which makes it difficult to compensate for the required inertia power and suffers from slow response speed. Fixed inertia constants are also difficult to adapt to the complex and diverse scenarios of the grid. Inappropriate inertia power compensation can even exacerbate grid frequency fluctuations, causing even more severe impacts. Therefore, improving the inertia compensation capability and adaptability of virtual inertia control strategies is a key issue. With the rapid development of artificial intelligence technology, the use of data-driven adjustment of inertia constants is a new approach. Summary of the Invention

[0004] In order to improve the compensation capability of multi-regional power grid energy storage inertia power, the present invention provides a data-driven multi-regional power grid energy storage adaptive virtual inertia control method and related devices.

[0005] The present invention is implemented by the following technical solutions:

[0006] A data-driven multi-region power grid energy storage adaptive virtual inertia control method includes the following steps:

[0007] Step 1: Establish at least two regional power grid models. Each regional power grid includes a wind power system, a thermal power plant, and an energy storage device. The wind power system is connected to the grid using constant power control as a disturbance source. The thermal power plant participates in the primary and secondary frequency regulation of the regional power grid. The load frequency controller uses an integral controller, and the energy storage device uses virtual inertia control to compensate for inertia power.

[0008] Step 2: Use the improved Deep Q Network algorithm under the competitive framework to adjust the inertia constants of the virtual inertial control units in the two regions and establish the action space, state space, and reward function required by the Dueling DQN agent;

[0009] Step 3: Based on the action space, state space, and reward function established in Step 2, establish a target loss function, iterate the neural network parameters of the Dueling DQN agent, and introduce an L1 parameter norm penalty term to improve the target loss function and enhance the generalization ability of the Dueling DQN agent.

[0010] Furthermore, the regional power grid model is expressed as:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017]

[0018] Where i=1,2; is the frequency deviation of region i, is the inertia time constant of region i, is the damping constant of region i, is the power increment output by the wind power system, is the inertia compensation power increment of energy storage output, is the load change of the power grid, is the output power increment of the thermal power plant, is the power increment output by the turbine system, is the time constant of the turbine system, is the time constant of the generator, It is the control signal of thermal power plant; is the adjustment coefficient; is the integration coefficient, is the coefficient of variation, is the tie line power change, is the coupling coefficient is the initial wind power variation, is the time constant of the wind turbine generator set, is the inertia time constant, is the energy storage time constant.

[0019] Furthermore, the Dueling DQN agent obtains frequency information of each region from the external power grid as the state space. The frequency information includes the frequency deviation of the regional power grid and the integral of the frequency deviation. The action space is the inertia constant of the virtual inertial control unit of each region. The reward function is established based on the frequency information obtained by the Dueling DQN agent to realize feedback from the external environment to the agent after the agent outputs an action.

[0020] Furthermore, in step 3, the L1 parameter norm penalty term is introduced to improve the target loss function, specifically including:

[0021] The L1 parameter norm penalty term is used to improve the traditional mean square loss function C0, add the structural risk term C1, and add regularization operation:

[0022]

[0023]

[0024]

[0025]

[0026] Among them, N is the number of samples, γ is the discount coefficient; χ is the penalty coefficient, and ω represents the weight in the neural network parameters.

[0027] Furthermore, the Dueling DQN agent network consists of two small networks, which are equivalent to three parameters, where θ is the parameter of the convolutional neural network shared by the two small networks; α and β are the network parameters of the small network itself; the Q function of the Dueling DQN agent is As shown in the following formula:

[0028]

[0029] To avoid the problem of unrecognizable action value when the state value and advantage function change while the sum of the two, Q function, remains unchanged, the Q function is modified to:

[0030]

[0031] in, is the maximum advantage function value, The action that maximizes the advantage function;

[0032] The Dueling DQN agent gradually updates the neural network parameters by minimizing the loss function, thereby gradually achieving an accurate estimate of the Q value:

[0033]

[0034] .

[0035] Furthermore, the Dueling DQN agent continuously interacts with the external power grid to obtain sample data. This sample data is then used to train the Q network until convergence. When the neural network can accurately estimate the Q value, the action that maximizes the Q value is selected, ultimately achieving data-driven adaptive virtual inertia control of multi-regional power grid energy storage.

[0036] .

[0037] Furthermore, the specific steps of extracting sample data to train the Q network are as follows:

[0038] Initialize the neural network parameters θ, α, β, and set the number of training episodes.

[0039] (1) In each Episode, initialize the first state s t ;

[0040] (2) Select action a t Act on the environment to get the next state s t+1 , environmental feedback reward r t ;

[0041] (3) This quadruple is stored in the experience replay set;

[0042] (4) Sampling samples from the experience replay set and updating the network through the objective function containing the structural risk term;

[0043] (5)s t = s t+1 , determine whether the maximum number of training steps has been reached, if so, go to step (6), otherwise go to step (2);

[0044] (6) Determine whether the maximum number of training episodes has been reached. If so, the training ends; otherwise, go to step (1).

[0045] A data-driven multi-regional power grid energy storage adaptive virtual inertial control system, comprising: a computer-readable storage medium and a processor;

[0046] The computer-readable storage medium is used to store executable instructions;

[0047] The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the data-driven multi-region power grid energy storage adaptive virtual inertia control method.

[0048] A non-transitory computer-readable storage medium stores a computer program, which, when executed by a processor, implements the data-driven multi-regional power grid energy storage adaptive virtual inertia control method.

[0049] Compared with the existing methods, the present invention has the following advantages:

[0050] The method proposed in this paper is data-driven. According to the changes in the power grid scenario, the data-driven Dueling DQN intelligent agent uses the frequency changes of the multi-regional power grid to adaptively adjust the inertia constants in the virtual inertial control units in different regions, achieving reasonable compensation of inertial power in different scenarios and coordinated cooperation between multi-regional energy storage, thereby jointly improving the stability of the multi-regional power grid. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to an embodiment of the present invention.

[0052] Figure 2 This is a multi-region power grid frequency control model according to an embodiment of the present invention.

[0053] Figure 3 This is the Q function composition of the Dueling DQN agent in an embodiment of the present invention.

[0054] Figure 4 This is a flowchart of Dueling DQN agent training according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0056] The embodiment of the present invention provides a data-driven multi-regional power grid energy storage adaptive virtual inertia control method, such as Figure 1 As shown, the method includes:

[0057] Step 1: First, establish at least two regional power grid models. Each regional power grid includes a wind power system, a thermal power plant, and an energy storage device. The wind power system is connected to the power grid in the form of constant power control as a disturbance source to maximize the use of clean energy. The thermal power plant participates in the primary and secondary frequency regulation of the regional power grid. The load frequency controller adopts an integral control controller. The energy storage device participates in the primary frequency regulation of the regional power grid due to its fast charging and discharging response speed. The control method is to adopt virtual inertia control to provide additional inertia power for the power grid and improve the stability of the frequency. This embodiment is illustrated by taking two regions as an example. The regional power grid model is expressed as follows:

[0058]

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065] Where i=1,2; is the frequency deviation of region i, is the inertia time constant of region i, is the damping constant of region i, is the power increment output by the wind power system, is the inertia compensation power increment of energy storage output, is the load change of the power grid, is the output power increment of the thermal power plant, is the power increment output by the turbine system, is the time constant of the turbine system, is the time constant of the generator, It is the control signal of thermal power plant; is the adjustment coefficient; is the integration coefficient, is the coefficient of variation, is the tie line power change, is the coupling coefficient is the initial wind power variation, is the time constant of the wind turbine generator set, is the inertia time constant, is the energy storage time constant.

[0066] The microgrids in the two regions are coupled to each other through interconnection lines. Each regional grid is equipped with a virtual inertial control unit that uses energy storage as energy. The unit's structural model and the multi-regional grid frequency control model are as follows: Figure 2 The frequency deviation and other information of the two regional power grids are input into the Dueling DQN agent through the communication line, and then the Dueling DQN agent outputs the adjustment instructions of the virtual inertia constants of the two regions.

[0067] Step 2: Using an improved Deep Q Network (Dueling DQN) algorithm within a competitive framework, the inertia constants of the virtual inertial control units (ICUs) in the two regional power grids are adjusted, establishing the action space, state space, and reward function required by the Dueling DQN agent. The Dueling DQN agent is positioned as the connecting line between the two regions, promoting coordination between the two energy storage VICs.

[0068] To achieve the coordination of the two-region virtual inertial control units, the Dueling DQN agent needs to obtain frequency information of regions 1 and 2 from the external power grid. Specifically, the frequency deviation and the integral of the frequency deviation of the two regional power grids are obtained. The state space is:

[0069]

[0070] The action space of the agent is the inertia constant of the virtual inertial control unit in the two regions:

[0071]

[0072] in, are the virtual inertia constants for region 1 and region 2, Size range between, Size range Since the action space of Dueling DQN agent is a discrete space, it is necessary to discretize the action variables. The interval is divided into 8 discrete actions, and the changes are , then the action set of Dueling DQN agent is:

[0073]

[0074]

[0075] In different scenarios and time periods, the virtual inertia constants of the virtual inertia control units in the two areas will be different. The inertia constant will be adaptively adjusted according to changes in the external power grid frequency information. The inertia control unit selects the appropriate virtual inertia constant at different stages.

[0076] Based on the frequency information obtained by the Dueling DQN agent, a reward function is established to realize the feedback of the external environment to the agent after the agent outputs an action:

[0077]

[0078]

[0079]

[0080] in, The reward function weights are chosen so that the greater the frequency deviation, the greater the weight, and the smaller the reward received by the Dueling DQN agent. Through the reward function, the Dueling DQN agent can judge the quality of its own actions and seek long-term rewards.

[0081] Step 3: Based on the action space, state space and reward function established in step 2, establish the target loss function and iterate the neural network parameters of the Dueling DQN agent. Dueling DQN is an improved algorithm for deep Q network. Different from the value function approximation method of DQN, Dueling DQN is based on the competition framework and the Q function By state value and advantage function The L1 parameter norm penalty term is introduced to improve the loss function and enhance the adaptability of the Dueling DQN agent.

[0082] The Dueling DQN agent network consists of two small networks, which can be equivalent to three parameters. θ is the parameter of the convolutional neural network shared by the two small networks; α and β are the network parameters of the small network itself. The Q function of the Dueling DQN agent is As follows, Figure 3 shown.

[0083]

[0084] To avoid the problem of unrecognizable action value when the state value and advantage function change while the sum of the two, Q function, remains unchanged, the Q function is modified to:

[0085]

[0086] in, is the maximum advantage function value, The action that maximizes the advantage function.

[0087] The Dueling DQN agent gradually updates the neural network parameters by minimizing the loss function, thereby gradually achieving an accurate estimation of the Q value.

[0088]

[0089]

[0090] The Dueling DQN agent has an experience replay pool and is trained offline. The agent randomly extracts a certain number of samples from the experience replay pool and updates the neural network based on the loss function. Therefore, the Dueling DQN agent has the ability to replay experience. When extracting samples, the loss function is:

[0091]

[0092]

[0093] Where N is the number of samples and γ is the discount coefficient.

[0094] The L1 parameter norm penalty term is used to improve the traditional mean square loss function C0, add the structural risk term C1, and add regularization operation:

[0095]

[0096]

[0097] χ is the penalty coefficient, and ω represents the weight in the neural network parameters.

[0098] By improving the mean squared loss function, the Dueling DQN agent's generalization capability is enhanced. The Dueling DQN agent continuously interacts with the external power grid to obtain sample data, extracting this data to train the Q network until convergence. Once the neural network can accurately determine the Q value, it selects the action that maximizes the Q value, ultimately achieving data-driven adaptive virtual inertia control of energy storage in multiple regions of the power grid.

[0099]

[0100] The training process is as follows: Figure 4 :

[0101] Initialize the neural network parameters θ, α, β, and set the number of training episodes.

[0102] (1) In each Episode, initialize the first state s t ;

[0103] (2) Select action at Act on the environment to get the next state s t+1 , environmental feedback reward r t ;

[0104] (3) This quadruple is stored in the experience replay set;

[0105] (4) Sampling samples from the experience replay set and updating the network through the objective function containing the structural risk term;

[0106] (5)s t = s t+1 , determine whether the maximum number of training steps has been reached, if so, go to step (6), otherwise go to step (2);

[0107] (6) Determine whether the maximum number of training episodes has been reached. If so, the training ends; otherwise, go to step (1).

[0108] After training, the data-driven Dueling DQN agent adjusts the inertia constants of the virtual inertial control units in the two regions according to the frequency changes of the two regional power grids, improving the inertia power compensation capability of the energy storage systems in the two regions, achieving coordinated cooperation between the two regions and jointly improving the stability of the power grids in the two regions.

[0109] Another aspect of the present invention provides a data-driven multi-regional power grid energy storage adaptive virtual inertial control system, comprising: a computer-readable storage medium and a processor;

[0110] The computer-readable storage medium is used to store executable instructions;

[0111] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the data-driven multi-regional power grid energy storage adaptive virtual inertia control method described in the first aspect.

[0112] Another aspect of the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for implementing the data-driven adaptive virtual inertia control method for multi-regional power grid energy storage described in the first aspect is implemented.

[0113] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A data-driven multi-regional power grid energy storage adaptive virtual inertia control method, characterized in that: The steps include: Step 1: Establish at least two regional power grid models. Each regional power grid includes a wind power system, a thermal power plant, and an energy storage device. The wind power system is connected to the grid using constant power control as a disturbance source. The thermal power plant participates in the primary and secondary frequency regulation of the regional power grid. The load frequency controller uses an integral controller, and the energy storage device uses virtual inertia control to compensate for inertia power. Step 2: Use the improved Deep Q Network algorithm under the competitive framework to adjust the inertia constants of the virtual inertial control units in the two regions and establish the action space, state space, and reward function required by the Dueling DQN agent; Step 3: Based on the action space, state space, and reward function established in Step 2, establish a target loss function, iterate the neural network parameters of the Dueling DQN agent, and introduce an L1 parameter norm penalty term to improve the target loss function and enhance the generalization ability of the Dueling DQN agent.

2. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 1, characterized in that: The regional power grid model is expressed as: Where i = 1, 2; Δf i is the frequency deviation of region i, H i is the inertia time constant of region i, D i is the damping constant of region i, ΔP W,i is the power increment of the wind power system output, ΔP V,i is the inertia compensation power increment of the energy storage output, ΔP L,i is the load change of the power grid, ΔP M,i is the output power increment of the thermal power plant, ΔP G,i is the power increment of the turbine system output, T t,i is the time constant of the turbine system, T g,i is the time constant of the generator, ΔP C,i is the control signal of the thermal power plant; R i is the adjustment coefficient; K S,i is the integral coefficient, β i is the coefficient of deviation, ΔP TIE is the tie line power change, T 12 is the coupling coefficient ΔP wind,i is the initial wind power change, T W,i is the time constant of the wind turbine generator set, K V,i is the inertia time constant, T E,i is the energy storage time constant.

3. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 1, characterized in that: The Dueling DQN agent obtains frequency information of each region from the external power grid as the state space. The frequency information includes the frequency deviation of the regional power grid and the integral of the frequency deviation. The action space is the inertia constant of the virtual inertial control unit of each region. The reward function is established based on the frequency information obtained by the Dueling DQN agent to realize the feedback of the external environment to the agent after the agent outputs an action.

4. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 1, characterized in that: In step 3, the L1 parameter norm penalty term is introduced to improve the target loss function, specifically including: The L1 parameter norm penalty term is used to improve the traditional mean square loss function C0, add the structural risk term C1, and add regularization operation: C=C0+C1 Among them, N is the number of samples, γ is the discount coefficient; χ is the penalty coefficient, and ω represents the weight in the neural network parameters.

5. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 1, characterized in that: The DuelingDQN agent network consists of two small networks, which are equivalent to three parameters, where θ is the parameter of the convolutional neural network part shared by the two small networks; α and β are the network parameters of the small network itself; the Q function Q(s, a; θ, α, β) of the Dueling DQN agent is shown as follows: Q(s,a;θ,α,β)=V(s;θ,β)+A(s,a;θ,α) To avoid the problem of unrecognizable action value when the state value and advantage function change while the sum of the two, Q function, remains unchanged, the Q function is modified to: Among them, maxA(s,a max ; θ,α) is the maximum advantage function value, a max The action that maximizes the advantage function; The Dueling DQN agent gradually updates the neural network parameters by minimizing the loss function, thereby gradually achieving an accurate estimate of the Q value: Loss=(yQ(s t ,a t ;θ,α,β)) 2 6. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 5, characterized in that: The Dueling DQN agent continuously interacts with the external power grid to obtain sample data. This sample data is then used to train the Q network until convergence. When the neural network can accurately estimate the Q value, the action that maximizes the Q value is selected, ultimately achieving data-driven adaptive virtual inertia control of multi-regional power grid energy storage.

7. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 6, characterized in that: The specific steps of extracting sample data to train the Q network are as follows: Initialize the neural network parameters θ, α, β, and set the number of training episodes. (1) In each Episode, initialize the first state s t ; (2) Select action a t Act on the environment to get the next state s t+1 , environmental feedback reward r t ; (3) t ,a t ,r t ,s t+1 ) This quadruple is stored in the experience replay set; (4) Sampling samples from the experience replay set and updating the network through the objective function containing the structural risk term; (5)s t =s t+1 , determine whether the maximum number of training steps has been reached, if so, go to step (6), otherwise go to step (2); (6) Determine whether the maximum number of training episodes has been reached. If so, the training ends; otherwise, go to step (1).

8. The data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to claim 7, characterized in that: After training, the data-driven Dueling DQN agent adjusts the inertia constant of the regional virtual inertial control unit according to the frequency changes of the regional power grid, improving the inertia power compensation capability of the energy storage systems in the two regions and achieving coordinated cooperation between the two regions.

9. A data-driven multi-regional power grid energy storage adaptive virtual inertial control system, comprising: Computer-readable storage medium and processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium and execute the data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for implementing the data-driven multi-regional power grid energy storage adaptive virtual inertia control method according to any one of claims 1 to 8 is implemented.

Images