Multi-type energy storage voltage and frequency short-time control method based on DQN algorithm
By adopting a multi-type energy storage voltage and frequency short-time control method based on DQN algorithm in the power system, the problem that traditional methods are difficult to stabilize voltage and frequency when facing renewable energy volatility and intermittentity is solved, and a more efficient control strategy is achieved, improving system performance and stability.
Patent Information
- Application Number
- CN202411885062.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-15
AI Technical Summary
When traditional power systems face renewable energy volatility and intermittent conditions, it is difficult to effectively stabilize voltage and frequency, and traditional energy storage control methods cannot effectively deal with the control imbalance caused by the SOC differences in energy storage units.
A multi-type energy storage voltage and frequency short-term control method based on DQN algorithm is adopted, and the charge and discharge power is limited by combining multi-type energy storage SOCs, and control instructions are pre-allocated based on the SOC equalization strategy, Markov decision-making process is established, the data-driven algorithm and physical model are combined, offline training of the agent is performed, and the optimal control solution is given.
An intelligent control strategy is realized, the charging and discharging power of the energy storage unit is dynamically adjusted, the overall performance of the system is improved, the stable control ability of voltage and frequency is enhanced, and the control complexity is reduced.
Smart Images
Figure CN119834318A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power systems, and in particular to a short-term control method for voltage and frequency of multiple types of energy storage based on a DQN algorithm. Background Art
[0002] With the rapid development of renewable energy, the application of energy storage technology in power systems is becoming increasingly important, especially in terms of frequency and voltage control. Traditional power systems mainly rely on conventional power sources such as thermal power and hydropower for regulation. However, due to the volatility and intermittency of renewable energy (such as wind power and solar power), the stability of voltage and frequency faces increasing challenges. In this context, various types of new energy storage systems (such as compressed air energy storage, flywheel energy storage and electrochemical energy storage) have become an important means to solve the frequency and voltage problems of power systems due to their flexibility and fast response speed.
[0003] In frequency and voltage control, the state of charge (SOC) of the energy storage system is crucial to its charging and discharging strategy. Traditional control methods mostly use fixed droop control strategies, which cannot effectively deal with the control imbalance problem caused by the SOC difference of energy storage units. Summary of the invention
[0004] In order to solve the above technical problems, the present invention proposes a short-term control method of multi-type energy storage voltage and frequency based on the DQN algorithm to realize an intelligent control strategy. This method uses big data and artificial intelligence technology to monitor and optimize the system operation in real time, and can dynamically adjust the charging and discharging power of the energy storage unit, thereby improving the overall performance of the system. In order to have a basic understanding of some aspects of the disclosed embodiments, a simple summary is given below. This summary is not a general review, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a preface to the detailed description that follows.
[0005] According to a first aspect of an embodiment of the present invention, a method for short-term control of voltage and frequency of multiple types of energy storages based on a DQN algorithm is provided.
[0006] In one embodiment, a method for short-term control of voltage and frequency of multiple types of energy storage based on a DQN algorithm includes:
[0007] Step S1, limiting the charging and discharging power in combination with multiple types of energy storage SOC, and pre-allocating control instructions based on the SOC balancing strategy;
[0008] Step S2, establishing a Markov decision process based on the physical model of multiple types of energy storage participating in frequency and voltage control, and realizing the mutual combination of data-driven algorithm and physical model;
[0009] Step S3, set the DQN network parameters, perform offline training of the intelligent agent, wait for the intelligent agent training reward to converge, perform online testing, and provide the optimal control solution based on the actual power system operation stability requirements.
[0010] Optionally, the step of pre-allocating control instructions based on the SOC balancing strategy includes:
[0011] The droop control coefficient expression is calculated according to formula (1):
[0012]
[0013] Where R0 is the rated droop coefficient of energy storage, SOC i Indicates the current state of charge of the i-th energy storage, SOC avg Represents the average state of charge of multiple types of energy storage systems, C max is the maximum capacity of the energy storage unit, C i is the current capacity of the energy storage unit, k is the acceleration factor, n is the convergence factor, i dic <0 means the energy storage unit is in charging state, i dic >0 indicates that the energy storage unit is in a discharging state.
[0014] Optionally, in step S2, a Markov decision process is established based on a physical model of frequency and voltage control involving multiple types of energy storage to achieve the mutual combination of data-driven algorithms and physical models, including: converting the dynamic optimization control of frequency and voltage of the new energy storage power system into a discrete-time Markov decision process, and using a reinforcement learning algorithm to solve the problem of dynamic optimization control of frequency and voltage.
[0015] Optionally, before establishing the Markov decision process, it is necessary to establish a physical model of multiple types of energy storage participating in frequency and voltage control, including:
[0016] Active power upper and lower limit constraints of energy storage units:
[0017] P i,min ≤P i ≤P i,max
[0018] Where P i ,P i,max ,P i,min They respectively represent the active output, upper limit of active output and lower limit of active output of the energy storage unit.
[0019] Optionally, before establishing the Markov decision process, it is necessary to establish a physical model of multiple types of energy storage participating in frequency and voltage control, including:
[0020] Reactive power upper and lower limit constraints:
[0021] Q i,min ≤Q i ≤Q i,max
[0022] In the formula, Q i ,Q i,max ,Q i,min They respectively represent the reactive output, reactive output upper limit, and reactive output lower limit of the energy storage unit.
[0023] Optionally, before establishing the Markov decision process, it is necessary to establish a physical model of multiple types of energy storage participating in frequency and voltage control, including:
[0024] State of charge upper and lower limit constraints:
[0025] SOC i,min ≤SOC i ≤SOC i,max
[0026] In the formula, SOC i ,SOC i,max ,SOC i,min They respectively represent the state of charge, upper limit of the state of charge, and lower limit of the state of charge of the energy storage unit.
[0027] Optionally, the establishing of the Markov decision process includes: establishing a system state space; establishing a system action space; establishing a system state transfer; and establishing a system reward function.
[0028] Optionally, the state space represents a finite set of states, including: active and reactive output of compressed air energy storage, active and reactive output of flywheel energy storage, and active and reactive output of electrochemical energy storage, as well as regional control deviation of power system frequency control index and central node voltage of voltage control index.
[0029] Optionally, the action space represents a finite set of behaviors, including: active and reactive output adjustment values of compressed air energy storage, active and reactive output adjustment values of flywheel energy storage, and active and reactive output adjustment values of electrochemical energy storage.
[0030] Optionally, the establishment of system state transition includes: the goal of the agent is to continuously observe the state s t , choose an appropriate action value a t , and according to the reward values r and r of the environment feedback t+1 , to optimize your behavior strategy
[0031] Optionally, the system reward function setting includes: the ACE control effect of the frequency control indicator, the node voltage control effect of the voltage control indicator, and the active and reactive power regulation output cost of the energy storage unit.
[0032] Optionally, the system reward function is as follows:
[0033]
[0034] In the formula, F ACE For ACE control reward, F Vp is the central node voltage control reward, γ is the penalty term - active and reactive output costs, f P,i (ΔP i ) is the active output cost function, f Q.i (ΔQ i ) is the reactive power output cost function, c1, c2, c3 are weight coefficients respectively. ΔP i ,ΔQ i Divided into active and reactive output adjustment values of multiple types of energy storage units.
[0035] Optionally, in step S3, DQN network parameters are set, the intelligent agent is trained offline, and after the intelligent agent training reward converges, an online test is performed, and an optimal control solution is given in combination with the actual power system operation stability requirements, specifically including: a working mode of frequency and voltage dynamic optimization control based on the DuelingDQN algorithm, including two stages: offline learning and online application.
[0036] Optionally, the training mode of the offline learning stage is as follows:
[0037] First, at the beginning of each control cycle, the control command acceptance range of each unit in the system at the initial moment is determined according to the unit power output, network lines and load of the offline system;
[0038] Then, read {P at time t from the environment caes,i ,Q caes,i ,P fess,i ,Q fess,i ,P bess,i ,Q bess,i ,ACE,V p}State observation and pass it to the agent;
[0039] Then, train the agent with DQN as the core algorithm to make action selection a t ;
[0040] Next, exchange with the power system link to obtain the next state s t+1 and reward r t ;
[0041] The above process is repeated, and new samples are continuously generated during the training process. After a fixed training time, a batch of samples are randomly selected from the experience pool to train the neural network parameters and improve the action strategy of the intelligent agent until it converges to the optimal strategy.
[0042] Optionally, in the online application phase, at the beginning of each decision moment, the DQN agent selects the optimal control scheme based on the trained actor parameters according to the relevant control parameters transmitted by the data monitoring system of the power grid control center.
[0043] According to a second aspect of an embodiment of the present invention, a computer device is provided.
[0044] In some embodiments, the computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0045] The technical solution provided by the embodiment of the present invention may have the following beneficial effects:
[0046] The physical model of frequency and voltage control of multiple types of new energy storage systems is combined with the data-driven model to realize a mixed digital-analog frequency and voltage control method. In the actual power grid frequency and voltage control, the event-driven method is used to obtain the ACE and node voltage deviation values as control instructions. The data-driven method of this application is used to solve the control strategy online, and it is uniformly controlled to issue an overall power output instruction, which is completed jointly by multiple types of new pure energy storage clusters, and the overall power only needs to be further distributed internally, making the regulation method simpler and more efficient.
[0047] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0048] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0049] Figure 1 is a flow chart of a method for short-term control of voltage and frequency of multiple types of energy storage based on a DQN algorithm according to an exemplary embodiment;
[0050] Figure 2 is a schematic diagram of a network structure of a DQN algorithm according to an exemplary embodiment;
[0051] Figure 3 The figure is a schematic diagram showing the structure of a computer device according to an exemplary embodiment. DETAILED DESCRIPTION
[0052] The following description and accompanying drawings fully illustrate the specific embodiments of this article so that those skilled in the art can practice them. Parts and features of some embodiments may be included in or replace parts and features of other embodiments. The scope of the embodiments of this article includes the entire scope of the claims, as well as all available equivalents of the claims. Herein, the terms "first", "second", etc. are only used to distinguish one element from another, without requiring or implying any actual relationship or order between these elements. In fact, the first element can also be called the second element, and vice versa. Moreover, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the structure, device or equipment including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also include elements inherent to such structure, device or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the structure, device or equipment including the elements. Each embodiment is described in a progressive manner herein, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other.
[0053] The terms "longitudinal", "lateral", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc. in this document indicate the orientation or position relationship based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing this document and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In the description of this document, unless otherwise specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a mechanical connection or an electrical connection, it can also be the internal communication of two elements, it can be a direct connection, or it can be an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.
[0054] As used herein, the term "plurality" means two or more than two, unless otherwise specified.
[0055] In this document, the character " / " indicates that the preceding and following objects are in an "or" relationship. For example, A / B means: A or B.
[0056] In this article, the term "and / or" is a description of the association relationship between objects, indicating that three relationships may exist. For example, A and / or B means: A or B, or, A and B.
[0057] It should be understood that, although the various steps in the flow chart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the figure may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0058] Each module in the device or system of the present application can be implemented in whole or in part by software, hardware, or a combination thereof. The above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to the above modules.
[0059] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.
[0060] In the prior art, the reinforcement learning method based on Markov decision process has been widely used to solve complex control problems. By establishing a physical model of the energy storage system, the frequency and voltage control problem is transformed into a state-action-reward form, and the dynamic behavior of the system is optimized using the reinforcement learning algorithm. This process not only takes into account the physical constraints of the energy storage unit, but also introduces the synergy between multiple energy storage types, improving the system's responsiveness and regulation accuracy.
[0061] In the specific implementation process, it is necessary to first reasonably limit the charging and discharging power of multiple types of energy storage, and pre-allocate control instructions through the SOC balancing strategy to ensure that various types of energy storage participate in frequency and voltage regulation in the best state. After establishing the Markov decision process, the definition of the system state space and action space becomes particularly important. The state space includes control indicators such as the active and reactive output of the energy storage unit, the system frequency and voltage, while the action space corresponds to the charging and discharging adjustment value of the energy storage unit. The advantage of using the DQN algorithm is that it can improve the stability and convergence speed of training through the introduction of the experience replay mechanism and the target network. Through continuous interaction with the system environment, the algorithm can adaptively learn the best control strategy. In the training stage, the intelligent agent selects the optimal action and provides feedback based on the real-time system state, and improves the effectiveness and reliability of the control strategy through continuous iteration. In addition, in actual engineering applications, the charging and discharging capacity of the energy storage system is limited by multiple factors such as inverter capacity and SOC. Therefore, the control strategy not only needs to consider the characteristics of the energy storage unit itself, but also must be optimized in the operating environment of the entire power system to achieve the best control effect. By introducing hierarchical control and event-driven mechanisms, this solution effectively improves the economy and coordination of the system, reduces the adjustment friction between control layers, and ensures steady-state operation of frequency and voltage.
[0062] In summary, the short-term voltage and frequency control method for multiple types of new energy storage systems based on the DQN algorithm proposed in this application not only organically combines advanced deep learning technology with energy storage system control, but also solves the shortcomings of traditional methods in SOC balancing by redefining the droop control strategy. Through this innovative control method, the dynamic response capability of the power system can be effectively improved, the optimization control complexity under small interference faults can be reduced, and strong technical support can be provided for the large-scale access of high-proportion new energy in the future.
[0063] like Figure 1 As shown, the present invention proposes a multi-type energy storage voltage and frequency short-term control method based on the DQN algorithm, comprising the following steps:
[0064] Step S1, limiting the charge and discharge power in combination with multiple types of energy storage SOC, and pre-allocating control instructions based on the SOC balancing strategy to ensure that each type of energy storage performs frequency and voltage regulation control in the optimal operating state.
[0065] Step S2, based on the physical model of multiple types of energy storage participating in frequency and voltage control, a Markov decision process is established to achieve the mutual combination of the data-driven algorithm and the physical model, thereby improving the confidence of the data-driven algorithm.
[0066] Step S3, set the DQN network parameters, perform offline training of the intelligent agent, wait for the intelligent agent training reward to converge, perform online testing, and provide the optimal control solution based on the actual power system operation stability requirements.
[0067] Therefore, the above-mentioned embodiment of the present invention solves the frequency and voltage control problem of multiple types of new energy storage systems, and at the same time combines the system frequency and voltage control of multiple types of new energy storage with the current big data intelligent computing to form a large-scale new energy storage control system with digital participation control. The method proposed in this application can achieve stable operation of the frequency and voltage of multiple types of new energy storage, effectively reduce the variable dimension and calculation complexity in the optimization control problem under small disturbance faults of the power system, improve the optimization control speed, and provide good technical support for multiple types of new energy storage to participate in the control of the power system.
[0068] The active and reactive power output control of various types of new energy storage involves a link of receiving control instructions and command distribution. The traditional power distribution method is mainly a droop control method. The conventional droop coefficient is generally a constant. If the initial SOC of the energy storage unit is different, the SOC autonomous balance cannot be achieved. In order to solve the problems existing in traditional droop control, this application proposes a step of pre-distributing control instructions based on the SOC balance strategy, including: The droop control coefficient expression can be calculated according to formula (1):
[0069]
[0070] Where R0 is the rated droop coefficient of energy storage, SOC i Indicates the current state of charge of the i-th energy storage, SOC avg Represents the average state of charge of multiple types of energy storage systems, C max is the maximum capacity of the energy storage unit, C i is the current capacity of the energy storage unit, k is the acceleration factor, n is the convergence factor, i dic <0 means the energy storage unit is in charging state, i dic >0 indicates that the energy storage unit is in a discharging state.
[0071] When the initial SOC of the two energy storage units is at SOC i >SOC j When the SOC change rate difference is less than 0, the SOC change rate of the i-th energy storage unit will be less than the change rate of the j-th energy storage unit, so the SOC difference of the two energy storage units will gradually decrease until they are the same. The present invention adopts an improved droop control strategy, by dynamically adjusting R i This adjusts the charging and discharging speed and achieves fast SOC balance between energy storage units. When charging, the larger the SOC, the higher the R i As it increases, the output current becomes smaller; the smaller the SOC, the lower the Ri As it decreases, the output current increases. Discharging is the opposite of charging.
[0072] Based on the hierarchical and regional coordinated control architecture of network-province-region coordination commonly constructed in China, the hierarchical control and event-driven mechanism are adopted to effectively improve the economic efficiency of system operation and reduce the reciprocating adjustment between control systems. The coordinated control system is decomposed into the highest control layer, the middle control layer and the bottom control layer. The optimization objectives of each control layer are also focused according to the different control objects, control means and time scales, so as to realize the decomposition and coordination of the coordinated optimization system from the time dimension, space dimension and target dimension. There is data information exchange between each control layer, and the control instructions are transmitted from top to bottom. At the same time, the system dispatcher can directly intervene in the instructions output by the highest layer to solve the problems of data transmission speed, frequency and voltage control cycle of the existing architecture. The core middle layer of the control architecture is based on the droop control coefficient of the SOC balancing strategy. Various types of new energy storage accept the active and reactive instructions issued by the highest control system according to their own capacity and SOC status for balanced distribution. And then the allocated instruction scheme is sent to the specific energy storage unit for execution.
[0073] Among them, different types of new energy storage power models and their SOC calculation methods are as follows:
[0074] (1) Compressed air energy storage
[0075] The mathematical model of compressed air energy storage includes multiple control parameters such as gas storage tank, expander, heat exchanger, valve, etc. In order to simplify the calculation, the compressed air energy storage P caes The following control model is used:
[0076]
[0077] Where W is the mass flow rate of the regulating threshold, c p are the parameters of the ideal air state, is the ambient input temperature, β is the expansion ratio of the expander, and k is the ideal specific heat capacity of air.
[0078] The calculation formula for the state of charge of the compressed air energy storage unit is as follows:
[0079]
[0080] In the formula, SOC caes is the current state of charge of the compressed air energy storage, SOC(0) caes is the initial charge state of compressed air energy storage, E N,caes It is the rated capacity of compressed air energy storage.
[0081] (2) Electrochemical energy storage
[0082] The state of charge calculation formula for electrochemical energy storage is as follows:
[0083]
[0084] In the formula, SOC bess is the current state of charge of the electrochemical energy storage, SOC(0) bees is the initial state of charge of electrochemical energy storage, α i Is the charging mark, which is 1 when in charging state, β i It is the discharge flag, which is 1 in the discharge state. It cannot be charged and discharged at the same time. cha is the charging efficiency of electrochemical energy storage, η dis is the discharge efficiency of electrochemical energy storage, P bess is the output of the electrochemical energy storage unit, T is the unit time period, E N,bess is the rated capacity of electrochemical energy storage.
[0085] (3) Flywheel energy storage
[0086] Flywheel energy storage controls the output power by driving the flywheel to accelerate or decelerate, completing the task of storing and releasing electrical energy. The specific power output model is as follows:
[0087] P out =T fess ·ω fess
[0088]
[0089] Where P out is the power output of the flywheel energy storage unit, T fess is the electromagnetic torque of the flywheel energy storage unit, J fess is the moment of inertia of the flywheel energy storage unit, ω fess is the angular velocity of the flywheel.
[0090] The charge state of the flywheel energy storage is calculated as follows:
[0091]
[0092] In the formula, SOC fess The current state of charge of the flywheel energy storage, SOC(0) fess is the initial charge state of the flywheel energy storage, E rate is the rated capacity of the flywheel energy storage.
[0093] In the engineering application of the multi-type new energy storage in the present invention, the actual charging and discharging power is subject to various restrictions, including the inverter capacity and SOC limit of the energy storage system. In steady state, the maximum charging / discharging active power is limited by the energy storage inverter capacity and the state of charge of the energy storage, while the reactive power is limited by the remaining capacity of the energy storage inverter. When there is remaining inverter capacity, it is considered to control the charging / discharging reactive power of the energy storage system to participate in the grid voltage regulation. Therefore, the maximum charging / discharging reactive power should not exceed the remaining inverter capacity, as shown in the following formula:
[0094]
[0095] Where P is the active output of the energy storage unit, S N is the rated capacity of the energy storage system inverter, and Q is the reactive output of the energy storage unit.
[0096] The step S2, according to the physical model of multiple types of energy storage participating in frequency and voltage control, establishes a Markov decision process to realize the mutual combination of data-driven algorithm and physical model, including: converting the frequency and voltage dynamic optimization control problem of the new energy storage power system into a discrete-time Markov decision process. The reinforcement learning algorithm is a random control solution method based on the Markov decision process. Therefore, the present invention uses the reinforcement learning algorithm to solve the frequency and voltage dynamic optimization control problem.
[0097] Before establishing the Markov decision process, it is necessary to establish a physical model for multiple types of energy storage to participate in frequency and voltage control, including:
[0098] Active power upper and lower limit constraints of energy storage units:
[0099] P i,min ≤P i ≤P i,max
[0100] Reactive power upper and lower limit constraints:
[0101] Q i,min ≤Q i ≤Q i,max
[0102] State of charge upper and lower limit constraints:
[0103] SOC i,min ≤SOC i ≤SOC i,max
[0104] Where P i ,P i,max ,P i,min They represent the active output, upper limit, lower limit of active output, and Q of the energy storage unit respectively. i ,Qi,max ,Q i,min Respectively represent the reactive power output, reactive power upper limit, reactive power lower limit, SOC i ,SOC i,max ,SOC i,min They respectively represent the state of charge, upper limit of the state of charge, and lower limit of the state of charge of the energy storage unit.
[0105] The Markov decision process based on the physical model of multiple types of new energy storage systems includes the following:
[0106] (1) Establishing the system state space
[0107] The state space represents a finite set of states. The state space of the present invention specifically includes: the active and reactive output of compressed air energy storage, the active and reactive output of flywheel energy storage, and the active and reactive output of electrochemical energy storage, as well as the power system frequency control index regional control deviation ACE, and the voltage control index central node voltage Vp, which are as follows:
[0108] {P caes,i ,Q caes,i ,P fess,i ,Q fess,i ,P bess,i ,Q bess,i ,ACE,V p}
[0109] (2) Establishing the system action space
[0110] The action space represents a finite set of behaviors. The state space of the present invention specifically includes: active and reactive output adjustment values of compressed air energy storage, active and reactive output adjustment values of flywheel energy storage, and active and reactive output adjustment values of electrochemical energy storage, as follows:
[0111] {ΔP caes,i ,ΔQ caes,i ,ΔP fess,i ,ΔQ fess,i ,ΔP bess,i ,ΔQ bess,i}
[0112] (3) Establishing system state transfer
[0113] The goal of the agent is to continuously observe the state s t , choose an appropriate action value a t , and according to the reward values r and r of the environment feedback t+1 , to optimize your behavioral strategy.
[0114] (4) Establishing the system reward function
[0115] The system reward function setting mainly includes three aspects: the ACE control effect of the frequency control index, the node voltage control effect of the voltage control index, and the active and reactive power regulation output cost of the energy storage unit. The specific reward function formula is as follows:
[0116]
[0117] In the formula, F ACE For ACE control reward, F Vp is the central node voltage control reward, γ is the penalty term - active and reactive output costs, f P,i (ΔP i ) is the active output cost function, f Q.i (ΔQ i ) is the reactive power output cost function, c1, c2, c3 are weight coefficients respectively. ΔP i ,ΔQ i Divided into active and reactive output adjustment values of multiple types of energy storage units.
[0118] The DQN algorithm uses a deep neural network to approximate the value function, which can effectively solve the "curse of dimensionality" problem of the traditional value function algorithm Q-leaning in large-scale environments. Its network structure is as follows: Figure 2 As shown in the figure, the input of the neural network is the state observation s of the system, and the output is the value Q(s,a) of each action under this state, with multiple hidden layers in the middle.
[0119] The DQN algorithm introduces an experience replay mechanism to break the correlation between data and improve the stability of neural network training. t ,a t ,r t ,s t+1 ) is put into the experience pool. When the neural network is trained, a fixed number of samples are randomly drawn from the experience pool to update the neural network weights. In addition, DQN also introduces two neural networks with the same structure to approximate the state-action value function and generate the target Q value. The former is called the current Q network, which is used to approximate the value function Q(s,a| θ ), which is used to help the agent choose the appropriate action, where θ represents the weight of the neural network. The latter is called the target Q network, which is used to generate the target Q value to guide the parameter update of the current Q network. The output of this network is usually expressed as Q(s,a| θ' ). The parameters of the current Q network are updated at each interaction, while the parameters of the target Q network are updated every certain number of steps. The DQN algorithm specifically includes the following steps:
[0120] (1) Get the current system status s t , and input into the current Q network;
[0121] (2) The Q network outputs the value Q(s,a| θ ), and select the corresponding action a t Act on the environment;
[0122] (3) The environment gives the current instant reward r t and the state quantity s at the next moment t+1 , update the current environment and transfer the sample (s t ,a t ,r t ,s t+1 ) is stored in the experience pool;
[0123] (4) Randomly select m groups of transfer samples (s i ,a i ,r i ,s i+1 ), i = 1, 2…m, and input them into the current Q network and the target Q network respectively;
[0124] (5) Calculated by the target Q network:
[0125]
[0126] In the formula, y i is the value function, r j is the reward function, γ DQN is the discount factor.
[0127] (6) Calculate the mean square error loss function:
[0128]
[0129] According to the back propagation algorithm of the neural network, the parameters θ of the current Q network are updated.
[0130] (7) After a fixed number of iterations, the parameters of the target Q network are replaced by the parameters θ of the current Q network.
[0131] In the above step S3, the DQN network parameters are set, the intelligent agent is trained offline, and the intelligent agent training reward converges, and then an online test is performed. In combination with the actual power system operation stability requirements, the optimal control scheme is given, which specifically includes: a working mode of frequency and voltage dynamic optimization control based on the DuelingDQN algorithm, including two stages: offline learning and online application.
[0132] The training mode of the offline learning stage is as follows: First, at the beginning of each control cycle, the control command acceptance range of each unit in the system at the initial moment is determined according to the system parameter values such as unit power output, network line and load of the offline system. Secondly, {P caes,i ,Q caes,i ,P fess,i ,Q fess,i ,P bess,i ,Q bess,i ,ACE,V p} state observations and pass them to the agent. Then, the agent with DQN as the core algorithm is trained to make action selections a t . Then exchange with the power system link to obtain the next state s t+1 and reward r t , and new samples are continuously generated during the training process. After a fixed training time, a batch of samples are randomly selected from the experience pool to train the neural network parameters and improve the action strategy of the intelligent agent until it converges to the optimal strategy.
[0133] In the online application stage, at the beginning of each decision moment, the data monitoring system of the power grid control center transmits relevant control parameters, and the DQN agent selects the optimal control scheme based on the trained actor parameters.
[0134] Therefore, the present invention combines the physical model of frequency and voltage control of energy storage in various types of new energy storage systems with the data-driven model, realizes a frequency and voltage control method with a mixed digital-analog combination, and uses an event-driven method to obtain the ACE and node voltage deviation values as control instructions in actual power grid frequency and voltage control. The data-driven method of the present application is used to solve the control strategy online, and it is uniformly controlled to issue an overall power output instruction, which is completed jointly by various types of new pure energy storage clusters, and the overall power only needs to be further distributed internally, making the regulation method simpler and more efficient.
[0135] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store static information and dynamic information data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, the steps in the above method embodiment are implemented.
[0136] Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the computer device to which the solution of the present invention is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0137] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiment when executing the computer program.
[0138] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above method embodiment are implemented.
[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0140] The present invention is not limited to the structures which have been described above and shown in the drawings, and various modifications and changes may be made without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.
Claims
1. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm, characterized in that: include: Step S1, limiting the charging and discharging power in combination with multiple types of energy storage SOC, and pre-allocating control instructions based on the SOC balancing strategy; Step S2, establishing a Markov decision process based on the physical model of multiple types of energy storage participating in frequency and voltage control, and realizing the mutual combination of data-driven algorithm and physical model; Step S3, set the DQN network parameters, perform offline training of the intelligent agent, wait for the intelligent agent training reward to converge, perform online testing, and provide the optimal control solution based on the actual power system operation stability requirements.
2. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 1, characterized in that: The step of pre-allocating control instructions based on the SOC balancing strategy includes: The droop control coefficient expression is calculated according to formula (1): Where R0 is the rated droop coefficient of energy storage, SOC i Indicates the current state of charge of the i-th energy storage, SOC avg Represents the average state of charge of multiple types of energy storage systems, C max is the maximum capacity of the energy storage unit, C i is the current capacity of the energy storage unit, k is the acceleration factor, n is the convergence factor, i dic <0 means the energy storage unit is in charging state, i dic >0 indicates that the energy storage unit is in a discharging state.
3. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 1, characterized in that: The step S2 establishes a Markov decision process based on the physical model of multiple types of energy storage participating in frequency and voltage control to achieve the mutual combination of data-driven algorithm and physical model, including: converting the dynamic optimization control of frequency and voltage of the new energy storage power system into a discrete-time Markov decision process, and using a reinforcement learning algorithm to solve the problem of dynamic optimization control of frequency and voltage.
4. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 3, characterized in that: Before establishing the Markov decision process, it is necessary to establish a physical model for multiple types of energy storage participating in frequency and voltage control, including: Active power upper and lower limit constraints of energy storage units: P i,min ≤P i ≤P i,max Where P i ,P i,max ,P i,min They respectively represent the active output, upper limit of active output and lower limit of active output of the energy storage unit.
5. The method for short-term control of voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 3, characterized in that: Before establishing the Markov decision process, it is necessary to establish a physical model for multiple types of energy storage participating in frequency and voltage control, including: Reactive power upper and lower limit constraints: Q i,min ≤Q i ≤Q i,max In the formula, Q i ,Q i,max ,Q i,min They respectively represent the reactive output, reactive output upper limit, and reactive output lower limit of the energy storage unit.
6. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 3, characterized in that: Before establishing the Markov decision process, it is necessary to establish a physical model for multiple types of energy storage participating in frequency and voltage control, including: State of charge upper and lower limit constraints: SOC i,min ≤SOC i ≤SOC i,max In the formula, SOC i ,SOC i,max ,SOC i,min They respectively represent the state of charge, upper limit of the state of charge, and lower limit of the state of charge of the energy storage unit.
7. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 3, characterized in that: The establishment of the Markov decision process includes: establishing a system state space; establishing a system action space; establishing a system state transfer; and establishing a system reward function.
8. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 7, characterized in that: The state space represents a finite set of states, including: active and reactive output of compressed air energy storage, active and reactive output of flywheel energy storage, and active and reactive output of electrochemical energy storage, as well as regional control deviation of power system frequency control index and central node voltage of voltage control index.
9. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 7, characterized in that: The action space represents a finite set of behaviors, including: active and reactive output adjustment values of compressed air energy storage, active and reactive output adjustment values of flywheel energy storage, and active and reactive output adjustment values of electrochemical energy storage.
10. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 7, characterized in that: The establishment of system state transfer includes: the goal of the agent is to continuously observe the state s t , choose an appropriate action value a t , and according to the reward values r and r of the environment feedback t+1 , to optimize your behavioral strategy.
11. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 7, characterized in that: The system reward function setting includes: the ACE control effect of the frequency control indicator, the node voltage control effect of the voltage control indicator, and the active and reactive power regulation output cost of the energy storage unit.
12. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 11, characterized in that: The system reward function is as follows: In the formula, F ACE For ACE control rewards, is the central node voltage control reward, γ is the penalty term - active and reactive output costs, f P,i (ΔP i ) is the active output cost function, f Q.i (ΔQ i ) is the reactive power output cost function, c1, c2, c3 are weight coefficients respectively. ΔP i ,ΔQ i Divided into active and reactive output adjustment values of multiple types of energy storage units.
13. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 1, characterized in that: The step S3 sets the DQN network parameters, performs offline training of the intelligent agent, waits for the intelligent agent training reward to converge, performs online testing, and provides an optimal control solution in combination with the actual power system operation stability requirements, specifically including: a working mode of frequency and voltage dynamic optimization control based on the DuelingDQN algorithm, including two stages: offline learning and online application.
14. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 13, characterized in that: The training mode of the offline learning stage is as follows: First, at the beginning of each control cycle, the control command acceptance range of each unit in the system at the initial moment is determined according to the unit power output, network lines and load of the offline system; Then, read {P at time t from the environment caes,i ,Q caes,i ,P fess,i ,Q fess,i ,P bess,i ,Q bess,i ,ACE,V p }State observation and pass it to the agent; Then, train the agent with DQN as the core algorithm to make action selection a t ; Next, exchange with the power system link to obtain the next state s t+1 and reward r t ; The above process is repeated, and new samples are continuously generated during the training process. After a fixed training time, a batch of samples are randomly selected from the experience pool to train the neural network parameters and improve the action strategy of the intelligent agent until it converges to the optimal strategy.
15. A short-term control method for voltage and frequency of multiple types of energy storage based on DQN algorithm as claimed in claim 13, characterized in that: In the online application phase, at the beginning of each decision moment, the data monitoring system of the power grid control center transmits relevant control parameters, and the DQN agent selects the optimal control solution based on the trained actor parameters.
16. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 15 are implemented.