Energy storage charge state balance secondary control method based on safety reinforcement learning
By employing a deep deterministic policy gradient agent based on secure reinforcement learning, combined with droop control and a virtual impedance model, the problem of charge state imbalance in energy storage systems is solved, achieving charge state balance and system stability, thereby improving the adaptability and safety of energy storage devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SHANGHAI UNIV
- Filing Date
- 2025-12-17
- Publication Date
- 2026-05-01
AI Technical Summary
The imbalance of state of charge in existing energy storage systems leads to a decrease in equipment efficiency, an increase in the risk of individual overcharging and over-discharging, and traditional control methods have weak generalization ability, strong model dependence and insufficient safety, making them unable to adapt to the needs and characteristics of different systems.
A deep deterministic policy gradient agent based on safe reinforcement learning is adopted. Combining droop control and virtual impedance models, a state space, action space and reward function are constructed. The control policy is learned directly from the system interaction through reinforcement learning. A soft update mechanism for the target network parameters of the deep deterministic policy gradient agent is established to achieve charge state balance and system stability.
It achieves state-of-charge balance and frequency stability in different systems, has good adaptability and generalization ability, avoids system instability, and improves the operating efficiency and safety of energy storage devices.
Smart Images
Figure CN121965656A_ABST
Abstract
Description
A Secondary Control Method for Energy Storage State-of-Charge Balance Based on Security Reinforcement Learning Technical Field
[0001] This invention relates to the field of energy storage technology, and in particular to a secondary control method for energy storage state of charge balance based on security reinforcement learning, and also to a corresponding control system. Background Technology
[0002] With the large-scale grid connection of renewable energy sources such as wind and solar power, the power system faces challenges such as severe power fluctuations on the source side and widening peak-valley load differences. To mitigate the uncertainty of renewable energy output and improve system stability, energy storage, as a regulating resource, has been widely deployed. It can balance source-load differences, suppress frequency shifts, and stabilize auxiliary voltage through the charging and discharging process, becoming an important component supporting the grid-friendly operation of renewable energy. Especially in regional power grids with tight source-grid-load coupling, the rational management of energy storage resources is crucial for the efficiency of renewable energy utilization and system security.
[0003] In multi-energy storage collaborative operation scenarios, state-of-charge (POC) imbalance can lead to decreased efficiency of energy storage devices, increased risk of individual overcharging and over-discharging, and impact on the overall system lifespan and economic viability. Currently, commonly used POC strategies are primarily based on proportional-integral (PI) controllers, adjusting reference power, droop coefficient, or virtual impedance to achieve POC consistency. However, these methods often rely on proportional and integral parameters in the POC secondary controller designed for a specific system, making it difficult to adapt to the needs and characteristics of different systems. This results in variations in control performance and stability across different systems. Therefore, a more adaptive control strategy is urgently needed, capable of learning and dynamically adjusting control parameters to adapt to the characteristics of different systems and ensure stable operation without relying on precise modeling of the secondary controller.
[0004] Reinforcement learning, as a model-free and self-learning intelligent control method, has been gradually applied in power systems in recent years, demonstrating strong adaptability and optimization capabilities, especially in areas such as renewable energy forecasting, intelligent dispatching, and energy storage control. Among these, deep deterministic policy gradient algorithms are suitable for power electronics control problems due to their ability to handle continuous action spaces. However, their safety and controllability in renewable energy scenarios remain challenges, particularly in tasks involving energy storage state-of-charge control that require meeting physical safety constraints. Issues such as poor policy interpretability, unrestricted actions, and unstable training processes urgently need to be addressed.
[0005] In summary, traditional control methods suffer from weak generalization ability, strong model dependence, and insufficient security of reinforcement learning strategies, thus failing to meet the needs for rational regulation of energy storage resources. Summary of the Invention
[0006] To address the aforementioned shortcomings in existing technologies, this invention provides a secondary control method for energy storage charge state balance based on security reinforcement learning, and also provides a corresponding control system.
[0007] According to one aspect of the present invention, a secondary control method for energy storage state-of-charge balance based on security reinforcement learning is provided, comprising: constructing an energy storage inverter model based on droop control and virtual impedance to describe power conversion and grid-connected operation; setting control objectives and constraints for a secondary controller based on the energy storage inverter model to achieve state-of-charge balance; establishing the state space, action space, and reward function of a deep deterministic policy gradient agent, as well as the policy network objective function and value network loss function, based on the control objectives and constraints of the secondary controller, and using the obtained deep deterministic policy gradient agent as the secondary controller; establishing a soft update mechanism for the target network parameters of the deep deterministic policy gradient agent; and establishing an energy storage state-of-charge balance control strategy based on the secondary control of the deep deterministic policy gradient agent and its target network parameter soft update mechanism.
[0008] Preferably, the construction of the energy storage inverter model based on droop control and virtual impedance includes: establishing the inverter main circuit structure, including: energy storage, a three-phase full-bridge DC-AC inverter circuit, an LC filter circuit, and line impedance; wherein, the DC power output from the energy storage is fed into the three-phase full-bridge DC-AC inverter circuit via a DC bus; the three-phase output of the three-phase full-bridge DC-AC inverter circuit passes sequentially through a filter inductor. Measuring the equivalent resistance of the inverter Then connect the filter capacitor This forms an LC filter circuit to suppress inverter switching harmonics; the filter capacitor node then passes through the line resistance of the line impedance. and line inductance The energy is connected to a three-phase voltage source in the power grid; wherein, the state of charge expression of the energy storage is:
[0009] In the formula, For energy storage exist The state of charge value at time t; For energy storage At the initial moment The electrical state value of the charge; For energy storage The capacity; For energy storage The charge and discharge efficiency; For energy storage exist The system continuously outputs active power; it establishes a primary control loop based on droop control and virtual impedance, including: droop control, virtual impedance control, voltage and current dual closed-loop control, and PWM control; firstly, it collects the voltage and current at the LC filter port, calculates the active and reactive power output of the primary controller, and the droop control generates the voltage outer loop reference voltage amplitude and phase based on the output active and reactive power; secondly, it generates d-axis and q-axis voltages through dq transformation, and superimposes the virtual impedance onto the d-axis and q-axis reference voltages of the voltage outer loop control input; the deviation between the reference voltage and the actual voltage value is used by the PI controller to generate the current inner loop d-axis and q-axis reference currents; then the current inner loop generates PWM modulation signals through the PI controller; finally, it achieves grid connection after PWM control; the equation expression of the primary control loop based on droop control and virtual impedance is:
[0010] In the formula, For energy storage exist Reactive power output at all times and Energy storage exist The droop coefficients of the active and reactive power loops in the constant droop control; and Energy storage exist Frequency and voltage at any given moment; and The system's reference frequency and voltage; For virtual resistor The resulting equivalent active power loss; For virtual reactance The resulting equivalent reactive power loss; For energy storage exist Current at any given moment.
[0011] Preferably, the setting of the control objectives and constraints of the secondary controller includes: establishing the control objectives of the secondary controller with the aim of achieving state-of-charge equilibrium for energy storage, and minimizing both the state-of-charge deviation and the frequency deviation. and for:
[0012]
[0013] In the formula, For energy storage exist The state of charge value at time t; For energy storage At the initial moment The state of charge value; For energy storage exist The droop coefficient of the active loop in the constant droop control; For energy storage exist Frequency of time; For energy storage exist Virtual resistance in real-time virtual impedance control; For energy storage exist Current at any given moment; For the amount of energy stored; For frequency deviation, the following constraints are established:
[0014] In the formula: and These are the upper and lower limits for adjusting the droop coefficient, respectively. and These are the upper and lower limits for adjusting the virtual resistance, respectively. and These are the upper and lower limits of the current, respectively; and These are the upper and lower limits of active power, respectively; and These are the upper and lower limits of the state of charge, respectively.
[0015] Preferably, , , and The value of is determined based on the energy storage inverter model, a small-signal model and state-space equations are established, and the upper and lower limits of the action space are obtained by analyzing the Nyquist stability criterion.
[0016] Preferably, establishing the state space of the deep deterministic policy gradient agent includes: selecting, based on the control objective of the secondary controller, the state space of the agent with the first... Taiwan's energy storage state of charge Frequency of grid connection points Output active power ,load Output current The droop coefficient of the active power loop in droop control. And virtual resistance in virtual impedance control As observations of the agent, a state space is established. , is represented as:
[0017] Establish the action space of a deep deterministic policy gradient agent, including: designing action variables as adjustment amounts for the droop coefficient and virtual resistance, then the action space... Represented as:
[0018] In the formula, and These are the adjustment amounts for the droop coefficient and the virtual resistance, respectively; based on the constraints of the secondary controller, the constraint range of the action space is set; the action boundary of this constraint range satisfies the following conditions:
[0019] Establishing the reward function for a deep deterministic policy gradient agent includes: constructing an unconstrained reward function based on the Lagrange objective function. , is represented as:
[0020] In the formula, , As a weighting factor, it is used to balance the state-of-charge equilibrium and frequency stability; Let be the Lagrange multipliers for each constraint, where Denotes the Lagrange multipliers for the upper bound constraint. Represents the Lagrange multiplier for the lower bound constraint.
[0021] Preferably, the objective function of the policy network of the deep deterministic policy gradient agent is established, including: using the reward function of the deep deterministic policy gradient agent to guide the value network. The value network is trained with the current state-action pair As input, the function outputs an estimate of the expected long-term return, with the learning objective being to approximate the cumulative discounted reward.
[0022] In the formula, The discount factor represents the discount factor for the future th... The discount weight of step rewards; For intelligent agents at any time Based on the immediate reward returned by the environment according to the current state and action; iterative training is performed using the Bellman equation form:
[0023] In the formula, The target Q value, constructed based on the Bellman equation, is used as the target label during the current value network update. and These are the target value network and the target strategy network, respectively. The immediate reward is calculated based on the current state and actions. Discount factor; The state at the next moment; based on this, select the optimal action. As the objective of the policy network, maximizing the Q-value of the value network output, the objective function of the policy network is... Defined as:
[0024] In the formula, Let it be the expected function; These are the parameters of the policy network; The state distribution sampled in the experience replay buffer; the value network loss function of the deep deterministic policy gradient agent is established, including: the value network used to estimate state-action pairs. The expected return, its loss function Defined as the mean square error between the current Q value and the target Q value, i.e.:
[0025] In the formula, Let it be the expected function; These are the parameters of the current value network.
[0026] Preferably, a soft update mechanism for the target network parameters of a deep deterministic policy gradient agent is established, comprising: the target policy network and the target value network correspond to the current policy network and the current value network, respectively, and the parameters of the target policy network and the target value network are iteratively updated through a soft update method; wherein, the soft update uses the following recursive formula for smooth parameter replacement:
[0027] In the formula, The soft update coefficient is much less than 1; These are the parameters for the current policy network and the current value network, respectively; These are the parameters for the target policy network and the target value network, respectively.
[0028] Preferably, the establishment of the energy storage charge state equalization control strategy based on deep deterministic policy gradient agent secondary control includes: the current policy network obtaining the current state from the energy storage environment. The current policy network calculates the current action based on the current state and sends it as a reference control quantity to the energy storage environment. The energy storage system executes the corresponding control operation based on the issued control quantity, runs for one time step, and then returns to the next state. and rewards And record the quadruple The data is entered into the experience replay pool; the deep deterministic policy gradient agent randomly samples a batch of data from the experience replay pool to update the value network and policy network; the target policy network outputs the target action based on the next state, which is fed into the target value network to calculate the target value, and the training target is constructed by combining the reward; the value network minimizes the loss between the target Q value and the predicted Q value, and updates the network parameters through backpropagation; the policy network maximizes the action value under the current policy to optimize policy performance; the target network parameters are adjusted through a soft update mechanism to improve training stability; the above steps are repeated until the training meets the termination condition.
[0029] According to another aspect of the present invention, a secondary control system for energy storage state-of-charge balance based on security reinforcement learning is provided, comprising: an energy storage inverter model module, which is used to construct an energy storage inverter model based on droop control and virtual impedance to describe power conversion and grid-connected operation; an agent construction module, which sets the control objectives and constraints of the secondary controller based on the energy storage inverter model to achieve state-of-charge balance; based on the control objectives and constraints of the secondary controller, establishing the state space, action space, and reward function of a deep deterministic policy gradient agent, as well as the policy network objective function and value network loss function, and using the obtained agent as the secondary controller; establishing a soft update mechanism for the target network parameters of the deep deterministic policy gradient agent; and an energy storage state-of-charge balance control module, which establishes an energy storage state-of-charge balance control strategy based on the deep deterministic policy gradient agent and its target network parameter soft update mechanism to achieve energy storage state-of-charge balance control.
[0030] Due to the adoption of the above technical solution, this invention has at least one of the following beneficial effects compared with the prior art: This invention does not rely on specific secondary controller models and parameters, and has good versatility and adaptability. Compared with the traditional SOC equalization method based on PI controllers, which requires parameter tuning for different systems, this invention directly learns control strategies from system interactions through reinforcement learning agents, possessing strong generalization and online adaptability, and is suitable for various energy storage structures and new energy grid connection scenarios.
[0031] In terms of safety, this invention constructs an action space constraint mechanism based on the system's small-signal stability analysis results, effectively limiting the policy output range and ensuring that the generated control commands operate within the physically feasible domain, thus avoiding system instability and other unsafe states. Furthermore, a reward function that comprehensively considers SOC differences and frequency stability is designed, enabling the control policy to quickly achieve state-of-charge equilibrium while also ensuring system stability and reliability.
[0032] This invention employs a deep deterministic policy gradient algorithm to construct an intelligent agent, and improves the stability and convergence speed of policy training through a target network, experience replay, and soft update mechanisms. The trained control policy can be modularly deployed in practical control platforms, exhibiting good portability and engineering implementation value. Attached Figure Description
[0033] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 is a schematic diagram of the main circuit structure of an energy storage inverter in one embodiment.
[0034] Figure 2 is a block diagram of the primary control of the inverter in one embodiment.
[0035] Figure 3 is a control block diagram of an energy storage state-of-charge equalization control strategy based on deep deterministic policy gradient agent secondary control in one embodiment.
[0036] Figure 4 shows a Nyquist plot in one embodiment: the evolution of the stability boundary under the change of the droop coefficient parameter.
[0037] Figure 5 shows a Nyquist plot in one embodiment: the evolution of the stability boundary under changes in virtual impedance parameters.
[0038] Figure 6 is a schematic diagram of the operation of a secondary control method for energy storage charge state balance based on security reinforcement learning in one embodiment. Detailed Implementation
[0039] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.
[0040] Existing energy storage state-of-charge (SOC) equalization control methods suffer from problems such as strong model dependence, poor adaptability of secondary controller parameters, and lack of stability and security guarantees. To address these issues, one embodiment of this invention provides a model-free secondary control method based on secure reinforcement learning for energy storage SOC equalization (hereinafter referred to as the secure reinforcement learning-based secondary control method for energy storage SOC equalization). This method can handle the SOC equalization problem of multiple energy storage systems. It employs a deep deterministic policy gradient agent combined with stability analysis results to constrain the action space, constructs a reward function with physical constraints and goal orientation, and obtains a generalizable and deployable control strategy through policy network training. It is applicable to various new energy grid-connected systems and possesses advantages such as strong adaptability, model independence, and high control accuracy, thereby achieving the SOC equalization control objective of multiple energy storage systems under dynamic operating conditions.
[0041] Specifically, as shown in Figure 6, the energy storage state-of-charge balance secondary control method based on security reinforcement learning provided in this embodiment may include: Step S1, constructing an energy storage inverter model based on droop control and virtual impedance to realize power conversion and grid-connected operation. In some preferred embodiments, this step may further include: Step S1.1, establishing the inverter main circuit structure shown in Figure 1: including energy storage, a three-phase full-bridge DC-AC inverter circuit, an LC filter circuit, and line impedance, wherein the DC power output from the energy storage is fed into the inverter circuit via the DC bus; the three-phase output of the inverter circuit passes sequentially through the filter inductor. Measuring the equivalent resistance of the inverter Then connect the filter capacitor This forms an LC filter circuit to suppress inverter switching harmonics; the filter capacitor node is then connected to the line resistance of the line impedance. and line inductance It is connected to a three-phase voltage source in the power grid. The state-of-charge expression for the energy storage is:
[0042] In the formula: For energy storage exist The state of charge value at time t; For energy storage At the initial moment State of charge (SOC) value; For energy storage The capacity; For energy storage The charge and discharge efficiency; For energy storage exist The active power output at any given time.
[0043] Step S1.2: Establish a primary control loop based on droop and virtual impedance as shown in Figure 2: droop control, virtual impedance control, voltage and current dual closed-loop control, and a PWM generator. The control flow is as follows: First, acquire the voltage and current at the LC filter port, calculate the active and reactive power outputs of the primary controller, and generate the voltage outer loop reference voltage amplitude and phase based on the output active and reactive power through the droop control loop. Next, generate the d-axis and q-axis voltages through dq transformation. Then, superimpose the virtual impedance onto the d-axis and q-axis reference voltages of the voltage outer loop control input through the virtual impedance control loop. The deviation between the reference voltage and the actual voltage is passed through the PI controller of the voltage outer loop to generate the reference currents for the d-axis and q-axis of the current inner loop. Then, generate the PWM modulation signal through the PI controller of the current inner loop. Finally, achieve grid connection after PWM control. The equation expression for the primary control loop based on droop control and virtual impedance is:
[0044] In the formula: and Energy storage exist The droop coefficients of the active and reactive power loops in the constant droop control; and Energy storage exist Frequency and voltage at any given moment; and The system's reference frequency and voltage; For virtual resistor The resulting equivalent active power loss; For virtual reactance The resulting equivalent reactive power loss; For energy storage exist Current at any given moment.
[0045] Step S2: Based on the energy storage inverter model, set the control objectives and constraints of the secondary controller to achieve state-of-charge (POC) balance. In some preferred embodiments, this step may further include: Step S2.1: Considering that the energy storage's POC balance control objective can be achieved by adjusting the parameters of the active power loop, and that adjusting the active power loop will affect the frequency, the following two control objectives are designed to minimize the POC deviation and the frequency deviation:
[0046]
[0047] In the formula, for exist The state of charge value at time t; for At the initial moment State of charge (SOC) value; For energy storage exist The droop coefficient of the active loop in the constant droop control; For energy storage exist Frequency of time; For energy storage exist Virtual resistance in real-time virtual impedance control; For energy storage exist Current at any given moment; This refers to the amount of energy stored.
[0048] Step S2.2, to meet the system's stability and security requirements, the following constraints must be satisfied:
[0049] In the formula: and These are the upper and lower limits for adjusting the droop coefficient, respectively. and These are the upper and lower limits for adjusting the virtual resistance, respectively. and These are the upper and lower limits of the current, respectively; and These are the upper and lower limits of active power, respectively; and These are the upper and lower limits of the state of charge, respectively; step S2.3, where , , and Based on the inverter model established in step S1, a small-signal model and state-space equations are established. The system transfer function is established based on the inverter's state-space equations, and the Nyquist curve of the system in the complex frequency domain is plotted, as shown in Figures 4 and 5. The stability trend of the system under different control parameters is analyzed. By determining whether the system's open-loop response bypasses the critical point (–1, 0j), the stable operating range of the droop coefficient and virtual impedance in the controller parameters is determined, and their feasible upper and lower boundaries are obtained. , , and ,in, , , and This is to ensure the stability and safety of the system under control and regulation.
[0050] Step S3: Establish the state space of the deep deterministic policy gradient agent. In some preferred embodiments, this step may further include: based on the goal of achieving energy storage charge state equilibrium and frequency stability in step S2, the state space variables are selected with the first... Taiwan's energy storage state of charge Frequency of grid connection points Output active power Output current The droop coefficient of the active loop in droop control Virtual resistance in virtual impedance control As observations for the agent, the state space is established: .
[0051] Step S4: Establish the action space of the deep deterministic policy gradient agent. In some preferred embodiments, this step may further include: the deep deterministic policy gradient agent acts as a secondary energy storage controller, and to achieve the objective in step S3, it designs action variables as adjustments to the droop coefficient and virtual resistance; then the action space is:
[0052] Based on the system stability analysis results in step S2.3, constraints are set for the action space to ensure that the reinforcement learning policy output meets the dynamic stability requirements of the system and the physical limitations of the device. The action boundaries should satisfy the following conditions: .
[0053] Step S5: Establish the reward function for the deep deterministic policy gradient agent. In some preferred embodiments, this step may further include: the reward function of the deep deterministic policy gradient agent is designed to promote the equilibrium of the state of charge of energy storage and suppress frequency fluctuations. Furthermore, to meet system operational constraints, this invention embeds it into a Lagrangian objective function to construct an unconstrained reward function, enabling reinforcement learning to automatically adjust the penalty intensity and approximate the feasible region boundary during training. Its expression is:
[0054] in, , As a weighting factor, it is used to balance the state-of-charge equilibrium and frequency stability; Let be the Lagrange multipliers for each constraint, where Denotes the Lagrange multipliers for the upper bound constraint. Represents the Lagrange multiplier for the lower bound constraint.
[0055] Step S6: Establish the objective function for the deep deterministic policy gradient agent policy network. In some preferred embodiments, this step may further include: Step S6.1: The above-mentioned reward function is used to guide the value network. The value network is trained with the current state-action pair As input, the function outputs an estimate of the expected long-term return, with the learning objective being to approximate the cumulative discounted reward.
[0056] Step S6.2, perform iterative training using the Bellman equation form:
[0057] in, and These are the target value network and the target strategy network, respectively. The immediate reward is calculated based on the current state and actions. As a discount factor, This represents the state at the next moment.
[0058] Step S6.3: Based on this, the goal of the policy network is to select the optimal action. This maximizes the Q-value of the value network output, thereby indirectly optimizing the reward. Its policy objective function is defined as:
[0059] in, Let it be the expected function; These are the parameters of the policy network; This represents the state distribution sampled in the experience playback buffer.
[0060] Step S7: Establish the loss function of the deep deterministic policy gradient agent value network. In some preferred embodiments, this step may further include: using the value network to estimate state-action pairs. The expected return is calculated, and its loss function is defined as the mean square error between the current Q value and the target Q value, i.e.:
[0061] in, Let it be the expected function; These are the parameters of the current value network.
[0062] Step S8: Establish a soft update mechanism for the target network parameters of the deep deterministic policy gradient agent. In some preferred embodiments, this step may further include: the target policy network and the target value network correspond to the current policy network and the current value network, respectively, and their parameters are iteratively updated through a soft update method to improve training stability. The soft update uses the following recursive formula for smooth parameter replacement:
[0063] in, The soft update coefficient is much less than 1, so it is set to 0.001. These are the parameters for the current policy network and the value network, respectively; These are the parameters for the target policy network and the value network, respectively.
[0064] Step S9: Based on the deep deterministic policy gradient agent and its target network parameter soft update mechanism, establish an energy storage charge state equalization control strategy based on the secondary control of the deep deterministic policy gradient agent. In some preferred embodiments, as shown in Figure 3, this step may further include: Step S9.1: The current policy network obtains the current state from the energy storage environment. And the actions of the previous moment:
[0065] Step S9.2, the current policy network determines the current state. Calculate the action And send it to the system environment as a reference control quantity; in step S9.3, the energy storage system sends the control quantity as a reference. Execute the corresponding control step, run for one time step, and then return to the next state. and rewards And record the quadruple Entering the experience replay pool; Step S9.4, the deep deterministic policy gradient agent randomly samples a batch of data from the experience pool to update the value network and policy network; Step S9.5, the target policy network adjusts according to the next state. Output target action The process involves: 1) Inputting the target value into the target value network to calculate the target value, and constructing the training target by combining the reward; 2) Step S9.6: Minimizing the loss between the target Q-value and the predicted Q-value in the value network, and updating the network parameters through backpropagation; 3) Step S9.7: Maximizing the action value under the current policy in the policy network to optimize policy performance; 4) Step S9.8: Adjusting the target network parameters through a soft update mechanism to improve training stability; 5) Step S9.9: Repeating the above steps until the training satisfies the SOC deviation being less than... And the frequency deviation is less than ,Right now:
[0066] More preferably, The value is 0.001%. The value is 0.005.
[0067] Based on the same inventive concept, one embodiment of the present invention also provides a secondary control system for energy storage charge state balance based on security reinforcement learning.
[0068] Specifically, the energy storage state-of-charge (SOC) balancing secondary control system based on security reinforcement learning provided in this embodiment may include: an energy storage inverter model module, which is used to construct an energy storage inverter model based on droop control and virtual impedance to describe power conversion and grid-connected operation; an agent construction module, which sets the control objectives and constraints of the secondary controller based on the energy storage inverter model to achieve SOC balancing; based on the control objectives and constraints of the secondary controller, establishing the state space, action space, reward function, policy network objective function, and value network loss function of a deep deterministic policy gradient agent, and using the obtained agent as the secondary controller; establishing a soft update mechanism for the target network parameters of the deep deterministic policy gradient agent; and an energy storage SOC balancing control module, which establishes an energy storage SOC balancing control strategy based on the deep deterministic policy gradient agent and its target network parameter soft update mechanism to achieve energy storage SOC balancing control.
[0069] It should be noted that the steps in the method provided by the present invention can be implemented using corresponding modules, devices, units, etc. in the system. Those skilled in the art can refer to the technical solution of the method to realize the composition of the system. That is, the embodiments in the method can be understood as preferred examples for building the system, and will not be elaborated here.
[0070] The energy storage state-of-charge (SOC) balancing secondary control method and system based on secure reinforcement learning provided in the above embodiments of the present invention are applicable to multi-energy storage collaborative control in new energy grid-connected scenarios. This method and system construct a deep deterministic policy gradient agent, generates control actions based on real-time system operating status information, and aims to quickly achieve dynamic SOC balancing among energy storage devices and system frequency stability. The method employs a technical solution that constructs a state space, action space, and reward function, establishes a policy network and a value network, designs an action space constraint mechanism using the Nyquist stability criterion to meet system stability requirements, and obtains control strategies through reinforcement learning interactive training, deploying them in the system to achieve online control. This approach eliminates the need for precise modeling of the secondary controller, possesses good adaptability and generalization capabilities, effectively improves energy storage operating efficiency and safety, and enables adaptive coordinated control of the SOC in various new energy power systems.
[0071] Any matters not covered in the above embodiments of the present invention are well-known in the art.
[0072] The specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.
Claims
1. A secondary control method for energy storage charge state balance based on security reinforcement learning, characterized in that, include: A model of an energy storage inverter based on droop control and virtual impedance is constructed to describe power conversion and grid-connected operation; Based on the energy storage inverter model, the control objectives and constraints of the secondary controller are set to achieve state-of-charge equilibrium. Based on the control objectives and constraints of the secondary controller, the state space, action space, reward function, policy network objective function, and value network loss function of the deep deterministic policy gradient agent are established, and the resulting deep deterministic policy gradient agent is used as the secondary controller. A soft update mechanism for the target network parameters of the deep deterministic policy gradient agent is established. Based on the deep deterministic policy gradient agent and its target network parameter soft update mechanism, an energy storage state-of-charge equilibrium control strategy based on the secondary control of the deep deterministic policy gradient agent is established.
2. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 1, characterized in that, The construction of the energy storage inverter model based on droop control and virtual impedance includes: establishing the inverter main circuit structure, including: energy storage, a three-phase full-bridge DC-AC inverter circuit, an LC filter circuit, and line impedance; wherein, the DC power output from the energy storage is fed into the three-phase full-bridge DC-AC inverter circuit via a DC bus; the three-phase output of the three-phase full-bridge DC-AC inverter circuit passes sequentially through a filter inductor. Measuring the equivalent resistance of the inverter Then connect the filter capacitor This forms an LC filter circuit to suppress inverter switching harmonics; the filter capacitor node then passes through the line resistance of the line impedance. and line inductance The energy storage is connected to a three-phase voltage source in the power grid; the state-of-charge expression of the energy storage is: In the formula, For energy storage exist The state of charge value at time t; For energy storage At the initial moment The electrical state value of the charge; For energy storage The capacity; For energy storage The charge and discharge efficiency; For energy storage exist The system continuously outputs active power; it establishes a primary control loop based on droop control and virtual impedance, including: droop control, virtual impedance control, voltage and current dual closed-loop control, and PWM control; firstly, it collects the voltage and current at the LC filter port, calculates the active and reactive power output of the primary controller, and the droop control generates the voltage outer loop reference voltage amplitude and phase based on the output active and reactive power; secondly, it generates d-axis and q-axis voltages through dq transformation, and superimposes the virtual impedance onto the d-axis and q-axis reference voltages of the voltage outer loop control input; the deviation between the reference voltage and the actual voltage is used by a PI controller to generate the current inner loop d-axis and q-axis reference currents; then, the current inner loop generates a PWM modulation signal through the PI controller; finally, it achieves grid connection after PWM control; the equation expression of the primary control loop based on droop control and virtual impedance is: In the formula, For energy storage exist Reactive power output at all times and Energy storage exist The droop coefficients of the active and reactive power loops in the constant droop control; and Energy storage exist Frequency and voltage at any given moment; and The system's reference frequency and voltage; For virtual resistor The resulting equivalent active power loss; For virtual reactance The resulting equivalent reactive power loss; For energy storage exist Current at any given moment.
3. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 1, characterized in that, The setting of control objectives and constraints for the secondary controller includes: aiming to achieve state-of-charge balance in energy storage, and establishing control objectives for the secondary controller based on minimizing state-of-charge deviation and frequency deviation, respectively. and for: In the formula, For energy storage exist The state of charge value at time t; For energy storage At the initial moment The state of charge value; For energy storage exist The droop coefficient of the active loop in the constant droop control; For energy storage exist Frequency of time; For energy storage exist Virtual resistance in real-time virtual impedance control; For energy storage exist Current at any given moment; For energy storage The capacity; For energy storage The charge and discharge efficiency; For energy storage exist The state of charge value at time t; For energy storage At the initial moment The electrical state value of the charge; For energy storage The capacity; For energy storage The charge and discharge efficiency; Energy storage exist The droop coefficient of the active loop in the constant droop control; Energy storage exist Frequency of time; This is the system's reference frequency; For energy storage exist Virtual resistance in real-time virtual impedance control; For energy storage exist Current at any given moment; This is for frequency deviation; To determine the energy storage capacity, the following constraints are established: In the formula: and These are the upper and lower limits for adjusting the droop coefficient, respectively. and These are the upper and lower limits for adjusting the virtual resistance, respectively. and These are the upper and lower limits of the current, respectively; and These are the upper and lower limits of active power, respectively; and These are the upper and lower limits of the state of charge, respectively; For energy storage exist The active power output at any given time.
4. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 3, characterized in that, 、 、 and The value of is determined based on the energy storage inverter model, a small-signal model and state-space equations are established, and the upper and lower limits of the action space are obtained by analyzing the Nyquist stability criterion.
5. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 1, characterized in that, Establishing the state space of a deep deterministic policy gradient agent includes: selecting the state space of the agent based on the control objective of the secondary controller. Taiwan's energy storage state of charge Frequency of grid connection points Output active power ,load Output current The droop coefficient of the active power loop in droop control. And virtual resistance in virtual impedance control As observations of the agent, a state space is established. , is represented as: Establish the action space of a deep deterministic policy gradient agent, including: designing action variables as adjustment amounts for the droop coefficient and virtual resistance, then the action space... Represented as: In the formula, and These are the adjustment amounts for the droop coefficient and the virtual resistance, respectively; based on the constraints of the secondary controller, the constraint range of the action space is set; the action boundary of this constraint range satisfies the following conditions: In the formula, and These are the upper and lower limits for adjusting the droop coefficient, respectively. and These represent the upper and lower limits for adjusting the virtual resistance, respectively; the reward function of the deep deterministic policy gradient agent is established, including: constructing an unconstrained reward function based on the Lagrangian objective function, thus defining the reward function of the deep deterministic policy gradient agent. , is represented as: In the formula, and The control objective of the secondary controller; For energy storage exist Current at any given moment; For energy storage exist The active power output at all times; For energy storage exist The state of charge value at time t; and These are the upper and lower limits of the current, respectively; and These are the upper and lower limits of active power, respectively; and These are the upper and lower limits of the state of charge, respectively; 、 As a weighting factor, it is used to balance the state-of-charge equilibrium and frequency stability; Let be the Lagrange multipliers for each constraint, where Denotes the Lagrange multipliers for the upper bound constraint. Represents the Lagrange multiplier for the lower bound constraint.
6. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 1, characterized in that, Establish the objective function of the policy network for a deep deterministic policy gradient agent, including: using the reward function of the deep deterministic policy gradient agent to guide the value network. The value network is trained with the current state-action pair As input, the function outputs an estimate of the expected long-term return, with the learning objective being to approximate the cumulative discounted reward. In the formula, The discount factor represents the discount factor for the future th... The discount weight of step rewards; For intelligent agents at any time Based on the immediate reward returned by the environment according to the current state and action; iterative training is performed using the Bellman equation form: In the formula, The target Q value, constructed based on the Bellman equation, is used as the target label during the current value network update. and These are the target value network and the target strategy network, respectively. The immediate reward is calculated based on the current state and actions. Discount factor; The state at the next moment; based on this, select the optimal action. As the objective of the policy network, maximizing the Q-value of the value network output, the objective function of the policy network is... Defined as: In the formula, Let it be the expected function; These are the parameters of the policy network; The state distribution sampled in the experience replay buffer; the value network loss function of the deep deterministic policy gradient agent is established, including: the value network used to estimate state-action pairs. The expected return, its loss function Defined as the mean square error between the current Q value and the target Q value, i.e.: In the formula, Let it be the expected function; These are the parameters of the current value network.
7. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 1, characterized in that, A soft update mechanism for the target network parameters of a deep deterministic policy gradient agent is established, comprising: a target policy network and a target value network corresponding to the current policy network and the current value network, respectively; and the parameters of the target policy network and the target value network are iteratively updated through a soft update method; wherein, the soft update uses the following recursive formula for smooth parameter replacement: In the formula, The soft update coefficient is much less than 1; These are the parameters for the current policy network and the current value network, respectively; These are the parameters for the target policy network and the target value network, respectively.
8. The secondary control method for energy storage state-of-charge balance based on security reinforcement learning according to claim 1, characterized in that, The establishment of the energy storage charge state equalization control strategy based on deep deterministic policy gradient agent secondary control includes: the current policy network obtaining the current state from the energy storage environment. The current policy network calculates the current action based on the current state and sends it as a reference control quantity to the energy storage environment. The energy storage system executes the corresponding control operation based on the issued control quantity, runs for one time step, and then returns to the next state. and rewards And record the quadruple The data is entered into the experience replay pool; the deep deterministic policy gradient agent randomly samples a batch of data from the experience replay pool to update the value network and policy network; the target policy network outputs the target action based on the next state, which is fed into the target value network to calculate the target value, and the training target is constructed by combining the reward; the value network minimizes the loss between the target Q value and the predicted Q value, and updates the network parameters through backpropagation; the policy network maximizes the action value under the current policy to optimize policy performance; the target network parameters are adjusted through a soft update mechanism to improve training stability; the above steps are repeated until the training meets the termination condition.
9. A secondary control system for energy storage charge state balance based on security reinforcement learning, characterized in that, include: The energy storage inverter model module is used to build an energy storage inverter model based on droop control and virtual impedance, which describes power conversion and grid-connected operation. The module comprises: an agent construction module, which sets the control objectives and constraints of the secondary controller based on the energy storage inverter model to achieve state-of-charge (SOC) equilibrium; a state space, action space, and reward function of the deep deterministic policy gradient agent, as well as the policy network objective function and value network loss function, based on the control objectives and constraints of the secondary controller, and uses the resulting agent as the secondary controller; and a soft update mechanism for the target network parameters of the deep deterministic policy gradient agent. The module also includes an energy storage SOC equilibrium control module, which, based on the deep deterministic policy gradient agent and its target network parameter soft update mechanism, establishes an energy storage SOC equilibrium control strategy based on the secondary control of the deep deterministic policy gradient agent to achieve energy storage SOC equilibrium control.