Three-level npc energy storage converter control method based on clf-cbf constraint residual reinforcement learning

Abstract

The application discloses a three-level NPC energy storage converter control method based on CLF-CBF constraint residual reinforcement learning, adopts a three-layer series control architecture: the bottom layer is a bidirectional PI baseline controller configured with direct-current voltage feedforward correction; the middle layer is a residual reinforcement learning correction module, which outputs a bounded residual voltage correction quantity through SOC adaptive action limiting and a multi-target reward function, and compensates modeling errors, power grid disturbances and parameter drift; the top layer is a CLF-CBF-QP safety filtering module, which covers multi-dimensional control barrier function constraints of grid-side electrical safety and battery-side operation safety, and performs minimum amplitude safety correction on the nominal control instruction by solving a quadratic programming problem. The application can significantly improve the current control performance and harmonic suppression effect under weak power grid and wide SOC conditions, and guarantees system operation reliability and battery safety through the layer-by-layer bottom-up architecture, and is suitable for energy storage power station scenes with high proportion of new energy penetration.

Description

Technical Field

[0001] This invention relates to the field of energy storage system control technology, specifically a control method for a three-level NPC energy storage converter in a weak power grid based on CLF-CBF constrained residual reinforcement learning. Background Technology

[0002] With a high proportion of new energy sources being integrated into the power system, the dual randomness on both the power supply and load sides has led to a significant deterioration in the system's inertia support capacity and frequency regulation characteristics. To mitigate power fluctuations, smooth peak and valley loads, and provide emergency power support, the installed capacity of new energy storage technologies, represented by electrochemical energy storage, has experienced explosive growth. As the core interface device between the battery and the grid, the power conversion system (PCS) directly determines the grid-connected power quality, system operational stability, and battery lifespan. In medium- and high-power energy storage power station applications, the three-phase three-level neutral point clamped (NPC) topology has become the mainstream technology due to its advantages of low device voltage stress, superior output harmonic characteristics, and large single-unit capacity.

[0003] In remote areas and end-point distribution networks with high penetration of renewable energy, the power grid generally exhibits weak grid characteristics, characterized by low short-circuit ratio (SCR), high and fluctuating grid impedance, and significant voltage distortion and frequency shift. This poses a severe challenge to the stable operation of NPC-type energy storage converters. Under weak grid conditions, the resonant point of the LCL filter is prone to shift with the grid impedance. At the same time, the inherent midpoint potential balance problem of the NPC topology is further amplified by disturbances, leading to uneven device stress, increased harmonic distortion, and in severe cases, even system instability.

[0004] Currently, the mainstream NPC in the industry generally adopts a fixed-parameter PI current controller. This solution has a simple structure, high reliability, and can achieve good current tracking effect under the nominal operating conditions of a strong power grid. However, it has inherent limitations: First, the PI parameters are tuned for nominal operating conditions. When the grid impedance fluctuates under weak grid conditions and the device parameters drift with temperature, the current tracking accuracy decreases, the total harmonic distortion (THD) increases, and the midpoint potential fluctuation intensifies. It cannot simultaneously take into account harmonic suppression and potential balance. In extreme cases, it may even cause LCL filter resonance, leading to system instability. Second, traditional PI control is mostly designed for unidirectional grid-connected inverters and is not well adapted to the bidirectional charging and discharging operation of the energy storage converter PCS. Distortion and current surges are prone to occur when the charging and discharging current crosses zero, accelerating battery aging. In addition, the control logic only focuses on the grid-connected side electrical indicators and the midpoint potential constraints of the topology itself. It does not incorporate battery-side safety constraints such as the state of charge (SOC) and charge / discharge rate into the real-time control closed loop. It relies on the battery management system's post-trip protection and cannot prevent overcharging, over-discharging, and current surge damage from the control source. For example, authorized patent CN202311176424.9 proposes a control method for a three-level energy storage converter. By establishing output voltage and midpoint voltage prediction models, it predicts and evaluates different switching states and selects the optimal space vector to achieve output control and DC side midpoint potential balance. However, this scheme is essentially still a predictive control based on an accurate model: on the one hand, its control performance is highly dependent on the accuracy of the system model. When the grid impedance fluctuates under weak grid conditions and the device parameters drift with temperature, model mismatch will directly lead to a decrease in current tracking accuracy, an increase in total harmonic distortion, and an aggravation of midpoint potential fluctuations; on the other hand, this scheme does not set up an independent real-time safety constraint layer. The operational safety of the battery side only relies on the ex-post trip protection of the battery management system, and cannot avoid overcharging, over-discharging, and current surge damage from the control source.

[0005] To improve the control performance of inverters under complex operating conditions, current research has begun to introduce reinforcement learning (RL) into the field of power electronic control. This mainly falls into two technical categories: one is a complete replacement approach, where a reinforcement learning controller directly replaces the traditional PI current loop. Some research has explored this approach for NPC topologies. While this approach has strong adaptability, it lacks a safety constraint mechanism. Exploratory actions during training and operation can easily lead to overcurrent, overvoltage, and midpoint potential exceeding limits. Furthermore, it lacks a reliable fallback control loop, resulting in insufficient interpretability and industrial reliability, making it unsuitable for safety-critical energy storage scenarios. The other approach is a gain-scheduling approach, where reinforcement learning is used to adjust the proportional and integral coefficients of the PI controller online. This approach retains the PI control structure, but its performance improvement potential is limited, and the real-time safety constraint problem remains unresolved. For example, authorized patent CN202410397056.9 proposes a low-level control method for modular multilevel converters based on reinforcement learning. This method directly generates low-level control actions through a reinforcement learning agent, and uses a reward function to balance voltage tracking, capacitor voltage balance, and switching losses. However, this solution adopts an architecture that completely replaces the traditional controller with reinforcement learning, without a stable baseline controller to back it up. It lacks interpretability and industrial reliability, and the exploration actions during training and operation are prone to causing overcurrent and overvoltage faults. At the same time, its safety constraints are only soft-guided through the reward function, which does not have the ability to satisfy the constraints deterministically. Furthermore, it has not made special designs for the bidirectional charging and discharging characteristics of the energy storage converter and the safety requirements of the battery side, and cannot be directly adapted to the engineering scenarios of NPC type energy storage converters.

[0006] Existing research on NPC topologies is geared towards unidirectional photovoltaic grid-connected scenarios. Safety constraints only cover electrical indicators on the grid-connected side, completely neglecting the bidirectional operation characteristics of energy storage systems and the operational safety of the battery side. Furthermore, it has not been deeply integrated with residual reinforcement learning architecture and NPC topology constraints, and has not yet formed a complete three-layer control system of "baseline fallback - performance optimization - safety enforcement". It also cannot simultaneously cover the triple objectives of grid-connected electrical safety, topology self-constraints, and battery operational safety.

[0007] In summary, existing technologies cannot simultaneously meet the three core requirements of NPC-type energy storage converters (PCS) under weak power grids: excellent current control and midpoint potential balance performance, reliable grid-connected electrical safety, and stringent battery operation safety. Therefore, there is an urgent need for a control method for NPC-type energy storage converters that is adaptable to bidirectional charge and discharge operation and takes into account the three constraints, in order to overcome the aforementioned deficiencies of existing technologies. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the technical problem this invention aims to solve is to provide a control method for three-level NPC energy storage converters in weak grid scenarios based on CLF-CBF constrained residual reinforcement learning. This method is designed for three-level NPC energy storage converters in weak grid environments and constructs a three-layer series control system consisting of PI baseline fallback, residual reinforcement learning performance optimization, and CLF-CBF-QP safety enforcement. Through decoupling and coordination at the architectural level, it can simultaneously address grid-connected electrical safety, topology constraints, and battery operational safety.

[0009] This invention is achieved using the following technical solution: A control method for a three-level NPC energy storage converter based on CLF-CBF constrained residual reinforcement learning is proposed. This method is applied to a three-phase three-level neutral-point clamped energy storage converter. The DC side of the energy storage converter is connected to a battery energy storage unit with a matching battery management system, and the AC side is connected to the power grid via an LCL filter. A three-layer series control architecture consisting of a PI baseline controller, a residual reinforcement learning correction module, and a CLF-CBF-QP safety filter module is employed. The method includes the following steps: S1. Real-time acquisition of three-phase current, three-phase grid voltage, DC bus voltage, battery state of charge and charge / discharge current, which are converted into electrical components in the synchronous rotating dq coordinate system via phase-locked loop and Park transformation. S2, the PI baseline controller receives d-axis and q-axis current reference commands and real-time feedback electrical components, and outputs baseline voltage control commands to serve as the system stability base to ensure closed-loop stable operation under nominal operating conditions; S3. The residual reinforcement learning correction module collects the full system state of the grid-connected side and the battery side, outputs a bounded residual voltage correction amount and superimposes it on the baseline voltage control command to obtain the nominal control command, in order to compensate for modeling errors, grid disturbances and device parameter drift. S4 and CLF-CBF-QP safety filter modules perform real-time verification of the nominal control command in each control cycle. By solving a quadratic programming problem, with the goal of minimizing the correction amount of the nominal control command, they force all constraints of grid-connected electrical safety and battery-side operational safety to be met, and output the final safety control command. S5. Input the final safety control command into the space vector pulse width modulation module, convert it into a drive signal for the power switching device, and drive the three-phase three-level neutral point clamping energy storage converter to operate.

[0010] This invention also protects a three-level NPC energy storage converter control system based on CLF-CBF constrained residual reinforcement learning that uses the above-described control method.

[0011] Compared with the prior art, the beneficial effects of the present invention are: This method adopts a three-layer serial control architecture, with three core solutions designed: PI baseline controller, residual reinforcement learning correction, and CLF-CBF-QP real-time safety filter. Following the design principle of "decoupling and division of labor, and layer-by-layer protection", the PI layer ensures the basic stability of the system, the RL layer improves the dynamic control performance, and the safety layer strictly adheres to the bottom line of operational safety. The failure of any layer will not lead to system collapse.

[0012] This invention employs a two-stage training process of "unconstrained warm start + constrained fine-tuning." In the first stage, the safety layer is closed, allowing the agent to quickly grasp the basic correction rules. In the second stage, the safety layer is opened, enabling the policy to actively adapt to the safety boundary, effectively solving the engineering challenges of oscillations and slow convergence during training under constraints. Simultaneously, the constraint boundaries of the three modules are unified. The SOC adaptive action limit of reinforcement learning is aligned with the threshold gradient of the CBF safety constraint. The nominal stable interval of the PI baseline serves as the basic feasible region for the CLF and CBF constraints, allowing the three layers of constraints to converge layer by layer from loose to tight, ensuring that a feasible solution always exists for the quadratic programming and avoiding constraint conflicts.

[0013] 1. PI Baseline Controller Module: The PI baseline controller is the core of the entire system's reliability. As the underlying stable base, it receives current reference commands and real-time sampled feedback, and outputs baseline voltage control commands to ensure closed-loop stability under nominal operating conditions.

[0014] This module adopts a bidirectional current control structure in a synchronous dq coordinate system, and is specifically optimized for the wide SOC fluctuation and bidirectional operation characteristics of energy storage converters: the grid voltage phase is obtained through a phase-locked loop, and the sampled three-phase grid current and three-phase grid voltage are converted into DC quantities in a synchronous rotating dq coordinate system through Park transformation. The d-axis current reference is positive during discharge and negative during charging, and bidirectional power control can be achieved through a single current loop; a proportional-integral control structure is adopted to achieve current tracking without steady-state error, and a dq-axis cross-coupling compensation term is added to cancel inter-axis coupling. A grid voltage feedforward term is introduced to accelerate the grid disturbance response speed. The integrator supports bidirectional saturation limiting and is adapted to bidirectional tracking without steady-state error for both positive and negative currents.

[0015] To address the characteristic of battery terminal voltage fluctuating over a wide range with SOC, DC voltage feedforward correction is introduced on the original PI output to offset the impact of DC voltage changes on PWM modulation gain, ensuring consistent equivalent gain of the modulation loop under different SOCs and DC voltage levels, and improving control consistency over a wide SOC range.

[0016] 2. Residual Reinforcement Learning Correction Module: This module serves as a performance optimization layer. It collects the full system state data from both the grid-connected and battery sides, and outputs a bounded residual voltage correction value, which is superimposed on the PI baseline command to obtain the nominal control command. This command is used to compensate for modeling errors, grid disturbances, and parameter drift, thereby improving current tracking accuracy and harmonic suppression. The module employs a near-end strategy optimization algorithm for training, and its core design is as follows: State space design: Nine physically measurable system variables are selected to form a state vector, covering d / q axis current tracking error, measured value of inverter-side inductor current, measured value of grid voltage, battery state of charge, real-time DC bus voltage, and active power tracking error. All state variables are normalized and mapped to the [-1,1] interval based on nominal values, so that the agent can simultaneously perceive complete operating information of the grid-connected side and the battery side.

[0017] Action space design: The agent outputs the d-axis and q-axis residual voltage correction values ​​and directly superimposes them onto the PI baseline output; it adopts a SOC adaptive dynamic limiting mechanism, which retains full correction authority in the nominal SOC working range, narrows the charging direction limit and widens the discharging direction limit in the high SOC range, and narrows the discharging direction limit and widens the charging direction limit in the low SOC range, thereby reducing the risk of battery overshooting from the action generation level.

[0018] Multi-objective reward function: Based on the principle of "prioritizing current accuracy, moderate intervention, and taking into account battery safety", it is composed of a weighted component of current tracking error penalty, residual action amplitude penalty, action smoothness penalty, power tracking error penalty, battery current change rate penalty, and SOC boundary risk penalty. All coefficients are normalized to balance the weights of each optimization objective.

[0019] Two-stage training strategy: The training process adopts unconstrained hot start and constrained fine-tuning. In the first stage, basic operating condition training is carried out with the safety layer closed within the nominal operating range of SOC, so that the agent can quickly master the basic correction rules. In the second stage, the full safety layer is opened, and the strategy is fine-tuned under extreme and complex operating conditions such as the entire SOC range, weak grid impedance fluctuations, and device parameter drift, so as to balance training efficiency and final operational safety.

[0020] 3. CLF-CBF-QP Real-time Safety Filtering Module: This module serves as a top-level independent safety defense line. It does not depend on the quality of PI parameters or reinforcement learning strategies. It performs real-time verification of the nominal control command in each control cycle. By solving a small-scale quadratic programming problem, it forces all safety constraints to be satisfied under the premise of minimizing the correction amount of the nominal control command, and outputs the final safety control command.

[0021] CLF stability constraints: Based on the dq-axis current tracking error, a quadratic control Lyapunov function is constructed to force the current error to converge exponentially, and relaxation variables are configured to avoid constraint conflicts under extreme operating conditions. This mathematically guarantees the current tracking stability of the closed-loop system and avoids oscillation instability caused by weak grid disturbances or residual correction actions.

[0022] CBF safety constraints: Based on the forward invariance principle of the control barrier function, five types of safety constraints are constructed for grid-connected electrical safety and battery-side operational safety: grid-connected side includes grid-connected current amplitude constraints, DC bus voltage safety constraints that change dynamically with SOC, and NPC midpoint potential balance constraints; battery side includes SOC safety domain constraints and charge / discharge rate constraints, ensuring that the system state is always within the safe range from the control source.

[0023] QP optimization solution: Taking minimizing the deviation between the final control command and the nominal control command as the objective function, the CLF stability constraint and all CBF safety constraints are transformed into linear inequality constraints, and a standard convex quadratic programming problem is constructed for real-time solution; the optimal solution obtained is the final safety control command, which is converted into a switching drive signal by the space vector pulse width modulation module to drive the power switching devices of the NPC converter.

[0024] This invention considers the bidirectional operation characteristics of energy storage systems and the operational safety of the battery side. The three types of constraints—grid-connected electrical systems, NPC topology, and battery safety—are highly coupled, making it impossible to achieve strict hard constraint satisfaction solely through reward functions. Fixed action limits cannot match the safety boundaries of dynamically changing SOC, leading to a difficult balance between performance and safety. This invention creatively applies Residual Reinforcement Learning (RRL), commonly used in slow-dynamic scenarios such as power outer loops and distribution network voltage regulation, to fast inner-loop current control (where the inner-loop control cycle is short, real-time requirements are extremely high, and conventional reinforcement learning inference delays are difficult to adapt to; power electronic systems are strongly coupled and highly nonlinear, making residual correction prone to triggering LCL resonance and disrupting closed-loop stability; strategy training convergence is difficult in fast-dynamic scenarios, and oscillations are likely to occur). Furthermore, it integrates CLF-CBF-QP safety filtering and a PI baseline controller to form an overall collaborative architecture. Performance optimization is achieved by superimposing a small residual correction on the baseline controller, balancing the stability of baseline control with the adaptive capabilities of reinforcement learning. By selecting low-dimensional physical measurable state variables and employing a lightweight fully connected network, the inference delay is controlled within the control cycle. A PI controller is used as a stable baseline to limit the magnitude of the residual correction, preserving the baseline stability fallback capability. A safety filter layer with CLF stability constraints is added, and the CLF-CBF-QP safety layer transforms multiple constraints into deterministic linear hard constraints, forming a dual-layer protection of "soft guidance + hard interception" with the reward function. This balances performance optimization and safety baseline, mathematically ensuring current error convergence, suppressing oscillation risks, and directly adapting to the engineering requirements of NPC-type energy storage converters.

[0025] Quadratic programming combining the control barrier function (CBF) and the control Lyapunov function (CLF). The QP (Quick Programming) framework enables deterministic constraint satisfaction and stability guarantees at the control end. Given the bidirectional charging and discharging of the energy storage converter, wide-range SOC fluctuations, and dynamic time-varying safety constraints, a dynamic CBF constraint is constructed to address the energy storage characteristics. The DC voltage and SOC safety domain are designed to adapt to changing operating conditions. Relaxation variables are configured for the CLF constraint to avoid solution infeasibility due to multiple constraint conflicts. It adapts to fast dynamic real-time requirements by transforming all stability and safety constraints into linear inequalities, constructing a small-scale convex quadratic programming problem, and using the effective set method for rapid solution, meeting the real-time requirements of short control cycles in the inner loop. A three-layer serial architecture achieves deep integration: CLF-CBF-QP serve as an independent post-safety layer, using minimal intervention as the correction principle, preserving the performance optimization benefits of RL while providing deterministic safety fallback. Combined with a two-stage training strategy, the RL strategy actively adapts to safety constraints, reducing ineffective corrections, forming a complete three-layer control system of "baseline fallback - performance optimization - safety enforcement," simultaneously covering the triple objectives of grid-connected electrical safety, topology self-constraints, and battery operation safety.

[0026] This application targets a three-level NPC energy storage converter for weak grid scenarios. It represents the first time that each module has been customized and improved specifically for energy storage scenarios, addressing three key technical issues of existing solutions: insufficient adaptability, lack of safety constraints, and incompatibility with bidirectional energy storage operation and battery safety. For bidirectional charging and discharging and wide SOC operation of the energy storage converter: a current loop structure supporting bidirectional saturation limiting is designed to adapt to bidirectional zero-steady-state error tracking of positive and negative currents, improving distortion during zero-crossing switching of charging and discharging currents; a new DC voltage feedforward correction stage is added to offset the impact of DC bus voltage changes caused by battery SOC fluctuations on PWM modulation gain, improving control consistency across the entire SOC range. This improvement resolves the performance degradation problem when traditional PI control is directly applied to energy storage PCS, demonstrating innovation in scenario adaptation. In residual reinforcement learning, the agent simultaneously perceives operational information from both the grid-connected side and the battery side, overcoming the limitation of existing solutions that only focus on grid-connected electrical quantities. It actively adapts to battery safety boundaries at the action generation level, achieving differentiated control that maintains performance within the nominal range and reduces risk within the boundary range. This application is the first to deeply integrate CLF-CBF-QP with the residual reinforcement learning architecture and apply it to NPC energy storage converter control. At the converter current loop control level, it simultaneously covers five types of safety constraints across three dimensions: grid-connected electrical safety, NPC topology self-constraints, and battery-side operational safety, overcoming the limitation of existing solutions that only cover grid-connected indicators. It constructs a DC bus voltage safety domain that dynamically changes with SOC, replacing traditional fixed threshold constraints, better aligning with the battery's charge-discharge voltage-current characteristics, and avoiding false limits and leakage protection. Adhering to the principle of "minimal intervention," it is used as an independent back-end safety layer, forming a dual safety protection of reward-based soft guidance and constraint-based hard interception with the front-end residual reinforcement learning. This ensures the deterministic satisfaction of safety constraints while maximizing the performance optimization benefits of reinforcement learning, addressing the pain point of insufficient safety reliability in existing reinforcement learning control. The three-layer architecture follows the design principles of decoupling and division of labor, and layer-by-layer fallback. The failure of any layer will not lead to system collapse, significantly improving the operational reliability in industrial scenarios. At the same time, the three layers work together to achieve the optimal balance of stability, performance and security, resulting in a technical effect of 1+1+1>3. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the main circuit topology of the three-phase three-level neutral point clamping energy storage converter involved in this invention; Figure 2 This is a schematic diagram of the overall principle of the three-layer serial control architecture involved in this invention; Figure 3 This is a schematic diagram comparing the total harmonic distortion rate of grid-connected current under various typical operating conditions involved in this invention. Figure 4 This is a schematic diagram comparing the d-axis and q-axis current tracking waveforms under the power step dynamic condition involved in this invention. Figure 5This is a schematic diagram comparing the total harmonic distortion rate of grid-connected current in different battery state-of-charge ranges involved in this invention. Figure 6 This is a schematic diagram illustrating the operating characteristics of the CLF-CBF-QP safety filter module under extreme combined stress conditions as described in this invention. Detailed Implementation

[0028] The present invention will be further explained below with reference to the embodiments and accompanying drawings, but this is not intended to limit the scope of protection of this application.

[0029] Figure 1 The diagram illustrates the connection topology of a three-phase three-level NPC inverter bridge, an LCL filter, and the power grid. The DC side employs a series-connected capacitor structure to form the DC bus, providing tiered DC voltage support for the inverter bridge. Each phase arm of the three-phase three-level neutral-point clamped inverter bridge consists of four power switches with anti-parallel diodes and two clamping diodes, achieving three-level voltage output through the switching combinations of the switches. The three-phase AC output terminals of the inverter bridge are connected to an LCL filter, which consists of an inverter-side filter inductor L, a star-connected filter capacitor C, and a grid-side filter inductor L, used to filter out switching subharmonics. The filter's grid-side port Eg is connected to the three-phase AC power grid, enabling bidirectional power interaction between the energy storage converter and the grid.

[0030] Figure 2 The diagram illustrates the complete signal flow and internal composition of each module in the three-layer serial control architecture of this invention. The top layer is the reference input and feedback sampling stage, which receives externally issued d-axis and q-axis current reference commands, collects real-time grid-connected current through feedback sensors, and calculates the current tracking error. The first layer is the PI baseline controller module, which incorporates a bidirectional PI current control loop and a DC voltage feedforward correction stage, outputting a baseline voltage control command as the system's stability base. The second layer is the residual reinforcement learning correction module, which includes state space design, action space design, a multi-objective reward function, a PPO policy optimization algorithm, an Actor-Critic network architecture, and a two-stage training strategy. It outputs a bounded residual voltage correction amount, which is superimposed on the PI baseline command to obtain the nominal control command. The third layer is the CLF-CBF-QP real-time safety filtering module, which incorporates CLF closed-loop stability constraints, grid-connected side electrical safety constraints, and battery-side operational safety constraints. It performs real-time safety verification and correction of the nominal control command through quadratic programming optimization, outputting the final safety control command.

[0031] The controlled object of this invention is a three-phase three-level neutral-point clamping energy storage converter. Its DC side is connected to a battery energy storage unit (with a BMS system, which can collect the SOC, battery terminal voltage, charging and discharging current and other statuses in real time), and its AC side is connected to a weak grid through an LCL filter. The system adopts a grid-following operation mode and achieves phase synchronization with the grid through a phase-locked loop (PLL).

[0032] This invention adopts a three-layer serial control architecture, consisting of a PI baseline controller, a residual reinforcement learning correction module, and a CLF-CBF-QP security filter module, from bottom to top. The overall signal flow is as follows: The PI baseline controller serves as the system stability base, receiving current reference commands and real-time sampled feedback, and outputting baseline voltage control commands to ensure closed-loop stability under nominal operating conditions. The residual reinforcement learning module collects the full system state from the grid-connected side and the battery side, outputs a bounded residual voltage correction, and superimposes it onto the PI baseline command to obtain the nominal control command. This command is used to compensate for modeling errors, grid disturbances, and device parameter drift, improving current tracking accuracy and harmonic suppression. The CLF-CBF-QP safety filter module serves as an independent safety defense line, performing real-time verification of the aforementioned nominal control command in each control cycle. By solving a small-scale quadratic programming problem, it forces all grid-connected side and battery side safety constraints to be satisfied while minimizing the correction amount of the nominal control command, and outputs the final safety control command. The final control command is converted into drive signals for the power switching devices via the space vector pulse width modulation (SVPWM) module, which then drives the power switching devices of the NPC converter to operate.

[0033] Example 1 A control method for a three-level NPC energy storage converter based on CLF-CBF constrained residual reinforcement learning is proposed. This method is applied to a three-phase three-level neutral-point clamped energy storage converter. The DC side of the energy storage converter is connected to a battery storage unit of a matching battery management system, and the AC side is connected to the power grid via an LCL filter. A three-layer series control architecture consisting of a PI baseline controller, a residual reinforcement learning correction module, and a CLF-CBF-QP safety filter module is employed. The method includes the following steps: Step 1: System-wide status acquisition and coordinate transformation Execution content: Real-time acquisition of three-phase grid-connected current, three-phase grid voltage, DC bus voltage, and midpoint potential offset on the grid-connected side via sensors; interactive acquisition of battery state of charge (SOC) and charging / discharging current via the battery management system; acquisition of grid voltage phase via phase-locked loop; conversion of three-phase AC quantities into DC quantities in a synchronous rotating dq coordinate system via Park transformation; and calculation of current tracking error.

[0034] Input sources: hardware sampling circuit, BMS communication data Output data: dq-axis current component, dq-axis grid voltage component, DC bus voltage, midpoint potential offset, battery SOC, current tracking error, power tracking error Downstream applications: The output full state quantity is simultaneously supplied to the PI baseline controller in step 2, the residual reinforcement learning module in step 3, and the CLF-CBF-QP safety filter module in step 4, serving as the unified data foundation for all control links.

[0035] The perception and standardized conversion of the full-dimensional operating status of the grid-connected side, topology side, and battery side are completed, providing a consistent and processable state input for subsequent control calculations. This is the perception foundation of the entire control closed loop and fully achieves the step objectives of "state acquisition and preprocessing".

[0036] Step 2: The PI baseline controller generates baseline voltage commands. Execution content: Receive the dq axis current reference command issued externally, combine it with the real-time current feedback output in step 1, and calculate it through a bidirectional proportional-integral control loop; superimpose the dq axis cross-coupling compensation term and the grid voltage feedforward term to offset the inter-axis coupling and grid disturbance; introduce DC voltage feedforward correction to offset the impact of SOC fluctuation on modulation gain; configure the integrator with bidirectional saturation limiting, and finally output the baseline voltage control command.

[0037] Input source: External current reference command + dq-axis current, mains voltage, and DC bus voltage output from step 1. Output data: dq axis baseline voltage control command Downstream application: As a stable base input in step 3, it is superimposed with the residual correction to generate the nominal control command.

[0038] It achieves zero steady-state error tracking and closed-loop stability of the current under nominal operating conditions, providing a reliable fallback control capability for the entire system. Even if all subsequent correction and safety modules fail, the output of this step can still independently support the basic grid-connected operation of the system, fully achieving the hierarchical goal of "baseline stability fallback".

[0039] Step 3: Residual reinforcement learning correction generates nominal control instructions Execution content: The 9-dimensional normalized system state quantity output from step 1 is input into the trained PPO policy network, and the dq axis residual voltage correction quantity is output. After being constrained by the SOC adaptive dynamic limiting mechanism, the correction quantity is directly superimposed on the baseline voltage command output from step 2 to obtain the nominal control command.

[0040] Input source: Full system state output from step 1 + Baseline voltage control command output from step 2 Output data: DQ axis nominal control voltage command Downstream application: As an optimization instruction to be verified, it is input into the security filtering module in step 4 for compliance verification.

[0041] Without compromising the stability of the PI baseline, adaptive compensation is performed for unmodeled dynamics, grid impedance fluctuations, and device parameter drift, improving current tracking accuracy and harmonic suppression under weak grid conditions. At the same time, SOC adaptive limiting reduces the risk of battery overshooting from the action generation end, fully achieving the level goal of "performance optimization and upgrade".

[0042] Step 4: CLF-CBF-QP security filtering generates final security control commands. Execution content: Using the nominal control command output in step 3 as the optimization benchmark, and combining the real-time state variables output in step 1, the CLF current stability constraint, the three electrical safety constraints on the grid-connected side, and the two operational safety constraints on the battery side are uniformly transformed into linear inequality constraints; a convex quadratic programming problem is constructed, and the solution is performed in real time with the goal of "minimizing the deviation between the final command and the nominal control command", and the final safety control command is output.

[0043] Input source: Nominal control command output from step 3 + status variables such as current error, current amplitude, DC voltage, midpoint potential, SOC, and current change rate output from step 1. Output data: DQ axis final safety control voltage command Downstream application: Input the modulation module in step 5 to convert it into a physical drive signal.

[0044] While preserving the performance of RL optimization to the maximum extent, it enforces closed-loop stability and all-dimensional operational safety through deterministic mathematical constraints, and makes only minimal corrections to the nominal control commands. It is the last independent safety line of defense for the entire system, and fully achieves the hierarchical goal of "mandatory safety fallback".

[0045] Example 2 The core design principle of the three-layer architecture in this embodiment is decoupling and division of labor, with each layer providing a safety net: the PI layer ensures basic stability, the RL layer improves dynamic performance, and the security layer safeguards the bottom line of security. Failure of any layer will not cause system collapse; that is, even if both the RL and security layers fail, the PI controller can still operate independently, ensuring the system's basic grid-connected functionality. When the RL policy is abnormal, the security layer can directly correct the action to a safe range, ensuring industrial-grade reliability from an architectural perspective.

[0046] 1. PI Baseline Controller Module: The PI baseline controller is the core of the entire system's reliability. It adopts a bidirectional current control structure under the synchronous dq coordinate system to address the wide SOC fluctuation and bidirectional operation characteristics of the energy storage converter PCS, and can independently achieve stable current control under all operating conditions.

[0047] Phase lock loop (PLL) is used to obtain the grid voltage phase. i The sampled three-phase grid-connected current i a , i b , i c Three-phase power grid voltage v a , v b , v c Converted to DC quantities in a synchronously rotating dq coordinate system via Park transformation: d-axis and q-axis current components i d , i q d-axis and q-axis grid voltage components v d , v q .

[0048] When the energy storage converter is discharging, the d-axis current reference i dref It is a positive value; during charging, i dref When the value is negative, bidirectional power control can be achieved through a single current loop. The current tracking error is defined as:

[0049] A proportional-integral (PI) control structure is employed to achieve zero steady-state error current tracking. Simultaneously, a dq-axis cross-coupling compensation term is added to counteract inter-axis coupling in the synchronous rotating coordinate system, thereby improving dynamic response speed. The controller output expression is:

[0050] In the formula, K p , K i These are the proportional and integral coefficients, respectively. v d , v q It serves as a grid voltage feedforward term, used to accelerate the response speed under grid disturbances; oh The angular frequency of the power grid. L For inverter-side filter inductance; ωLi q , ωLi d The cross-coupling compensation term is used to decouple dq axis control; the integrator supports bidirectional saturation limiting and can be adapted to bidirectional zero steady-state error tracking of positive and negative currents.

[0051] To address the wide fluctuation of battery terminal voltage with SOC, a DC voltage feedforward correction is introduced on the original PI output to offset the impact of DC voltage variations on PWM modulation gain.

[0052] in V dcn The rated DC bus voltage, V dcr This is the DC bus voltage sampled in real time.

[0053] This correction ensures that the equivalent gain of the modulation loop remains consistent under different SOC and DC voltage levels, avoiding control performance degradation caused by DC voltage fluctuations, improving control consistency over a wide SOC range, and ensuring underlying security.

[0054] The DC voltage feedforward correction in this application is designed to address the wide-range fluctuations in the battery terminal voltage of the energy storage converter with varying State of Charge (SOC). Its target is the wide-range quasi-steady-state variation of the DC bus voltage caused by changes in battery SOC, rather than short-term transient fluctuations. The aim is to offset the impact of DC voltage level variations on the PWM modulation gain, ensuring that the equivalent gain of the modulation loop remains consistent across different SOCs and DC voltage steady-state operating points. This improves the consistency of current control performance over a wide SOC range and prevents the PI control effect from deteriorating when the SOC deviates from the nominal value. Simultaneously, this correction, combined with a bidirectional PI current loop design, can adapt to both charging and discharging operation, maintaining stable modulation gain even when both positive and negative currents flow in both directions.

[0055] 2. Residual Reinforcement Learning Correction Module: This module does not replace the PI controller; it only outputs a small residual voltage correction amount superimposed on the PI baseline output. This enables adaptive compensation for unmodeled dynamics, grid impedance fluctuations, and nonlinear coupling, improving current control performance and battery operation friendliness under weak grid conditions without compromising baseline stability. The module uses the Proximal Policy Optimization (PPO) algorithm for training and includes five core design components: state space, action space, reward function, network architecture, and a two-stage training process.

[0056] The state vector is composed entirely of physically measurable system variables, covering the grid-connected electrical state and the battery-side operating state, totaling 9 dimensions, ensuring that the agent obtains complete operational information.

[0057] In the formula, i de , i qe The d-axis and q-axis current tracking errors reflect the degree of deviation in current tracking. i d , iq These are the measured values ​​of the inductor current on the d-axis and q-axis inverter sides, reflecting the current operating point of the system. v d , v q The values ​​represent the measured grid voltages along the d and q axes, reflecting the grid disturbance state; SOC represents the battery state of charge. V dc This is the real-time DC bus voltage, reflecting the battery voltage fluctuation status. P ref The active power tracking error reflects the accuracy of power command tracking.

[0058] All state variables are normalized based on nominal values ​​and mapped to the [-1,1] interval to improve the numerical stability and convergence speed of neural network training. The first 6 dimensions are general states for the grid-connected side, and the last 3 dimensions are energy storage-specific states, enabling the agent to perceive the battery operating boundary and actively adapt to energy storage operation requirements.

[0059] Action space design: The agent's output is the residual voltage correction amount along the d and q axes.

[0060] This correction value is directly added to the PI baseline output to obtain the nominal control command:

[0061] The superimposed nominal control command is directly input into the subsequent CLF-CBF-QP real-time safety filtering module as a reference nominal value for the quadratic programming solution. In the objective function of the safety filtering module, this nominal control command serves as the optimization benchmark. The solution process aims to minimize the deviation between the final safety control command and this nominal control command. Under the premise of forcibly satisfying all CLF stability constraints and CBF safety constraints, it retains as much of the performance optimization effect brought by the PI baseline and residual correction as possible. The final safety control command obtained after verification and correction by the safety filtering module is then sent to the space vector pulse width modulation module to be converted into a switching drive signal, completing the output of the entire control link.

[0062] To address the battery safety requirements of energy storage, this invention employs a SOC adaptive dynamic limiting mechanism to replace the traditional fixed limiting mechanism. When the SOC ∈ [20%, 80%] within the nominal operating range, the limit is ±0.15 pu, preserving full calibration authority and prioritizing control performance. In the high SOC range (SOC > 90%), the charging direction limit is narrowed to ±0.08 pu, while the discharging direction limit is widened to ±0.18 pu, actively limiting aggressive calibration in the charging direction. In the low SOC range (SOC < 10%), the discharging direction limit is narrowed to ±0.08 pu, while the charging direction limit is widened to ±0.18 pu, actively limiting aggressive calibration in the discharging direction. Here, "pu" represents the per-unit value.

[0063] This mechanism reduces the risk of battery overshooting from the action generation level, forming a dual protection of "prevention before the event + interception after the event" with the back-end CBF safety layer. At the same time, it retains full permissions within the nominal range without sacrificing the performance optimization capability under normal operating conditions.

[0064] Multi-objective reward function: The reward function is designed around the principles of "prioritizing current accuracy, moderate intervention, and considering battery safety," and consists of 6 weighted factors. The reward value rt for each control step is: Where rt is the reward value. a、b、l、g、m、n These represent the weighting coefficients for the current tracking error penalty, residual action amplitude penalty, action smoothness penalty, power tracking error penalty, battery current change rate penalty, and SOC boundary risk penalty, respectively. i de , i qe These represent the current tracking errors along the d and q axes, respectively. These are the residual voltage corrections for the d and q axes, respectively; Δ i d The single-step change in the d-axis current; SOC risk The degree to which the SOC deviates from the nominal range; LPF(•) is a first-order low-pass filter operator; P ref This refers to the active power tracking error.

[0065] The physical meanings and functions of each item are as follows: Current tracking error penalty term (coefficient) α =1.0): The square of the penalty current tracking deviation is the core term for improving the accuracy of current control, driving the agent to reduce steady-state error and transient deviation.

[0066] Residual motion amplitude penalty term (coefficient) β =0.1): The magnitude of the penalty residual correction is increased to guide the agent to minimize intervention, maintain the dominance of the PI controller, and avoid aggressive actions that could cause system instability.

[0067] Action smoothness penalty (coefficient) l =0.05): penalizes high-frequency abrupt changes in residual actions, where LPF(•) is a first-order low-pass filter operator to ensure smooth control commands and avoid exciting LCL filter resonance and switching harmonics.

[0068] Power tracking error penalty term (coefficient) c =0.2): penalize active power tracking deviation, adapt to application scenarios of energy storage participating in grid frequency regulation and power dispatch, and improve the response accuracy and tracking speed of power commands.

[0069] Battery current change rate penalty term (coefficient) m =0.15): Δ i d It represents the single-step change in d-axis current, penalizing rapid changes in current to prevent large current surges from damaging the battery electrodes and extending battery cycle life.

[0070] SOC Boundary Risk Penalty Item (Coefficient) n =0.3): SOC risk The value represents the degree to which the SOC deviates from the nominal range. When the SOC is in the range of [20%, 80%], the value is 0. The closer the SOC is to the upper or lower limit, the larger the value becomes, guiding the agent to actively move away from the SOC safety boundary.

[0071] All coefficients have been normalized to balance the weights of each optimization objective and prevent any single factor from dominating the learning strategy.

[0072] The network architecture employs a dual-network structure of Actor and Critic. Both networks share an input state vector but are structurally independent, being fully connected feedforward neural networks. The Actor network takes a 9-dimensional customized state vector as input, which passes through two hidden layers of 256 and 128 neurons respectively, using the ReLU activation function. The final output is the Gaussian distribution mean of the residual actions on the d and q axes. The action standard deviation is designed with independent trainable parameters to support exploration during the training phase. The output is mapped to the SOC adaptive dynamic limiting range using the Tanh activation function. The Critic network maintains the same input dimension and hidden layer structure as the Actor network, and its output is the expected discounted reward scalar value for the corresponding state, used to calculate the generalized dominance function and guide policy network updates. The combination of a 9-dimensional customized input dimension, dual-branch output adapted to dq-axis synchronous rotating coordinate system control, and the dynamic limiting mechanism ensures inference speed and control accuracy in fast-dynamic scenarios. Using a 9-dimensional physical measurable state vector that integrates grid-connected and battery-side information as input, rather than a single grid-connected electrical quantity input as in general scenarios, the intelligent agent can perceive energy storage-specific operating information such as battery state of charge and DC bus voltage fluctuations, and actively adapt to bidirectional charging and discharging characteristics and battery safety boundary requirements.

[0073] Key training hyperparameters were all referenced to typical engineering values ​​for the PPO algorithm in the continuous control field, and were specifically adapted for this scenario. The specific configuration is as follows: For the learning rate, both the Actor network and the Critic network use 3×10⁻⁶. -4 The initial value is set, and a linear decay strategy is used to gradually reduce the learning rate in the later stages of training to improve policy stability. This tuning logic is designed for fast dynamic control scenarios in power electronics, avoiding training oscillations caused by an excessively large learning rate and slow convergence caused by an excessively small learning rate. The discount factor is set to 0.99, which is used to calculate the long-term discount reward and is a common value for continuous control tasks. The generalized dominance estimation parameter is set to 0.95, which can balance the bias and variance of the dominance estimation and adapt to the fast dynamic and multi-disturbance operating characteristics of this scenario. The batch size is set to 2048; the number of update rounds per batch is set to 10; and the pruning coefficient is set to... A value of 0.2 is used to limit the magnitude of a single policy update, preventing system instability caused by sudden policy changes. The entropy coefficient is initially set to 0.01, linearly decaying to 0.001 during training. This encourages extensive exploration in the early stages to avoid getting trapped in local optima, while reducing the entropy weight later to improve policy stability. The action standard deviation is initially set to 0.1 per unit, gradually decaying during training to ensure sufficient exploration capability in the early stages, and for actions to become more stable after policy convergence. These parameter settings, while ensuring nonlinear fitting capability, compress the single-step inference latency to an acceptable range within a single control cycle, avoiding the inference timeout problem caused by the large number of layers and parameters in general deep networks, thus adapting to industrial-grade real-time control requirements.

[0074] The residual reinforcement learning correction module adopts a two-stage training strategy: in the first stage of unconstrained warm start, a relatively high initial learning rate and entropy coefficient are used to accelerate convergence; in the second stage of constrained fine-tuning, the learning rate and exploration range are reduced to adapt to safety constraints. This is a key setting to ensure that the final performance and safety of this solution are both achieved. The above network structure and hyperparameters are typical feasible implementations in this scenario. The specific values ​​can be fine-tuned according to the actual hardware platform and operating conditions to fully meet the feasibility requirements of the patent.

[0075] A dedicated Proximal Policy Optimization (PPO) algorithm is designed for the fast dynamic control scenario within the inner loop of a three-level NPC energy storage converter. The residual policy is trained using PPO, which demonstrates stable and robust training in continuous control tasks. By pruning the target, the algorithm limits the policy update magnitude, preventing system instability caused by sudden policy changes.

[0076] The PPO trimming and substitution objective function is:

[0077] in: i These are the trainable parameters of the policy network; The ratio of the probability of actions under the new and old strategies; A t The generalized advantage estimation (GAE) method is used to estimate the advantage function. This is the pruning factor, typically 0.2, used to limit the magnitude of a single policy update and ensure stability during training.

[0078] The total loss function consists of three parts: policy pruning loss, value function regression loss, and entropy regularization term. The entropy regularization term is used to encourage sufficient exploration in the early stages of training and to prevent the policy from converging to a local optimum too early.

[0079] Actor Network (Policy Network): Input is a 9-dimensional state vector. s t The data is sequentially processed through two hidden layers (256 neurons / 128 neurons, ReLU activation function), ultimately outputting the Gaussian distribution mean of the residual actions on the d and q axes. The action standard deviation is parameterized separately for exploration during the training phase. The output is mapped to the action limiting range using the Tanh activation function to generate the final residual correction. The Actor network directly outputs the residual voltage correction in the synchronously rotated dq coordinate system, which is fully aligned with the output dimension and physical meaning of the PI baseline controller, allowing for direct superposition to complete the correction without additional signal conversion. Simultaneously, the Tanh activation function at the output end is deeply integrated with the SOC adaptive dynamic limiting mechanism proposed in this invention, replacing the fixed limiting logic in general solutions. This enables differentiated action range constraints in different SOC intervals, retaining full correction authority under nominal operating conditions and actively limiting the action amplitude in risk directions within boundary intervals, balancing control performance and battery operation safety.

[0080] Critic Network (Value Network): Input is a 9-dimensional state vector s t After passing through two hidden layers of the same size as the Actor, the expected discount reward scalar value for that state is output. V ( s t This is used to evaluate the quality of a state, calculate the advantage function, and guide the policy network update.

[0081] The Critic network adopts an independent fully connected structure of the same size as the Actor, decoupling the value evaluation and policy update functions. Combined with a two-stage training process, it can accelerate policy convergence in the unconstrained warm start stage and stabilize the policy update process in the constrained fine-tuning stage, thus better adapting to the training needs of reinforcement learning under safety constraints.

[0082] This invention employs a two-stage training process of "unconstrained hot start + constrained fine-tuning," with all training scenarios designed specifically for energy storage operating conditions. This addresses the engineering problems of difficult, slow-converging, and volatile reinforcement learning training with safety constraints. In this embodiment, the specific process of the two-stage training is as follows: Phase 1: Unconstrained Warm Startup Phase (150,000 steps in total) With the CLF-CBF safety layer disabled, training is conducted within the nominal operating range of 20% to 80% of SOC, covering basic operating conditions such as constant current charging, constant current discharging, and small-amplitude power steps. This stage allows the agent to explore freely in an unconstrained environment, quickly master the basic current correction rules, converge to a better performance level, and avoid the low learning efficiency caused by safety constraints.

[0083] Phase Two: Fine-tuning with Safety Constraints (120,000 steps in total) The complete CLF-CBF safety layer is enabled, expanding the training scenarios to the full SOC range (10%~90%), weak grid impedance fluctuations (SCR3~10), sensor noise, device parameter drift, and other extreme composite conditions. (SCR=3~10 uses uniform random sampling within the range, with the corresponding short-circuit ratio randomly selected in each training round, covering all operating conditions from extremely weak to strong grids; the device parameter drift ratio is set to a random fluctuation of ±20% of the nominal values ​​of key parameters such as filter inductors and capacitors, simulating temperature drift and aging deviations; sensor noise uses Gaussian white noise with an amplitude of ±1% of the nominal sampling value to match the noise level of the actual sampling circuit.) This stage allows the agent to continue optimizing the strategy under safety constraints, adapting to all energy storage operation scenarios, ensuring that the final strategy has both excellent performance and always complies with safety rules in actual deployment.

[0084] The two-stage training ensures training efficiency and the final performance ceiling, while also guaranteeing operational security after deployment, effectively solving the practical obstacle of difficulty in convergence of reinforcement learning under security constraints.

[0085] 3. CLF-CBF-QP Real-time Safety Filtering Module: This module is the last independent safety defense line of the entire control architecture. It does not depend on the quality of PI parameters and reinforcement learning (RL). By transforming stability requirements and safety constraints into linear inequalities, it solves small-scale quadratic programming problems, achieves deterministic constraint satisfaction in each control cycle, and provides safety guarantees that do not depend on the learned strategy.

[0086] The core design principle is the principle of minimal intervention: when the nominal control command itself satisfies all constraints, the safety layer does not make any modifications and fully preserves the performance of RL optimization; only when the nominal control command has a risk of violation will the nominal control command be modified to the minimum extent to bring it back to the safe and feasible domain, thus achieving a balance between "safety first and performance guaranteed".

[0087] CLF stability constraints are used to construct a quadratic control Lyapunov function based on the dq-axis current tracking error, which mathematically guarantees the current tracking stability of the closed-loop system.

[0088] in e =[ i de , i qe ] T Let be the current tracking error vector. This function is positive definite with respect to the error and radially unbounded, satisfying the basic requirements of a Lyapunov function.

[0089] To ensure the convergence of the current error exponent, the following CLF constraint condition must be met:

[0090] in: c >0 represents the desired convergence rate coefficient, which can be tuned according to the dynamic response requirements; d These are slack variables used to avoid QP infeasibility caused by constraint conflicts under extreme conditions, thus ensuring the robustness of the solution.

[0091] This constraint guarantees that, in the absence of constraint conflicts, the current tracking error will converge to zero at an exponential rate, and the system will always remain closed-loop stable.

[0092] CBF safety constraints, tailored to the operational characteristics of energy storage PCS, construct five categories of safety constraints: grid-connected side electrical safety and battery side operational safety, all encoded as control obstacle functions. Utilizing the forward invariance principle of CBF: for the safety set... S ={ x | h ( x If )≥0}, if satisfying ( k (where >0 is the constraint strictness coefficient), then the set S It has forward invariance, meaning that as long as the initial state is in the safe zone, the state will never go beyond the safe boundary during the entire process.

[0093] Grid-connected side electrical safety constraints (3 items): (1) Grid-connected current amplitude constraint The combined amplitude of the grid-connected current is limited to the maximum allowable value of the power devices to avoid overcurrent damage. The barrier function is defined as:

[0094] in i max This represents the maximum allowable current amplitude.

[0095] (2) DC bus voltage safety constraints To address the characteristic of DC voltage fluctuations in energy storage with State of Charge (SOC), a voltage safety domain that dynamically changes with SOC is constructed. Compared to a fixed voltage threshold, this better reflects the actual operating characteristics of the battery, avoiding false limiting or leakage protection. The barrier function is defined as:

[0096] in V dcmax (SOC) V dcmin (SOC) represents the upper and lower limits of the allowable DC bus voltage under the corresponding SOC, which are determined by the battery charge and discharge voltage-current characteristics.

[0097] (3) Potential balance constraint at the midpoint of NPC To ensure voltage balance between the upper and lower DC capacitors in a three-level NPC converter, it is crucial to prevent uneven device stress and harmonic degradation caused by midpoint potential shift. The barrier function is defined as:

[0098] Where Δ V np Δ represents the real-time offset of the midpoint potential. V npmax This is the maximum allowed offset.

[0099] Battery-side operational safety constraints (2 items): (1) SOC security domain constraints To prevent overcharging and over-discharging at the control level, thus protecting battery life and safety, two barrier functions are constructed: upper and lower limits.

[0100] SOC max SOC min These are the upper and lower limits of the battery's permissible SOC (typical values ​​are 90% and 10%).

[0101] The direction and amplitude of the charging and discharging current are constrained by the CBF condition: when the SOC is close to the upper limit, the charging current amplitude is automatically limited; when the SOC is close to the lower limit, the discharging current amplitude is automatically limited, ensuring that the SOC is always in a safe range from the control source, without waiting for the BMS to trip.

[0102] (2) Charge / discharge rate constraint Limiting the rate of change of battery current avoids current surges during power jumps, thereby extending battery cycle life. The barrier function is defined as:

[0103] in yesmax The maximum allowable rate of current change within a single control step is determined by the battery's maximum charge / discharge rate and the cell's tolerance characteristics. This constraint ensures a smooth current change and avoids large fluctuations. yes / dt Impact damage to battery electrodes.

[0104] QP optimization solution: In each control cycle, the above CLF stability constraints and all CBF safety constraints are transformed into constraints on the control input. u dq Given the linear inequality constraints, a standard quadratic programming problem is constructed for real-time solution. The entire optimization problem follows the principle of minimal intervention, preserving the performance optimization effects brought by the PI baseline and residual RL module as much as possible while strictly satisfying all stability and security constraints.

[0105] (1) Objective function The optimization objective is to minimize the deviation between the final control command and the nominal control command, which can be expressed mathematically as follows:

[0106] In the formula: The final d-axis and q-axis safety control voltage vectors to be solved; u dqn The nominal control voltage vector is the sum of the PI baseline and the residual RL output; ∥•∥ 2 The square of the vector's second norm is equivalent to the sum of the squares of the voltage deviations of the two paths, d and q.

[0107] The design logic of the objective function is as follows: when the nominal control command itself satisfies all constraints, the optimal solution is completely consistent with the nominal control command, the safety layer does not intervene in any way, and the current tracking accuracy improvement and harmonic suppression effect brought about by reinforcement learning optimization are fully preserved; only when there is a risk of violation in the nominal control command will the nominal control command be modified with the minimum magnitude to minimize the performance loss brought about by the safety layer.

[0108] (2) Constraints Constraints are divided into two categories: closed-loop stability constraints and safety boundary constraints, which together constitute the feasible region of the optimization problem.

[0109] ① CLF closed-loop stability constraints

[0110] In the formula: V ( e () is the control Lyapunov function constructed based on the current tracking error; This is the time derivative of the Lyapunov function; c This is the CLF convergence rate coefficient, used to set the desired convergence rate of the current error; d These are slack variables used to avoid unsolvable optimization problems caused by constraint conflicts under extreme conditions, thus ensuring the robustness of the solution.

[0111] The physical meaning of this constraint is to ensure that the charging and discharging current tracking error is always in an exponentially convergent state, keeping the system closed-loop stable and avoiding oscillations and instability caused by weak grid disturbances or residual correction actions. Under ideal conditions without constraint conflicts, the current error will increase at a rate... c It continues to decay until it reaches zero.

[0112] ② CBF safety boundary constraints

[0113] In the formula: h k ( x ) is the first k Each control barrier function corresponds to a runtime safety constraint. Let be the time derivative of the barrier function; k k For the first k The convergence coefficient of a constraint is used to control the strictness of the constraint enforcement.

[0114] This constraint is based on the forward invariance principle of the control barrier function: as long as the initial state of the system is within the safe set ( h k ( x If the system state is greater than or equal to 0 and always satisfies the above inequality, then the system state will always remain within the safe set during the entire operation process and will not exceed the safe boundary.

[0115] This invention sets a total of 5 safety constraints, among which k =1,2,3 correspond to the electrical safety constraints on the grid-connected side (current amplitude, DC bus voltage, NPC midpoint potential). k =4,5 corresponds to the battery-side operational safety constraints (SOC safety domain, charge / discharge rate) specific to energy storage.

[0116] (3) Solution and Output The aforementioned quadratic programming problem is a standard convex optimization problem with a unique global optimal solution, which can be solved in a very short time using the effective set method. The optimal solution obtained is the final safety control command, which is directly sent to the space vector pulse width modulation (SVPWM) module to generate the switching transistor drive signal.

[0117] Figure 3The diagram uses a bar chart to compare three control schemes: traditional PI control, PI+residual reinforcement learning control, and PI+residual RL+CLF-CBF-QP control. The total harmonic distortion (THD) of the grid-connected current is compared under eight typical energy storage operating conditions: ideal operation, high temperature and high power, extreme temperature, weak grid, extremely weak grid, combined stress, power transient, and sensor noise. The vertical axis represents the THD, and the horizontal axis represents different operating condition categories. This visually demonstrates that the proposed solution significantly reduces the current harmonic distortion level under most operating conditions. While the harmonic index slightly increases under extreme conditions such as extremely weak grids due to safety constraints, the system maintains stable operation, achieving a balance between performance and safety. The proposed PI+residual RL+CLF-CBF-QP control is universally applicable to extreme conditions, especially under extremely weak grid conditions. In these cases, PI+residual reinforcement learning control performs worse than traditional PI control, while the proposed PI+residual RL+CLF-CBF-QP control significantly enhances the performance of PI+residual reinforcement learning control.

[0118] Figure 4 The diagram is divided into two sub-figures: the left figure shows the d-axis current tracking waveform, and the right figure shows the q-axis current tracking waveform. It compares the dynamic response processes of four control curves—reference command, traditional PI control, PI + residual reinforcement learning control, and PI + residual RL + CLF-CBF-QP control—under active power step conditions. The vertical axis represents the current amplitude, and the horizontal axis represents the running time. This visually demonstrates how the proposed solution effectively shortens the current settling time, reduces current overshoot, suppresses coupling oscillations between the d and q axes, and significantly improves the response speed and stability of current tracking during power step processes.

[0119] Figure 5 The graph compares three control schemes—traditional PI control, PI + residual reinforcement learning control, and PI + residual RL + CLF-CBF-QP control—within the entire operating range of battery state of charge (SOC), using a line graph to illustrate the changing trend of the total harmonic distortion (THD) of the grid-connected current. The vertical axis represents the THD of the grid-connected current, and the horizontal axis represents the battery SOC. This demonstrates the U-shaped fluctuation characteristic of traditional PI control, where harmonic performance increases significantly as SOC deviates from the nominal range, while the proposed scheme effectively narrows the performance fluctuation amplitude across the entire SOC range, improving control consistency and power quality stability over a wide SOC operating range.

[0120] Figure 6The graph is divided into six sub-graphs, arranged in three rows and two columns, corresponding to the control waveforms on the d-axis and q-axis, respectively. The first row shows the nominal residual command waveform of RL before QP filtering, indicating the per-unit peak range of the nominal action. The second row shows the actual safe action waveform after QP safety filtering, indicating the per-unit peak value of the final output action. The third row shows the intervention correction waveform of the safety filter, distinguishing between QP-increased and QP-decreased correction directions, and indicating the QP activation rate and maximum intervention value. The vertical axis represents the control command in per-unit form, and the horizontal axis represents the running time. This visually demonstrates the real-time correction effect of the safety filtering module on the nominal control command, verifying that the safety layer can retain the RL optimization performance to the maximum extent while constraining the control action within the safe and feasible domain.

[0121] Conventional unidirectional grid-connected inverters' PI current loops only need to adapt to the integral saturation limiting of the unidirectional current. When directly migrated to bidirectional charging and discharging scenarios in energy storage, the problem of untimely integral saturation reset during the zero-crossing switching phase of the charging and discharging current is prone to occur, causing current distortion and impact. Moreover, the zero-steady-state error tracking of bidirectional current places higher demands on the integrator limiting logic and anti-saturation design. On the other hand, the parameter matching between DC voltage feedforward correction and the current loop is more difficult. Changes in battery SOC will cause the DC bus voltage to fluctuate over a wide range. If the feedforward gain is not properly matched with the PI parameters of the current loop, it will introduce additional disturbances into the control loop. In scenarios with weak grid impedance fluctuations, DC voltage fluctuations and grid-side disturbances are prone to coupled oscillations, further exacerbating the difficulty of parameter tuning. At the same time, the PI parameters and feedforward coefficients tuned under nominal operating conditions will degrade in performance after the SOC deviates from the nominal range, making it difficult to maintain a consistent control effect across the entire SOC operating range. This application adapts bidirectional current tracking to bidirectional current tracking by configuring the integrator with bidirectional saturation limiting logic. At the same time, it adopts a feedforward correction method with rated DC voltage normalization to ensure consistent equivalent gain of the modulation loop. In addition, it uses a subsequent residual reinforcement learning module to compensate for parameter mismatch errors under all operating conditions, effectively solving the above difficulties and realizing the coupling adaptation of bidirectional operating characteristics and wide-range voltage fluctuations.

[0122] In this application, CLF-CBF-QP serves as an independent top-level safety defense line in a three-layer control architecture. Its core function is to perform minimal safety corrections on the nominal control commands output by the front end in each control cycle. It covers multiple objectives, including closed-loop stability, grid-connected electrical safety, NPC topology constraints, and battery operation safety. It can provide deterministic safety guarantees that do not rely on reinforcement learning strategies, while maximizing the performance optimization benefits brought by residual reinforcement learning. It is a customized application for the inner-loop control scenario of NPC-type energy storage converters, meeting the real-time requirements of fast dynamic systems. It avoids the problem in existing technologies where excessive safety correction can completely negate the performance optimization effect of reinforcement learning, while insufficient correction can not provide a reliable safety backup, making it difficult to balance the relationship between performance preservation and safety constraints.

[0123] This invention adheres to the design principles of decoupling, division of labor, and layered fallback, clearly defining the functional boundaries of the three-layer module. The PI layer is fully responsible for basic stability, the residual reinforcement learning layer only outputs small-amplitude corrections without overturning the baseline control logic, and the safety layer strictly follows the principle of "minimal intervention," only correcting illegal instructions. Each layer performs its function without interfering with the others, and the failure of any layer will not cause system collapse, ensuring operational reliability from an architectural perspective. The reinforcement learning module uses a 9-dimensional low-dimensional measurable state paired with a lightweight fully connected network, compressing inference latency to the microsecond level to meet the real-time requirements of fast dynamics within the inner loop. Simultaneously, all safety and stability constraints are transformed into a small-scale convex quadratic programming problem, solved quickly using the effective set method, with the overall computation completed within a single control cycle.

[0124] Any aspects not covered in this invention are applicable to existing technologies.

Claims

1. A control method for a three-phase, three-level, NPC energy storage converter based on CLF-CBF constrained residual reinforcement learning, applied to a three-phase, three-level, neutral-point clamped energy storage converter, wherein the DC side of the energy storage converter is connected to a battery energy storage unit with a matching battery management system, and the AC side is connected to the power grid via an LCL filter, characterized in that... The method employs a three-layer serial control architecture consisting of a PI baseline controller, a residual reinforcement learning correction module, and a CLF-CBF-QP security filter module, and includes the following steps: S1. Real-time acquisition of three-phase current, three-phase grid voltage, DC bus voltage, battery state of charge and charge / discharge current, which are converted into electrical components in the synchronous rotating dq coordinate system via phase-locked loop and Park transformation. S2, the PI baseline controller receives d-axis and q-axis current reference commands and real-time feedback electrical components, and outputs baseline voltage control commands to serve as the system stability base to ensure closed-loop stable operation under nominal operating conditions; S3. The residual reinforcement learning correction module collects the full system state of the grid-connected side and the battery side, outputs a bounded residual voltage correction amount and superimposes it on the baseline voltage control command to obtain the nominal control command, in order to compensate for modeling errors, grid disturbances and device parameter drift. S4 and CLF-CBF-QP safety filter modules perform real-time verification of the nominal control command in each control cycle. By solving a quadratic programming problem, with the goal of minimizing the correction amount of the nominal control command, they force all constraints of grid-connected electrical safety and battery-side operational safety to be met, and output the final safety control command. S5. Input the final safety control command into the space vector pulse width modulation module, convert it into a drive signal for the power switching device, and drive the three-phase three-level neutral point clamping energy storage converter to operate.

2. The control method according to claim 1, characterized in that, In step S2, the PI baseline controller adopts a bidirectional current control structure in the synchronous dq coordinate system: it uses proportional-integral control to achieve current tracking without steady-state error, adds a dq axis cross-coupling compensation term to cancel inter-axis coupling, introduces a grid voltage feedforward term to improve the disturbance response speed, and configures the integrator with bidirectional saturation limiting to adapt to bidirectional charging and discharging operation of positive and negative currents; at the same time, it introduces DC voltage feedforward correction on the basis of the original PI output to cancel the influence of DC bus voltage fluctuation with battery state of charge on PWM modulation gain.

3. The control method according to claim 1, characterized in that, In step S3, the residual reinforcement learning correction module is trained using a near-end policy optimization algorithm. Its state space contains 9 physically measurable system variables, specifically: d-axis current tracking error, q-axis current tracking error, d-axis inverter-side inductor current, q-axis inverter-side inductor current, d-axis grid voltage, q-axis grid voltage, battery state of charge, real-time DC bus voltage, and active power tracking error. All state variables are normalized to the [-1,1] interval based on nominal values. The residual reinforcement learning correction module outputs d-axis and q-axis residual voltage correction values ​​in its action space, employing a battery state-of-charge adaptive dynamic limiting mechanism. When the battery's state of charge is within the nominal operating range, the residual correction is limited across the entire range. When the battery is in the high SOC range, the residual correction limit in the charging direction is narrowed, and the residual correction limit in the discharging direction is widened. When the battery is in the low SOC range, the residual correction limit in the discharge direction is narrowed, and the residual correction limit in the charging direction is widened. The reward function of the residual reinforcement learning correction module is composed of a weighted sum of current tracking error penalty, residual action amplitude penalty, action smoothness penalty, power tracking error penalty, battery current change rate penalty, and SOC boundary risk penalty. The weight coefficients of each penalty are normalized. The residual reinforcement learning correction module adopts a two-stage training process: The first stage is the unconstrained hot start stage. The CLF-CBF-QP safety filter module is turned off, and training is carried out within the SOC nominal working range, covering constant current charging and discharging and power step basic operating conditions, so that the agent can quickly master the basic current correction law. The second stage is the constrained fine-tuning stage, which activates the CLF-CBF-QP safety filtering module and expands the training scenario to a complex working condition involving the entire SOC range, weak grid impedance fluctuations, sensor noise, and device parameter drift, thereby completing the safety constraint adaptation and performance optimization of the strategy.

4. The control method according to claim 3, characterized in that, In the SOC adaptive dynamic limiting mechanism, the limiting range is ±0.15pu in the full range; in the high SOC range (SOC>90%), the limiting is narrowed to ±0.08pu in the charging direction and widened to ±0.18pu in the discharging direction; in the low SOC range (SOC<10%), the limiting is narrowed to ±0.08pu in the discharging direction and widened to ±0.18pu in the charging direction; pu represents the per-unit value. The residual reinforcement learning correction module employs an Actor-Critic dual-network structure. The Actor network takes a 9-dimensional customized state vector as input, which passes through two hidden layers of 256 and 128 neurons respectively, and is activated by the ReLU function. The final output is the Gaussian distribution mean of the residual action on the d and q axes. The action standard deviation uses an independently trainable parameter. The output is mapped to the range of SOC adaptive dynamic limiting through the Tanh activation function; the input dimension and hidden layer structure of the Critic network are the same as those of the Actor network, and the output is the expected discounted reward scalar value of the corresponding state.

5. The control method according to claim 3, characterized in that, The reward function is: ， Where rt is the reward value. α, β, λ, γ, μ, ν These represent the weighting coefficients for the current tracking error penalty, residual action amplitude penalty, action smoothness penalty, power tracking error penalty, battery current change rate penalty, and SOC boundary risk penalty, respectively. i de , i qe These represent the current tracking errors along the d and q axes, respectively. These are the residual voltage corrections for the d and q axes, respectively; Δ i d The single-step change in the d-axis current; SOC risk The degree to which the SOC deviates from the nominal range; LPF(•) is a first-order low-pass filter operator; P ref This refers to the active power tracking error.

6. The control method according to claim 1, characterized in that, In step S4, the CLF stability constraint of the CLF-CBF-QP safety filter module is specifically as follows: a quadratic control Lyapunov function is constructed based on the dq axis current tracking error to force the current tracking error to converge exponentially, and slack variables are configured to avoid the infeasibility of the solution caused by constraint conflicts under extreme working conditions. CBF safety constraints include three types of grid-connected side electrical safety constraints: Grid-connected current amplitude constraint, limiting the combined amplitude of grid-connected current to not exceed the maximum allowable value of power devices; DC bus voltage safety constraints are implemented to construct a voltage safety domain that dynamically changes with the battery's state of charge, matching the battery's charge-discharge voltage-current characteristics. The NPC midpoint potential balance constraint limits the real-time offset of the midpoint potential of the three-level converter to no more than the maximum allowable offset. CBF safety constraints include two types of battery-side operational safety constraints: SOC safety domain constraints are constructed by establishing upper and lower SOC barrier functions. By constraining the direction and amplitude of the charging and discharging current, overcharging and over-discharging of the battery are prevented from the control level. Charge / discharge rate constraints limit the rate of current change within a single control step, suppressing current surges during power jumps.

7. The control method according to claim 6, characterized in that, Barrier function for grid-connected current amplitude constraint for: ； in i max The maximum allowable current amplitude; i d , i q These are the measured values ​​of the inductor current on the d-axis and q-axis inverter sides. Barrier function for DC bus voltage safety constraints for: ， in V dcmax (SOC) V dcmin (SOC) represents the upper and lower limits of the DC bus voltage under the corresponding SOC; V dc This is the real-time DC bus voltage; Barrier function for NPC midpoint potential balance constraint for: ， Where Δ V np Δ represents the real-time offset of the midpoint potential. V npmax The maximum allowable offset; The two barrier functions for the upper and lower bounds of the SOC safety region constraint are as follows: ， SOC max SOC min These are the upper and lower limits of the battery's permissible SOC; Represents the upper limit barrier function; The lower limit barrier function is represented; SOC is the battery state of charge. Barrier function for charge / discharge rate constraint for: ， in di max t represents the maximum allowable rate of change of current within a single control step; t represents time. x For variables.

8. The control method according to claim 1, characterized in that, In step S4, the quadratic programming problem takes minimizing the L2 deviation between the final safety control command and the nominal control command as the objective function. The CLF stability constraint and all CBF safety constraints are transformed into linear inequality constraints about the control input. The globally optimal safety control command is obtained in real time by using the effective set method. When the nominal control command itself satisfies all constraints, the optimal solution is completely consistent with the nominal control command. The safety layer does not intervene in any way, and fully retains the current tracking accuracy improvement and harmonic suppression effect brought about by reinforcement learning optimization. Only when there is a risk of violation in the nominal control command should the nominal control command be modified to the minimum extent possible, so as to minimize the performance loss caused by the safety layer.

9. A three-level NPC energy storage converter control system based on CLF-CBF constrained residual reinforcement learning, characterized in that, The system uses the control method described in any one of claims 1-8.

Images