Reinforced learning-power hardware-in-loop training system and method for power converter

Abstract

The invention discloses a reinforcement learning-power hardware-in-the-loop training system and method for a power converter, and belongs to the technical field of power electronic control and artificial intelligence crossing. The system comprises a real-time digital simulator, a power amplifier, a controlled power converter and an AI accelerator. The method comprises the steps of S1, system initialization and environment configuration; s2, performing safe online training; and S3, strategy convergence and solidification. Through the power hardware-in-the-loop architecture, the risk of reinforcement learning exploration is completely avoided, and hardware damage caused by overvoltage and overcurrent is fundamentally eradicated in combination with a real-time hardware protection strategy. According to the method, the real object power converter controller is incorporated into a training closed loop, so that an intelligent agent strategy directly learns and adapts to the nonlinear dynamic state of real hardware, the gap from simulation to reality is thoroughly filled up, and it is ensured that a training result can be directly applied to an actual system. According to the invention, a full-real-time training system is constructed, and synchronous concurrent processing of data acquisition, strategy execution and model updating is realized.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of power electronics control and artificial intelligence, specifically relating to a reinforcement learning-power hardware-in-the-loop training system and method for power converters. Background Technology

[0002] Power electronic converters are facing increasingly complex operating conditions, such as grid background harmonics, unknown load characteristics, and time-varying system parameters, which places extremely high demands on the adaptability and robustness of controllers. Online reinforcement learning control, as a data-driven intelligent control method, can autonomously learn and optimize control strategies through continuous interaction with the environment, without relying on a precise mathematical model of the controlled object, demonstrating great potential in handling such complex nonlinear control problems. However, applying online reinforcement learning to power electronic converter control faces two major challenges:

[0003] First, there are the risks and costs of conducting exploratory learning directly on a fully physical platform. Reinforcement learning agents inevitably perform numerous suboptimal or random exploratory actions during the initial training phase. If learning is conducted directly on a real power converter and its connected power grid or load, these exploratory actions are highly likely to cause overvoltage and overcurrent faults, leading to hardware damage or system outages. This poses an extremely high risk and is not engineering feasible.

[0004] Second, there is a performance gap between pure digital simulation and real-world applications. To avoid the aforementioned risks, most current research is limited to pure digital simulation environments. However, simulation models struggle to accurately reproduce all nonlinear characteristics of actual power hardware (such as switching device dynamics, parasitic parameters, and measurement noise). This often leads to performance degradation or even instability when control strategies trained in simulation are transferred to real-world applications, resulting in a significant "simulation-to-reality" gap. Some studies have attempted to narrow this gap through high-precision modeling, but the model identification cost is high, and accurate modeling across the entire domain is difficult to achieve for nonlinear and time-varying systems, thus raising practicality issues.

[0005] Therefore, there is an urgent need in this field for a system architecture and training method that can balance training security and model fidelity, so as to provide a training environment for reinforcement learning agents that can both explore safely and reflect the dynamics of real hardware with high accuracy, thereby effectively bridging the above gap and promoting the practical application of reinforcement learning in power electronic control. Summary of the Invention

[0006] To address the security risks of online reinforcement learning training and the performance gap between simulation and reality in pure digital simulation, this invention provides a reinforcement learning-power hardware-in-the-loop training system and method for power converters. This system provides a high-fidelity training environment for the reinforcement learning controller of power converters that can safely explore and incorporate real hardware dynamics.

[0007] The technical solution adopted in this invention is:

[0008] A reinforcement learning-power hardware-in-the-loop training system for power converters, including

[0009] Real-time digital simulator: performs high-precision simulation of the external dynamic environment that the power converter needs to connect to in practical applications;

[0010] Power amplifier: Receives signals from the real-time digital simulator and amplifies them to the actual voltage and current levels required by the controlled power converter. At the same time, it can monitor the electrical status of the power converter port in real time and provide feedback for system closed-loop control.

[0011] Controlled power converter: includes main power circuit and controller to be trained. The main power circuit interacts with the power amplifier in real energy, and the controller is responsible for running the agent decision-making algorithm and driving the main circuit.

[0012] AI Accelerator: Runs reinforcement learning agents to autonomously learn the optimal control strategy.

[0013] Furthermore, the power amplifier and the real-time digital simulator communicate bidirectionally through an AD / DA data conversion unit. The real-time digital simulator converts the environmental conditions it calculates from digital command signals into analog signals through the AD / DA data conversion unit and sends them to the power amplifier. At the same time, the power amplifier feeds back the voltage and current signals of the controlled power converter port it samples to the real-time simulator through the AD / DA data conversion unit as input for its next calculation.

[0014] Furthermore, the main power amplifier circuit of the power amplifier and the power circuit of the controlled power converter perform real power level energy interaction.

[0015] Furthermore, the controller of the controlled power converter and the AI ​​accelerator exchange data bidirectionally via the TCP / IP high-speed communication protocol. The controlled power converter receives the strategy signals sent by the AI ​​accelerator and simultaneously feeds back the electrical state of the main circuit and the agent reward to the AI ​​accelerator.

[0016] Furthermore, the system implements a security strategy consisting of real-time hardware protection and secure agent guidance.

[0017] Furthermore, the real-time hardware protection strategy is as follows: In the real-time digital simulator, the maximum safe voltage and current thresholds allowed at the port of the controlled power converter are preset. Within each real-time simulation step, the real-time digital simulator compares its port voltage and current with the maximum safe voltage and current thresholds. When the port voltage and current exceed the maximum safe voltage and current thresholds, the real-time digital simulator will actively clamp and only output the maximum safe voltage and current thresholds. The voltage and current at the port of the controlled power converter are always limited to a safe range through the power amplifier.

[0018] Furthermore, the agent's safety guidance strategy includes an overvoltage and overcurrent penalty term added to the design of the reinforcement learning reward function. When the port voltage and current approach the maximum safe voltage and current threshold, the agent will receive a negative reward.

[0019] A method for reinforcement learning-power hardware-in-the-loop training system for power converters, the specific process of which is as follows:

[0020] S1. System initialization and environment configuration;

[0021] S2. Secure online training;

[0022] S3. Strategy convergence and solidification.

[0023] Furthermore, S1 includes the following steps:

[0024] S11. Load the mathematical model into the real-time digital simulator and set the safe voltage and current thresholds;

[0025] S12. The AI ​​accelerator initializes the reinforcement learning agent and constructs the master policy network and master value network;

[0026] S13. The master strategy network is synchronized to the controller of the controlled power converter.

[0027] Furthermore, S2 is a real-time training loop. Training is completed and the loop ends when the control performance of the agent converges and stabilizes.

[0028] Includes the following steps:

[0029] S21. The controlled power converter acquires state variables s t Calculate and execute action variable a t ;

[0030] S22. In the next control cycle, the controller acquires the new state s. t+1 To calculate the reward r t ;

[0031] S23. Translate the empirical tuple (s t , a t, r t , s t+1 Experiences uploaded to the AI ​​accelerator's replay pool;

[0032] S24. The AI ​​accelerator randomly extracts batches of data from the experience replay pool;

[0033] S25. Execute the backpropagation algorithm to update the network parameters;

[0034] The S26.AI accelerator will synchronize the updated master strategy network parameters to the controlled power converter.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] 1. Safe and reliable training: This invention completely avoids the risks of reinforcement learning exploration through a power hardware-in-the-loop architecture. Combined with real-time hardware protection strategies, it fundamentally eliminates hardware damage caused by overvoltage and overcurrent, providing an absolutely safe training environment for online learning.

[0037] 2. High model fidelity: By incorporating the physical power converter controller into the training closed loop, this invention enables the agent strategy to directly learn and adapt to the nonlinear dynamics of real hardware, completely bridging the gap between simulation and reality, and ensuring that the training results can be directly applied to the actual system.

[0038] 3. Highly Efficient Training Process: This invention constructs a fully real-time training system that achieves synchronous and concurrent processing of data acquisition, policy execution, and model updates. This ensures that the time scale of the training environment is consistent with that of the real-world scenario, greatly accelerating policy convergence and efficiently producing control policies that can directly address actual dynamic working conditions. Attached Figure Description

[0039] Figure 1 This is a system block diagram of the present invention;

[0040] Figure 2 This is a flowchart of the method of the present invention. Detailed Implementation

[0041] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0042] This invention is a system architecture and training method that integrates online reinforcement learning and power hardware-in-the-loop simulation for safe and efficient training of power converter controllers.

[0043] 1. For example Figure 1As shown, this invention provides a power hardware-in-the-loop simulation training system for online reinforcement learning of power converters. This system integrates a real-time digital simulator, a power amplifier, and an AI accelerator to build a high-fidelity and high-security training environment, aiming to solve the hardware risks and performance gap between simulation and reality in traditional training methods.

[0044] The system consists of the following four core components:

[0045] Real-time digital simulator: includes a real-time processor, AD / DA data conversion unit, and circuit mathematical model;

[0046] This module uses a real-time processor to perform high-precision simulations of the external dynamic environment that the power converter needs to connect to in practical applications. The complex operating conditions simulated include, but are not limited to, grid background harmonics, voltage sags, frequency fluctuations, load step changes, and nonlinear load characteristics. These scenarios provide a rich and diverse training environment for reinforcement learning agents.

[0047] Power amplifier: includes main power amplifier, AD / DA data conversion unit, sensor, and signal conditioning module;

[0048] The primary function of the power amplifier is to receive signals from a real-time digital simulator and amplify them through the main power amplifier circuit to the actual voltage and current levels required by the controlled power converter. Simultaneously, its built-in high-precision sensors and signal conditioning circuits can monitor the electrical status of the power converter ports in real time, providing feedback for system closed-loop control.

[0049] Controlled power converter: includes main power circuit, controller to be trained, and Ethernet interface module;

[0050] The main power circuit interacts with the power amplifier in real energy, and its controller is responsible for running the agent decision-making algorithm and driving the main circuit.

[0051] AI Accelerator: Includes AI computing unit, reinforcement learning agent, and Ethernet interface module;

[0052] The AI ​​accelerator runs a reinforcement learning agent that autonomously learns the optimal control strategy. During training, it continuously optimizes controller performance to adapt to various dynamic operating conditions.

[0053] The architecture creates a training environment with real hardware by placing the high-risk external environment in a real-time digital simulator, separating the core algorithm to be trained from the physical controller, and using a power amplifier as a bridge.

[0054] 2. To ensure close collaboration within the above system architecture, the components within the system form a closed-loop training environment through the following interaction logic:

[0055] Interaction between the real-time digital simulator and the power amplifier: The power amplifier and the real-time digital simulator communicate bidirectionally via an AD / DA conversion unit. The real-time digital simulator converts its calculated environmental conditions (such as grid voltage) from digital command signals to analog signals via the DA conversion unit and sends them to the power amplifier. Simultaneously, the power amplifier feeds back the voltage and current signals sampled at the ports of the controlled power converter to the real-time simulator via the DA conversion unit, serving as input for its next calculation.

[0056] Interaction between power amplifier and controlled power converter: The main power amplifier circuit of the power amplifier and the power circuit of the controlled power converter perform real power level energy interaction.

[0057] Interaction between the controlled power converter and the AI ​​accelerator: The controller of the controlled power converter and the AI ​​accelerator exchange data bidirectionally via the TCP / IP high-speed communication protocol. The controlled power converter receives policy signals sent by the AI ​​accelerator, and simultaneously feeds back the electrical state of the main circuit and the agent's reward to the AI ​​accelerator.

[0058] 3. To achieve secure training of the above system architecture, this invention implements a security strategy consisting of real-time hardware protection and secure guidance of intelligent agents.

[0059] Real-time hardware protection strategy: In the real-time digital simulator, the maximum safe voltage and current thresholds allowed at the controlled power converter ports are preset. Within each real-time simulation step, the real-time digital simulator compares its port voltage and current with the maximum safe voltage and current thresholds. When the port voltage and current exceed the maximum safe voltage and current thresholds, the real-time digital simulator actively clamps the voltage and current, outputting only the maximum safe voltage and current thresholds. This ensures that the voltage and current at the controlled power converter ports are always limited within a safe range through power amplifiers. This strategy, through real-time monitoring and active intervention, constructs a safety boundary for the training environment at the physical level.

[0060] The agent's safety guidance strategy incorporates overvoltage and overcurrent penalties into the reinforcement learning reward function. When the port voltage and current approach the maximum safe voltage and current threshold, the agent receives a negative reward. Through this negative feedback, the agent avoids actions that could lead to overvoltage and overcurrent. This strategy guides the agent at the algorithmic level, enabling it to proactively avoid risks.

[0061] 4. Based on the above system architecture, interaction logic, and security strategy, this invention proposes an online reinforcement learning training method based on power hardware-in-the-loop simulation. For example... Figure 2 As shown, the specific process is as follows:

[0062] S1: System Initialization and Environment Configuration. Initialize the power hardware-in-the-loop simulation training system. Load the mathematical model of the target application scenario into the real-time digital simulator, set the maximum safe voltage / current threshold, and activate the real-time hardware protection strategy. Initialize the reinforcement learning agent in the AI ​​accelerator, and construct the main policy network and the main value network. Subsequently, synchronize the initial parameters of the main policy network to the replica policy network running in the controlled power converter controller.

[0063] S2: Secure online training system starts, initiating real-time training. The system concurrently executes the following processes:

[0064] 1. The controller of the controlled power converter collects the electrical state of its main power circuit, forming the state variable s. t It is then input into its replica policy network, which calculates the action variable a in real time. t And execute immediately, driving the power circuit to interact with the power amplifier;

[0065] 2. In the next control cycle, the system acquires the new state s. t+1 And calculate the reward r according to the set reward function. t Subsequently, the empirical tuple (s) t , a t , r t , s t+1 Experiences uploaded to the AI ​​accelerator's replay pool;

[0066] 3. The AI ​​accelerator periodically extracts batch data randomly from the experience replay pool, calculates the gradient based on this, and updates the parameters of its main policy network and main value network. After the update is completed, the AI ​​accelerator synchronously sends the updated main policy network parameters to the controlled power converter controller, overwriting its replica policy network, and completing one policy iteration.

[0067] S3: Policy Convergence and Consolidation. After the continuous interactive training in step S2, training is complete when the agent's control performance converges and stabilizes. The power hardware-in-the-loop simulation training system is stopped, the AI ​​accelerator training is halted, and the agent network parameters are exported.

[0068] Example: The following is in conjunction with the appendix Figure 2 This paper provides a detailed description of an online reinforcement learning training method based on power hardware-in-the-loop simulation, as described in this invention. This embodiment uses the training of a controller for a three-phase inverter connected to a resistive load as an example, but the application of this invention is not limited to this.

[0069] 1. System construction and initialization phase (corresponding to the steps from "Start" to "Start System" in the flowchart):

[0070] First, the power hardware-in-the-loop simulation training system is built. Specifically:

[0071] In a real-time digital simulator, a digital simulation model with a resistive load is used to simulate the external operating environment of the inverter. Simultaneously, the simulator sets the maximum allowable safe voltage threshold (e.g., 310V) and maximum safe current threshold (e.g., 10A) for the inverter's AC port, thereby activating real-time hardware protection strategies.

[0072] In AI accelerators (which can be servers equipped with GPUs or edge computing devices such as NVIDIA Jetson), deep learning frameworks such as PyTorch or TensorFlow are used to initialize reinforcement learning agents (e.g., using DDPG or SAC algorithms). These agents consist of a principal policy network (Actor) and a principal value network (Critic), with network parameters randomly initialized.

[0073] The main body of the controlled power converter is a three-phase grid-connected inverter. An embedded program capable of running neural network inference is burned into the controller. This program carries a replica policy network of the reinforcement learning agent, as well as a reward function containing control objectives and overvoltage and overcurrent penalties.

[0074] After the hardware connection is completed, the initialization command is executed. The AI ​​accelerator synchronizes the initial parameters of its main policy network to the replica policy network in the controlled power converter in one go through TCP / IP Ethernet communication, ensuring the initial consistency of the policies at both ends.

[0075] 2. Online training loop phase (corresponding to the loop from "Start System" to "Has the agent training converged?" in the flowchart):

[0076] The system starts and enters the real-time training loop. The specific steps are as follows:

[0077] 2.1 Real-time decision-making and execution of the controlled power converter:

[0078] In each control cycle, the controlled power converter collects electrical quantities such as output current and grid voltage through its ADC module, forming the state variable s. t Its controller utilizes its local replica policy network to manage the state variable s. t Perform forward propagation calculations and output the control action variable 'a' in real time. t (e.g., voltage reference value). It is then modulated into a PWM drive signal and applied to the switching transistor, thereby enabling actual energy interaction with the power amplifier.

[0079] 2.1 Controlled Power Converter Data Collection and Reward Calculation:

[0080] In controlling action variable a tIn the next control cycle after execution, the controlled power converter acquires a new state s. t+1 And calculate the reward r according to the designed reward function. t Subsequently, the complete experience tuple (s) generated from this interaction will be... t , a t , r t ,s t+1 Experiences are uploaded to the AI ​​accelerator's playback pool via TCP / IP protocol for storage.

[0081] 2.3 Optimization of Asynchronous Strategy for AI Accelerator:

[0082] Steps 2.1 and 2.2.1 above continue, constantly generating new interaction data. Simultaneously, an independent process within the AI ​​accelerator is periodically activated. This process randomly samples a small batch of experience data from the experience replay pool and calculates the policy gradient and value loss using the update rules of a reinforcement learning algorithm (such as DDPG). The parameters of the AI ​​accelerator's local master policy network and master value network are updated via backpropagation. After the network update is complete, the AI ​​accelerator sends the master policy network parameters to the controller of the controlled power converter via a TCP / IP network, overwriting its original replica policy network parameters. This completes one policy iteration, and the controlled power converter continues to interact using the updated policy.

[0083] 3. Strategy convergence and solidification phase (corresponding to the steps after the "Yes" branch in the flowchart):

[0084] The convergence of the agent is determined by monitoring the average reward value or key performance indicators (such as output current THD). Training is considered complete when this indicator stabilizes above a predetermined threshold for several consecutive cycles. Subsequently, the entire power hardware-in-the-loop simulation training system is stopped. The final trained master policy network parameters are exported from the AI ​​accelerator and burned into the controller of the controlled power converter. Afterward, the power converter can operate independently under real loads, detached from the training system.

[0085] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A reinforcement learning - power hardware-in-the-loop training system for a power converter, characterized by: Comprising Real-time digital simulator: high-precision simulation of the external dynamic environment required for the power converter to be connected in actual application; Power amplifier: receiving signals from the real-time digital simulator and amplifying them to the actual voltage, current level required by the controlled power converter, while being able to monitor the electrical state of the power converter port in real time to provide feedback for system closed-loop and control; Controlled power converter: including the main power circuit and the controller to be trained, the main power circuit interacts with the power amplifier in real energy, and the controller is responsible for running the agent decision algorithm and driving the main circuit; AI accelerator: running reinforcement learning agents to autonomously learn optimal control strategies.

2. The reinforcement learning - power hardware-in-the-loop training system for power converters of claim 1, wherein: The power amplifier and the real-time digital simulator communicate bidirectionally through an AD / DA data conversion unit. The real-time digital simulator converts the environmental conditions calculated from digital command signals into analog signals through the AD / DA data conversion unit and sends them to the power amplifier. At the same time, the power amplifier feeds back the voltage and current signals sampled from the controlled power converter port to the real-time simulator through the AD / DA data conversion unit as input for the next calculation.

3. The reinforcement learning - power hardware-in-the-loop training system for power converters of claim 2, wherein: The main power amplifier circuit of the power amplifier and the power circuit of the controlled power converter interact in real power level energy.

4. The reinforcement learning - power hardware-in-the-loop training system for power converters of claim 3, wherein: The controller of the controlled power converter and the AI accelerator exchange data bidirectionally through the TCP / IP high-speed communication protocol. The controlled power converter receives the strategy signal sent by the AI accelerator and feeds back the electrical state of the main circuit and the agent reward to the AI accelerator.

5. The reinforcement learning - power hardware-in-the-loop training system for power converters of claim 1, wherein: The system implements a set of safety strategies composed of real-time hardware protection and agent safety guidance.

6. The reinforcement learning - power hardware-in-the-loop training system for power converters of claim 5, wherein: The real-time hardware protection strategy: in the real-time digital simulator, the maximum safe voltage and current threshold allowed for the controlled power converter port is preset. In each real-time simulation step, the real-time digital simulator compares the port voltage and current with the maximum safe voltage and current threshold. When the port voltage and current is greater than the maximum safe voltage and current threshold, the real-time digital simulator will actively clamp and only output the maximum safe voltage and current threshold, limiting the voltage and current of the controlled power converter port within the safe range through the power amplifier.

7. The reinforcement learning - power hardware-in-the-loop training system for power converters of claim 5, wherein: The agent safety guidance strategy: a over-voltage and over-current penalty term is added in the design of the reinforcement learning reward function. When the port voltage and current approaches the maximum safe voltage and current threshold, the agent will receive a negative reward.

8. A method for using the reinforcement learning-power hardware-in-the-loop training system for power converters according to any one of claims 1-7, characterized in that: The specific process is as follows: S1. System initialization and environment configuration; S2. Online training in safety; S3. Strategy convergence and solidification.

9. The reinforcement learning - power hardware-in-the-loop training method for power converters according to claim 8, characterized in that: S1 includes the following steps: S11. Load the mathematical model in the real-time digital simulator and set the safe voltage and current threshold; S12. Initialize the reinforcement learning agent in the AI accelerator and build the main policy network and the main value network; S13. Synchronize the main policy network to the controller of the controlled power converter.

10. The reinforcement learning - power hardware-in-the-loop training method for power converters according to claim 9, characterized in that: S2 is a real-time training loop. When the control performance of the agent converges and stabilizes, the training is complete, and the loop ends; Including the following steps: S21. The controlled power converter acquires state variables s t , calculates and executes action variables a t ; S22. The controller of the next control period acquires the new state s t+1 to calculate the reward r t ; S23. upload the experience tuple (s t , a t , r t , s t+1 ) to an experience replay pool of the AI accelerator; S24. The AI accelerator randomly extracts batch data from the experience replay pool; S25. Perform a back propagation algorithm to update network parameters; S26. The AI accelerator synchronizes the updated master strategy network parameters to the controlled power converter.

Images