A single-stage multi-port inverter control method and system based on online interactive deep reinforcement learning

Abstract

The application discloses a single-stage multi-port inverter control method and system based on online interaction deep reinforcement learning, relates to the technical field of micro-grid, and solves the technical problems that micro-grid control depends on offline training, has insufficient generalization ability, and lacks online adaptive ability. The method comprises the following steps: mathematically formalizing the control problem of the single-stage multi-port inverter, defining the state and action space, designing the safe constraint multi-objective reward function, and constructing the deep neural network architecture; collecting offline data and pre-training through imitation learning to provide an initial control strategy for an agent; constructing a main training loop of deep reinforcement learning and an online interaction learning framework closed with the single-stage multi-port inverter, and performing real-time closed-loop online interaction learning training between a high-performance computer and a real-time control computer. Through the real-time closed-loop online interaction learning between the high-performance computer and the real-time control computer, the demand of the micro-grid is effectively met.

Description

Technical Field

[0001] This invention relates to the field of microgrid technology, and in particular to a single-stage multi-port inverter control method and system based on online interactive deep reinforcement learning. Background Technology

[0002] With the rapid development of new energy technologies, distributed energy, centered on photovoltaic power generation and energy storage systems, is gradually replacing traditional centralized power generation models. In this process, microgrids, as an important technological form capable of enabling the coordinated operation of multiple energy sources, have become a significant development direction for modern power systems. Especially in islanded operation mode, microgrids can supply power independently without relying on the main grid, playing an irreplaceable role in extreme environments, areas with weak grids, and critical load power supply scenarios. In islanded microgrids, the coordinated operation of photovoltaic power generation systems and battery energy storage systems is crucial for achieving power balance and ensuring power quality. This has led to the emergence of Single-Stage Multi-Port Inverters (SSMPI), which, through a single-stage power conversion circuit, simultaneously connects multiple DC source ports and one / multiple AC ports to directly complete power conversion, power distribution, and inverter output, without an independent intermediate DC / DC conversion stage or a common DC bus.

[0003] Single-stage multi-port inverter topologies integrate multiple DC energy ports and AC output terminals into a single conversion structure, enabling direct energy coupling and transmission. This reduces intermediate conversion stages, significantly improving system efficiency and reducing hardware complexity, but also increases control difficulty. With the development of artificial intelligence technology, Deep Reinforcement Learning (DRL) has gradually been introduced into the field of power electronics control. By interacting with the environment to learn control strategies, it can effectively handle nonlinear, uncertain, and multi-objective optimization problems without relying on precise mathematical models.

[0004] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art: Microgrid control based on deep reinforcement learning relies on offline training, has insufficient generalization ability, lacks online adaptive capability, and is difficult to meet the needs of microgrids. Summary of the Invention

[0005] The purpose of this invention is to provide a single-stage multi-port inverter control method and system based on online interactive deep reinforcement learning, to solve the technical problems of existing deep reinforcement learning-based microgrid control that rely on offline training, have insufficient generalization ability, lack online adaptive capability, and are difficult to adapt to the needs of microgrids. The various technical effects of the preferred solutions among the many technical solutions provided by this invention are detailed below.

[0006] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a control method for a single-stage multi-port inverter based on online interactive deep reinforcement learning, comprising the following steps: S100: Mathematically formalizing the control problem of the single-stage multi-port inverter, defining the state and action spaces, designing a safety-constrained multi-objective reward function, and constructing a deep neural network architecture; S200: Collecting offline data through a simulation environment and a real physical platform, performing imitation learning pre-training on the collected data to provide an initial control strategy for the agent; S300: Constructing an online interactive learning framework that integrates the main training loop of deep reinforcement learning with the closed loop of the single-stage multi-port inverter, and performing online interactive learning in each control cycle. The high-performance computer and the real-time control computer perform real-time closed-loop online interactive learning and training based on TD3 to continuously refine the control strategy of the single-stage multi-port inverter; when executing the TD3 closed-loop online training loop, it is based on the current state. ,action ,award and the next state Composition of state transition tuples And added to the experience replay pool In the middle, the TD3 algorithm uses the experience replay pool The process involves randomly sampling small batches of samples to perform multi-step gradient updates.

[0007] Preferably, step S100 specifically includes: S110: formalizing the control problem of a single-stage multi-port inverter into a Markov decision process, thereby defining a seven-dimensional state space. and one-dimensional continuous action space S120: Design a multi-objective reward function with security constraints based on dense feedback, sparse evaluation and security penalty mechanism; S130: Construct an actor-critic network structure based on a deep neural network architecture.

[0008] Preferably, in step S110, the state space The expression is: ,in, The power allocation coefficient at the current moment; The normalized tracking error for photovoltaic power represents historical information; The difference in photovoltaic power between adjacent time steps represents the prediction information; To normalize instantaneous power and characterize real-time information; The virtual duty cycle of the three-phase system is equivalent to two-level modulation; the operating space. The expression is: ,in This action The zero-vector action time allocation factor in the corresponding space vector modulation strategy is the only control variable that determines the power distribution between the photovoltaic port and the battery port.

[0009] Preferably, in step S120, the multi-objective reward function The expression is: ;in, This represents a dense reward for tracking error, based on the normalized tracking deviation of photovoltaic power. Pre-defined piecewise linear mapping function; To represent the reward for suppressing power fluctuations, a sliding window with a length of 20 control cycles is introduced, and the standard deviation of photovoltaic power within the window is calculated in real time. With peak The normalized ripple score is then used to determine the ripple distribution. The range in which it is located is evaluated using a tiered fluctuation assessment based on pre-defined conditional logic. This represents a sparse reward for steady-state accuracy, if the current photovoltaic tracking error satisfies... Then a fixed positive reward will be given in one go. ; Indicates the safety penalty for smooth motion, and calculates the motion at the current moment. Actions in the previous moment absolute value of the difference ; This indicates a supervised boundary violation safety penalty, where a pre-deployed PI reference controller generates a reference action. Real-time comparison of agent outputs Compared with this benchmark value, if If the action is deemed potentially unsafe, a fixed penalty will be imposed. The forced strategy is to explore within a safe area; These represent the weight coefficients for dense tracking error reward, power fluctuation suppression reward, sparse steady-state accuracy reward, smooth motion safety penalty, and supervised boundary crossing safety penalty, respectively.

[0010] Preferably, in step S130, in the actor-critic network structure, the actor network contains two hidden layers with 80 and 64 neurons respectively, both connected to the ReLU activation function. The output layer consists of a single neuron followed by the Tanh activation function, used to constrain the action within the [-1,1] interval and then linearly map it to [0,1]. The critic network adopts a dual-pathway structure, processing the state input and action input independently. Each input is mapped to a 48-dimensional feature space through a fully connected layer with 100 neurons. The two feature vectors are then concatenated and processed again through a fully connected layer with 48 neurons and ReLU activation, finally outputting a scalar state-action value. value.

[0011] Preferably, step S200 specifically includes: S210: collecting system state-control action pair data from two sources: a high-fidelity Simulink simulation environment and a real physical platform regulated by a PI controller, to construct an expert dataset; S220: constructing an artificial neural network with the exact same structure as the agent actor network, using the system state as input and the control action as output, and performing supervised learning pre-training on the expert dataset by minimizing the mean square error between its output and the expert action; S230: directly transferring the network weights of the pre-trained artificial neural network to the actor network as its initial parameters. .

[0012] Preferably, step S300 specifically includes: S310: Building an online interactive system architecture with a high-performance computer as the strategy training center and a real-time control computer as the real-time control and data interaction interface; the real-time control computer communicates with the dSPACE control desktop for execution control; before starting online interactive learning, collecting several state transition data from the ideal simulation environment to pre-fill the experience playback pool on the high-performance computer; S320: Establishing a wireless bidirectional communication link based on the TCP / IP protocol between the high-performance computer and the real-time control computer; S330: Executing a closed-loop online TD3 training cycle to continuously compensate for environmental nonlinear factors and cross-coupling effects, enabling the control strategy to continuously evolve and adapt in real-time in the real physical environment, thereby improving the adaptive performance in real working conditions.

[0013] Preferably, in step S330, in each control cycle The following steps are executed sequentially to complete one learning and control loop: S331: The real-time control computer receives the actor network from the high-performance computer. Based on the current state Calculated actions Superimposed with cropped Gaussian exploration noise Generate the final execution action ,in This represents the clipping function. This indicates the hyperparameters and sends them to the dSPACE control desktop for execution, affecting the single-stage multi-port inverter system; S332: Current system status. Transition to the next state Then, the real-time control computer calculates the immediate reward using a multi-objective reward function. S333: Will be determined by the current state ,action ,award and the next state A complete state transition tuple The state transition tuple is transmitted in real time to the high-performance computer via a TCP / IP link; 334: The high-performance computer receives the state transition tuple and appends it to the experience replay pool in a first-in-first-out manner. In this process, the simulated data in the pool is gradually replaced by real physical data; S335: High-performance computers replay the data from the experience pool. In step S336, a small batch of samples are randomly sampled, and the TD3 algorithm is used to perform multi-step gradient updates. After the high-performance computer completes this round of optimization, it immediately sends the updated actor network parameters θ to the real-time control computer through the TCP / IP link for action calculation in the next control cycle. Steps S331 to S335 are executed cyclically until the control process ends.

[0014] Preferably, step S335 specifically includes: using a target strategy to smooth and regularize the calculation of the target action, truncating the double-Q learning to calculate the temporal difference target value; and updating the network parameters of the two commentators accordingly. , Then every interval Step 1: Update the actor's network parameters. and with soft update coefficient Smoothly update all target network parameters.

[0015] A single-stage multi-port inverter control system based on online interactive deep reinforcement learning is disclosed, used to run the single-stage multi-port inverter control method based on online interactive deep reinforcement learning described above. The system includes a single-stage multi-port inverter main circuit module, a photovoltaic power supply module, a battery energy storage module, an AC-side load, an LC filter module, a high-performance computer, a dSPACE control desktop, and a real-time control computer. The photovoltaic power supply module is connected to the upper DC port of the single-stage multi-port inverter, and the battery energy storage module is connected to the lower DC port. The AC side of the single-stage multi-port inverter is connected to the load via the LC filter module. A control execution unit is connected to the drive signal terminal of the single-stage multi-port inverter to generate PWM control signals. The dSPACE control desktop is the real-time control execution unit and is connected to the real-time control computer. A bidirectional data interaction link is established between the high-performance computer and the real-time control computer via TCP / IP protocol. The real-time control computer executes real-time signal acquisition and control command issuance, while the high-performance computer performs strategy iteration optimization and network parameter updates, collaboratively performing full-chain closed-loop adaptive control.

[0016] Implementing one of the above-described technical solutions of the present invention has the following advantages or beneficial effects: This invention utilizes the TD3 real-time closed-loop online interactive learning between a high-performance computer and a real-time control computer to feed back the real-time state transition data generated by the physical system in each control cycle to the strategy optimization loop, thereby achieving continuous online updates of the control strategy. This enables the strategy network to continuously optimize for real-world environmental characteristics. This mechanism allows the strategy network to continuously adapt to modeling errors, parameter drift, measurement noise, and unmodeled dynamics during real hardware operation, fundamentally narrowing the performance gap between simulation and reality and endowing the control system with the ability to continuously evolve under time-varying conditions. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings: Figure 1 This is a flowchart of a single-stage multi-port inverter control method based on online interactive deep reinforcement learning, according to an embodiment of the present invention. Figure 2 This is a flowchart of step S100 in a single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to Embodiment 1 of the present invention; Figure 3 This is a flowchart of step S200 in a single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to Embodiment 1 of the present invention; Figure 4 This is a flowchart of step S300 in a single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to Embodiment 1 of the present invention; Figure 5 This is a flowchart of step S330 in a single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of a single-stage multi-port inverter control system based on online interactive deep reinforcement learning, according to Embodiment 2 of the present invention. Figure 7 This is a diagram showing the implementation result of a single-stage multi-port inverter control system based on online interactive deep reinforcement learning, according to Embodiment 2 of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the present invention clearer, various exemplary embodiments described below will be referenced to the accompanying drawings, which form part of the exemplary embodiments, illustrating various exemplary embodiments that may be used to implement the present invention. Unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. It should be understood that they are merely examples of processes, methods, and apparatuses consistent with some aspects of the present invention disclosed as detailed in the appended claims, and other embodiments may be used, or structural and functional modifications may be made to the embodiments listed herein without departing from the scope and spirit of the present invention.

[0019] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," etc., indicate the orientation or positional relationship based on the accompanying drawings, and are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the referred element must have a specific orientation, or be constructed and operated in a specific orientation. The terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. The term "multiple" means two or more. The terms "connected" and "linked" should be interpreted broadly, for example, they can be fixed connections, detachable connections, integral connections, mechanical connections, electrical connections, communication connections, direct connections, indirect connections through an intermediate medium, and can be the internal connection of two elements or the interaction relationship between two elements. The term "and / or" includes any and all combinations of one or more of the related listed items. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0020] To illustrate the technical solution described in this invention, specific embodiments are described below, showing only the parts related to the embodiments of this invention.

[0021] Example 1: like Figure 1As shown, this invention provides a control method for a single-stage multi-port inverter based on online interactive deep reinforcement learning, including the following steps: S100: The control problem of the single-stage multi-port inverter is mathematically formalized, defining the state and action spaces, designing a multi-objective reward function with safety constraints, and constructing a deep neural network architecture. This reward function guides the strategy from the root to pursue high performance while adhering to safe operating boundaries. The multi-objective reward function avoids compressing complex objectives into a single scalar reward, but instead explicitly models the trade-offs between objectives, enabling the agent to find a reasonable balance between mutually constraining objectives such as safety, efficiency, and cost, and prioritizing the protection of hard safety constraints. S200: An initial control strategy is provided to the agent through offline data collection and imitation learning pre-training; a safe and reliable initial strategy is provided to the deep reinforcement learning agent through data-driven imitation learning, avoiding control instability caused by random exploration in the early stages of online learning. S300: Constructs an online interactive learning framework that integrates the main training loop of deep reinforcement learning with the closed-loop operation of a single-stage multi-port inverter. Through real-time closed-loop online interactive learning training using a high-performance computer and a real-time control computer, it feeds back the actual state transition data generated by the physical system in each control cycle to the strategy optimization loop in real time. This enables continuous online updates of the control strategy, continuous compensation for environmental nonlinearities and cross-coupling effects, and continuous refinement of the single-stage multi-port inverter control strategy. This improves the adaptive performance under real-world operating conditions, allowing the inverter to achieve more precise voltage waveform regulation under different operating conditions. The optimal approach is to optimize the control strategy in each control cycle. The high-performance computer and the real-time control computer perform real-time closed-loop online interactive learning and training based on TD3. When executing the TD3 (Twin Delayed Deep Deterministic Policy Gradient) closed-loop online training loop, it is based on the current state. ,action ,award and the next state Composition of state transition tuples And added to the experience replay pool In the middle, the TD3 algorithm uses the experience replay pool Multi-step gradient updates are performed by randomly sampling small batches of samples. By gradually introducing real-time state transition data collected during actual operation into the replay buffer and participating in the TD3 algorithm update, the policy network can continuously optimize for real-world environmental characteristics. Real-time closed-loop online interactive learning achieves a complete educational closed loop of "input-interaction-feedback-optimization" by constructing a dynamic feedback system covering the entire learning process, transforming one-way teaching into a data-driven personalized learning experience, significantly improving the efficiency and relevance of knowledge absorption. This invention uses TD3 real-time closed-loop online interactive learning between a high-performance computer and a real-time control computer to feed back the real state transition data generated by the physical system in each control cycle to the policy optimization loop in real time, realizing continuous online updates of the control policy, enabling the policy network to continuously optimize for real-world environmental characteristics. This mechanism allows the policy network to continuously adapt to modeling errors, parameter drift, measurement noise, and unmodeled dynamics during real hardware operation, fundamentally narrowing the performance gap between simulation and reality, and giving the control system the ability to continuously evolve under time-varying conditions.

[0022] As an optional implementation method, such as Figure 2 As shown, step S100 specifically includes: S110: The control problem of a single-stage multi-port inverter is formalized into a Markov decision process (a mathematical framework describing an agent maximizing long-term cumulative returns through sequential decisions in a stochastic environment; its core characteristic is that the next state depends only on the current state and action, and is independent of historical paths, thus transforming the complex sequential decision problem into a computable optimization problem), thereby defining a seven-dimensional state space. and one-dimensional continuous action space The state space describes all possible observable states of the environment, while the action space describes all possible action options that the agent can execute. S120: Based on dense feedback, sparse evaluation, and a safety penalty mechanism, a multi-objective reward function with safety constraints is designed to balance control performance and online learning safety, improving the steady-state control accuracy of the microgrid under complex operating conditions. The dense reward provides continuous feedback signals to the power tracking error through piecewise linear mapping and introduces a power ripple suppression term based on sliding window statistics to avoid high-frequency oscillations caused by control actions. The sparse reward is used to strengthen the long-term steady-state high-precision control objective. The safety constraint penalty term explicitly constrains the exploration range of the action space by limiting the amplitude of action changes and detecting abnormal actions based on prior controller monitoring signals. This hybrid reward structure, through reasonable weighting, ensures that the agent always adheres to the safe operating boundary while optimizing power allocation performance, effectively solving the risk of system instability caused by random actions during online exploration. S130: Based on a deep neural network architecture, an actor-critic network structure is constructed. The actor network directly generates the policy, and the critic network evaluates the value of the state or action. The optimization is achieved through the interaction of gradient signals. By combining the division of labor between the two, the limitations of a single method are overcome. It is suitable for tasks with continuous action space and takes into account both the flexibility of policy gradient and the stability of value function methods.

[0023] As an optional implementation, in step S110, the state space The expression is: ,in, The power allocation coefficient at the current moment; The normalized tracking error for photovoltaic power represents historical information; The difference in photovoltaic power between adjacent time steps represents the prediction information; To normalize instantaneous power and characterize real-time information; The three-phase virtual duty cycle is equivalent to two-level modulation; the state space is designed to simulate the "experience-current state-trend" learning paradigm in human cognition, providing the policy network with rich inputs containing system dynamics characteristics. Action space. The expression is: ,in This action The zero-vector action time allocation factor in the corresponding space vector modulation strategy (which achieves precise control of AC output voltage by synthesizing a basic voltage vector and significantly reduces current harmonics) is the only control variable that determines the power distribution between the photovoltaic port and the battery port.

[0024] As an optional implementation, in step S120, the multi-objective reward function... The expression is: ;in, This represents a dense reward for tracking error, based on the normalized tracking deviation of photovoltaic power. Pre-defined piecewise linear mapping function, in Within the positive reward interval, the slope of each sub-interval decreases to encourage high-precision tracking; Within the penalty interval, the slope of each segment increases monotonically, thus imposing a rapidly increasing penalty on larger deviations. To represent the reward for suppressing power fluctuations, a sliding window with a length of 20 control cycles is introduced, and the standard deviation of photovoltaic power within the window is calculated in real time. With peak And normalize it. , Represents the photovoltaic reference power, based on the normalized ripple score. Within the given interval, a tiered fluctuation evaluation is provided based on pre-defined conditional logic, and the normalized ripple score is... A higher positive reward is given when the score is lower, and the normalized ripple score is increased. When the value is large, apply the normalized ripple score. A linear penalty proportional to power effectively suppresses power oscillations. This represents a sparse reward for steady-state accuracy, if the current photovoltaic tracking error satisfies... Then a fixed positive reward will be given in one go. This enhances long-term, high-precision, steady-state performance. Indicates the safety penalty for smooth motion, and calculates the motion at the current moment. Actions in the previous moment absolute value of the difference ,like Then a punishment will be imposed. This is done to constrain drastic changes in control quantities and ensure smooth output. This indicates a supervised boundary violation safety penalty, where a pre-deployed PI reference controller generates a reference action. Real-time comparison of agent outputs Compared with this benchmark value, if If the action is deemed potentially unsafe, a fixed penalty will be imposed. The forced strategy is to explore within a safe area; These represent the weight coefficients for dense rewards for tracking errors, rewards for suppressing power fluctuations, rewards for sparse rewards for steady-state accuracy, safety penalties for smooth action, and safety penalties for supervised out-of-bounds violations, respectively. These weight coefficients can be configured according to the control objective. By constructing a multi-objective reward function with safety constraints, the dense rewards for power tracking errors are organically integrated with the power ripple suppression mechanism, enabling the agent to autonomously learn a control strategy that balances fast response and stable output during online interaction. Simultaneously, the online learning mechanism allows the control strategy to continuously optimize network parameters using real-time operating data in a realistic hardware environment, effectively compensating for modeling errors, parameter drift, and unmodeled dynamics between the simulation model and the actual system. This embodiment directly pushes the decision space of deep reinforcement learning down to the converter-level control layer, using the power allocation factor in the modulation strategy as the control action. It integrates photovoltaic and battery power allocation and grid-side AC voltage regulation into the state space and reward signal design. Through online interactive learning, the agent autonomously captures the complex nonlinear mapping relationship between the power allocation factor and the power flow at multiple terminals of the system, achieving coordinated optimization control of power allocation and voltage regulation without explicit decoupling mechanisms. This effectively solves the inherent strong coupling control problem of single-stage multi-port inverters due to the lack of an intermediate buffer stage. Experimental results show that when the photovoltaic port reference power is 800W, this embodiment reduces the port power ripple to 29.1W, a reduction of approximately 66.24% compared to the existing linear control scheme (86.2W) and approximately 24.81% compared to the purely offline training deep reinforcement learning scheme (38.7W). When the photovoltaic port reference power is 1200W, the power ripple of this embodiment is further reduced to 26.5W, a reduction of approximately 66.28% compared to the linear scheme and approximately 42.76% compared to the purely offline scheme. Therefore, this embodiment uses a multi-objective reward function to maintain extremely low power fluctuations under different power distribution conditions, which significantly improves the steady-state control accuracy of the microgrid under complex conditions.

[0025] As an optional implementation, in step S130, the actor-critic network structure includes two hidden layers with 80 and 64 neurons respectively, both connected to ReLU activation functions. The output layer consists of a single neuron followed by a Tanh activation function, which linearly maps the action to [0,1] after constraining it to the [-1,1] interval. The critic network employs a dual-pathway structure, processing the state input and action input independently. Each input is mapped to a 48-dimensional feature space via a fully connected layer with 100 neurons. The two feature vectors are then concatenated and processed again via a fully connected layer with 48 neurons and ReLU activation, ultimately outputting a scalar state-action value. value.

[0026] As an optional implementation method, such as Figure 3As shown, step S200 specifically includes: S210: Collect system state-control action pair data from two sources: a high-fidelity Simulink simulation environment (the Simulink simulation environment is a graphical multi-domain simulation and model-based design platform based on MATLAB, which realizes the visualization modeling, simulation verification and automatic code generation of dynamic systems through modular block diagrams, significantly improving the development efficiency of complex engineering problems such as control systems and signal processing, and making the virtual model and physical entity highly consistent in dynamic response, boundary behavior and fault modes) and a real physical platform regulated by a PI controller (a closed-loop physical platform containing real controlled objects, sensors, actuators and hardware limiting constraints). Establish a "perception-decision" mapping relationship through the system state-control action pair data, which must meet the requirements of strict time alignment, comprehensive state coverage and accurate action correspondence, and construct an expert dataset; select an expert dataset that covers various operating conditions such as steady state, load change, and reference signal step, forming a hybrid expert dataset containing more than one million samples. S220: Construct an artificial neural network with the exact same structure as the agent-actor network. Using the system state as input and control actions as output, supervised learning pre-training is performed on an expert dataset by minimizing the mean square error between its output and expert actions. This allows implicit stable control knowledge to be distilled from the data. In the initial stage of online training in this embodiment, no severe divergence or unstable actions occurred, indicating that the imitation learning initialization strategy effectively avoids unsafe actions that might be introduced during the purely random exploration phase. This significantly reduces the risk of microgrid system instability during training and improves the engineering safety of the algorithm when deployed on actual devices. S230: The network weights of the pre-trained artificial neural network are directly transferred to the agent network as its initial parameters. This approach endows the agent with a near-optimal initial policy, avoiding the random exploration phase and ensuring operational safety during the initial stages of online training. Before starting online training for deep reinforcement learning, a simulation learning dataset is constructed using a large amount of operational data from the simulation environment and the real PI controller. Supervised pre-training of the policy network is then performed, and the pre-trained network weights are directly transferred to the actor network of deep reinforcement learning as the initial policy. This enables the agent to possess near-optimal behavioral capabilities upon its first interaction with the physical system, avoiding the inherent high risk and high sample consumption problems of purely random initial policies exploring on real hardware. This achieves a simultaneous improvement in training safety, initial control quality, and convergence efficiency.

[0027] As an optional implementation method, such as Figure 4As shown, step S300 specifically includes: S310: Building an online interactive system architecture with a high-performance computer as the strategy training center and a real-time control computer as the real-time control and data interaction interface. The real-time control computer communicates with the dSPACE control desktop for execution control (the dSPACE control desktop is the core interactive platform for experiment management, data monitoring, hardware control, and automated testing, integrating experiment control, data monitoring, and automated testing into a single environment, connecting the Simulink model with physical hardware, realizing real-time verification of control algorithms, and significantly reducing efficiency losses caused by tool switching; preferably, it communicates with the real-time control computer through the ASAM MCD-3 automation interface. ASAM MCD-3 is a remote automation control interface specification designed specifically for measurement and calibration systems in the ASAM standard system. Its core function is to enable the upper-level automation system to call the functions of calibration tools in a unified way without having to pay attention to the underlying communication protocol or the specific implementation details of the tools). Before starting online interactive learning, a number of state transition data (preferably 500,000 in this embodiment) are collected from the ideal simulation environment to pre-fill the experience playback pool on the high-performance computer to smooth the transition from simulation learning to hardware adaptation. S320: A wireless bidirectional communication link based on the TCP / IP protocol is established between the high-performance computer and the real-time control computer, enabling low-latency uploading of measurement data and distribution of updated network parameters. The high-performance computer and the real-time control computer implement a closed-loop control process of real-time status acquisition, remote strategy optimization, and online parameter distribution. Offline simulation data is only used for initial filling of the playback buffer, and is gradually replaced by real interactive data through a first-in-first-out mechanism. This allows the strategy to smoothly transition from the simulation environment to actual hardware, significantly reducing the performance gap between simulation and reality. This hardware-software collaborative working method enables complex deep reinforcement learning algorithms to run stably on embedded power electronic control platforms (such as dSPACEMicroLabBox), balancing the dual requirements of high-performance computing resource utilization and real-time control response, laying the foundation for further engineering deployment on actual digital signal processor platforms. S330: Executes a closed-loop online TD3 training loop to continuously compensate for environmental nonlinear factors and cross-coupling effects, enabling the control strategy to continuously evolve and adapt in real-time in the real physical environment, improving adaptive performance in real-world operating conditions. During real-time interaction with the environment, the intelligent agent continuously collects new experiences and dynamically updates the policy network and value network, forming a complete learning loop of "execution-feedback-optimization," enabling the control strategy to continuously evolve and adapt in real-time to the real physical environment. In this embodiment, the online interactive learning mechanism is used to continuously compensate for environmental nonlinear factors and cross-coupling effects, allowing the inverter to achieve more precise voltage waveform regulation under different operating conditions.Specifically, under the condition of a photovoltaic port reference power of 1200W, the present invention reduces the total harmonic distortion (THD) of the AC side voltage to 1.14%, which is approximately 39.68% lower than the linear controller scheme (1.89%) and approximately 26.92% lower than the purely offline training deep reinforcement learning scheme (1.56%). This effect is not achieved by adjusting the filter parameters, but by the control strategy itself's better adaptation to nonlinear dynamics and real-time adjustment, fundamentally improving the power output quality of the system. Furthermore, relying on a closed-loop online interactive learning framework, this embodiment directly incorporates the real-time feedback of the physical system into the policy update loop, without relying on a fixed offline training model. In dynamic tests such as step changes in photovoltaic port reference power (800W to 1200W and reverse changes) and grid-side load switching (load power switching from 1250W to 1000W and then back), both photovoltaic power and battery power can smoothly transition and accurately track their respective reference values. Grid-side voltage and current remain stable. The agent's actions show rapid and accurate dynamic adjustments, indicating that this embodiment has fully captured the strong coupling relationship between power distribution and AC voltage regulation. Even under rapidly changing operating conditions, it can still maintain the accuracy of decoupled control and the dynamic stability of the system, possessing robust adaptability that is difficult to achieve with traditional linear controllers and pure offline reinforcement learning controllers.

[0028] As an optional implementation method, such as Figure 5 As shown, in step S330, in each control cycle The following steps are executed sequentially to complete one learning and control loop: S331: The real-time control computer receives the actor network from the high-performance computer. Based on the current state Calculated actions Superimposed clipped Gaussian exploration noise Generate the final execution action ,in This represents the clipping function, used to prevent gradient explosion during training. This indicates the hyperparameters and sends them to the dSPACE control desktop for execution, affecting the single-stage multi-port inverter system; S332: Current system status. Transition to the next state Then, the real-time control computer calculates the immediate reward using a multi-objective reward function. While optimizing control precision, it imposes hard safety constraints on exploration behavior; S333: will change from the current state ,action ,award and the next state A complete state transition tuple The data is transmitted in real time to a high-performance computer via a TCP / IP link to achieve experience storage and transmission; S334: The high-performance computer receives the state transition tuple and appends it to the experience replay pool in a first-in-first-out manner. In this process, simulated data in the pool is gradually replaced by real physical data. A first-in, first-out (FIFO) mechanism is used to gradually replace simulated data with transition data collected from real physical systems, enabling the policy network to smoothly transition from the simulated feature space to the real environment feature space. This ensures the continuity of the training process while effectively avoiding performance degradation or instability caused by drastic changes in observation distribution in the initial stage. S335: High-performance computer from the experience replay pool A small batch of randomly sampled data was used, and the TD3 algorithm was employed to perform multi-step gradient updates. By gradually introducing real-time state transition data collected during actual operation into the replay buffer and participating in the TD3 algorithm update, the policy network could continuously optimize for real-world environmental features. Experimental data showed that the average reward of the online training scheme converged and reached a high level after about 80 rounds, while the average reward of the pure offline training scheme remained significantly lower, and the reward distribution in each round was highly discrete, failing to converge effectively. This indicates that online interactive learning successfully overcame the performance bottleneck of offline training. By learning the nonlinear mapping relationship in the real-world environment distribution, the control policy achieved a leap in both optimization efficiency and final performance. S336: After the high-performance computer completed this round of optimization, the updated actor network parameters were immediately updated. The data is transmitted to the real-time control computer via TCP / IP link for action calculation in the next control cycle. Steps S331 to S335 are executed cyclically until the control process ends. This enables the control strategy to continuously evolve and adapt in real-time in the real physical environment. It achieves closed-loop online adaptive optimal control of a strongly coupled, nonlinear system such as a single-stage multi-port inverter, combining the safety of initial learning, the real-time nature of strategy updates, and the optimality of long-term operation.

[0029] As an optional implementation, step S335 specifically includes: calculating the target action using target policy smoothing regularization (injecting pruning noise into the target action to suppress value function spikes), calculating the temporal difference target value by truncating double-Q learning (taking the minimum value among the two critics' target network outputs); and updating the network parameters of the two critics accordingly. , Then every interval Step 1: Update the actor's network parameters. and with soft update coefficient Smoothly update all target network parameters.

[0030] The embodiment is merely a specific example and does not indicate that this is the only way to implement the present invention.

[0031] Example 2: A single-stage multi-port inverter control system based on online interactive deep reinforcement learning is provided for running the single-stage multi-port inverter control method based on online interactive deep reinforcement learning described in Example 1, such as... Figure 6 As shown, the system includes a single-stage multi-port inverter main circuit module, a photovoltaic power supply module, a battery energy storage module, an AC-side load, an LC filter module, a high-performance computer, a dSPACE control desktop, and a real-time control computer. The photovoltaic power supply module is connected to the upper DC port of the single-stage multi-port inverter, and the battery energy storage module is connected to the lower DC port of the single-stage multi-port inverter. The AC side of the single-stage multi-port inverter is connected to the load via the LC filter module. The control execution unit is connected to the drive signal terminal of the single-stage multi-port inverter to generate PWM control signals. The dSPACE control desktop is the real-time control execution unit and is connected to the real-time control computer. A bidirectional data interaction link is established between the high-performance computer and the real-time control computer via the TCP / IP protocol. The real-time control computer executes real-time signal acquisition and control command issuance, while the high-performance computer performs strategy iteration optimization and network parameter updates, collaboratively performing full-chain closed-loop adaptive control.

[0032] In this embodiment, the following experimental platform is used: the photovoltaic side voltage is simulated by an IT6526C programmable DC power supply (380-420V), the battery side voltage is simulated by a Keysight RP7961A programmable DC power supply (180-220V), and the AC port is connected to a 110V / 50Hz islanded AC bus and a configurable resistive load group via an LC filter (3mH inductor, 10μF capacitor). The control algorithm is deployed on the dSPACE MicroLabBox DS1202 real-time hardware platform, and both the control frequency and the switching frequency are 10kHz.

[0033] Dynamic photovoltaic power reference change tests were conducted based on the above experimental platform. For example... Figure 7 As shown, in this test, the DC terminal V PV =400V, V Bat =200V, grid-side load power P eref Set to 1000W. Figure 7 (a) in the text represents the change in the reference value for photovoltaic power generation from 800W to 1200W. Figure 7 (b) shows the photovoltaic power generation reference value changing from 1200W to 800W. From top to bottom, the waveforms correspond to the grid-side voltage / current. v ga / i ga Photovoltaic power generation PPV Battery power P Bat And the agent's action values. During the test, the load power on the grid side was fixed at 1000W, and the photovoltaic port reference power jumped from 800W to 1200W, then back to 800W. During this process, the photovoltaic power at the upper port accurately tracked the change in the reference value, while the battery power at the lower port smoothly transitioned from 200W to -200W (the battery switched from discharging to charging) to compensate for the power deficit, and the grid-side voltage and current remained stable. Simultaneously, the agent's output action quantity showed rapid and accurate adaptive adjustment according to changes in operating conditions. The test results show that this embodiment can effectively achieve decoupling control between power allocation and AC voltage regulation under conditions of large step changes in photovoltaic power reference. Photovoltaic and battery power tracking is accurate and the transition is smooth, and the AC power quality is unaffected by disturbances. This dynamic performance is achieved through the closed-loop online interactive learning mechanism's accurate capture of the nonlinear mapping relationship between the power allocation factor and operating conditions, and the continuous optimization of the policy network under real-time data-driven conditions, verifying that the method of this invention has outstanding adaptive capability and robustness under dynamic operating conditions.

[0034] The above description is merely a preferred embodiment of the present invention. Those skilled in the art will understand that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the present invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A control method for a single-stage multi-port inverter based on online interactive deep reinforcement learning, characterized in that, Includes the following steps: S100: The control problem of a single-stage multi-port inverter is mathematically formalized, including the definition of state and action spaces, the design of a safety-constrained multi-objective reward function, and the construction of a deep neural network architecture. S200: Collects offline data through a simulation environment and a real physical platform, performs imitation learning pre-training on the collected data, and provides initial control strategies for the intelligent agent; S300: Constructs an online interactive learning framework that integrates the main training loop of deep reinforcement learning with the closed loop of a single-stage multi-port inverter, in each control cycle. The high-performance computer and the real-time control computer conduct real-time closed-loop online interactive learning and training based on TD3 to continuously refine the control strategy of the single-stage multi-port inverter. When executing the TD3 closed-loop online training loop, based on the current state ,action ,award and the next state Composition of state transition tuples And added to the experience replay pool In the middle, the TD3 algorithm uses the experience replay pool Randomly sample small batches of samples to perform multi-step gradient updates; Step S100 specifically includes: S110: formalizing the control problem of a single-stage multi-port inverter into a Markov decision process, thereby defining a seven-dimensional state space. and one-dimensional continuous action space S120: Design a multi-objective reward function with security constraints based on dense feedback, sparse evaluation, and security penalty mechanism; S130: Construct an actor-critic network structure based on a deep neural network architecture; state space The expression is: ,in, The power allocation coefficient at the current moment; The normalized tracking error for photovoltaic power represents historical information; The difference in photovoltaic power between adjacent time steps represents the prediction information; To normalize instantaneous power and characterize real-time information; The virtual duty cycle of the three-phase system is equivalent to two-level modulation; the operating space. The expression is: ,in This action The zero-vector action time allocation factor in the corresponding space vector modulation strategy is the only control variable that determines the power distribution between the photovoltaic port and the battery port. Step S200 specifically includes: S210: Collecting system state-control action pair data from two sources: a high-fidelity Simulink simulation environment and a real physical platform regulated by a PI controller, to construct an expert dataset; S220: Constructing an artificial neural network with the exact same structure as the agent actor network, using system state as input and control action as output, and performing supervised learning pre-training on the expert dataset by minimizing the mean square error between its output and expert actions; S230: Directly transferring the network weights of the pre-trained artificial neural network to the actor network as its initial parameters. .

2. The single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to claim 1, characterized in that, In step S120, the multi-objective reward function The expression is: ; in, This represents a dense reward for tracking error, based on the normalized tracking deviation of photovoltaic power. Pre-defined piecewise linear mapping function; To represent the reward for suppressing power fluctuations, a sliding window with a length of 20 control cycles is introduced, and the standard deviation of photovoltaic power within the window is calculated in real time. With peak The normalized ripple score is then used to determine the ripple distribution. The range in which it is located is evaluated using a tiered fluctuation assessment based on pre-defined conditional logic. This represents a sparse reward for steady-state accuracy, if the current photovoltaic tracking error satisfies... Then a fixed positive reward will be given in one go. ; Indicates the safety penalty for smooth motion, and calculates the motion at the current moment. Actions in the previous moment absolute value of the difference ; This indicates a supervised boundary violation safety penalty, where a pre-deployed PI reference controller generates a reference action. Real-time comparison of agent outputs With reference action The value, if If the action is deemed potentially unsafe, a fixed penalty will be imposed. The forced strategy is to explore within a safe area; These represent the weight coefficients for dense tracking error reward, power fluctuation suppression reward, sparse steady-state accuracy reward, smooth motion safety penalty, and supervised boundary crossing safety penalty, respectively.

3. The single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to claim 1, characterized in that, In step S130, the actor-critic network structure consists of two hidden layers with 80 and 64 neurons respectively, both connected to ReLU activation functions. The output layer is a single neuron followed by a Tanh activation function, which linearly maps the action to [0,1] after constraining it to the [-1,1] interval. The critic network employs a dual-pathway structure, processing the state input and action input independently. Each input is mapped to a 48-dimensional feature space via a fully connected layer with 100 neurons. The two feature vectors are then concatenated and processed again by a fully connected layer with 48 neurons and ReLU activation, ultimately outputting a scalar state-action value. value.

4. The single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to claim 1, characterized in that, The S300 steps specifically include: S310: Build an online interactive system architecture with a high-performance computer as the strategy training center and a real-time control computer as the interface for real-time control and data interaction. The real-time control computer communicates with the dSPACE control desktop for execution control. Before starting online interactive learning, collect several state transition data from the ideal simulation environment to pre-fill the experience playback pool on the high-performance computer. S320: Establishes a wireless two-way communication link based on the TCP / IP protocol between a high-performance computer and a real-time control computer; S330: Executes a closed-loop online TD3 training cycle to continuously compensate for environmental nonlinear factors and cross-coupling effects, enabling the control strategy to continuously evolve and adapt in real-time in the real physical environment, thereby improving the adaptive performance in real working conditions.

5. The single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to claim 4, characterized in that, In step S330, during each control cycle To complete a learning and control loop, follow these steps sequentially: S331: Real-time control computer receives actor network data from high-performance computer. Based on the current state Calculated actions Superimposed with cropped Gaussian exploration noise Generate the final execution action ,in This represents the clipping function. This represents the hyperparameters and sends them to the dSPACE control desktop for execution, affecting the single-stage multi-port inverter system. S332: Current system status Transition to the next state Then, the real-time control computer calculates the immediate reward using a multi-objective reward function. ; S333: Will be determined by the current state ,action ,award and the next state A complete state transition tuple It is transmitted in real time to a high-performance computer via a TCP / IP link; S334: The high-performance computer receives the state transition tuple and appends it to the experience replay pool in a first-in-first-out manner. In this process, the simulated data in the pool is gradually replaced by real physical data; S335: High-performance computers from the experience replay pool Randomly sample small batches of samples and use the TD3 algorithm to perform multi-step gradient updates. S336: After completing this round of optimization, the high-performance computer will immediately update the actor network parameters. The data is transmitted to the real-time control computer via a TCP / IP link for action calculation in the next control cycle. Steps S331 to S335 are executed cyclically until the control process ends.

6. The single-stage multi-port inverter control method based on online interactive deep reinforcement learning according to claim 5, characterized in that, The S335 steps specifically include: The target action is computed using a target-oriented strategy with smooth regularization, and the temporal difference target value is computed by truncating the double-Q learning; the network parameters of the two critics are updated accordingly. , Then every interval Step 1: Update the actor's network parameters. and with soft update coefficient Smoothly update all target network parameters.

7. A single-stage multi-port inverter control system based on online interactive deep reinforcement learning, characterized in that, The method for controlling a single-stage multi-port inverter based on online interactive deep reinforcement learning, as described in any one of claims 1-6, comprises a single-stage multi-port inverter main circuit module, a photovoltaic power supply module, a battery energy storage module, an AC-side load, an LC filter module, a high-performance computer, a dSPACE control desktop, and a real-time control computer. The photovoltaic power supply module is connected to the upper DC port of the single-stage multi-port inverter, and the battery energy storage module is connected to the lower DC port of the single-stage multi-port inverter. The AC side of the single-stage multi-port inverter is connected to the load via the LC filter module. A control execution unit is connected to the drive signal terminal of the single-stage multi-port inverter to generate PWM control signals. The dSPACE control desktop is the real-time control execution unit and is connected to the real-time control computer. A bidirectional data interaction link is established between the high-performance computer and the real-time control computer via the TCP / IP protocol. The real-time control computer executes real-time signal acquisition and control command issuance, while the high-performance computer performs strategy iteration optimization and network parameter updates, collaboratively performing full-chain closed-loop adaptive control.

Images