Power electronic equipment efficient online reinforcement learning control system and method based on distributed architecture

Abstract

The invention discloses a power electronic equipment efficient online reinforcement learning control system and method based on a distributed architecture. According to the invention, a distributed system architecture in which online training and real-time control are separated is provided, and the fundamental contradiction between the computational intensive demand of reinforcement learning model training and the strong real-time demand of power electronic equipment control is fundamentally solved through functional decoupling and physical platform separation. A non-blocking asynchronous pipeline processing mechanism is designed for the online training subsystem, receiving and processing of high-frequency and small-batch empirical data and the model training process are decoupled through multi-thread parallel cooperation, and seamless and continuous online evolution of a control strategy in the system operation process is achieved. A special strategy network for power electronic equipment control is embedded in the real-time control subsystem, and a progressive parameter updating strategy is designed. On the premise of guaranteeing the punctual output of the control instruction, the online updating of the control strategy is realized.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of power electronic equipment control and artificial intelligence, specifically relating to an efficient online reinforcement learning control system and method for power electronic equipment based on a distributed architecture. Background Technology

[0002] With the rapid development of the energy internet, power electronic equipment (such as photovoltaic inverters, energy storage converters, and flexible DC transmission converter valves) plays an increasingly crucial role in energy conversion and distribution. These devices typically operate under complex and variable conditions, placing extremely high demands on the dynamic response speed, stability, and adaptability of the control system, with control step lengths often needing to be as short as hundreds of microseconds.

[0003] Traditionally, power electronic devices have primarily employed methods such as proportional-integral (PI) controllers and model predictive control (MPC). PI controllers are simple in structure and highly reliable, but their fixed parameters result in poor dynamic performance and difficulty in achieving optimal control when dealing with nonlinear, wide-range operating controlled objects. While MPC offers superior control performance, its effectiveness heavily relies on an accurate mathematical model of the controlled object. However, models of real-world power systems often contain uncertainties or are difficult to model accurately, limiting its application.

[0004] In recent years, artificial intelligence technology, especially deep reinforcement learning (DRL) technology which does not require precise models and has autonomous learning capabilities, has provided new ideas for the control of complex power electronic equipment. DRL agents learn optimal strategies through trial and error by interacting with the environment, and theoretically can adapt to complex nonlinear systems and achieve high-performance control. However, directly applying DRL to the real-time control of power electronic equipment faces severe challenges: a complete DRL training step (including environment interaction, empirical data sampling, gradient calculation, and network parameter updates) has enormous computational overhead and cannot be completed within extremely short control steps (e.g., 100 microseconds). Forcing training within the control cycle will directly lead to delays in control command output, disrupt system stability, and even cause equipment failure.

[0005] To address this contradiction, existing technologies typically employ an "offline training, online application" model. This involves first training the DRL model for an extended period in a simulation environment, then fixing the trained model parameters, and finally deploying it to actual equipment for static control. While this method avoids the risks of real-time training, it cannot adapt to system characteristic drift caused by device aging, load fluctuations, and changes in the external environment during actual operation. It also lacks online self-optimization capabilities, and control performance degrades over time.

[0006] Therefore, there is an urgent need in this field for an innovative technical solution that can enable power electronic equipment to adaptively solve various environmental changes and its own parameter deviations that occur during actual operation, while ensuring the millisecond / microsecond-level real-time control performance of power electronic equipment, so as to achieve long-term stable operation. Summary of the Invention

[0007] This invention aims to overcome the shortcomings of existing technologies and provide an efficient online reinforcement learning control method and system for power electronic devices based on a distributed architecture. By separating the training process from the control process on the physical platform and time scale, and designing corresponding data communication and processing mechanisms, it achieves continuous online optimization of the control strategy while firmly ensuring the strong real-time control of power electronic devices.

[0008] The technical solution adopted in this invention is:

[0009] An efficient online reinforcement learning control system for power electronic devices based on a distributed architecture, including

[0010] The real-time control subsystem, deployed on the real-time control platform, is responsible for high-frequency, real-time interaction and control with the power electronic equipment environment;

[0011] The online training subsystem, deployed on the algorithm computing platform, is responsible for performing online training of computationally intensive deep reinforcement learning algorithms.

[0012] The real-time transmission subsystem is responsible for establishing a reliable and efficient two-way data channel between the real-time control subsystem and the online training subsystem.

[0013] Furthermore, the real-time control subsystem includes an environment interaction module, a strategy execution module, and a data caching module.

[0014] Furthermore, the model structure of the online training subsystem includes an Actor policy network and a Critic value network. The Critic network adopts a two-branch input structure: the state branch receives the current state of the system, and the action branch receives the actions taken by the agent. The outputs of the two branches are fused in the subsequent hidden layer to finally output an action value Q. The online training process uses the received empirical data and the Q value calculated by the Critic network to guide and optimize the Actor policy network. After optimization, the system periodically sends the updated Actor network parameters back to the real-time control subsystem, thus forming a complete online learning closed loop.

[0015] An efficient online reinforcement learning control method for power electronic devices based on a distributed architecture includes the following steps:

[0016] S1. The real-time control subsystem collects the operating status of power electronic equipment in each microsecond-level control cycle, generates and executes control actions using the embedded strategy network, and caches the experience data generated by the interaction.

[0017] S2. The real-time transmission subsystem continuously and reliably transmits these high-frequency, small-batch experience data from the real-time control subsystem to the online training subsystem;

[0018] S3. The online training subsystem receives and stores experience data, performs asynchronous and batch model training based on deep reinforcement learning algorithms, and continuously optimizes policy network parameters.

[0019] S4. At fixed intervals, the optimized policy network parameters are sent back to the real-time control subsystem via the real-time transmission subsystem, which then safely updates the parameters without interrupting the real-time control task.

[0020] Furthermore, the implementation process of the real-time control subsystem is as follows:

[0021] Step 1.1: Within each preset very short control step, the strategy execution module reads the current operating state of the power electronic equipment. The strategy execution module has a built-in strategy network. The strategy network takes the current system state Si as input and the control action aj applied to the system as output. The action space and state space of the strategy network are configured according to the specific application scenario. The number of hidden layers and nodes are set according to the control accuracy requirements. The strategy network maps the state to the corresponding control action based on the current parameters and executes it. At the same time, it records the experience data generated by this interaction, including the current state, executed action, immediate reward, next state and termination flag.

[0022] Step 1.2: The data caching module is responsible for putting experience data into a local cache queue for the real-time transmission subsystem to send to the online training subsystem; and is also responsible for receiving updated network parameters from the training subsystem to update the network parameters in the policy execution module.

[0023] Furthermore, the online training subsystem adopts a non-blocking asynchronous pipeline processing mechanism, which is specifically implemented by the parallel collaboration of the data receiving thread, the data processing and storage thread, and the model training thread.

[0024] Furthermore, the implementation process of the non-blocking asynchronous pipeline is as follows:

[0025] Step 2.1: The data receiving thread runs independently and is specifically responsible for listening to the network port. It continuously receives raw experience data packets from the real-time control subsystem in a non-blocking I / O manner and puts them into a thread-safe first-in-first-out queue to ensure the continuity and low latency of data reception.

[0026] Step 2.2: The data processing and storage thread runs independently. It retrieves the original data packets from the queue, parses and verifies their validity, and then stores the processed experience data into a large circular experience playback buffer.

[0027] Step 2.3: The model training thread runs independently. It continuously samples batches of data randomly from the experience replay buffer, calculates the loss functions of the Actor network and Critic network in the TD3 or DDPG algorithm, and updates the network parameters through backpropagation. The operation of this thread is asynchronous and parallel to the data receiving and processing threads, and they do not block each other.

[0028] Furthermore, the data transmission process of the real-time transmission subsystem is as follows:

[0029] Step 3.1: Data Packet Construction and Transmission Strategy: When transmitting data, the sending end constructs data packets according to the following steps: First, a data structure containing a header identifier, a data body length field, a data body field, and a packet tail identifier is constructed; when transmitting large-scale data such as network parameters, the sending end first serializes the parameters and fills the data packet according to a preset fixed block size, making its length an integer multiple of the fixed block size, and explicitly identifies the original data length in the packet header; for high-frequency empirical data sent from the control end to the training end, the same unified data packet structure is used to ensure transmission reliability;

[0030] Step 3.2: Data block reception and accumulation strategy: The receiving end adopts a block reception mechanism, processing only one fixed-size data block in each control cycle, storing it in a buffer and accumulating it;

[0031] Step 3.3: Data packet verification and parsing strategy: When the total amount of received data reaches the length of the complete data packet indicated by the packet header, the system performs data packet verification, parsing and parameter reconstruction uniformly during the idle period of the control cycle.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] The technical effects of this invention are mainly reflected in the following aspects, which are verified through the following figures and embodiments:

[0034] 1. Achieved high-frequency and stable uplink transmission of experience data: For microsecond-level high-frequency experience data generated by the real-time control terminal, the distributed architecture and communication mechanism designed in this invention can ensure that the data is continuously, stably and without loss transmitted to the training terminal.

[0035] 2. Solved the real-time problem of large-scale parameter downlink: The innovative block reception and unpacking strategy of this invention effectively overcomes the challenge of resource-constrained real-time control terminals being unable to handle large-scale network parameter updates.

[0036] 3. Achieved online reinforcement learning training while ensuring strict real-time control: The core effect of this invention lies in successfully separating the computationally intensive model training task from the microsecond-level real-time control loop through a distributed architecture. In real-time simulation of a three-phase inverter, while the system stably executes control cycles every 100 microseconds, the background online training subsystem can continuously collect experience data, update the policy network, and synchronize parameters. During the training process, the real-time control task experiences no blocking or timeouts, as shown in the attached diagram. Figure 4 The reward curve shows that the reward value converges to the optimum, proving the effectiveness of online learning; see attached. Figure 5 As shown in the output waveform THD analysis, under the enhanced control of this distributed architecture, the output harmonic distortion is 1.35%, demonstrating the superior control performance. This fully verifies that the distributed architecture can achieve online and continuous performance self-optimization while ensuring the safe and stable operation of power electronic equipment. Attached Figure Description

[0037] Figure 1 This is a system block diagram of the present invention;

[0038] Figure 2 This is the Actor network structure in this invention;

[0039] Figure 3 This is the Critic network structure in this invention;

[0040] Figure 4 This is the reward curve of the present invention during model training;

[0041] Figure 5 It is an output waveform THD analysis; Detailed Implementation

[0042] To better understand the purpose, structure, and function of this invention, the invention will be described in further detail below with reference to the accompanying drawings.

[0043] This invention is an online reinforcement learning control technology suitable for strong real-time requirements. Specifically, it is a method and system that separates the reinforcement learning model training process from the real-time control process of the equipment on the physical platform and time scale, so as to realize continuous online optimization of power electronic equipment during operation.

[0044] In view of the fundamental contradiction between the high computational cost of online model training and the strong real-time requirements of equipment control when applying deep reinforcement learning (DRL) directly to the control of power electronic equipment, as described in the background art, this invention aims to solve the following specific technical problems:

[0045] 1. The contradiction between online training and real-time control in traditional DRL controllers: Traditional online training of DRLs is a computationally intensive task, taking far longer than the extremely short control steps (e.g., 100 microseconds) required by power electronic devices. Training cannot be completed within the control cycle, and forcing its implementation would compromise the real-time performance and stability of the control. However, the commonly adopted "offline training, online application" model, used to ensure real-time performance, renders the controller a "static" model that cannot adaptively optimize, making it difficult to cope with performance degradation caused by changes in system characteristics during actual operation. Therefore, the core technical problem this invention aims to solve is how to achieve continuous online learning and optimization of the control strategy without affecting the real-time control of the equipment.

[0046] This invention addresses this contradiction through a distributed architecture, separating computationally intensive training tasks from the real-time control loop and deploying them on independent computing nodes. However, this distributed solution introduces new technical challenges.

[0047] 2. Challenges in Reliable and Efficient Transmission and Processing of High-Frequency Bidirectional Data Streams in Distributed Architectures: To achieve the separation of training and control, a highly reliable data path needs to be established between the control subsystem and the training subsystem. This presents a challenge of bidirectional data streams, and the following two core issues need to be addressed.

[0048] 2.1 High-frequency, small-batch data transmission of experience data from the control end to the training end. The control end generates one piece of experience data at each step, forming a continuous data stream. The training subsystem requires an efficient, non-blocking mechanism to continuously receive, parse, and store massive amounts of small data packets, avoiding data loss or training pauses due to I / O blocking or processing delays.

[0049] 2.2 Reliable Reception of Large-Scale Parameter Update Packets from Training End to Control End. The parameter packet data generated during training is enormous, while the real-time control subsystem has limited computing resources. Attempting to receive and parse the entire parameter packet within a single microsecond control step would severely consume CPU resources and compromise real-time performance. Therefore, a mechanism is needed to enable the resource-constrained control end to smoothly receive and complete large-scale parameter updates without affecting the real-time control task.

[0050] To achieve the above objectives, the present invention adopts the following technical solution:

[0051] 1. A high-efficiency online reinforcement learning control system for power electronic devices based on a distributed architecture, including...

[0052] The real-time control subsystem, deployed on a real-time control platform (such as a real-time simulator or embedded hardware), is responsible for high-frequency, real-time interaction and control with the power electronic equipment environment.

[0053] The online training subsystem, deployed on a separate algorithm computing platform (such as a server), is responsible for performing online training of computationally intensive deep reinforcement learning algorithms.

[0054] The real-time transmission subsystem, based on network protocols such as TCP / IP, is responsible for establishing a reliable and efficient bidirectional data channel between the real-time control subsystem and the online training subsystem.

[0055] The real-time control subsystem and the online training subsystem are separated in terms of physical platform and time scale, deployed on different computing platforms, and connected by a network, with bidirectional data communication achieved by the real-time transmission subsystem.

[0056] The three subsystems work together through the following collaborative workflow (combined) Figure 1 As shown), this constitutes a complete online reinforcement learning loop:

[0057] (1) In each microsecond-level control cycle, the real-time control subsystem collects the operating status of the power electronic equipment through the environmental interaction module, generates and executes control actions using the strategy network embedded in the strategy execution module, and sends the experience data generated by the interaction to the data cache module for caching.

[0058] (2) The real-time transmission subsystem continuously and reliably transmits these high-frequency, small-batch experience data from the real-time control subsystem to the online training subsystem;

[0059] (3) The online training subsystem receives and stores experience data, performs asynchronous and batch model training based on deep reinforcement learning algorithms, and continuously optimizes policy network parameters.

[0060] (4) The optimized strategy network parameters are sent back to the real-time control subsystem through the real-time transmission subsystem at fixed intervals, and the subsystem can safely update the parameters without interrupting the real-time control task.

[0061] This architecture separates the training process from the control process on both the physical platform and the time scale, enabling collaboration between computationally intensive and real-time tasks. This fundamentally resolves the conflict between the two on the time scale, ensuring the feasibility of online learning and the stability of system control.

[0062] Furthermore, the three subsystems mentioned above are further refined:

[0063] 2. The real-time control subsystem specifically includes: an environment interaction module, a policy execution module, and a data caching module;

[0064] The implementation process is as follows:

[0065] Step 1.1: The strategy execution module reads the current operating state of the power electronic equipment within each preset very short control step. This module has a built-in policy network (Actor network), the structure of which is as follows: Figure 2 As shown, the network takes the current system state Si (such as voltage, current, and their reference values) as input and the control action aj applied to the system as output (such as PWM duty cycle). The network's action space and state space can be configured according to the specific application scenario, while the number of hidden layers and nodes is set according to the control accuracy requirements. The policy network maps the state to the corresponding control action based on the current parameters and executes it, while recording the experience data generated by this interaction, including the current state, executed action, immediate reward, next state, and termination flag.

[0066] Step 1.2: The data caching module is responsible for putting experience data into a local cache queue for the real-time transmission subsystem to send to the online training subsystem; and for receiving updated network parameters from the training subsystem to update the network parameters in the policy execution module.

[0067] 3. Online Training Subsystem: Deployed on an independent algorithm computing platform, this subsystem is responsible for executing online training of the deep reinforcement learning model. It receives empirical data from the real-time control subsystem via the real-time transmission subsystem and employs algorithms such as TD3 or DDPG as the core training algorithm. Its model structure includes an Actor policy network (consistent with the policy network described in step 1.1) and a Critic value network. The Critic network is as follows... Figure 3 As shown, it adopts a dual-branch input structure: the state branch receives the current state of the system, and the action branch receives the actions taken by the agent; the outputs of the two branches are fused in the subsequent hidden layer to finally output an action value Q. The online training process uses the received empirical data and the Q value calculated by the Critic network to guide and optimize the Actor policy network; after optimization, the system periodically sends the updated Actor network parameters back to the real-time control subsystem, thus forming a complete online learning loop.

[0068] To efficiently process the high-frequency, small-batch experience data streams transmitted from the control terminal, the online training subsystem adopts a non-blocking asynchronous pipeline processing mechanism, which is specifically implemented by the parallel collaboration of the data receiving thread, the data processing and storage thread, and the model training thread.

[0069] The implementation process of the non-blocking asynchronous pipeline is as follows:

[0070] Step 2.1: The data receiving thread runs independently and is specifically responsible for listening to the network port. It continuously receives raw experience data packets from the real-time control subsystem in a non-blocking I / O manner and puts them into a thread-safe first-in-first-out queue to ensure the continuity and low latency of data reception.

[0071] Step 2.2: The data processing and storage thread runs independently. It retrieves the original data packets from the queue, parses and verifies their validity, and then stores the processed experience data into a large circular experience playback buffer.

[0072] Step 2.3: The model training thread runs independently. It continuously samples batches of data randomly from the experience replay buffer, calculates the loss functions of the Actor network and Critic network in the TD3 or DDPG algorithm, and updates the network parameters through backpropagation. The operation of this thread is asynchronous and parallel to the data receiving and processing threads, and they do not block each other.

[0073] The innovation of this online training subsystem lies in combining deep reinforcement learning online training tasks with a specific asynchronous pipeline data processing mechanism. Through multi-threaded parallel collaboration, the reception, storage, and consumption of high-frequency data streams are logically decoupled, avoiding data loss or training interruption caused by delays in a single processing stage. This provides a stable and efficient data stream and training capability for the continuous online optimization of power electronic equipment.

[0074] 4. The real-time transmission subsystem is based on a communication protocol designed specifically for distributed reinforcement learning architectures. This protocol adapts to the constraints of high-frequency bidirectional data streams and real-time control resources through specific data packet structure design and transmission strategies.

[0075] The data transmission process of the real-time transmission subsystem is as follows:

[0076] Step 3.1: Data Packet Construction and Transmission Strategy: When transmitting data, the sending end constructs data packets according to the following steps: First, a data structure containing a header identifier, a data body length field, a data body field, and a packet tail identifier is constructed; when transmitting large-scale data such as network parameters, the sending end first serializes the parameters and fills the data packet according to a preset fixed block size, making its length an integer multiple of the fixed block size, and explicitly identifies the original data length in the packet header; for high-frequency empirical data sent from the control end to the training end, the same unified data packet structure is used to ensure transmission reliability.

[0077] Step 3.2: Data block reception and accumulation strategy: The receiving end adopts a block reception mechanism, which processes only one fixed-size data block in each control cycle, stores it in the buffer and accumulates it; this strategy is specifically designed for the resource constraints of the real-time control subsystem to ensure that large-scale parameter updates do not consume all the computing resources of a single control cycle.

[0078] Step 3.3: Data packet verification and parsing strategy: When the total amount of received data reaches the length of the complete data packet identified by the packet header, the system performs data packet verification (checking header and footer identifiers), parsing, and parameter reconstruction uniformly during the idle period of the control cycle. This strategy ensures that the parsing process does not block the real-time control task, while guaranteeing the integrity of data transmission.

[0079] The innovation of the real-time transmission subsystem lies in the fact that, through the specially optimized communication protocol and transmission strategy, it effectively solves the resource competition problem between high-frequency, small-batch experience data uploading and large-scale parameter downlink updates in the distributed reinforcement learning architecture. It is particularly suitable for the stringent real-time requirements of power electronic equipment control and provides communication assurance for efficient collaboration between the online training subsystem and the real-time control subsystem.

[0080] The innovation of this invention lies in:

[0081] Architectural Innovation: A distributed system architecture that separates online training from real-time control is proposed. By decoupling functions and separating the physical platform, the fundamental contradiction between the computationally intensive requirements of reinforcement learning model training and the strong real-time requirements (microsecond level) of power electronic equipment control is fundamentally resolved, laying the foundation for achieving continuous online optimization.

[0082] Innovative training mechanism: The online training subsystem is designed with a non-blocking asynchronous pipeline processing mechanism. Through multi-threaded parallel collaboration, the reception and processing of high-frequency, small-batch experience data are decoupled from the model training process, ensuring the stability and efficiency of the training data stream and realizing the seamless and continuous online evolution of the control strategy during system operation.

[0083] Real-time control innovation: The real-time control subsystem embeds a dedicated policy network (Actor network) for power electronic device control and designs a progressive parameter update strategy. This strategy ensures that the resource-constrained real-time control terminal can smoothly and without blocking receive and apply large-scale network parameter updates from the training terminal, realizing online updates of the control policy while ensuring timely output of control commands.

[0084] Collaborative Communication Innovation: The realization of the above core innovations benefits from the collaborative communication guarantee provided by the real-time transmission subsystem. Its customized communication protocol and targeted data packaging / unpacking strategy are key enabling technologies that support the reliable and efficient operation of the core process of high-frequency experience data uplink and large-scale parameter downlink updates.

[0085] This invention enables power electronic devices to maintain extremely high real-time control while possessing online self-learning and adaptive optimization capabilities, significantly improving their control performance, stability, and long-term robustness under complex and variable operating conditions.

[0086] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0087] This embodiment uses the real-time control of a three-phase inverter as the application scenario, employing the TD3 algorithm as the reinforcement learning algorithm. The online training subsystem is deployed in a Python environment (using the PyTorch framework) under the Ubuntu 22.04 operating system, and the real-time control subsystem is deployed on the RTscale real-time simulator. The two are connected via Gigabit Ethernet. Those skilled in the art should understand that the following embodiments are merely examples, and the application of this invention is not limited to this scenario.

[0088] 1. System Overall Architecture and Initialization

[0089] The system comprises three subsystems: an online training subsystem (Python interface), a real-time control subsystem (RTscale interface), and a real-time transmission subsystem (TCP / IP network). The system startup process is as follows:

[0090] Step S101: Initialize the online training subsystem. The online training subsystem is implemented in Python code, with the main_control() function as the main entry point. The system first initializes the TD3 agent, setting the hyperparameters: state_dim=6 (including reference current, output current, and current error), action_dim=3 (PWM duty cycle), learning rate actor_lr=0.0001, critic_lr=0.001, discount factor gamma=0.99, and experience pool capacity buffer_size=1,000,000. After the agent is created, it attempts to load an existing model for breakpoint retraining.

[0091] Step S102: Initialize the real-time transmission subsystem. The communication server starts in Python and binds its IP address and port. The server waits for RTscale to connect in blocking mode, with a timeout of 5 minutes. After the connection is established, both parties enter the data exchange phase.

[0092] Step S103: Start the data processing thread.

[0093] The online training subsystem creates the following threads:

[0094] Data receiving thread: Responsible for continuously receiving experience data packets sent by the RTscale end, running in non-blocking I / O mode, parsing data packets and storing them in a thread-safe queue.

[0095] Data processing thread: Retrieves experience data from the queue and adds it to the experience replay buffer.

[0096] Training thread: Periodically samples from the experience pool, calculates gradients, and updates TD3 network parameters. The training frequency is asynchronous with data reception.

[0097] Parameter sending thread: Packs up the updated Actor network parameters and sends them to the RTscale end at fixed intervals.

[0098] 2. Real-time control subsystem workflow (RTscale end)

[0099] The real-time control subsystem is implemented in the RTscale real-time simulator, with a control step size set to 100 microseconds. Its workflow is as follows:

[0100] Step S201: Environmental Interaction and Data Acquisition. Within each control step, the RTscale terminal reads the real-time state of the three-phase inverter, forming a state vector s_t. The embedded Actor network maps s_t to an action a_t (PWM duty cycle) based on the current parameters and executes this action to control the inverter. Simultaneously, the system calculates the immediate reward r_t (the negative value of the tracking error) and detects the termination flag done_t.

[0101] Step S202: Packaging and Sending Experience Data. The RTscale client packages the experience data tuples according to a custom protocol. The data packet structure includes: Header Magic Number (4 bytes, fixed identifier), Data Body Length (4 bytes), Data Body (converting floating-point numbers such as status, action, and reward into a uint32 array, little-endian), and Tail Magic Number (4 bytes, fixed identifier). After packaging, it is sent to the Python client in real time via TCP socket.

[0102] Step S203: Chunked reception and parsing of parameter update packets. When the Python side sends Actor network parameters, the RTscale side adopts a chunked reception strategy: data packets are padded to multiples of 100 uint32 bytes. The RTscale side receives only 100 uint32 data blocks in each control step and stores them in a buffer. The reception process is non-blocking, ensuring that control tasks take priority. When the cumulative received data volume reaches the full packet length, the packets are unpacked uniformly during the idle period of the control cycle, and the embedded Actor network parameters are updated after verifying the magic numbers at the beginning and end.

[0103] 3. Online Training Subsystem Workflow (Python Side)

[0104] The Python side employs a non-blocking asynchronous pipelined processing mechanism to ensure parallel training and data reception.

[0105] Step S301: Experience Data Reception and Parsing. The data receiving thread listens to the socket and continuously receives raw data packets. Each received complete packet is parsed: the magic number is checked, and fields such as status, action, and reward are extracted and converted into a numerical array. The parsed experience data is placed in a queue.

[0106] Step S302: Experience data storage. The data processing thread retrieves experience data from the queue and stores it in the experience pool. The experience pool uses a circular buffer structure, automatically overwriting old data when the data volume exceeds the capacity.

[0107] Step S303: Online training of the TD3 algorithm. The training thread performs network updates: randomly samples a batch of data from the experience pool, calculates the loss of the dual Critic network, and updates the Critic parameters through backpropagation; periodically updates the Actor network to maximize the Q value; and synchronizes the target network parameters using a soft update method. The training loss is recorded and monitored using TensorBoard.

[0108] Step S304: Parameter Packaging and Sending. The parameter sending thread periodically extracts parameters from the Actor network, flattens them into floating-point arrays in layer order, converts them to uint32 arrays and fills them to a multiple of 100 in length, constructs a sending packet and sends it to the RTscale end via socket.

[0109] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0110] It is understood that the present invention has been described through some embodiments, and those skilled in the art will recognize that various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, under the teachings of the present invention, these features and embodiments can be modified to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of the present invention.

Claims

1. A highly efficient online reinforcement learning control system for power electronic devices based on a distributed architecture, characterized in that: include The real-time control subsystem, deployed on the real-time control platform, is responsible for high-frequency, real-time interaction and control with the power electronic equipment environment; The online training subsystem, deployed on the algorithm computing platform, is responsible for performing online training of computationally intensive deep reinforcement learning algorithms. Real-time transmission subsystem; Responsible for establishing a reliable and efficient two-way data channel between the real-time control subsystem and the online training subsystem.

2. The efficient online reinforcement learning control system for power electronic devices based on a distributed architecture as described in claim 1, characterized in that: The real-time control subsystem includes an environment interaction module, a strategy execution module, and a data caching module.

3. The efficient online reinforcement learning control system for power electronic devices based on a distributed architecture as described in claim 1, characterized in that: The model structure of the online training subsystem includes an Actor policy network and a Critic value network. The Critic network adopts a two-branch input structure: the state branch receives the current state of the system, and the action branch receives the actions taken by the agent. The outputs of the two branches are fused in the subsequent hidden layer to finally output an action value Q. The online training process uses the received empirical data and the Q value calculated by the Critic network to guide and optimize the Actor policy network. After optimization, the system periodically sends the updated Actor network parameters back to the real-time control subsystem, thus forming a complete online learning closed loop.

4. A method for an efficient online reinforcement learning control system for power electronic devices based on a distributed architecture as described in any one of claims 1 to 3, characterized in that: Includes the following steps: S1. The real-time control subsystem collects the operating status of power electronic equipment in each microsecond-level control cycle, generates and executes control actions using the embedded strategy network, and caches the experience data generated by the interaction. S2. The real-time transmission subsystem continuously and reliably transmits these high-frequency, small-batch experience data from the real-time control subsystem to the online training subsystem; S3. The online training subsystem receives and stores experience data, performs asynchronous and batch model training based on deep reinforcement learning algorithms, and continuously optimizes policy network parameters. S4. At fixed intervals, the optimized policy network parameters are sent back to the real-time control subsystem via the real-time transmission subsystem, which then safely updates the parameters without interrupting the real-time control task.

5. The efficient online reinforcement learning control method for power electronic devices based on a distributed architecture as described in claim 4, characterized in that: The implementation process of the real-time control subsystem is as follows: Step 1.1: Within each preset very short control step, the strategy execution module reads the current operating state of the power electronic equipment. The strategy execution module has a built-in strategy network. The strategy network takes the current system state Si as input and the control action aj applied to the system as output. The action space and state space of the strategy network are configured according to the specific application scenario. The number of hidden layers and nodes are set according to the control accuracy requirements. The strategy network maps the state to the corresponding control action based on the current parameters and executes it. At the same time, it records the experience data generated by this interaction, including the current state, executed action, immediate reward, next state and termination flag. Step 1.2: The data caching module is responsible for putting experience data into a local cache queue for the real-time transmission subsystem to send to the online training subsystem; and is also responsible for receiving updated network parameters from the training subsystem to update the network parameters in the policy execution module.

6. The efficient online reinforcement learning control method for power electronic devices based on a distributed architecture as described in claim 4, characterized in that: The online training subsystem adopts a non-blocking asynchronous pipeline processing mechanism, which is specifically implemented by the parallel collaboration of the data receiving thread, the data processing and storage thread, and the model training thread.

7. The efficient online reinforcement learning control method for power electronic devices based on a distributed architecture as described in claim 6, characterized in that: The implementation process of the non-blocking asynchronous pipeline is as follows: Step 2.1: The data receiving thread runs independently and is specifically responsible for listening to the network port. It continuously receives raw experience data packets from the real-time control subsystem in a non-blocking I / O manner and puts them into a thread-safe first-in-first-out queue to ensure the continuity and low latency of data reception. Step 2.2: The data processing and storage thread runs independently. It retrieves the original data packets from the queue, parses and verifies their validity, and then stores the processed experience data into a large circular experience playback buffer. Step 2.3: The model training thread runs independently. It continuously samples batches of data randomly from the experience replay buffer, calculates the loss functions of the Actor network and Critic network in the TD3 or DDPG algorithm, and updates the network parameters through backpropagation. The operation of this thread is asynchronous and parallel to the data receiving and processing threads, and they do not block each other.

8. The efficient online reinforcement learning control method for power electronic devices based on a distributed architecture as described in claim 4, characterized in that: The data transmission process of the real-time transmission subsystem is as follows: Step 3.1: Data Packet Construction and Transmission Strategy: When transmitting data, the sending end constructs data packets according to the following steps: First, a data structure containing a header identifier, a data body length field, a data body field, and a packet tail identifier is constructed; when transmitting large-scale data such as network parameters, the sending end first serializes the parameters and fills the data packet according to a preset fixed block size, making its length an integer multiple of the fixed block size, and explicitly identifies the original data length in the packet header; for high-frequency empirical data sent from the control end to the training end, the same unified data packet structure is used to ensure transmission reliability; Step 3.2: Data block reception and accumulation strategy: The receiving end adopts a block reception mechanism, processing only one fixed-size data block in each control cycle, storing it in a buffer and accumulating it; Step 3.3: Data packet verification and parsing strategy: When the total amount of received data reaches the length of the complete data packet indicated by the packet header, the system performs data packet verification, parsing and parameter reconstruction uniformly during the idle period of the control cycle.

Images