Network construction type converter control method based on deep reinforcement learning and related equipment

Abstract

This application provides a control method and related equipment for a grid-connected converter based on deep reinforcement learning, belonging to the field of power electronics technology. The method includes: acquiring the current active power, current active power change rate, current reactive power, and current output voltage of the grid-connected converter, as well as the current angular frequency change and current angular frequency change rate of the power grid, to form a state observation vector; inputting the state observation vector into a pre-trained target deep reinforcement learning model for forward inference to obtain optimal virtual synchronous generator control parameter values; analyzing the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power, and current output voltage of the grid-connected converter using a virtual synchronous generator control strategy to generate a three-phase reference voltage; and controlling the grid-connected converter based on the three-phase reference voltage. This application can improve the adaptability and stability of the grid-connected converter under power grid operating conditions.

Description

Technical Field

[0001] This application relates to the field of power electronics technology, and in particular to a grid-type converter control method and related equipment based on deep reinforcement learning. Background Technology

[0002] Grid-based converters actively provide voltage and frequency support to the power grid by simulating the operating mechanism of synchronous generators, fundamentally solving the problem of system strength degradation caused by the high proportion of new energy integration. Virtual synchronous generator technology is the most typical solution for realizing grid-based control. However, the control parameters of virtual synchronous generators mainly rely on engineers' experience and offline simulation to be predetermined. With the continuous increase in the penetration rate of intermittent power sources such as wind power and photovoltaics, the fixed-parameter virtual synchronous generator control strategy is not effective in dealing with the increasingly complex and variable power grid operating conditions. Summary of the Invention

[0003] The main purpose of this application is to propose a control method and related equipment for grid-type converters based on deep reinforcement learning, which can improve the adaptability and stability of grid-type converters under grid operating conditions.

[0004] To achieve the above objectives, one aspect of this application proposes a network converter control method based on deep reinforcement learning, the method comprising: The current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the grid, are obtained to form a state observation vector. The state observation vector is input into the pre-trained target deep reinforcement learning model for forward inference to obtain the optimal virtual synchronous generator control parameter values. Based on the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter, a virtual synchronous generator control strategy is used for analysis to generate a three-phase reference voltage. The grid-type converter is controlled based on the three-phase reference voltage.

[0005] Furthermore, the current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter are obtained in the following way: Obtain the current three-phase voltage and current three-phase current at the grid connection point; The current three-phase voltage and current three-phase current at the grid connection point are sequentially subjected to Clark transformation and Park transformation to obtain the current dq-axis voltage and current dq-axis current. The current active power, current reactive power, and current output voltage of the grid-type converter are calculated based on the current dq-axis voltage and the current dq-axis current. The historical active power of the grid-type converter, calculated in the previous control cycle, is retrieved and combined with the current active power of the grid-type converter to calculate the current active power change rate of the grid-type converter.

[0006] Furthermore, the current angular frequency change and the current angular frequency change rate of the power grid are obtained in the following way: The current angular frequency of the power grid is obtained, and then the change in the current angular frequency of the power grid is calculated in combination with the rated angular frequency of the power grid. The historical angular frequency of the power grid obtained in the previous control cycle is retrieved and then combined with the current angular frequency of the power grid to calculate the current angular frequency change rate of the power grid.

[0007] Furthermore, the target deep reinforcement learning model is obtained in the following way: A virtual synchronous generator control grid-connected converter model was built in the MATLAB / Simulink environment, and various disturbance scenarios were set to form a simulation environment; In the simulation environment, the pre-constructed initial deep reinforcement learning model is interactively trained using the deep deterministic policy gradient algorithm to obtain the target deep reinforcement learning model.

[0008] Furthermore, the step of analyzing the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power, and current output voltage of the grid-connected converter using a virtual synchronous generator control strategy to generate a three-phase reference voltage includes: Based on the optimal virtual synchronous generator control parameter values, the parameters of the pre-built virtual synchronous generator controller are updated; The newly determined active power reference value, reactive power reference value, and output voltage reference value of the virtual synchronous generator are called, and together with the current active power, current reactive power, and current output voltage of the grid-type converter, they are input to the updated virtual synchronous generator controller for calculation to obtain the virtual power angle and virtual electromotive force. The three-phase reference voltage is synthesized based on the virtual power angle and the virtual electromotive force.

[0009] Furthermore, controlling the grid-type converter based on the three-phase reference voltage includes: Based on the three-phase reference voltage, a pulse width modulation signal is generated; The switching transistors included in the grid converter are controlled to turn on and off according to the pulse width modulation signal.

[0010] To achieve the above objectives, another aspect of this application proposes a network converter control system based on deep reinforcement learning, the system comprising: The first module is used to obtain the current active power, current active power change rate, current reactive power and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the grid, in order to form a state observation vector. The second module is used to input the state observation vector into the pre-trained target deep reinforcement learning model for forward inference to obtain the optimal virtual synchronous generator control parameter values. The third module is used to analyze the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter using a virtual synchronous generator control strategy to generate a three-phase reference voltage. The fourth module is used to control the grid-type converter based on the three-phase reference voltage.

[0011] To achieve the above objectives, another aspect of this application proposes an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-described deep reinforcement learning-based network converter control method.

[0012] To achieve the above objectives, another aspect of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described deep reinforcement learning-based network converter control method.

[0013] To achieve the above objectives, another aspect of this application proposes a computer program product comprising a computer program that, when executed by a processor, implements the aforementioned deep reinforcement learning-based network converter control method.

[0014] This application includes at least the following beneficial effects: By introducing an adaptive control mechanism, the current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the grid, are first obtained to form a state observation vector. Then, the state observation vector is input into a pre-trained target deep reinforcement learning model for forward inference, so that the virtual synchronous generator control parameters can be dynamically adjusted with the system state. Then, based on the optimal virtual synchronous generator control parameter values ​​obtained by inference and the current active power, current reactive power, and current output voltage of the grid-type converter, a virtual synchronous generator control strategy is used for analysis. Subsequently, the grid-type converter is controlled based on the three-phase reference voltage generated by the analysis. Compared with the prior art, this can improve the adaptability and stability of the grid-type converter under grid operating conditions. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating a network converter control method based on deep reinforcement learning provided in an embodiment of this application. Figure 2 This is a schematic diagram of the composition of a network converter control system based on deep reinforcement learning provided in an embodiment of this application; Figure 3 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of systems and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0017] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0018] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0020] Microgrids, as the core form of highly reliable distributed energy systems, are crucial for building modern energy systems. Among them, grid-forming converters (GFMs) actively provide voltage and frequency support to the grid by simulating the operating mechanisms of synchronous generators (such as inertia, damping, and frequency and voltage regulation characteristics), fundamentally solving the problem of system strength degradation caused by the high proportion of renewable energy integration. Virtual synchronous generator (VSG) technology is the most typical solution for realizing grid-forming control; however, the control parameters of VSGs mainly rely on engineers' experience and offline simulations to be predetermined. With the increasing penetration rate of intermittent power sources such as wind and solar power, the fixed-parameter VSG control strategy is not effective in dealing with increasingly complex and variable grid operating conditions. For example, a large virtual inertia value helps suppress the rate of frequency change but increases frequency deviation and recovery time. A large virtual damping value helps suppress oscillations but slows down the response speed. It is evident that fixed parameters cannot achieve dynamic optimal balance during transient processes, resulting in long system recovery time or large overshoot. Under weak grid conditions or different grid impedances, virtual synchronous generators with fixed parameters are prone to interacting with the grid, causing subsynchronous or high-frequency oscillations, which threaten system safety. When the system operating point deviates from the design conditions, the controller performance degrades significantly.

[0021] In view of this, embodiments of this application provide a grid-type converter control method and related equipment based on deep reinforcement learning. This scheme introduces an adaptive control mechanism, first acquiring the current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the power grid, to form a state observation vector. Then, the state observation vector is input into a pre-trained target deep reinforcement learning model for forward inference, enabling the virtual synchronous generator control parameters to be dynamically adjusted according to the system state. Finally, based on the optimal virtual synchronous generator control obtained through inference... The system analyzes the control parameters, current active power, current reactive power, and current output voltage of the grid-connected converter using a virtual synchronous generator control strategy. Then, it controls the grid-connected converter based on the three-phase reference voltage generated by the analysis. Compared with existing technologies, this avoids the reliance on expert experience and cumbersome simulation parameter tuning process, and can resolve the contradiction between transient and steady-state performance of fixed parameters. This improves the adaptability and stability of the grid-connected converter under grid operating conditions, enabling it to adaptively operate at the optimal or near-optimal operating point under various complex scenarios such as load fluctuations and fault disturbances.

[0022] This application provides a deep reinforcement learning-based network converter control method, relating to the field of power electronics technology. It can be applied to terminals, servers, or software running on either a terminal or server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc., but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the above method, but is not limited to these forms.

[0023] Please see Figure 1 , Figure 1 This is an optional flowchart illustrating a deep reinforcement learning-based network converter control method provided in an embodiment of this application. The method may include, but is not limited to, the following steps S101 to S104: Step S101: Obtain the current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the power grid, to form a state observation vector; Step S102: Input the state observation vector into the pre-trained target deep reinforcement learning model for forward inference to obtain the optimal virtual synchronous generator control parameter values; Step S103: Based on the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter, the virtual synchronous generator control strategy is used for analysis to generate a three-phase reference voltage. Step S104: Control the grid-type converter according to the three-phase reference voltage.

[0024] Steps S101 to S104 as shown in the embodiments of this application introduce a deep reinforcement learning (DRL) algorithm, which enables the control parameters of the virtual synchronous generator to be dynamically adjusted according to the system state, thereby improving the adaptability and stability of the grid-connected converter under grid operating conditions.

[0025] In step S101 of some embodiments, the grid-connected converter can adopt a three-phase LCL grid-connected inverter topology to realize power conversion. The process of determining the current active power, current active power change rate, current reactive power and current output voltage of the grid-connected converter may include, but is not limited to, steps S201 to S204, as specifically explained below.

[0026] Step S201: Obtain the current three-phase voltage and current three-phase current at the grid connection point. This grid connection point can be understood as the common coupling point between the power grid and the grid-connected converter. Specifically, the original three-phase voltage and original three-phase current at the grid connection point are collected by Hall sensors at the current moment, and then the original three-phase voltage and original three-phase current at the grid connection point at the current moment are filtered to remove high-frequency noise, thereby obtaining the current three-phase voltage and current at the grid connection point.

[0027] Step S202: Perform Clark transformation and Park transformation on the current three-phase voltage and current three-phase current at the grid connection point respectively to obtain the current dq axis voltage and current dq axis current; Specifically, Clark transformation is performed on the current three-phase voltage and current at the grid connection point to obtain the current αβ-axis voltage and current, which can be achieved using the following mathematical expression: , ; In the formula, , and Collectively referred to as the current three-phase voltage, and Collectively referred to as the current αβ axis voltage, , and Collectively referred to as the current three-phase current, and Collectively referred to as the current αβ axis current, The current moment; The current dq-axis voltage and current can be obtained by performing Park transformations on the current αβ-axis voltage and current respectively, which can be achieved using the following mathematical expressions: , ; In the formula, and Collectively referred to as the current dq axis voltage, and Collectively referred to as the current dq-axis current, The virtual power angle is generated by the VSG control strategy adopted in the previous control cycle. To control the cycle, it is preferable to set it to 1ms.

[0028] Step S203: Calculate the current active power, current reactive power, and current output voltage of the grid-type converter based on the current dq-axis voltage and current dq-axis current. This can be achieved using the following mathematical expressions: , , ; In the formula, This represents the current active power of the grid-type converter. This represents the current reactive power of the grid-connected converter. This is the current output voltage of the grid-type converter.

[0029] Step S204: The historical active power of the grid-type converter calculated in the previous control cycle is retrieved, and then combined with the current active power of the grid-type converter to calculate the current active power change rate of the grid-type converter. This can be achieved using the following mathematical expression: ; In the formula, The current active power change rate of the grid-type converter. This refers to the historical active power of the grid-type converter.

[0030] In some embodiments, the process of determining the current angular frequency change and the current angular frequency change rate of the power grid in step S101 may include, but is not limited to, steps S301 to S302, as detailed below.

[0031] Step S301: Obtain the current angular frequency of the power grid, and then calculate the change in the current angular frequency of the power grid by combining it with the rated angular frequency of the power grid. Specifically, this can be achieved using the following mathematical expression: ; In the formula, This represents the current change in the angular frequency of the power grid. The current angular frequency of the power grid. This is the rated angular frequency of the power grid, typically taken as 314 rad / s.

[0032] Step S302: The historical angular frequency of the power grid acquired in the previous control cycle is retrieved and combined with the current angular frequency of the power grid to calculate the current angular frequency change rate of the power grid. This can be achieved using the following mathematical expression: ; In the formula, The current rate of change of the angular frequency of the power grid. This represents the historical angular frequency of the power grid.

[0033] In step S102 of some embodiments, the training method for the target deep reinforcement learning model may include, but is not limited to, the following: first, building a virtual synchronous generator control grid-connected converter grid-connected model in the MATLAB / Simulink environment and setting various disturbance scenarios to form a simulation environment, including load change scenarios, voltage drop scenarios, grid impedance change scenarios, etc.; then, using the Deep Deterministic Policy Gradient (DDPG) algorithm in the simulation environment to interactively train the pre-built initial deep reinforcement learning model to obtain the target deep reinforcement learning model, wherein the initial deep reinforcement learning model is built using the existing Actor-Critic architecture, the Actor network (i.e., the policy network) is used to output the optimal action according to the input state, and the Critic network (i.e., the value network) is used to evaluate the long-term value generated by the input state-optimal action to guide the update of the Actor network.

[0034] Specifically, the VSG parameter optimization problem is modeled as a standard Markov decision process. The core elements involved in this process include the state space, action space, and reward function, where: The state space is defined as follows: ; In the formula, Refers to the state space, used to comprehensively characterize the system's operating conditions. This represents the change in the angular frequency of the power grid. The rate of change of the angular frequency of the power grid. The active power output of the grid-type converter. The voltage output by the grid-type converter. The active power change rate of a grid-type converter. The reactive power output of the grid-connected converter. It is the transpose symbol; Define the action space as: ; In the formula, Refers to the action space, which is the set of optimal control parameters used to comprehensively represent the output of a DRL (Deep Reinforcement Learning) agent. The preferred safe range for virtual inertia and its physical feasibility is... Simulate the rotational inertia of a synchronous generator rotor. The larger the value, the slower the system frequency changes and the better the inertia. The smaller the value, the faster the frequency response. The preferred safe range for a virtual damping coefficient and its physical feasibility is... Simulate the damping effect on a synchronous generator. The larger the value, the faster the system power oscillation subsides. The preferred active-frequency droop factor and its physically feasible safe range are... , The reactive power-voltage droop factor, and its physically feasible safe range, is preferably set to . ; The reward function is used to guide the DRL agent in learning the target quantitative index. It can be designed as a weighted sum of system stability and power quality indicators, aiming to penalize deviations, oscillations, and exceeding limits, and reward rapid recovery and stable operation. Its corresponding mathematical expression is: ; In the formula, Refers to the reward function, for The weight corresponding to the item is preferably set to 1.0. for The weight corresponding to the item is preferably set to 0.5. The voltage deviation of a grid-connected converter can be understood as the difference between the output voltage of the grid-connected converter and the no-load rated voltage. for The weight corresponding to the item is preferably set to 0.8. for The weight corresponding to the item is preferably set to 0.3. This is the oscillation suppression value. for The weight corresponding to the item is preferably set to 0.6. To reward fine-tuning values. Under normal circumstances, it is necessary to... item, item, item, Item and Each item is normalized to remove dimensions, and then weighted summation and fine-tuning are performed to obtain the reward value.

[0035] Optionally, for oscillation suppression value The determination process includes: if the oscillation amplitude of the active power output of the grid-type converter exceeds the preset power oscillation amplitude threshold during the current control cycle, then adopt... This formula calculates the initial oscillation suppression value, where, This is the initial oscillation suppression value. This is a preset power penalty coefficient used to characterize the intensity of power oscillation penalty. This is the moving average power, used to filter out steady-state fluctuations. The preset power oscillation amplitude threshold can be set to 5% of the rated active power. If power oscillation is also detected in the previous control cycle, the final calculated oscillation suppression value from the previous control cycle will be compared with the initial oscillation suppression value. The values ​​are accumulated to obtain the final calculated oscillation suppression value within the current control cycle. If no power oscillation is detected in the previous control cycle, the initial oscillation suppression value will be adjusted. This value is directly used as the final oscillation suppression value calculated within the current control cycle. .

[0036] Optionally, for oscillation suppression value The determination process includes: if the angular frequency change of the power grid is detected to exceed the preset angular frequency change threshold during the current control cycle, then... This formula calculates the initial oscillation suppression value, where, This is the initial oscillation suppression value. This is a preset angular frequency penalty coefficient, used to characterize the intensity of the angular frequency oscillation penalty. This is the preset critical value for angular frequency change; if angular frequency oscillation is also detected in the previous control cycle, the final calculated oscillation suppression value from the previous control cycle will be compared with the initial oscillation suppression value. The values ​​are accumulated to obtain the final calculated oscillation suppression value within the current control cycle. If no angular frequency oscillation is detected in the previous control cycle, the initial oscillation suppression value will be adjusted. This value is directly used as the final oscillation suppression value calculated within the current control cycle. .

[0037] Adjustment value for rewards The determination method includes: when all indicators are detected to fall within their respective corresponding safety ranges, all indicators include , , and Adjust the reward value The stability maintenance reward value is set to 1.0; when the grid quickly recovers stability after detecting a disturbance, the reward value is fine-tuned. Set the rapid recovery reward value, preferably 0.5; when at least one key indicator is detected to exceed its corresponding safety range, the key indicators include... , and , Exceeding the limit may trigger protection actions or cause the load to disconnect from the grid. Both excessively high and low temperatures may cause equipment damage or system crashes. Exceeding the limit may trigger frequency protection in a weak power grid, resulting in a reward fine-tuning value. Set the penalty value for violations, with a preferred setting of -2.0.

[0038] It should be noted that rapid recovery to stability after a grid disturbance refers to the grid system's ability to smoothly return to a steady-state operating state within a short period of time after experiencing external disturbances such as load surges, voltage drops, or short-circuit faults. This can be measured by at least one of the following dimensions: recovery time, overshoot, and number of oscillations. Specifically: recovery time refers to the time elapsed from the occurrence of the disturbance until all indicators recover to their respective safe ranges. A rapid recovery to stability after a grid disturbance is determined when the recovery time is less than or equal to a preset baseline recovery time. Overshoot can be either power overshoot or voltage overshoot. Power overshoot refers to the maximum deviation of the active power output of the grid-connected converter during the recovery process, while voltage overshoot refers to the maximum deviation of the voltage output of the grid-connected converter during the recovery process. A rapid recovery to stability after a grid disturbance is determined when either the power overshoot is less than or equal to a preset power deviation threshold, or when the voltage overshoot is less than or equal to a preset voltage deviation threshold. The number of oscillations refers to the number of repeated fluctuations in the grid during the recovery process. A rapid recovery to stability after a grid disturbance is determined when the number of oscillations is less than or equal to a preset allowable number of oscillations.

[0039] In some embodiments, the initial deep reinforcement learning model autonomously learns a general mapping strategy from the state of a complex system to the optimal control parameters through an interactive-store-learning loop in the simulation environment. This interactive-store-learning loop includes an initialization phase, a round-based interaction and experience collection phase, and a policy learning and network update phase. The specific descriptions of each of the above phases are as follows.

[0040] During the initialization phase, an experience replay pool is created, and the Actor network, Critic network, and experience replay pool are initialized. At the same time, the simulation environment is reset to its initial state.

[0041] In the round-based interaction and experience collection phase, model training is performed cyclically in rounds. Each round begins with a reset of the simulation environment. After obtaining the initial state observation vector, the DRL agent performs the following operations in each discrete time step within each round: First, the Actor network is used to observe the vector based on the current state. Output a basic action vector To encourage exploration, in this basic action vector Add random noise This random noise Gaussian noise can be used to form the current action vector. Available This expression achieves the goal; Next, the current action vector The virtual synchronous generator control grid-connected converter model is applied to this simulation environment, and then the simulation is advanced by one step to calculate the new state observation vector. And calculate the instant reward based on the preset reward function. Simultaneously, it determines whether the current discrete time step leads to round termination, i.e., whether system instability occurs in the current discrete time step or whether the current discrete time step is the preset maximum discrete time step, in order to generate a termination flag. Optionally, if it causes the round to terminate, then set If it does not cause the round to terminate, then set ; The complete empirical tuples will then be generated through interaction in the current discrete time step. The data is stored in the experience replay pool, which serves as a data buffer to store a large amount of historical interaction experience data for subsequent offline learning.

[0042] The above process is repeated in a loop within a single round until the round ends. Then the simulation environment is reset and the training for the next round begins. Through a large number of rounds of interaction, the experience replay pool accumulates rich experience data on successful and unsuccessful cases under various working conditions, laying a data foundation for subsequent strategy learning.

[0043] During the policy learning and network update phase, when the amount of data accumulated in the experience replay pool reaches the preset batch size, training enters the learning and update phase. The DRL agent randomly samples a batch of historical experience data from the experience replay pool and uses the existing DDPG algorithm to optimize its Actor and Critic networks in parallel. A batch of historical experience data contains... There are 1 historical experience data point, and each historical experience data point is represented as . , The corresponding network optimization methods include the following: (1) With the objective of minimizing the mean squared error between the value predicted by the current Critic network and the target value, the parameters of the current Critic network are updated through backpropagation. The parameters of the current Critic network can be the bias of each neuron in the current Critic network and the connection weight between each pair of neurons, so that the value estimate generated by the current Critic network is closer to the real return. The corresponding objective function is: , in: ; In the formula, Refers to the objective function required for Critic network optimization. For the first Historical experience data The corresponding target value, Refers to the current Critic network. Represents the observation vector of the current state. - Current action vector The value lies in the observation of the current state vector. and the current action vector The predictions are obtained by inputting the data into the current Critic network. As a discount factor, The new action vector is obtained by transforming the new state observation vector. The results are obtained by inputting the data into the target Actor network and analyzing it. Refers to the target Critic network, which is usually a delayed copy of the Critic network used for stable training. Represents the observation vector of the new state. - New final motion vector The value lies in the new state observation vector. and the new final action vector The prediction is obtained by inputting the data into the target Critic network. (2) To maximize the value estimated by the current Critic network, the parameters of the current Actor network are updated using the gradient ascent method. The parameters of the current Actor network can be the bias of each neuron in the current Actor network and the connection weights between each pair of neurons, so that the action vector output by the current Actor network can obtain the maximum value estimated by the current Critic network. The corresponding objective function is: ; Here, the gradient formula for updating the parameters of the current Actor network, determined according to the chain rule, is as follows: ; In the formula, Refers to the objective function required for optimizing the Actor network. Refers to the parameters of the current Actor network. Refers to the parameters of the current Critic network. Refers to the current state observation vector The action vector is obtained by inputting it into the current Actor network for analysis. Refers to the current state observation vector and the action vectors obtained from the analysis The value obtained by inputting the data into the current Critic network for prediction. Represents the objective function gradient, This represents the action vector obtained from the analysis of the current Critic network. gradient, That is , This represents the gradient of the current Actor network with respect to its parameters.

[0044] To improve training stability, this application employs a soft update approach to slowly track the parameters of both the target Actor network and the target Critic network. This is achieved using the following mathematical expression: , ; In the formula, Refers to the parameters of the target Actor network. The parameters refer to the target Critic network. For preset update coefficients and .

[0045] In some embodiments, to ensure that the target deep reinforcement learning model can run efficiently on resource-constrained embedded hardware (such as DSPs and FPGAs), some optimizations can be performed on the target deep reinforcement learning model before deployment. Specifically, this includes: converting the weights and activation values ​​involved in the target deep reinforcement learning model from FP32 floating-point precision during training to FP16 floating-point precision or INT8 fixed-point precision to obtain a quantized target deep reinforcement learning model, thereby reducing model storage usage and improving model computation speed; then, by analyzing the importance of each weight involved in the quantized target deep reinforcement learning model, redundant neurons or weight connections are structurally removed to obtain a pruned target deep reinforcement learning model, thereby compressing the model size and reducing model computational complexity while basically maintaining model performance. Based on this, the pruned target deep reinforcement learning model is hardware-aware compiled using a dedicated toolchain (such as TensorRT for NVIDIA Jetson) and integrated into the real-time inference engine of the embedded hardware. This real-time inference engine typically runs as a high-priority interrupt service routine, and its execution cycle is generally strictly synchronized with the control cycle set by the network converter control system.

[0046] In some embodiments, if it is detected that the optimal virtual synchronous generator control parameter values ​​determined in step S102 all fall within the range of [-1, 1], then the optimal virtual synchronous generator control parameter values ​​need to be mapped according to the safety range of the virtual synchronous generator control parameters before they can be applied to step S103. The numerical mapping formula for each virtual synchronous generator control parameter is as follows: ; In the formula, The optimal virtual synchronous generator control parameter values ​​falling within the range [-1, 1] are... These are the optimal virtual synchronous generator control parameter values ​​after mapping. This refers to the safe range of the control parameters for a virtual synchronous generator, which include virtual inertia. Virtual damping coefficient Active power-frequency droop coefficient and reactive power-voltage droop factor .

[0047] In step S103 of some embodiments, the analysis of generating a three-phase reference voltage by using a virtual synchronous generator control strategy based on the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter can be implemented in ways that include, but are not limited to, steps S401 to S403, as detailed below.

[0048] Step S401: Update the parameters of the pre-constructed virtual synchronous generator controller according to the optimal virtual synchronous generator control parameter values. The basic principle of the virtual synchronous generator controller is to simulate the electromechanical transient characteristics of a synchronous generator, enabling the power electronic converter to autonomously support the power grid. It internally includes an active-frequency control loop and a reactive-voltage control loop, which are explained below: (1) The active-frequency control loop involves rotor motion characteristic elements and a virtual speed governor. The corresponding rotor motion equation is: ; in: ; In the formula, The output angular frequency of the VSG. The input mechanical power of the VSG, For VSG's virtual power angle, This is the reference value for the active power of the VSG; (2) In the reactive power-voltage control loop, the regulation of the output reactive power to the terminal voltage is mainly achieved by simulating the excitation regulation of the synchronous generator. The expression used in this implementation process is: ; In the formula, The virtual electromotive force output by VSG. This is the reactive power reference value for VSG. The preset voltage regulation reactive power coefficient, This is the reference value for the output voltage of the VSG.

[0049] Step S402: Call the latest determined active power reference value, reactive power reference value, and output voltage reference value of the virtual synchronous generator, and input them together with the current active power, current reactive power, and current output voltage of the grid-type converter to the updated virtual synchronous generator controller for calculation to obtain the virtual power angle and virtual electromotive force; wherein, the active power reference value, reactive power reference value, and output voltage reference value of the virtual synchronous generator can be provided by the upper-level dispatch or set locally.

[0050] Step S403: Based on the virtual power angle and virtual electromotive force, synthesize the three-phase reference voltage, which can be achieved using the following mathematical expression: ; In the formula, This is the three-phase reference voltage.

[0051] In step S104 of some embodiments, the content regarding controlling the grid-type converter according to the three-phase reference voltage can be implemented in ways that include, but are not limited to: first generating a pulse width modulation (PWM) signal based on the three-phase reference voltage; specifically, using an existing PWM modulator to convert the three-phase reference voltage into a corresponding pulse width modulation signal; and then controlling the switching transistors included in the grid-type converter based on the pulse width modulation signal to achieve grid-type support for the power grid.

[0052] In some embodiments, steps S101 to S104 are executed cyclically according to a fixed control cycle to achieve continuous online adaptive optimization control of the grid-type converter.

[0053] Please see Figure 2 , Figure 2 This is an optional schematic diagram of a networked converter control system based on deep reinforcement learning provided in an embodiment of this application. It can implement the aforementioned networked converter control method based on deep reinforcement learning. The system may include, but is not limited to, the following: The first module 501 is used to obtain the current active power, current active power change rate, current reactive power and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the grid, in order to form a state observation vector. The second module 502 is used to input the state observation vector into the pre-trained target deep reinforcement learning model for forward inference to obtain the optimal virtual synchronous generator control parameter values. The third module 503 is used to analyze the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter using the virtual synchronous generator control strategy to generate a three-phase reference voltage. The fourth module 504 is used to control the grid-type converter based on the three-phase reference voltage.

[0054] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those implemented in the above method embodiments, and the beneficial effects achieved by this system embodiment are also the same as those achieved by the above method embodiments.

[0055] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described deep reinforcement learning-based network converter control method. This electronic device can include any smart terminal such as a tablet computer or desktop computer.

[0056] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those implemented by the above method embodiments, and the beneficial effects achieved by the present device embodiments are also the same as those achieved by the above method embodiments.

[0057] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the hardware structure of an electronic device according to another embodiment. The electronic device includes: The processor 601 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory 602 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 602 can store the operating system and other applications. When the technical solutions provided in the embodiments of this application are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called and executed by the processor 601. The input / output interface 603 is used to implement information input and output; The communication interface 604 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 605 transmits information between various components of the device (e.g., processor 601, memory 602, input / output interface 603, and communication interface 604); The processor 601, memory 602, input / output interface 603 and communication interface 604 are connected to each other within the device via bus 605.

[0058] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described deep reinforcement learning-based network converter control method.

[0059] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented by this storage medium embodiment are the same as those implemented by the above method embodiments, and the beneficial effects achieved by this storage medium embodiment are also the same as those achieved by the above method embodiments.

[0060] This application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described deep reinforcement learning-based network converter control method.

[0061] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those implemented by the above method embodiments, and the beneficial effects achieved by the embodiments of this program product are also the same as those achieved by the above method embodiments.

[0062] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0063] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0064] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0065] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0066] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0067] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0068] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0069] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0070] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0073] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A control method for a networked converter based on deep reinforcement learning, characterized in that, The method includes: The current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the grid, are obtained to form a state observation vector. The state observation vector is input into the pre-trained target deep reinforcement learning model for forward inference to obtain the optimal virtual synchronous generator control parameter values. Based on the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter, a virtual synchronous generator control strategy is used for analysis to generate a three-phase reference voltage. The grid-type converter is controlled based on the three-phase reference voltage.

2. The network converter control method based on deep reinforcement learning according to claim 1, characterized in that, The current active power, current active power change rate, current reactive power, and current output voltage of the grid-type converter are obtained in the following ways: Obtain the current three-phase voltage and current three-phase current at the grid connection point; The current three-phase voltage and current three-phase current at the grid connection point are sequentially subjected to Clark transformation and Park transformation to obtain the current dq-axis voltage and current dq-axis current. The current active power, current reactive power, and current output voltage of the grid-type converter are calculated based on the current dq-axis voltage and the current dq-axis current. The historical active power of the grid-type converter, calculated in the previous control cycle, is retrieved and combined with the current active power of the grid-type converter to calculate the current active power change rate of the grid-type converter.

3. The network converter control method based on deep reinforcement learning according to claim 1, characterized in that, The current angular frequency change and the current angular frequency change rate of the power grid are obtained in the following way: The current angular frequency of the power grid is obtained, and then the change in the current angular frequency of the power grid is calculated in combination with the rated angular frequency of the power grid. The historical angular frequency of the power grid obtained in the previous control cycle is retrieved and then combined with the current angular frequency of the power grid to calculate the current angular frequency change rate of the power grid.

4. The network converter control method based on deep reinforcement learning according to claim 1, characterized in that, The target deep reinforcement learning model is obtained in the following way: A virtual synchronous generator control grid-connected converter model was built in the MATLAB / Simulink environment, and various disturbance scenarios were set to form a simulation environment; In the simulation environment, the pre-constructed initial deep reinforcement learning model is interactively trained using the deep deterministic policy gradient algorithm to obtain the target deep reinforcement learning model.

5. The network converter control method based on deep reinforcement learning according to claim 1, characterized in that, The step of analyzing the optimal virtual synchronous generator control parameters and the current active power, current reactive power, and current output voltage of the grid-connected converter using a virtual synchronous generator control strategy to generate a three-phase reference voltage includes: Based on the optimal virtual synchronous generator control parameter values, the parameters of the pre-built virtual synchronous generator controller are updated; The newly determined active power reference value, reactive power reference value, and output voltage reference value of the virtual synchronous generator are called, and together with the current active power, current reactive power, and current output voltage of the grid-type converter, they are input to the updated virtual synchronous generator controller for calculation to obtain the virtual power angle and virtual electromotive force. The three-phase reference voltage is synthesized based on the virtual power angle and the virtual electromotive force.

6. The network converter control method based on deep reinforcement learning according to claim 1, characterized in that, The control of the grid-type converter based on the three-phase reference voltage includes: Based on the three-phase reference voltage, a pulse width modulation signal is generated; The switching transistors included in the grid converter are controlled to turn on and off according to the pulse width modulation signal.

7. A network converter control system based on deep reinforcement learning, characterized in that, The system includes: The first module is used to obtain the current active power, current active power change rate, current reactive power and current output voltage of the grid-type converter, as well as the current angular frequency change and current angular frequency change rate of the grid, in order to form a state observation vector. The second module is used to input the state observation vector into the pre-trained target deep reinforcement learning model for forward inference to obtain the optimal virtual synchronous generator control parameter values. The third module is used to analyze the optimal virtual synchronous generator control parameter values ​​and the current active power, current reactive power and current output voltage of the grid-type converter using a virtual synchronous generator control strategy to generate a three-phase reference voltage. The fourth module is used to control the grid-type converter based on the three-phase reference voltage.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the deep reinforcement learning-based network converter control method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the network converter control method based on deep reinforcement learning as described in any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the network converter control method based on deep reinforcement learning as described in any one of claims 1 to 6.

Images