Power distribution network robust control method and device based on adversarial reinforcement learning

Abstract

The application discloses a power distribution network robust control method and device based on adversarial reinforcement learning, relates to the technical field of power grid operation control, and comprises the following steps: obtaining operation state data of a power distribution network, generating a control action based on the operation state data through a control strategy model, and generating source-load power disturbance parameters based on the operation state data and the control action through a disturbance strategy model; determining a system operation deviation based on the control action and the source-load power disturbance parameters, and performing adversarial optimization on the control strategy model and the disturbance strategy model based on the system operation deviation to obtain a target control strategy model; and generating a target control action based on real-time operation state data of the power distribution network according to the target control strategy model, so that the photovoltaic inverter in the power distribution network executes the target control action. Through the bidirectional adversarial training of the control strategy model and the disturbance strategy module, the robustness and reliability of the control strategy are improved.

Description

Technical Field

[0001] This application relates to the field of power grid operation and control technology, specifically to a robust control method and device for distribution networks based on adversarial reinforcement learning. Background Technology

[0002] With the integration of distributed energy resources and electric vehicle loads, the penetration rate of power electronic devices in power systems has experienced explosive growth. However, the inherent low inertia, weak damping characteristics, and high sensitivity to control parameters of power electronic devices make the system highly susceptible to broadband oscillations, voltage exceedances, and even cascading grid disconnections when faced with disturbances such as new energy fluctuations, load abrupt changes, or extreme weather, affecting the safe and stable operation of the distribution network. To address the difficulty of control strategies based on fixed models or linearized assumptions adapting to the highly nonlinear, strongly coupled, and highly uncertain operating environment of modern distribution networks, existing technologies propose control based on reinforcement learning. This leverages the model-free nature and sequential decision-making capabilities of reinforcement learning data-driven approaches, enabling photovoltaic inverter control strategies to autonomously uncover dynamic patterns through online interaction. However, existing reinforcement learning-based distribution network control methods generally suffer from insufficient robustness. On the one hand, they rely on ideal simulation environments for training, failing to fully encompass real disturbances and making it difficult to accurately characterize the complex dynamic coupling risks of the distribution network. On the other hand, while conservative static boundary or simple disturbance models can ensure stability under extreme conditions, they sacrifice response speed and regulation accuracy under normal operating conditions. More importantly, existing strategies are mostly passive defenses, lacking explicit defense mechanisms against worst-case disturbances, which limits their application in high-reliability scenarios. Summary of the Invention

[0003] The purpose of this application is to address the problem that existing reinforcement learning-based distribution network control methods rely on training in ideal environments and lack proactive exploration and explicit defense mechanisms against extreme disturbances, leading to overfitting and poor robustness in actual operation. This application proposes a robust distribution network control method and device based on adversarial reinforcement learning. By constructing an adversarial reinforcement learning framework that includes both control and disturbance strategies, the two are jointly optimized based on the adversarial objectives of minimizing and maximizing system operating deviations. This allows the control strategy model to continuously evolve under the most severe and targeted source-load fluctuation scenarios, effectively overcoming the overfitting problem and improving the voltage / frequency support capability and control robustness of the distribution network when facing unknown and complex source-load disturbances.

[0004] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, embodiments of this application provide a robust control method for distribution networks based on adversarial reinforcement learning, the method comprising: The system acquires the operating status data of the distribution network, generates control actions based on the operating status data using a control strategy model, and generates source-load power disturbance parameters based on the operating status data and the control actions using a disturbance strategy model. Based on the control actions and the source-load power disturbance parameters, the system operating deviation is determined, and the control strategy model and the disturbance strategy model are optimized in an adversarial manner based on the system operating deviation to obtain a target control strategy model. According to the target control strategy model, a target control action is generated based on the real-time operating status data of the distribution network to control the photovoltaic inverters in the distribution network to execute the target control action.

[0005] Optionally, the operating status data includes the voltage phase angle, frequency, voltage amplitude, photovoltaic output, and load demand of each node in the distribution network; the control actions include the active power control signal and reactive power control signal of the photovoltaic inverter at each node; and the source-load power disturbance parameters include the photovoltaic active power output, photovoltaic reactive power output, node active load, and node reactive load at the next moment.

[0006] Optionally, the step of generating control actions based on operating state data through a control strategy model and generating source-load power disturbance parameters based on operating state data and the control actions through a disturbance strategy model includes: defining the control process of the distribution network as a Markov decision process with minimizing frequency deviation and voltage deviation as the distribution network operation control objective, and establishing a control strategy model with the distribution network operation control objective as the reward term; obtaining the state vector of the control strategy model based on the operating state data, and generating control actions based on the state vector; defining the disturbance in the distribution network control process as a Markov decision process with maximizing frequency deviation and voltage deviation as the distribution network disturbance objective, and establishing a disturbance strategy model with the distribution network disturbance objective as the reward term; obtaining the state vector of the disturbance strategy model based on the control actions and the operating state data, and generating source-load power disturbance parameters based on the state vector.

[0007] Optionally, the step of determining the system operating deviation based on the control action and the source-load power disturbance parameters, and performing adversarial optimization on the control strategy model and the disturbance strategy model based on the system operating deviation, includes: obtaining the frequency and voltage amplitude of each node in the distribution network under the combined action of the control action and the source-load power disturbance parameters, calculating the difference with the rated frequency of the distribution network and the reference value of the rated voltage of each node to obtain the system frequency deviation and voltage deviation; calculating the operating deviation reward and adversarial reward based on the frequency deviation and voltage deviation of each node; updating the parameters of the control strategy model with minimizing the operating deviation reward as the optimization objective; and updating the parameters of the disturbance strategy model with maximizing the adversarial reward as the optimization objective.

[0008] Optionally, when updating the parameters of the control strategy model with the optimization objective of minimizing the running deviation reward, the following steps are also performed: extracting data samples from the control experience pool of the control strategy model; calculating the policy loss function and evaluation loss function of the control strategy model based on the data samples; updating the policy network parameters of the control strategy model based on the policy loss function; and updating the evaluation network parameters of the control strategy model based on the evaluation loss function.

[0009] Optionally, when updating the parameters of the perturbation policy model with the optimization objective of maximizing the adversarial reward, the following steps are also performed: extracting data samples based on the perturbation experience pool of the perturbation policy model; calculating the policy loss function and evaluation loss function of the perturbation policy model based on the data samples; updating the policy network parameters of the perturbation policy model based on the policy loss function; and updating the evaluation network parameters of the perturbation policy model based on the evaluation loss function.

[0010] Optionally, both the control strategy model and the disturbance strategy model include a strategy network for generating control signals, an evaluation network for evaluating the value of the output of the strategy network and adjusting its direction, and a target evaluation network for providing a value benchmark for the training objective of the evaluation network; the strategy network takes state variables as inputs and outputs action variables, and the evaluation network and the target evaluation network take state variables and action variables as inputs and outputs cumulative discount rewards.

[0011] Optionally, the step of generating target control actions based on the real-time operating status data of the distribution network according to the target control strategy model, so as to control the photovoltaic inverters in the distribution network to execute the target control actions, includes: using the real-time operating status data of the distribution network as input parameters of the target control strategy model, outputting the active power control signal and reactive power control signal at the current moment through forward propagation; using the active power control signal and the reactive power control signal as the target control actions, and sending them to the corresponding photovoltaic inverters for execution control.

[0012] Optionally, the method further includes: during the adversarial optimization of the control strategy model and the disturbance strategy model, the control strategy model and the disturbance strategy model interact with the distribution network control environment; the distribution network control environment is constructed based on the distribution network power flow model and the photovoltaic inverter dynamic model, and is used to calculate the operating status data at the next moment according to the control action and the source load power disturbance parameters and provide feedback rewards; wherein, the photovoltaic inverter dynamic model includes a primary control frequency dynamic response process and voltage droop control characteristics, the frequency dynamic response process generates the frequency change rate based on the frequency droop coefficient and the active power control signal output by the control strategy model, and the voltage droop control characteristics generate the voltage amplitude change rate based on the voltage droop coefficient and the reactive power control signal output by the control strategy model.

[0013] Secondly, embodiments of this application provide a robust control device for a power distribution network based on adversarial reinforcement learning, comprising: The basic training module is used to acquire the operating status data of the distribution network, generate control actions based on the operating status data through the control strategy model, and generate source-load power disturbance parameters based on the operating status data and the control actions through the disturbance strategy model. The adversarial optimization module is used to determine the system operating deviation based on the control actions and the source-load power disturbance parameters, and perform adversarial optimization on the control strategy model and the disturbance strategy model based on the system operating deviation to obtain the target control strategy model. The grid control module is used to generate target control actions based on the real-time operating status data of the distribution network according to the target control strategy model, so as to control the photovoltaic inverters in the distribution network to execute the target control actions.

[0014] The beneficial effects of this application are: 1. This application constructs a joint framework of adversarial game between the control strategy model and the disturbance strategy model, transforming the traditional one-way mode of passively adapting to environmental disturbances in reinforcement learning control into a two-way adversarial mode that actively explores system vulnerabilities. The disturbance strategy model autonomously generates photovoltaic output disturbances and load disturbances, continuously exploring the safety boundary of the distribution network. This forces the control strategy model to continuously optimize its control strategy under worst-case disturbances, effectively overcoming the robustness gap and overfitting problems caused by existing methods relying on ideal simulation environments for training. This enhances the adaptive capability and control resilience of the distribution network when facing unknown disturbances beyond the training data. 2. By embedding the power flow model of the distribution network and the dynamic model of the photovoltaic inverter into the environmental interaction process during the adversarial optimization process, each update of the control strategy model and the disturbance strategy model is strictly limited by the physical operation law of the power system. This ensures that the control actions and disturbance injections do not deviate from the actual adjustment capabilities and physical safety boundaries of the equipment, thereby overcoming the problem that the simplified static disturbance model is difficult to characterize the complex dynamic coupling risks. Attached Figure Description

[0015] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings. The drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0016] Figure 1 This is a flowchart of a robust control method for power distribution networks based on adversarial reinforcement learning, as described in an embodiment of this application.

[0017] Figure 2 This is a framework diagram of the adversarial reinforcement learning model in an embodiment of this application.

[0018] Figure 3 This is a structural diagram of a robust control device module for power distribution networks based on adversarial reinforcement learning, according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description of this application is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely one preferred embodiment of this application and are only used to explain this application. They do not limit the scope of protection of this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] Example 1: As Figure 1 As shown, a robust control method for distribution networks based on adversarial reinforcement learning includes steps S1-S3, wherein: S1. Obtain the operating status data of the distribution network, generate control actions based on the operating status data through the control strategy model, and generate source-load power disturbance parameters based on the operating status data and the control actions through the disturbance strategy model.

[0021] Specifically, the operating status data includes the voltage phase angle, frequency, voltage amplitude, photovoltaic output, and load demand of each node in the distribution network; the control actions include the active power control signal and reactive power control signal of the photovoltaic inverters at each node; and the source-load power disturbance parameters include the photovoltaic active power output, photovoltaic reactive power output, node active power load, and node reactive power load at the next moment.

[0022] Specifically, in step S1, a control action is generated based on the operating state data using a control strategy model, and source load power disturbance parameters are generated based on the operating state data and the control action using a disturbance strategy model, including: Minimizing frequency deviation and voltage deviation are the control objectives of the distribution network operation. The control process of the distribution network is defined as a Markov decision process, and a control strategy model with the distribution network operation control objective as the reward term is established. The state vector of the control strategy model is obtained based on the operating state data, and control actions are generated based on the state vector. With maximizing frequency deviation and voltage deviation as the distribution network disturbance objectives, the disturbance in the distribution network control process is defined as a Markov decision process, and a disturbance strategy model with the aforementioned distribution network disturbance objectives as the reward term is established. The state vector of the disturbance strategy model is obtained based on the control action and the operating state data, and the source load power disturbance parameters are generated based on the state vector.

[0023] In some embodiments, the core objective of distribution network operation control is to maintain system frequency and node voltage near their rated values. After modeling the control process as a Markov decision process, during the training phase, the control strategy model observes the environmental state at each discrete time step. Select Action The environment transitions to the next state and returns a reward. The state variable... Including voltage phase angle, frequency, node voltage amplitude, photovoltaic active and reactive power output, node active and reactive load, and action variables. For the active power control signal and reactive power control signal of the photovoltaic inverter at each node, the reward function is... Defined as a negative weighted value of frequency deviation and voltage deviation, it is used to guide the control policy agent to learn a policy that minimizes this deviation.

[0024] The objective of the disturbance strategy model is exactly the opposite of that of the control strategy model. The disturbance strategy model aims to find the source-load disturbance combination that maximizes the deviation of the system frequency and voltage from their rated values. The state variables of the disturbance strategy model are... This includes voltage phase angle, frequency, node voltage amplitude, photovoltaic active and reactive power output, node active and reactive load, and action variables output by the control strategy model. Select action variables based on the combination of environmental variables. This refers to the source-load perturbation combination, including the photovoltaic active power output, nodal active load, photovoltaic reactive power output, and nodal reactive load at the next time step. These parameters are superimposed as perturbations onto the current source-load parameters to form the system injection quantity at the next time step. It should be understood that the construction of the above dual Markov decision process is the formal basis of the adversarial reinforcement learning framework. By defining control and perturbation as decision problems with opposite optimization directions, a clear mathematical modeling basis is provided for subsequent joint training. The model structure is a conventional setting.

[0025] In some examples, the operational control objective of a distribution network can be expressed as: (1); in, and The weights for frequency and voltage targets are respectively used to balance the relative importance of the two types of control targets. In this embodiment... It is 1.0. The value is 1.0, which can be flexibly adjusted according to operational requirements in practical applications; For nodes exist Frequency of time; To control the cycle; The set of nodes in the power distribution network; node exist Voltage amplitude at any given moment; This is the reference value for the voltage of the power distribution network; Reward function of control strategy model It can be represented as: (2); Reward function of perturbation policy model It can be represented as: (3); In this embodiment, a two-way adversarial mode is formed by the reward of the control strategy model and the reward of the disturbance strategy model, which forces the control strategy model to continuously optimize its control strategy under the worst-case disturbance. This effectively overcomes the robustness gap and overfitting problem caused by the reliance on ideal simulation environment training in existing methods, and significantly improves the adaptive capability and control resilience of the distribution network when facing unknown disturbances outside the training data.

[0026] S2. Determine the system operating deviation based on the control action and the source load power disturbance parameters, and perform adversarial optimization on the control strategy model and the disturbance strategy model based on the system operating deviation to obtain the target control strategy model.

[0027] Specifically, system operating deviation reflects the degree to which the system deviates from its rated operating point under the control actions output by the control strategy model and the disturbances output by the disturbance strategy model. This can include, for example, frequency and voltage deviations at various nodes in the distribution network. The optimization objective of the control strategy model is to minimize this deviation, while the optimization objective of the disturbance strategy model is to maximize it; the two form an adversarial game. In this process, the control strategy model continuously learns how to maintain system stability under the worst-case disturbances, while the disturbance strategy model continuously explores the system's weak points. Through bidirectional adversarial optimization, they alternately update the network parameters of both models, ensuring that the optimized target control strategy model possesses robustness against various potential disturbances.

[0028] In this embodiment, by establishing an adversarial optimization mechanism between the control strategy model and the perturbation strategy model, the traditional one-way passive defense is transformed into an active risk mining mode. The perturbation strategy model automatically generates and injects perturbations during the training phase, forcing the control strategy model to continuously optimize its strategy under worst-case conditions. This effectively overcomes the robustness gap and overfitting problems caused by existing methods' reliance on ideal simulation environments.

[0029] Specifically, step S2 includes: The frequency and voltage amplitude of each node in the distribution network are obtained under the combined action of the control action and the source load power disturbance parameters. The difference between these values ​​and the rated frequency and the rated voltage reference value of each node in the distribution network are calculated to obtain the frequency deviation and voltage deviation of the system. Calculate operational deviation rewards and adversarial rewards based on the frequency and voltage deviations of each node; The parameters of the control strategy model are updated with the optimization objective of minimizing the operational deviation reward. The parameters of the perturbation policy model are updated with the optimization objective of maximizing the adversarial reward. Based on the control strategy model after the adversarial optimization is completed, the target control strategy model is obtained.

[0030] In some embodiments, obtaining the frequency and voltage amplitude of each node in the distribution network under the coordinated operation of the control action and the source-load power disturbance parameters includes: Based on the active and reactive power control signals output by the control strategy model, and the photovoltaic active power output, photovoltaic reactive power output, node active load, and node reactive load output by the disturbance strategy model at the next moment, the active and reactive power distribution of the entire network at the next moment is calculated under the combined effect of control actions and source-load disturbances, relying on the distribution network power flow model, to obtain the preliminary node voltage correlation state. Subsequently, combined with the photovoltaic inverter dynamic model, the active and reactive power control commands received by the inverter are substituted into the dynamic adjustment logic to simulate the dynamic process of inverter low-pass filter response, frequency droop control, and voltage droop control, and the real-time frequency and voltage amplitude of each node in the entire distribution network at the next moment are derived. The actual frequency and actual voltage amplitude of each node at the next moment are calculated by comparing them with the preset rated frequency and rated voltage reference values ​​of each node in the distribution network, respectively, to finally obtain the frequency deviation and voltage deviation of the system.

[0031] In some examples, the power flow model of the distribution network is represented as: (4); (5); in, The set of nodes in a power distribution network; To control the cycle; and They are nodes exist Real-time photovoltaic active power output and nodal active power load; and They are nodes exist The photovoltaic reactive power output and nodal reactive load at any given time; if the node If there is no photovoltaic inverter at that location, then ,but ,in, It is the set of nodes in the distribution network that contain photovoltaic inverters; and They are nodes and nodes exist Voltage amplitude at any given moment; and These are the nodes in the admittance matrix of the distribution network nodes. and nodes The real and imaginary parts of the corresponding element; For nodes and nodes exist The voltage phase angle difference at time t, i.e. ,in, and They are nodes and nodes exist The voltage phase angle at a given moment.

[0032] In some examples, the dynamic model of a photovoltaic inverter is represented as: (6); (7); (8); in, For nodes exist The derivative of the voltage phase angle at time t; For nodes exist Frequency of time; For the node The time constant of the low-pass filter in a photovoltaic inverter; and They are nodes exist The derivative of the frequency with respect to time at any given moment and the derivative of the voltage amplitude with respect to time; and They are located at the nodes respectively Frequency droop coefficient and voltage droop coefficient of primary control of photovoltaic inverter; and They are located at the nodes respectively Photovoltaic inverters in The active and reactive power control signals at any given time are the control actions output by the control strategy model. This is a reference value for the distribution network voltage.

[0033] In some embodiments, the operating deviation reward is calculated based on the reward functions of the control strategy model and the disturbance strategy model, i.e., the above formulas (2) and (3). and combat rewards Therefore, the network parameters of the control strategy model and the disturbance strategy model are updated according to the operational deviation reward and the adversarial reward, respectively.

[0034] In this embodiment, by acquiring the system operation deviation under the combined effect of control actions and source load power disturbance parameters, the operation control reward and adversarial reward are calculated to update the corresponding model parameters, realizing synchronous alternating updates of the network parameters of both parties in opposite gradient directions. This mechanism ensures the stable evolution of the adversarial game process, enabling the control strategy model to improve in the direction of reducing the worst-case deviation with each parameter update, thereby gradually converging to a robust control strategy with good suppression capabilities against various disturbances.

[0035] Specifically, in step S2, when updating the parameters of the control strategy model with the optimization objective of minimizing the running deviation reward, the following is also performed: Data samples are extracted from the control experience pool based on the control strategy model; Calculate the strategy loss function and evaluation loss function of the control strategy model based on the data sample; The policy network parameters of the control policy model are updated based on the policy loss function. The evaluation network parameters of the control strategy model are updated based on the evaluation loss function.

[0036] Specifically, after each decision, the control policy model stores the resulting quadruple into the control experience pool. The quadruple includes a state variable, an action variable, a control reward, and a next state variable. During the model update phase, a batch of samples is randomly sampled from the experience pool as training samples to calculate the loss functions of the policy network and evaluation network in the model, thereby updating the corresponding policy network parameters and evaluation network parameters. This experience replay mechanism breaks the strong correlation between time-series samples, making the optimization process more stable. The sample size can be flexibly adjusted according to computational resources and problem size.

[0037] In some examples, the policy loss function of the control policy model It can be represented as: (9); in, The size of the randomly selected data sample; For the next state variable; For the policy network in the control policy model Initialization parameters; It is the entropy coefficient; The entropy regularization term of the policy loss function. This represents the probability distribution of the policy network's output actions. This represents taking the natural logarithm of the probability, used to convert probability multiplication into addition to simplify gradient calculation; This represents the Q-value, or cumulative discount reward value, of the evaluation network, which takes state variables and action variables as input.

[0038] It should be noted that the principle behind the calculation formula for this policy loss function is: combining the deterministic policy gradient theorem with the maximum entropy principle, designing the policy loss to maximize the expected cumulative reward while simultaneously maximizing the policy entropy. Specifically, Entropy reward is used to maximize the output distribution entropy of the policy in a given state while optimizing the cumulative reward. It forces the policy to maintain a certain degree of randomness and exploration diversity, and prevents the policy from becoming too deterministic, i.e., overfitting to a specific action. Derived from the deterministic policy gradient theorem, it guides policy selection in the gradient direction to produce actions that generate the maximum Q value, i.e., reducing frequency and voltage deviations. By minimizing these deviations, it encourages movement toward higher-value actions.

[0039] The initial parameters of the policy network are updated based on the calculated policy loss function. The update process can be expressed as follows: ,in, For parameters The learning rate, for example, is determined based on expert experience. 1.0×10 -4 .

[0040] Evaluation loss function of control strategy model It can be represented as: (10); in, Initialization parameters for evaluating the network in the control strategy model; To evaluate the target reward value during network training, its calculation method can be expressed as: (11); in, The discount rate for future rewards is set at 0.99 in this embodiment based on experience. , representing the next state variable The action variables under the next state are the active and reactive control signals output by the policy network in the next state. As an immediate reward, it is a reward for operational deviations after performing a control action in the current state; To evaluate the cumulative discount reward value output by the network, taking the next state variable and the next action variable as inputs; This is the entropy penalty term, which is the negative value of the logarithm of the probability of the policy network's output action at the next time step multiplied by the entropy coefficient. It represents a penalty for the entropy of the policy in the next state. Since the optimization objective of the policy in the maximum entropy reinforcement learning framework is to maximize the weighted sum of the cumulative reward and the policy entropy, the entropy term at the next time step is subtracted from the objective value. This allows the evaluation network to convert the exploration redundancy of the future policy into a value loss when estimating the current value, thereby guiding the policy network to actively suppress excessive randomness while pursuing high rewards, achieving a dynamic balance between exploration and exploitation. The initial parameters of the evaluation network are updated based on the calculated evaluation loss function. The update process can be expressed as follows: ,in, For parameters The learning rate, for example, is determined based on expert experience. 5.0×10 -4 .

[0041] It should be noted that the design principle of the evaluation loss function is based on the Bellman optimality equation and temporal difference learning. The Bellman optimality equation originates from reinforcement learning and states that the optimal Q-value of the current state-action pair is... It must equal the immediate reward plus the discount factor multiplied by the expected optimal value of the next state, i.e. The training objective of the evaluation network is to minimize the evaluation loss function so that the predictions of the current policy network continuously approach the temporal difference target value. .

[0042] Specifically, in step S2, when updating the parameters of the perturbation policy model with the optimization objective of maximizing the adversarial reward, the following is also performed: Data samples are extracted from the perturbation experience pool based on the perturbation strategy model; Calculate the policy loss function and evaluation loss function of the perturbation policy model based on the data sample; The policy network parameters of the perturbation policy model are updated based on the policy loss function. The evaluation network parameters of the perturbation policy model are updated based on the evaluation loss function.

[0043] Specifically, consistent with the control strategy model described above, the perturbation strategy model stores the resulting quadruple in the perturbation experience pool after each decision. The quadruple includes a state variable, an action variable, a control reward, and a next state variable. During the model update phase, a batch of samples is randomly sampled from the experience pool as training samples to calculate the loss functions of the policy network and evaluation network in the model, thereby updating the corresponding policy network parameters and evaluation network parameters. It should be understood that the calculation methods for the policy loss function and evaluation loss function, as well as the parameter update methods, of the perturbation strategy model are consistent with those of the control strategy model described above; only the state variables and action variables differ, and will not be reiterated here.

[0044] It should be noted that, regarding the entropy reward term in the policy network, in adversarial optimization, for the control policy model, it is used to prevent the policy from overfitting to a few specific typical perturbation patterns too early, and to encourage it to explore a wider range of defense policy spaces; for the perturbation policy model, it encourages the perturbation policy to continuously output diverse perturbation combinations, avoiding it from getting trapped in local optima of repeatedly outputting the same failed perturbation, thereby continuously providing challenging hard samples for the control policy model.

[0045] In this embodiment, historical interaction samples are randomly drawn from the experience pool for batch learning. The policy loss function and evaluation loss function are calculated to update the parameters of the policy network and evaluation network respectively. This breaks the strong correlation between samples over consecutive time, effectively suppressing parameter oscillations and overestimation problems in deep reinforcement learning training. The experience replay mechanism allows training samples to be reused, improving data efficiency. Combined with the synergistic optimization of policy loss and evaluation loss, it ensures stable and efficient iterative updates of the policy networks for both the control policy model and the perturbation policy model during adversarial training.

[0046] Specifically, in combination Figure 2 As shown, both the control strategy model and the disturbance strategy model include a policy network, an evaluation network, a target evaluation network, and an experience pool. The policy network is used to generate control signals, the evaluation network is used to evaluate the value of the output of the policy network and adjust the optimization direction, and the target evaluation network is used to provide a value benchmark for the training objective of the evaluation network. The policy network takes state variables as inputs and outputs action variables, and the evaluation network and the target evaluation network take state variables and action variables as inputs and outputs cumulative discount rewards.

[0047] In this embodiment, the policy network takes state variables as input and outputs action variables to achieve rapid decision-making. For the control policy model, the action variables are active and reactive power control signals; for the disturbance policy model, the action variables are source load power disturbance parameters. The initialization parameters of the target evaluation network are consistent with those of the evaluation network. During updates, the parameters of the target evaluation network are updated based on the updated evaluation network parameters.

[0048] Furthermore, the core role of the evaluation network is real-time value assessment and policy guidance. It takes the current state and action variables as input and outputs the expected cumulative discounted reward value (Q-value) for that state-action pair. This Q-value is directly used to calculate the loss function of the policy network, providing a quantitative optimization direction for the gradient update of the policy network parameters. It is the direct basis for driving the control policy model to improve towards reducing frequency and voltage deviations, and for driving the perturbation policy model to improve towards increasing deviations. The core role of the target evaluation network is to stabilize the training process and mitigate overestimation. It does not directly participate in the real-time update of the policy network but is used to generate the target Q-value, as described above. By using a soft update mechanism, the parameters of the target evaluation network are lagging behind those of the evaluation network. That is, the initial parameters of the evaluation network are updated first, and then the updated parameters of the evaluation network are used to update the parameters of the target evaluation network. This keeps the target Q value relatively stable in multiple iterations and avoids the violent oscillation of the target value caused by rapid changes in the evaluation network parameters.

[0049] The evaluation network and the target evaluation network work together to decouple target generation and real-time evaluation. The evaluation network is responsible for responding to the latest interaction data and quickly exploring optimization directions, while the target evaluation network is responsible for anchoring to a relatively fixed optimization benchmark. This dual-network structure can effectively suppress Q-value overestimation and gradient vanishing problems in agent training, providing a smooth and robust convergence guarantee for the adversarial game between the control policy model and the perturbation policy model. It should be understood that although this embodiment uses a dual-evaluation network structure to further suppress Q-value overestimation, in practical applications, if the problem size is small or higher computational efficiency is required, a single evaluation network structure can also be used. In this case, the two evaluation networks can simply be merged into one. The network structure parameters mentioned above, including the number of hidden layers and the number of neurons, are conventional settings and are not specifically limited in this embodiment. In actual deployment, they can be flexibly adjusted according to the complexity of the control problem and the hardware computing power.

[0050] S3. Based on the target control strategy model, generate target control actions based on the real-time operating status data of the distribution network, so as to control the photovoltaic inverters in the distribution network to execute the target control actions.

[0051] Specifically, step S3 includes: The real-time operating status data of the distribution network is used as the input parameters of the target control strategy model, and the active power control signal and reactive power control signal at the current moment are output through forward propagation. The active power control signal and the reactive power control signal are used as the target control action and sent to the corresponding photovoltaic inverter for execution control.

[0052] In some embodiments, after the control strategy model and the disturbance strategy model have completed adversarial optimization, only the strategy network after training convergence is retained in the control strategy model. At the beginning of each control cycle, real-time measurement data of each node in the distribution network are collected, and a state vector is constructed according to the same specifications as during training. This state vector is input into the strategy network, and after layer-by-layer forward propagation calculation through a multi-layer fully connected neural network, the final output layer generates an action vector corresponding to the normalized value of the photovoltaic inverter, which includes active power control signals and reactive power control signals.

[0053] In some embodiments, the active power control signal and reactive power control signal are converted into actual physical quantities through inverse normalization and then sent to each photovoltaic inverter. The photovoltaic inverter uses the control signal as a reference command for primary control to adjust the inverter's active power output and reactive power. This achieves an end-to-end response from state perception to control command generation. This process does not require online iterative solution of optimization problems, does not rely on disturbance prediction information, and only involves simple forward computation of neural networks, significantly reducing the communication bandwidth requirements and computational resource consumption of online control.

[0054] Specifically, the method further includes: During the adversarial optimization of the control strategy model and the disturbance strategy model, the control strategy model and the disturbance strategy model interact with the distribution network control environment; The distribution network control environment is constructed based on the distribution network power flow model and the photovoltaic inverter dynamic model. It is used to calculate the operating status data of the next moment according to the control action and the source load power disturbance parameters and to feed back the reward. The photovoltaic inverter dynamic model includes a primary control frequency dynamic response process and a voltage droop control characteristic. The frequency dynamic response process generates the frequency change rate based on the frequency droop coefficient and the active power control signal output by the control strategy model. The voltage droop control characteristic generates the voltage amplitude change rate based on the voltage droop coefficient and the reactive power control signal output by the control strategy model.

[0055] In this embodiment, the dynamic model of the photovoltaic inverter accurately characterizes the dynamic response of the primary control frequency and the voltage droop control characteristics, ensuring that any control action and disturbance parameter generated by the control strategy model and the disturbance strategy model are mapped to the actual physically achievable power regulation range. This makes the training convergence control strategy not only optimal in terms of economic or stability indicators, but also strictly follow the operating laws of the power system and the physical limits of the equipment.

[0056] In this embodiment, by embedding the power flow model of the distribution network and the dynamic model of the photovoltaic inverter into the adversarial training environment, the aim is to establish a causal mapping between the abstract control signal and the actual frequency and voltage response of the system, and to internalize the physical characteristics of the inverter such as inertial delay, droop coupling and ramping constraints into environmental transfer rules. This forces the disturbance strategy model to generate worst-case disturbances only within the physically feasible domain, thereby guiding the control strategy model to adapt to the equipment adjustment limits during the training phase, ensuring that the control strategy still has executability and safety margin under the extreme operating conditions of the real distribution network.

[0057] Based on the same inventive concept, this application also provides an adversarial reinforcement learning-based robust control device for distribution networks, corresponding to the adversarial reinforcement learning-based robust control method for distribution networks. Figure 3 As shown, the device includes: The basic training module is used to acquire the operating status data of the distribution network, generate control actions based on the operating status data through the control strategy model, and generate source load power disturbance parameters based on the operating status data and the control actions through the disturbance strategy model. The adversarial optimization module is used to determine the system operating deviation based on the control action and the source load power disturbance parameters, and to perform adversarial optimization on the control strategy model and the disturbance strategy model based on the system operating deviation to obtain the target control strategy model. The power grid control module is used to generate target control actions based on the real-time operating status data of the distribution network according to the target control strategy model, so as to control the photovoltaic inverters in the distribution network to execute the target control actions.

[0058] In this embodiment, a complete automated closed loop from data acquisition and adversarial training to online deployment is constructed through the coordinated operation of the basic training module, the adversarial optimization module, and the power grid control module. The basic training module drives the control and disturbance strategy models to continuously interact with the environment to generate training data, enabling the control strategy models to continuously face evolving and severe operating conditions during the training process. The adversarial optimization module alternately updates the network parameters of both sides based on system operating deviations to search for robust control strategies. The power grid control module deploys the converged target strategy model in the online environment and generates control commands in real time. This effectively overcomes the robustness gap and strategy overfitting problems caused by existing methods that rely on ideal simulation environments for training, and significantly improves the adaptive control capability and operational safety of the distribution network when facing unknown disturbances in actual operation.

[0059] The above-described embodiments are preferred embodiments of this application and are not intended to limit the specific scope of this application. The scope of this application includes but is not limited to the specific embodiments described above. All equivalent changes made in accordance with the shape, structure, and method of this application are within the protection scope of this application.

Claims

1. A robust control method for power distribution networks based on adversarial reinforcement learning, characterized in that: Includes the following steps: The system acquires the operating status data of the distribution network, generates control actions based on the operating status data using a control strategy model, and generates source-load power disturbance parameters based on the operating status data and the control actions using a disturbance strategy model. The system aims to minimize frequency and voltage deviations as the distribution network operation control objectives, defines the distribution network control process as a Markov decision process, and establishes a control strategy model with the distribution network operation control objectives as the reward term. The system then acquires the state vector of the control strategy model based on the operating status data, and generates control actions based on the state vector. With maximizing frequency deviation and voltage deviation as the distribution network disturbance objectives, the disturbance in the distribution network control process is defined as a Markov decision process, and a disturbance strategy model with the distribution network disturbance objective as the reward term is established; the state vector of the disturbance strategy model is obtained according to the control action and the operating state data, and the source load power disturbance parameters are generated according to the state vector. The system operating deviation is determined based on the control action and the source load power disturbance parameters, and the control strategy model and the disturbance strategy model are counter-optimized based on the system operating deviation to obtain the target control strategy model. Based on the target control strategy model, target control actions are generated using real-time operating status data of the distribution network to control the photovoltaic inverters in the distribution network to execute the target control actions.

2. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 1, characterized in that: The operational status data includes the voltage phase angle, frequency, voltage amplitude, photovoltaic output, and load demand of each node in the distribution network; The control actions include active power control signals and reactive power control signals for the photovoltaic inverters at each node. The source-load power disturbance parameters include the photovoltaic active power output, photovoltaic reactive power output, node active load, and node reactive load at the next moment.

3. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 1, characterized in that: The process of determining the system operating deviation based on the control action and the source load power disturbance parameters, and performing adversarial optimization on the control strategy model and the disturbance strategy model based on the system operating deviation, includes: The frequency and voltage amplitude of each node in the distribution network are obtained under the combined action of the control action and the source load power disturbance parameters. The difference between these values ​​and the rated frequency and the rated voltage reference value of each node in the distribution network are calculated to obtain the frequency deviation and voltage deviation of the system. Calculate operational deviation rewards and adversarial rewards based on the frequency and voltage deviations of each node; The parameters of the control strategy model are updated with the optimization objective of minimizing the operational deviation reward. The parameters of the perturbation policy model are updated with the optimization objective of maximizing the adversarial reward.

4. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 3, characterized in that: When updating the parameters of the control strategy model with the optimization objective of minimizing the running deviation reward, the following is also performed: Data samples are extracted from the control experience pool based on the control strategy model; Calculate the policy loss function and evaluation loss function of the control strategy model based on the data sample; The policy network parameters of the control policy model are updated based on the policy loss function. The evaluation network parameters of the control strategy model are updated based on the evaluation loss function.

5. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 3, characterized in that: When updating the parameters of the perturbation policy model with the optimization objective of maximizing the adversarial reward, the following is also performed: Data samples are extracted from the perturbation experience pool based on the perturbation strategy model; Calculate the policy loss function and evaluation loss function of the perturbation policy model based on the data sample; The policy network parameters of the perturbation policy model are updated based on the policy loss function. The evaluation network parameters of the perturbation policy model are updated based on the evaluation loss function.

6. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 3, characterized in that: Both the control strategy model and the disturbance strategy model include a strategy network for generating control signals, an evaluation network for evaluating the value of the output of the strategy network and optimizing its direction, and a target evaluation network for providing a value benchmark for the training objectives of the evaluation network. The policy network takes state variables as inputs and outputs action variables, while the evaluation network and the target evaluation network take state variables and action variables as inputs and outputs cumulative discount rewards.

7. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 1, characterized in that: The step of generating target control actions based on the real-time operating status data of the distribution network according to the target control strategy model, and controlling the photovoltaic inverters in the distribution network to execute the target control actions, includes: The real-time operating status data of the distribution network is used as the input parameters of the target control strategy model, and the active power control signal and reactive power control signal at the current moment are output through forward propagation. The active power control signal and the reactive power control signal are used as the target control action and sent to the corresponding photovoltaic inverter for execution control.

8. The robust control method for distribution networks based on adversarial reinforcement learning according to claim 1, characterized in that: The method further includes: During the adversarial optimization of the control strategy model and the disturbance strategy model, the control strategy model and the disturbance strategy model interact with the distribution network control environment; The distribution network control environment is constructed based on the distribution network power flow model and the photovoltaic inverter dynamic model. It is used to calculate the operating status data of the next moment according to the control action and the source load power disturbance parameters and to feed back the reward. The photovoltaic inverter dynamic model includes a primary control frequency dynamic response process and a voltage droop control characteristic. The frequency dynamic response process generates the frequency change rate based on the frequency droop coefficient and the active power control signal output by the control strategy model. The voltage droop control characteristic generates the voltage amplitude change rate based on the voltage droop coefficient and the reactive power control signal output by the control strategy model.

9. A robust control device for power distribution networks based on adversarial reinforcement learning, characterized in that: The robust control method for distribution networks based on adversarial reinforcement learning as described in any one of claims 1-8 includes: The basic training module is used to acquire the operating status data of the distribution network, generate control actions based on the operating status data through the control strategy model, and generate source load power disturbance parameters based on the operating status data and the control actions through the disturbance strategy model. The adversarial optimization module is used to determine the system operating deviation based on the control action and the source load power disturbance parameters, and to perform adversarial optimization on the control strategy model and the disturbance strategy model based on the system operating deviation to obtain the target control strategy model. The power grid control module is used to generate target control actions based on the real-time operating status data of the distribution network according to the target control strategy model, so as to control the photovoltaic inverters in the distribution network to execute the target control actions.

Images