Power distribution network dispatching method, computer device and storage medium

Abstract

The application provides a power distribution network scheduling method, computer equipment and a storage medium. The method comprises the following steps: acquiring real-time observation data of a power distribution network; calling a preset target agent to process the real-time observation data, and obtaining a scheduling action instruction of the power distribution network, wherein a long short-term memory (LSTM) neural network in the target agent is pre-trained according to a plurality of reference data pairs in a constraint Markov decision process corresponding to the power distribution network, and is obtained through adversarial alternating training according to the constraint Markov decision process corresponding to the power distribution network, a constraint strategy optimization algorithm corresponding to the constraint Markov decision process, and an alternating training algorithm based on an adversarial opponent, and the reference data pair comprises a reference state of the power distribution network and a reference scheduling action instruction corresponding to the reference state; and scheduling the power distribution network according to the scheduling action instruction. Based on the application scheme, the safety, robustness and economy of the power distribution network scheduling can be improved.

Description

Technical Field

[0001] This application relates to the field of power grid technology, and more specifically, to a distribution network dispatching method, computer equipment, and storage medium. Background Technology

[0002] In distribution network dispatching technologies, optimization based on mathematical models (such as convex relaxation AC power flow models) is the primary approach. However, these methods have significant drawbacks: convex relaxation introduces model bias, potentially leading to calculations that violate safety constraints such as voltage and line capacity (e.g., voltage exceeding limits, line overload); while non-convex optimization methods (such as the primal dual interior-point method) can improve accuracy, their computational complexity is high in large-scale distribution networks, making them unsuitable for real-time dispatching. Furthermore, these methods do not adequately consider inter-time constraint coupling relationships (e.g., the continuity of energy storage charging and discharging states), making it difficult to capture the dynamic evolution of the distribution network. Especially when facing fluctuations in renewable energy output and load forecasting errors, traditional methods lack effective mechanisms to ensure robust decision-making. Due to these shortcomings, it is difficult for related technologies to simultaneously achieve high computational efficiency, low constraint violation rates, and strong uncertainty handling capabilities for real-time distribution network dispatching. Summary of the Invention

[0003] This application provides a power distribution network dispatching method, computer equipment, and storage medium.

[0004] The first aspect of this application provides a power distribution network dispatching method.

[0005] Obtain real-time observation data of the power distribution network;

[0006] The preset target intelligent agent is invoked to process the real-time observation data to obtain the dispatch action instructions of the distribution network. The long short-term memory LSTM neural network in the target intelligent agent is obtained by imitating and pre-training multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network, and by adversarial alternating training based on the constrained Markov decision process corresponding to the distribution network, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm. The reference data pairs include the reference state of the distribution network and the reference dispatch action instructions corresponding to the reference state.

[0007] The power distribution network is dispatched according to dispatching action instructions.

[0008] A second aspect of this application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of any of the above methods.

[0009] A third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the method as described in any of the above.

[0010] The beneficial effects of the technical solutions provided by some embodiments of this application include at least the following: First, by acquiring real-time observation data of the distribution network, basic input is provided for subsequent scheduling decisions; then, a preset target agent is invoked to process the real-time observation data and generate scheduling action instructions for the distribution network. The Long Short-Term Memory (LSTM) neural network in the target agent is trained in two stages: the first stage is to perform imitation pre-training based on multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network. The reference data pairs include the reference state of the distribution network and the reference scheduling action instructions corresponding to the reference state. This stage rapidly improves the initial policy performance of the agent by utilizing expert knowledge; the second stage is to perform adversarial alternating training based on the constrained Markov decision process corresponding to the distribution network, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm. This stage simulates the uncertainty of renewable energy and load by introducing an adversarial disturbance mechanism and combines it with the constrained policy optimization algorithm to ensure the safety of scheduling decisions; finally, the distribution network is scheduled according to the scheduling action instructions to achieve real-time optimized operation. This method significantly shortens the initial exploration time of the agent and improves training efficiency by mimicking pre-training; it effectively reduces the risk of violating safety constraints such as voltage and line capacity during scheduling through a constraint policy optimization algorithm; it enhances the agent's robustness against renewable energy output fluctuations and load forecasting errors through an adversarial alternating training algorithm; and it solves the problem of constraint coupling relationships between time periods by capturing the dynamic evolution characteristics of the distribution network through a Long Short-Term Memory (LSTM) neural network. Therefore, this invention can simultaneously achieve real-time distribution network scheduling with high computational efficiency, low constraint violation rate, and strong uncertainty handling capabilities, significantly improving the security, robustness, and economy of distribution network scheduling. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0012] Figure 1 A flowchart illustrating a power distribution network dispatching method provided in one embodiment of this application;

[0013] Figure 2 A flowchart for obtaining a target intelligent agent is provided as an embodiment of this application;

[0014] Figure 3 A schematic diagram illustrating the linearization process of branch power constraints provided in one embodiment of this application;

[0015] Figure 4 is a neural network structure diagram provided in an embodiment of this application;

[0016] Figure 5 This is a diagram illustrating the training process of three sets of experiments provided in one embodiment of this application.

[0017] Figure 6 The image shows the effect of four sets of experiments on the test set provided in one embodiment of this application.

[0018] Figure 7 This is a diagram illustrating the training process of an adversarial agent according to one embodiment of this application.

[0019] Figure 8 An overall framework diagram provided for one embodiment of this application.

[0020] Figure 9 This is a schematic diagram of a computer device structure provided in one embodiment of this application. Detailed Implementation

[0021] The technical solutions in this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in the text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0022] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0023] In distribution network dispatching technologies, optimization methods based on mathematical models are primarily relied upon, such as using convex relaxation AC power flow models for optimization calculations. These methods reduce computational complexity and improve efficiency by convexly relaxing nonlinear constraints. However, this approach has significant drawbacks: First, the convex relaxation process introduces model bias, potentially leading to optimization results that violate safety constraints such as voltage and line capacity, resulting in voltage overruns or line overloads, thus affecting the safe operation of the distribution network. Second, while non-convex optimization methods (such as the primal-dual interior-point method, gradient method, and heuristic algorithms) can improve solution accuracy to some extent, their computational complexity increases significantly when dealing with large-scale distribution networks, making it difficult to meet the demands of high-frequency, real-time dispatching. Furthermore, these technologies do not fully consider the constraint coupling relationships between different time periods in distribution network operation, such as the continuity of energy storage device charging and discharging states and generator ramping constraints, making it difficult to effectively capture the dynamic evolution characteristics of the system and limiting their applicability in complex scenarios. Especially when facing fluctuations in renewable energy output and load forecasting errors, related technologies lack effective mechanisms to ensure robust decision-making and cannot cope with the challenges posed by source-load uncertainty. Specifically, renewable energy output (such as wind and solar power) exhibits significant randomness and volatility, while load demand also suffers from forecasting errors. These uncertainties directly affect the accuracy and reliability of dispatch decisions. Related technologies typically employ deterministic optimization models, failing to fully consider the impact of these uncertainties, potentially leading to dispatch deviations or safety risks in actual operation. Furthermore, related technologies also have shortcomings in handling certain observability issues; for example, in cases of incomplete or noisy observation information, accurate dispatch decisions are difficult to make. Based on these deficiencies, related technologies struggle to simultaneously achieve high computational efficiency, low constraint violation rates, and strong uncertainty handling capabilities in real-time distribution network dispatch, failing to meet the demands of modern power systems for efficient, safe, and robust dispatch. Therefore, there is an urgent need for a real-time distribution network dispatch method that effectively combines computational efficiency, safety constraints, and robustness to address the challenges posed by renewable energy and load uncertainties and ensure the safe and stable operation of the power system.

[0024] To address the aforementioned issues, this application provides a distribution network scheduling method. This method first acquires real-time observation data of the distribution network to provide basic input for subsequent scheduling decisions. Then, a pre-defined target agent is invoked to process the real-time observation data, generating scheduling action instructions for the distribution network. The Long Short-Term Memory (LSTM) neural network in the target agent is trained in two stages: the first stage involves simulated pre-training based on multiple reference data pairs under the constrained Markov decision process (CPM) corresponding to the distribution network. These reference data pairs include the reference state of the distribution network and the corresponding reference scheduling action instructions. This stage rapidly improves the agent's initial strategy performance by utilizing expert knowledge. The second stage involves adversarial alternating training based on the CPM corresponding to the distribution network, the constrained policy optimization algorithm corresponding to the CPM, and an adversarial alternating training algorithm. This stage simulates the uncertainty of renewable energy and load by introducing an adversarial disturbance mechanism and combines it with the constrained policy optimization algorithm to ensure the safety of scheduling decisions. Finally, the distribution network is scheduled according to the scheduling action instructions to achieve real-time optimized operation. This method significantly shortens the initial exploration time of the agent and improves training efficiency by mimicking pre-training; it effectively reduces the risk of violating safety constraints such as voltage and line capacity during scheduling through a constraint policy optimization algorithm; it enhances the agent's robustness against renewable energy output fluctuations and load forecasting errors through an adversarial alternating training algorithm; and it solves the problem of constraint coupling relationships between time periods by capturing the dynamic evolution characteristics of the distribution network through a Long Short-Term Memory (LSTM) neural network. Therefore, this invention can simultaneously achieve real-time distribution network scheduling with high computational efficiency, low constraint violation rate, and strong uncertainty handling capabilities, significantly improving the security, robustness, and economy of distribution network scheduling.

[0025] The following will combine Figure 1 - Figure 8 This paper provides a detailed description of the power distribution network dispatching method provided in the embodiments of this application.

[0026] Please see Figure 1 , Figure 1 This is a flowchart illustrating a power distribution network dispatching method according to one embodiment of this application. Figure 1 As shown, the method in this application embodiment may include the following steps S101-S103.

[0027] S101, acquire real-time observation data of the power distribution network.

[0028] Specifically, the distribution network involved in this embodiment refers to a power system network composed of multiple power nodes, power branches, conventional generator sets, renewable energy generator sets (such as wind turbines and photovoltaic generators), energy storage devices, and loads. This distribution network is connected to the upper-level power grid through a common coupling point, enabling the transmission and distribution of electrical energy. Real-time observation data of the distribution network refers to various electrical quantity data reflecting the operating status of the distribution network collected at specific times, including but not limited to: active and reactive loads of each node, active and reactive power output of each conventional generator set, predicted active power output of each wind turbine and photovoltaic generator set, energy status of each energy storage device, voltage amplitude and phase angle of each node, active and reactive power of each branch, and active and reactive power of the common coupling point. This real-time observation data comprehensively reflects the current operating status of the distribution network, providing a basic input for subsequent dispatching decisions.

[0029] The process of acquiring real-time observation data of the distribution network can be implemented in several possible ways, including the following steps: First, sensors and data acquisition devices are deployed at various nodes and branches of the distribution network to monitor and record electrical quantities in real time. These sensors and data acquisition devices can include voltage transformers, current transformers, power measurement devices, and power quality monitoring devices. Second, the acquired data is transmitted to a data processing center via a communication network, which can include wired communication (such as fiber optic communication) and wireless communication (such as 4G / 5G communication). Then, the received data is preprocessed at the data processing center, including data cleaning, data verification, and data format conversion, to eliminate noise and outliers and ensure data accuracy and consistency. Finally, the preprocessed data is stored in a database and organized according to time series for subsequent dispatch decision-making modules to access and analyze. In some possible implementations, the real-time observation data of the distribution network is pre-stored by other devices, allowing direct retrieval of this data from those devices. These steps enable efficient and accurate acquisition of real-time observation data of the distribution network, providing reliable data support for subsequent dispatch decisions.

[0030] S102, the preset target agent is invoked to process the real-time observation data to obtain the dispatch action instructions of the distribution network. The long short-term memory (LSTM) neural network in the target agent is obtained by imitating and pre-training multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network, and by adversarial alternating training based on the constrained Markov decision process corresponding to the distribution network, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm. The reference data pairs include the reference state of the distribution network and the reference dispatch action instructions corresponding to the reference state.

[0031] Specifically, the target agent involved in this embodiment is a pre-set agent used for real-time optimization and scheduling of the distribution network. The target agent includes an LSTM neural network, specifically: the LSTM policy network, LSTM reward value network, and LSTM constraint cost value network of the target agent all adopt the LSTM neural network structure to capture the time series characteristics of the distribution network operation, enhance the agent's ability to remember historical information, and thus make more accurate decisions in an environment with incomplete observability.

[0032] The process of acquiring the target agent is as follows: First, an initial agent and an adversary agent are constructed. The initial agent includes an LSTM policy network, an LSTM reward value network, and an LSTM constrained cost value network. The adversary agent includes a policy network and a reward value network, and the parameters of these networks are initialized. Second, based on the linear power flow optimization algorithm corresponding to the distribution network, multiple reference data pairs under the constrained Markov decision process of the distribution network are acquired. These reference data pairs include the reference state of the distribution network and the reference scheduling action instructions corresponding to the reference state. Then, based on the multiple reference data pairs, the initial agent is pre-trained by imitation. Specifically, the reference data pairs are divided into multiple batches, and the neural network parameters of the initial agent are updated using the gradient descent algorithm until a preset stopping condition is met, resulting in a pre-trained agent. Finally, based on the Constrained Markov Decision Process (CMM), the corresponding constrained policy optimization algorithm, and the adversarial alternating training algorithm, the pre-trained agent and the adversary agent are called for adversarial alternating training. Specifically, this involves fixing the parameters of the adversary agent, executing decisions and collecting trajectory data through the pre-trained agent, and updating the parameters of the pre-trained agent; then fixing the parameters of the pre-trained agent again, applying adversarial perturbations through the adversary agent and collecting trajectory data, and updating the parameters of the adversary agent, until the preset stopping condition is met, thus obtaining the target agent.

[0033] After acquiring real-time observation data of the distribution network, a pre-defined target agent can be invoked to process the data and obtain dispatching instructions for the distribution network. Specifically, this process involves inputting the real-time observation data into the LSTM policy network of the target agent, which then outputs dispatching instructions for the distribution network based on the input real-time observation data and historical information.

[0034] Through the above process, the target agent can generate reasonable dispatching action commands based on real-time observation data of the distribution network, achieving real-time optimized dispatching of the distribution network. By imitating pre-training and adversarial alternating training, the decision-making efficiency and robustness of the agent are significantly improved, effectively addressing the uncertainties caused by renewable energy output fluctuations and load forecasting errors. At the same time, the constrained policy optimization algorithm ensures the safety of dispatching decisions, providing an efficient, safe, and robust solution for real-time optimized dispatching of the distribution network.

[0035] S103, dispatches the distribution network according to dispatch action instructions.

[0036] Specifically, the scheduling action commands involved in this embodiment refer to commands generated by the target intelligent agent based on real-time observation data of the distribution network, used to adjust the operating status of various devices in the distribution network. These commands include, but are not limited to: active and reactive power scheduling commands for conventional generator sets, active and reactive power scheduling commands for wind turbine generator sets, active and reactive power scheduling commands for photovoltaic generator sets, and active and reactive power scheduling commands for energy storage devices. These scheduling action commands are used to control the operating status of various devices in the distribution network to achieve real-time optimized scheduling of the distribution network, reduce operating costs, and ensure the safety and stability of system operation.

[0037] Regarding the process of dispatching the distribution network according to dispatch action instructions, some possible implementations can be achieved through the following steps: First, the dispatch action instructions generated by the target intelligent agent are transmitted to the control center of the distribution network. The control center generates specific control signals based on the dispatch action instructions. Second, the control signals are sent to various devices in the distribution network, including conventional generator sets, wind turbine generator sets, photovoltaic generator sets, and energy storage devices, through a communication network. Then, each device adjusts its operating status according to the received control signals, such as adjusting the active and reactive power output of conventional generator sets, adjusting the active and reactive power output of wind turbine generator sets and photovoltaic generator sets, and adjusting the charging and discharging power of energy storage devices. Finally, during the dispatching process, the operating status of each device and the overall operation of the distribution network are monitored in real time to ensure that the execution effect of the dispatch action instructions meets expectations and to make dynamic adjustments based on the actual situation. Through the above steps, the distribution network can be dispatched efficiently and accurately according to dispatch action instructions, achieving real-time optimized operation of the distribution network.

[0038] In this embodiment, real-time observation data of the distribution network is first acquired to provide basic input for subsequent scheduling decisions. Then, a pre-defined target agent is invoked to process the real-time observation data and generate scheduling action instructions for the distribution network. The LSTM neural network in the target agent is trained in two stages: the first stage involves simulated pre-training based on multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network. These reference data pairs include the reference state of the distribution network and the reference scheduling action instructions corresponding to the reference state. This stage utilizes expert knowledge to quickly improve the initial policy performance of the agent. The second stage involves adversarial alternating training based on the constrained Markov decision process corresponding to the distribution network, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and an adversarial alternating training algorithm. This stage introduces an adversarial disturbance mechanism to simulate the uncertainty of renewable energy and load, and combines it with the constrained policy optimization algorithm to ensure the safety of scheduling decisions. Finally, the distribution network is scheduled according to the scheduling action instructions to achieve real-time optimized operation. This embodiment significantly shortens the initial exploration time of the agent and improves training efficiency by mimicking pre-training; it effectively reduces the risk of violating safety constraints such as voltage and line capacity during scheduling through a constraint policy optimization algorithm; it enhances the agent's robustness against renewable energy output fluctuations and load forecasting errors through an adversarial alternating training algorithm; and it solves the problem of constraint coupling relationships between time periods by capturing the dynamic evolution characteristics of the distribution network through an LSTM neural network. Therefore, this embodiment can simultaneously achieve real-time distribution network scheduling with high computational efficiency, low constraint violation rate, and strong uncertainty handling capabilities, significantly improving the security, robustness, and economy of distribution network scheduling.

[0039] Please see Figure 2 , Figure 2 This application provides a schematic diagram of a process for obtaining a target intelligent agent according to one embodiment of the present application, such as... Figure 2 As shown, the method in this application embodiment may include the following steps S201-S204, and steps S201-S204 may be performed in... Figure 1 The procedure is performed before step S101 in the illustrated embodiment.

[0040] S201, Construct the initial agent and the adversary agent, wherein the initial agent includes multiple corresponding LSTM neural networks, and the adversary agent includes multiple corresponding neural networks.

[0041] S202, based on the linear optimization power flow algorithm corresponding to the distribution network, obtains multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network;

[0042] S203, based on multiple reference data pairs, the initial agent is pre-trained by imitation to obtain a pre-trained agent;

[0043] S204. Based on the constrained Markov decision process, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm, the pre-trained agent and the adversary agent are called to perform adversarial alternating training to obtain the target agent, wherein the target agent is formed by updating the parameters of the pre-trained agent through the neural network.

[0044] Specifically, the initial agent in this embodiment refers to the agent constructed in the early stages of training. It includes an LSTM policy network, an LSTM reward value network, and an LSTM constrained cost value network. These networks all employ an LSTM neural network structure to capture time-series characteristics of the distribution network operation, enhancing the agent's ability to remember historical information and thus making more accurate decisions in environments with incomplete observability. The adversary agent refers to the agent used to simulate renewable energy and load uncertainty. It includes a policy network and a reward value network. These networks employ a fully connected neural network structure to generate adversarial disturbances, interfering with the initial agent's decision-making process and thereby enhancing the initial agent's robustness.

[0045] Regarding the process of constructing the initial agent and the adversary agent, some possible implementations can be achieved through the following steps: First, construct the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network of the initial agent. These networks consist of an input layer, an LSTM layer, a fully connected layer, and an output layer. The LSTM layer captures time-series features, the fully connected layer extracts nonlinear features, and the output layer generates the policy or value estimate. Second, initialize the parameters of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network, specifically including the initialization of the weight matrix and bias vector. This can be done using random initialization or pre-training initialization methods. Then, construct the policy network and reward value network of the adversary agent. These networks consist of an input layer, a fully connected layer, and an output layer. The fully connected layer extracts nonlinear features, and the output layer generates adversarial perturbations or value estimates. Finally, initialize the parameters of the adversary agent's policy network and reward value network, specifically including the initialization of the weight matrix and bias vector. This can be done using random initialization or pre-training initialization methods. Through the above steps, the initial agent and the adversary agent can be constructed efficiently and accurately, providing a foundation for subsequent imitation pre-training and adversarial alternating training.

[0046] Regarding the linear power flow optimization algorithm for the distribution network, the process of obtaining multiple reference data pairs under the constrained Markov decision process (CDM) of the distribution network can be implemented through the following steps in some possible methods: First, the linear power flow optimization algorithm iteratively solves the historical operating data of the distribution network to generate multiple reference states of the distribution network and reference scheduling action instructions corresponding to each reference state. The linear power flow optimization algorithm transforms nonlinear constraints into linear forms through Taylor expansion, thereby significantly reducing the computational burden and facilitating the generation of large amounts of expert knowledge. Second, based on each reference state and the corresponding reference scheduling action instructions, multiple reference data pairs are determined. These reference data pairs include the reference states of the distribution network and the corresponding reference scheduling action instructions. Through the above steps, multiple reference data pairs under the constrained Markov decision process of the distribution network can be obtained efficiently and accurately, providing a foundation for subsequent simulation pre-training.

[0047] Regarding the process of obtaining a pre-trained agent by imitating the initial agent based on multiple reference data pairs, some possible implementations can be achieved through the following steps: First, during the imitation pre-training process of the initial agent, the multiple reference data pairs are divided into multiple batches, with each batch containing a certain number of reference data pairs. Second, for any reference data pair in the current batch, the reference state of the current batch is input into the initial agent, resulting in the output of the initial agent of the predicted scheduling action instruction corresponding to the reference state in the current batch. Then, loss calculation is performed based on the predicted scheduling action instruction corresponding to the reference state in the current batch and the reference scheduling action instruction in the current batch, yielding the mean squared error loss for the current batch. Finally, based on the mean squared error loss of the current batch, the neural network parameters of the initial agent are updated using the gradient descent algorithm. If the preset first stopping condition is not met, the initial agent is pre-trained by imitation using the reference data of the next batch. If the preset first stopping condition is met, the pre-training process of the initial agent ends, and the initial agent is determined as the pre-trained agent. Through the above steps, the initial agent can be pre-trained efficiently and accurately, significantly shortening the initial exploration time of the agent and improving training efficiency.

[0048] Regarding the process of obtaining a target agent by invoking a pre-trained agent and an adversary agent through adversarial alternating training based on a Constrained Markov Decision Process (CMM), the corresponding constraint policy optimization algorithm, and an adversarial alternating training algorithm, some possible implementations can be achieved through the following steps: First, invoking the pre-trained agent and the adversary agent for multiple iteration cycles of adversarial alternating training. Second, for any target iteration cycle within these multiple iteration cycles, fixing the neural network parameters of the adversary agent. Then, in the distribution network environment defined by the CMM, executing decisions through the pre-trained agent and collecting its raw trajectory data. Next, updating the neural network parameters of the pre-trained agent based on the raw trajectory data and the constraint policy optimization algorithm corresponding to the CMM. Subsequently, fixing the updated neural network parameters of the pre-trained agent. Finally, applying adversarial perturbations to the pre-trained agent through the adversary agent and collecting the perturbation trajectory data of the adversary agent. Finally, based on the perturbation trajectory data and the constrained policy optimization algorithm corresponding to the constrained Markov decision process, the neural network parameters of the adversary agent are updated. If the preset second stopping condition is not met, the process proceeds to the next iteration cycle of the target iteration cycle, where the pre-trained agent and the adversary agent are called for adversarial alternating training. If the preset second stopping condition is met, the adversarial alternating training process between the pre-trained agent and the adversary agent ends, and the pre-trained agent is identified as the target agent. Through the above steps, adversarial alternating training can be carried out efficiently and accurately, significantly enhancing the robustness of the agent in dealing with renewable energy output fluctuations and load forecasting errors.

[0049] In this embodiment, an initial agent and an adversary agent are first constructed to lay the foundation for subsequent imitation pre-training and adversarial alternating training. Then, based on the linear power flow optimization algorithm corresponding to the distribution network, multiple reference data pairs under the constrained Markov decision process (CPM) of the distribution network are obtained, providing a basis for imitation pre-training. Next, based on the multiple reference data pairs, the initial agent is subjected to imitation pre-training to obtain a pre-trained agent, significantly shortening the initial exploration time and improving training efficiency. Finally, according to the constrained Markov decision process, the corresponding constraint policy optimization algorithm, and the adversarial alternating training algorithm, the pre-trained agent and the adversary agent are called for adversarial alternating training to obtain the target agent, significantly enhancing the agent's robustness against renewable energy output fluctuations and load forecasting errors. This embodiment, through imitation pre-training and adversarial alternating training, significantly improves the agent's decision-making efficiency and robustness, effectively addressing the uncertainties caused by renewable energy output fluctuations and load forecasting errors. Simultaneously, the constraint policy optimization algorithm ensures the safety of scheduling decisions, providing an efficient, safe, and robust solution for real-time optimized scheduling of the distribution network.

[0050] In one embodiment, for Figure 2 Step S201 of the illustrated embodiment can be further refined to include the following steps:

[0051] Construct an LSTM policy network, an LSTM reward value network, and an LSTM constrained cost value network;

[0052] Initialize the parameters of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network to obtain an initial agent composed of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network.

[0053] Construct policy networks and reward value networks;

[0054] Initialize the parameters of the policy network and the reward value network to obtain the adversary agent composed of the policy network and the reward value network.

[0055] Specifically, the LSTM policy network involved in this embodiment refers to a neural network used to generate scheduling action instructions. Its structure includes an input layer, an LSTM layer, a fully connected layer, and an output layer. The input layer is used to receive real-time observation data of the distribution network, the LSTM layer is used to capture time-series features, the fully connected layer is used to extract nonlinear features, and the output layer is used to generate scheduling action instructions. The LSTM reward value network refers to a neural network used to estimate reward values. Its structure includes an input layer, an LSTM layer, a fully connected layer, and an output layer. The input layer is used to receive real-time observation data of the distribution network, the LSTM layer is used to capture time-series features, the fully connected layer is used to extract nonlinear features, and the output layer is used to estimate reward values. The LSTM constraint cost value network refers to a neural network used to estimate constraint violation costs. Its structure includes an input layer, an LSTM layer, a fully connected layer, and an output layer. The input layer is used to receive real-time observation data of the distribution network, the LSTM layer is used to capture time-series features, the fully connected layer is used to extract nonlinear features, and the output layer is used to estimate constraint violation costs. A policy network refers to a neural network used to generate adversarial disturbances. Its structure includes an input layer, a fully connected layer, and an output layer. The input layer receives real-time observation data from the distribution network, the fully connected layer extracts nonlinear features, and the output layer generates adversarial disturbances. A reward value network refers to a neural network used to estimate the reward value for adversarial disturbances. Its structure also includes an input layer, a fully connected layer, and an output layer. The input layer receives real-time observation data from the distribution network, the fully connected layer extracts nonlinear features, and the output layer estimates the reward value for adversarial disturbances.

[0056] The process of constructing the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network can be implemented in several possible ways, specifically through the following steps: First, determine the input layer dimensions of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network. These input layer dimensions should be consistent with the dimensions of the real-time observation data of the distribution network. Second, determine the LSTM layer dimensions of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network. These LSTM layer dimensions are determined based on the time-series characteristics and complexity of the distribution network. Then, determine the dimensions of the fully connected layers of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network. These fully connected layer dimensions are determined based on the nonlinear characteristics and complexity of the distribution network. Finally, determine the output layer dimensions of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network. These output layer dimensions should be consistent with the dimensions of the scheduling action commands.

[0057] The initialization of the parameters of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network can be implemented in several possible ways, specifically through the following steps: First, initialize the weight matrices of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network. This initialization can be achieved using random initialization or pre-training initialization methods. Second, initialize the bias vectors of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network. This initialization can be achieved using zero initialization or pre-training initialization methods. These steps efficiently and accurately initialize the parameters of the LSTM policy network, LSTM reward value network, and LSTM constrained cost value network, providing a foundation for subsequent imitation pre-training.

[0058] Regarding the construction of the policy network and reward value network, some possible implementations can be achieved through the following steps: First, determine the input layer dimensions of the policy network and reward value network, ensuring these dimensions match the dimensions of the real-time observation data of the distribution network. Second, determine the dimensions of the fully connected layers of the policy network and reward value network, based on the nonlinear complexity of the distribution network. Finally, determine the output layer dimensions of the policy network and reward value network, ensuring these dimensions match the dimensions used to combat disturbances.

[0059] The process of initializing the parameters of the policy network and the reward / value network can be implemented in several possible ways, specifically through the following steps: First, initialize the weight matrices of the policy network and the reward / value network. The weight matrices can be initialized using random initialization or pre-training initialization methods. Second, initialize the bias vectors of the policy network and the reward / value network. The bias vectors can be initialized using zero initialization or pre-training initialization methods. These steps efficiently and accurately initialize the parameters of the policy network and the reward / value network, providing a foundation for subsequent adversarial alternation training.

[0060] For example, when constructing an LSTM policy network, the input layer dimension is D, the LSTM layer dimension is H, the fully connected layer dimension is F, and the output layer dimension is A; the weight matrix of the LSTM policy network is initialized as W1, and the bias vector is b1. When constructing an LSTM reward value network, the input layer dimension is D, the LSTM layer dimension is H, the fully connected layer dimension is F, and the output layer dimension is 1; the weight matrix of the LSTM reward value network is initialized as W2, and the bias vector is b2. When constructing an LSTM constrained cost value network, the input layer dimension is D, the LSTM layer dimension is H, the fully connected layer dimension is F, and the output layer dimension is 1; the weight matrix of the LSTM constrained cost value network is initialized as W3, and the bias vector is b3. When constructing a policy network, the input layer dimension is D, the fully connected layer dimension is F, and the output layer dimension is A; the weight matrix of the policy network is initialized as W4, and the bias vector is b4. When constructing the reward value network, the input layer dimension is D, the fully connected layer dimension is F, and the output layer dimension is 1; the weight matrix of the reward value network is initialized to W5, and the bias vector is b5. According to the formula...

[0061]

[0062] Where a t This represents the action at time t. This represents the active power dispatch command for conventional generator set k at time t. This represents the reactive power dispatch command for conventional generator set k at time t. This represents the active power dispatch command for wind turbine generator k at time t. This represents the reactive power dispatch command for wind turbine generator k at time t. This represents the active power dispatch command for photovoltaic generator set k at time t. This represents the reactive power dispatch command for photovoltaic generator set k at time t. This represents the active power dispatch command for energy storage device k at time t. This represents the reactive power scheduling instruction for energy storage device k at time t. The LSTM policy network outputs the scheduling action instruction. According to the formula...

[0063] r t (s t ,a t )=-(C1+C2+C3+C4),

[0064] Where r t Let C1 represent the reward at time t, C2 represent the cost of conventional generator power generation, C3 represent the penalty cost of wind and solar curtailment, C4 represent the cost of purchasing electricity from the upper-level grid at the common coupling point, and C5 represent the energy consumption cost of the energy storage device. The LSTM reward value network outputs the reward value. According to the formula...

[0065] c t (s t ,a t )=(C V +C S +C P +C PCC ),

[0066] Where c t C represents the constraint violation cost at time t. V Indicates the cost of violating voltage constraints, C S Indicates the cost of violating line capacity constraints, C P This indicates that the constraint of conventional unit ramp-up violates the cost, C PCC The constraint violation cost represents the energy transmission capacity between the upstream power grid and the common coupling point. The LSTM constraint cost value network outputs the constraint violation cost. The policy network outputs the disturbance mitigation reward value, and the reward value network outputs the disturbance mitigation reward value.

[0067] In this embodiment, an LSTM policy network, an LSTM reward value network, and an LSTM constrained cost value network are first constructed to lay the foundation for subsequent imitation pre-training. Then, the parameters of these networks are initialized to obtain an initial agent composed of these networks. Next, the policy network and reward value network are constructed to lay the foundation for subsequent adversarial alternation training. Finally, the parameters of the policy network and reward value network are initialized to obtain an adversarial agent composed of these networks. This embodiment, by constructing and initializing the LSTM policy network, LSTM reward value network, LSTM constrained cost value network, and policy network, provides an efficient and accurate foundation for subsequent imitation pre-training and adversarial alternation training, significantly improving the agent's decision-making efficiency and robustness. It can effectively cope with the uncertainties caused by renewable energy output fluctuations and load forecasting errors, while ensuring the safety of scheduling decisions through constrained policy optimization algorithms. This provides an efficient, safe, and robust solution for real-time optimized scheduling of distribution networks.

[0068] In one embodiment, for Figure 2 Step S202 of the illustrated embodiment can be further refined to include the following steps:

[0069] Based on the linear optimization power flow algorithm corresponding to the distribution network, the historical operation data of the distribution network is iteratively solved to generate multiple reference states of the distribution network and reference scheduling action instructions corresponding to each reference state.

[0070] Based on each reference state and the corresponding reference scheduling action instruction, multiple reference data pairs are determined.

[0071] Specifically, the linear power flow optimization algorithm involved in this embodiment refers to an optimization algorithm that transforms nonlinear constraints into linear forms through Taylor expansion, thereby significantly reducing computational complexity. This algorithm iteratively solves the historical operating data of the distribution network to generate multiple reference states of the distribution network and corresponding reference scheduling action commands for each reference state. The historical operating data of the distribution network includes, but is not limited to: active and reactive loads of each node, active and reactive power outputs of each conventional generator set, predicted active power outputs of each wind turbine and photovoltaic generator set, energy status of each energy storage device, voltage amplitude and phase angle of each node, active and reactive power of each branch, and active and reactive power at the common coupling point. A reference state refers to the operating state of the distribution network at a specific moment, including the active and reactive loads of each node, predicted active power outputs of each wind turbine and photovoltaic generator set, active power output of each conventional generator set, energy status of each energy storage device, and timing information. Reference dispatching instructions refer to the dispatching instructions for various devices in the distribution network at a specific time, including active and reactive power dispatching instructions for conventional generator sets, active and reactive power dispatching instructions for wind turbine generator sets and photovoltaic generator sets, and active and reactive power dispatching instructions for energy storage devices. Reference data pairs refer to data pairs consisting of reference states and reference dispatching instructions, used for simulation pre-training.

[0072] Regarding the process of iteratively solving historical operating data of the distribution network based on the linear power flow optimization algorithm corresponding to the distribution network, and generating multiple reference states of the distribution network and reference scheduling action instructions corresponding to each reference state, in some possible implementations, it can be achieved through the following steps: First, acquire historical operating data of the distribution network, including active and reactive loads of each node, active and reactive power outputs of each conventional generator set, predicted active power outputs of each wind turbine and photovoltaic generator set, energy status of each energy storage device, voltage amplitude and phase angle of each node, active and reactive power of each branch, active and reactive power of the common coupling point, etc. Second, iteratively solve the historical operating data based on the linear power flow optimization algorithm to generate multiple reference states of the distribution network, including active and reactive loads of each node, predicted active power outputs of each wind turbine and photovoltaic generator set, active power outputs of each conventional generator set, energy status of each energy storage device, and time-series information, etc. Then, based on the solution results of the linear power flow algorithm, reference scheduling action instructions corresponding to each reference state are generated, including active and reactive power scheduling instructions for conventional generator sets, active and reactive power scheduling instructions for wind turbine generator sets and photovoltaic generator sets, and active and reactive power scheduling instructions for energy storage devices.

[0073] Regarding the process of determining multiple reference data pairs based on each reference state and its corresponding reference scheduling action instruction, some possible implementations can be achieved through the following steps: First, pair each reference state with its corresponding reference scheduling action instruction to form multiple reference data pairs. Second, clean and validate the reference data pairs to ensure their accuracy and consistency. Finally, store the cleaned and validated reference data pairs in a database for subsequent imitation pre-training calls.

[0074] For example, assume that the historical operating data of the distribution network includes the active load of node 1. Reactive load of node 1 Active power output of conventional generator set 1 Predicted active power output of wind turbine generator set 1 Predicted active power output of photovoltaic generator unit 1 Energy state E of energy storage device 1 1,t-1 Voltage amplitude V at node 1 1,t Phase angle θ of node 1 1,t Active power of branch 1 reactive power of branch 1 Active power P at the common coupling point t PCC and the reactive power at the common coupling point First, the historical operating data is iteratively solved using a linear power flow optimization algorithm (also known as a linear optimal algorithm) to generate the reference state s of the distribution network. t , where s t Defined as:

[0075]

[0076] Where aux t This represents time-series information, including time t and its trigonometric function values. The linear optimization power flow algorithm is a power flow calculation method based on a linearized model. By linearizing the nonlinear power flow equations, it can efficiently solve for the operating state of the distribution network, making it suitable for real-time optimization and scheduling of large-scale systems. Then, based on the solution results of the linear optimization power flow algorithm, a reference state s is generated. t The corresponding reference scheduling action instruction a t , where a t Defined as:

[0077]

[0078] in This represents the active power dispatch command for conventional generator set 1 at time t. This represents the reactive power dispatch command for conventional generator set 1 at time t. This represents the active power dispatch command for wind turbine generator set 1 at time t. This represents the reactive power dispatch command for wind turbine generator set 1 at time t. This represents the active power dispatch command for photovoltaic generator unit 1 at time t. This represents the reactive power dispatch command for photovoltaic generator unit 1 at time t. This represents the active power dispatch command for energy storage device 1 at time t. This represents the reactive power dispatch command for energy storage device 1 at time t. Next, the reference state s... t With reference scheduling action instruction a t Pairing to form reference data pairs (s t ,a t Finally, for the reference data pair (s) t ,a t Data cleaning and verification are performed to ensure the accuracy and consistency of the data, and the data is stored in the database for subsequent imitation pre-training calls.

[0079] In this embodiment, the historical operating data of the distribution network is first iteratively solved using a linear power flow optimization algorithm based on the distribution network to generate multiple reference states of the distribution network and reference scheduling action instructions corresponding to each reference state, providing a foundation for subsequent imitation pre-training. Then, based on each reference state and its corresponding reference scheduling action instructions, multiple reference data pairs are determined, providing direct reference for imitation pre-training. This embodiment generates reference data pairs using a linear power flow optimization algorithm, significantly reducing computational complexity while ensuring data accuracy and consistency. This provides an efficient and reliable foundation for subsequent imitation pre-training, significantly improving the initial policy performance of the agent, shortening the initial exploration time, and increasing training efficiency.

[0080] In one embodiment, for Figure 2 Step S203 of the illustrated embodiment can be further refined to include the following steps:

[0081] During the imitation pre-training process of the initial agent, multiple reference data pairs are divided into multiple batches of reference data pairs;

[0082] For any reference data pair in the current batch among multiple batches of reference data pairs, the reference state in the reference data pair of the current batch is input into the initial agent, and the predictive scheduling action instruction corresponding to the reference state in the reference data pair of the current batch is output by the initial agent.

[0083] The mean square error loss of the current batch is obtained by calculating the loss between the predicted scheduling action command corresponding to the reference state based on the reference data of the current batch and the reference scheduling action command based on the reference data of the current batch.

[0084] Based on the mean squared error loss of the current batch, the neural network parameters of the initial agent are updated using the gradient descent algorithm;

[0085] If the preset first stopping condition is not met, the initial agent will be pre-trained by imitation based on the reference data pair of the next batch of the current batch.

[0086] If the preset first stopping condition is met, the imitation pre-training process of the initial agent ends, and the initial agent is identified as the pre-trained agent.

[0087] Specifically, in this embodiment, the multiple batches of reference data pairs refer to dividing multiple reference data pairs into multiple subsets according to a certain number, with each subset called a batch. The reference data pair of the current batch refers to the batch currently being processed during the imitation pre-training process. The predicted scheduling action instruction refers to the scheduling action instruction generated by the initial agent based on the input reference state. The mean squared error loss refers to the mean squared error between the predicted scheduling action instruction and the reference scheduling action instruction, used to measure the prediction accuracy of the initial agent. The gradient descent algorithm refers to an optimization algorithm that minimizes the loss function by calculating the gradient of the loss function with respect to the neural network parameters and updating the parameters in the opposite direction of the gradient. The preset first stopping condition refers to the condition used to determine whether to stop training during the imitation pre-training process, such as the number of training epochs reaching a preset value or the loss function converging to a preset threshold.

[0088] Regarding the process of dividing multiple reference data pairs into multiple batches during the imitation pre-training of the initial agent, some possible implementations can be achieved through the following steps: First, determine the number of reference data pairs in each batch, with the batch size weighed against computational resources and training efficiency. Second, divide the multiple reference data pairs into multiple batches according to the batch size, ensuring that each batch contains the same number of reference data pairs. Finally, store the divided batches of reference data pairs in memory or on disk for subsequent use in imitation pre-training.

[0089] Regarding the process of inputting the reference state of any current batch of reference data pairs from multiple batches into the initial agent to obtain the predictive scheduling action instructions corresponding to the reference state of the current batch of reference data pairs output by the initial agent, some possible implementations can be achieved through the following steps: First, extract the reference state from the current batch of reference data pairs. The reference state includes the active and reactive loads of each node, the predicted active power output of each wind turbine and photovoltaic generator, the active power output of each conventional generator, the energy status of each energy storage device, and timing information, etc. Second, input the extracted reference state into the LSTM policy network of the initial agent. The LSTM policy network outputs the predictive scheduling action instructions corresponding to the reference state of the current batch of reference data pairs based on the input reference state and historical information. These instructions include active and reactive power scheduling instructions for conventional generators, active and reactive power scheduling instructions for wind turbines and photovoltaic generators, and active and reactive power scheduling instructions for energy storage devices, etc.

[0090] Regarding the process of calculating the mean square error loss of the current batch by aligning the predicted scheduling action commands corresponding to the reference state with the reference scheduling action commands aligned with the reference data of the current batch, in some possible implementations, this can be achieved through the following steps: First, extract the reference scheduling action commands from the reference data pair of the current batch. These reference scheduling action commands include active and reactive power scheduling commands for conventional generator sets, active and reactive power scheduling commands for wind turbine generator sets and photovoltaic generator sets, and active and reactive power scheduling commands for energy storage devices, etc. Second, calculate the mean square error between the predicted scheduling action commands and the reference scheduling action commands, which is used as the mean square error loss of the current batch.

[0091] Regarding the process of updating the neural network parameters of the initial agent using the gradient descent algorithm based on the mean squared error loss of the current batch, some possible implementations can be achieved through the following steps: First, calculate the gradient of the mean squared error loss of the current batch with respect to the LSTM policy network parameters of the initial agent. Second, update the parameters of the LSTM policy network according to the gradient direction, specifically including updating the weight matrix and bias vector. Then, repeat the above process until all reference data pairs of the current batch have been processed.

[0092] Regarding the process of performing imitation pre-training on the initial agent based on the reference data pairs of the next batch if the preset first stopping condition is not met, in some possible implementations, this can be achieved through the following steps: First, determine whether the current training round number has reached a preset value or whether the loss function has converged to a preset threshold. Second, if the preset value has not been reached or the function has not converged to the preset threshold, continue processing the next batch of reference data pairs to perform imitation pre-training on the initial agent.

[0093] Regarding the process of ending the imitation pre-training process of the initial agent and determining the initial agent as the pre-trained agent if the preset first stopping condition is met, in some possible implementations, this can be achieved through the following steps: First, determine whether the current training round number has reached a preset value or whether the loss function has converged to a preset threshold. Second, if the preset value is reached or the preset threshold is converged, end the imitation pre-training process and determine the initial agent as the pre-trained agent.

[0094] For example, during the imitation pre-training process of the initial agent, the expert dataset D, containing M reference data pairs, is divided into K batches, each batch containing N reference data pairs (satisfying M = K × N). For the reference data pairs in the j-th batch...

[0095] Predictive scheduling action instruction generation:

[0096] Reference state Input the policy network π of the initial agent θ Generate predictive scheduling action instructions:

[0097]

[0098] in This represents the device scheduling instruction vector.

[0099] Mean squared error loss calculation:

[0100] Calculate prediction instructions With expert instructions L2 norm loss:

[0101]

[0102] in For expert action instructions, N is the number of samples in the current batch.

[0103] Neural network parameter updates:

[0104] The policy network parameters are updated using a gradient descent algorithm with a learning rate of α.

[0105]

[0106] This represents the gradient of the loss function with respect to the parameter θ.

[0107] Stop condition determination:

[0108] If satisfied (ò is a preset threshold) or j = K max If the maximum batch index is reached, then training ends and the initial agent is determined as the pre-trained agent; otherwise, the above process is repeated for the (j+1)th batch.

[0109] In this embodiment, multiple reference data pairs are first divided into multiple batches to provide an efficient data processing method for imitation pre-training. Then, for the current batch of reference data pairs, the reference state is input into the initial agent to generate predictive scheduling action instructions, and the prediction accuracy is measured by calculating the mean squared error loss. Next, based on the mean squared error loss, the neural network parameters of the initial agent are updated using the gradient descent algorithm to improve prediction accuracy. Finally, a preset first stopping condition is used to determine whether to continue training until the condition is met, at which point the imitation pre-training process ends, resulting in a pre-trained agent. This embodiment significantly improves the efficiency of imitation pre-training through batch processing and the gradient descent algorithm. Simultaneously, the mean squared error loss and preset stopping condition ensure the prediction accuracy and training stability of the pre-trained agent, providing an efficient and reliable foundation for subsequent adversarial alternation training.

[0110] In one embodiment, for Figure 2 Step S204 of the illustrated embodiment can be further refined to include the following steps:

[0111] The pre-trained agent and the adversary agent are called to perform adversarial training for multiple iterations;

[0112] For any target iteration cycle among multiple iteration cycles, the neural network parameters of the adversary agent are fixed.

[0113] In a power distribution network environment defined by a constrained Markov decision process, decisions are made by a pre-trained agent and raw trajectory data of the pre-trained agent is collected.

[0114] Based on the original trajectory data and the constraint policy optimization algorithm corresponding to the constrained Markov decision process, the neural network parameters of the pre-trained agent are updated.

[0115] Fix the neural network parameters of the pre-trained agent after updating;

[0116] The opponent agent applies adversarial perturbations to the pre-trained agent and collects the perturbation trajectory data of the opponent agent.

[0117] The neural network parameters of the adversary agent are updated based on perturbation trajectory data and the constraint policy optimization algorithm corresponding to the constrained Markov decision process.

[0118] If the preset second stopping condition is not met, the next iteration cycle of the target iteration cycle will be entered, so that the pre-trained agent and the adversary agent can be called for adversarial alternating training in the next iteration cycle.

[0119] If the preset second stopping condition is met, the adversarial training process between the pre-trained agent and the opponent agent ends, and the pre-trained agent is identified as the target agent.

[0120] Specifically, the adversarial alternating training involving multiple iteration cycles in this embodiment refers to enhancing the robustness of the pre-trained agent by alternately fixing the neural network parameters of the pre-trained agent and the adversary agent, and updating the neural network parameters of the other party respectively. The target iteration cycle refers to the iteration cycle currently being processed during the adversarial alternating training process. The original trajectory data refers to the data collected after the pre-trained agent executes decisions in the power distribution network environment, including state, actions, rewards, and constraint violation costs. The perturbation trajectory data refers to the data collected after the adversary agent applies adversarial perturbations to the pre-trained agent, including state, actions, rewards, and constraint violation costs. The preset second stopping condition refers to the condition used to determine whether to stop training during the adversarial alternating training process, such as the number of training rounds reaching a preset value or the reward function converging to a preset threshold.

[0121] Regarding the process of invoking a pre-trained agent and an adversary agent for adversarial training over multiple iterations, some possible implementations can be achieved through the following steps: First, determine the number of iterations for adversarial training, balancing training effectiveness and computational resources. Second, according to the preset number of iterations, invoke the pre-trained agent and the adversary agent for alternating training, with each iteration including fixing the parameters of one agent and updating the parameters of the other.

[0122] Regarding the process of fixing the neural network parameters of the adversary agent for any target iteration cycle among multiple iteration cycles, some possible implementations can be achieved through the following steps: First, at the beginning of the target iteration cycle, lock the parameters of the adversary agent's policy network and reward value network to ensure that the adversary agent's parameters remain unchanged when updating the pre-trained agent's parameters. Second, set the adversary agent's parameters to an untrainable state to prevent the adversary agent's parameters from being updated during gradient calculation.

[0123] Regarding the process of executing decisions and collecting raw trajectory data of a pre-trained agent in a distribution network environment defined by a Constrained Markov Decision Process (CDM), one possible implementation involves the following steps: First, input real-time observation data of the distribution network into the LSTM policy network of the pre-trained agent to generate scheduling action instructions. Second, control the operating status of each device in the distribution network according to the scheduling action instructions, and collect the raw trajectory data of the pre-trained agent, including status, actions, rewards, and constraint violation costs.

[0124] Regarding the process of updating the neural network parameters of a pre-trained agent based on the original trajectory data and the constraint policy optimization algorithm corresponding to the constrained Markov decision process, some possible implementations can be achieved through the following steps: First, calculate the reward value and constraint violation cost of the pre-trained agent based on the original trajectory data. Second, based on the constraint policy optimization algorithm, calculate the parameter update directions of the LSTM policy network, LSTM reward value network, and LSTM constraint cost value network of the pre-trained agent. Then, update the neural network parameters of the pre-trained agent using the gradient descent algorithm to maximize the reward value and minimize the constraint violation cost.

[0125] Regarding the process of fixing the neural network parameters of the pre-trained agent after updates, some possible implementations can be achieved through the following steps: First, after updating the neural network parameters of the pre-trained agent, lock the parameters of the LSTM policy network, LSTM reward value network, and LSTM constraint cost value network of the pre-trained agent to ensure that the parameters of the pre-trained agent remain unchanged when the parameters of the adversary agent are updated. Second, set the parameters of the pre-trained agent to an untrainable state to prevent the parameters of the pre-trained agent from being updated during gradient calculation.

[0126] Regarding the process of applying adversarial perturbations to a pre-trained agent and collecting the perturbation trajectory data of the adversary agent, some possible implementations can be achieved through the following steps: First, input real-time observation data of the distribution network into the policy network of the adversary agent to generate adversarial perturbations. Second, apply the adversarial perturbations to the observation data of the pre-trained agent to interfere with its decision-making process. Then, collect the perturbation trajectory data of the adversary agent, including its state, actions, rewards, and constraint violation costs.

[0127] Regarding the process of updating the neural network parameters of an adversary agent based on perturbation trajectory data and the constraint policy optimization algorithm corresponding to a constrained Markov decision process, some possible implementations can be achieved through the following steps: First, calculate the reward value and constraint violation cost of the adversary agent based on the perturbation trajectory data. Second, calculate the parameter update directions of the policy network and reward value network of the adversary agent based on the constraint policy optimization algorithm. Then, update the neural network parameters of the adversary agent using the gradient descent algorithm to maximize the effect of adversarial perturbation on the pre-trained agent.

[0128] Regarding the process of proceeding to the next iteration cycle of the target iteration cycle if the preset second stopping condition is not met, and then invoking the pre-trained agent and the adversary agent for adversarial alternating training in the next iteration cycle, some possible implementations can be achieved through the following steps: First, determine whether the current training round number has reached a preset value or whether the reward function has converged to a preset threshold. Second, if the preset value has not been reached or the function has not converged to the preset threshold, continue to the next iteration cycle and invoking the pre-trained agent and the adversary agent for adversarial alternating training.

[0129] Regarding the process of ending the adversarial training process between the pre-trained agent and the adversary agent and identifying the pre-trained agent as the target agent if the preset second stopping condition is met, some possible implementations can be achieved through the following steps: First, determine whether the current training round number has reached a preset value or whether the reward function has converged to a preset threshold. Second, if the preset value has been reached or the preset threshold has been converged, end the adversarial training process and identify the pre-trained agent as the target agent.

[0130] For example, during adversarial alternation training:

[0131] The neural network parameters of the opposing agent are fixed:

[0132] Let the current iteration period be k=1, and the pre-trained agent policy be... The opponent's strategy is

[0133] Fixed φ k constant.

[0134] Update the pre-trained agent:

[0135] In a distribution network environment defined by a constrained Markov decision process, Execution decision generation trajectory (T = 96 steps).

[0136] The gradient g, constraint gradient b, Hessian matrix H, and constraint value z = J of constrained policy optimization (CPO) are calculated based on τ.C (π k )-d(d=0.01).

[0137] If feasible, solve the dual problem. λ * ,υ * Update parameters

[0138] If this is not feasible, use a recovery update.

[0139] The neural network parameters of the pre-trained agent are fixed:

[0140] Fixed update θ k+1 constant.

[0141] Update the adversary agent:

[0142] opponent Apply perturbation to the observations Generate disturbance trajectory

[0143] Update the opponent parameter φ using Proximity Policy Optimization (PPO). k+1 Minimize loss in This is a generalized advantage estimate.

[0144] Iteration termination judgment:

[0145] If k < K max (K max =100) and constraint violation cost J C (π k+1 )>ò c (ò c =10 -4 If k ← k+1, then return to step 1;

[0146] Otherwise, terminate training and output the target agent.

[0147] In this embodiment, adversarial alternating training is first conducted over multiple iterations using a pre-trained agent and an adversary agent, laying the foundation for enhancing the robustness of the pre-trained agent. Then, for a target iteration cycle, the neural network parameters of the adversary agent are fixed, and the pre-trained agent executes decisions and collects raw trajectory data. The neural network parameters of the pre-trained agent are updated based on a constraint policy optimization algorithm. Next, the updated neural network parameters of the pre-trained agent are fixed, and adversarial perturbations are applied by the adversary agent, and perturbation trajectory data is collected. The neural network parameters of the adversary agent are updated again based on the constraint policy optimization algorithm. Finally, a preset second stopping condition is used to determine whether to continue training, until the condition is met, ending the adversarial alternating training process and obtaining the target agent. This embodiment, through adversarial alternating training and constraint policy optimization algorithms, significantly enhances the robustness of the pre-trained agent against renewable energy output fluctuations and load forecasting errors, while ensuring the security of scheduling decisions. It provides an efficient, safe, and robust solution for real-time optimized scheduling of distribution networks.

[0148] In one embodiment, the formula corresponding to the above-mentioned constrained Markov decision process is:

[0149] (S,A,R,P,C);

[0150] Where S represents the set of states, A represents the set of actions, R represents the reward function, P represents the state transition function, and C represents the constraint violation cost function.

[0151] In one embodiment, the formula corresponding to the above state set S is expressed as:

[0152] S={s t};

[0153] Among them, s t s represents the state at time t. t Defined as:

[0154]

[0155] in, This represents the active load of node i at time t;

[0156] This represents the reactive load of node i at time t;

[0157] This represents the predicted maximum active power output of wind turbine generator k at time t.

[0158] This represents the predicted maximum active power output of photovoltaic generator set k at time t;

[0159] This represents the active power output of conventional generator set k at time t-1;

[0160] E k,t-1 This represents the energy of energy storage device k at time t-1;

[0161] aux t It represents time series information, including time t and its trigonometric function values.

[0162] In one embodiment, the formula corresponding to the above action set A is expressed as:

[0163] A={a t};

[0164] Among them, a t Let a represent the action at time t. t Defined as:

[0165]

[0166] in, This represents the active power dispatch command for conventional generator set k at time t;

[0167] This represents the reactive power dispatch command for conventional generator set k at time t;

[0168] This represents the active power dispatch command for wind turbine generator k at time t;

[0169] This represents the reactive power dispatch command for wind turbine generator k at time t;

[0170] This represents the active power dispatch instruction for photovoltaic generator set k at time t;

[0171] This represents the reactive power dispatch instruction for photovoltaic generator set k at time t;

[0172] This represents the active power dispatch command for energy storage device k at time t;

[0173] This represents the reactive power dispatch command for energy storage device k at time t.

[0174] In one embodiment, the formula corresponding to the above reward function R is expressed as:

[0175] r t (s t ,a t ) = -(C1 + C2 + C3 + C4);

[0176] Among them, s t This represents the state at time t;

[0177] a t Indicates the action at time t;

[0178] r t This represents the reward at time t;

[0179] C1 represents the power generation cost of conventional generating units;

[0180] C2 represents the penalty cost of curtailing wind and solar power;

[0181] C3 represents the cost of purchasing electricity from the upstream power grid at the common coupling point;

[0182] C4 represents the energy consumption cost of the energy storage device.

[0183] In one embodiment, the formula corresponding to the above state transition function P is expressed as:

[0184] P(s t+1 |s t ,a t );

[0185] Among them, P executes action a t After state from s t Transfer to s t+1 The probability of;

[0186] s t+1 This represents the state at time t+1;

[0187] s t This represents the state at time t;

[0188] a t This represents the action at time t.

[0189] In one embodiment, the formula corresponding to the constraint violation cost function C is expressed as:

[0190] c t (s t ,a t )=(C V +C S +C P +C PCC );

[0191] Among them, c t This represents the cost of violating the constraint at time t;

[0192] s t This represents the state at time t;

[0193] a tIndicates the action at time t;

[0194] C V This indicates the cost of violating voltage constraints;

[0195] C S This indicates that the cost of violating line capacity constraints is high.

[0196] C P This indicates that the cost of violating the constraints of ramp-up for conventional units is high.

[0197] C PCC This represents the cost of violating the constraint on the power transmission capacity between the upper-level power grid and the common coupling point.

[0198] In the aforementioned embodiments related to the Constrained Markov Decision Process (CMM), the real-time optimization scheduling problem of the distribution network is modeled as a combination of a state set, an action set, a reward function, a state transition function, and a constraint violation cost function. The state set includes the active and reactive loads of each node, the predicted maximum active power output of wind turbines and photovoltaic generators, the active power output of conventional generators, the energy of energy storage devices, and time-series information. The action set includes active and reactive power scheduling commands for conventional generators, wind turbines, photovoltaic generators, and energy storage devices. The reward function is the generation cost of conventional units and the penalty for wind and solar curtailment. The model incorporates negative values ​​for costs, point-of-common coupling (POCC) electricity purchase costs, and energy storage device energy consumption costs. A state transition function describes the probability of a state transition after an action is performed. The constraint violation cost function includes the constraint violation costs for voltage, line capacity, conventional unit ramping, and PCC power transmission capacity. Through this modeling, the operating state, dispatch instructions, operating costs, and safety constraints of the distribution network can be comprehensively considered. By combining the reward function and constraint violation cost function, the system's operational safety is ensured while optimizing operating costs. Furthermore, the dynamic evolution characteristics of the system are captured through the state transition function, thereby achieving real-time optimized dispatching of the distribution network and significantly improving the accuracy, safety, and robustness of dispatching decisions.

[0199] To better understand the role of the scheme in this application, please refer to [further details]. Figures 3 to 8 .

[0200] Specifically, this implementation verifies the effectiveness of the invention through simulation experiments. The simulation experiments were conducted on a modified IEEE30 system, using Python 3.10 of PyTorch v2.1.2 as the programming environment.

[0201] Step 1 of the embodiment introduces the model configuration of the data center aggregate, including the composition of the data center, time-of-use electricity pricing, operation and maintenance cost coefficients, and the parameter design and structure of the Long Short-Term Memory (LSTM) neural network.

[0202] Step 1: The modified IEEE 30 system includes 20 loads, 6 conventional generators, 1 wind turbine, 1 photovoltaic generator and 2 energy storage devices. The load demand and renewable energy output data are from measured data in Belgium

[41] , and the dataset contains 35,040 data points (covering 365 days). The data from the first 345 days is used as the training set and the data from the last 20 days is used as the test set. Based on this, the node loads and wind and solar power output in the test system are appropriately scaled according to the load demand and renewable energy output data. The hyperparameter settings of the proposed algorithm are detailed in Tables 1, 2 and 3. The number of iterations for linear optimal power flow calculation is set to 4, and the Gurobi solver is used for solving.

[0203] Table 1 shows the parameters of the agent:

[0204]

[0205] Step 2: Based on the training set data, expert data is generated using the linear optimal power flow calculation method. This expert data is then divided into a training set and a test set for imitation learning at a 4:1 ratio. Subsequently, supervised training is performed on networks without Long Short-Term Memory (LSTM) and networks with LSTM, respectively.

[0206] Step 3: To verify the effects of imitation learning, secure reinforcement learning, and Long Short-Term Memory (LSTM) networks, three sets of experiments were set up: Experiment (1): Training was performed directly using the secure reinforcement learning algorithm without imitation learning (without LSTM); Experiment (2): Training was performed using the secure reinforcement learning algorithm accelerated by imitation learning (without LSTM); Experiment (3): Training was performed using the secure reinforcement learning algorithm accelerated by imitation learning (using LSTM). The training process for the three sets of experiments is as follows: Figure 5 As shown in Table 2, the results of the three experimental groups on the test set are as follows:

[0207] Table 2 shows the performance of the three experimental groups on the test set:

[0208] experiment Average reward Average constraint violation (1) -12498 0.00150 (2) -12670 0.00061 (3) -12587 0.00002

[0209] according to Figure 5As a result, Experiment (1) performed worse than Experiments (2) and (3) at the beginning of training, especially in terms of average constraint violation, which was significantly higher than Experiments (2) and (3). This is because Experiment (1) did not use imitation learning. Imitation learning can provide initial policy guidance in reinforcement learning training, helping the agent to quickly acquire the ability to complete basic tasks in the early stages, thereby accelerating the learning process. In addition, Experiment (3) performed better than Experiment (2) at the beginning of training, further verifying that networks containing Long Short-Term Memory (LSTM) have better fitting effects than networks without LSTM.

[0210] According to Table 2, Experiment (3) outperforms Experiment (2). This further illustrates the advantages of networks incorporating Long Short-Term Memory (LSTM). Comparing Experiment (1) and Experiment (3), Experiment (1) has a larger average reward, but Experiment (3) has a smaller average constraint violation. The average constraint violation of all three groups of experiments in the test set is less than 0.01, indicating that the constraint policy optimization algorithm can effectively handle the constraint optimization problem in reinforcement learning, ensuring that the constraint is not violated while optimizing the policy.

[0211] Step 4: Load demand and renewable energy output data for 10 days were randomly selected from the test set, and the initial energy value of the energy storage devices was set to 0. After excluding random factors, the agents trained in experiments (1), (2), and (3) were used to make scheduling decisions for 10 episodes. At the same time, the linear optimal power flow calculation method was used for comparison, as experiment (4). The scheduling results are as follows: Figure 6 As shown.

[0212] according to Figure 6 Overall, Experiment (3) performed best, followed by Experiment (1), then Experiment (2), and Experiment (4) performed worst. Specifically, the constraint violation costs of Experiments (1), (2), and (3) were all less than 0.01, while Experiment (4) frequently violated constraints, especially when power flow exceeded limits. This is related to the approximate calculation method of the linear optimal power flow algorithm. This indicates that the safety reinforcement learning algorithm can effectively ensure the safe operation of the system. Furthermore, Experiment (4) had the smallest reward, indicating that the linear optimal power flow calculation method is not as effective as the safety reinforcement learning method in reducing system operating costs.

[0213] Step 5: To verify whether the opponent's interference can effectively reduce the agent's performance, experiment (5) was set up: In this experiment, the agent trained in experiment (2) was used, and an opponent was introduced. Only the opponent's policy network was updated, while the agent's policy network was not updated. The training results are as follows: Figure 7 As shown.

[0214] according to Figure 7As a result, as the adversary trains, the cost of constraint violation by the agent continuously increases, while the reward does not decrease significantly. This is because weighting coefficients are set in the normalized reward and cost, with cost having a larger proportion. Therefore, the adversary's main goal is to interfere with the agent's decision-making by increasing the agent's cost. Figure 7 The results show that adversaries trained using the PPO algorithm can effectively interfere with the agent's decision-making process.

[0215] Step 6: To verify the effectiveness of robust reinforcement learning for agent decision-making performance under uncertain conditions, the following four sets of experiments were set up: Experiment (6): The maximum deviation of the load demand forecast was set to 5%, and the maximum deviation of the renewable energy forecast was set to 10%. The robust and secure reinforcement learning algorithm accelerated by imitation learning was used for training (without using Long Short-Term Memory LSTM); Experiment (7): The maximum deviation of the load demand forecast was set to 5%, and the maximum deviation of the renewable energy forecast was set to 10%. The robust and secure reinforcement learning algorithm accelerated by imitation learning was used for training (using Long Short-Term Memory LSTM); Experiment (8): The maximum deviation of the load demand forecast was set to 10%, and the maximum deviation of the renewable energy forecast was set to 20%. The robust and secure reinforcement learning algorithm accelerated by imitation learning was used for training (without using Long Short-Term Memory LSTM); Experiment (9): The maximum deviation of the load demand forecast was set to 10%, and the maximum deviation of the renewable energy forecast was set to 20%. The robust and secure reinforcement learning algorithm accelerated by imitation learning was used for training (using Long Short-Term Memory LSTM).

[0216] Step 7: In the test set, test the decision robustness of the six agents trained in experiments (2), (3), and (6)-(9) under the following two conditions: (1) The actual value of load demand is evenly distributed within ±5% of the predicted value, and the actual value of renewable energy is evenly distributed within ±10% of the predicted value; (2) The actual value of load demand is evenly distributed within ±10% of the predicted value, and the actual value of renewable energy is evenly distributed within ±20% of the predicted value. The test results are shown in Table 3:

[0217] Table 3 shows the performance of the 6 experimental groups on the test set:

[0218]

[0219] According to Table 3, the constraint violation costs of experiments (2) and (3) are significantly greater than 0.01. However, without considering uncertainty, the constraint violation costs of experiments (2) and (3) are less than 0.01. Furthermore, the reward in scenario (2) is lower than that in scenario (1), while the cost in scenario (2) is higher than that in scenario (1). This indicates that the difference between the predicted value and the true value can effectively interfere with the agent's decision-making, and the agent's decision-making performance decreases as the accuracy of the predicted value decreases. Meanwhile, the cost of experiments (6) or (8) is significantly lower than that of experiment (2), and the cost of experiments (7) or (9) is significantly lower than that of experiment (3). This indicates that the Alternating Training with Learned Adversaries (ATLA) algorithm can significantly improve the robustness of the agent's decision-making. The reward of experiment (6) is higher than that of experiment (7), but its cost is also higher than that of experiment (7). This indicates that the network containing Long Short-Term Memory (LSTM) performs better in terms of robustness, but its optimization performance is slightly lower than that of the network without LSTM. Further comparison of experiments (6) and (8) or (7) and (9) revealed that the agent trained in scenario (2) had lower costs but also lower rewards compared to the agent trained in scenario (1). This indicates that agents trained in environments with greater uncertainty are more robust, but this also leads to a decrease in rewards.

[0220] The overall framework of this method is as follows: Figure 7 As shown in the figure. Compared with existing methods, this invention proposes a robust and secure reinforcement learning method that combines imitation learning to accelerate training, aiming to solve the problem of real-time optimization operation of distribution networks. This method ensures the stable and secure operation of the power system by employing a constrained policy optimization algorithm. This invention utilizes a linear optimal power flow calculation method to construct an expert knowledge base and uses imitation learning to pre-train the agent's policy network, thereby accelerating the training process and improving convergence speed. Furthermore, this invention introduces the ATLA algorithm to enhance the robustness of the agent's decision-making and combines it with a Long Short-Term Memory (LSTM) network to further improve the agent's performance. The proposed method demonstrates significant advantages in reducing power system operating costs, ensuring system security, and coping with load fluctuations and renewable energy uncertainties; these advantages have been fully verified in theoretical and numerical simulations.

[0221] For a better understanding of the embodiments of this application, please refer to [link / reference]. Figure 8 . Figure 8The overall framework diagram provided for one embodiment of this application illustrates the overall scheme of a robust and secure reinforcement learning framework that combines imitation learning and adversarial training. It includes four core components: a storage unit, a trajectory generation module, an adversary module, and an agent module. These components work collaboratively to achieve real-time dispatching of the power distribution network. The specific process is as follows:

[0222] (1) Storage unit:

[0223] Input: Receive a complete data packet from the trajectory generation module, including: observation state o, action a, reward r, constraint violation cost c, reward value, and network hidden state h. φ Constraint cost value network hidden state h ω Strategy network hidden state h θ True state s, opponent's actions Opponent's reward

[0224] Output: Pass s to the adversary module Pass o, a, r, c, h to the agent module φ h ω h θ .

[0225] Function: Serves as a data hub, caching interaction trajectories for subsequent training.

[0226] (2) Trajectory Generation Module:

[0227] Input: No external input; internal self-closed loop operation.

[0228] Output: Output complete trajectory data o, a, r, c, h to the storage unit. φ h ω h θ ,s,

[0229] Internal processes:

[0230] 1. Action Generation: The agent's policy network outputs action distribution parameters μ. t and Σ t Through Gaussian distribution N[μ t ,Σ t Sampling generation action a t .

[0231] 2. Power system interaction: a t The dispatch instructions are input into the power system environment. The environment solves the power flow problem using Newton's method and outputs the next state s. t+1 and current state s t .

[0232] 3. Countermeasures against disturbance injection: s t Input the adversary network, output the adversary's action parameters and pass Sample generation of opponent actions The disturbance module will With s t Combined, generate perturbation observations. t (For example: Where k bias,L =0.05].

[0233] 4. Closed-loop feedback: o t Input actor-critic adversarial network, output new motion parameters μ t , Σ t and hidden state The latter was updated to

[0234] (3) Opponent Module:

[0235] Input: s provided by the receiving storage unit

[0236] Internal processes:

[0237] PPO Algorithm: Optimizes and updates actor-adversarial network parameters θ using a proximal strategy. v The reward function is: Where k r ,k c r is the weighting coefficient. t and c t Normalization is required.

[0238] Network update: Based on trajectory data D v ={τ v Calculate the policy gradient and optimize the opponent's policy.

[0239] (4) Agent module:

[0240] Input: o, a, r, c, h provided by the receiving storage unit φ h ω h θ .

[0241] Internal processes:

[0242] 1. Imitation learning pre-training: Expert database (state-action pairs generated by linear optimal power flow) [s] IL ,a IL Pre-trained initial actor network. Loss function:

[0243] 2. Critics' Network Update: Calculating the Target Value and Update reward value network parameter L φ and constrained cost value network parameters L ω Output gradient and

[0244] 3. Actor Network Update (Constraint Policy Optimization):

[0245] Importance sampling: Estimating expectation:

[0246] Approximate solution: Estimate gradients g and b, Hessian matrix H, constraint value z = J C [π k -d. If feasible, solve the dual problem: Where r = g T H -1 b, β = b T H -1 b. If this is not feasible, proceed with a recovery update:

[0247] Backtracking search: Determining the final parameter θ k+1 It satisfies the KL divergence constraint.

[0248] Global workflow:

[0249] 1. Initialization: The agent's LSTM policy network θ, reward value network φ, constraint cost value network ω, and opponent's policy network θ v .

[0250] 2. Expert pre-training: Use linear optimal power flow to generate expert data to pre-train the agent policy network.

[0251] 3. Alternating resistance training cycle:

[0252] Phase 1 (Fixed Opponent): The agent makes decisions in the environment, collects the trajectory D, and optimizes and updates θ, φ, ω through constraint policies.

[0253] Phase 2 (Fixed Agent): The opponent generates a perturbation trajectory D v θ is updated using the PPO algorithm. v .

[0254] 4. Termination Condition: Repeat alternating training until the policy converges (meeting the preset stopping condition). This framework accelerates initial convergence through imitation learning, enhances robustness through adversarial training, and uses LSTM networks to handle temporal dependencies, ultimately achieving safe, efficient, and disturbance-resistant real-time scheduling of the distribution network.

[0255] It is understood that the solutions based on the embodiments of this application can improve the security, robustness, and economy of power distribution network dispatching.

[0256] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0257] In one embodiment, a computer device is provided, the internal structure of which can be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and the database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the above-described power distribution network dispatching method. It includes: a memory and a processor; the memory stores the computer program; and the processor executes the computer program to implement any step of the above-described power distribution network dispatching method.

[0258] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, can perform any of the steps in the above power distribution network dispatching method.

[0259] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0260] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0261] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0262] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0263] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0264] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A distribution network dispatching method, characterized in that, include: Obtain real-time observation data of the power distribution network; The preset target agent is invoked to process the real-time observation data to obtain the dispatch action instructions for the distribution network. The long short-term memory (LSTM) neural network in the target agent is obtained by imitating and pre-training multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network, and by adversarial alternating training based on the constrained Markov decision process corresponding to the distribution network, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm. The reference data pairs include the reference state of the distribution network and the reference dispatch action instructions corresponding to the reference state. The power distribution network is dispatched according to the dispatching action command; Before acquiring real-time observation data of the distribution network, the following steps are included: Construct an initial intelligent agent and an adversary intelligent agent, wherein the initial intelligent agent includes multiple corresponding LSTM neural networks, and the adversary intelligent agent includes multiple corresponding neural networks; Based on the linear optimized power flow algorithm corresponding to the distribution network, multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network are obtained. Based on multiple reference data pairs, the initial agent is subjected to imitation pre-training to obtain a pre-trained agent; Based on the constrained Markov decision process, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm, the pre-trained agent and the adversary agent are called to perform adversarial alternating training to obtain the target agent, wherein the target agent is formed by updating the parameters of the pre-trained agent through the neural network. The step of performing imitation pre-training on the initial agent based on multiple reference data pairs to obtain a pre-trained agent includes: During the imitation pre-training process of the initial agent, the multiple reference data pairs are divided into multiple batches of reference data pairs; For any one of the reference data pairs in the current batch among the multiple batches of reference data pairs, the reference state in the reference data pair in the current batch is input into the initial agent to obtain the prediction scheduling action instruction corresponding to the reference state in the reference data pair in the current batch output by the initial agent. The mean squared error loss of the current batch is obtained by calculating the loss between the predicted scheduling action command corresponding to the reference state of the reference data of the current batch and the reference scheduling action command of the reference data of the current batch. Based on the mean squared error loss of the current batch, the neural network parameters of the initial agent are updated using the gradient descent algorithm; If the preset first stopping condition is not met, then the initial agent is pre-trained by imitation based on the reference data pair of the next batch of the current batch. If the preset first stopping condition is met, the imitation pre-training process of the initial agent ends, and the initial agent is identified as a pre-trained agent.

2. The method according to claim 1, characterized in that, The construction of the initial intelligent agent and the adversary intelligent agent includes: Construct an LSTM policy network, an LSTM reward value network, and an LSTM constrained cost value network; Initialize the parameters of the LSTM policy network, the LSTM reward value network, and the LSTM constrained cost value network to obtain an initial agent composed of the LSTM policy network, the LSTM reward value network, and the LSTM constrained cost value network; Construct policy networks and reward value networks; Initialize the parameters of the policy network and the reward value network to obtain an adversary agent composed of the policy network and the reward value network.

3. The method according to claim 1, characterized in that, The linear power flow optimization algorithm based on the distribution network obtains multiple reference data pairs under the constrained Markov decision process corresponding to the distribution network, including: Based on the linear optimized power flow algorithm corresponding to the distribution network, the historical operation data of the distribution network is iteratively solved to generate multiple reference states of the distribution network and reference scheduling action instructions corresponding to each of the multiple reference states; Based on each of the reference states and the corresponding reference scheduling action instructions, multiple reference data pairs are determined.

4. The method according to claim 1, characterized in that, The step of obtaining the target agent by invoking the pre-trained agent and the adversary agent for adversarial alternating training based on the constrained Markov decision process, the constrained policy optimization algorithm corresponding to the constrained Markov decision process, and the adversarial alternating training algorithm includes: The pre-trained agent and the adversary agent are called to perform adversarial training for multiple iteration cycles; For any target iteration cycle among the multiple iteration cycles, the neural network parameters of the adversary agent are fixed; In the power distribution network environment defined by the constrained Markov decision process, the pre-trained agent executes decisions and collects the raw trajectory data of the pre-trained agent. Based on the original trajectory data and the constraint policy optimization algorithm corresponding to the constrained Markov decision process, the neural network parameters of the pre-trained agent are updated. The neural network parameters of the pre-trained agent are fixed after the update; The adversary agent applies adversarial perturbations to the pre-trained agent and collects the perturbation trajectory data of the adversary agent. The neural network parameters of the adversary agent are updated based on the perturbation trajectory data and the constraint policy optimization algorithm corresponding to the constraint Markov decision process. If the preset second stopping condition is not met, the next iteration cycle of the target iteration cycle will be entered, so that the pre-trained agent and the adversary agent can be called to conduct adversarial alternating training in the next iteration cycle. If the preset second stopping condition is met, the adversarial alternation training process between the pre-trained agent and the opponent agent ends, and the pre-trained agent is identified as the target agent.

5. The method according to claim 1, characterized in that, The formula corresponding to the constrained Markov decision process is: ； in, Represents a set of states. Represents a set of actions. Represents the reward function, Represents the state transition function. This indicates that the constraint violates the cost function.

6. The method according to claim 5, characterized in that, The formula corresponding to the state set S is expressed as follows: ； in, Indicates at time state, Defined as: ； in, Represents a node At any moment Active load; Represents a node At any moment reactive load; Indicates wind turbine generator set At any moment The predicted maximum active power output; Indicates photovoltaic generator set At any moment The predicted maximum active power output; Indicates conventional generator set At any moment Those who have made meritorious contributions; Indicates energy storage devices At any moment Energy; Represents timing information, including time. And its trigonometric function values.

7. The method according to claim 5, characterized in that, The formula corresponding to the action set A is expressed as follows: ； in, Indicates at time The action, Defined as: ； in, Indicates conventional generator set At any moment Active power dispatch instructions; Indicates conventional generator set At any moment reactive power dispatch instructions; Indicates wind turbine generator set At any moment Active power dispatch instructions; Indicates wind turbine generator set At any moment reactive power dispatch instructions; Indicates photovoltaic generator set At any moment Active power dispatch instructions; Indicates photovoltaic generator set At any moment reactive power dispatch instructions; Indicates energy storage devices At any moment Active power dispatch instructions; Indicates energy storage devices At any moment Reactive power dispatching instructions.

8. The method according to claim 5, characterized in that, The formula corresponding to the reward function R is expressed as follows: ； in, Indicates at time The state; Indicates at time The action; Indicates at time The reward; This indicates the cost of generating electricity from a conventional generating unit; This indicates the penalty cost for abandoning wind and solar power. This represents the cost of purchasing electricity from the upstream power grid at the common coupling point; This indicates the energy consumption cost of energy storage equipment.

9. The method according to claim 5, characterized in that, The formula corresponding to the state transition function P is expressed as follows: ； in, Execute action After state from Transferred to The probability of; Indicates at time The state; Indicates at time The state; Indicates at time The action.

10. The method according to claim 5, characterized in that, The formula corresponding to the constraint violation cost function C is expressed as follows: ； in, Indicates at time The cost of violating constraints; Indicates at time The state; Indicates at time The action; This indicates the cost of violating voltage constraints; This indicates that the cost of violating line capacity constraints is high. This indicates the cost of violating the ramp-up constraint for conventional units; This represents the cost of violating the constraint on the power transmission capacity between the upper-level power grid and the common coupling point.

11. A computer device, comprising: The method includes a memory and a processor, the memory storing a computer program, characterized in that the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 10.

Images