Two-stage cloud-edge collaborative scheduling method and device for active distribution network based on deep reinforcement learning

Abstract

The application provides a kind of active distribution network two-stage cloud edge collaborative scheduling method and device based on deep reinforcement learning, belongs to distribution network dispatching field;The method comprises the following steps: ADN is divided into multiple regions, each region is an agent, and an edge server is equipped for each agent;Build upper ADN day-ahead scheduling model and lower ADN day-ahead scheduling model;Build upper ADN DDPG model and lower ADN MADDPG model;Cloud server uses historical power data to train day-ahead scheduling model by improved DDPG algorithm in upper ADN DDPG model;In the day-ahead training stage, cloud server trains multi-agent day-ahead scheduling model for different time periods based on day-ahead scheduling results using hierarchical MADDPG model training method in lower ADN MADDPG model;The application can effectively solve the problem of mixed action and model constraint.

Description

Technical Field

[0001] This application relates to the field of distribution network dispatching technology, and in particular to a two-stage cloud-edge collaborative dispatching method and device for active distribution networks based on deep reinforcement learning. Background Technology

[0002] In recent years, with the large-scale integration of distributed power sources, active distribution networks (ADNs) have faced challenges such as declining operational economics, increased voltage fluctuations, and deteriorating power quality, placing higher demands on their economic operation and rapid response capabilities. Data-driven deep reinforcement learning methods can not only adapt to dynamic changes such as load fluctuations and the randomness of new energy sources, but also leverage their real-time learning and inference capabilities to quickly adjust control strategies, improving system efficiency and stability. Combining edge-cloud collaboration with deep reinforcement learning for the operation and scheduling of active distribution networks can fully leverage the advantages of both, thereby effectively improving the overall operational performance and reliability of ADNs.

[0003] In terms of economic dispatch, the paper "Active Distribution Network Hierarchical Optimization Economic Dispatch Based on ADMM Algorithm" proposes an active distribution network hierarchical optimization economic dispatch method based on the alternating direction multiplier method to address the economic issues of the overall operation of the distribution network while ensuring the safe operation of the system. The paper "Active Distribution Network Multi-Time Scale Optimization Dispatch Considering the Three-Dimensional Characteristics of Voltage Step-Down Energy Saving and Photovoltaic Errors" proposes a multi-time scale optimization dispatch method for ADN considering voltage step-down energy saving regulation and distributed photovoltaic prediction errors to solve the safety and economic operation problems of ADN caused by the stochasticity of high-proportion distributed photovoltaics. With the access of power electronic equipment, the diversified regulation resources of ADN provide more means for its refined and flexible control, but also increase the complexity of control, such as differences in control methods and inconsistent dynamic responses, as well as discrete control problems such as on-load tap-changing transformers (OLTC), capacitor bank access, and topology reconfiguration. However, current ADN dispatch based on deep reinforcement learning mainly focuses on continuous variables, and there is limited research on dispatching of mixed variables in ADN. Existing mixed variable dispatching methods based on deep reinforcement learning (DRL) are still very immature.

[0004] Regarding ADN node voltage over-limit control, the paper "Two-stage multi-mode voltage control for distribution networks: a deep reinforcement learning approach based on multiple intelligences" aims to mitigate ADN voltage fluctuations and achieve coordinated operation among different devices, proposing a two-stage multi-mode voltage control strategy for distribution networks based on deep reinforcement learning. The paper "Active Active-Reactive Coordinated Optimization Scheduling for Distribution Networks Based on Deep Deterministic Policy Gradients" proposes an active-reactive coordinated optimization scheduling strategy for ADN based on deep deterministic policy gradients to improve the reliability and economy of distribution network power supply. While centralized control of ADN can effectively achieve global optimization, it heavily relies on reliable communication and is susceptible to single-point failures and high communication burdens. Therefore, distributed and decentralized control strategies have received widespread attention. With decreasing control timescales, such as in intraday control phases, the centralized control mode of cloud servers struggles to meet the demands of short control cycles and high communication bandwidth. Furthermore, with decreasing control timescales, the DRL method has limitations in multi-step feature training, making it difficult to adapt to changes in the ADN environment, thus hindering the convergence of the control model training. Summary of the Invention

[0005] To address the aforementioned technical issues, this application proposes a two-stage cloud-edge collaborative scheduling method and device for active distribution networks based on deep reinforcement learning. The method utilizes a two-stage deep reinforcement learning approach to achieve day-ahead and intraday scheduling of the active distribution network, thereby optimizing its operation and scheduling strategy.

[0006] The technical solution adopted in this application is: a two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning, comprising the following steps:

[0007] S1: Divide the ADN into multiple regions, each region is an agent, and equip each agent with an edge server. All edge servers communicate bidirectionally with the cloud server.

[0008] S2: Construct a two-stage scheduling model for ADN, which includes the day-ahead scheduling model of the upper-layer ADN and the intraday scheduling model of the lower-layer ADN.

[0009] S3: Construct a two-stage deep reinforcement learning model for the ADN. The two-stage deep reinforcement learning model includes the DDPG model of the upper-layer ADN used to train the day-ahead economic scheduling model of the upper-layer ADN and the MADDPG model of the lower-layer ADN used to train the intraday scheduling model of the lower-layer ADN.

[0010] S4: Model Training: First, the cloud server uses historical power data to train the day-ahead scheduling model using the improved DDPG algorithm in the upper-layer ADN's DDPG model. Then, in the intraday training phase, the cloud server uses the hierarchical MADDPG model training method in the lower-layer ADN's MADDPG model based on the day-ahead scheduling results to train the multi-agent intraday scheduling model for different time periods.

[0011] Furthermore, the goal of the upper-level ADN day-ahead scheduling model is to minimize the daily operating cost of the ADN while ensuring safe operation. The decision variables include controllable devices within the ADN, and the daily operating cost of the ADN includes the main grid cost, energy storage operating cost, branch network loss cost, and penalty cost.

[0012] Controllable devices within an ADN include energy storage systems, capacitor banks, and on-load tap changers;

[0013] The main grid cost includes the cost of purchasing electricity from the main grid, carbon emission costs, and the cost of power interaction penalties with the ADN;

[0014] The operating cost of energy storage is obtained through a levelized cost of electricity (LCOE) model.

[0015] The penalty costs include the penalty costs for ADN interaction with the main grid and the penalty costs for node voltage fluctuations.

[0016] Furthermore, in each control cycle of the day-ahead dispatch, the constraints that the safe and economical operation of the ADN must meet include: power flow constraints of the entire distribution network, transmission power constraints of the PCC nodes, power and state of charge constraints of the energy storage system, state of charge constraints of the energy storage system at the end of each day, on-load tap changer constraints, capacitor bank constraints, and voltage constraints of each node of the ADN.

[0017] Furthermore, the goal of the lower-level ADN intraday scheduling model is to cope with intraday random power fluctuations. By adjusting the fast-adjustable reactive power devices, it ensures that the voltage of each node in the distribution network meets the requirements for safe operation, with the objective function being to reduce the difference between the voltage of all nodes in the ADN and the expected voltage.

[0018] Furthermore, during each control cycle of intraday operation, the intraday scheduling of ADN needs to meet the following constraints: power flow constraints of the entire distribution network, transmission power constraints of PCC nodes, voltage constraints of each node of ADN, and reactive power output constraints of energy storage systems and new energy sources.

[0019] Furthermore, the two-stage scheduling model of ADN is modeled as a Markov decision process. The day-ahead scheduling model is trained by the DDPG model of the upper layer ADN, and the intraday scheduling model is trained by the MADDPG model of the lower layer ADN.

[0020] The DDPG model of the upper-layer ADN includes the system's state, actions, reward design, and improved DDPG algorithm. The system's state space includes the charge state set of the energy storage in the ADN at time t, the active and reactive power demand in the ADN at time t, the active and reactive power demand of the main grid, and the sum of voltage fluctuations at each node. The system's action space includes the day-ahead scheduling control variables of the ADN, which is a mixed discrete and continuous action space. The system's reward function is the operating cost of the ADN at time t.

[0021] The MADDPG model of the lower-layer ADN includes the system's state, actions, reward design, and improved MADDPG training model. The system's state space is the state space of each agent, including the active and reactive power demand of each agent at time t and the difference between all nodes of each agent and the expected voltage. The system's action space includes the reactive power output of the energy storage converter and the new energy converter in each agent. The system's reward is the difference between the nodes of each agent and the expected voltage.

[0022] Furthermore, the improved DDPG algorithm includes improvements to the handling of mixed actions and model constraints;

[0023] The process for handling mixed actions is as follows: First, a discrete action network containing the voltage regulation amplitude of the adjustable voltage regulator and the number of capacitor banks is added to the DDPG algorithm. The discrete action network is based on the action network and the output is discretized through the softmax function. Next, discrete actions are generated through the discrete action network, and continuous actions containing the active power output of energy storage are generated through the original continuous action network. Finally, the mixed actions are used as the input of the environment to obtain an experience.

[0024] Then, in the network update, the critic network is still updated by minimizing the current... Values ​​and Targets The error is realized by maximizing the current value in both discrete and continuous action networks.

[0025] Model constraints include action space constraints and state space constraints. Action space constraints are obtained by linearly mapping the neural network output to the corresponding action range, and the energy storage charge state constraints are obtained through corresponding transformations. State space constraints are achieved by adding a penalty term to the reward function.

[0026] Furthermore, the improved MADDPG training model is a hierarchical MADDPG model training method that trains the control model at different time scales.

[0027] The basic model for training a day is divided into 24 time periods over a long time scale, which constitutes one round. A general model is trained based on the predicted power of new energy sources and loads, so that it can basically adapt to the active and reactive power changes of ADN at different times of the day.

[0028] At a short timescale, multiple network models are trained based on power changes in different time periods; the output of the general model is combined with historical power data for that time period to further train the model.

[0029] Furthermore, it also includes: S5: During the model execution phase, the cloud server distributes the trained intraday scheduling model to each edge server. In each intraday control cycle, the edge server controls the corresponding equipment actions based on the real-time local observation information of each agent. During the execution process, each agent makes independent decisions.

[0030] A computer device includes a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method.

[0031] The advantages of this application over the prior art are as follows:

[0032] (1) During the day-ahead scheduling phase, an ADN scheduling model considering the interaction cost of the main power grid and voltage fluctuations was constructed, and an improved deep deterministic policy gradient algorithm was proposed to solve the problem of mixed actions and model constraints.

[0033] (2) During the intraday scheduling phase, an objective function for the intraday scheduling ADN was established, and a multi-agent distributed training model was constructed. Furthermore, a hierarchical training framework was designed to improve model convergence by training the model in different time periods at different time scales.

[0034] (3) The two-stage cloud-edge collaborative scheduling strategy of the active distribution network realizes the two-stage scheduling of ADN through cloud training and edge execution collaboration. It not only meets the real-time requirements of the intraday control cycle, but also effectively reduces the communication burden of ADN. Attached Figure Description

[0035] The following description, in conjunction with the accompanying drawings, further illustrates this application:

[0036] Figure 1 This is a power distribution network structure diagram provided for an embodiment of this application;

[0037] Figure 2 A two-stage edge-cloud collaborative scheduling architecture diagram provided for embodiments of this application;

[0038] Figure 3 A flowchart illustrating the two-stage model training process provided in this application embodiment;

[0039] Figure 4 The diagram showing the day-to-day running results provided for the embodiments of this application;

[0040] Figure 5 The reactive power output diagrams for each converter using the method described in this application are shown below.

[0041] Figure 6 A comparison chart of the voltage at each node on a given day under different methods provided in the embodiments of this application. Detailed Implementation

[0042] like Figures 1 to 6 As shown, this application provides a two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning, which is a two-stage cloud-edge collaborative scheduling strategy for active distribution networks based on deep reinforcement learning. It schedules the controllable resources of the Active Distribution Network (ADN) at two time scales: day-ahead and intraday, to achieve economical scheduling and voltage safety operation.

[0043] In this embodiment, the ADN is divided into multiple regions, each region is a smart agent, and each smart agent is equipped with an edge server. The cloud server performs two-stage model training, and the edge server uses the trained model to achieve real-time information collection and control.

[0044] During the day-ahead scheduling phase, the scheduling resources include the active power output of energy storage, the number of capacitor banks switched on and off, and the voltage regulation amplitude of the on-load tap changer (OLTC). Considering that the day-ahead scheduling resources are a set of mixed integer variables, this application proposes an improved deep deterministic policy gradient (DDPG) algorithm to handle the scheduling constraints of mixed actions and ADN (Active Data Network) to meet the day-ahead scheduling requirements.

[0045] During the intraday scheduling phase, based on the day-ahead scheduling results, the intraday scheduling resources, including reactive power output from energy storage and new energy sources, are a set of continuous variables. The Multi-Agent DDPG algorithm is employed to meet the intraday control cycle and communication bandwidth requirements. However, the DDPG training model is difficult to converge over the entire scheduling cycle and struggles to adapt to the characteristics of different time periods within a day. Therefore, this application also proposes a hierarchical training method. The model is trained at different time scales. A basic intraday network model is trained on a long-term scale. Based on this, intraday scheduling models for different time periods are obtained through time-segmented training.

[0046] During the model execution phase, the multi-agent model trained in the cloud is distributed to the edge servers of each agent. Each edge server executes control of its sub-region based on real-time local observation information. The various regions work together to achieve the goals of day-ahead economic scheduling and intraday safe operation.

[0047] Taking the improved IEEE 33-node as an example, the distribution network structure studied in this application is as follows: Figure 1 As shown, the two-stage edge-cloud collaborative scheduling architecture is as follows: Figure 2 As shown, based on this, the specific implementation steps of the scheduling method in this application include:

[0048] S1: The ADN is divided into multiple regions, each region being an agent, and each agent is equipped with an edge server. All edge servers communicate bidirectionally with the cloud server. The cloud server (CS) performs two-stage model training, and the edge server (ES) uses the trained model to achieve real-time information acquisition and control.

[0049] S2: Construct a two-stage scheduling model for ADN, which includes the day-ahead economic scheduling model of the upper-layer ADN and the intraday scheduling model of the lower-layer ADN.

[0050] S3: Construct a two-stage deep reinforcement learning model for the ADN. The two-stage deep reinforcement learning model includes the DDPG model of the upper-layer ADN used to train the day-ahead economic scheduling model of the upper-layer ADN and the MADDPG model of the lower-layer ADN used to train the intraday scheduling model of the lower-layer ADN.

[0051] S4: The two-stage scheduling model is trained using a two-stage deep reinforcement learning model. In the model training phase, firstly, the cloud server uses historical power data to train the day-ahead scheduling model using an improved DDPG algorithm from the upper-layer ADN's DDPG model. Subsequently, in the intraday training phase, based on the day-ahead scheduling results, the cloud server uses the hierarchical MADDPG model training method from the lower-layer ADN's MADDPG model to train the multi-agent intraday scheduling model for different time periods.

[0052] S5: During the model execution phase, the cloud server distributes the trained intraday scheduling model to each edge server. In each intraday control cycle, the edge servers control the actions of corresponding devices based on the real-time local observation information of each agent. During execution, each agent makes independent decisions without exchanging information with others. This not only meets the real-time requirements of the intraday control cycle but also effectively reduces the communication burden on the ADN.

[0053] The two-stage scheduling model and the two-stage deep reinforcement learning model will be explained in detail below.

[0054] 1) Upper-layer ADN day-ahead economic scheduling model

[0055] The day-ahead scheduling objective of an ADN is to minimize its daily operating cost while ensuring safe operation. Decision variables include controllable devices within the ADN, such as active power from energy storage, reactive power from capacitor banks, and on-load tap changers (OLTCs). (Daily operating cost of the ADN) Including main grid costs Energy storage operating costs Branch network loss cost And the penalty cost, which is expressed as follows:

[0056] (1);

[0057] in, The purpose of penalizing the power grid is to improve the utilization rate of renewable energy sources such as ADN. The purpose of penalizing node voltage fluctuations is to reduce node voltage fluctuations in the ADN to meet safe operation requirements. This refers to the scheduling period of a day. The first one represents the distribution network Each node.

[0058] The costs are as follows:

[0059] (1) Main power grid cost . This includes the cost of purchasing electricity from the main grid and the cost of carbon emissions:

[0060] (2);

[0061] in, The unit electricity price on the main grid (¥ / kWh), The active power of the main grid (the power injected into the ADN from the main grid is positive). The carbon emission cost of the main power grid To control the cycle. If The carbon emissions per unit time are positive. Exceeding the maximum allowed value The carbon emission cost of ADN is:

[0062] (3);

[0063] in, Cost per unit of carbon emissions (¥ / kg) The calculation formula is as follows:

[0064] (4);

[0065] in, The CO2 emission factor (kg / kWh) of the main power grid.

[0066] (2) Energy storage operating costs . The levelized cost of energy (LCOE) model yields the following:

[0067] (5);

[0068] in, The unit energy conversion cost of energy storage (¥ / kWh), The active power output for energy storage.

[0069] (3) Branch network loss cost The network loss cost for all ADN branches is:

[0070] (6);

[0071] in, For the current in each branch, For each branch resistance.

[0072] (4) In order to promote the distributed renewable energy consumption of ADN, a penalty cost is set for interaction between ADN and the main grid. :

[0073] (7);

[0074] in, The penalty coefficient per unit of energy interaction (¥ / kWh).

[0075] (5) To reduce voltage fluctuations at various nodes of the distribution network, a node voltage fluctuation penalty cost is set:

[0076] (8);

[0077] in, This is the voltage fluctuation penalty coefficient. This is the sum of voltage fluctuations at each node. This represents the expected voltage (per-unit value) for each node, typically set to 1. Let be the voltage at node k.

[0078] In each control cycle of the daytime scheduling The safe and economical operation of an ADN should meet certain constraints, including:

[0079] (1) Power flow constraints of the entire distribution network, i.e.

[0080] (9);

[0081] In the formula, Let t be the reactive power of the active distribution network; , These are the active power from photovoltaic (PV) and wind power (WT), respectively. , For the active and reactive power of each node; Let k be the voltage at node k. Let be the phase difference between node k and node m; and Let K be the branch conductance and susceptance from node k to node m; This represents the voltage regulation range of the on-load transformer. In this embodiment, the output power of each power source is specified as positive. q represents the q-th energy storage system, s represents the s-th photovoltaic system, r represents the r-th wind turbine, and w represents the w-th capacitor. u represents the u-th load, and k represents the k-th node. Let be the voltage at the k-th node at time t, and m be the voltage at the m-th node. Let Ω be the voltage of the m-th node at time t, and Ω be the set of nodes.

[0082] (2) Transmission power constraints of PCC nodes:

[0083] (10);

[0084] in, These represent the upper and lower limits of active and reactive power that are allowed to be transmitted by the main network nodes.

[0085] PCC (Point of Common Coupling) refers to the common connection point.

[0086] (3) Power and state of charge (SOC) constraints of energy storage system.

[0087] (11);

[0088] (12);

[0089] in, , , These represent the power and upper and lower limits of the SOC of the energy storage system, respectively.

[0090] (4) The SOC of stored energy at the end of each day must be within a reasonable range, that is:

[0091] (13);

[0092] Where, β0, β 24 These are the constraint coefficients, with values ​​ranging from [0.9 to 1.1]. and The SOCs are the start and end times, respectively.

[0093] (5) OLTC Constraints. In each control cycle, the voltage regulation amplitude of the OLTC should meet the following requirements:

[0094] (14);

[0095] in, , These are the minimum and maximum adjustable ranges of the OLTC, respectively.

[0096] (6) Capacitor Bank Constraints. The number of capacitor banks in a node. It should meet the following requirements:

[0097] (15);

[0098] in, This represents the maximum number of capacitor banks w.

[0099] (7) Voltage constraints at each node of ADN:

[0100] (16);

[0101] in, These are the upper and lower limits of the voltage amplitude required for safe operation of node k.

[0102] 2) Intraday scheduling model of lower-layer ADN

[0103] Intraday dispatching of the ADN (Active Distribution Network) aims to address intraday random power fluctuations by adjusting rapidly adjustable reactive power devices such as energy storage converters and renewable energy converters to ensure that the voltage at all nodes of the distribution network meets safe operation requirements. Intraday adjustments reduce the voltage difference between all nodes of the ADN and the expected voltage. As the objective function, that is:

[0104] (17).

[0105] During each control cycle of intraday operation, the intraday scheduling of ADN should meet certain constraints, specifically including:

[0106] (1) Reactive power output constraints for energy storage and new energy sources. Both must satisfy the converter's capacity constraints, i.e.:

[0107] (18);

[0108] in, For PV, WT or energy storage, These represent the upper and lower limits of reactive power output for new energy sources and energy storage, respectively. This refers to the rated capacity of the converter.

[0109] (2) Other constraints are consistent with those of the upper-level operation, such as equations (9), (10), and (16).

[0110] The established two-stage scheduling model of ADN can be modeled as a Markov decision process (MDP) and can be used to realize the cloud-edge collaborative operation control of the distribution network through centralized deep deterministic policy gradient (DDPG) and distributed multi-agent DDPG (MADDPG) algorithms, respectively.

[0111] 1) The DDPG model of the upper-layer ADN includes the system's state, actions, reward design, and improved DDPG algorithm. The system's state, actions, and reward design are as follows:

[0112] (1) The state space of the system The design is as follows:

[0113] (19);

[0114] Wherein, SOC is the set of charged states of energy stored in the ADN at time t; , These represent the active and reactive power demands within the ADN at time t (i.e., the sum of load and renewable energy power).

[0115] (2) Action space ADN's day-ahead scheduling control variables include active power from energy storage. Number of capacitor banks connected to the grid and OLTC regulation It is a mixed discrete and continuous action space:

[0116] (20);

[0117] in, , These are the sets of active power stored in the ADN and the switching quantities of the capacitor bank, respectively.

[0118] (3) Rewards. Reward function The operating cost of the ADN at time t:

[0119] (twenty one).

[0120] Improved DDPG algorithm:

[0121] The DDPG algorithm primarily addresses optimization problems in high-dimensional continuous action spaces; however, the action space in this embodiment is a hybrid action space. To address this characteristic, this application improves the DDPG algorithm. The improved DDPG algorithm mainly includes two aspects: hybrid action processing and the addition of model constraints.

[0122] (1) Mixed motion processing

[0123] First, a discrete action network is added to the DDPG algorithm. This network is based on an actor network and discretizes the output using a softmax function. Furthermore, discrete actions are generated using this network. And generate continuous actions through the original continuous action network (actor). Ultimately, by using the mixed actions as input to the environment, an experience is obtained. ,in This refers to the current state in an experience. For the current action space, For the current reward, To obtain a new state after an action is input from the environment.

[0124] Then, in the network update, the critic network is still updated by minimizing the error between the current action value function and the target value function, that is:

[0125] (twenty two);

[0126] in, For discrete action networks, For continuous action networks, For the objective value function, The current action value function;

[0127] Both discrete action networks and continuous action networks are updated by maximizing the current action. Values ​​are used to achieve this, that is:

[0128] (twenty three).

[0129] (2) Handling of model constraints

[0130] The constraints in the DDPG training model are divided into two categories: action space constraints and state space constraints. Action space constraints can be linearly mapped to the corresponding action range by the neural network output, as shown in equations (11), (14), and (15). The SOC constraint for energy storage (equation (12)) can be transformed as follows:

[0131] (twenty four);

[0132] in, This refers to the energy storage capacity.

[0133] State space constraints, such as equations (10), (13), and (16), can be achieved by adding a penalty term to the reward function, i.e., equation (21) can be modified as follows:

[0134] (25);

[0135] in, To constrain the penalty for exceeding the limit:

[0136] (26);

[0137] in, , , These are the over-limit penalty coefficients for PCC, SOC, and node voltage, respectively.

[0138] 2) The MADDPG model of the lower layer ADN includes the system's actions, states, rewards, and an improved MADDPG training model.

[0139] Intraday scheduling reduces node voltage fluctuations by adjusting reactive power output devices. Since intraday scheduling has a short control cycle, using cloud server control could lead to excessive resource consumption and even fail to meet control requirements. Therefore, this application employs the MADDPG algorithm based on distributed control. The algorithm is centrally trained on a cloud server and distributed control is implemented on edge servers to achieve intraday scheduling.

[0140] System state, action, and reward design

[0141] (1) Each intelligent agent The state space design is as follows:

[0142] (27);

[0143] (2) Action Space. The action space of each agent includes the controllable devices within its area:

[0144] (28);

[0145] in, , These are the reactive power outputs of the energy storage converters and new energy converters within each intelligent device.

[0146] (3) Rewards. The reward function for each agent is the difference between the node of each agent and the expected voltage (i.e., the intraday operating objective function (17)):

[0147] (29).

[0148] Improved MADDPG training model

[0149] In intraday scheduling, due to the short control cycle (5 minutes in this embodiment), each episode requires 288 steps if traditional training methods are used. Too many steps can lead to difficulties in model training convergence and increase the difficulty of adapting the model to changes in ADN power at different time periods. Therefore, this application proposes a hierarchical MADDPG model training method to train the control model at different time scales.

[0150] (1) Training a basic model for a day on a long time scale. Divide a day into 24 time periods, which constitute one round. Train a general model based on the day-ahead renewable energy and load forecast power so that it can basically adapt to the active and reactive power changes of ADN at different time periods of the day.

[0151] (2) At a short time scale, multiple network models are trained based on power changes in different time periods. The output of the general model is combined with historical power data for that time period to further train the model, thereby enhancing its adaptability to the characteristics of each time period.

[0152] Hierarchical training methods can reduce the number of training steps per round for the network while improving its adaptability at different times. During training, the final action of the input environment... Configure it as follows:

[0153] (30);

[0154] in, and These represent the network's output actions at long and short time scales, respectively. These are the weighting coefficients.

[0155] The training process of the DRL-based two-stage active distribution network scheduling model proposed in this embodiment is as follows: Figure 3 As shown.

[0156] To verify the effectiveness of the proposed two-stage cloud-edge collaborative scheduling strategy for active distribution networks based on DRL, a case study analysis was conducted using an improved IEEE 33-node distribution network system. The proposed method was implemented using the PyTorch framework for model training and execution, while some comparative analyses were performed in the MATLAB environment.

[0157] Improved IEEE 33-node system, such as Figure 1 As shown in Table 1. Model training is performed on a cloud server, dividing the distribution network into three agents, each controlled by a different edge server. System parameters adopt the IEEE 33 standard node parameters, and the renewable energy power comes from a dataset from a wind, solar, and energy storage demonstration base. Parameter settings are shown in Table 1. Power flow calculation uses the PyPower package. The hyperparameters of DRL include the learning rate. , All default values ​​are used.

[0158] Table 1 Simulation Parameter Settings

[0159]

[0160] The previous day's scheduling results from the upper level are as follows: Figure 4 As shown in Table 2.

[0161] Table 2. Dispatch Results (Previous Day)

[0162]

[0163] In Table 2, * indicates the average for one day. .

[0164] As shown by the day-ahead dispatch data, all proposed methods can achieve day-ahead economic dispatch, and voltage fluctuations meet operational requirements. Grid power remains roughly around 0 between 0:00 and 18:00, indicating that the control model can effectively coordinate the source-load balance of the distribution network. After 18:00, grid power rises sharply because the power of new energy sources is limited and cannot meet the energy demand of the ADN throughout the day, while the energy storage system needs to maintain the SOC requirement at the end of the day. Therefore, after 18:00, the ADN mainly relies on the main grid.

[0165] Based on the results from the upper layer, a hierarchical MADDPG algorithm is used for intraday scheduling, with a control period of 5 minutes and a segment time of 3 hours. For comparison, three scenarios are set up for intraday scheduling, and four methods are used for intraday scheduling. (1) The proposed hierarchical MADDPG method. (2) The MADDPG method without improvement. (3) The centralized method (COA). (4) The no-control method. With a 1-hour period, the traditional model-based optimization method is used to centrally control the ADN using intraday actual power information (the optimization objective remains unchanged). The results under different methods are as follows. Figures 5-6 As shown in Table 3.

[0166] Table 3. Intraday scheduling results under different control methods

[0167]

[0168] From Table 3, Figure 5 and Figure 6It can be seen that the hierarchical MADDPG method, MADDPG, and COA method can all ensure that the node voltage does not exceed the limit within a day. Under the hierarchical MADDPG method, the maximum and average node voltage fluctuations over 24 hours are reduced by 17.7% and 26.1% respectively compared to the MADDPG method, far lower than the uncontrolled case. The proposed hierarchical MADDPG decomposes intraday scheduling into global and local features and achieves daily control through different network weights, thereby improving the adaptability of the control model. The COA method shows the best performance across all aspects. However, its results can only be used as a reference because it consumes a large amount of communication resources and affects the real-time performance of control. Therefore, this method is not suitable for practical applications.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning, characterized in that: Includes the following steps: S1: Divide the ADN into multiple regions, each region is an agent, and equip each agent with an edge server. All edge servers communicate bidirectionally with the cloud server. S2: Construct a two-stage scheduling model for ADN, which includes the day-ahead scheduling model of the upper-layer ADN and the intraday scheduling model of the lower-layer ADN. The two-stage scheduling model of ADN is modeled as a Markov decision process. The day-ahead scheduling model is trained by the DDPG model of the upper layer ADN, and the intraday scheduling model is trained by the MADDPG model of the lower layer ADN. The upper-layer ADN's DDPG model includes the system's state, actions, reward design, and an improved DDPG algorithm. The system's state space includes... t The set of states of charge of energy stored in the ADN at any given time. t The system's active and reactive power demand within the ADN at any given time, the active and reactive power demand of the main power grid, and the sum of voltage fluctuations at each node; the system's action space includes the day-ahead scheduling control variables of the ADN, belonging to a mixed discrete and continuous action space; the system's reward function is taken as... t The operating cost of ADN at any given time; The lower-layer ADN's MADDPG model includes the system's state, actions, reward design, and an improved MADDPG training model. The system's state space is the state space of each agent, including... t The system's active and reactive power demands within each intelligent entity at all times, as well as the difference between all nodes of each intelligent entity and the expected voltage; the system's action space includes the reactive power output of the energy storage converter and the new energy converter within each intelligent entity; the system's reward is the difference between the nodes of each intelligent entity and the expected voltage. The improved MADDPG training model is a hierarchical MADDPG model training method that trains the control model at different time scales. The basic model for training a day is divided into 24 time periods over a long time scale, which constitutes one round. A general model is trained based on the predicted power of new energy sources and loads, so that it can basically adapt to the active and reactive power changes of ADN at different times of the day. At a short timescale, multiple network models are trained based on power changes in different time periods; the output of the general model is combined with historical power data for that time period to further train the model; S3: Construct a two-stage deep reinforcement learning model for the ADN. The two-stage deep reinforcement learning model includes the DDPG model of the upper-layer ADN used to train the day-ahead economic scheduling model of the upper-layer ADN and the MADDPG model of the lower-layer ADN used to train the intraday scheduling model of the lower-layer ADN. S4: Model Training: First, the cloud server uses historical power data to train the day-ahead scheduling model using the improved DDPG algorithm in the upper-layer ADN DDPG model; then, in the intraday training phase, the cloud server uses the hierarchical MADDPG model training method in the lower-layer ADN MADDPG model based on the day-ahead scheduling results to train the multi-agent intraday scheduling model for different time periods. The improved DDPG algorithm includes improvements to the handling of mixed actions and model constraints; The process for handling mixed actions is as follows: First, a discrete action network containing the voltage regulation amplitude of the adjustable voltage regulator and the number of capacitor banks is added to the DDPG algorithm. The discrete action network is based on the action network and the output is discretized through the softmax function. Next, discrete actions are generated through the discrete action network, and continuous actions containing the active power output of energy storage are generated through the original continuous action network. Finally, the mixed actions are used as the input of the environment to obtain an experience. Then, in the network update, the critic network is still updated by minimizing the current... Q Values ​​and Targets Q The error realization of the value, the updates of both discrete action networks and continuous action networks are achieved by maximizing the current value. Q Values ​​are used to achieve this, where Q The value is the action value function; Model constraints include action space constraints and state space constraints. Action space constraints are obtained by linearly mapping the neural network output to the corresponding action range, and the energy storage charge state constraints are obtained through corresponding transformations. State space constraints are achieved by adding a penalty term to the reward function.

2. The two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning according to claim 1, characterized in that: The goal of the upper-level ADN day-ahead scheduling model is to minimize the daily operating cost of the ADN while ensuring safe operation. The decision variables include controllable devices within the ADN. The daily operating cost of the ADN includes the main grid cost, energy storage operating cost, branch network loss cost, and penalty cost. Controllable devices within an ADN include energy storage systems, capacitor banks, and on-load tap changers; The main grid cost includes the cost of purchasing electricity from the main grid, carbon emission costs, and the cost of power interaction penalties with the ADN; The operating cost of energy storage is obtained through a levelized cost of electricity (LCOE) model. The penalty costs include the penalty costs for ADN interaction with the main grid and the penalty costs for node voltage fluctuations.

3. The two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning according to claim 2, characterized in that: During each control cycle of the day-ahead dispatch, the constraints that the safe and economical operation of the ADN must meet include: power flow constraints of the entire distribution network, transmission power constraints of the PCC node, power and state of charge constraints of the energy storage system, state of charge constraints of the energy storage system at the end of each day, on-load tap changer constraints, capacitor bank constraints, and voltage constraints of each node of the ADN.

4. The two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning according to claim 1, characterized in that: The goal of the lower-level ADN intraday dispatch model is to cope with intraday random power fluctuations. By adjusting the fast-adjustable reactive power devices, it ensures that the voltage of each node in the distribution network meets the requirements for safe operation, with the objective function being to reduce the difference between the voltage of all nodes in the ADN and the expected voltage.

5. The two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning according to claim 4, characterized in that: During each control cycle of intraday operation, the intraday scheduling of ADN needs to meet the following constraints: power flow constraints of the entire distribution network, transmission power constraints of PCC nodes, voltage constraints of each ADN node, and reactive power output constraints of energy storage systems and new energy sources.

6. A two-stage cloud-edge collaborative scheduling method for active distribution networks based on deep reinforcement learning according to any one of claims 1-5, characterized in that: Also includes: S5: During the model execution phase, the cloud server distributes the trained intraday scheduling model to each edge server. In each intraday control cycle, the edge server controls the corresponding equipment actions based on the real-time local observation information of each agent. During the execution process, each agent makes independent decisions.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1-6.

Images