Main-distribution cooperative scheduling optimization method based on deep reinforcement learning

Abstract

The invention belongs to the technical field of power systems, and more specifically relates to a main and distribution cooperative scheduling method based on deep reinforcement learning. Firstly, a power system model in the active power distribution network is established, various resources contained in the power system model are determined, a mathematical model is constructed, and a power system scheduling agent optimization model is established; then interacting with an active power distribution network environment through an intelligent agent to obtain historical interaction information, and storing the historical interaction information into an experience pool for updating a strategy network and a value network; and finally, regulating and controlling various resources in the power distribution network by utilizing the strategy network. According to the method, the autonomy of the power distribution network is fully considered, an auxiliary decision is provided for scheduling of an upper-layer main network, the peak regulation pressure brought to the main network is reduced, the power deviation of a tie line is reduced, meanwhile, end-to-end input and output control can be achieved through the deep reinforcement learning algorithm based on data driving, and the efficiency in a main and distribution cooperative scheduling task is effectively improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of power systems, and more specifically, relates to a master - distribution collaborative scheduling optimization method based on deep reinforcement learning. Background Art

[0002] With the large - scale access of photovoltaic power, energy storage devices, and flexible loads, the distribution network has already possessed many operating characteristics of the main grid, such as having source - storage - load links and multi - directional power flow. It is necessary to fully consider better exerting initiative on the distribution side, making use of the autonomy of the distribution network to provide auxiliary decision - making for the scheduling of the upper - layer main grid, reducing the peak - shaving pressure on the main grid, and realizing the optimization of master - distribution collaborative scheduling.

[0003] However, the access of a large number of distributed energy sources has led to an increase in decision variables, greatly improving the difficulty of solution and the difficulty of accurate modeling, bringing huge challenges to the application of traditional scheduling optimization methods. The data - driven deep reinforcement learning algorithm has certain advantages in solving such problems. It has characteristics such as not relying on models and adaptive learning, can cope with the complex uncertainty of source and load, and has a good supporting role in the optimal regulation of active distribution networks. Therefore, in - depth research on the active distribution network optimal scheduling method based on deep reinforcement learning for master - distribution collaboration has important theoretical significance and application value. Summary of the Invention

[0004] Aiming at the deficiencies in the existing technology, the present invention proposes a master - distribution collaborative scheduling optimization method based on deep reinforcement learning. This method can achieve the safe, stable, and economic operation of the main - distribution power system under the influence of uncertainty, reduce the power deviation of the tie - line, and reduce the peak - shaving pressure of the distribution network on the main grid.

[0005] To achieve the above - mentioned purpose, the present invention adopts the following technical solutions:

[0006] The master - distribution collaborative scheduling optimization method based on deep reinforcement learning is characterized in that the method includes the following steps:

[0007] Step 1: Determine various types of resources included in the active distribution network and construct its cost model;

[0008] Step 2: Determine the optimization objective of the power system scheduling task and establish its agent optimization model;

[0009] Step 3: According to the model established in Step 1, the agent interacts with the active distribution network environment, obtains historical interaction information and stores it in the experience pool, and uses an improved feature extraction network to realize the iterative update and optimization of the policy network and value network;

[0010] Step 4: Realize the control of the active distribution network according to the policy network obtained in Step 3.

[0011] This technical solution is further optimized. The active distribution network system described in step 1 includes N DG gas turbines, N pv photovoltaic power stations, N ess energy storage power stations, non-adjustable rigid loads, and adjustable flexible loads.

[0012] This technical solution is further optimized. The construction of the cost function of the active distribution network described in step 1 is as follows:

[0013] minC DN = C dg + C ess + C buy + C fl

[0014] where C DN represents the cost function of the active distribution network, and the ultimate goal is to minimize the daily operating cost within this area. C dg represents the operating cost of the gas turbine, C ess represents the cost of the energy storage power station, C buy represents the cost of purchasing electricity from the main grid, C fl represents the cost of flexible load compensation.

[0015] This technical solution is further optimized. The specific expressions of each part in the objective function are as follows:

[0016]

[0017] where a day is divided into K time periods, each time period interval is Δt, a g , b g and c g are the operating cost coefficients of the g-th gas turbine in the active distribution network; is the output of the g-th gas turbine at time k;

[0018]

[0019] where represents the unit charge and discharge price of energy storage at time k; is the charge and discharge power of the e-th energy storage at time k; η ess is the charge and discharge efficiency of the energy storage; and are the charge efficiency and discharge efficiency of the energy storage respectively;

[0020]

[0021] where represents the unit power purchase price of the active distribution network at time k; The interactive power of the active distribution network at time k; Indicates that the distribution network purchases electricity from the main grid, Indicates selling electricity to the main grid.

[0022]

[0023]

[0024] Among them, the flexible load needs to be reduced within a certain range, and the schedulable part is elastically adjustable with the incentive price. At time k, under the incentive price , the proportion of the reducible part that can participate in the grid scheduling in the total flexible load is expressed as Γ(ζ r,k ). λ fl,min and λ fl,max are the minimum incentive price and the maximum incentive price respectively, and β and γ are the elasticity coefficients, and are the maximum reducible amount of the flexible load and the total flexible load at time k respectively. The maximum scheduling amount of the flexible load at time k under the incentive price is expressed as represents the actual reduction amount of the flexible load at time k,

[0025] Further optimizing this technical solution, the constraint conditions of the active distribution network model described in step 1 include power balance constraint, upper and lower limits of gas turbine output, state of charge constraint of energy storage power station, upper and lower limits of reducible load reduction amount, upper and lower limits of interactive power between main and distribution networks, and line power flow constraint, which are specifically as follows:

[0026] The power balance constraint:

[0027]

[0028] Among them are the outputs of the thermal power unit, photovoltaic power plant, and energy storage power station at time k respectively; is the interactive power with the main grid at time k, P load,k is the total load at time k, is the actual reduction amount of the flexible load at time k.

[0029] The state of charge constraint of the energy storage power station:

[0030]

[0031] Among them, is the state of charge of the energy storage at time period k, V ess is the energy storage capacity; and are the upper and lower limits of the charge and discharge power; and are the minimum and maximum values of the energy storage state of charge, respectively. and represent the energy storage state of charge at the initial and termination times.

[0032] The upper and lower limits of the load curtailment amount of the load to be curtailed:

[0033]

[0034] The upper and lower limits of the active distribution network-main grid interactive power:

[0035]

[0036] where and are the minimum and maximum values of the active power interaction between the active distribution network and the main grid, respectively.

[0037] The line power flow constraint:

[0038]

[0039] where and are the power transfer distribution coefficients of the thermal power unit, energy storage power station, photovoltaic power plant, and load at the node to line l; F l,max is the upper limit of the power flow of line l; N b is the total number of grid nodes; is the load value of node b at time k after the flexible load participates in the scheduling, satisfying:

[0040]

[0041] This technical solution is further optimized. The optimization objective of the active distribution network system described in step 2 is specifically as follows:

[0042] The optimization objective of the active distribution network system is to ensure that the constraints are met within the region and the operating cost is the lowest, while improving the utilization rate of new energy and reducing the power fluctuation of the tie line. Specifically:

[0043] C = λ1C DN + λ2C Abandon + λ3C Cut + λ4C Pf + λ5C Diff

[0044] where C DN 、C Abandon 、C Cut 、C Pf and C DiffThey are respectively the operation cost of the distribution network, the penalty for abandoned new energy, the penalty for load shedding, the penalty for power flow over - limit, and the penalty for tie - line power fluctuation. λ1, λ2, λ3, λ4, and λ5 are the weight coefficients of each cost.

[0045] R=-C

[0046] Among them, the optimization goal of the agent is to maximize the reward R, so it is set as the negative value of the cost.

[0047] The penalty for abandoned new energy:

[0048]

[0049] Among them, is the penalty coefficient for abandoned new energy, and P unba,k is the power imbalance at time k, which can be expressed as:

[0050]

[0051] The penalty for load shedding:

[0052]

[0053] Among them, is the penalty coefficient for load shedding.

[0054] The penalty for power flow over - limit:

[0055]

[0056] Among them, is the penalty coefficient for power flow over - limit, and μ k indicates whether there is an over - limit situation at time k, which is a 0 - 1 variable. μ k =1 indicates that there is an over - limit, and μ k =0 indicates that all branches are not over - limit.

[0057] The penalty for tie - line power fluctuation:

[0058]

[0059] Among them, is the penalty coefficient for tie - line power fluctuation, is the tie - line power plan issued by the main grid.

[0060] This technical solution is further optimized for the state space and action space of the agent optimization model described in step 2, specifically as follows:

[0061] The system operation state vector s at time k of the active distribution network k consists of time k, the output of the gas turbine the predicted power of the photovoltaic power station Energy storage state Actual load value P load,k and the maximum reducible amount of flexible load which can be expressed as:

[0062]

[0063] Action vector a at the k-th moment k composed of the adjustment amounts of each unit in the distribution network Energy storage charging and discharging power and the reduction amount of flexible load which can be expressed as:

[0064]

[0065] This technical solution is further optimized. The improved feature extraction network described in step 3 is specifically as follows:

[0066] In deep reinforcement learning, a feature extraction network based on a fully connected layer is commonly used. In this network structure, all neurons are fully connected through two hidden layers. After training, the agent can achieve a direct mapping from input to output. Considering that the state information in the state space is a sequence that changes over time and the input state information has a temporal relationship, a convolutional neural network is used as the feature extraction network in this paper, which is more suitable for the extraction of temporal information in network design and has a faster convergence speed.

[0067] This technical solution is further optimized. Using the deep reinforcement learning algorithm for training to achieve iterative update and optimization of the policy network and value network, the specific steps are as follows:

[0068] Step 3.1: Initialize parameters: Determine the total number of decision-making periods K within a day, the total number of training rounds E for each agent max , the capacity N of the experience pool B , the number N of batch training samples S , the discount factor γ, the learning rate α of the Actor network A and the learning rate α of the Critic network C ; Initialize the current training round number e = 0; Randomly initialize the parameters θ of the target Actor network μ' , the parameters θ of the Actor network μ and the parameters θ of the target Critic network Q' , the parameters θ of the Critic network Q ;

[0069] Step 3.2: Initialize the decision-making moment k = 0, initialize the operation environment of the active distribution network, and the agent inputs the current state s k , outputs the action a k , and obtains the immediate reward rk and transfer to the next state s k+1 , and store the generated samples in the experience pool;

[0070] Step 3.3: Let k = k + 1. If k < K, return to Step 3.2; if k = K, execute Step 3.4;

[0071] Step 3.4: Randomly select N S batch of quadruple sample data [s k , a k , r k , s k+1 ;

[0072] Step 3.5: Calculate a' k+1 through the target Actor network from s k+1 , and then calculate q' k+1 through the target Critic network from a' k+1 and s k+1 . y can be calculated as:

[0073] y = R + γQ'(s k+1 , μ'(s k+1 |θ μ' )|θ Q' )

[0074] = R + γQ'(s k+1 , a' k+1 |θ Q' )

[0075] = R + γq' k+1

[0076] The loss function L of the Critic network can be expressed as:

[0077]

[0078] The update gradient of the Actor network can be expressed as:

[0079]

[0080] Step 3.6: Update the target Actor network and the target Critic network in a soft update manner:

[0081] θ Q' ← τθ Q + (1 - τ)θ Q'

[0082] θ μ' ← τθ μ + (1 - τ)θ μ'

[0083] Step 3.7: Let \(e = e + 1\); if \(e \lt E\) max , then return to Step 3.2, if \(e = E\) max then stop learning and the network training is completed.

[0084] Different from the prior art, the beneficial effects are mainly manifested in that: the large-scale access of photovoltaic, energy storage devices and flexible loads in the active distribution network can reduce the peak regulation pressure on the upper-level power grid. However, due to the volatility, randomness and uncontrollability of new energy power generation output, and at the same time, the increase in decision variables leads to a decrease in computational efficiency, and the increase in its installed capacity will lead to an increase in the uncertainty and regulation difficulty of the distribution network. To address this problem, the present invention provides a main- and distribution-network collaborative scheduling optimization method based on deep reinforcement learning. The data-driven deep reinforcement learning algorithm has a powerful adaptive learning ability, is applicable to high-dimensional continuous action spaces, can adapt to the uncertainty of the environment, and effectively improves the efficiency of main- and distribution-network collaborative optimal scheduling. BRIEF DESCRIPTION OF THE DRAWINGS

[0085] Figure 1 is a flowchart of the scheduling optimization for the active distribution network;

[0086] Figure 2 is a schematic diagram of the scheduling optimization for the active distribution network based on deep reinforcement learning;

[0087] Figure 3 is the structural framework of the improved feature extraction network.

[0088] Figure 4 is a comparison chart of the reward values of different feature networks in the present invention.

[0089] Figure 5 is a scheduling result chart of the policy network in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0090] To describe in detail the technical content, structural features, achieved objectives and effects of the technical solution, the following is a detailed description in conjunction with specific embodiments and with reference to the accompanying drawings.

[0091] Refer to as Figure 1 shown, a main- and distribution-network collaborative scheduling optimization method based on deep reinforcement learning includes the following steps:

[0092] Step 1: Determine various resources included in the active distribution network and construct their cost models;

[0093] The active distribution network system includes N DG gas turbines, N pv photovoltaic power stations, N ess energy storage power stations, non-adjustable rigid loads and adjustable flexible loads. The construction of the active distribution network operating cost function is as follows:

[0094] minC DN = C dg + C ess + C buy + C fl

[0095] where C DN represents the cost function of the active distribution network, and the ultimate goal is to minimize the daily operating cost within this area. C dg represents the operating cost of the gas turbine, C ess represents the cost of the energy storage power station, C buy represents the cost of purchasing electricity from the main grid, C fl represents the cost of flexible load compensation. The specific representation is as follows:

[0096]

[0097] where a day is divided into K time periods, each time period interval is Δt, a g , b g and c g are the operating cost coefficients of the gth gas turbine in the active distribution network; is the output of the gth gas turbine at time k;

[0098]

[0099] where represents the unit charge and discharge price of the energy storage at time k; is the charge and discharge power of the eth energy storage at time k; η ess is the charge and discharge efficiency of the energy storage; and are the charge efficiency and discharge efficiency of the energy storage respectively;

[0100]

[0101] where represents the unit power purchase price of the active distribution network at time k; is the interactive power of the active distribution network at time k; represents that the distribution network purchases electricity from the main grid, represents that the distribution network sells electricity to the main grid.

[0102]

[0103] where the electricity demand of the flexible load can be reduced within a certain range, and the part that can participate in the grid dispatching shows an elastic adjustable characteristic related to the incentive price. At time k, at the incentive price Under the condition, the proportion of the reducible part that can participate in power grid dispatching in the total flexible load is expressed as Γ(ζ r,k ). λ fl,min and λ fl,max are the minimum incentive price and the maximum incentive price respectively, and β and γ are the elasticity coefficients. and are the maximum reducible amount and the total flexible load at time k respectively. The maximum dispatching amount of the flexible load at time k under the incentive price is expressed as represents the actual reducible amount of the flexible load at time k.

[0104] The constraint conditions of the active distribution network model include power balance constraint, upper and lower limits of gas turbine output, state of charge constraint of energy storage power station, upper and lower limits of reducible load reduction amount, upper and lower limits of interactive power between main and distribution networks, and line power flow constraint, which are specifically as follows:

[0105] The power balance constraint:

[0106]

[0107] Among them are the outputs of thermal power units, photovoltaic power plants, and energy storage power stations at time k respectively; is the interactive power with the main grid at time k, and P load,k is the total load at time k, is the actual reducible amount of the flexible load at time k.

[0108] The state of charge constraint of the energy storage power station:

[0109]

[0110] Among them, is the state of charge of the energy storage at time k, and V ess is the energy storage capacity; and are the upper and lower limits of the charge and discharge power; and are the minimum and maximum values of the state of charge of the energy storage respectively. and represent the state of charge of the energy storage at the initial time and the termination time.

[0111] The upper and lower limits of the reducible load reduction amount constraint:

[0112]

[0113] The upper and lower limits of the interactive power between the main and distribution networks constraint:

[0114]

[0115] wherein and are the minimum and maximum values of the active power interaction between the active distribution network and the main grid, respectively.

[0116] The line power flow constraint is as follows:

[0117]

[0118] wherein and are the power transmission distribution coefficients of the thermal power unit, energy storage power station, photovoltaic power plant, and the node where the load is located to line l; F l,max is the upper limit of the power flow of line l; N b is the total number of grid nodes; is the load value of node b at time k after the flexible load participates in the scheduling, satisfying:

[0119]

[0120] Step 2: Determine the optimization objective of the power system scheduling task and establish its intelligent agent optimization model;

[0121] The optimization objective of the active distribution network system is to ensure that the constraints are met within the region and the operating cost is the lowest, while improving the utilization rate of new energy and reducing the power fluctuation of the tie line. Specifically:

[0122] C = λ1C DN + λ2C Abandon + λ3C Cut + λ4C Pf + λ5C Diff

[0123] wherein, C DN , C Abandon , C Cut , C Pf and C Diff are the operating cost of the distribution network, new energy abandonment penalty, load shedding penalty, power flow over-limit penalty, and tie line power fluctuation penalty, respectively. λ1, λ2, λ3, λ4, and λ5 are the weight coefficients of each cost.

[0124] R = -C

[0125] wherein, the optimization objective of the intelligent agent is to maximize the reward R, so it is set as the negative value of the cost.

[0126] The new energy abandonment penalty is as follows:

[0127]

[0128] wherein is the penalty coefficient for new energy curtailment, P unba,k is the power imbalance at time k, which can be expressed as:

[0129]

[0130] The load shedding penalty:

[0131]

[0132] where is the load shedding penalty coefficient.

[0133] The power flow violation penalty:

[0134]

[0135] where is the power flow violation penalty coefficient, μ k indicates whether there is a violation at time k and is a 0-1 variable. μ k = 1 indicates that there is a violation, μ k = 0 indicates that all branches are not violated.

[0136] The tie-line power fluctuation penalty:

[0137]

[0138] where is the tie-line power fluctuation penalty coefficient, is the tie-line power plan issued by the main grid.

[0139] The state space and action space of the agent optimization model are as follows:

[0140] The system operation state vector s at time k of the active distribution network k consists of time k, the output of the gas turbine the predicted power of the photovoltaic power station the energy storage state the actual load value P load,k and the maximum reducible amount of the flexible load and can be expressed as:

[0141]

[0142] The action vector a at time k k consists of the adjustment amounts of each unit in the distribution network the charge and discharge power of the energy storage and the reduction amount of the flexible load and can be expressed as:

[0143]

[0144] Step 3: Use the DDPG algorithm for training to achieve iterative update and optimization of the policy network and value network.

[0145] In deep reinforcement learning, a feature extraction network based on a fully connected layer is commonly used. This network structure connects all neurons through two hidden layers, and after training, the agent can achieve a direct mapping from input to output. Considering that the state information in the state space is a sequence that changes over time and the input state information has a temporal relationship, the present invention uses a convolutional neural network as the feature extraction network, which is more suitable for extracting temporal information in network design and has a faster convergence speed.

[0146] The specific steps are as follows:

[0147] Step 3.1: Initialize parameters: Determine the total number of decision-making periods K within a day, the total number of training episodes E for each agent max , the capacity N of the experience pool B , the number N of batch training samples S , the discount factor γ, the learning rate α of the Actor network A and the learning rate α of the Critic network C ; Initialize the current training episode e = 0; Randomly initialize the parameters θ of the target Actor network μ' , the parameters θ of the Actor network μ and the parameters θ of the target Critic network Q' , the parameters θ of the Critic network Q ;

[0148] Step 3.2: Initialize the decision-making moment k = 0, initialize the operation environment of the active distribution network, and the agent inputs the current state s k , outputs the action a k , obtains the immediate reward r k and transfers to the next state s k+1 , and stores the generated samples in the experience pool;

[0149] Step 3.3: Let k = k + 1. If k < K, return to Step 3.2; if k = K, execute Step 3.4;

[0150] Step 3.4: Randomly select N S batch quadruple sample data [s k , a k , r k , s k+1 ;

[0151] Step 3.5: Calculate a' k+1 through the target Actor network from s k+1 , and then from a'k+1 and s k+1 Calculate q' through the target Critic network k+1 , and y can be calculated as follows:

[0152] y = R + γQ'(s k+1 , μ'(s k+1 |θ μ' )|θ Q' )

[0153] = R + γQ'(s k+1 , a' k+1 |θ Q' )

[0154] = R + γq' k+1

[0155] The loss function L of the Critic network can be expressed as:

[0156]

[0157] The update gradient of the Actor network can be expressed as:

[0158]

[0159] Step 3.6: Update the target Actor network and the target Critic network in a soft update manner:

[0160] θ Q' ← τθ Q + (1 - τ)θ Q'

[0161] θ μ' ← τθ μ + (1 - τ)θ μ'

[0162] Step 3.7: Let e = e + 1; if e < E max , then return to Step 3.2, if e = E max then stop learning and the network training is completed.

[0163] Step 4: Implement the control of the active distribution network according to the policy network obtained in Step 3:

[0164] Using the construction of the above active distribution network environment and agent optimization model, verify the feasibility of the DDPG algorithm. At any decision-making moment during the system operation, normalize the information such as the operating state and input it into the policy network. After forward propagation, denormalize the action information output by the policy network to obtain the actions that the system can take at the current decision-making moment, and observe the reward value after repeating the above operations within a scheduling period.

[0165] Refer to, for example Figure 3 as shown, the structural framework of the improved feature extraction network is explained as follows:

[0166] First, the input state space passes through the feature extraction network to obtain a feature vector. The Actor network flattens the spatial features obtained by extraction in time series and sends them into a fully connected network, and then outputs action A. The Critic network sends the feature vector and action A into the fully connected network at the same time to obtain the value Q of the state-action.

[0167] Refer to, for example Figure 4 as shown, after multiple trainings, both the improved feature extraction network and the traditional feature extraction network based on fully connected layers can achieve convergence. However, the agent using the convolutional network can better extract the time series information in the input state, which is convenient for the agent to make more reasonable action outputs according to the current state. Its convergence speed is faster and the result is better than that of the fully connected layer.

[0168] Refer to, for example Figure 5 as shown, since the photovoltaic power starts to output in the active distribution network at the 20th period, the distribution network can meet part of the load demand, and the net load decreases, so the tie line power drops significantly. From the 55th period, the photovoltaic power output in the distribution network starts to decline, and the load demand is at a high level. To meet the load balance in its area, the main network power purchase needs to be increased, and the tie line power shows an obvious upward trend. According to Figure 5 the display, it can be clearly seen that there is two-way power flow between the active distribution network and the main network, and power support can be provided to the upper-level power grid during some periods, reducing the peak regulation pressure on the main network.

[0169] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or terminal device. Without further limitation, the elements defined by the statement "including..." or "comprising..." do not exclude the existence of additional elements in the process, method, article or terminal device including the said elements. In addition, in this article, "greater than", "less than", "exceeding", etc. are understood not to include the present number; "above", "below", "within", etc. are understood to include the present number.

[0170] Although the above-described embodiments have been described, those skilled in the art can make additional changes and modifications once they know the basic creative concept. Therefore, the above are only the embodiments of the present invention, and do not limit the patent protection scope of the present invention. Any equivalent structural or equivalent process transformation made by using the content of the specification and drawings of the present invention, or directly or indirectly applied in other related technical fields, is similarly included in the patent protection scope of the present invention.

Claims

1. A master-slave collaborative scheduling optimization method based on deep reinforcement learning, characterized in that, The method includes the following steps: Step 1: Determine various types of resources included in the active distribution network and construct their cost models; Step 2: Determine the optimization objectives of the power system scheduling tasks and establish their agent optimization models; Step 3: According to the model established in Step 1, the agent interacts with the active distribution network environment, obtains historical interaction information and stores it in the experience pool, and uses the improved feature extraction network to realize the iterative update and optimization of the policy network and the value network; Step 4: Implement the control of the active distribution network according to the policy network obtained in Step 3.

2. The main and accessory collaborative scheduling optimization method based on deep reinforcement learning according to claim 1, wherein The active distribution network system described in step 1 includes N DG gas turbines, N pv photovoltaic power stations, N ess energy storage stations, non-adjustable rigid loads, and adjustable flexible loads.

3. The main and distribution collaborative scheduling optimization method based on deep reinforcement learning according to claim 2, characterized in that The construction of the active distribution network cost function in Step 1 is as follows: minC DN = C dg + C ess + C buy + C fl Among them, C DN represents the cost function of the active distribution network, and the ultimate goal is to minimize the daily operating cost within this area. C dg represents the operating cost of the gas turbine, C ess represents the cost of the energy storage power station, C buy represents the cost of purchasing electricity from the main grid, C fl represents the cost of flexible load compensation.

4. The master-slave collaborative scheduling optimization method based on deep reinforcement learning according to claim 3, wherein The specific expressions of each part in the objective function are as follows: Among them, a day is divided into K time periods, and the interval of each time period is Δt, a g , b g and c g are the operating cost coefficients of the gth gas turbine in the active distribution network; is the output of the gth gas turbine at time k; Among them, represents the unit charge-discharge price of energy storage at time k; is the charge-discharge power of the e-th energy storage at time k; η ess is the charge-discharge efficiency of the energy storage; and are the charge efficiency and discharge efficiency of the energy storage respectively; Among them, represents the unit power purchase price of the active distribution network at time k; is the interactive power of the active distribution network at time k; indicates that the distribution network purchases electricity from the main grid, indicates selling electricity to the main grid; Among them, the flexible load needs to be reduced within a certain range. The part that can participate in scheduling has an elastic adjustability with the incentive price. At time k, under the incentive price , the proportion of the reducible part that can participate in the power grid scheduling in the total flexible load is expressed as Γ(ζ r,k ). λ fl,min and λ fl,max are the minimum incentive price and the maximum incentive price respectively, and β and γ are the elasticity coefficients respectively. and are the maximum reducible amount of the flexible load and the total flexible load at time k respectively. The maximum scheduling amount of the flexible load at time k under the incentive price is expressed as represents the actual reduction amount of the flexible load at time k.

5. The main and distribution collaborative scheduling optimization method based on deep reinforcement learning according to claim 4, characterized in that The constraint conditions of the active distribution network model in Step 1 include power balance constraints, upper and lower limits of gas turbine output, state of charge constraints of energy storage power stations, upper and lower limits of the reduction amount of curtailable loads, upper and lower limits of the interactive power between the main and distribution networks, and line power flow constraints, which are specifically as follows: The power balance constraint: where are the outputs of the thermal power unit, photovoltaic power plant, and energy storage power station at time k, respectively; is the interactive power with the main grid at time k, P load,k is the total load at time k, is the actual reduction amount of the flexible load at time k; The state of charge constraint of the energy storage power station: Among them, is the state of charge of energy storage in the k-th period, and V ess is the energy storage capacity; and are the upper and lower limits of the charge and discharge power; and are the minimum and maximum values of the state of charge of energy storage respectively, and represent the state of charge of energy storage at the initial and termination times; The upper and lower limits of the reduction amount of curtailable loads: The upper and lower limits of the interactive power between the main and distribution networks: wherein and are respectively the minimum value and the maximum value of the active power interaction between the active distribution network and the main grid; The line power flow constraint: Among them and are the power transfer distribution coefficients of the thermal power unit, energy storage power station, photovoltaic power plant, and the node where the load is located to line l; F l,max is the upper limit of the power flow of line l; N b is the total number of grid nodes; is the load value of node b at time k after the flexible load participates in the scheduling, satisfying:

6. The main and distribution collaborative scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that The optimization objectives of the active distribution network system in Step 2 are specifically as follows: The optimization objective of the active distribution network system is to ensure that the constraints are met within the region and the operating cost is the lowest, while improving the utilization rate of new energy and reducing the power fluctuation of the tie line. Specifically: C = λ1C DN + λ2C Abandon + λ3C Cut + λ4C Pf + λ5C Diff Among them, C DN , C Abandon , C Cut , C Pf and C Diff are respectively the operation cost of the distribution network, the penalty for abandoned new energy, the penalty for load shedding, the penalty for power flow over-limit and the penalty for tie-line power fluctuation, and λ1, λ2, λ3, λ4 and λ5 are the weight coefficients of each cost; R = -C Among them, the optimization objective of the agent is to maximize the reward R, so it is set as the negative value of the cost; The penalty for abandoned new energy: Among them, is the penalty coefficient for new energy curtailment, and P unba,k is the power imbalance in the k-th period, which can be expressed as: The penalty for load shedding: Among them, is the loss-of-load penalty factor; The penalty for power flow over-limit: Among them, is the penalty coefficient for power flow limit violation, μ k indicates whether there is a limit violation in the k-th period, which is a 0-1 variable. μ k = 1 indicates that there is a limit violation, and μ k = 0 indicates that all branches have no limit violation; The penalty for tie line power fluctuation: Among them, is the penalty coefficient for the power fluctuation of the tie line, is the tie line power plan issued by the main grid.

7. The main and distribution collaborative scheduling optimization method based on deep reinforcement learning according to claim 6, characterized in that The state space and action space of the agent optimization model in Step 2 are specifically as follows: The system operation state vector s of the active distribution network at time k k At time k, the output of the gas turbine The predicted power of the photovoltaic power station The state of the energy storage The actual value of the load P load,k And the maximum reducible amount of the flexible load Composed of, which can be expressed as: Action vector a at the k-th moment k Adjustment amount of each unit in the distribution network Energy storage charge and discharge power And flexible load reduction amount Composed of, can be expressed as:

8. The main and distribution collaborative scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that In the improved feature extraction network in Step 3, a convolutional neural network is used as the feature extraction network.

9. The main and distribution collaborative scheduling optimization method based on deep reinforcement learning according to claim 1, characterized in that Step 3 uses a deep reinforcement learning algorithm for training to realize the iterative update and optimization of the policy network and the value network, The specific steps are as follows: Step 3.1: Initialize parameters: Determine the total number of decision-making periods K within a day, the total number of training rounds E for each agent max , the capacity N of the experience pool B , the number N of batch training samples S , the discount factor γ, the learning rate α of the Actor network A and the learning rate α of the Critic network C ; Initialize the current training round number e = 0; Randomly initialize the parameters θ of the target Actor network μ' , the parameters θ of the Actor network μ and the parameters θ of the target Critic network Q' , the parameters θ of the Critic network Q ; Step 3.2: Initialize the decision-making moment \(k = 0\), initialize the operating environment of the active distribution network, and the agent inputs the current state \(s\) k , and outputs the action \(a\) k , and obtains the immediate reward \(r\) k and transfers to the next state \(s\) k+1 , and stores the generated samples in the experience pool; Step 3.3: Let k = k + 1. If k < K, return to Step 3.

2. If k = K, execute Step 3.4; Step 3.4: Randomly select N S batch of quadruple sample data [s k , a k , r k , s k+1 ; Step 3.5: From s k+1 Calculate a' through the target Actor network k+1 , and then from a' k+1 and s k+1 Calculate q' through the target Critic network k+1 , it can be calculated that y is: y = R + γQ'(s k+1 , μ'(s k+1 |θ μ' )|θ Q' ) = R + γQ'(s k+1 , a' k+1 |θ Q' ) = R + γq' k+1 The loss function L of the Critic network can be expressed as: The update gradient of the Actor network can be expressed as: Step 3.6: The target Actor network and the target Critic network are updated in a soft update manner: θ Q' ← τθ Q +(1 - τ)θ Q' θ μ' ← τθ μ +(1 - τ)θ μ' Step 3.7: Let e = e + 1; if e < E max , then return to Step 3.2, if e = E max then stop learning and the network training is completed.

Images