A power distribution network double-layer optimization scheduling method based on deep reinforcement learning

Abstract

The application discloses a power distribution network double-layer optimization scheduling method based on deep reinforcement learning, and comprises the following steps: 1, obtaining a power distribution network distributed power output and load prediction model established by taking the minimum node power prediction deviation as an objective function; 2, converting the power distribution network distributed power output and load prediction model into a corresponding Markov decision model by using a deep deterministic policy gradient algorithm, and solving the Markov decision model by using the deep deterministic policy gradient algorithm to obtain the power distribution network distributed power output and load prediction value; 3, establishing a power distribution network upper-layer optimization scheduling model containing distributed power regulation and control cost in a long time scale, and solving the model by using a particle swarm algorithm to obtain the optimal output of the distributed power and load in the long time scale; and 4, designing a power distribution network lower-layer power optimization distribution function based on online rolling optimization in a short time scale, and distributing the active power of the power distribution network distributed power and load in a distributed optimal manner in the short time scale.

Description

Technical Field

[0001] This invention belongs to the field of distribution network optimization scheduling technology, specifically involving a two-layer optimization scheduling method for distribution networks based on deep reinforcement learning. Background Technology

[0002] With the depletion of fossil fuels, renewable energy sources, represented by wind power and solar power, have developed rapidly, and their proportion in distribution networks has gradually increased in recent years. However, the widespread integration of new energy sources such as wind and solar power into distribution networks has brought enormous pressure and risks to the operation and control of these networks due to their inherent randomness and volatility. Therefore, addressing the dispatching challenges arising from the volatility and randomness of new energy sources in distribution networks has become a problem that power grid dispatching needs to solve. Summary of the Invention

[0003] This invention addresses the shortcomings of existing technologies by proposing a deep reinforcement learning-based solution that enables precise power scheduling of distribution networks across multiple time scales. This leads to the acquisition of the optimal power scheduling strategy, ensuring the high-quality and efficient operation of the distribution network. The specific technical solution is as follows: A two-layer optimal scheduling method for distribution networks based on deep reinforcement learning, the specific steps of which are as follows: Step 1: Obtain the distributed generation output and load forecasting model for the distribution network with the objective function of minimizing the node power prediction deviation; Step 2: Convert the distributed generation output and load forecasting model of the distribution network into a corresponding Markov decision model, and solve the Markov decision model using the deep deterministic strategy gradient algorithm to obtain the distributed generation output and load forecast values ​​of the distribution network. Step 3: Establish an upper-level optimal scheduling model for the distribution network that includes the cost of distributed generation regulation over a long time scale. Based on the output and load forecast values ​​of the distributed generation in the distribution network obtained in Step 2, the particle swarm optimization algorithm is used to solve the problem and obtain the optimal output of the distributed generation and load over a long time scale. Step 4: Design a power optimization allocation function for the lower layer of the distribution network based on online rolling optimization under short time scale. Perform distributed optimal allocation of active power of distributed power sources and loads in the distribution network under short time scale to achieve accurate tracking of the optimal output of active power of distributed power sources and loads in Step 3.

[0004] As an optimization, the objective function of the distributed generation output and load forecasting model in step one of the distribution network is: (1) Among them: Among them, Indicates the number of prediction results; Indicates the predicted value; Represents the actual value.

[0005] As an optimization: Step two specifically involves... 2.1 Set the state space to the current active power of each distributed power source and load; set the action space to the changes in active power output of each distributed power source and the changes in load; set the reward function to the quadratic sum of the actual active power value and the predicted active power value of each node. The state space is as follows: (2) in, For the first The active power value of a distributed generation unit or load. , This represents the total number of distributed power sources and loads in the distribution network. The action space refers to the changes in the active power output of the distributed power sources and loads integrated into each node. ,in , and They are nodes The minimum and maximum active power of the distributed power source and load are incorporated. This represents the current active power of distributed power sources and loads. The reward function is: (3) in: For the first The actual active power output of each distributed power source and load. For the first Predicted active power output of distributed power sources and loads. This is the weight matrix; 2.2: The DDPG algorithm is used to solve the distributed generation output and load prediction model of the distribution network to obtain the distributed generation output and load prediction values ​​of the distribution network.

[0006] As an optimization: Step 2.2 involves using the DDPG algorithm to solve the distributed generation output and load forecasting model of the distribution network to obtain the distributed generation output and load forecast values ​​of the distribution network, specifically as follows: 2.2.1 Initialize the parameters of the main network and the target network, initialize the memory, and observe the active power of each distributed power source and load as the initial state; 2.2.2: Select an action based on the behavioral strategy and issue it to the environment to execute the action, thereby obtaining a reward value and a new state; 2.2.3: Store the state transition process obtained in 2.2.2 into the memory bank, and randomly sample the transition data in the memory bank as training data for the policy master network and the evaluation master network; 2.2.4: Update the parameters of the evaluation master network using the gradient descent method, and use the average value of continuous observations to soft-update the parameters of the master network to the target network; 2.2.5: Repeat steps 2.2.2 to 2.2.4 until the iteration ends, and obtain the output and load forecast values ​​of the distributed power source.

[0007] As an optimization, step three specifically includes the following steps: Step 3.1: The time scale of the upper-level optimization scheduling model for the distribution network under the long-term scale is on the hourly level. Specifically, it includes the objective functions for the cost objectives of distributed new energy, energy storage, and load regulation. The regulation cost objective function can be specifically expressed as: (4) In equation (4), , and They represent t Time of the first i The operating revenue of the first distributed new energy source, the first j The operating cost of the first energy storage and the second l Operating cost per load; , and These represent collections of distributed renewable energy sources, energy storage, and loads, respectively. , and It can be expressed in the form of a quadratic function: (5) In equation (5), , and For the first i The operating benefit coefficient of a distributed new energy source; , and For the first j The operating cost coefficient of an energy storage system; , and For the first l Operating cost coefficient for each load; , and They are respectively t Time of the first i The active power output of the first distributed new energy source, the first j The active power output of the first energy storage and the firstl The active power output of each load; The aforementioned control cost objective function (4) should satisfy the following constraints: (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) Equations (6) and (7) represent the upper and lower limits of active power constraints for distributed renewable energy and energy storage, respectively. and They are respectively Time of the first Active power output of distributed new energy sources and energy storage and For the first Upper and lower limits of active power of distributed renewable energy sources. and For the first The upper and lower limits of active power for each energy storage system, among which The power output and load prediction model of the distributed power generation in the distribution network described in step one; Equation (8) represents the upper and lower limits of the active power of the load. for Time of the first Active power demand of each load and For the first The upper and lower limits of active power demand for each load, of which The power output and load forecasting model of the distributed power generation in the distribution network described in step one; Equation (9) represents the active power ramping constraint of the load. and For the first The active power ramp-up limits for each load; Equations (12) and (13) represent the active and reactive power balance constraints between adjacent nodes, respectively. and They are respectively Time of the first The injected active power and injected reactive power of each node and They are respectively Time of the first The injected active power and injected reactive power of each node and They are respectively Time of the first The active power loss and reactive power loss of each node. and They are respectively Time of the first Active power output of distributed new energy sources. For the position located at the A collection of distributed new energy sources with nodes. for Time of the first The active power output of the energy storage unit. and They are located at the th A collection of energy storage and load at each node. for Time of the first The active power consumed by the load of each node; Equation (14) is the SOC constraint for energy storage. for Time of the first A single energy storage SOC, and For the first The upper and lower limits of the SOC of the energy storage; Equation (15) is the active power balance constraint of the system; Step 3.2: The particle swarm optimization algorithm is used to solve the objective function of regulation cost established in Step 3.1, so as to obtain the optimal output of distributed new energy, energy storage and load over a long time scale.

[0008] As an optimization: step 4 specifically involves; After solving the upper-level optimization scheduling objective function in step 3.2, an active power regulation objective function for distributed generation units and loads is established within a short-time scale, i.e., a minute-level response interval. The active power output is then adjusted through online rolling optimization. (16) In equation (16), For the prediction domain length, This represents the total number of distributed generation and load units on a short timescale. Let be the cost function of the i-th distributed generation or load unit; The constraints of the objective function (16) are as follows: (17) (18) in, Let be the power value of the i-th distributed generation or load unit predicted at time k+j. The maximum active power of the i-th distributed generation or load unit predicted at time k+j is derived from the distributed power output and load prediction model of the distribution network described in step one. The optimal output of distributed renewable energy, energy storage, and load over a long time scale is obtained from the solution in step 3-2; By continuously optimizing the objective function (16) at each moment in a short time scale and issuing the optimal command to the distributed generation or load unit, the active power of the distributed generation or load unit can be precisely adjusted.

[0009] The beneficial effects of this invention are as follows: This invention uses a deep reinforcement learning algorithm to predict the active power of distributed generation units and loads in the distribution network, reducing the impact of uncertainties at both the source and load ends on the optimal scheduling of the system. At the same time, it uses a two-layer optimal scheduling algorithm at different time scales to adjust the active power output at the source and load ends of the system, which can make full use of the output of distributed power sources and adjust the active power demand of the load. It also ensures the economy, flexibility and speed of power scheduling of the distribution network when the output of distributed power sources is unstable, and has good dynamic response performance. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the process of the present invention. Detailed Implementation

[0011] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0012] like Figure 1 As shown: 1. A two-layer optimal scheduling method for distribution networks based on deep reinforcement learning, the specific steps of which are as follows: Step 1: Obtain the distributed generation output and load forecasting model for the distribution network with the objective function of minimizing the node power prediction deviation; The objective function of the distributed generation output and load forecasting model in step one is: (1) Among them: Among them, Indicates the number of prediction results; Indicates the predicted value; Represents the actual value.

[0013] Step 2: Convert the distributed generation output and load forecasting model of the distribution network into a corresponding Markov decision model, and solve the Markov decision model using the deep deterministic strategy gradient algorithm to obtain the distributed generation output and load forecast values ​​of the distribution network. 2.1 Set the state space to the current active power of each distributed power source and load; set the action space to the changes in active power output of each distributed power source and the changes in load; set the reward function to the quadratic sum of the actual active power value and the predicted active power value of each node. The state space is as follows: (2) in, For the first The active power value of a distributed generation unit or load. , This represents the total number of distributed power sources and loads in the distribution network. The action space refers to the changes in the active power output of the distributed power sources and loads integrated into each node. ,in , and They are nodes The minimum and maximum active power of the distributed power source and load are incorporated. This represents the current active power of distributed power sources and loads. The reward function is: (3) in: For the first The actual active power output of each distributed power source and load. For the first Predicted active power output of distributed power sources and loads. This is the weight matrix; 2.2: The DDPG algorithm is used to solve the distributed generation output and load prediction model of the distribution network to obtain the distributed generation output and load prediction values ​​of the distribution network.

[0014] Step 2.2, which involves using the DDPG algorithm to solve the distributed generation output and load forecasting model of the distribution network to obtain the distributed generation output and load forecast values ​​of the distribution network, specifically involves: 2.2.1 Initialize the parameters of the main network and the target network, initialize the memory, and observe the active power of each distributed power source and load as the initial state; 2.2.2: Select an action based on the behavioral strategy and issue it to the environment to execute the action, thereby obtaining a reward value and a new state; 2.2.3: Store the state transition process obtained in 2.2.2 into the memory bank, and randomly sample the transition data in the memory bank as training data for the policy master network and the evaluation master network; 2.2.4: Update the parameters of the evaluation master network using the gradient descent method, and use the average value of continuous observations to soft-update the parameters of the master network to the target network; 2.2.5: Repeat steps 2.2.2 to 2.2.4 until the iteration ends, and obtain the output and load forecast values ​​of the distributed power source.

[0015] Step 3: Establish an upper-level optimal scheduling model for the distribution network that includes the cost of distributed generation regulation over a long time scale. Based on the output and load forecast values ​​of the distributed generation in the distribution network obtained in Step 2, the particle swarm optimization algorithm is used to solve the problem and obtain the optimal output of the distributed generation and load over a long time scale. Step three specifically includes the following steps: Step 3.1: The time scale of the upper-level optimization scheduling model for the distribution network under the long-term scale is on the hourly level. Specifically, it includes the objective functions for the cost objectives of distributed new energy, energy storage, and load regulation. The regulation cost objective function can be specifically expressed as: (4) In equation (4), , and They represent t Time of the first i The operating revenue of the first distributed new energy source, the first j The operating cost of the first energy storage and the second l Operating cost per load; , and These represent collections of distributed renewable energy sources, energy storage, and loads, respectively. , and It can be expressed in the form of a quadratic function: (5) In equation (5), , and For the first i The operating benefit coefficient of a distributed new energy source; , and For the first j The operating cost coefficient of an energy storage system; , and For the first l Operating cost coefficient for each load; , and They are respectively t Time of the first i The active power output of the first distributed new energy source, the first j The active power output of the first energy storage and the first l The active power output of each load; The aforementioned control cost objective function (4) should satisfy the following constraints: (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) Equations (6) and (7) represent the upper and lower limits of active power constraints for distributed renewable energy and energy storage, respectively. and They are respectively Time of the first Active power output of distributed new energy sources and energy storage and For the first Upper and lower limits of active power of distributed renewable energy sources. and For the first The upper and lower limits of active power for each energy storage system, among which The power output and load prediction model of the distributed power generation in the distribution network described in step one; Equation (8) represents the upper and lower limits of the active power of the load. for Time of the first Active power demand of each load and For the first The upper and lower limits of active power demand for each load, of which The power output and load forecasting model of the distributed power generation in the distribution network described in step one; Equation (9) represents the active power ramping constraint of the load. and For the first The active power ramp-up limits for each load; Equations (12) and (13) represent the active and reactive power balance constraints between adjacent nodes, respectively. and They are respectively Time of the first The injected active power and injected reactive power of each node and They are respectively Time of the first The injected active power and injected reactive power of each node and They are respectively Time of the first The active power loss and reactive power loss of each node. and They are respectively Time of the first Active power output of distributed new energy sources. For the position located at the A collection of distributed new energy sources with nodes. for Time of the first The active power output of the energy storage unit. and They are located at the th A collection of energy storage and load at each node. for Time of the first The active power consumed by the load of each node; Equation (14) is the SOC constraint for energy storage. for Time of the first A single energy storage SOC, and For the first The upper and lower limits of the SOC of the energy storage; Equation (15) is the active power balance constraint of the system; Step 3.2: The particle swarm optimization algorithm is used to solve the objective function of regulation cost established in Step 3.1, so as to obtain the optimal output of distributed new energy, energy storage and load over a long time scale.

[0016] Step 4: Design a power optimization allocation function for the lower layer of the distribution network based on online rolling optimization under short time scale. Perform distributed optimal allocation of active power of distributed power sources and loads in the distribution network under short time scale to achieve accurate tracking of the optimal output of active power of distributed power sources and loads in Step 3.

[0017] Step 4 specifically involves: After solving the upper-level optimization scheduling objective function in step 3.2, an active power regulation objective function for distributed generation units and loads is established within a short-time scale, i.e., a minute-level response interval. The active power output is then adjusted through online rolling optimization. (16) In equation (16), For the prediction domain length, This represents the total number of distributed generation and load units on a short timescale. Let be the cost function of the i-th distributed generation or load unit; The constraints of the objective function (16) are as follows: (17) (18) in, Let be the power value of the i-th distributed generation or load unit predicted at time k+j. The maximum active power of the i-th distributed generation or load unit predicted at time k+j is derived from the distributed power output and load prediction model of the distribution network described in step one. The optimal output of distributed renewable energy, energy storage, and load over a long time scale is obtained from the solution in step 3-2; By continuously optimizing the objective function (16) at each moment in a short time scale and issuing the optimal command to the distributed generation or load unit, the active power of the distributed generation or load unit can be precisely adjusted.

[0018] This invention presents a two-layer optimal scheduling method for distribution networks based on deep reinforcement learning. It proposes an optimal scheduling strategy for distribution networks that considers the uncertainties of renewable energy generation and multi-energy load forecasting. By utilizing source-load power forecasting based on deep reinforcement learning to address the uncertainties of renewable energy sources and loads within the system, the reliability of the strategy is improved. Furthermore, this invention employs a two-layer optimal scheduling algorithm to adjust the active power output at the source and load ends of the system. This fully utilizes the output of distributed generation sources and ensures the economy, flexibility, and speed of power regulation under conditions of unstable distributed generation output and load fluctuations.

[0019] The present invention has been disclosed above with reference to preferred embodiments, but it is not intended to limit the present invention. All technical solutions obtained by adopting equivalent substitutions or equivalent transformations fall within the protection scope of the present invention.

Claims

1. A two-layer optimal scheduling method for distribution networks based on deep reinforcement learning, characterized in that, The specific steps are as follows: Step 1: Obtain the distributed generation output and load forecasting model for the distribution network with the objective function of minimizing the node power prediction deviation; Step 2: Convert the distributed generation output and load forecasting model of the distribution network into a corresponding Markov decision model, and solve the Markov decision model using the deep deterministic strategy gradient algorithm to obtain the distributed generation output and load forecast values ​​of the distribution network. Step 3: Establish an upper-level optimal scheduling model for the distribution network that includes the cost of distributed generation regulation over a long time scale. Based on the output and load forecast values ​​of the distributed generation in the distribution network obtained in Step 2, the particle swarm optimization algorithm is used to solve the problem and obtain the optimal output of the distributed generation and load over a long time scale. Step 3.1: The time scale of the upper-level optimization scheduling model for the distribution network under the long-term scale is hourly, specifically including the objective functions of distributed new energy, energy storage, and load regulation costs. The regulation cost objective function can be specifically expressed as: (4) In equation (4), , and They represent t Time of the first i The operating revenue of the first distributed new energy source, the first j The operating cost of the first energy storage and the second l Operating cost per load; , and These represent collections of distributed renewable energy sources, energy storage, and loads, respectively. , and It can be expressed in the form of a quadratic function: （5） In equation (5), , and For the first i The operating benefit coefficient of a distributed new energy source; , and For the first j The operating cost coefficient of an energy storage system; , and For the first l Operating cost coefficient for each load; , and They are respectively t Time of the first i The active power output of the first distributed new energy source, the first j The active power output of the first energy storage and the first l The active power output of each load; The aforementioned control cost objective function (4) should satisfy the following constraints: （6） （7） （8） （9） （10） （11） （12） （13） （14） （15） Equations (6) and (7) represent the upper and lower limits of active power constraints for distributed renewable energy and energy storage, respectively. and They are respectively Time of the first Active power output of distributed new energy sources and energy storage and For the first Upper and lower limits of active power of distributed renewable energy sources. and For the first The upper and lower limits of active power for each energy storage system, among which The power output and load prediction model of the distributed power generation in the distribution network described in step one; Equation (8) represents the upper and lower limits of the active power of the load. for Time of the first Active power demand of each load and For the first The upper and lower limits of active power demand for each load, of which The power output and load forecasting model of the distributed power generation in the distribution network described in step one; Equation (9) represents the active power ramping constraint of the load. and For the first The active power ramp-up limits for each load; Equations (12) and (13) represent the active and reactive power balance constraints between adjacent nodes, respectively. and They are respectively Time of the first The injected active power and injected reactive power of each node and They are respectively Time of the first The injected active power and injected reactive power of each node and They are respectively Time of the first The active power loss and reactive power loss of each node. and They are respectively Time of the first Active power output of distributed new energy sources. For the position located at the A collection of distributed new energy sources with nodes. for Time of the first The active power output of the energy storage unit. and They are located at the th A collection of energy storage and load at each node. for Time of the first The active power consumed by the load of each node; Equation (14) is the SOC constraint for energy storage. for Time of the first A single energy storage SOC, and For the first The upper and lower limits of the SOC of the energy storage; Equation (15) is the active power balance constraint of the system; Step 3.2: The particle swarm optimization algorithm is used to solve the control cost objective function established in Step 3.1, so as to obtain the optimal output of distributed new energy, energy storage and load over a long time scale; Step 4: Design a power optimization allocation function for the lower layer of the distribution network based on online rolling optimization under short time scale. Perform distributed optimal allocation of active power of distributed power sources and loads in the distribution network under short time scale to achieve accurate tracking of the optimal active power output of distributed power sources and loads in Step 3. After solving the upper-level optimization scheduling objective function in step 3.2, an active power regulation objective function for distributed generation units and loads is established within a short-time scale, i.e., a minute-level response interval. The active power output is then adjusted through online rolling optimization. （16） In equation (14), For the prediction domain length, This represents the total number of distributed generation and load units on a short timescale. Let be the cost function of the i-th distributed generation or load unit; The constraints of the objective function (16) are as follows: （17） （18） in, Let be the power value of the i-th distributed generation or load unit predicted at time k+j. The maximum active power of the i-th distributed generation or load unit predicted at time k+j is derived from the distributed power output and load prediction model of the distribution network described in step one. The optimal output of distributed renewable energy, energy storage, and load over a long time scale is obtained from the solution in step 3-2; By continuously optimizing the objective function (16) at each moment in a short time scale and issuing the optimal command to the distributed generation or load unit, the active power of the distributed generation or load unit can be precisely adjusted.

2. The two-layer optimal scheduling method for distribution networks based on deep reinforcement learning according to claim 1, characterized in that: The objective function of the distributed generation output and load forecasting model in step one of the distribution network is: (1) Among them: Among them, Indicates the number of prediction results; Indicates the predicted value; Represents the actual value.

3. The two-layer optimal scheduling method for distribution networks based on deep reinforcement learning according to claim 1, characterized in that: The second step specifically involves, 2.1 Set the state space to the current active power of each distributed power source and load; set the action space to the changes in active power output of each distributed power source and the changes in load; set the reward function to the quadratic sum of the actual active power value and the predicted active power value of each node. The state space is as follows: （2） in, For the first The active power value of a distributed generation unit or load. , This represents the total number of distributed power sources and loads in the distribution network. The action space refers to the changes in the active power output of the distributed power sources and loads integrated into each node. ,in , and They are nodes The minimum and maximum active power of the distributed power source and load are incorporated. This represents the current active power of distributed power sources and loads. The reward function is: （3） in: For the first The actual active power output of each distributed power source and load. For the first Predicted active power output of distributed power sources and loads. This is the weight matrix; 2.2: The DDPG algorithm is used to solve the distributed generation output and load prediction model of the distribution network to obtain the distributed generation output and load prediction values ​​of the distribution network.

4. The two-layer optimal scheduling method for distribution networks based on deep reinforcement learning according to claim 3, characterized in that: Step 2.2, which involves using the DDPG algorithm to solve the distributed generation output and load forecasting model of the distribution network to obtain the distributed generation output and load forecast values ​​of the distribution network, specifically involves: 2.2.1 Initialize the parameters of the main network and the target network, initialize the memory, and observe the active power of each distributed power source and load as the initial state; 2.2.2: Select an action based on the behavioral strategy and issue it to the environment to execute the action, thereby obtaining a reward value and a new state; 2.2.3: Store the state transition process obtained in 2.2.2 into the memory bank, and randomly sample the transition data in the memory bank as training data for the policy master network and the evaluation master network; 2.2.4: Update the parameters of the evaluation master network using the gradient descent method, and use the average value of continuous observations to soft-update the parameters of the master network to the target network; 2.2.5: Repeat steps 2.2.2 to 2.2.4 until the iteration ends, and obtain the output and load forecast values ​​of the distributed power source.

Images