A novel multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning

Abstract

The present invention discloses a novel multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning, which relates to the field of optimization of wind power / photovoltaic and other new energy power generation connected to the power grid. Specifically, a multi-objective two-stage robust optimization mathematical model for a wind power / photovoltaic / thermal power hybrid energy system is first modeled, and then the two-stage robust optimization problem is solved by multi-agent reinforcement learning, and the multi-objective optimization problem is solved by meta-reinforcement learning, and finally the two are combined to form an end-to-end solution for multi-objective two-stage robust optimization of a wind power / photovoltaic / thermal power hybrid energy system. The present invention is used to solve the multi-objective two-stage robust optimization problem of new energy power generation connected to the power grid, and through advanced artificial intelligence algorithms, a more efficient, flexible and optimal solution to the scheduling strategy is achieved. The present invention can save optimization experience and can directly use the trained neural network model, which will greatly save computing cost and computing time in actual production applications.

Description

Technical Field

[0001] The present invention relates to the field of optimization of wind power / photovoltaic and other renewable energy power generation connected to the power grid, and specifically to a novel multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning. Background Art

[0002] An energy structure that is overly dependent on fossil fuels has posed enormous challenges to climate and environmental issues. As global climate and environmental issues have become increasingly severe in recent years, countries around the world are paying more and more attention to the progress of energy structure adjustments. Replacing traditional fossil fuel energy with clean renewable energy has become a global development consensus.

[0003] Wind and solar power, as the mainstays of renewable energy generation, have seen their share of the power system steadily increase in recent years. This is due to the widespread distribution of wind and solar resources, enabling their application in diverse geographical environments. However, their intermittent and volatile nature as non-stationary power sources presents significant challenges for grid integration. Developing solutions to address the multi-objective and uncertainties inherent in optimizing the scheduling of renewable energy generation for grid integration is key to improving the regulatory capabilities of emerging power systems.

[0004] Traditional solutions based on swarm intelligence optimization algorithms have problems such as an inability to preserve optimization experience, short scheduling time intervals, and difficulty coping with the high complexity of the problem space. Therefore, an optimization algorithm that can solve these problems is needed. Summary of the Invention

[0005] In order to solve the problems of traditional solutions based on swarm intelligence optimization algorithms, such as the inability to save optimization experience, short scheduling time intervals, and difficulty in coping with the high complexity of the problem space, the present invention provides a new multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning.

[0006] The present invention is achieved through the following technical solution: The present invention studies the use of reinforcement learning-related algorithms to solve the multi-objective uncertainty optimization scheduling problem of power systems, and provides a new multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning, including the following steps:

[0007] S1: A multi-objective two-stage robust optimization mathematical model for a wind power / photovoltaic / thermal power hybrid energy system was modeled. The optimization objectives in the mathematical model include minimizing economic costs, minimizing carbon emissions, and minimizing the active power loss of the power grid. Considering the uncertainty of wind power generation and photovoltaic power generation, an uncertainty set mathematical model was constructed as shown in Equations (1) and (2). Yes, the true value. is the predicted value, is an uncertainty:

[0008]

[0009]

[0010] The economic scheduling objectives for the first stage are shown in equations (3) to (7), where a i 、b i 、c i is the cost coefficient of the thermal power plant, is the cost coefficient of the wind power station, is the cost coefficient of the photovoltaic power station, m t is the electricity price at time t:

[0011]

[0012]

[0013]

[0014]

[0015] C=C T +C W +C PV +C ext (7)

[0016] The first-stage carbon emission scheduling target is shown in Equations (8) and (9), ρ i , τ i 、ω i , α i , β i is the calculation coefficient of carbon emissions, given by the specific thermal power plant;

[0017]

[0018]

[0019] The first-stage target of line active power loss is shown in formula (10):

[0020]

[0021] The overall goal of the first stage is expressed as formula (11):

[0022]

[0023] For the two-stage dispatching objectives, the two-stage objectives of economy, carbon emissions, and line loss are shown in Equations (12)-(13), where a new coefficient h is added. i The operation and maintenance cost coefficient adjusted in real time for the second stage;

[0024]

[0025]

[0026] The second-stage goal is to find the uncertainty u that maximizes the target in the uncertainty set U, and at the same time find the stage decision variable y that minimizes the target, as shown in formula (14):

[0027]

[0028] The constraints of the dispatching strategy include power balance constraints, maximum and minimum power constraints, and thermal power station power ramp constraints, as shown in Equations (15)-(17):

[0029]

[0030]

[0031]

[0032] Because the two-stage objective is a max-min form objective that is not easy to solve directly, the Lagrange multiplier formula is used to introduce the KKT condition to construct the Lagrange dual problem of the two-stage objective. At the same time, the weights (λ1, λ2, λ3) are introduced to weight the three objectives, and the final multi-objective two-stage robust optimization mathematical model is obtained, as shown in Equations (18)-(20).

[0033]

[0034]

[0035]

[0036] The optimization variable in one stage is still the daily / weekly / monthly scheduling strategy The two-stage scheduling strategy becomes the uncertainty u and the Lagrange multiplier γ 0 ,

[0037] S2: Solve the two-stage robust optimization problem by multi-agent reinforcement learning and solve the multi-objective optimization problem by using meta-reinforcement learning; by setting up two heterogeneous agents Agent1 and Agent2, Agent1 is responsible for minimizing the overall optimization goal, and Agent2 is responsible for maximizing the two-stage goal. Specifically, Agent1 receives the operating status information state of the current power grid at time t t , output the first-stage solution x t , Agent2 accepts state tand the first-stage solution x t , output the second-stage solution y t ,Agent2 makes the one-stage scheduling policy output by Agent1 more robust by affecting the reward function of Agent1;

[0038] S2-1: Agent 1 obtains the current state information s of the power grid t And output the one-stage scheduling strategy x t ;

[0039] S2-2: Agent 1 obtains the current state information s of the power grid t And the one-stage strategy x output by Agent1 t , output the second-stage uncertainty u and the Lagrange multiplier γ;

[0040] S2-3: The simulation environment accepts the actions output by the two agents, returns the rewards of the two agents at this moment, and enters the next state s t+1 ; Repeat S2-1 to S2-3 until t = T;

[0041] S2-4: Update the two agents based on the data generated in steps S2-1 to S2-3.

[0042] S3: Combining multi-agent reinforcement learning with meta-reinforcement learning to develop an end-to-end solution for multi-objective two-stage robust optimization of wind / photovoltaic / thermal hybrid energy systems. The details are as follows:

[0043] S3-1: Meta Learning stage:

[0044] S3-1-1: First, initialize a meta model, which includes the policy network and value network of Agent 1 and the policy network and value network of Agent 2;

[0045] S3-1-2: Then, a number of weights (λ1, λ2, λ3) are uniformly sampled and substituted into the mathematical model to construct a multi-agent reinforcement learning environment;

[0046] S3-1-3: Next, we use the Reptile algorithm framework for meta-learning. The IPPO algorithm is used for adversarial learning between multiple agents, and the PPO-Clip algorithm is used to update the policy network and value network of each agent.

[0047] S3-1-4: When the preset number of meta-learning rounds is reached, end meta-learning and return to metamodel;

[0048] S3-2: Fine Tuning stage:

[0049] S3-2-1: First, accept the meta model after meta learning and copy the meta model Z times;

[0050] S3-2-2: Then, a number of weights (λ1, λ2, λ3) are sampled at equal intervals and substituted into the mathematical model to construct a multi-agent reinforcement learning environment;

[0051] S3-2-3: Use the IPPO algorithm and PPO-Clip algorithm to fine-tune each copy of the meta model in a multi-agent reinforcement learning environment with different weights;

[0052] S3-2-4: After the number of fine-tuning rounds is reached, the fine-tuned model can be used to solve specific optimization scheduling problems and obtain the Pareto optimal solution;

[0053] S3-2-5: Use all the obtained Pareto optimal solutions to construct the Pareto frontier through the non-domination principle.

[0054] Compared with the prior art, the present invention has the following beneficial effects: the present invention provides a new multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning. ① Compared with the traditional multi-objective solution method based on swarm intelligence optimization algorithm, thanks to the reinforcement learning's focus on long-term rewards, it can handle scheduling tasks with longer time steps. The scheduling time of the traditional method is generally at the daily level (24 hours), while this method can handle scheduling tasks at the weekly level (7*24) or even the monthly level (30*24); ② Thanks to the powerful nonlinear processing capability of the deep neural network, this method can handle nonlinear information in complex power systems compared with traditional methods, which makes the method less likely to fall into local optimality, thereby obtaining a better Pareto optimal solution; ③ Thanks to the learning ability of the neural network, the method can save optimization experience, which makes it possible to directly use the trained neural network model for the new time period, instead of re-running the optimization program like the traditional algorithm, which will greatly save computing cost and computing time in actual production applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 FIG. 1 is a schematic diagram of the IEEE structure of a specific embodiment of the present invention.

[0056] Figure 2 Schematic diagram of solving a two-stage robust optimization problem for multi-agent reinforcement learning.

[0057] Figure 3 Flowchart for solving multi-objective optimization problems using meta-reinforcement learning. DETAILED DESCRIPTION

[0058] The present invention will be further described below with reference to specific embodiments.

[0059] This embodiment uses the IEEE 39-node example for verification, in which 2 wind power stations, 3 photovoltaic power stations, 4 thermal power stations and 21 loads are set up. Figure 1 As shown in Figure 1, a new multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning includes the following steps:

[0060] S1: A multi-objective two-stage robust optimization mathematical model for a wind power / photovoltaic / thermal power hybrid energy system was modeled. The optimization objectives in the mathematical model include minimizing economic costs, minimizing carbon emissions, and minimizing the active power loss of the power grid. Considering the uncertainty of wind power generation and photovoltaic power generation, an uncertainty set mathematical model was constructed as shown in Equations (1) and (2). Yes, the true value. is the predicted value, is an uncertainty:

[0061]

[0062]

[0063] The economic scheduling objectives for the first stage are shown in equations (3) to (7), where a i 、b i 、c i is the cost coefficient of the thermal power plant, is the cost coefficient of the wind power station, is the cost coefficient of the photovoltaic power station, m t is the electricity price at time t:

[0064]

[0065]

[0066]

[0067]

[0068] C=C T +C W +C PV +C ext (7)

[0069] The first-stage carbon emission scheduling target is shown in Equations (8) and (9), ρ i , τ i 、ω i , α i , βi is the calculation coefficient of carbon emissions, given by the specific thermal power plant;

[0070]

[0071]

[0072] The first-stage target of line active power loss is shown in formula (10):

[0073]

[0074] The overall goal of the first stage is expressed as formula (11):

[0075]

[0076] For the two-stage dispatching objectives, the two-stage objectives of economy, carbon emissions, and line loss are shown in Equations (12)-(13), where a new coefficient h is added. i The operation and maintenance cost coefficient adjusted in real time for the second stage;

[0077]

[0078]

[0079] The second-stage goal is to find the uncertainty u that maximizes the target in the uncertainty set U, and at the same time find the stage decision variable y that minimizes the target, as shown in formula (14):

[0080]

[0081] The constraints of the dispatching strategy include power balance constraints, maximum and minimum power constraints, and thermal power station power ramp constraints, as shown in Equations (15)-(17):

[0082]

[0083]

[0084]

[0085] Because the two-stage objective is a max-min form objective that is not easy to solve directly, the Lagrange multiplier formula is used to introduce the KKT condition to construct the Lagrange dual problem of the two-stage objective. The weights (λ1, λ2, λ3) are then introduced to weight the three objectives, and the final multi-objective two-stage robust optimization mathematical model is obtained, as shown in Equations (18)-(20).

[0086]

[0087]

[0088]

[0089] The optimization variable in one stage is still the daily / weekly / monthly scheduling strategy The two-stage scheduling strategy becomes the uncertainty u and the Lagrange multiplier γ 0 ,

[0090] S2: Solve the two-stage robust optimization problem by multi-agent reinforcement learning and solve the multi-objective optimization problem by using meta-reinforcement learning; by setting up two heterogeneous agents Agent1 and Agent2, Agent1 is responsible for minimizing the overall optimization goal, and Agent2 is responsible for maximizing the two-stage goal. Specifically, Agent1 receives the operating status information state of the current power grid at time t t , output the first-stage solution x t , Agent2 accepts state t and the first-stage solution x t , output the second-stage solution y t ,like Figure 2 As shown in Figure 2, Agent2 makes the one-stage scheduling policy output by Agent1 more robust by affecting the reward function of Agent1;

[0091] S2-1: Agent 1 obtains the current state information s of the power grid t And output the one-stage scheduling strategy x t ;

[0092] S2-2: Agent 1 obtains the current state information s of the power grid t And the one-stage strategy x output by Agent1 t , output the second-stage uncertainty u and the Lagrange multiplier γ;

[0093] S2-3: The simulation environment accepts the actions output by the two agents, returns the rewards of the two agents at this moment, and enters the next state s t+1 ; Repeat S2-1 to S2-3 until t = T;

[0094] S2-4: Update the two agents based on the data generated in steps S2-1 to S2-3.

[0095] S3: Combine multi-agent reinforcement learning with meta-reinforcement learning to form an end-to-end solution for multi-objective two-stage robust optimization of wind power / photovoltaic / thermal power hybrid energy systems. The process is as follows: Figure 3 As shown, the details are as follows:

[0096] S3-1: Meta Learning stage:

[0097] S3-1-1: First, initialize a meta model, which includes the policy network and value network of Agent 1 and the policy network and value network of Agent 2;

[0098] S3-1-2: Then, a number of weights (λ1, λ2, λ3) are uniformly sampled and substituted into the mathematical model to construct a multi-agent reinforcement learning environment;

[0099] S3-1-3: Next, we use the Reptile algorithm framework for meta-learning. The IPPO algorithm is used for adversarial learning between multiple agents, and the PPO-Clip algorithm is used to update the policy network and value network of each agent.

[0100] S3-1-4: When the preset number of meta-learning rounds is reached, end meta-learning and return to metamodel;

[0101] S3-2: Fine Tuning stage:

[0102] S3-2-1: First, accept the meta model after meta learning and copy the meta model Z times;

[0103] S3-2-2: Then, a number of weights (λ1, λ2, λ3) are sampled at equal intervals and substituted into the mathematical model to construct a multi-agent reinforcement learning environment;

[0104] S3-2-3: Use the IPPO algorithm and PPO-Clip algorithm to fine-tune each copy of the meta model in a multi-agent reinforcement learning environment with different weights;

[0105] S3-2-4: After the number of fine-tuning rounds is reached, the fine-tuned model can be used to solve specific optimization scheduling problems and obtain the Pareto optimal solution;

[0106] S3-2-5: Use all the obtained Pareto optimal solutions to construct the Pareto frontier through the non-domination principle.

[0107] The scope of protection claimed by the present invention is not limited to the above specific embodiments. For those skilled in the art, the present invention may have various variations and modifications. Any modifications, improvements and equivalent substitutions made within the concept and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A novel multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning, characterized by: The steps include: S1: A multi-objective two-stage robust optimization mathematical model for a wind power / photovoltaic / thermal power hybrid energy system was modeled; The multi-objective two-stage robust optimization mathematical model is as follows: S2: Solve the two-stage robust optimization problem through multi-agent reinforcement learning and the multi-objective optimization problem through meta-reinforcement learning; the details are as follows: By setting up two heterogeneous agents Agent1 and Agent2, Agent1 is responsible for minimizing the overall optimization goal, and Agent2 is responsible for maximizing the second-stage goal. Specifically, Agent1 receives the operating status information of the power grid at the current time t, state t , output the first-stage solution x t , Agent2 accepts state t and the first-stage solution x t , output the second-stage solution y t ,Agent2 makes the one-stage scheduling policy output by Agent1 more robust by affecting the reward function of Agent1; S2-1: Agent 1 obtains the current state information s of the power grid t And output the one-stage scheduling strategy x t ; S2-2: Agent 1 obtains the current state information s of the power grid t And the one-stage strategy x output by Agent1 t , output the second-stage uncertainty u and the Lagrange multiplier γ; S2-3: The simulation environment accepts the actions output by the two agents, returns the rewards of the two agents at this moment, and enters the next state s t+1 ; Repeat S2-1 to S2-3 until t = T; S2-4: Update the two agents based on the data generated in steps S2-1 to S2-3; S3: Combining multi-agent reinforcement learning with meta-reinforcement learning to develop an end-to-end solution for multi-objective two-stage robust optimization of wind / photovoltaic / thermal hybrid energy systems; the details are as follows: S3-1: Meta Learning stage: S3-1-1: First, initialize a meta model, which includes the policy network and value network of Agent 1 and the policy network and value network of Agent 2; S3-1-2: Then, a number of weights λ1, λ2, λ3 are uniformly sampled and substituted into the mathematical model to construct a multi-agent reinforcement learning environment; S3-1-3: Next, we use the Reptile algorithm framework for meta-learning. The IPPO algorithm is used for adversarial learning between multiple agents, and the PPO-Clip algorithm is used to update the policy network and value network of each agent. S3-1-4: When the preset number of meta learning rounds is reached, meta learning ends and returns to the meta model; S3-2: Fine Tuning stage: S3-2-1: First, accept the meta model after meta learning and copy the meta model Z times; S3-2-2: Then, sample several weights λ1, λ2, λ3 at equal intervals and substitute them into the mathematical model to construct a multi-agent reinforcement learning environment; S3-2-3: Use the IPPO algorithm and PPO-Clip algorithm to fine-tune each copy of the meta model in a multi-agent reinforcement learning environment with different weights; S3-2-4: After the number of fine-tuning rounds is reached, the fine-tuned model can be used to solve specific optimization scheduling problems and obtain the Pareto optimal solution; S3-2-5: Use all the obtained Pareto optimal solutions to construct the Pareto frontier through the non-domination principle.

2. The novel multi-objective robust optimization method for power systems based on meta-multi-agent deep reinforcement learning according to claim 1 is characterized by: Step S1 is specifically as follows: The optimization objectives in the mathematical model include minimizing economic costs, minimizing carbon emissions, and minimizing the active power loss of the power grid. Considering the uncertainty of wind power generation and photovoltaic power generation, the uncertainty set mathematical model is constructed as shown in equations (1) and (2). Yes, the true value. is the predicted value, is an uncertainty: The economic scheduling objectives for the first stage are shown in equations (3) to (7), where a i 、b i 、c i is the cost coefficient of the thermal power plant, is the cost coefficient of the wind power station, is the cost coefficient of the photovoltaic power station, m t is the electricity price at time t: C=C T +C W +C PV +C ext (7) The first-stage carbon emission scheduling target is shown in Equations (8) and (9), ρ i , τ i 、ω i , α i , β i is the calculation coefficient of carbon emissions: The first-stage target of line active power loss is shown in formula (10): The overall goal of the first stage is expressed as formula (11): For the two-stage dispatching objectives, the two-stage objectives of economy, carbon emissions, and line loss are shown in Equations (12)-(13), where a new coefficient h is added. i The operation and maintenance cost coefficient adjusted in real time for the second stage; The second-stage goal is to find the uncertainty u that maximizes the target in the uncertainty set U, and at the same time find the stage decision variable y that minimizes the target, as shown in formula (14): The constraints of the dispatching strategy include power balance constraints, maximum and minimum power constraints, and thermal power station power ramp constraints, as shown in Equations (15)-(17): Through the Lagrange multiplier formula, the KKT condition is introduced to construct the Lagrange dual problem of the two-stage objectives. At the same time, the weights λ1, λ2, and λ3 are introduced to weight the three objectives, and the final multi-objective two-stage robust optimization mathematical model is obtained, as shown in Equations (18)-(20); The optimization variable in one stage is still the daily / weekly / monthly scheduling strategy The two-stage scheduling strategy becomes the uncertainty u and the Lagrange multiplier γ 0 ,

Images