Multi-agent reinforcement learning scheduling method based on optimality guarantee

Abstract

The invention belongs to the technical field of power grids, and particularly relates to a multi-agent reinforcement learning scheduling method based on optimality assurance, which simulates the actual operation condition of a power grid by establishing a power grid simulation system based on a mathematical model, and provides a multi-agent reinforcement learning scheduling method based on optimality assurance according to the requirements and requirements of power grid scheduling. Constructing a scheduling target and a constraint to guide the intelligent agent to make a proper decision in the learning process; a state space, an action space and a reward function are constructed for each agent, a reinforcement learning algorithm is combined with a simulation environment to train a strategy of the agent, and in the training process, the agent continuously optimizes the strategy through interaction of the simulation environment, so that the optimal scheduling effect is achieved; moreover, the intelligent agent can sense the dynamic change of the power grid in real time, and carries out the real-time adjustment and optimization of the scheduling scheme according to the real-time information, thereby guaranteeing the timeliness and accuracy of the power grid scheduling, and effectively solving a problem that a conventional scheduling method is lack of flexibility and real-time performance.

Description

Technical Field

[0001] The present application belongs to the field of power grid technology, and more specifically, to a multi-agent reinforcement learning scheduling method based on optimality guarantee. Background Art

[0002] As one of the important infrastructures of modern society, the operation status of the power system is directly related to economic development and social stability. In the power system, grid dispatching is a key link to ensure the safe, stable and efficient operation of the power system. However, with the continuous expansion of the scale of the power system and the growth of electricity demand, the traditional grid dispatching method has been unable to meet the requirements of complex and large-scale power grids.

[0003] At present, the technical solutions for optimizing power grid dispatch mainly include rule-based static dispatch methods and optimization algorithms based on mathematical models. Rule-based static dispatch methods often rely on experience and fixed rules, such as formulating dispatch plans according to load forecast results, or setting thresholds based on historical data for safety control.

[0004] The static dispatch method based on rules has certain limitations in dealing with the large scale and complexity of power grids, and often cannot achieve the optimal dispatch effect. Although the optimization algorithm based on mathematical models can provide more accurate dispatch solutions, the computational complexity is high and it is difficult to quickly solve in a real-time environment.

[0005] Traditional dispatching methods often lack flexibility and real-time performance, and cannot effectively cope with the dynamic changes in the power grid's operating status. In power systems, real-time performance is crucial because any delay or error may lead to grid instability or even accidents. Summary of the invention

[0006] The present invention provides a multi-agent reinforcement learning scheduling method based on optimality guarantee, which is intended to solve the technical problem that the current static scheduling method is limited in processing large-scale and complex data of power grids.

[0007] The multi-agent reinforcement learning scheduling method based on optimality guarantee includes the following steps:

[0008] Step 1: Obtain the relationship and parameter data between the grid topology and various elements in the grid, and establish a grid simulation system based on a mathematical model based on the acquired data;

[0009] Step 2: Construct scheduling goals and constraints to guide the decision-making of the agent during learning;

[0010] Step 3: Construct the state space, action space and reward function of each intelligent agent to realize the perception, decision-making and learning of the power grid status;

[0011] Step 4: Use reinforcement learning algorithm to train the agent's strategy based on the constructed scheduling objectives, constraints, agent's state space, action space and reward function, combined with the simulation environment;

[0012] Step 5: Generate a real-time dispatch plan based on the trained intelligent agent strategy, and adjust and optimize the dispatch plan in real time based on the dynamic changes of the power grid.

[0013] The present invention establishes a power grid simulation system based on a mathematical model to simulate the actual operation of the power grid, and constructs scheduling goals and constraints to guide the intelligent agent to make appropriate decisions during the learning process according to the needs and requirements of the power grid scheduling; then for each intelligent agent, a state space, an action space and a reward function are constructed, and a reinforcement learning algorithm is used in combination with a simulation environment to train the strategy of the intelligent agent. During the training process, the intelligent agent continuously optimizes the strategy through interaction with the simulation environment to achieve the optimal scheduling effect; and the intelligent agent can perceive the dynamic changes of the power grid in real time, and adjust and optimize the scheduling plan in real time according to the real-time information, thereby ensuring the timeliness and accuracy of the power grid scheduling, and effectively solving the problem of the lack of flexibility and real-time performance of the traditional scheduling method.

[0014] Preferably, the step 1 comprises the following steps:

[0015] Step 1.1: Obtain the topological structure of the power grid and the relationship and parameter data between the various elements in the power grid, organize the obtained power grid data into the form of nodes and edges, where the nodes represent the various devices in the power grid, and the nodes represent the connection relationship between the devices, and determine the direct connection relationship between the various nodes according to the topological structure;

[0016] Step 1.2: Based on the power flow calculation method, establish the node power flow equation to describe the power balance relationship between each node in the power grid;

[0017] Step 1.3: Integrate the node power flow equations into a nonlinear equation system and construct a power flow calculation model;

[0018] Step 1.4: Construct the software implementation of the power grid simulation system based on the established power flow calculation model.

[0019] Preferably, the node power flow equation is as follows:

[0020]

[0021] Where: P i and Q i are the active power and reactive power of the ith node respectively; V i represents the voltage amplitude of the ith node; V j represents the voltage amplitude of the jth node; θ irepresents the voltage phase angle of the ith node; G ij and B ij are the real and imaginary parts of the admittance between the i-th and j-th nodes respectively; θ j represents the voltage phase angle of the jth node; N represents the total number of nodes in the power grid.

[0022] Preferably, the nonlinear equations are as follows:

[0023] F(V,θ)=0;

[0024] Where: V represents the voltage amplitude vector of all nodes in the power grid; θ represents the voltage phase angle vector of all nodes in the power grid.

[0025] Preferably, the scheduling target is the minimum cost, and the objective function is as follows:

[0026]

[0027] Where: w1, w2 and w3 represent the weights of various costs; V gen represents the cost of power generation; C loss represents the transmission loss cost; C imbalance represents the load unbalance cost; V represents the node voltage vector; θ represents the node phase angle vector; P g Represents the generator active power vector; Q g represents the generator reactive power vector.

[0028] Preferably, the constraints include:

[0029] Voltage stability limit:

[0030]

[0031] Where: V min and V max Indicates the minimum and maximum voltage amplitude allowed for each node; V i Represents the voltage amplitude of node i; Transmission line capacity limitation:

[0032]

[0033] Where: P l and Q l Respectively represent the active and reactive power flows on each transmission line; S l Indicates the rated capacity of the transmission line;

[0034] Power balance constraints:

[0035] P g -P d -P loss =0;

[0036] Q g -Q d =0;

[0037] Where: P g Represents the total active power generated by all generators; P d Indicates the total active power consumed by all loads; Q g Represents the total reactive power generated by all generators; Q d Represents the total reactive power consumed by all loads; P loss Represents the total active power loss in the transmission line.

[0038] Preferably, step 3 comprises the following steps:

[0039] Step 3.1: Represent the state of the power grid as a vector and represent the state of the power grid as the state space of the agent;

[0040] Step 3.2: Represent the scheduling decision action of each agent with a vector, and represent the vector as the action space of the agent;

[0041] Step 3.3: With cost minimization as the objective of the reward function, reward the agent for taking actions that reduce cost.

[0042] Preferably, the step 3.1 comprises the following steps:

[0043] For each time step, the voltage amplitude and phase angle of all nodes in the power grid are obtained;

[0044] The voltage amplitude and the phase angle are arranged in order to form a state vector; and the formed state vector is used as the state space of the intelligent agent.

[0045] Preferably, the step 3.2 comprises the following steps:

[0046] For each agent, an action is chosen at each time step, which corresponds to an action vector;

[0047] Each element in the action vector represents a scheduling parameter, and the scheduling parameters are arranged in order to form an action vector.

[0048] The beneficial effects of the present invention include:

[0049] The present invention establishes a power grid simulation system based on a mathematical model to simulate the actual operation of the power grid, and constructs scheduling goals and constraints to guide the intelligent agent to make appropriate decisions during the learning process according to the needs and requirements of the power grid scheduling; then for each intelligent agent, a state space, an action space and a reward function are constructed, and a reinforcement learning algorithm is used in combination with a simulation environment to train the strategy of the intelligent agent. During the training process, the intelligent agent continuously optimizes the strategy through interaction with the simulation environment to achieve the optimal scheduling effect; and the intelligent agent can perceive the dynamic changes of the power grid in real time, and adjust and optimize the scheduling plan in real time according to the real-time information, thereby ensuring the timeliness and accuracy of the power grid scheduling, and effectively solving the problem of the lack of flexibility and real-time performance of the traditional scheduling method. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0051] Figure 1 A schematic diagram of the overall steps provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0053] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0054] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0055] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0056] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0057] The multi-agent reinforcement learning scheduling method based on optimality guarantee includes the following steps:

[0058] Step 1: Obtain the relationship and parameter data between the grid topology and various elements in the grid, and establish a grid simulation system based on a mathematical model based on the acquired data;

[0059] The step 1 comprises the following steps:

[0060] Step 1.1: Obtain the topological structure of the power grid and the relationship and parameter data between the various elements in the power grid, organize the obtained power grid data into the form of nodes and edges, where the nodes represent the various devices in the power grid, and the nodes represent the connection relationship between the devices, and determine the direct connection relationship between the various nodes according to the topological structure;

[0061] The topological structure information of the power grid can be obtained through existing power grid drawings, geographic information system (GIS) data or field surveys.

[0062] The grid topology information is represented in the form of a graph, where nodes represent devices in the grid (such as generators, transformers, loads, etc.) and edges represent the connection relationships between devices (such as transmission lines).

[0063] Collect the relationship and parameter data between various elements in the power grid, including but not limited to:

[0064] Parameters of transmission lines such as resistance, reactance, capacitance, etc.;

[0065] Generator's maximum output, minimum output, start time, stop time and other information;

[0066] Transformer ratio, rated capacity, loss and other information;

[0067] Information such as load consumption characteristics and demand curve.

[0068] Step 1.2: Based on the power flow calculation method, establish the node power flow equation to describe the power balance relationship between each node in the power grid;

[0069] The node power flow equation is as follows:

[0070]

[0071] Where: P i and Q i are the active power and reactive power of the ith node respectively; V i represents the voltage amplitude of the ith node; V j represents the voltage amplitude of the jth node; θ i represents the voltage phase angle of the ith node; G ij and B ij are the real and imaginary parts of the admittance between the i-th and j-th nodes respectively; θ j represents the voltage phase angle of the jth node; N represents the total number of nodes in the power grid.

[0072] Step 1.3: Integrate the node power flow equations into a nonlinear equation system and construct a power flow calculation model;

[0073] The nonlinear equations are as follows:

[0074] F(V,θ)=0;

[0075] Where: V represents the voltage amplitude vector of all nodes in the power grid; θ represents the voltage phase angle vector of all nodes in the power grid.

[0076] The specific expression of the nonlinear equations is as follows:

[0077]

[0078] Where: F i (V,θ) represents the node power flow equation of the i-th node; it represents the relationship between the active power and reactive power of the node and the voltage and admittance parameters between the adjacent nodes.

[0079] Step 1.4: Construct the software implementation of the power grid simulation system based on the established power flow calculation model.

[0080] Step 2: Construct scheduling goals and constraints to guide the decision-making of the agent during learning;

[0081] The scheduling objective is to minimize the cost, and the objective function is as follows:

[0082]

[0083] Where: w1, w2 and w3 represent the weights of various costs; C gen represents the cost of power generation; C loss represents the transmission loss cost; C imbalancerepresents the load unbalance cost; V represents the node voltage vector; θ represents the node phase angle vector; P g Represents the generator active power vector; Q g represents the generator reactive power vector.

[0084] The constraints include:

[0085] Voltage stability limit:

[0086]

[0087] Where: V min and V max Indicates the minimum and maximum voltage amplitude allowed for each node; V i Represents the voltage amplitude of node i; Transmission line capacity limitation:

[0088]

[0089] Where: P l and Q l Respectively represent the active and reactive power flows on each transmission line; S l Indicates the rated capacity of the transmission line;

[0090] Power balance constraints:

[0091] P g -P d -P loss =0;

[0092] Q g -Q d =0;

[0093] Where: P g Represents the total active power generated by all generators; P d Indicates the total active power consumed by all loads; Q g Represents the total reactive power generated by all generators; Q d Represents the total reactive power consumed by all loads; P loss Represents the total active power loss in the transmission line.

[0094] Step 3: Construct the state space, action space and reward function of each intelligent agent to realize the perception, decision-making and learning of the power grid status;

[0095] The step 3 comprises the following steps:

[0096] Step 3.1: Represent the state of the power grid as a vector and represent the state of the power grid as the state space of the agent;

[0097] Step 3.2: Represent the scheduling decision action of each agent with a vector, and represent the vector as the action space of the agent;

[0098] Step 3.3: With cost minimization as the objective of the reward function, reward the agent for taking actions that reduce cost.

[0099] The step 3.1 comprises the following steps:

[0100] For each time step, the voltage amplitude and phase angle of all nodes in the power grid are obtained;

[0101] The voltage amplitude and the phase angle are arranged in order to form a state vector; and the formed state vector is used as the state space of the intelligent agent.

[0102] The state vector can be expressed as:

[0103] S=[V1,θ1,V2,θ1,...,V N ,θ N ];

[0104] Where: V i represents the voltage amplitude of the i-th node; θ i represents the voltage phase angle of the i-th node;

[0105] The step 3.2 comprises the following steps:

[0106] For each agent, an action is chosen at each time step, which corresponds to an action vector;

[0107] The action vector can be expressed as:

[0108] a=[a1,a2,…,a m ];

[0109] In the formula, m represents the length of the action vector, a i represents the i-th scheduling parameter.

[0110] Each element in the action vector represents a scheduling parameter, and the scheduling parameters are arranged in order to form an action vector.

[0111] The reward function is as follows:

[0112] R(α)=-(w1·C gen +w2·C loss +w3·C imbalance );

[0113] Where: R(a) represents the reward value calculated based on the action a taken by the agent.

[0114] Step 4: Use reinforcement learning algorithm to train the agent's strategy based on the constructed scheduling objectives, constraints, agent's state space, action space and reward function, combined with the simulation environment;

[0115] Specifically, a policy gradient method is adopted as the reinforcement learning algorithm to train the strategy of the agent according to the current power grid state, the state space and action space of the agent, and the reward function.

[0116] Through interaction with the simulation environment, the agent will gradually optimize its decision-making strategy according to the guidance of the reward function, with the goal of minimizing the total cost.

[0117] The policy gradient update formula is as follows:

[0118]

[0119] Where: θ represents the parameters of the agent strategy; J(θ) represents the expected return of the strategy; α represents the learning rate; represents the policy gradient.

[0120] In each training cycle, the agent interacts with the simulation environment multiple times to collect experience data;

[0121] Use experience data to update the agent's strategy and optimize it gradually;

[0122] Training continues until a certain number of iterations is reached or a stopping condition is met.

[0123] Step 5: Generate a real-time dispatch plan based on the trained intelligent agent strategy, and adjust and optimize the dispatch plan in real time based on the dynamic changes of the power grid.

[0124] Generate real-time scheduling plan:

[0125] Using the trained agent strategy, a preliminary real-time dispatching plan is generated according to the current grid status. This includes determining dispatching parameters such as the generator output of each node and the switch status of the line to achieve the dispatching target under the current grid status.

[0126] Real-time adjustments and optimizations:

[0127] Monitor the actual operating status of the power grid: The monitoring system collects the operating data of the power grid in real time, including the voltage and power flow of each node, to reflect the actual operating status of the power grid in real time.

[0128] Agent decision-making: The current power grid status is input into the trained agent model. The agent makes decision inferences based on the current status and its learned strategies to generate a new scheduling plan.

[0129] Implementation of the dispatching plan: Apply the new dispatching plan generated by the intelligent agent to the actual power grid, that is, adjust the generator output, control the line switch status, etc., to achieve dynamic adjustment of the power grid.

[0130] Real-time feedback and optimization: Based on the implementation effect of the dispatching plan and the real-time feedback information of the power grid, the performance of the current dispatching plan is evaluated, including indicators such as cost and stability. If necessary, the feedback information will be used to adjust the strategy of the intelligent agent to optimize the real-time performance and effect of the dispatching plan.

[0131] Through the above steps, it is possible to generate real-time dispatching plans based on the trained intelligent agent strategy, and make real-time adjustments and optimizations under the dynamic changes of the power grid, thereby ensuring the flexibility, real-time and efficiency of power grid dispatching, and effectively solving the limitations of traditional dispatching methods.

[0132] The present invention establishes a power grid simulation system based on a mathematical model to simulate the actual operation of the power grid, and constructs scheduling goals and constraints to guide the intelligent agent to make appropriate decisions during the learning process according to the needs and requirements of the power grid scheduling; then for each intelligent agent, a state space, an action space and a reward function are constructed, and a reinforcement learning algorithm is used in combination with a simulation environment to train the strategy of the intelligent agent. During the training process, the intelligent agent continuously optimizes the strategy through interaction with the simulation environment to achieve the optimal scheduling effect; and the intelligent agent can perceive the dynamic changes of the power grid in real time, and adjust and optimize the scheduling plan in real time according to the real-time information, thereby ensuring the timeliness and accuracy of the power grid scheduling, and effectively solving the problem of the lack of flexibility and real-time performance of the traditional scheduling method.

[0133] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-agent reinforcement learning scheduling method based on optimality guarantee, characterized in that: The following steps are involved: Step 1: Obtain the relationship and parameter data between the grid topology and various elements in the grid, and establish a grid simulation system based on a mathematical model based on the acquired data; Step 2: Construct scheduling goals and constraints to guide the decision-making of the agent during learning; Step 3: Construct the state space, action space and reward function of each intelligent agent to realize the perception, decision-making and learning of the power grid status; Step 4: Use reinforcement learning algorithm to train the agent's strategy based on the constructed scheduling objectives, constraints, agent's state space, action space and reward function, combined with the simulation environment; Step 5: Generate a real-time dispatch plan based on the trained intelligent agent strategy, and adjust and optimize the dispatch plan in real time based on the dynamic changes of the power grid.

2. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 1, characterized in that: The step 1 comprises the following steps: Step 1.1: Obtain the topological structure of the power grid and the relationship and parameter data between the various elements in the power grid, organize the obtained power grid data into the form of nodes and edges, where the nodes represent the various devices in the power grid, and the nodes represent the connection relationship between the devices, and determine the direct connection relationship between the various nodes according to the topological structure; Step 1.2: Based on the power flow calculation method, establish the node power flow equation to describe the power balance relationship between each node in the power grid; Step 1.3: Integrate the node power flow equations into a nonlinear equation system and construct a power flow calculation model; Step 1.4: Construct the software implementation of the power grid simulation system based on the established power flow calculation model.

3. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 2 is characterized in that: The node power flow equation is as follows: Where: P i and Q i are the active power and reactive power of the ith node respectively; V i represents the voltage amplitude of the ith node; V j represents the voltage amplitude of the jth node; θ i represents the voltage phase angle of the ith node; G ij and B ij are the real and imaginary parts of the admittance between the i-th and j-th nodes respectively; θ j represents the voltage phase angle of the jth node; N represents the total number of nodes in the power grid.

4. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 2 is characterized in that: The nonlinear equations are as follows: F(V,θ)=0; Where: V represents the voltage amplitude vector of all nodes in the power grid; θ represents the voltage phase angle vector of all nodes in the power grid.

5. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 1, characterized in that: The scheduling objective is to minimize the cost, and the objective function is as follows: Where: w1, w2 and w3 represent the weights of various costs; C gen represents the cost of power generation; C loss represents the transmission loss cost; C imbalance represents the load unbalance cost; V represents the node voltage vector; θ represents the node phase angle vector; P g Represents the generator active power vector; Q g represents the generator reactive power vector.

6. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 1, characterized in that: The constraints include: Voltage stability limit: Where: V min and V max Indicates the minimum and maximum voltage amplitude allowed for each node; V i represents the voltage amplitude of node i; Transmission line capacity limitations: Where: P l and Q l Respectively represent the active and reactive power flows on each transmission line; S l Indicates the rated capacity of the transmission line; Power balance constraints: P g -P d -P loss =0; Q g -Q d =0; Where: P g Represents the total active power generated by all generators; P d Indicates the total active power consumed by all loads; Q g Represents the total reactive power generated by all generators; Q d Represents the total reactive power consumed by all loads; P loss Represents the total active power loss in the transmission line.

7. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 1, characterized in that: The step 3 comprises the following steps: Step 3.1: Represent the state of the power grid as a vector and represent the state of the power grid as the state space of the agent; Step 3.2: Represent the scheduling decision action of each agent with a vector, and represent the vector as the action space of the agent; Step 3.3: With cost minimization as the objective of the reward function, reward the agent for taking actions that reduce cost.

8. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 7 is characterized in that: The step 3.1 comprises the following steps: For each time step, the voltage amplitude and phase angle of all nodes in the power grid are obtained; The voltage amplitude and the phase angle are arranged in order to form a state vector; and the formed state vector is used as the state space of the intelligent agent.

9. The multi-agent reinforcement learning scheduling method based on optimality guarantee according to claim 7, characterized in that: The step 3.2 comprises the following steps: For each agent, an action is chosen at each time step, which corresponds to an action vector; Each element in the action vector represents a scheduling parameter, and the scheduling parameters are arranged in order to form an action vector.

Images