Multi-agent reinforcement learning method based on operation cost optimization

Abstract

The invention belongs to the technical field of power systems, and relates to a multi-agent reinforcement learning method based on operation cost optimization, a power grid system is divided into a plurality of areas, each area corresponds to an independent agent, a complex global problem is converted into a plurality of local problems, and the feasibility of decision making is improved; establishing a cost function for each agent to comprehensively describe the cost influence caused by the agent executing a certain action; a deep neural network is adopted as a decision model of each agent, and the agents can be helped to learn an optimal action decision strategy through interaction with the environment; a reinforcement learning framework based on a multi-agent evolution strategy algorithm is constructed, and each agent learns an optimal action strategy according to the state of the agent and decisions of other agents; optimally, a method based on the game theory is adopted to further coordinate action decisions of all agents, and it is ensured that the system is converged to a global optimal state; and the overall operation efficiency and economy of the power grid system are improved.

Description

Technical Field

[0001] The present application belongs to the technical field of power systems, and more specifically, to a multi-agent reinforcement learning method based on operation cost optimization. Background Art

[0002] With the development of smart grids, how to improve the operating efficiency and reliability of power systems has become one of the key issues. Power systems are usually composed of many power plants, transmission lines, substations, etc., forming a complex network system. During the operation of the system, it is necessary to coordinate the power generation, dispatching, distribution and other decisions of each node to meet the power demand and minimize the overall operating cost.

[0003] Traditional power grid dispatching methods mainly rely on manual experience to make decisions, which is difficult to adapt to the increasingly complex power system environment. In recent years, reinforcement learning technology has shown good application prospects in power system optimization. Reinforcement learning can automatically learn the optimal decision-making strategy through interaction with the environment, thereby improving the operation efficiency of the power grid.

[0004] However, there are multiple independent decision-making entities in the power system, such as power plants, transmission and distribution companies, etc., each of which has its own goals and decision-making methods. How to achieve the global optimal dispatch of the power grid system by coordinating and optimizing the decisions of each entity in a multi-agent environment has become a key issue that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a multi-agent reinforcement learning method based on operation cost optimization, which aims to achieve optimal scheduling of the power grid system at the lowest cost by coordinating and optimizing the decision of each subject in a multi-agent environment.

[0006] A multi-agent reinforcement learning method based on operation cost optimization includes the following steps:

[0007] Step 1: Divide the entire power grid system into multiple areas, each area corresponds to an intelligent agent; define the state vector of each intelligent agent, determine the actions that each intelligent agent can take, and establish a cost function for each intelligent agent to describe the cost of executing the action;

[0008] Step 2: Use a deep neural network as the decision model for each agent, and use a deep reinforcement learning algorithm to train the decision model of each agent, and output the action decision of each agent through the decision model;

[0009] Step 3: Construct a multi-agent reinforcement learning framework based on the multi-agent evolutionary strategy algorithm, define an independent reward function for each agent, and establish information exchange and coordinated decision-making between agents. Through the multi-agent reinforcement learning framework, optimize the action decision of each agent to minimize the cost;

[0010] Step 4: Coordinate the action decisions of each intelligent agent to achieve the global optimum based on the game theory method; output the global optimal action decision, and each intelligent agent performs the corresponding action based on the output action decision.

[0011] The present invention divides the power grid system into multiple areas, each area corresponds to an independent intelligent agent, converts complex global problems into multiple local problems, and improves the feasibility of decision-making; establishes a cost function for each intelligent agent to comprehensively describe the cost impact of the intelligent agent performing a certain action; and adopts a deep neural network as the decision model of each intelligent agent, and uses a deep reinforcement learning algorithm to train the decision model; through interaction with the environment, the intelligent agent can help learn the optimal action decision strategy and gradually reduce its own operating cost; and constructs a reinforcement learning framework based on a multi-agent evolutionary strategy algorithm, constructs a separate reward function for each intelligent agent, so that it can learn the optimal action strategy according to its own state and the decision of other intelligent agents; optimally adopts a game theory-based method to further coordinate the action decisions of each intelligent agent to ensure that the system converges to the global optimal state; improves the overall operating efficiency and economy of the power grid system.

[0012] Preferably, the constructed cost function is as follows:

[0013]

[0014] Where: w 1 、w 2 and w 3 is the corresponding weight coefficient, C i (s i ,A i ) represents the total cost function of agent i; represents the power generation cost function of agent i; represents the grid loss cost function of agent i; represents the emission cost function of agent i, that is, the environmental cost caused by the pollutant emissions generated during the power generation process; S i Indicates the state of agent i; A i Represents the initial decision of agent i.

[0015] Preferably, the structure of the decision model is as follows:

[0016] Input layer: The input is the state vector of the agent;

[0017] Hidden layer: The decision model adopts a multi-layer perceptron structure, which consists of multiple fully connected hidden layers; each fully connected hidden layer uses the ReLU activation function as the activation function;

[0018] Output layer: The output layer outputs the actions that the agent can take; if the output of the agent is a discrete action, the output layer uses a fully connected layer + Softmax activation structure to output the probability distribution of each action; if the output of the agent is a continuous action, a fully connected layer + Tanh activation is used to output the certainty value of the action; for the output probability distribution, the action with the highest probability is selected.

[0019] Preferably, the training steps of the multi-agent reinforcement learning framework are as follows:

[0020] a. Each agent outputs action decisions through its own decision model based on the current state and the current state information of other agents;

[0021] b. After the agent executes the action decision, it obtains timely rewards through the reward function, which is the opposite of the agent's cost function;

[0022] c. Using the multi-agent evolutionary strategy algorithm, update the parameters of each agent's decision model, so that the total cost of the overall system continues to decrease;

[0023] d. Repeat steps a to d above until the system reaches a stable global optimal state.

[0024] Preferably, step c comprises the following steps:

[0025] Initialize the decision model parameters θ of each agent i ;

[0026] At the current parameter θ i Next, sample K action sequences:

[0027] For each sampled action sequence Execute the action sequence to get the cumulative reward:

[0028]

[0029] Where: Indicates that the action sequence is being executed After that, the cumulative reward obtained by agent i; represents the immediate reward obtained by agent i at the tth time step;

[0030] Calculate the performance indicators of the action sequence:

[0031]

[0032] Where: Representing action sequence The corresponding performance indicator is the total cost of agent i;

[0033] Using the sampled performance indicators, update the parameters θ of agent i i :

[0034]

[0035] Where: θ i represents the decision model parameters of agent i; θ i ' represents the updated decision model parameters of agent i; α represents the learning rate, which controls the step size of parameter update; Indicates that for θ i The gradient of ; K represents the number of samples.

[0036] Preferably, step 4 comprises the following steps:

[0037] Step 4.1: Model the entire multi-agent as a non-cooperative game model; each agent is a participant in the game, and the goal is to minimize its own cost function;

[0038] Step 4.2: Find the Nash equilibrium solution of each agent when the decisions of other agents are determined, that is, the optimal decision solution for itself;

[0039] Step 4.3: Based on game theory analysis, check whether the current action decision has reached Nash equilibrium;

[0040] Step 4.4: If Nash equilibrium is not reached, decision coordination is performed until it finally converges to a Nash equilibrium solution.

[0041] Preferably, the Nash equilibrium solution in step 4.3 should satisfy the following conditions:

[0042]

[0043] Where: A i * represents the optimal decision of agent i; A i represents the initial decision of agent i; C i (S i ,A i *) represents the total cost function of agent i; C i (S i ,A i *) represents the cost calculated by agent i based on the optimal decision; C i (Si ,A i ) represents the cost calculated by agent i based on the initial decision.

[0044] Preferably, the decision coordination in step 4.4 includes the following steps:

[0045] Each agent calculates its own optimal decision based on the current state and the decisions of other agents;

[0046] The optimal decisions of each agent are exchanged and compared until the decisions of all agents can no longer be improved, and then the final global optimal action decision is output.

[0047] The beneficial effects of the present invention include:

[0048] The present invention divides the power grid system into multiple areas, each area corresponds to an independent intelligent agent, converts complex global problems into multiple local problems, and improves the feasibility of decision-making; establishes a cost function for each intelligent agent to comprehensively describe the cost impact of the intelligent agent performing a certain action; and adopts a deep neural network as the decision model of each intelligent agent, and uses a deep reinforcement learning algorithm to train the decision model; through interaction with the environment, the intelligent agent can help learn the optimal action decision strategy and gradually reduce its own operating cost; and constructs a reinforcement learning framework based on a multi-agent evolutionary strategy algorithm, constructs a separate reward function for each intelligent agent, so that it can learn the optimal action strategy according to its own state and the decision of other intelligent agents; optimally adopts a game theory-based method to further coordinate the action decisions of each intelligent agent to ensure that the system converges to the global optimal state; improves the overall operating efficiency and economy of the power grid system. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0050] Figure 1 An overall step block diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0052] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0053] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0055] See also Figure 1 As shown, the best embodiment of the present invention is further described;

[0056] A multi-agent reinforcement learning method based on operation cost optimization includes the following steps:

[0057] Step 1: Divide the entire power grid system into multiple areas, each area corresponds to an intelligent agent; define the state vector of each intelligent agent, determine the actions that each intelligent agent can take, and establish a cost function for each intelligent agent to describe the cost of executing the action;

[0058] Regional division:

[0059] The entire power grid system is divided into several regions according to factors such as the geographical location of the power grid system and the scope of power supply. Each region is a relatively independent subsystem with its own power plant, transmission and distribution facilities, etc.

[0060] This regional division method can make full use of the distributed characteristics of the power grid system, transform the complex global optimization problem into multiple relatively independent local optimization problems, and improve the feasibility of decision-making.

[0061] Agent definition:

[0062] Each region corresponds to an intelligent agent, representing the decision-making body of the power system in that region.

[0063] Define the state vector S of each agent i , including the power load situation, power generation capacity, transmission line status and other information in the area, reflecting the system status of the intelligent agent.

[0064] Determine the actions A that each agent can take i , including dispatching the output of power plants, controlling the power transmission of transmission lines, etc. These action decisions directly affect the operation of the power grid in the region.

[0065] The cost function of the construction is as follows:

[0066]

[0067] Where: w 1 、w 2 and w 3 is the corresponding weight coefficient, C i (S i ,A i ) represents the total cost function of agent i; represents the power generation cost function of agent i; represents the grid loss cost function of agent i; represents the emission cost function of agent i, that is, the environmental cost caused by the pollutant emissions generated during the power generation process; S i Indicates the state of agent i; A i Represents the initial decision of agent i.

[0068] The regional division and intelligent agent modeling methods can make full use of the distributed characteristics of the power grid system, while also incorporating comprehensive factors such as power generation costs, grid losses and environmental impacts, laying the foundation for subsequent optimization decisions.

[0069] Step 2: Use a deep neural network as the decision model for each agent, and use a deep reinforcement learning algorithm to train the decision model of each agent, and output the action decision of each agent through the decision model;

[0070] The structure of the decision model is as follows:

[0071] Input Layer:

[0072] The input is the state vector S of agent i i , contains information such as the power load, power generation capacity, and transmission line status of the area.

[0073] Hidden Layer:

[0074] The multi-layer perceptron (MLP) structure is used as the hidden layer, which consists of multiple fully connected hidden layers.

[0075] Each fully connected hidden layer uses ReLU (Rectified Linear Unit) as the activation function, which can effectively solve the gradient vanishing problem and improve the training efficiency of the model.

[0076] Output layer:

[0077] For discrete action spaces, the output layer uses a fully connected layer + Softmax$activation structure to output the probability distribution of each optional action. In actual decision-making, the action with the highest probability is selected.

[0078] For continuous action space, the output layer uses a fully connected layer + Tanh$activation structure to directly output the deterministic value of the action.

[0079] During the training process, a deep reinforcement learning algorithm is used to optimize the decision model. Specifically, a policy gradient-based optimization method such as the REINFORCE algorithm can be used.

[0080] Step 3: Construct a multi-agent reinforcement learning framework based on the multi-agent evolutionary strategy algorithm, define an independent reward function for each agent, and establish information exchange and coordinated decision-making between agents. Through the multi-agent reinforcement learning framework, optimize the action decision of each agent to minimize the cost;

[0081] The training steps of the multi-agent reinforcement learning framework are as follows:

[0082] a. Each agent outputs action decisions through its own decision model based on its current state and the current state information of other agents;

[0083] b. After the agent executes the action decision, it obtains timely rewards through the reward function, which is the opposite of the agent's cost function;

[0084] c. Using the multi-agent evolutionary strategy algorithm, update the parameters of each agent's decision model, so that the total cost of the overall system continues to decrease;

[0085] The step c comprises the following steps:

[0086] Initialize the decision model parameters θ of each agent i ;

[0087] At the current parameter θ i Next, sample K action sequences:

[0088] For each sampled action sequence Execute the action sequence to get the cumulative reward:

[0089]

[0090] Where: Indicates that the action sequence is being executed After that, the cumulative reward obtained by agent i; represents the immediate reward obtained by agent i at the tth time step;

[0091] Calculate the performance indicators of the action sequence:

[0092]

[0093] Where: Representing action sequence The corresponding performance indicator is the total cost of agent i;

[0094] Using the sampled performance indicators, update the parameters θ of agent i i :

[0095]

[0096] Where: θ i represents the decision model parameters of agent i; θ i ' represents the updated decision model parameters of agent i; α represents the learning rate, which controls the step size of parameter update; Indicates that for θ i The gradient of ; K represents the number of samples.

[0097] d. Repeat steps a to d above until the system reaches a stable global optimal state.

[0098] Step 4: Coordinate the action decisions of each intelligent agent to achieve the global optimum based on the game theory method; output the global optimal action decision, and each intelligent agent performs the corresponding action based on the output action decision.

[0099] The step 4 comprises the following steps:

[0100] Step 4.1: Model the entire multi-agent as a non-cooperative game model; each agent is a participant in the game, and the goal is to minimize its own cost function;

[0101] Define the game model participants:

[0102] The agents in each region are considered as independent game players. Each agent has its own cost function C i (S i ,A i ), the goal is to minimize its own operating costs.

[0103] Determine the decision variables of the game model:

[0104] Each agent can take action decisions A i , including dispatching the output of power plants and controlling the power transmission of transmission lines, which directly affects the operation of the power grid in the area.

[0105] The overall decision vector of the game model is {A 1 ,A 2 ,…,A N}, where N is the number of agents.

[0106] The objective function of establishing the game model is:

[0107] The goal of each agent i is to minimize its own cost function C i (S i ,A i ), the goal of the entire game model is to find the Nash equilibrium solution of each agent's decision, that is, all agents cannot reduce their own costs by unilaterally changing their own decisions.

[0108] Step 4.2: Find the Nash equilibrium solution of each agent when the decisions of other agents are determined, that is, the optimal decision solution for itself;

[0109] Determine the reward function for each agent:

[0110] For each agent i, the reward function is a negative cost function: R i (S i ,A i )=-C i (S i ,A i ); The reward function reflects the goal that agent i wants to maximize;

[0111] Analyze other agents' decisions:

[0112] Assume that the decisions of the other N-1 agents have been determined, that is, A i (j≠i) are all fixed values. In this case, agent i needs to find the optimal decision that maximizes its own benefits;

[0113] Solve the optimal decision of agent i:

[0114] The decision A of other agents i (j≠i) is considered as a known parameter, then the optimal decision of agent i is obtained by solving the following:

[0115] A i = argmaxR i (S i ,A i )=argmax-C i (Si ,A i );

[0116] That is, find a decision A i *, so that when the decisions of other agents are determined, the cost function C of agent i i minimize;

[0117] Step 4.3: Based on game theory analysis, check whether the current action decision has reached Nash equilibrium;

[0118] The Nash equilibrium solution in step 4.3 should satisfy the following conditions:

[0119]

[0120] Where: A i * represents the optimal decision of agent i; A i represents the initial decision of agent i; C i (S i ,A i *) represents the total cost function of agent i; C i (S i ,A i *) represents the cost calculated by agent i based on the optimal decision; C i (S i ,A i ) represents the cost calculated by agent i based on the initial decision.

[0121] Step 4.4: If Nash equilibrium is not reached, decision coordination is performed until it finally converges to a Nash equilibrium solution.

[0122] The decision coordination in step 4.4 includes the following steps:

[0123] Each agent calculates its own optimal decision based on the current state and the decisions of other agents;

[0124] The optimal decisions of each agent are exchanged and compared until the decisions of all agents can no longer be improved, and then the final global optimal action decision is output.

[0125] Exemplary:

[0126] Initial decision making

[0127] Each regional intelligent agent calculates its own optimal power generation scheduling plan based on its own power generation costs, grid losses, emissions and other factors.

[0128] Decision exchange and comparison

[0129] Agent 1 combines its own power generation scheduling plan with the plans of other agents to calculate the total cost of the entire power grid system.

[0130] Let other agents exchange their solutions in turn and calculate the total cost.

[0131] Decision Improvement and Convergence

[0132] Compare the total costs under various decision exchange combinations and select the global power generation scheduling scheme that minimizes the total cost.

[0133] If a better total cost solution is found, the global decision is updated and the decision exchange and comparison are repeated.

[0134] If no better solution is found, it means that Nash equilibrium has been reached, and the current global optimal power generation scheduling plan is output.

[0135] The present invention divides the power grid system into multiple areas, each area corresponds to an independent intelligent agent, converts complex global problems into multiple local problems, and improves the feasibility of decision-making; establishes a cost function for each intelligent agent to comprehensively describe the cost impact of the intelligent agent performing a certain action; and adopts a deep neural network as the decision model of each intelligent agent, and uses a deep reinforcement learning algorithm to train the decision model; through interaction with the environment, the intelligent agent can help learn the optimal action decision strategy and gradually reduce its own operating cost; and constructs a reinforcement learning framework based on a multi-agent evolutionary strategy algorithm, constructs a separate reward function for each intelligent agent, so that it can learn the optimal action strategy according to its own state and the decision of other intelligent agents; optimally adopts a game theory-based method to further coordinate the action decisions of each intelligent agent to ensure that the system converges to the global optimal state; improves the overall operating efficiency and economy of the power grid system.

[0136] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-agent reinforcement learning method based on operation cost optimization, characterized in that: The following steps are involved: Step 1: Divide the entire power grid system into multiple areas, each area corresponds to an intelligent agent; define the state vector of each intelligent agent, determine the actions that each intelligent agent can take, and establish a cost function for each intelligent agent to describe the cost of executing the action; Step 2: Use a deep neural network as the decision model for each agent, and use a deep reinforcement learning algorithm to train the decision model of each agent, and output the action decision of each agent through the decision model; Step 3: Construct a multi-agent reinforcement learning framework based on the multi-agent evolutionary strategy algorithm, define an independent reward function for each agent, and establish information exchange and coordinated decision-making between agents. Through the multi-agent reinforcement learning framework, optimize the action decision of each agent to minimize the cost; Step 4: Coordinate the action decisions of each intelligent agent to achieve the global optimum based on the game theory method; output the global optimal action decision, and each intelligent agent performs the corresponding action based on the output action decision.

2. A multi-agent reinforcement learning method based on operation cost optimization according to claim 1, characterized in that: The cost function of the construction is as follows: Where: w1, w2 and w3 are the corresponding weight coefficients, C i (S i ,A i ) represents the total cost function of agent i; represents the power generation cost function of agent i; represents the grid loss cost function of agent i; represents the emission cost function of agent i, that is, the environmental cost caused by the pollutant emissions generated during the power generation process; S i Indicates the state of agent i; A i Represents the initial decision of agent i.

3. The multi-agent reinforcement learning method based on operation cost optimization according to claim 1, characterized in that: The structure of the decision model is as follows: Input layer: The input is the state vector of the agent; Hidden layer: The decision model adopts a multi-layer perceptron structure, which consists of multiple fully connected hidden layers; each fully connected hidden layer uses ReLU activation function as the activation function; Output layer: The output layer outputs the actions that the agent can take; if the output of the agent is a discrete action, the output layer adopts a fully connected layer + Softmaxactivation structure to output the probability distribution of each action; if the output of the agent is a continuous action, a fully connected layer + Tanhactivation is used to output the certainty value of the action; for the output probability distribution, the action with the highest probability is selected.

4. The multi-agent reinforcement learning method based on operation cost optimization according to claim 1, characterized in that: The training steps of the multi-agent reinforcement learning framework are as follows: a. Each agent outputs action decisions through its own decision model based on its current state and the current state information of other agents; b. After the agent executes the action decision, it obtains timely rewards through the reward function, which is the opposite of the agent's cost function; c. Using the multi-agent evolutionary strategy algorithm, update the parameters of each agent's decision model, so that the total cost of the overall system continues to decrease; d. Repeat steps a to d above until the system reaches a stable global optimal state.

5. A multi-agent reinforcement learning method based on operation cost optimization according to claim 4, characterized in that: The step c comprises the following steps: Initialize the decision model parameters θ of each agent i ; At the current parameter θ i Next, sample K action sequences: For each sampled action sequence Execute the action sequence to get the cumulative reward: Where: Indicates that the action sequence is being executed After that, the cumulative reward obtained by agent i; represents the immediate reward obtained by agent i at the tth time step; Calculate the performance indicators of the action sequence: Where: Representing action sequence The corresponding performance indicator is the total cost of agent i; Using the sampled performance indicators, update the parameters θ of agent i i : Where: θ i represents the decision model parameters of agent i; θ i ′ represents the updated decision model parameters of agent i; α represents the learning rate, which controls the step size of parameter update; Indicates that for θ i The gradient of ; K represents the number of samples.

6. The multi-agent reinforcement learning method based on operation cost optimization according to claim 1, characterized in that: The step 4 comprises the following steps: Step 4.1: Model the entire multi-agent as a non-cooperative game model; each agent is a participant in the game, and the goal is to minimize its own cost function; Step 4.2: Find the Nash equilibrium solution of each agent when the decisions of other agents are determined, that is, the optimal decision solution for itself; Step 4,3: Based on game theory analysis, check whether the current action decision has reached Nash equilibrium; Step 4.4: If Nash equilibrium is not reached, decision coordination is performed until it finally converges to a Nash equilibrium solution.

7. A multi-agent reinforcement learning method based on operation cost optimization according to claim 6, characterized in that: The Nash equilibrium solution in step 4.3 should satisfy the following conditions: Where: A i * represents the optimal decision of agent i; A i represents the initial decision of agent i; C i (S i ,A i *) represents the total cost function of agent i; C i (S i ,A i *) represents the cost calculated by agent i based on the optimal decision; C i (S i ,A i ) represents the cost calculated by agent i based on the initial decision.

8. The multi-agent reinforcement learning method based on operation cost optimization according to claim 6, characterized in that: The decision coordination in step 4.4 includes the following steps: Each agent calculates its own optimal decision based on the current state and the decisions of other agents; The optimal decisions of each agent are exchanged and compared until the decisions of all agents can no longer be improved, and then the final global optimal action decision is output.

Images