Power grid investment dynamic game optimization method and system based on state space decomposition and agent reinforcement learning

Abstract

The invention provides a power grid investment dynamic game optimization method and system based on state space decomposition and agent reinforcement learning. According to the method, the power grid investment decision problem is modeled into the intelligent agent dynamic game problem, and the reinforcement learning technology is combined, so that the power grid investment scheme can be effectively optimized, and the economical efficiency and stability of the system are improved. According to the method, a state space decomposition strategy is adopted, the power grid investment problem is divided into a plurality of sub-problems, the dimensionality is reduced, and the calculation efficiency is improved. Meanwhile, intelligent agent reinforcement learning is introduced, so that intelligent agents (such as a power grid company, a power generation enterprise, a supervision mechanism and the like) optimize investment decisions through interactive learning, and finally a global optimal investment scheme is realized.

Description

Technical Field

[0001] The present invention provides a method and system for dynamic game optimization of power grid investment based on state space decomposition and intelligent agent reinforcement learning, specifically relates to a dynamic decision-making optimization method for power grid investment based on intelligent algorithms, and belongs to the intersection of smart grid technology and computational intelligence. Background Art

[0002] Currently, power grid investment decisions are usually made using optimization model-based methods, which mainly include:

[0003] Static optimization method: Grid investment decisions are made based on traditional mathematical optimization methods such as linear programming (LP), integer programming (IP), and dynamic programming (DP), but they cannot effectively cope with the dynamic uncertainty of grid development.

[0004] Game theory approach: Some studies have introduced game theory to model grid investment, but they usually assume that investors have perfect information and cannot cope with market uncertainty.

[0005] Optimization methods based on reinforcement learning (RL): Reinforcement learning is used for power grid scheduling and investment optimization. However, since power grid investment decisions involve high-dimensional state spaces and agent interactions, single-agent reinforcement learning faces problems such as difficult convergence and high computational complexity in high-dimensional complex systems. Summary of the Invention

[0006] Based on this, the purpose of this invention is to improve the investment returns of power grids, reduce carbon emissions, and enhance the adaptability of investment strategies through key technologies such as state space decomposition, reinforcement learning optimization, and dynamic game solving.

[0007] The present invention proposes a power grid investment dynamic game optimization method based on state space decomposition and agent reinforcement learning, which includes the following steps:

[0008] Step S1: The power grid investment decision problem is represented as an agent Markov game problem, and a parameter model for the power grid investment decision is established. The parameter model for the MDP power grid investment decision of a single agent i includes: the state of the power grid is represented by the state space S; the investment decision of the power grid is represented by the action space A; and the investment income of the power grid is calculated by the reward function R(s,a).

[0009] Step S2: Divide the state space S of the power grid;

[0010] Step S3: Decompose the action space A of the power grid; derive the reward function R(s,a);

[0011] Step S4: Using a centralized training and distributed execution strategy, multiple investment entities are trained through reinforcement learning. This includes optimizing the strategy of each investment entity through proximal strategy optimization methods and using attention mechanisms and graph neural networks to enhance the information exchange capabilities between agents.

[0012] Step S5: using an adversarial learning mechanism to solve the dynamic game;

[0013] Step S6: Output the optimal decision for grid investment.

[0014] Furthermore, step S1 includes the following contents:

[0015] Step S11: The state space S of the power grid includes the components including the power transmission line state of the power grid, including the operating state of each transmission line and the power flow P l ; The substation status of the power grid includes the voltage amplitude v of the substation b and phase angle θ b The load demand of the power grid includes the real-time active load p of each node d b and reactive load q d b ; The market electricity price of the power grid includes the real-time electricity price p market The market price of the power grid includes: the total output of renewable energy in the region t ;

[0016] Step S12: The investment decision of the power grid is represented by the action space A; wherein the action space A includes the transmission investment A of the investment decision of the power grid grid , Power generation investment decision A gen and demand response strategies for grid investment decisions demand ;

[0017] Step S13: The investment income of the power grid is calculated by the reward function R(s,a), where the investment income of the power grid includes economic income, investment cost and social income;

[0018] Furthermore, step S2 includes the following contents:

[0019] Step S21: Decompose the state space S of the power grid into global state and local state, including the following contents:

[0020] S=S global ∪{S1,S2,…S j ,…S n}

[0021] Among them, the global state of the power grid is represented by S global Indicates that the global state of the power grid S globalIncluding the topology of the power grid and the total load of the power grid Where d represents the index of the power plant connected to the grid, b is the total number of power plants; and the market electricity price p market ; The regional state of the power grid of agent i is represented by S j express;

[0022] Step S22: Use the self-encoding dimension reduction technology to analyze the regional state S of the power grid. j Perform dimensionality reduction, including the following:

[0023] h j =σ(W enc S j +b enc )

[0024] S j low =f autoencoder (S j )=σ(W dnc h j +b enc )

[0025] Among them, σ(·) is the activation function, W enc is the weight matrix of the encoder, W dec Decoder weight matrix, b enc is the bias vector; f autoencoder (·) represents the dimensionality reduction process of the autoencoder;

[0026] Step S23: The autoencoder is obtained by minimizing the reconstruction error:

[0027]

[0028] Among them L recon represents the reconstruction error; Indicates the state after reconstruction.

[0029] Furthermore, step S3 includes the following contents:

[0030] Step S31: Decompose the action space A of the power grid: The investment decisions that can be taken in the action space A of each intelligent agent i include transmission investment, generation investment, demand response strategy, etc. The final action space is expressed as: A = {A grid ,A gen ,A demand};

[0031] Furthermore, in step S2, three types of investment decisions of agent i are defined:

[0032] A i grid={(add,l,c),(upgrade,l,c),(remove,l)}

[0033] Among them, (add, l, c) means adding a new transmission line l with a capacity of c; (upgrade, l, c) means expanding the capacity of line l; (remove, l) means removing line l;

[0034] A i gen ={(add,g,c),(upgrade,g,c),(retire,g)}

[0035] Among them, (add,g,c) means adding a new unit g with a capacity of c; (upgrade,g,c) means expanding the capacity of unit g; (retire,g) means retiring unit g;

[0036] A i demand ={(price,p),(incentive,β),(storage,c)}

[0037] Among them, price represents the adjusted electricity price p; incentive represents the user response incentive parameter β; storage represents the investment in the energy storage system, capacity c.

[0038] Furthermore, step S3 also includes the following contents:

[0039] Step S32: The economic benefit term of the reward function R(s,a) of the single agent i includes:

[0040]

[0041] Where Riprofit is the cumulative economic benefit obtained by agent i during the investment cycle; T is the time span of investment decision, that is, the total number of cycles for the investment subject to optimize the decision; γ is the discount factor, which ranges from 0 < γ < 1 and is used to measure the impact of future returns; pilocal is the electricity price of the i-th investment subject in the local market, and the unit of electricity price is: yuan / kWh; is the current electricity sales volume of the i-th investment entity, and the unit of electricity sales volume is kWh; is the operating cost of the i-th investment entity, including daily maintenance, management and operation costs (unit: yuan);

[0042] Step S33: The investment cost term of the reward function R(s,a) includes the following:

[0043]

[0044] in, is the total investment cost of the i-th investment entity; is the infrastructure construction cost of the i-th investment entity (unit: yuan); is the long-term maintenance cost of the i-th investment entity (unit: yuan); Subsidies provided by the government or policy agencies to encourage new energy investment (unit: yuan);

[0045] Step S33: The carbon emission penalty term of the reward function R(s,a) includes the following:

[0046]

[0047] in, is the penalty imposed on the i-th investment entity for carbon emissions (unit: yuan); λ is the carbon emission penalty coefficient, which represents the fee to be paid per unit of carbon emissions (or carbon tax, unit: yuan / ton); is the carbon emissions of the i-th investment entity (unit: ton), which is proportional to the power generation of the coal-fired unit.

[0048] Furthermore, the step S4 includes the following contents:

[0049] Step S41: The reinforcement learning method adopts a centralized training-distributed execution (CTDE) architecture, which enables multiple agents to share experience in the training phase and make independent decisions in the execution phase;

[0050] In the training phase, all agents share the global state S and use the Proximal Policy Optimization (PPO) algorithm to optimize the strategy. The specific formula is as follows:

[0051]

[0052] in:

[0053] clip() represents the clipping function, E t Represents the mathematical expectation of the state-action-reward sample data collected during training;

[0054]

[0055] ∈=0.2

[0056] Step S42: In the execution phase, agent i observes the local i Independent decision-making:

[0057] a i ~π i (a|o i )

[0058] oi =g(S,i)

[0059] Where: g(·) is the local observation function, which extracts the state variables of region j;

[0060] Step S43: The reinforcement learning method uses an attention mechanism and a graph neural network; the graph neural network includes the following contents:

[0061] h i =σ(WAGG({h j}∪{h i})+b)

[0062] Among them, parameter b represents the bias term used to adjust the output of the neural network layer; h i represents the hidden state of agent i; h j Represents the hidden state of the region; WAGG represents the feature used to combine the neighbor region j with h i The combined aggregate weight matrix, ; σ represents the nonlinear activation parameter;

[0063] The attention mechanism weights include the following:

[0064]

[0065] Among them, W q Represents the query weight matrix, h i Convert to query vector q i =W q h i , W k represents the key weight matrix, d represents the dimension of the query and key vectors, and the function softmax represents the normalization function that converts the score into a probability distribution.

[0066] Furthermore, step S5 includes the following contents:

[0067] Step S51: Adopting an adversarial learning mechanism to solve the dynamic game includes using a generative adversarial network (GAN) model to simulate market changes, wherein the game between the GAN generator and the discriminator is:

[0068]

[0069] Among them, the parameters Represents the expectation of the real data distribution, D(S) represents the output probability of the discriminator for the real sample S, Represents the discriminator's response to the generator sample The output probability of .

[0070] Furthermore, step S6 includes the following contents:

[0071] Step S61: Outputting the agent strategy optimization target, including the following:

[0072]

[0073] Among them, the parameter E represents the average expectation of all possible trajectories, the parameter T represents the time step or the maximum number of steps in the round, and the parameter J(π i ) is represented as strategy π i The expected total discounted reward of .

[0074] According to the second aspect of the present invention, a dynamic game optimization system for power grid investment based on state space decomposition and agent reinforcement learning includes an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor, and is characterized in that when the processor executes the computer program, it implements a dynamic game optimization method for power grid investment based on state space decomposition and agent reinforcement learning as described in any one of the items mentioned in the present invention.

[0075] According to the third aspect of the present invention, a dynamic game optimization system for power grid investment based on state space decomposition and agent reinforcement learning includes a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements a dynamic game optimization method for power grid investment based on state space decomposition and agent reinforcement learning as described in any one of the items mentioned in the present invention.

[0076] Compared with the prior art, the present invention has the following beneficial effects:

[0077] First, this invention utilizes reinforcement learning, enabling intelligent agents to autonomously learn optimal investment strategies, maximizing investment returns while reducing unnecessary costs. Second, intelligent investment can be used to regulate load, increase renewable energy penetration, and reduce carbon emissions from thermal power plants. Furthermore, state-space decomposition and agent-based reinforcement learning reduce computational complexity, making it suitable for large-scale power grid optimization scenarios. Finally, by introducing intelligent agent-based game theory, grid investors can make autonomous decisions, optimize market mechanisms, and achieve supply and demand balance.

[0078] The present invention can be applied to fields such as smart grid investment optimization, new energy access planning, and power market mechanism design, providing technical support for optimal investment and efficient regulation of power systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 Flowchart of a method according to an embodiment of the present invention.

[0080] Figure 2 This is a reinforcement learning training curve diagram of the present invention.

[0081] Figure 3 This is a comparison chart of the economic and environmental performance of different methods of the present invention.

[0082] Figure 4 This is an analysis diagram of voltage stability, return on investment, and load response after reinforcement learning of the present invention. DETAILED DESCRIPTION

[0083] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0084] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0085] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0086] The present invention proposes a power grid investment dynamic game optimization method based on state space decomposition and agent reinforcement learning, which includes the following steps:

[0087] Step S1: The power grid investment decision problem is represented as an agent Markov game problem, and a parameter model for the power grid investment decision is established. The parameter model for the MDP power grid investment decision of a single agent i includes: the state of the power grid is represented by the state space S; the investment decision of the power grid is represented by the action space A; and the investment income of the power grid is calculated by the reward function R(s,a).

[0088] Step S2: Divide the state space S of the power grid;

[0089] Step S3: Decompose the action space A of the power grid; derive the reward function R(s,a);

[0090] Step S4: Using a centralized training and distributed execution strategy, multiple investment entities are trained through reinforcement learning. This includes optimizing the strategy of each investment entity through proximal strategy optimization methods and using attention mechanisms and graph neural networks to enhance the information exchange capabilities between agents.

[0091] Step S5: using an adversarial learning mechanism to solve the dynamic game;

[0092] Step S6: Output the optimal decision for grid investment.

[0093] Furthermore, step S1 includes the following contents:

[0094] Step S11: The state space S of the power grid includes the components including the power transmission line state of the power grid, including the operating state of each transmission line and the power flow P l ; The substation status of the power grid includes the voltage amplitude v of the substation b and phase angle θ b The load demand of the power grid includes the real-time active load p of each node d b and reactive load q d b ; The market electricity price of the power grid includes the real-time electricity price p market The market price of the power grid includes: the total output of renewable energy in the region t ;

[0095] Step S12: The investment decision of the power grid is represented by the action space A; wherein the action space A includes the transmission investment A of the investment decision of the power grid grid , Power generation investment decision A gen and demand response strategies for grid investment decisions demand ;

[0096] Step S13: The investment income of the power grid is calculated by the reward function R(s,a), where the investment income of the power grid includes economic income, investment cost and social income;

[0097] Furthermore, step S2 includes the following contents:

[0098] Step S21: Decompose the state space S of the power grid into global state and local state, including the following contents:

[0099] S=S global ∪{S1,S2,…S j ,…S n}

[0100] Among them, the global state of the power grid is represented by S global Indicates that the global state of the power grid S global Including the topology of the power grid and the total load of the power grid Where d represents the index of the power plant connected to the grid, b is the total number of power plants; and the market electricity price p market ; The regional state of the power grid of agent i is represented by S j express;

[0101] Step S22: Use the self-encoding dimension reduction technology to analyze the regional state S of the power grid. j Perform dimensionality reduction, including the following:

[0102] h j =σ(W enc S j +b enc )

[0103] S j low =f autoencoder (S j )=σ(W dnc h j +b enc )

[0104] Among them, σ(·) is the activation function, W enc is the weight matrix of the encoder, W dec Decoder weight matrix, b enc is the bias vector; f autoencoder (·) represents the dimensionality reduction process of the autoencoder;

[0105] Step S23: The autoencoder is obtained by minimizing the reconstruction error:

[0106]

[0107] Among them L recon represents the reconstruction error; Indicates the state after reconstruction.

[0108] Furthermore, step S3 includes the following contents:

[0109] Step S31: Decompose the action space A of the power grid: The investment decisions that can be taken in the action space A of each intelligent agent i include transmission investment, generation investment, demand response strategy, etc. The final action space is expressed as: A = {A grid ,A gen ,A demand};

[0110] Furthermore, in step S2, three types of investment decisions of agent i are defined:

[0111] A i grid ={(add,l,c),(upgrade,l,c),(remove,l)}

[0112] Among them, (add, l, c) means adding a new transmission line l with a capacity of c; (upgrade, l, c) means expanding the capacity of line l; (remove, l) means removing line l;

[0113] A i gen={(add,g,c),(upgrade,g,c),(retire,g)}

[0114] Among them, (add,g,c) means adding a new unit g with a capacity of c; (upgrade,g,c) means expanding the capacity of unit g; (retire,g) means retiring unit g;

[0115] A i demand ={(price,p),(incentive,β),(storage,c)}

[0116] Among them, price represents the adjusted electricity price p; incentive represents the user response incentive parameter β; storage represents the investment in the energy storage system, capacity c.

[0117] Furthermore, step S3 also includes the following contents:

[0118] Step S32: The economic benefit term of the reward function R(s,a) of the single agent i includes:

[0119]

[0120] Where Riprofit is the cumulative economic benefit obtained by agent i during the investment cycle; T is the time span of investment decision, that is, the total number of cycles for the investment subject to optimize the decision; γ is the discount factor, which ranges from 0 < γ < 1 and is used to measure the impact of future returns; pilocal is the electricity price of the i-th investment subject in the local market, and the unit of electricity price is: yuan / kWh; is the current electricity sales volume of the i-th investment entity, and the unit of electricity sales volume is kWh; is the operating cost of the i-th investment entity, including daily maintenance, management and operation costs (unit: yuan);

[0121] Step S33: The investment cost term of the reward function R(s,a) includes the following:

[0122]

[0123] in, is the total investment cost of the i-th investment entity; is the infrastructure construction cost of the i-th investment entity (unit: yuan); is the long-term maintenance cost of the i-th investment entity (unit: yuan); Subsidies provided by the government or policy agencies to encourage new energy investment (unit: yuan);

[0124] Step S33: The carbon emission penalty term of the reward function R(s,a) includes the following:

[0125]

[0126] in, is the penalty imposed on the i-th investment entity for carbon emissions (unit: yuan); λ is the carbon emission penalty coefficient, which represents the fee to be paid per unit of carbon emissions (or carbon tax, unit: yuan / ton); is the carbon emissions of the i-th investment entity (unit: ton), which is proportional to the power generation of the coal-fired unit.

[0127] Furthermore, the step S4 includes the following contents:

[0128] Step S41: The reinforcement learning method adopts a centralized training-distributed execution (CTDE) architecture, which enables multiple agents to share experience in the training phase and make independent decisions in the execution phase;

[0129] In the training phase, all agents share the global state S and use the Proximal Policy Optimization (PPO) algorithm to optimize the strategy. The specific formula is as follows:

[0130]

[0131] in:

[0132] clip() represents the clipping function, E t Represents the mathematical expectation of the state-action-reward sample data collected during training;

[0133]

[0134] ∈=0.2

[0135] Step S42: In the execution phase, agent i observes the local i Independent decision-making:

[0136] a i ~π i (a|o i )

[0137] o i =g(S,i)

[0138] Where: g(·) is the local observation function, which extracts the state variables of region j;

[0139] Step S43: The reinforcement learning method uses an attention mechanism and a graph neural network; the graph neural network includes the following contents:

[0140] hi =σ(WAGG({h j}∪{h i})+b)

[0141] Among them, parameter b represents the bias term used to adjust the output of the neural network layer; h i represents the hidden state of agent i; h j Represents the hidden state of the region; WAGG represents the feature used to combine the neighbor region j with h i The combined aggregate weight matrix, ; σ represents the nonlinear activation parameter;

[0142] The attention mechanism weights include the following:

[0143]

[0144] Among them, W q Represents the query weight matrix, h i Convert to query vector q i =W q h i , W k represents the key weight matrix, d represents the dimension of the query and key vectors, and the function softmax represents the normalization function that converts the score into a probability distribution.

[0145] Furthermore, step S5 includes the following contents:

[0146] Step S51: Adopting an adversarial learning mechanism to solve the dynamic game includes using a generative adversarial network (GAN) model to simulate market changes, wherein the game between the GAN generator and the discriminator is:

[0147]

[0148] Among them, the parameters Represents the expectation of the real data distribution, D(S) represents the output probability of the discriminator for the real sample S, Represents the discriminator's response to the generator sample The output probability of .

[0149] Furthermore, step S6 includes the following contents:

[0150] Step S61: Outputting the agent strategy optimization target, including the following:

[0151]

[0152] Among them, the parameter E represents the average expectation of all possible trajectories, the parameter T represents the time step or the maximum number of steps in the round, and the parameter J(π i ) is represented as strategy π iThe expected total discounted reward of .

[0153] According to the second aspect of the present invention, a dynamic game optimization system for power grid investment based on state space decomposition and agent reinforcement learning includes an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor, and is characterized in that when the processor executes the computer program, it implements a dynamic game optimization method for power grid investment based on state space decomposition and agent reinforcement learning as described in any one of the items mentioned in the present invention.

[0154] According to the third aspect of the present invention, a dynamic game optimization system for power grid investment based on state space decomposition and agent reinforcement learning includes a computer-readable storage medium, and the computer-readable storage medium stores a computer program, and is characterized in that when the computer program is executed by a processor, it implements a dynamic game optimization method for power grid investment based on state space decomposition and agent reinforcement learning as described in any one of the items mentioned in the present invention.

[0155] In this embodiment, the specific contents of step S1 are as follows: based on the IEEE 14-bus test system, the grid investment entities include the grid company, power generation enterprises, and demand-side management agencies; 14-dimensional voltage data is used to represent the grid state space S, and a discrete 4-choice 1 is used to represent the action space A; the reward function is defined as R(s,a) = 0.5 × benefit - 0.3 × investment cost + 0.2 × carbon emission penalty.

[0156] Furthermore, in this embodiment, step S2 specifically involves splitting the grid action space A and state space S using a regional partitioning method. The grid investment optimization problem is modeled using a Markov decision process (MDP). The reinforcement learning agent's action space A consists of four discrete actions: Action A1 (no investment): maintaining the current grid state and not taking any investment action; Action A2 (new transmission line): adding a new transmission line to increase grid capacity; Action A3 (expanding power generation capacity): expanding or enhancing generators to increase power supply capacity; and Action A4 (adjusting electricity prices): adjusting electricity prices to influence load demand.

[0157] The state space S is decomposed into two types of states: the global state Sglobal, which represents the overall information of the power grid, such as the total system load, total investment, and market electricity prices; and the local state Si, which represents the local power grid status for each intelligent agent (grid company, power generation enterprise, or demand-side management agency), including local load, voltage level, and transmission line conditions within the region.

[0158] During the agent interaction process, after each action is performed, the environment will update the grid status based on the power flow calculation and calculate the corresponding reward value for reinforcement learning training.

[0159] Furthermore, in this embodiment, step S3 specifically involves optimizing the agent's investment decision using the Proximal Policy Optimization (PPO) reinforcement learning method. PPO is a reinforcement learning algorithm that uses experience replay and clipping techniques to improve training stability. Through repeated interactions with the environment, the agent adjusts its strategy based on the reward function and gradually finds the optimal grid investment decision. The training parameters are set as follows: discount factor γ = 0.99, clipping factor ∈ = 0.2, and maximum number of training rounds N = 1000.

[0160] During training, the agent makes decisions based on the current state of the grid (such as voltage level, power demand, investment strategy, etc.). After each action is executed, the grid state is updated, and rewards are calculated based on the new state and fed back to the agent to update its strategy.

[0161] Furthermore, in this embodiment, the specific content of step S4 is: a dynamic game solving method is introduced in the reinforcement learning process. Through the adversarial learning mechanism, the intelligent agent can continuously optimize its own strategy in a competitive environment, and the strategy function is updated: Where α is the learning rate, is the gradient of the policy function. The agent learns interactively in a dynamic environment and optimizes individual benefits, while taking into account system carbon emissions and economic costs, and gradually reaches game equilibrium. The training process is strengthened as follows Figure 1 and Figure 2 As shown in the figure, the light blue curve is the profit per round, and the dark blue curve is the moving average profit.

[0162] Furthermore, in this embodiment, the specific content of step S5 is: by analyzing the long-term reward, the total system income and the carbon emission index, it is determined whether the agent strategy has achieved the optimization goal. The specific results are as follows: Figure 3 and Figure 4 As shown. Among them, Figure 3 The interpretation of the results is: Left: The total benefit under the method proposed by the present invention is the highest; Middle: The investment cost of this method is the lowest; Right: The carbon emissions are the smallest, with significant green benefits. Figure 4The results are interpreted as follows: the top figure shows the voltage distribution at each node, with values ​​between 0.95 and 1.1, indicating system stability; the middle figure shows the return on investment (ROI) under the four strategies, with action 4 performing best; the bottom figure shows that under the "adjust electricity price" strategy, off-peak load is significantly increased, effectively achieving peak load shaving and valley filling.

[0163] Further, if Figure 4 As shown, the upper figure shows the voltage distribution of each node, with values ​​between 0.95–1.1, indicating a stable system; the middle figure shows the return on investment (ROI) under the four strategies, with action 4 performing best; the lower figure shows that under the "adjust electricity price" strategy, the off-peak load is significantly improved, effectively achieving load peak shaving and valley filling.

[0164] Furthermore, in this embodiment, step S6 specifically involves outputting the final optimal investment decision. After the strategy training converges, this embodiment conducts testing and evaluation, outputting the optimal action distribution and environmental response data shown in Table 1 based on the agent, and outputting optimization recommendations. These recommendations specifically include: a transmission optimization tendency: recommending line expansion to improve network capacity and stability; and a demand-side optimization tendency: recommending dynamic adjustments to electricity prices to influence load response.

[0165] action Number of times selected Proportion A1 13 13％ A2 41 41％ A3 17 17％ A4 29 29％

[0166] Table 1 Distribution of final strategy action selections

[0167] The above description is merely a simulation embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A dynamic game optimization method for power grid investment based on state space decomposition and agent reinforcement learning, characterized in that: The following steps are involved: Step S1: The power grid investment decision problem is represented as an agent Markov game problem, and a parameter model for the power grid investment decision is established. The parameter model for the power grid investment decision of agent i includes: the state of the power grid is represented by the state space S; the investment decision of the power grid is represented by the action space A; and the investment income of the power grid is calculated by the reward function R(s,a). Step S2: Divide the state space S of the power grid; Step S3: Decompose the action space A of the power grid; derive the reward function R(s,a); Step S4: Using a centralized training and distributed execution strategy, multiple investment entities are trained through reinforcement learning, including optimizing the strategy of each investment entity through a proximal strategy optimization method and improving the information interaction capability of agent i using an attention mechanism and a graph neural network. Step S5: using an adversarial learning mechanism to solve the dynamic game; Step S6: Output the optimal decision for grid investment.

2. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 1 is characterized in that: The step S1 includes the following contents: Step S11: The state space S of the power grid includes the components including the power transmission line state of the power grid, including the operating state of each transmission line and the power flow P l ; The substation status of the power grid includes the voltage amplitude v of the substation b and phase angle θ b The load demand of the power grid includes the real-time active load p of each node d b and reactive load q d b ; The market electricity price of the power grid includes the real-time electricity price p market ; The market price of the power grid includes: the total output of renewable energy in the region r t ; Step S12: The investment decision of the power grid is represented by the action space A; wherein the action space A includes the transmission investment A of the investment decision of the power grid grid , Power generation investment decision A gen and demand response strategies for grid investment decisions demand ; Step S13: The investment income of the power grid is calculated by the reward function R(s, a), where the investment income of the power grid includes economic income, investment cost and social income.

3. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 2 is characterized in that: The step S2 includes the following contents: Step S21: Decompose the state space S of the power grid into global state and local state, including the following contents: S=S global ∪{S1,S2,…S j ,…S n } Among them, the global state of the power grid is represented by S global Indicates that the global state of the power grid S global Including the topology of the power grid and the total load of the power grid Where d represents the index of the power plant connected to the grid, b is the total number of power plants; and the market electricity price p market ; The regional state of the power grid of agent i is represented by S j express; Step S22: Use the self-encoding dimension reduction technology to analyze the regional state S of the power grid. j Perform dimensionality reduction, including the following: h j =σ(W enc S j +b enc ) Among them, σ(·) is the activation function, W enc is the weight matrix of the encoder, W dec Decoder weight matrix, b enc is the bias vector; f autoencoder (·) represents the dimensionality reduction process of the autoencoder; Step S23: The autoencoder is obtained by minimizing the reconstruction error: Among them L recon represents the reconstruction error; Indicates the state after reconstruction.

4. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 2 is characterized in that: The step S3 includes the following contents: Step S31: Decompose the action space A of the power grid: The investment decisions that can be taken in the action space A of each intelligent agent i include transmission investment, generation investment, demand response strategy, etc. The final action space is expressed as: A = {A grid ,A gen ,A demand }; Furthermore, in step S2, three types of investment decisions of agent i are defined: A i grid ={(add,l,c),(upgrade,l,c),(remove,l)} Among them, (add, l, c) means adding a new transmission line l with a capacity of c; (upgrade, l, c) means expanding the capacity of line l; (remove, l) means removing line l; A i gen ={(add,g,c),(upgrade,g,c),(retire,g)} Among them, (add,g,c) means adding a new unit g with a capacity of c; (upgrade,g,c) means expanding the capacity of unit g; (retire,g) means retiring unit g; A i demand ={(price,p),(incentive,β),(storage,c)} Among them, price represents the adjusted electricity price p; incentive represents the user response incentive parameter β; storage represents the investment in the energy storage system, capacity c.

5. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 2, characterized in that: The step S3 further includes the following contents: Step S32: The economic benefit term of the reward function R(s,a) of the single agent i includes: Where Riprofit is the cumulative economic benefit obtained by agent i during the investment cycle; T is the time span of investment decision, that is, the total number of cycles for the investment subject to optimize the decision; γ is the discount factor, which ranges from 0 < γ < 1 and is used to measure the impact of future returns; pilocal is the electricity price of the i-th investment subject in the local market; is the current electricity sales volume of the i-th investment entity, and the unit of electricity sales volume is kWh; is the operating cost of the i-th investment entity, including daily maintenance, management and operation costs; Step S33: The investment cost term of the reward function R(s,a) includes the following: in, is the total investment cost of the i-th investment entity; is the infrastructure construction cost of the i-th investment entity; is the long-term maintenance cost of the i-th investment entity; Subsidies provided by governments or policy agencies to encourage investment in new energy; Step S33: The carbon emission penalty term of the reward function R(s,a) includes the following: in, is the penalty imposed on the i-th investment entity for carbon emissions; λ is the carbon emission penalty coefficient, which represents the cost to be paid per unit of carbon emissions; is the carbon emission of the i-th investment entity, which is proportional to the power generation of the coal-fired unit.

6. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 1, characterized in that: The step S4 includes the following contents: Step S41: The reinforcement learning method adopts a centralized training-distributed execution architecture, which enables multiple agents to share experience during the training phase and make independent decisions during the execution phase; In the training phase, all agents share the global state S and use the Proximal Policy Optimization (PPO) algorithm to optimize the strategy. The specific formula is as follows: in: clip() represents the clipping function, E t Represents the mathematical expectation of the state-action-reward sample data collected during training; ∈＝0.2 Step S42: In the execution phase, agent i observes the local i Independent decision-making: to i ~π i (a|o i ) o i =g(S,i) Where: g(·) is the local observation function, which extracts the state variables of region j; Step S43: The reinforcement learning method uses an attention mechanism and a graph neural network; the graph neural network includes the following contents: h i =σ(WAGG({h j }∪{h i })+b) Among them, parameter b represents the bias term used to adjust the output of the neural network layer; h i represents the hidden state of agent i; h j Represents the hidden state of the region; WAGG represents the feature used to combine the neighbor region j with h i The combined aggregate weight matrix, ; σ represents the nonlinear activation parameter; The attention mechanism weights include the following: Among them, W q Represents the query weight matrix, h i Convert to query vector q i =W q h i , W k represents the key weight matrix, d represents the dimension of the query and key vectors, and the function softmax represents the normalization function that converts the score into a probability distribution.

7. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 1, characterized in that: The step S5 includes the following contents: Step S51: Adopting an adversarial learning mechanism to solve the dynamic game includes using a generative adversarial network (GAN) model to simulate market changes, wherein the game between the GAN generator and the discriminator is: Among them, the parameters Represents the expectation of the real data distribution, D(S) represents the output probability of the discriminator for the real sample S, Represents the discriminator's response to the generator sample The output probability of .

8. The method for dynamic game optimization of power grid investment based on state space decomposition and agent reinforcement learning according to claim 1, characterized in that: The step S6 includes the following contents: Step S61: Output the agent strategy optimization goal, Includes the following: Among them, the parameter E represents the average expectation of all possible trajectories, the parameter T represents the time step or the maximum number of steps in the round, and the parameter J(π i ) is represented as strategy π i The expected total discounted reward of .

9. A power grid investment dynamic game optimization system based on state space decomposition and agent reinforcement learning, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a dynamic game optimization method for power grid investment based on state space decomposition and agent reinforcement learning as described in any one of claims 1 to 8.

10. A power grid investment dynamic game optimization system based on state space decomposition and agent reinforcement learning, comprising a computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, it implements a power grid investment dynamic game optimization method based on state space decomposition and agent reinforcement learning as described in any one of claims 1 to 8.

Images