Source network load storage collaborative optimization method and device based on graph convolution reinforcement learning
Through the method of graph convolution reinforcement learning based on graph convolution, the graph structure of the target distribution network is established and divided into subnet diagrams, which solves the real-time and accuracy of reactive power regulation of photovoltaic inverter in traditional methods, and improves the power system voltage regulation capability and stability.
Patent Information
- Application Number
- CN202411180714.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The traditional source network load storage collaborative optimization method is difficult to control the reactive power of the photovoltaic inverter in real time and accurately, resulting in insufficient voltage regulation capabilities of the power system, affecting the stability and reliability of the system.
Using a method based on graph convolution reinforcement learning, the graph structure of the target distribution network is established and divided into multiple subnet diagrams. Each photovoltaic inverter node corresponds to an agent, and the reactive power is regulated through the coordinated control strategy of the agent. Use graph convolutional networks and action networks to process data and make decisions.
By decomposing the large image into sub-graphs, the data volume and calculation complexity of the agent process are reduced, the calculation efficiency and response speed are improved, the accurate regulation of the reactive power of the photovoltaic inverter is ensured, and the real-time performance of the system is enhanced. stability.
Smart Images

Figure CN119093390A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of voltage control of distribution networks, and in particular to a source-grid-load-storage collaborative optimization method and device based on graph convolution reinforcement learning. Background Art
[0002] In the power system, the coordinated optimization of source, grid, load and storage is a complex and critical issue, which involves the coordinated operation of power sources, grids, loads and energy storage equipment. With the large-scale access of renewable energy and the rapid development of smart grids, the difficulty and complexity of the coordinated optimization of source, grid, load and storage are increasing, and traditional optimization methods are gradually exposing many limitations. In the process of coordinated optimization of source, grid, load and storage, a large amount of data needs to be processed and analyzed in real time to formulate accurate optimization strategies. The traditional approach is to upload this data to the cloud computing center for processing, but as the amount of data continues to increase, the burden on the cloud computing center is also increasing, resulting in slower processing speeds and even data loss or processing errors.
[0003] In addition, existing source-grid-load-storage collaborative optimization methods often only focus on the global optimization goal, while ignoring the characteristics and requirements of local equipment. For example, photovoltaic inverters are important equipment in the power system, and their reactive power regulation has an important impact on the voltage stability and power quality of the power system.
[0004] However, in order to ensure efficient coordinated control between nodes, traditional reactive power control methods often incorporate the overall data into an intelligent model for calculation, which requires a large amount of computing resources, has low computing efficiency, and is difficult to respond and regulate quickly. This may cause the algorithm to perform poorly in the face of complex power grid environments and changing operating conditions, and it is impossible to control the reactive power of photovoltaic inverters in real time and accurately. This will lead to insufficient voltage regulation capabilities of the power system, affecting the stability and reliability of the system. Summary of the invention
[0005] The purpose of the present invention is to solve the problem mentioned in the above background technology that the reactive power of the photovoltaic inverter cannot be controlled in real time and accurately, and to propose a source-grid-load-storage collaborative optimization method and device based on graph convolution reinforcement learning.
[0006] In a first aspect of the present invention, a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning is provided, the method comprising:
[0007] In the strategy learning stage, a first graph structure of the target distribution network is established; the first graph structure includes a plurality of photovoltaic inverter nodes, transmission nodes and load nodes; each photovoltaic inverter node corresponds to a graph convolutional reinforcement learning agent;
[0008] Calculating a voltage influence coefficient matrix between nodes in the first graph structure, and dividing the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs;
[0009] Determine the local observation range of the target intelligent agent according to the subnet diagram of the photovoltaic inverter corresponding to the target intelligent agent; the target intelligent agent is any one of the multiple intelligent agents;
[0010] Centrally training multiple agents based on the global observation quantity of the target distribution network to obtain a collaborative control strategy of the target agent;
[0011] In the collaborative optimization process of source, grid, load and storage, the reactive power of photovoltaic inverter is regulated according to the collaborative control strategy of multiple intelligent agents.
[0012] Optionally, the calculating a voltage influence coefficient matrix between nodes in the first graph structure, and dividing the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs includes:
[0013] According to the first graph structure, power flow calculation is performed to obtain node voltage;
[0014] The reactive power of each photovoltaic inverter node is disturbed respectively, and the power flow calculation is updated to obtain the voltage change of each node;
[0015] According to the voltage change, the voltage disturbance matrix is established:
[0016]
[0017] Among them, A ij is the element of the i-th row and j-th column of the voltage disturbance matrix; ΔU ij is the voltage change of the jth node when the reactive power of the i-th PV inverter node changes; ΔRP i is the change in reactive power of the i-th PV inverter node;
[0018] Normalizing the voltage disturbance matrix to obtain a voltage influence coefficient matrix;
[0019] The first graph structure is divided according to the voltage influence coefficient matrix to obtain a plurality of subnetwork graphs.
[0020] Optionally, the target agent includes a graph convolutional network, a first action network, a target action network, a first evaluation network and a target evaluation network;
[0021] The centralized training of multiple agents to obtain a coordinated control strategy for a target agent includes:
[0022] Obtaining the current global observation amount, and extracting the local observation amount of the target agent according to the distribution of the subnetwork graph, using the local observation amount as the input of the graph convolutional network to obtain the local state embedding feature;
[0023] Obtaining a first action according to the first action network of the target agent and its corresponding local state embedding features;
[0024] Integrate the first actions of multiple agents to obtain a first joint action, and obtain the next global observation value and immediate reward according to the first joint action;
[0025] Using the global observation and the first joint action as inputs of the first evaluation network to obtain a first state-action value;
[0026] According to the next global observation value and the target action network of each agent, a second joint action is obtained; and the next global observation value and the second joint action are used as inputs of the target evaluation network to obtain the target state action value;
[0027] According to the first state action value and the target state action value, updating the parameters of the first evaluation network and the first action network;
[0028] Repeating the learning process of the target agent until the target agent reaches a preset convergence condition;
[0029] After the training is completed, the target action network of the target intelligent agent is extracted as the coordination control strategy of the target intelligent agent.
[0030] Optionally, the first action is a continuous value; and obtaining the next global state and the immediate reward according to the first joint action includes:
[0031] According to the current global state and the first joint action, the reactive power of each photovoltaic inverter is calculated:
[0032]
[0033] Among them, RP j is the reactive power of the jth PV inverter; a j is the action chosen by the jth agent; SP j is the rated capacity of the jth PV inverter; AP j is the active power of the jth PV inverter;
[0034] According to the reactive power of each photovoltaic inverter node, the voltage amplitude of each node is calculated to obtain the voltage vector;
[0035] The last active power, the last reactive power, the current active power, the current reactive power and the voltage vector of each photovoltaic inverter are used as the next global state.
[0036] Optionally, obtaining the next global state and the immediate reward according to the first joint action further includes:
[0037] According to the next global state and the preset target voltage, the immediate reward of the target agent is calculated:
[0038]
[0039] Among them, r t is the desired immediate reward; n is the number of nodes in the next global state; V i is the voltage value of the ith node; f L (V i ) is the reward and penalty value of the voltage deviation of the i-th node from the target voltage; V i,0 is the target voltage of the ith node; is the probability density function of a normal distribution; w, b, c, h, and k are constants.
[0040] A second aspect of the present invention provides a source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning, the device comprising:
[0041] A graph building module, used to build a first graph structure of a target distribution network in a strategy learning phase; the first graph structure includes a plurality of photovoltaic inverter nodes, transmission nodes and load nodes; each photovoltaic inverter node corresponds to a graph convolutional reinforcement learning agent;
[0042] A partitioning module, used to calculate a voltage influence coefficient matrix between nodes in the first graph structure, and divide the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs;
[0043] An agent initialization module, used to determine the local observation range of the target agent according to the subnet diagram of the photovoltaic inverter corresponding to the target agent; the target agent is any one of the multiple agents;
[0044] A training module, used for centrally training multiple agents according to the global observation quantity of the target distribution network to obtain a collaborative control strategy of the target agent;
[0045] The execution module is used to regulate the reactive power of the photovoltaic inverter according to the collaborative control strategy of multiple intelligent agents during the collaborative optimization process of source, grid, load and storage.
[0046] Optionally, the partition module includes:
[0047] A power flow calculation module, used to perform power flow calculation according to the first graph structure to obtain node voltage;
[0048] The voltage change acquisition module is used to disturb the reactive power of each photovoltaic inverter node, update the power flow calculation, and obtain the voltage change of each node;
[0049] The voltage disturbance matrix establishment module is used to establish the voltage disturbance matrix according to the voltage variation:
[0050]
[0051] Among them, A ij is the element of the i-th row and j-th column of the voltage disturbance matrix; ΔU ij is the voltage change of the jth node when the reactive power of the i-th PV inverter node changes; ΔRP i is the change in reactive power of the i-th PV inverter node;
[0052] A voltage influence coefficient matrix determination module is used to normalize the voltage disturbance matrix to obtain a voltage influence coefficient matrix;
[0053] A subgraph division module is used to divide the first graph structure according to the voltage influence coefficient matrix to obtain multiple subnetwork graphs.
[0054] Optionally, the target agent includes a graph convolutional network, a first action network, a target action network, a first evaluation network and a target evaluation network;
[0055] The training module includes:
[0056] A feature extraction module, used to obtain the current global observation amount, and extract the local observation amount of the target agent according to the distribution of the subnetwork graph, and use the local observation amount as the input of the graph convolutional network to obtain the local state embedding feature;
[0057] An action selection module, configured to obtain a first action according to a first action network of a target agent and its corresponding local state embedding features;
[0058] A state transfer module, used to integrate the first actions of multiple agents to obtain a first joint action, and obtain the next global observation value and immediate reward according to the first joint action;
[0059] a first value evaluation module, configured to use the global observation and the first joint action as inputs of the first evaluation network to obtain a first state action value;
[0060] A target value evaluation module is used to obtain a second joint action according to the next global observation value and the target action network of each agent; and use the next global observation value and the second joint action as inputs of the target evaluation network to obtain a target state action value;
[0061] A parameter updating module, used for updating parameters of the first evaluation network and the first action network according to the first state action value and the target state action value;
[0062] A stop judgment module is used to repeat the learning process of the target agent until the target agent reaches a preset convergence condition;
[0063] The strategy confirmation module is used to extract the target action network of the target intelligent agent as the coordination control strategy of the target intelligent agent after completing the training.
[0064] Optionally, the first action is a continuous value; and the state transfer module includes:
[0065] An action execution module is used to calculate the reactive power of each photovoltaic inverter according to the current global state and the first joint action:
[0066]
[0067] Among them, RP j is the reactive power of the jth PV inverter; a j is the action chosen by the jth agent; SP j is the rated capacity of the jth PV inverter; AP j is the active power of the jth PV inverter;
[0068] The voltage change module is used to calculate the voltage amplitude of each node according to the reactive power of each photovoltaic inverter node to obtain the voltage vector;
[0069] The state integration module is used to take the last active power, the last reactive power, the current active power, the current reactive power and the voltage vector of each photovoltaic inverter as the next global observation quantity.
[0070] Optionally, the state transfer module further includes:
[0071] A reward calculation module is used to calculate the instantaneous reward of the target intelligent agent according to the next global state and the preset target voltage:
[0072]
[0073] Among them, r t is the desired immediate reward; n is the number of nodes in the next global state; Vi is the voltage value of the ith node; f L (V i ) is the reward and penalty value of the voltage deviation of the i-th node from the target voltage; V i,0 is the target voltage of the ith node; is the probability density function of a normal distribution; w, b, c, h, and k are constants.
[0074] Beneficial effects of the present invention:
[0075] The present invention proposes a source-grid-load-storage collaborative optimization method based on graph convolution reinforcement learning, the method comprising: in a strategy learning stage, establishing a first graph structure of a target distribution network; the first graph structure comprises a plurality of photovoltaic inverter nodes, a transmission node and a load node; each photovoltaic inverter node corresponds to a graph convolution reinforcement learning agent; calculating a voltage influence coefficient matrix between nodes in the first graph structure, and dividing the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnetwork graphs; determining a local observation range of a target agent according to a subnetwork graph where a photovoltaic inverter corresponding to the target agent is located; the target agent is any one of a plurality of agents; according to a global observation quantity of the target distribution network, centrally training a plurality of agents to obtain a collaborative control strategy of the target agent; in a collaborative optimization process of the source-grid-load-storage, regulating the reactive power of the photovoltaic inverter according to the collaborative control strategy of the plurality of agents.
[0076] By decomposing the originally complex large graph into multiple smaller subgraphs, the amount of data that each agent needs to process is effectively reduced, the computational complexity is reduced, and the overall computational efficiency is improved. In addition, the agent only needs to communicate with other nodes in its subgraph, which reduces the communication overhead of the overall system and improves the real-time performance and response speed of the control system. During the centralized training process, global data can be used for high-quality strategy evaluation and verification, so that deviations and errors in the strategy can be discovered and corrected in a timely manner, ensuring that the agent can make more accurate decisions during execution. BRIEF DESCRIPTION OF THE DRAWINGS
[0077] The present invention will be further described below in conjunction with the accompanying drawings.
[0078] Figure 1 A flowchart of a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning is provided for an embodiment of the present invention;
[0079] Figure 2 A structural diagram of a source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning is provided for an embodiment of the present invention. DETAILED DESCRIPTION
[0080] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0081] The embodiment of the present invention provides a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning. Figure 1 , Figure 1 A flow chart of a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning provided in an embodiment of the present invention. The method comprises the following steps:
[0082] S101, in the strategy learning phase, a first graph structure of a target distribution network is established.
[0083] S102, calculating a voltage influence coefficient matrix between nodes in a first graph structure, and dividing the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs.
[0084] S103, determining the local observation range of the target intelligent agent according to the subnet map where the photovoltaic inverter corresponding to the target intelligent agent is located.
[0085] S104, based on the global observation quantity of the target distribution network, multiple intelligent agents are centrally trained to obtain a collaborative control strategy for the target intelligent agent.
[0086] S105, in the collaborative optimization process of source, grid, load and storage, the reactive power of the photovoltaic inverter is regulated according to the collaborative control strategy of multiple intelligent agents.
[0087] Among them, the first graph structure includes multiple photovoltaic inverter nodes, transmission nodes and load nodes; each photovoltaic inverter node corresponds to a graph convolutional reinforcement learning agent; the target agent is any one of the multiple agents.
[0088] Based on a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning provided by an embodiment of the present invention, by decomposing the originally complex large graph into multiple smaller subgraphs, the amount of data that each intelligent agent needs to process is effectively reduced, the computational complexity is reduced, and the overall computational efficiency is improved. In addition, the intelligent agent only needs to communicate with other nodes in its subgraph, which reduces the communication overhead of the overall system and improves the real-time performance and response speed of the system. During the centralized training process, global data can be used for high-quality strategy evaluation and verification, so that deviations and errors in the strategy can be discovered and corrected in a timely manner, ensuring that the intelligent agent can make more accurate decisions during execution.
[0089] In one embodiment, step S102 includes:
[0090] Step 1: Perform power flow calculation according to the first graph structure to obtain node voltage.
[0091] Step 2: disturb the reactive power of each photovoltaic inverter node respectively, update the power flow calculation, and obtain the voltage change of each node.
[0092] Step 3: Establish the voltage disturbance matrix according to the voltage variation:
[0093]
[0094] Among them, A ij is the i-th row and j-th column element of the voltage disturbance matrix; ΔU ij is the voltage change of the jth node when the reactive power of the i-th PV inverter node changes; ΔRP i is the change in reactive power of the i-th PV inverter node.
[0095] Step 4: Normalize the voltage disturbance matrix to obtain the voltage influence coefficient matrix.
[0096] Step five: divide the first graph structure according to the voltage influence coefficient matrix to obtain multiple subnet graphs.
[0097] In one implementation, by dividing the subgrid diagram, a complex power system can be decomposed into several relatively independent subsystems to simplify management and control.
[0098] In one implementation method, by disturbing the reactive power, the influence of each photovoltaic inverter node on the voltage of the entire system can be clarified, and the area is partitioned according to the influence of each photovoltaic inverter, that is, the local observation range of the intelligent agent is divided, which can ensure the rationality of the regional division and the reliability of the decision-making basis of the intelligent agent.
[0099] In one implementation, the nodes can be clustered by setting the voltage coupling threshold. Each photovoltaic inverter node is used as a cluster head. The voltage influence coefficient matrix is denoted as B, then B ij means the voltage influence coefficient of the i-th photovoltaic inverter on the j-th node; select the largest element B in each column km , if the element is greater than the preset voltage coupling threshold, the mth node is included in the cluster where node k is located. This cluster classification method has certain flexibility and scalability, and can meet the needs of photovoltaic systems of different scales and application scenarios.
[0100] In one embodiment, the target agent includes a graph convolutional network, a first action network, a target action network, a first evaluation network, and a target evaluation network;
[0101] Step S104 includes:
[0102] Step 1: Get the current global observations, and extract the local observations of the target agent according to the distribution of the subnetwork graph. Use the local observations as the input of the graph convolutional network to obtain the local state embedding features.
[0103] Step 2: Obtain the first action based on the first action network of the target agent and its corresponding local state embedding features.
[0104] Step three: Integrate the first actions of multiple agents to obtain the first joint action, and obtain the next global observation value and immediate reward based on the first joint action.
[0105] Step 4: Use the global observation and the first joint action as the input of the first evaluation network to obtain the first state action value.
[0106] Step 5: According to the next global observation and the target action network of each agent, the second joint action is obtained; and the next global observation and the second joint action are used as inputs of the target evaluation network to obtain the target state action value.
[0107] Step six, according to the first state action value and the target state action value, update the parameters of the first evaluation network and the first action network.
[0108] Step seven, repeat the learning process of the target agent until the target agent reaches the preset convergence condition.
[0109] Step 8: After the training is completed, the first action network of the target intelligent agent is extracted as the coordination control strategy of the target intelligent agent.
[0110] In one implementation, the action network of the agent makes decisions based on the local state, and the evaluation network evaluates based on the global state, thereby optimizing the action network. In this way, while reducing the complexity of the algorithm and improving the computational efficiency, it ensures the coordination between the action selection strategies of multiple agents, so that the actions meet the global optimum.
[0111] In one implementation, during initialization, the parameters of the target action network are consistent with those of the first action network; the parameters of the target evaluation network are consistent with those of the first evaluation network; during training, the target network is regularly updated by a soft update method:
[0112] θ(π′)←τθ(π)+(1-τ)θ(π′)
[0113] θ(q′)←τθ(q)+(1-τ)θ(q′)
[0114] Among them, θ(π′) is the parameter of the target action network; θ(π) is the parameter of the first action network; θ(q′) is the parameter of the target evaluation network; θ(q) is the parameter of the first evaluation network; τ is the network update rate, which can be 0.001. By setting the target network, the fluctuation in the training process can be slowed down to a certain extent, the stability can be improved, and it helps the agent converge to better performance.
[0115] In one embodiment, the first action is a continuous value; and according to the first joint action, obtaining the next global state and the immediate reward comprises:
[0116] Step 1: Calculate the reactive power of each photovoltaic inverter based on the current global state and the first joint action:
[0117]
[0118] Among them, RP j is the reactive power of the jth PV inverter; a j is the action chosen by the jth agent; SP j is the rated capacity of the jth PV inverter; AP j is the active power of the jth PV inverter.
[0119] Step 2: According to the reactive power of each photovoltaic inverter node, the voltage amplitude of each node is calculated to obtain the voltage vector.
[0120] Step 3: The last active power, the last reactive power, the current active power, the current reactive power and the voltage vector of each photovoltaic inverter are used as the next global state.
[0121] In one implementation, the range of action values is selected based on the load capacity of the lines and transformers in the distribution network to ensure the safe operation of the photovoltaic inverter. j When it is greater than zero, it means that the photovoltaic inverter injects reactive power into the distribution network; when a j When it is less than zero, it means that the PV inverter absorbs reactive power from the distribution network.
[0122] In one embodiment, according to the first joint action, obtaining the next global state and the immediate reward further includes:
[0123] According to the next global state and the preset target voltage, calculate the immediate reward of the target agent:
[0124]
[0125] Among them, r t is the desired immediate reward; n is the number of nodes in the next global state; V i is the voltage value of the ith node; f L(V i ) is the reward and penalty value of the voltage deviation of the i-th node from the target voltage; V i,0 is the target voltage of the ith node; is the probability density function of a normal distribution; w, b, c, h, and k are constants.
[0126] In one implementation, It can be a normal distribution N(V i,0 , 1) is the probability density function.
[0127] In one implementation, after executing the action given by the agent, if the node voltage exceeds the preset range, a negative reward is given. If it does not exceed the range, a positive reward is given according to the amount of deviation from the target voltage. The larger the deviation, the smaller the reward. This encourages the agent to learn in the direction of stabilizing the node voltage.
[0128] The embodiment of the present invention provides a source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning. Figure 2 , Figure 2 A structural diagram of a source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning provided in an embodiment of the present invention. The device includes:
[0129] The graph building module is used to build the first graph structure of the target distribution network during the strategy learning phase.
[0130] The partitioning module is used to calculate the voltage influence coefficient matrix between nodes in the first graph structure, and divide the first graph structure according to the voltage influence coefficient matrix to obtain multiple subnet graphs.
[0131] The agent initialization module is used to determine the local observation range of the target agent according to the subnet diagram where the photovoltaic inverter corresponding to the target agent is located.
[0132] The training module is used to centrally train multiple intelligent agents according to the global observation quantity of the target distribution network to obtain the collaborative control strategy of the target intelligent agent.
[0133] The execution module is used to regulate the reactive power of the photovoltaic inverter according to the collaborative control strategy of multiple intelligent agents during the collaborative optimization process of source, grid, load and storage.
[0134] Among them, the first graph structure includes multiple photovoltaic inverter nodes, transmission nodes and load nodes; each photovoltaic inverter node corresponds to a graph convolutional reinforcement learning agent; the target agent is any one of the multiple agents.
[0135] Based on a source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning provided by an embodiment of the present invention, by decomposing the originally complex large graph into multiple smaller subgraphs, the amount of data that each intelligent agent needs to process is effectively reduced, the computational complexity is reduced, and the overall computational efficiency is improved. In addition, the intelligent agent only needs to communicate with other nodes in its subgraph, which reduces the communication overhead of the overall system and improves the real-time performance and response speed of the system. During the centralized training process, global data can be used for high-quality strategy evaluation and verification, so that deviations and errors in the strategy can be discovered and corrected in a timely manner, ensuring that the intelligent agent can make more accurate decisions during execution.
[0136] In one embodiment, the partition module includes:
[0137] The power flow calculation module is used to perform power flow calculation according to the first graph structure to obtain the node voltage.
[0138] The voltage change acquisition module is used to disturb the reactive power of each photovoltaic inverter node respectively, update the power flow calculation, and obtain the voltage change of each node.
[0139] The voltage disturbance matrix establishment module is used to establish the voltage disturbance matrix according to the voltage variation:
[0140]
[0141] Among them, A ij is the i-th row and j-th column element of the voltage disturbance matrix; ΔU ij is the voltage change of the jth node when the reactive power of the i-th PV inverter node changes; ΔRP i is the change in reactive power of the i-th PV inverter node.
[0142] The voltage influence coefficient matrix determination module is used to normalize the voltage disturbance matrix to obtain the voltage influence coefficient matrix.
[0143] The subgraph division module is used to divide the first graph structure according to the voltage influence coefficient matrix to obtain multiple subnetwork graphs.
[0144] In one embodiment, the target agent includes a graph convolutional network, a first action network, a target action network, a first evaluation network, and a target evaluation network;
[0145] The training modules include:
[0146] The feature extraction module is used to obtain the current global observation and extract multiple local observations according to the distribution of the subnetwork graph. The local observations are used as the input of the graph convolutional network to obtain the local state embedding features.
[0147] The action selection module is used to obtain the first action according to the first action network of the target intelligent agent and its corresponding local state embedding features.
[0148] The state transfer module is used to integrate the first actions of multiple agents to obtain a first joint action, and obtain the next global observation value and immediate reward based on the first joint action.
[0149] The first value evaluation module is used to use the global observation and the first joint action as inputs of the first evaluation network to obtain the first state action value.
[0150] The target value evaluation module is used to obtain the second joint action according to the next global observation and the target action network of each intelligent agent; and use the next global observation and the second joint action as the input of the target evaluation network to obtain the target state action value.
[0151] The parameter updating module is used to update the parameters of the first evaluation network and the first action network according to the first state action value and the target state action value.
[0152] The stop judgment module is used to repeat the learning process of the target agent until the target agent reaches the preset convergence condition.
[0153] The strategy confirmation module is used to extract the first action network of the target intelligent agent as the coordination control strategy of the target intelligent agent after completing the training.
[0154] In one embodiment, the first action is a continuous value; the state transfer module includes:
[0155] The action execution module is used to calculate the reactive power of each photovoltaic inverter according to the current global state and the first joint action:
[0156]
[0157] Among them, RP j is the reactive power of the jth PV inverter; a j is the action chosen by the jth agent; SP j is the apparent power of the jth PV inverter; AP j is the active power of the jth PV inverter.
[0158] The voltage change module is used to calculate the voltage amplitude of each node according to the reactive power of each photovoltaic inverter node to obtain the voltage vector.
[0159] The state integration module is used to take the last active power, the last reactive power, the current active power, the current reactive power and the voltage vector of each photovoltaic inverter as the next global observation quantity.
[0160] In one embodiment, the state transfer module further includes:
[0161] The reward calculation module is used to calculate the immediate reward of the target agent based on the next global state and the preset target voltage:
[0162]
[0163] Among them, r t is the desired immediate reward; n is the number of nodes in the next global state; V i is the voltage value of the ith node; f L (V i ) is the reward and penalty value of the voltage deviation of the i-th node from the target voltage; V i,0 is the target voltage of the ith node; is the probability density function of a normal distribution; w, b, c, h, and k are constants.
[0164] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.
Claims
1. A source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning, characterized in that: The method comprises: In the strategy learning stage, a first graph structure of the target distribution network is established; the first graph structure includes a plurality of photovoltaic inverter nodes, transmission nodes and load nodes; each photovoltaic inverter node corresponds to a graph convolutional reinforcement learning agent; Calculating a voltage influence coefficient matrix between nodes in the first graph structure, and dividing the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs; Determine the local observation range of the target intelligent agent according to the subnet diagram of the photovoltaic inverter corresponding to the target intelligent agent; the target intelligent agent is any one of the multiple intelligent agents; Centrally training multiple agents based on the global observation quantity of the target distribution network to obtain a collaborative control strategy of the target agent; In the collaborative optimization process of source, grid, load and storage, the reactive power of photovoltaic inverter is regulated according to the collaborative control strategy of multiple intelligent agents.
2. According to claim 1, a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning is characterized in that: The step of calculating a voltage influence coefficient matrix between nodes in the first graph structure and dividing the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs includes: According to the first graph structure, power flow calculation is performed to obtain node voltage; The reactive power of each photovoltaic inverter node is disturbed respectively, and the power flow calculation is updated to obtain the voltage change of each node; According to the voltage change, the voltage disturbance matrix is established: Among them, A ij is the element of the i-th row and j-th column of the voltage disturbance matrix; ΔU ij is the voltage change of the jth node when the reactive power of the i-th PV inverter node changes; ΔRP i is the change in reactive power of the i-th PV inverter node; Normalizing the voltage disturbance matrix to obtain a voltage influence coefficient matrix; The first graph structure is divided according to the voltage influence coefficient matrix to obtain a plurality of subnetwork graphs.
3. According to the method of source-grid-load-storage collaborative optimization based on graph convolutional reinforcement learning in claim 1, it is characterized in that: The target agent includes a graph convolutional network, a first action network, a target action network, a first evaluation network and a target evaluation network; The centralized training of multiple agents to obtain a coordinated control strategy for a target agent includes: Obtaining the current global observation amount, and extracting the local observation amount of the target agent according to the distribution of the subnetwork graph, using the local observation amount as the input of the graph convolutional network to obtain the local state embedding feature; Obtaining a first action according to the first action network of the target agent and its corresponding local state embedding features; Integrate the first actions of multiple agents to obtain a first joint action, and obtain the next global observation value and immediate reward according to the first joint action; Using the global observation and the first joint action as inputs of the first evaluation network to obtain a first state-action value; According to the next global observation value and the target action network of each agent, a second joint action is obtained; and the next global observation value and the second joint action are used as inputs of the target evaluation network to obtain the target state action value; According to the first state action value and the target state action value, updating the parameters of the first evaluation network and the first action network; Repeating the learning process of the target agent until the target agent reaches a preset convergence condition; After the training is completed, the target action network of the target intelligent agent is extracted as the coordination control strategy of the target intelligent agent.
4. According to claim 3, a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning is characterized in that: The first action is a continuous value; and obtaining the next global state and the immediate reward according to the first joint action includes: According to the current global state and the first joint action, the reactive power of each photovoltaic inverter is calculated: Among them, RP j is the reactive power of the jth PV inverter; a j is the action chosen by the jth agent; SP j is the rated capacity of the jth PV inverter; AP j is the active power of the jth PV inverter; According to the reactive power of each photovoltaic inverter node, the voltage amplitude of each node is calculated to obtain the voltage vector; The last active power, the last reactive power, the current active power, the current reactive power and the voltage vector of each photovoltaic inverter are used as the next global state.
5. According to claim 3, a source-grid-load-storage collaborative optimization method based on graph convolutional reinforcement learning is characterized in that: The obtaining of the next global state and the immediate reward according to the first joint action further comprises: According to the next global state and the preset target voltage, the immediate reward of the target agent is calculated: Among them, r t is the desired immediate reward; n is the number of nodes in the next global state; V i is the voltage value of the ith node; f L (V i ) is the reward and penalty value of the voltage deviation of the i-th node from the target voltage; V i,0 is the target voltage of the ith node; is the probability density function of a normal distribution; w, b, c, h, and k are constants.
6. A source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning, characterized in that: The device comprises: A graph building module, used to build a first graph structure of a target distribution network in a strategy learning phase; the first graph structure includes a plurality of photovoltaic inverter nodes, transmission nodes and load nodes; each photovoltaic inverter node corresponds to a graph convolutional reinforcement learning agent; A partitioning module, used to calculate a voltage influence coefficient matrix between nodes in the first graph structure, and divide the first graph structure according to the voltage influence coefficient matrix to obtain a plurality of subnet graphs; An agent initialization module, used to determine the local observation range of the target agent according to the subnet diagram of the photovoltaic inverter corresponding to the target agent; the target agent is any one of the multiple agents; A training module, used for centrally training multiple agents according to the global observation quantity of the target distribution network to obtain a collaborative control strategy of the target agent; The execution module is used to regulate the reactive power of the photovoltaic inverter according to the collaborative control strategy of multiple intelligent agents during the collaborative optimization process of source, grid, load and storage.
7. The source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning according to claim 6, characterized in that: The partition module comprises: A power flow calculation module, used to perform power flow calculation according to the first graph structure to obtain node voltage; The voltage change acquisition module is used to disturb the reactive power of each photovoltaic inverter node, update the power flow calculation, and obtain the voltage change of each node; The voltage disturbance matrix establishment module is used to establish the voltage disturbance matrix according to the voltage variation: Among them, A ij is the element of the i-th row and j-th column of the voltage disturbance matrix; ΔU ij is the voltage change of the jth node when the reactive power of the i-th PV inverter node changes; ΔRP i is the change in reactive power of the i-th PV inverter node; A voltage influence coefficient matrix determination module is used to normalize the voltage disturbance matrix to obtain a voltage influence coefficient matrix; A subgraph division module is used to divide the first graph structure according to the voltage influence coefficient matrix to obtain multiple subnetwork graphs.
8. The source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning according to claim 6, characterized in that: The target agent includes a graph convolutional network, a first action network, a target action network, a first evaluation network and a target evaluation network; The training module includes: A feature extraction module, used to obtain the current global observation amount, and extract the local observation amount of the target agent according to the distribution of the subnetwork graph, and use the local observation amount as the input of the graph convolutional network to obtain the local state embedding feature; An action selection module, configured to obtain a first action according to a first action network of a target agent and its corresponding local state embedding features; A state transfer module, used to integrate the first actions of multiple agents to obtain a first joint action, and obtain the next global observation value and immediate reward according to the first joint action; a first value evaluation module, configured to use the global observation and the first joint action as inputs of the first evaluation network to obtain a first state action value; A target value evaluation module is used to obtain a second joint action according to the next global observation value and the target action network of each agent; and use the next global observation value and the second joint action as inputs of the target evaluation network to obtain a target state action value; A parameter updating module, used for updating parameters of the first evaluation network and the first action network according to the first state action value and the target state action value; A stop judgment module is used to repeat the learning process of the target agent until the target agent reaches a preset convergence condition; The strategy confirmation module is used to extract the target action network of the target intelligent agent as the coordination control strategy of the target intelligent agent after completing the training.
9. The source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning according to claim 8, characterized in that: The first action is a continuous value; the state transition module includes: An action execution module is used to calculate the reactive power of each photovoltaic inverter according to the current global state and the first joint action: Among them, RP j is the reactive power of the jth PV inverter; a j is the action chosen by the jth agent; SP j is the rated capacity of the jth PV inverter; AP j is the active power of the jth PV inverter; The voltage change module is used to calculate the voltage amplitude of each node according to the reactive power of each photovoltaic inverter node to obtain the voltage vector; The state integration module is used to take the last active power, the last reactive power, the current active power, the current reactive power and the voltage vector of each photovoltaic inverter as the next global observation quantity.
10. The source-grid-load-storage collaborative optimization device based on graph convolutional reinforcement learning according to claim 8, characterized in that: The state transfer module also includes: A reward calculation module is used to calculate the instantaneous reward of the target intelligent agent according to the next global state and the preset target voltage: Among them, r t is the desired immediate reward; n is the number of nodes in the next global state; V i is the voltage value of the ith node; f L (V i ) is the reward and penalty value of the voltage deviation of the i-th node from the target voltage; V i,0 is the target voltage of the ith node; is the probability density function of a normal distribution; w, b, c, h, and k are constants.
Citation Information
Patent Citations
Active power distribution network real-time voltage control method based on multi-agent deep reinforcement learning
CN117674160A
