An agent and power grid environment interactive learning system and method, and a storage medium

Abstract

This invention relates to an interactive learning system and method for intelligent agents and power grid environments, as well as a storage medium. Using cross-sectional power flow files as an intermediate medium, the reinforcement learning agent, state estimation module, and power flow calculation module interact and link through interfaces to train the reinforcement learning agent. This method can effectively reuse advanced application software such as state estimation and dispatcher power flow calculation in the control system, reducing modeling complexity while making the agent training environment closer to the real power grid environment and improving accuracy.

Description

Technical Field

[0001] This invention relates to the field of power grid control technology, specifically to an intelligent agent and power grid environment interactive learning system and method, and storage medium. Background Technology

[0002] With the continuous development of my country's market economy, the construction of ultra-high voltage power grids and AC / DC hybrid power grids, the grid connection and operation of distributed microgrids, and the high-proportion continuous integration of renewable energy, the operation and control of power grids are facing increasing challenges. Reinforcement learning technology offers a new solution for solving highly nonlinear, high-dimensional, and real-time power grid control and decision-making problems. In power grid control and decision-making tasks, reinforcement learning agents cannot interact with the real power grid environment during training, leading to inflexible operation and an inability to cope with abnormal environments. Therefore, it is usually necessary to construct a simulation environment based on the real power grid.

[0003] Due to the large number of components and the complexity of topology and constraints in the real power grid environment, constructing a simulation environment is extremely time-consuming and labor-intensive. Existing methods use simulation environments to build agent interaction environments, which differ significantly from the real power grid environment, resulting in poor accuracy of control and decision-making results. If the trained agent model is applied to the actual power grid production environment for online decision-making, a large amount of migration and adaptation work is still required. Summary of the Invention

[0004] This invention provides an interactive learning system and method for intelligent agents and power grid environments. Addressing the complex and cumbersome construction of power grid simulation environments, especially power flow solvers, this invention utilizes cross-sectional power flow files as an intermediate medium. Modules such as reinforcement learning agents, state estimation modules, and power flow calculation modules interact and link through interfaces to train the agent. This method effectively reuses advanced application software such as state estimation and dispatcher power flow calculation in control systems, reducing modeling complexity while making the agent training environment closer to the real power grid environment and improving accuracy.

[0005] The present invention provides the following solution to the above-mentioned technical problems: an intelligent agent and power grid environment interactive learning system, the system comprising a state estimation module, an interaction service module, a reinforcement learning intelligent agent, and a power flow calculation module; the interaction service module is connected to the state estimation module, the reinforcement learning intelligent agent, and the power flow calculation module respectively through interfaces;

[0006] The state estimation module is used to read the power grid information at each historical time point, generate historical cross-sectional power flow files based on the power grid information, and obtain a set of historical cross-sectional power flow files corresponding to the expected target by classifying the historical cross-sectional power flow files at each historical time point according to the expected target.

[0007] The interactive service module is used to randomly select a target historical cross-section power flow file from the set of historical cross-section power flow files corresponding to the expected target learned by the agent, parse the target historical cross-section power flow file to obtain the target historical power grid state information, and then send the target historical power grid state information to the reinforcement learning agent.

[0008] The reinforcement learning agent is used to predict the action to be performed based on the target historical power grid state information, and sends the predicted action to the interaction service module so that the interaction service module can modify the target historical cross-section power flow file according to the predicted action, and send the modified target historical cross-section power flow file to the power flow calculation module.

[0009] The power flow calculation module is used to calculate the historical power flow based on the modified target historical cross-section power flow file, and generate a new historical cross-section power flow file by combining the calculation results of the historical power flow, and send the newly generated historical cross-section power flow file to the interactive service module.

[0010] Preferably, the interactive service module is further configured to parse the newly generated historical cross-sectional power flow file to form new power grid state information, generate a reward value based on the new power grid state information and the predicted execution action, determine whether the round has ended based on the new power grid state information, and send the reward value and the result of whether the round has ended to the reinforcement learning agent, so that the reinforcement learning agent can perform corresponding operations according to the result of whether the round has ended and determine whether the training meets the preset termination condition based on the reward value.

[0011] Preferably, the method for generating the reward value includes:

[0012] When the interactive service module determines that the predicted action is an illegal action, the reward value is set to negative; when the interactive service module determines that the predicted action is not an illegal action, the power grid efficiency value in the new power grid status information is extracted to generate a reward value. The higher the power grid efficiency value, the larger the reward value, and the lower the power grid efficiency value, the smaller the reward value.

[0013] Grid efficiency is the ratio of the total demand of all electrical loads to the total supply of all generating units. Unauthorized actions include reconnecting transmission lines during maintenance or modifying settings while components are cooling down.

[0014] Preferably, the method for determining whether a round has ended includes:

[0015] The interactive service module determines whether the reinforcement learning agent can manage the power grid normally based on the new power grid status information. If it can, the round ends and the result is sent to the reinforcement learning agent, allowing the agent to enter the next round. If it cannot, the round does not end and the result is sent to the reinforcement learning agent, allowing the agent to return to the action steps predicted based on the target's historical power grid status information and execute subsequent actions until the round ends.

[0016] The reinforcement learning agent is unable to continue to manage the power grid normally in the following two situations: First, when there are too many disconnected transmission lines, the transmission lines cannot transmit the power required by the load in time, which cannot meet the power consumption, resulting in forced power restriction or even large-scale power outages, or more than one generator unit being disconnected from the power grid; Second, the agent generates an erroneous action, causing the power grid environment to be disintegrated, forming several isolated and unconnected power grid topology subgraphs.

[0017] Preferably, the power grid information includes power grid model parameters, wiring connections, and a set of redundant telemetry measurements and remote signaling switch statuses.

[0018] Preferably, the state estimation module is used to generate cross-sectional power flow files through network topology analysis and calculation using the power grid information.

[0019] Preferably, the preset training termination conditions include reaching the preset maximum number of training steps, training to the last round, and the reward value no longer increasing.

[0020] A method for intelligent agents to learn through interaction with the power grid environment includes the following steps:

[0021] The state estimation module reads the power grid information at each historical time point and generates historical cross-sectional power flow files based on the power grid information. Based on the historical cross-sectional power flow files at each historical time point, it classifies them according to the expected target learned by the agent to obtain a set of historical cross-sectional power flow files corresponding to the expected target.

[0022] The interactive service module randomly selects a target historical cross-section power flow file from the set of historical cross-section power flow files corresponding to the expected target learned by the agent, parses the target historical cross-section power flow file to obtain the target historical power grid state information, and then sends the target historical power grid state information to the reinforcement learning agent.

[0023] The reinforcement learning agent predicts and executes actions based on the target's historical power grid state information and sends the results to the interaction service module.

[0024] The interactive service module modifies the target historical section power flow file according to the action to be executed, and then sends the modified target historical section power flow file to the power flow calculation module.

[0025] The power flow calculation module calculates the historical power flow based on the modified target historical section power flow file, and generates a new historical section power flow file by combining the calculation results of the historical power flow. The newly generated historical section power flow file is then sent to the interactive service module.

[0026] The interactive service module receives and parses the newly generated historical cross-sectional power flow file to form new power grid state information. Based on the new power grid state information and the predicted execution actions, it generates a reward value, determines whether the round has ended based on the new power grid state information, and sends the reward value and the result of whether the round has ended to the reinforcement learning agent.

[0027] The reinforcement learning agent receives the reward value and the result of whether the round has ended, executes the action of whether the round has ended, and determines whether the training meets the preset termination condition based on the reward value.

[0028] The present invention also provides a method for intelligent agents to interact with the power grid environment, comprising:

[0029] Obtain real-time power grid information;

[0030] Real-time power grid information is input into the intelligent agent and the power grid environment interaction system, and power grid control operation actions are output.

[0031] The intelligent agent-grid environment interaction system is trained using an intelligent agent-grid environment interaction learning method as described above.

[0032] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the intelligent agent and power grid environment interaction learning method described above.

[0033] The present invention also provides an electronic device, including a memory and a processor:

[0034] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, they implement the steps of the intelligent agent and power grid environment interaction learning method described above.

[0035] The beneficial effects of this invention are as follows:

[0036] 1. This invention addresses the complex and cumbersome construction of power grid simulation environments, especially power flow solvers. It utilizes cross-sectional power flow files as an intermediate medium, enabling interactive linkage between reinforcement learning agents, state estimation modules, and dispatcher power flow calculation modules through interfaces to train the reinforcement learning agent. This method effectively reuses advanced application software such as state estimation and dispatcher power flow calculation in the control system, reducing modeling complexity while enabling rapid construction of a power grid reinforcement learning simulation environment. This saves significant time in building power flow solvers and ensures that the agent training environment more closely resembles the real power grid environment.

[0037] 2. This invention enables targeted training of the model by generating power flow profile files according to the expected objectives.

[0038] 3. This invention improves the accuracy of the model by setting reward values ​​and whether the round ends.

[0039] 4. When the trained agent is applied to the actual power grid production environment for online decision-making, the agent-power grid environment interaction method constructed by this invention can be smoothly adapted to the advanced application software of the control system in the actual environment, thereby improving the efficiency of migration and adaptation.

[0040] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it according to the contents of the specification, the preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. Specific embodiments of the present invention are given in detail below with reference to the accompanying drawings. Attached Figure Description

[0041] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention.

[0042] Figure 1 This is a system diagram of the method described in an embodiment of the present invention. Detailed Implementation

[0043] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0044] Example 1

[0045] A learning system for interaction between an intelligent agent and the power grid environment, comprising a state estimation module, an interaction service module, a reinforcement learning intelligent agent, and a power flow calculation module;

[0046] The state estimation module is used to read power grid information at various historical time points, and generate historical cross-sectional power flow files based on the power grid information through network topology analysis and calculation. Based on the historical cross-sectional power flow files at each historical time point, a set of historical cross-sectional power flow files corresponding to the desired target is obtained. The power grid information includes power grid model parameters, wiring connections, and a set of redundant telemetry measurements and remote signaling switch states.

[0047] The interactive service module is used to randomly select a target historical cross-section power flow file from the set of historical cross-section power flow files corresponding to the expected target learned by the agent, parse the target historical cross-section power flow file to obtain the target historical power grid state information, and then send the target historical power grid state information to the reinforcement learning agent.

[0048] The reinforcement learning agent is used to predict the action to be performed based on the target historical power grid state information, and sends the predicted action to the interaction service module so that the interaction service module can modify the target historical cross-section power flow file according to the predicted action, and send the modified target historical cross-section power flow file to the power flow calculation module.

[0049] The power flow calculation module is used to calculate the historical power flow based on the modified target historical cross-section power flow file, and generate a new historical cross-section power flow file by combining the calculation results of the historical power flow, and send the newly generated historical cross-section power flow file to the interactive service module.

[0050] The interactive service module is also used to parse the newly generated historical cross-sectional power flow file to form new power grid state information, generate reward values ​​based on the new power grid state information and predicted execution actions, determine whether the round has ended based on the new power grid state information, and send the reward value and the result of whether the round has ended to the reinforcement learning agent, so that the reinforcement learning agent can perform corresponding operations according to the result of whether the round has ended and determine whether the training meets the preset termination conditions based on the reward value.

[0051] The reward value is generated in the following ways:

[0052] When the interactive service module determines that the predicted action is an illegal action, the reward value is set to negative; when the interactive service module determines that the predicted action is not an illegal action, the power grid efficiency value in the new power grid status information is extracted to generate a reward value. The higher the power grid efficiency value, the larger the reward value, and the lower the power grid efficiency value, the smaller the reward value.

[0053] The methods for determining whether a round has ended include:

[0054] The interactive service module determines whether the reinforcement learning agent can manage the power grid normally based on the new power grid status information. If it can, the round ends and the result is sent to the reinforcement learning agent, allowing the agent to enter the next round. If it cannot, the round does not end and the result is sent to the reinforcement learning agent, allowing the agent to return to the action steps predicted based on the target's historical power grid status information and execute subsequent actions until the round ends.

[0055] The preset training termination conditions include reaching the preset maximum number of training steps, training to the last round, and the reward value no longer increasing.

[0056] Example 2

[0057] A method for intelligent agents to learn through interaction with the power grid environment includes the following steps:

[0058] Step 1: The state estimation module reads power grid information from various historical time points collected by the data acquisition and monitoring control system in the power grid environment through an interface. It then filters the power grid information provided by the data acquisition and monitoring system to improve data accuracy and eliminate interference from erroneous information, thereby generating historical cross-sectional power flow files. The historical cross-sectional power flow files from various historical time points are then summarized according to the expected goals of the reinforcement learning agent (i.e., adding category labels to the power flow files, such as including new energy units, including power grid anomalies and faults, etc.), forming a set of historical cross-sectional power flow files, such as S = {QS1, QS2, QS3, ..., QS...}. n}, where QS is a commonly used cross-sectional power flow file format in power systems.

[0059] Step 2: The interactive service module selects a set of historical cross-sectional power flow files S corresponding to the expected goals of the agent training (e.g., the task goal is renewable energy consumption, prioritizing cross-sectional power flow files related to renewable energy equipment such as wind power and photovoltaics). It then randomly selects one historical cross-sectional power flow file A from the set, parses it to form the target historical grid status information, such as state = {"gen_p":[100,200,100,150,100],"gen_v":[220.5,220,220,219.5,220],"line_status":[0,1,1,1,1]……}, where gen_p is the active power output of the generator unit, gen_v is the generator voltage amplitude, and line_status is the line status. This target historical grid status information is then sent to the reinforcement learning agent.

[0060] Step 3: The reinforcement learning agent receives the target's historical power grid state information, performs calculations on it, and predicts the action to be performed, such as action={"adjust_gen_p":[1.17,-0.11,2.01,1.65,0],"adjust_gen_v":[-0.05,-0.05,-0.06,0.06.5,0.12]……}, where adjust_gen_p is the unit's active power output adjustment value and adjust_gen_v is the unit's voltage adjustment value, and sends it to the interactive service module.

[0061] Step 4: The interactive service module receives the action to be executed, modifies the corresponding target historical section power flow file A (e.g., according to the output action, the active power output of generator No. 1 is increased by 1.17 MW, then the active power value of generator No. 1 in the power flow file is increased by 1.17), generates a new historical section power flow file B, and then sends the historical section power flow file B to the power flow calculation module.

[0062] Step 5: The power flow calculation module calculates the historical power flow based on the historical cross-sectional power flow file B, and generates a new cross-sectional power flow file C by combining the calculation results of the historical power flow (such as power flow convergence, overload, and heavy load). The newly generated historical cross-sectional power flow file C is then sent to the interactive service module.

[0063] Step 6: The interactive service module parses the cross-sectional power flow file C, ultimately forming new power grid state information, generating reward values ​​based on the new power grid state information and predicted execution actions, determining whether the round has ended, and sending the reward value and the result of whether the round has ended to the reinforcement learning agent.

[0064] The reward value is generated in the following ways:

[0065] When the interactive service module determines that the predicted action is an illegal action, the reward value is set to negative; when the interactive service module determines that the predicted action is not an illegal action, the power grid efficiency value in the new power grid status information is extracted to generate a reward value. The higher the power grid efficiency value, the larger the reward value, and the lower the power grid efficiency value, the smaller the reward value.

[0066] The methods for determining whether a round has ended include:

[0067] The interactive service module determines whether the reinforcement learning agent can manage the power grid normally based on the new power grid status information. If it can, the round ends; if it cannot, the round does not end.

[0068] Step 7: The reinforcement learning agent receives the reward value and the result of whether the round has ended. If the current round has ended, it enters the next interactive round. The interactive service module randomly selects a new historical cross-section flow file for initialization until the agent's preset training end condition is met. If the current round has not ended, it enters the loop of steps 3-6 until the round ends. The preset training end condition includes reaching the preset maximum number of training steps, training to the last round, and the reward value no longer increasing.

[0069] Example 3

[0070] A method for intelligent agents to interact with the power grid environment includes the following steps:

[0071] Obtain real-time power grid information;

[0072] Real-time power grid information is input into the intelligent agent and the power grid environment interaction system, and power grid control operation actions are output.

[0073] The intelligent agent-grid environment interaction system is trained by the intelligent agent-grid environment interaction learning method described in Example 2.

[0074] Example 4

[0075] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a method for interactive learning between an intelligent agent and a power grid environment as described in Embodiment 2.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Those skilled in the art can readily implement the present invention based on the accompanying drawings and the above description. However, any modifications, alterations, or variations made by those skilled in the art without departing from the scope of the present invention, utilizing the disclosed technical content, are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, or variations made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the present invention.

Claims

1. A learning system for interaction between an intelligent agent and the power grid environment, characterized in that, The system includes a state estimation module, an interaction service module, a reinforcement learning agent, and a power flow calculation module. The state estimation module is used to read the power grid information at each historical time point, and generate historical cross-sectional power flow files based on the power grid information through network topology analysis and calculation. Based on the historical cross-sectional power flow files at each historical time point, a set of historical cross-sectional power flow files corresponding to the expected target is obtained according to the expected target. Among them, the power grid information includes power grid model parameters, connection relationships, and a set of redundant telemetry measurements and remote signaling switch states. The interactive service module is used to randomly select a target historical cross-section power flow file from the set of historical cross-section power flow files corresponding to the expected target learned by the agent, parse the target historical cross-section power flow file to obtain the target historical power grid state information, and then send the target historical power grid state information to the reinforcement learning agent. The reinforcement learning agent is used to predict the action to be performed based on the target historical power grid state information, and sends the predicted action to the interaction service module so that the interaction service module can modify the target historical cross-section power flow file according to the predicted action, and send the modified target historical cross-section power flow file to the power flow calculation module. The power flow calculation module is used to calculate the historical power flow based on the modified target historical cross-section power flow file, and generate a new historical cross-section power flow file by combining the calculation results of the historical power flow, and send the newly generated historical cross-section power flow file to the interactive service module.

2. The intelligent agent and power grid environment interactive learning system according to claim 1, characterized in that, The interactive service module is also used to parse the newly generated historical cross-sectional power flow file to form new power grid state information, generate reward values ​​based on the new power grid state information and predicted execution actions, determine whether the round has ended based on the new power grid state information, and send the reward value and the result of whether the round has ended to the reinforcement learning agent, so that the reinforcement learning agent can perform corresponding operations according to the result of whether the round has ended and determine whether the training meets the preset termination conditions based on the reward value.

3. The intelligent agent and power grid environment interactive learning system according to claim 2, characterized in that, The reward value is generated in the following ways: When the interactive service module determines that the predicted action is an illegal action, the reward value is set to negative; when the interactive service module determines that the predicted action is not an illegal action, the power grid efficiency value in the new power grid status information is extracted to generate a reward value. The higher the power grid efficiency value, the larger the reward value, and the lower the power grid efficiency value, the smaller the reward value.

4. The intelligent agent and power grid environment interactive learning system according to claim 2, characterized in that, The methods for determining whether a round has ended include: The interactive service module determines whether the reinforcement learning agent can manage the power grid normally based on the new power grid status information. If it can, the round ends and the result is sent to the reinforcement learning agent, allowing the agent to enter the next round. If it cannot, the round does not end and the result is sent to the reinforcement learning agent, allowing the agent to return to the action steps predicted based on the target's historical power grid status information and execute subsequent actions until the round ends.

5. The intelligent agent and power grid environment interactive learning system according to claim 2, characterized in that, The preset training termination conditions include reaching the preset maximum number of training steps, training to the last round, and the reward value no longer increasing.

6. A method for intelligent agents to learn through interaction with the power grid environment, characterized in that, The method is applied to an intelligent agent and power grid environment interaction learning system according to any one of claims 1-5, and the method includes the following steps: The state estimation module reads the power grid information at each historical time point, and generates historical cross-sectional power flow files based on the power grid information through network topology analysis and calculation. Based on the historical cross-sectional power flow files at each historical time point, it classifies them according to the expected target learned by the agent to obtain a set of historical cross-sectional power flow files corresponding to the expected target. Among them, the power grid information includes power grid model parameters, connection relationships, and a set of redundant telemetry measurements and remote signaling switch states. The interactive service module randomly selects a target historical cross-section power flow file from the set of historical cross-section power flow files corresponding to the expected target learned by the agent, parses the target historical cross-section power flow file to obtain the target historical power grid state information, and then sends the target historical power grid state information to the reinforcement learning agent. The reinforcement learning agent predicts the action to be performed based on the target's historical power grid state information and sends it to the interaction service module; The interactive service module modifies the target historical section power flow file according to the action to be executed, and then sends the modified target historical section power flow file to the power flow calculation module. The power flow calculation module calculates the historical power flow based on the modified target historical section power flow file, and generates a new historical section power flow file by combining the calculation results of the historical power flow. The newly generated historical section power flow file is then sent to the interactive service module. The interactive service module receives and parses the newly generated historical cross-sectional power flow file to form new power grid state information. Based on the new power grid state information and the predicted execution actions, it generates a reward value, determines whether the round has ended based on the new power grid state information, and sends the reward value and the result of whether the round has ended to the reinforcement learning agent. The reinforcement learning agent receives the reward value and the result of whether the round has ended, executes the action of whether the round has ended, and determines whether the training meets the preset termination condition based on the reward value.

7. A method for interaction between an intelligent agent and a power grid environment, characterized in that, include: Obtain real-time power grid information; Real-time power grid information is input into the intelligent agent and the power grid environment interaction system, and power grid control operation actions are output. The intelligent agent-grid environment interaction system is trained by the intelligent agent-grid environment interaction learning method as described in claim 6.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent agent and power grid environment interaction learning method as described in claim 6.

Images