Method and device for intelligently adjusting operation mode of power grid

Abstract

The invention relates to the technical field of power system operation mode control and adjustment, in particular to a power grid operation mode intelligent adjustment method and device, and the method comprises the steps: obtaining power grid operation data, and constructing a state vector and environment of an initial double-Q network; fitting a value function by using a graph Transform, and constructing a main Q network; determining an action selection strategy, selecting an optimal action meeting a certain action condition, and generating an optimal state vector; calculating a target Q value, and constructing a target Q network; correcting a reward function in the environment, and constructing a non-difference target to obtain a correction action; and constructing a final double-Q network, and adjusting the current operation mode to obtain an operation mode meeting a certain operation condition. Therefore, the problems that manual adjustment is low in efficiency, blindness exists, a deep reinforcement learning model is insufficient in sensing capacity for node voltage, line power flow and the like and is easily influenced by confusion variables, so that strategy decision deviation is caused, and actual requirements of a large-scale power grid are difficult to meet are solved.

Description

Technical Field

[0001] The present application relates to the technical field of power system operation mode control and adjustment, and in particular to a method and device for intelligent adjustment of power grid operation mode. Background Art

[0002] With the advancement of the "dual carbon" goals and the transformation of energy structure, the power system is developing towards a clean, low-carbon, flexible and efficient direction.

[0003] In related technologies, adjustments to the grid's operating mode can be achieved by relying on manual empirical rules and repeated simulation calculations. Alternatively, a power flow adjustment model can be designed based on algorithms such as DQN (Deep Q-Network). By simulating manual adjustment logic and utilizing a state-action-reward mechanism to guide generator start-up and shutdown or output adjustment, the power flow convergence problem in the operating mode can be resolved.

[0004] However, in related technologies, manual adjustments are inefficient and blind, suitable for low-dimensional action spaces, and have weak processing capabilities for high-dimensional action space data, making it difficult to adapt to the dynamic changes and operation mode adjustment requirements brought about by the widespread access of new energy sources. Deep reinforcement learning models mostly use fully connected networks or traditional convolutional networks to process grid status data, failing to fully explore the correlation characteristics of the grid topology, resulting in insufficient model perception of key features such as node voltage and line flow. At the same time, related models are easily affected by confounding variables in the value function estimation process, resulting in overestimation of value, leading to strategic decision-making deviations, and are difficult to meet the actual needs of large-scale power grids, and are in urgent need of improvement. Summary of the Invention

[0005] This application provides a method and device for intelligently adjusting the operation mode of a power grid to address the problems in related technologies, such as low efficiency and blindness in manual adjustment, which makes it difficult to adapt to the dynamic changes and operation mode adjustment requirements brought about by the widespread access of new energy sources; deep reinforcement learning models fail to fully explore the correlation characteristics of the power grid topology, resulting in insufficient perception of key features such as node voltage and line flow, and are easily affected by confounding variables during the value function estimation process, resulting in strategic decision-making deviations and difficulty in meeting the actual needs of large-scale power grids.

[0006] In a first aspect, an embodiment of the present application provides a method for intelligently adjusting an operation mode of a power grid, comprising the following steps: obtaining operation data of the power grid under a current operation mode, using the operation data to construct a state vector of an initial dual-Q network, and constructing an environment of the initial dual-Q network based on the state vector; inputting the state vector into a pre-constructed graph transformer to fit a value function of the initial dual-Q network, and using the value function to calculate a main Q value of a main Q network in the initial dual-Q network to construct the main Q network; determining an action selection strategy for the action space based on an action space and the main Q value in the environment, and selecting an optimal action that satisfies a preset action condition according to the action selection strategy; And generate an optimal state vector corresponding to the state vector after executing the optimal action; based on the optimal action, the optimal state vector and the main Q network, calculate the target Q value of the target Q network in the initial dual-Q network to construct the target Q network; based on the target Q value and the optimal action, correct the reward function in the environment to obtain a corrected reward function, and construct an indifference target based on the corrected reward function to correct the action deviation of the target Q network in assisting the main Q network in executing the optimal action to obtain a corrected action; construct a final dual-Q network based on the corrected action and the initial dual-Q network, so as to use the final dual-Q network to adjust the current operating mode until an operating mode that meets the preset operating conditions is obtained.

[0007] Optionally, in one embodiment of the present application, the state vector is input into a pre-constructed graph transformer to fit the value function of the initial dual-Q network, including: using the pre-constructed graph transformer to perform feature extraction on the state vector to generate an initial node feature vector of each node in the power grid and an initial feature matrix composed of the node feature vector; based on the connection data between different nodes, determining the first adjacency matrix in the pre-constructed graph transformer; based on the flow impact data between different nodes, determining the second adjacency matrix in the pre-constructed graph transformer; based on the initial node feature vector, the first adjacency matrix and the second adjacency matrix, determining the attention weights between the different nodes; based on the attention weights, using the pre-constructed graph transformer to update the initial node feature vector to obtain an updated node feature vector; based on the node feature vector, updating the initial feature matrix to obtain an updated feature matrix, and extracting features of the feature matrix to obtain global graph features in the pre-constructed graph transformer; based on the global graph features, using the pre-constructed graph transformer to fit the value function.

[0008] Optionally, in one embodiment of the present application, constructing the environment of the initial dual-Q network based on the state vector includes: constructing a state space in the environment based on the state vector; constructing a first action space of the action space based on the switching actions and capacitors in the power grid; constructing a second action space of the action space based on the output of the generator in the power grid; constructing a reward function in the environment based on the operating data; and constructing the environment based on the state space, the action space and the reward function.

[0009] Optionally, in one embodiment of the present application, the use of the operating data to construct the state vector of the initial dual-Q network includes: obtaining the power flow distribution data of the power grid based on the operating data; and converting the power flow distribution data into a vector to construct the state vector.

[0010] Optionally, in one embodiment of the present application, the expression of the indifference target may be, but is not limited to: , in, is the modified reward function, is the target Q value, is the discount factor.

[0011] The second aspect of the present application provides an intelligent adjustment device for the operation mode of a power grid, including: a first construction module for obtaining the operation data of the power grid under the current operation mode, so as to use the operation data to construct the state vector of the initial dual-Q network, and based on the state vector, construct the environment of the initial dual-Q network; a second construction module for inputting the state vector into a pre-constructed graph Transformer to fit the value function of the initial dual-Q network, so as to use the value function to calculate the main Q value of the main Q network in the initial dual-Q network, so as to construct the main Q network; a generation module for determining the action selection strategy of the action space based on the action space and the main Q value in the environment, so as to select the optimal action that meets the preset action conditions according to the action selection strategy, and Generate an optimal state vector corresponding to the state vector after executing the optimal action; a third construction module is used to calculate the target Q value of the target Q network in the initial dual-Q network based on the optimal action, the optimal state vector and the main Q network, so as to construct the target Q network; a correction module is used to correct the reward function in the environment based on the target Q value and the optimal action to obtain a corrected reward function, and construct an indifference target based on the corrected reward function to correct the action deviation of the target Q network in assisting the main Q network in executing the optimal action to obtain a corrected action; an adjustment module is used to construct a final dual-Q network based on the corrected action and the initial dual-Q network, so as to use the final dual-Q network to adjust the current operating mode until an operating mode that meets the preset operating conditions is obtained.

[0012] Optionally, in one embodiment of the present application, the second construction module includes: a first generating unit for performing feature extraction on the state vector using the pre-built graph Transformer to generate an initial node feature vector of each node in the power grid and an initial feature matrix composed of the node feature vectors; a first determining unit for determining a first adjacency matrix in the pre-built graph Transformer based on connection data between different nodes; a second determining unit for determining a second adjacency matrix in the pre-built graph Transformer based on flow impact data between different nodes; and a third determining unit for determining a second adjacency matrix in the pre-built graph Transformer based on the initial node feature vectors. The first adjacency matrix and the second adjacency matrix are used to determine the attention weights between the different nodes; the first update unit is used to update the initial node feature vector based on the attention weight using the pre-built graph Transformer to obtain an updated node feature vector; the second update unit is used to update the initial feature matrix based on the node feature vector to obtain an updated feature matrix, and extract the features of the feature matrix to obtain the global graph features in the pre-built graph Transformer; the fitting unit is used to fit the value function based on the global graph features using the pre-built graph Transformer.

[0013] Optionally, in one embodiment of the present application, the first construction module includes: a first construction unit for constructing a state space in the environment based on the state vector; a second construction unit for constructing a first action space of the action space based on the switching actions and capacitors in the power grid; a third construction unit for constructing a second action space of the action space based on the output of the generator in the power grid; a fourth construction unit for constructing a reward function in the environment based on the operating data; and a fifth construction unit for constructing the environment based on the state space, the action space and the reward function.

[0014] Optionally, in one embodiment of the present application, the first construction module includes: a second generation unit, used to obtain the power flow distribution data of the power grid based on the operation data; and a conversion unit, used to convert the power flow distribution data into a vector to construct the state vector.

[0015] Optionally, in one embodiment of the present application, the expression of the indifference target may be, but is not limited to: , in, is the modified reward function, is the target Q value, is the discount factor.

[0016] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for intelligently adjusting the operation mode of a power grid as described in the above embodiment.

[0017] A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned method for intelligent adjustment of the power grid operation mode.

[0018] The fifth aspect of the present application provides a computer program product, including a computer program, which, when executed, implements the above-mentioned method for intelligent adjustment of power grid operation mode.

[0019] The embodiment of the present application can use the operating data of the power grid under the current operating mode to construct the state vector of the initial dual-Q network, and based on the state vector, construct the environment of the initial dual-Q network, and use the pre-constructed graph Transformer value function to construct the main Q network, and then select the optimal action that meets certain action conditions according to the action selection strategy, and generate the optimal state vector corresponding to the state vector after executing the optimal action, thereby constructing the target Q network, and constructing the indifference target by modifying the reward function to obtain the modified action, and then constructing the final dual-Q network, and using the final dual-Q network to adjust the current operating mode until an operating mode that meets certain operating conditions is obtained. By introducing the graph Transformer and causal reinforcement learning mechanism to improve the dual-Q network, the dual-Q network has a stronger ability to perceive the state of the power grid operation mode, guide the dual-Q network to take correct actions, and realize efficient, reliable, and economical intelligent adjustment of the power grid operation mode. This solves the technical problems in related technologies, such as the low efficiency and blindness of manual adjustments, which make it difficult to adapt to the dynamic changes and operational mode adjustment requirements brought about by the widespread access of new energy; the deep reinforcement learning model fails to fully explore the correlation characteristics of the power grid topology, resulting in insufficient perception of key features such as node voltage and line flow, and is easily affected by confounding variables in the value function estimation process, resulting in strategic decision-making deviations and difficulty in meeting the actual needs of large-scale power grids.

[0020] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which: Figure 1This is a flow chart of a method for intelligently adjusting a power grid operation mode according to an embodiment of the present application; Figure 2 A flowchart of fitting a value function using a graph transformer according to one embodiment of the present application is provided; Figure 3 This is a flow chart of the working principle of the method for intelligently adjusting the operation mode of a power grid provided according to one embodiment of the present application; Figure 4 Schematic diagram of a block diagram of a device for intelligently adjusting a power grid operation mode according to an embodiment of the present application; Figure 5 A schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0023] The following describes the method and device for intelligently adjusting the operation mode of the power grid of an embodiment of the present application with reference to the accompanying drawings. In view of the fact that manual adjustment mentioned in the above background technology is inefficient and blind, and difficult to adapt to the dynamic changes and operation mode adjustment requirements brought about by the widespread access of new energy; the deep reinforcement learning model fails to fully explore the correlation characteristics of the power grid topology, resulting in insufficient perception of key features such as node voltage and line flow, and is easily affected by confounding variables in the value function estimation process, resulting in strategic decision deviations, and difficulty in meeting the actual needs of large-scale power grids. The present application provides a method for intelligently adjusting the operation mode of the power grid. In this method, the operation data of the power grid under the current operation mode can be used to construct the state vector of the initial dual-Q network, and based on the state vector, the environment of the initial dual-Q network is constructed, and the pre-constructed graph T is used. The algorithm uses a random value function to construct a main Q network, and then selects the optimal action that meets certain action conditions according to the action selection strategy, and generates the optimal state vector corresponding to the optimal action after executing the optimal action, thereby constructing a target Q network. By modifying the reward function to construct an indifferent target, a modified action is obtained, and then the final dual-Q network is constructed. The final dual-Q network is used to adjust the current operating mode until an operating mode that meets certain operating conditions is obtained. By introducing a graph transformer and a causal reinforcement learning mechanism to improve the dual-Q network, the dual-Q network has a stronger ability to perceive the state of the power grid operation mode, guiding the dual-Q network to make correct actions, and realizing efficient, reliable, and economical intelligent adjustment of the power grid operation mode. This solves the technical problems in related technologies, such as the low efficiency and blindness of manual adjustment, which makes it difficult to adapt to the dynamic changes and operation mode adjustment requirements brought about by the widespread access of new energy; the deep reinforcement learning model fails to fully explore the correlation characteristics of the power grid topology, resulting in insufficient perception of key features such as node voltage and line flow, and is easily affected by confounding variables in the value function estimation process, resulting in policy decision deviation and difficulty in meeting the actual needs of large-scale power grids.

[0024] Specifically, Figure 1 This is a flow chart of a method for intelligently adjusting the operation mode of a power grid provided according to an embodiment of the present application.

[0025] like Figure 1 As shown, the method for intelligently adjusting the grid operation mode includes the following steps: In step S101 , operation data of the power grid in the current operation mode is acquired, so as to construct a state vector of an initial dual-Q network using the operation data, and to construct an environment of the initial dual-Q network based on the state vector.

[0026] It is understood that the embodiments of the present application can implement intelligent adjustment of the grid operation mode and output an action set after inputting the operation data of the grid under the current operation mode. This method of intelligent adjustment of the grid operation mode mainly realizes automatic adjustment of the grid operation mode under conditions such as fluctuations in renewable energy output and load, and optimizes the grid power flow distribution. Specific applications can be divided into the following two typical scenarios: (1) When the output of renewable energy increases, it is very likely that the power transmission of certain lines in the power grid will exceed the limit, resulting in the inability to absorb the renewable energy. At this time, the embodiment of the present application can automatically reduce the power transmission of the lines with excessive power flow, and absorb the excess renewable energy power through other backup channels or local load side while ensuring the safety margin of system operation, thereby improving the new energy absorption capacity.

[0027] (2) When the output of renewable energy is at a low point but the load demand is high, the embodiment of the present application can automatically increase the power flow transmission of some lines and automatically adjust the units closest to the load center to increase the output to achieve reliable and economical adjustment of the grid operation mode.

[0028] In some embodiments, the embodiments of the present application can obtain operating data of the power grid under the current operating mode, and use the operating data to construct the state vector of the initial dual-Q network, and then construct the environment of the initial dual-Q network based on the state vector.

[0029] For example, in the embodiment of the present application, an actual grid model of the power grid can be built in the power flow calculation software, such as xx simulation software, and operating data such as the on / off status of the unit, output upper and lower limits, etc. can be set based on the actual grid model. After completing the grid modeling and operating data setting, the power flow calculation module is run to obtain power flow distribution data, wherein the power flow distribution data can be represented by data entries, which can be represented by, but not limited to: , in, For nodes The injected active power, For nodes The injected reactive power, For nodes The outflow active power, For nodes The outflow reactive power, For capacitors The amount of input, For nodes The voltage amplitude, For the line tidal power.

[0030] Optionally, in one embodiment of the present application, the state vector of the initial dual-Q network is constructed using operating data, including: obtaining power flow distribution data of the power grid based on the operating data; and converting the power flow distribution data into a vector to construct the state vector.

[0031] In some embodiments, the embodiments of the present application can convert the power flow distribution data into a vector and then construct a state vector.

[0032] For example, the embodiment of the present application can splice the flow distribution data into a one-dimensional vector in element order and convert it into the state vector required for deep reinforcement learning , which can be expressed as, but not limited to:

[0033] in, For nodes The injected active power, For nodes The injected reactive power, For nodes The outflow active power, For nodes The outflow reactive power, For capacitors The amount of input, For nodes The voltage amplitude, For the line tidal power.

[0034] Optionally, in one embodiment of the present application, an environment of an initial dual-Q network is constructed based on a state vector, including: constructing a state space in the environment based on the state vector; constructing a first action space of the action space based on the switching actions and capacitors in the power grid; constructing a second action space of the action space based on the output of the generator in the power grid; constructing a reward function in the environment based on operating data; and constructing the environment based on the state space, the action space, and the reward function.

[0035] It is understood that in the embodiments of the present application, the environment of the initial dual-Q network may include, but is not limited to, a state space, a first action space, a second action space, and a reward function, and this application does not impose specific restrictions. The action space may include, but is not limited to, a first action space and a second action space, wherein the first action space can be understood as a discrete action space, and the second action space can be understood as a continuous action space.

[0036] In some embodiments, the state space in the embodiments of the present application may be composed of the state vector in step S101.

[0037] In some embodiments, the first action space, i.e., the discrete action space, in the embodiments of the present application can be used to represent the switching of capacitors, which can be represented by, but not limited to: , in, For action, For the The switching action of a capacitor, Indicates that a capacitor is put into operation. To remove a capacitor, To maintain the current status, is the number of controllable capacitors in the system, is the total number of controllable capacitors in the system.

[0038] In some embodiments, the second action space in the embodiment of the present application is a continuous action space, which can be used to represent the adjustment of the generator The output can be expressed as, but not limited to: , in, For action, 、 For the generator output adjustment range, For generators The current output, 、 For generators Upper and lower limits of output.

[0039] Furthermore, the embodiment of the present application can combine the first action space with the second action space to construct a unified action space.

[0040] In some embodiments, the expression of the reward function in the embodiments of the present application can be, but is not limited to: , in, is the node voltage deviation, For nodes The actual voltage, is the reference value, is the line power flow deviation, For the line The actual trend is the thermal stability limit, 、 、 is the weight coefficient, is the total network loss of the system, and its calculation formula can be but is not limited to: , in, For the line The resistance, 、 are the line active power and reactive power, For nodes The actual voltage, 、 For nodes 、 The voltage phase angle.

[0041] In step S102, the state vector is input into a pre-built graph transformer to fit the value function of the initial dual-Q network, so as to calculate the main Q value of the main Q network in the initial dual-Q network using the value function to construct the main Q network.

[0042] It can be understood that in the embodiments of the present application, the graph transformer is a type of neural network model for processing graph-structured data. It combines the transformer architecture with graph learning technology, and realizes efficient representation and prediction of nodes, edges and graphs through the self-attention mechanism and graph inductive bias.

[0043] In some embodiments, the present invention can use a pre-built graph transformer to fit the value function of the initial dual-Q network, and then use the value function to calculate the main Q value of the main Q network in the initial dual-Q network, thereby constructing the main Q network. The calculation formula can be but not limited to: , in, is the value function of the graph Transformer fitting, is a correction term based on global graph features, Main Q network parameters.

[0044] Optionally, in one embodiment of the present application, the state vector is input into a pre-built graph transformer to fit the value function of the initial dual-Q network, including: using the pre-built graph transformer to extract features of the state vector to generate an initial node feature vector of each node in the power grid and an initial feature matrix composed of node feature vectors; based on the connection data between different nodes, determining the first adjacency matrix in the pre-built graph transformer; based on the flow impact data between different nodes, determining the second adjacency matrix in the pre-built graph transformer; based on the initial node feature vector, the first adjacency matrix and the second adjacency matrix, determining the attention weights between different nodes; based on the attention weights, using the pre-built graph transformer to update the initial node feature vector to obtain an updated node feature vector; based on the node feature vector, updating the initial feature matrix to obtain an updated feature matrix, and extracting features of the feature matrix to obtain global graph features in the pre-built graph transformer; based on the global graph features, using the pre-built graph transformer to fit the value function.

[0045] In some embodiments, the present application embodiment utilizes the process of pre-built graph transformer fitting value function as follows: Figure 2 As shown, the main contents are: Step S201: Perform feature extraction to generate an initial node feature vector and an initial feature matrix composed of the node feature vectors.

[0046] Among them, the embodiment of the present application can input the state vector into the graph Transformer to extract features and generate the initial node feature vector of each node in the power grid And the initial characteristic matrix of the power grid , its expression can be but not limited to: , , in, For nodes The voltage amplitude, For nodes The voltage phase angle, For nodes The injected active power, For nodes The injected reactive power, For nodes The initial node feature vector of .

[0047] Step S202: Generate a first adjacency matrix.

[0048] In this embodiment of the present application, the first adjacency matrix may reflect the connection relationship between different nodes, and its expression may be, but is not limited to,: , in, Representation node With node Whether to connect directly.

[0049] Step S203: Generate a second adjacency matrix.

[0050] The second adjacency matrix in the embodiment of the present application can reflect the power flow impact data between different nodes, and its expression can be, but is not limited to,: , in, Representation node For Node Whether there is a direct impact on the trend.

[0051] Step S204: Determine the attention weights between different nodes.

[0052] The expression of attention weights between different nodes in the embodiment of the present application can be, but is not limited to,: , in, For nodes and The attention weight, The function ensures that the attention weights are normalized. is the feature dimension, For nodes Features before update, For nodes Features before the update.

[0053] Step S205: Update the initial node feature vector.

[0054] The initial node feature vector update process in the embodiment of the present application may be, but is not limited to: , in, For nodes Updated features, is the learnable weight matrix.

[0055] Step S206: Update the initial feature matrix.

[0056] In this embodiment of the present application, the node feature vector is updated to update the initial feature matrix to obtain a deep feature matrix, the expression of which can be but is not limited to: , in, Refers to the node Deep features after feature update.

[0057] Step S207: extracting global graph features.

[0058] Among them, the embodiment of the present application is through the deep features in the deep feature matrix Extract global graph features, whose expressions can be but not limited to: , Step S208: Fitting the value function.

[0059] The expression of the fitting value function in the embodiment of the present application can be, but is not limited to,: , in, is the weight of the Transformer output layer, is the global feature weight, is the action embedding weight, is the global graph feature, is the activation function of the Transformer graph, here we choose function.

[0060] In step S103, based on the action space and the main Q value in the environment, an action selection strategy of the action space is determined to select the optimal action that meets the preset action conditions according to the action selection strategy, and generate an optimal state vector corresponding to the state vector after executing the optimal action.

[0061] In some embodiments, the embodiments of the present application can make the dual Q network structure not only suitable for making discrete actions, but also effectively make continuous actions. The system divides actions into continuous and discrete actions, and determines corresponding action selection strategies to select the optimal continuous and discrete actions that meet certain action conditions. The optimal action is derived by integrating the optimal continuous and discrete actions, and a state vector is generated to generate the optimal state vector corresponding to the execution of the optimal action. The specific action conditions can be set by those skilled in the art based on actual circumstances and are not specifically limited in this application.

[0062] Among them, when determining the continuous action, the embodiment of the present application selects the action selection strategy as the strategy network, thereby obtaining the optimal continuous action , its expression can be but not limited to: , in, is the policy function, is the policy network parameter set, According to the status The best continuous action, is the state vector.

[0063] In the embodiment of the present application, when determining discrete actions, the action selection strategy selected is the ϵ-greedy strategy, thereby obtaining the optimal discrete action , its expression can be but not limited to:

[0064] in, is the optimal discrete action, is the exploration rate, The optimal action selection is when the value of the fitted value function is maximized.

[0065] In step S104 , based on the optimal action, the optimal state vector and the main Q network, the target Q value of the target Q network in the initial dual Q network is calculated to construct the target Q network.

[0066] In some embodiments, the embodiments of the present application can construct a target Q network by calculating a target Q value and synchronizing the target Q network with the main Q network.

[0067] The target Q value may be calculated as follows, but is not limited to: , in, is the grid state after executing the optimal action selection, are the parameters of the target Q network. For optimal action selection, The process of synchronizing the target Q network with the main Q network may be, but is not limited to: , in, are the parameters of the main Q network, are the parameters of the target Q network, is the update coefficient.

[0068] In step S105, based on the target Q value and the optimal action, the reward function in the environment is modified to obtain a modified reward function. Based on the modified reward function, an indifference target is constructed to correct the action deviation of the target Q network in assisting the main Q network in performing the optimal action, thereby obtaining a corrected action. The expression of the indifference target can be, but is not limited to,: , in, is the modified reward function, is the target Q value, is the discount factor.

[0069] In some embodiments, the present invention can modify the reward function in the environment based on the optimal action to obtain a modified reward function, namely the counterfactual reward , its expression can be but not limited to: , in, is the reward function, For action, is the confounding factor, For a given confounding factor Reward under the conditions The expected value of For action Conduct causal intervention Post-reward The expected value of For counterfactual rewards.

[0070] Furthermore, the present embodiment can utilize counterfactual rewards Building causal indifference goals , its expression can be but not limited to: , in, is the modified reward function, is the target Q value, is the discount factor.

[0071] That is, the embodiment of the present application uses counterfactual rewards to remove the influence of confusion factors and construct an indifferent target , which enables the target Q network to assist the main Q network in making corrections when making the optimal action, reducing the deviation of making the optimal action according to the value function, and obtaining the corrected action. Its expression can be, but is not limited to: , In step S106 , a final dual-Q network is constructed according to the correction action and the initial dual-Q network, so as to adjust the current operation mode using the final dual-Q network until an operation mode that meets the preset operation conditions is obtained.

[0072] In some embodiments, embodiments of the present application can construct a final dual-Q network based on the corrective actions and the initial dual-Q network, and then use the final dual-Q network to adjust the current operating mode until an operating mode that meets certain operating conditions is achieved. The certain operating conditions can be set by those skilled in the art based on actual circumstances and are not specifically limited in this application.

[0073] The working principle of the intelligent adjustment method for the power grid operation mode proposed in the embodiment of the present application is introduced below with reference to a specific embodiment.

[0074] in, Figure 3 The present invention is a flowchart illustrating the working principle of a method for intelligently adjusting the operation mode of a power grid provided according to an embodiment of the present application.

[0075] Step S301: collecting and processing grid operation data.

[0076] Among them, the embodiment of the present application can obtain the operating data of the power grid under the current operating mode, and use the operating data to construct the state vector of the initial dual-Q network.

[0077] Step S302: constructing an initial dual-Q network environment.

[0078] In the embodiment of the present application, the environment of the initial dual-Q network may include, but is not limited to, a state space, a first action space, a second action space, and a reward function, and the present application does not impose any specific restrictions.

[0079] Step S303: Fitting the value function.

[0080] The process of fitting the value function in the embodiment of the present application is as follows: Figure 2 shown.

[0081] Step S304: Determine an action selection strategy and select the optimal action.

[0082] In this embodiment of the present application, when determining continuous actions, the action selection strategy selected is the policy network; when determining discrete actions, the action selection strategy selected is the ϵ-greedy strategy.

[0083] Step S305: Correct the optimal action to obtain a corrected action.

[0084] Step S306: Determine whether the operation mode meets certain operation conditions.

[0085] If the conditions are met, execute step S307; otherwise, execute step S303.

[0086] Step S307: Output the correction action set.

[0087] According to the intelligent adjustment method for the power grid operation mode proposed in the embodiment of the present application, the operating data of the power grid under the current operation mode can be used to construct the state vector of the initial dual-Q network, and based on the state vector, the environment of the initial dual-Q network can be constructed. The pre-constructed graph Transformer value function is used to construct the main Q network, and then the optimal action that meets certain action conditions is selected according to the action selection strategy, and the optimal state vector corresponding to the state vector after the optimal action is executed is generated, thereby constructing the target Q network. By modifying the reward function to construct the indifference target, the corrected action is obtained, and then the final dual-Q network is constructed. The final dual-Q network is used to adjust the current operation mode until an operation mode that meets certain operating conditions is obtained. By introducing the graph Transformer and causal reinforcement learning mechanism to improve the dual-Q network, the dual-Q network has a stronger ability to perceive the state of the power grid operation mode, guides the dual-Q network to take correct actions, and realizes efficient, reliable, and economical intelligent adjustment of the power grid operation mode. This solves the technical problems in related technologies, such as the low efficiency and blindness of manual adjustments, which make it difficult to adapt to the dynamic changes and operational mode adjustment requirements brought about by the widespread access of new energy; the deep reinforcement learning model fails to fully explore the correlation characteristics of the power grid topology, resulting in insufficient perception of key features such as node voltage and line flow, and is easily affected by confounding variables in the value function estimation process, resulting in strategic decision-making deviations and difficulty in meeting the actual needs of large-scale power grids.

[0088] Next, the intelligent adjustment device for the operation mode of the power grid proposed in accordance with the embodiment of the present application will be described with reference to the accompanying drawings.

[0089] Figure 4 This is a block diagram of a device for intelligently adjusting the operation mode of a power grid provided according to an embodiment of the present application.

[0090] like Figure 4 As shown, the intelligent adjustment device 10 for the power grid operation mode includes: a first construction module 100 , a second construction module 200 , a generation module 300 , a third construction module 400 , a correction module 500 and an adjustment module 600 .

[0091] The first construction module 100 is used to obtain operation data of the power grid under the current operation mode, to construct the state vector of the initial dual-Q network using the operation data, and to construct the environment of the initial dual-Q network based on the state vector.

[0092] The second construction module 200 is used to input the state vector into a pre-constructed graph transformer to fit the value function of the initial dual-Q network, and to calculate the main Q value of the main Q network in the initial dual-Q network using the value function to construct the main Q network.

[0093] The generation module 300 is used to determine the action selection strategy of the action space based on the action space and the main Q value in the environment, select the optimal action that meets the preset action conditions according to the action selection strategy, and generate the optimal state vector corresponding to the state vector after executing the optimal action.

[0094] The third construction module 400 is configured to calculate a target Q value of a target Q network in the initial dual Q network based on the optimal action, the optimal state vector, and the main Q network, so as to construct the target Q network.

[0095] The correction module 500 is used to correct the reward function in the environment based on the target Q value and the optimal action to obtain a corrected reward function, and to construct an indifference target based on the corrected reward function to correct the action deviation of the target Q network in assisting the main Q network in performing the optimal action to obtain a corrected action.

[0096] The adjustment module 600 is configured to construct a final dual-Q network according to the correction action and the initial dual-Q network, so as to adjust the current operation mode using the final dual-Q network until an operation mode that meets preset operation conditions is obtained.

[0097] Optionally, in one embodiment of the present application, the second construction module 200 includes: a first generating unit, a first determining unit, a second determining unit, a third determining unit, a first updating unit, a second updating unit and a fitting unit.

[0098] The first generation unit is used to extract features from the state vector using a pre-built graph transformer to generate an initial node feature vector of each node in the power grid and an initial feature matrix composed of the node feature vectors.

[0099] The first determining unit is configured to determine a first adjacency matrix in a pre-built graph transformer based on connection data between different nodes.

[0100] The second determining unit is configured to determine a second adjacency matrix in a pre-constructed graph transformer based on the power flow impact data between different nodes.

[0101] The third determining unit is used to determine the attention weights between different nodes based on the initial node feature vector, the first adjacency matrix and the second adjacency matrix.

[0102] The first updating unit is used to update the initial node feature vector based on the attention weight using a pre-built graph Transformer to obtain an updated node feature vector.

[0103] The second updating unit is used to update the initial feature matrix based on the node feature vector to obtain an updated feature matrix, and extract features of the feature matrix to obtain global graph features in the pre-built graph transformer.

[0104] The fitting unit is used to fit the value function based on the global graph features using the pre-built graph Transformer.

[0105] Optionally, in one embodiment of the present application, the first building module 100 includes: a first building unit, a second building unit, a third building unit, a fourth building unit and a fifth building unit.

[0106] The first construction unit is used to construct a state space in the environment based on the state vector.

[0107] The second constructing unit is configured to construct a first action space of the action space based on switching actions and capacitors in the power grid.

[0108] The third constructing unit is configured to construct a second action space of the action space based on the output of the generator in the power grid.

[0109] The fourth construction unit is used to construct a reward function in the environment based on the running data.

[0110] The fifth building unit is used to build an environment based on the state space, action space and reward function.

[0111] Optionally, in one embodiment of the present application, the first building module 100 includes: a second generating unit and a converting unit.

[0112] The second generating unit is used to obtain power flow distribution data of the power grid based on the operating data.

[0113] The conversion unit is used to convert the power flow distribution data into a vector to construct a state vector.

[0114] Optionally, in one embodiment of the present application, the expression of the indifference target may be, but is not limited to: , in, is the modified reward function, is the target Q value, is the discount factor.

[0115] It should be noted that the aforementioned explanation of the embodiment of the method for intelligently adjusting the operation mode of a power grid is also applicable to the intelligent adjustment device for the operation mode of a power grid of this embodiment, and will not be repeated here.

[0116] According to the intelligent adjustment device for the power grid operation mode proposed in the embodiment of the present application, the operating data of the power grid under the current operation mode can be used to construct the state vector of the initial dual-Q network, and based on the state vector, the environment of the initial dual-Q network can be constructed. The pre-constructed graph Transformer value function is used to construct the main Q network, and then the optimal action that meets certain action conditions is selected according to the action selection strategy, and the optimal state vector corresponding to the state vector after the optimal action is executed is generated, thereby constructing the target Q network. By modifying the reward function to construct the indifference target, the corrected action is obtained, and then the final dual-Q network is constructed. The final dual-Q network is used to adjust the current operation mode until an operation mode that meets certain operating conditions is obtained. By introducing the graph Transformer and causal reinforcement learning mechanism to improve the dual-Q network, the dual-Q network has a stronger ability to perceive the state of the power grid operation mode, guides the dual-Q network to take correct actions, and realizes efficient, reliable, and economical intelligent adjustment of the power grid operation mode. This solves the technical problems in related technologies, such as the low efficiency and blindness of manual adjustments, which make it difficult to adapt to the dynamic changes and operational mode adjustment requirements brought about by the widespread access of new energy; the deep reinforcement learning model fails to fully explore the correlation characteristics of the power grid topology, resulting in insufficient perception of key features such as node voltage and line flow, and is easily affected by confounding variables in the value function estimation process, resulting in strategic decision-making deviations and difficulty in meeting the actual needs of large-scale power grids.

[0117] Figure 5 This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. The electronic device may include: Memory 501 , processor 502 , and computer programs stored in the memory 501 and executable on the processor 502 .

[0118] When the processor 502 executes the program, the method for intelligently adjusting the operation mode of the power grid provided in the above embodiment is implemented.

[0119] Furthermore, the electronic device further includes: The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0120] The memory 501 is used to store computer programs that can be run on the processor 502 .

[0121] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0122] If the memory 501, processor 502, and communication interface 503 are implemented independently, the communication interface 503, memory 501, and processor 502 can be interconnected via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0123] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0124] The processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0125] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned method for intelligently adjusting the operation mode of a power grid.

[0126] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed, implements the above-mentioned method for intelligently adjusting the operation mode of the power grid.

[0127] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0128] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0129] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0130] The logic and / or steps represented in a flowchart or otherwise described herein, for example, can be considered a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" is any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (not exhaustive) of computer-readable media include: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically by optically scanning the paper or other medium and then editing, interpreting, or otherwise processing in a suitable manner as necessary, and then storing it in a computer memory.

[0131] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, it can be implemented using any one or a combination of the following technologies known in the art: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having suitable combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0132] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0133] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0134] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for intelligently adjusting the operation mode of a power grid, characterized in that: The following steps are involved: Acquiring operating data of the power grid in a current operating mode, using the operating data to construct a state vector of an initial dual-Q network, and constructing an environment of the initial dual-Q network based on the state vector; Inputting the state vector into a pre-built graph transformer to fit the value function of the initial dual-Q network, and calculating the main Q value of the main Q network in the initial dual-Q network using the value function to construct the main Q network; Determining an action selection strategy for the action space based on the action space and the main Q value in the environment, selecting an optimal action that satisfies a preset action condition according to the action selection strategy, and generating an optimal state vector corresponding to the state vector after executing the optimal action; Calculating a target Q value of a target Q network in the initial dual-Q network based on the optimal action, the optimal state vector, and the main Q network to construct the target Q network; Based on the target Q value and the optimal action, modifying the reward function in the environment to obtain a modified reward function, and constructing an indifference target based on the modified reward function to modify the action deviation of the target Q network in assisting the main Q network in executing the optimal action to obtain a modified action; A final dual-Q network is constructed according to the correction action and the initial dual-Q network, so as to adjust the current operation mode by using the final dual-Q network until an operation mode that meets preset operation conditions is obtained.

2. The method according to claim 1, characterized in that Inputting the state vector into a pre-built graph transformer to fit the value function of the initial double-Q network includes: Performing feature extraction on the state vector using the pre-built graph transformer to generate an initial node feature vector of each node in the power grid and an initial feature matrix composed of the node feature vectors; Determining a first adjacency matrix in the pre-constructed graph transformer based on connection data between different nodes; Determining a second adjacency matrix in the pre-constructed graph transformer based on power flow impact data between different nodes; Determining attention weights between the different nodes based on the initial node feature vector, the first adjacency matrix, and the second adjacency matrix; Based on the attention weight, using the pre-built graph Transformer to update the initial node feature vector to obtain an updated node feature vector; Based on the node feature vector, the initial feature matrix is ​​updated to obtain an updated feature matrix, and features of the feature matrix are extracted to obtain global graph features in a pre-built graph transformer; Based on the global graph features, the value function is fitted using the pre-built graph Transformer.

3. The method according to claim 1, characterized in that The step of constructing the initial dual-Q network environment based on the state vector includes: constructing a state space in the environment based on the state vector; constructing a first action space of the action space based on switching actions and capacitors in the power grid; constructing a second action space of the action space based on the output of the generator in the power grid; constructing a reward function in the environment based on the operating data; The environment is constructed based on the state space, the action space, and the reward function.

4. The method according to claim 1, wherein The constructing the state vector of the initial dual-Q network using the operating data includes: Based on the operation data, obtaining power flow distribution data of the power grid; The power flow distribution data is converted into a vector to construct the state vector.

5. The method according to claim 1, wherein The expression of the indifference target is: ， in, is the modified reward function, is the target Q value, is the discount factor.

6. An intelligent adjustment device for power grid operation mode, characterized in that: include: A first construction module is configured to obtain operating data of the power grid under a current operating mode, construct a state vector of an initial dual-Q network using the operating data, and construct an environment of the initial dual-Q network based on the state vector; a second construction module, configured to input the state vector into a pre-constructed graph transformer to fit the value function of the initial dual-Q network, and calculate the main Q value of the main Q network in the initial dual-Q network using the value function to construct the main Q network; a generation module, configured to determine an action selection strategy for the action space based on the action space in the environment and the main Q value, select an optimal action that satisfies a preset action condition according to the action selection strategy, and generate an optimal state vector corresponding to the state vector after executing the optimal action; a third construction module, configured to calculate a target Q value of a target Q network in the initial dual-Q network based on the optimal action, the optimal state vector, and the main Q network, so as to construct the target Q network; a correction module, configured to correct the reward function in the environment based on the target Q value and the optimal action to obtain a corrected reward function, and construct an indifference target based on the corrected reward function to correct the action deviation of the target Q network in assisting the main Q network in executing the optimal action to obtain a corrected action; An adjustment module is configured to construct a final dual-Q network according to the correction action and the initial dual-Q network, so as to adjust the current operation mode using the final dual-Q network until an operation mode that meets preset operation conditions is obtained.

7. The device according to claim 6, characterized in that The second building block includes: a first generating unit, configured to perform feature extraction on the state vector using the pre-built graph transformer to generate an initial node feature vector of each node in the power grid and an initial feature matrix composed of the node feature vectors; a first determining unit, configured to determine a first adjacency matrix in the pre-constructed graph transformer based on connection data between different nodes; a second determining unit, configured to determine a second adjacency matrix in the pre-constructed graph transformer based on power flow impact data between different nodes; a third determining unit, configured to determine the attention weights between the different nodes based on the initial node feature vector, the first adjacency matrix, and the second adjacency matrix; A first updating unit, configured to update the initial node feature vector using the pre-built graph Transformer based on the attention weight to obtain an updated node feature vector; A second updating unit is configured to update the initial feature matrix based on the node feature vector to obtain an updated feature matrix, and extract features of the feature matrix to obtain global graph features in a pre-constructed graph transformer; A fitting unit is used to fit the value function using the pre-built graph Transformer based on the global graph features.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for intelligently adjusting the operation mode of a power grid according to any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method for intelligently adjusting the operation mode of a power grid as described in any one of claims 1 to 5.

10. A computer program product, characterized in that It comprises a computer program, which, when executed, is used to implement the method for intelligently adjusting the operation mode of a power grid as claimed in any one of claims 1 to 5.

Images