Power grid active constraint optimization method and device based on deep reinforcement learning

Abstract

The invention discloses a power grid active constraint optimization method and device based on deep reinforcement learning, and the method comprises the steps: building a mirror image simulation environment, and generating different power flow operation condition data sets; training the initial system sensitivity model by using the data set to obtain a system sensitivity model; a sensitive regulation and control unit is screened, and an optimal active adjustment strategy is obtained through reinforcement learning training in a mirror image environment; and verifying strategy performance in a mirror image environment, and deploying in a power grid after the strategy performance reaches the standard. According to the power grid active constraint optimization method and device based on deep reinforcement learning, offline processing is realized through a mirror image simulation environment, key units are screened by using a system sensitivity model to reduce dimensions, and the complexity and time consumption of online calculation can be reduced by combining a reinforcement learning training strategy and offline verification; effective control of the power grid is achieved, and the problems that an existing active correction control technology is complex in solution, long in time consumption and difficult to meet the real-time decision-making requirement of the power grid can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of power grid safety and stability control, in particular to a power grid active power constraint optimization method and device based on deep reinforcement learning. BACKGROUND

[0002] With high proportion of renewable energy access and continuous complex operation mode of power grid, power grid safety and stability operation faces increasingly severe challenges; among them, line power over-limit is the core problem threatening the static security of power grid, so it is urgent to develop power grid active power correction control technology with rapidity and intelligence. Existing active power correction control technology mainly includes two categories: one is the traditional optimization model active power correction method, which is based on accurate power grid physical model, converts the active power correction problem into linear programming or quadratic programming problem, calculates the power transmission distribution factor or sensitivity matrix to construct the constraint, and calls the CPLEX solver to output the unit adjustment instruction, which is a classic scheme; the other is the original deep reinforcement learning decision method, which models the problem as a Markov decision process, the agent takes all controllable unit active adjustment as the action space and real-time power grid data as the state space, and learns the control strategy through trial and error interaction in the power grid simulation environment to maximize the cumulative reward, and the reward function punishes line over-limit and adjustment amplitude.

[0003] However, the existing active power correction strategy based on reinforcement learning often includes all controllable units in the action space, and the high-dimensional adjustment action output by the agent has a "black box" characteristic, making the decision logic difficult to understand; the dispatcher cannot verify the selection basis of the unit combination, which not only reduces the trust in the agent's decision, but also is not conducive to actual deployment; in addition, due to blind search of all units, the agent has difficulty in prioritizing low-cost and high-efficiency units, and is prone to produce control schemes with poor economic efficiency, which is contrary to the demand for lean operation of power grid; if efficient control units are determined by sensitivity analysis, traditional physical mechanism calculation is inefficient and time-consuming in large-scale power grid, which is difficult to meet the real-time decision-making time efficiency. SUMMARY

[0004] The present application provides a power grid active power constraint optimization method and device based on deep reinforcement learning to solve the problem that existing active power correction control technology is complex and time-consuming to solve, and difficult to meet the real-time decision-making requirements of power grid.

[0005] To achieve the above purpose, the present application provides a power grid active power constraint optimization method based on deep reinforcement learning, comprising:

[0006] acquiring physical and operating data of the power grid;

[0007] establishing a mirror simulation environment according to the physical and operating data, and generating a data set for different power flow operating conditions;

[0008] training a preset initial system sensitivity model according to the data set, to obtain a system sensitivity model;

[0009] screening a control unit whose absolute value of sensitivity of an out-of-limit line ranks within a preset proportion according to the system sensitivity model, and performing reinforcement learning training on the control unit to eliminate the out-of-limit scenario in the mirror simulation environment, to obtain an optimal active adjustment strategy;

[0010] verifying control performance of the optimal active adjustment strategy in a random environment in the mirror simulation environment, and deploying the power grid according to the optimal active adjustment strategy if a preset requirement is met.

[0011] The application establishes a mirror simulation environment according to physical and operation data, can transfer high-difficulty work that needs to be completed online to a virtual scene without risk, can actively simulate various power flow operation conditions without risk, and can avoid online calculation conflicts caused by occupation of real-time computing power of a real power grid; the generated data set covers different scene information, and provides structured samples associated with out-of-limit and control for subsequent model training. Training a system sensitivity model using a data set can avoid high calculation cost of a traditional method that needs to derive complex power flow equations to analyze the influence of a unit; secondly, the core control unit of an out-of-limit line is screened based on sensitivity, and after screening, only a few units that are most effective for eliminating out-of-limit can be focused on, and the problem complexity is significantly reduced; finally, reinforcement learning training is performed on the screened units in the mirror environment, since the action space has been simplified, reinforcement learning can efficiently explore the optimal adjustment strategy in a smaller state and action space, and an active adjustment strategy that accurately eliminates out-of-limit is quickly obtained. Verifying the performance of the strategy in a random scene in the mirror environment can avoid high risk and time consumption of trial and error in a real power grid, and can ensure the universality and effectiveness of the strategy.

[0012] Compared with the prior art, the application realizes offline processing through a mirror simulation environment, uses a system sensitivity model to screen key units for dimension reduction, combines reinforcement learning training strategy and offline verification, can reduce the complexity and time consumption of online calculation, realizes real-time and efficient decision-making of a power grid, and can solve the problem that existing active correction control technology is complex and time-consuming to solve, and is difficult to meet the real-time decision-making requirements of a power grid.

[0013] As a preferred scheme, a mirror simulation environment is established according to the physical and operation data, and a data set is generated for different power flow operation conditions, specifically:

[0014] A mirror simulation environment is established according to the physical and operation data, and a plurality of derived conditions are generated in the mirror simulation environment through security verification;

[0015] For each of the several types of derived working conditions, the power balance state data under the corresponding working condition is obtained by solving the power flow equation; wherein the power flow equation is established according to the grid node injection power, voltage amplitude and node admittance matrix;

[0016] The power flow equation is deduced and transformed to generate the sensitivity matrix under several types of derived working conditions;

[0017] Combined with the power balance state data and the basic attribute data of the power grid, the graph structure data representing the state of the power grid is constructed;

[0018] The graph structure data is used as input features, and the sensitivity matrix under the several types of derived working conditions is used as target labels to generate the data set.

[0019] The preferred scheme constructs a mirror simulation environment through physical and operating data, generates derived working conditions in combination with safety verification, and derives a sensitivity matrix based on the power flow equation, and finally forms a data set corresponding to the graph structure data and the sensitivity matrix. The mirror simulation environment can accurately reproduce the physical characteristics of the power grid, the derived working conditions cover various power flow operating scenarios, and the authenticity and comprehensiveness of the data set are ensured; and the graph structure data retains the topological association of the nodes and branches of the power grid, and the sensitivity matrix is directly associated with the state of the power grid and the control influence, so that the subsequent system sensitivity model can more accurately capture the dynamic characteristics of the power grid, providing high-quality data support for screening key control units and reinforcement learning training, and improving the adaptability of the optimization method to complex power grid working conditions.

[0020] As a preferred scheme, the several types of derived working conditions are generated through safety verification in the mirror simulation environment, specifically:

[0021] Through single-element and double-element fault safety verification in the mirror simulation environment, the power flow operating conditions under different power grid topologies are constructed;

[0022] For each of the constructed power flow operating conditions, the output fluctuation of the power source and the load is simulated in the mirror simulation environment to generate the several types of derived working conditions.

[0023] The fault verification of the preferred scheme can ensure that the operating scenarios under common topology changes of the power grid are covered, and the power source and load fluctuation simulation is close to the dynamic characteristics such as new energy output fluctuation and load random change in actual operation, so that the generated working conditions include different power grid topologies, contain overruns and no overruns, etc. This diversified working condition design avoids the scene limitation of model training, so that the subsequent system sensitivity model and reinforcement learning strategy can adapt to more complex actual power grid environment, and enhances the robustness and generalization ability of the optimization method.

[0024] As a preferred solution, the power flow equation is derived and transformed to generate a sensitivity matrix under several derived working conditions, specifically:

[0025] The power flow equation is mathematically abstracted to obtain a power balance relationship;

[0026] The power balance relationship is corrected according to the deviation amounts generated by the state variables and control variables to obtain a deviation balance relationship;

[0027] The deviation balance relationship is Taylor series expanded and the target high-order term is removed to obtain a linearized approximation equation about the state variable deviation and control variable deviation;

[0028] The linearized approximation equation is arranged in combination with the correlation between the line power flow and the state variable deviation to obtain the sensitivity matrix under the several derived working conditions.

[0029] The preferred solution derives the sensitivity matrix through power balance relationship correction, Taylor series expansion linearization, etc., taking into account the calculation accuracy and efficiency. On the one hand, the derivation process based on the power flow equation ensures that the sensitivity matrix can truly reflect the correlation between unit output adjustment and line power flow change; on the other hand, the linearization processing of Taylor series expansion removes high-order terms, simplifies the calculation while retaining the core influence law, so that the sensitivity matrix can not only accurately guide the regulation direction, but also is easy to be learned by the model. This accurate and simplified sensitivity matrix provides a reliable basis for subsequent screening of key regulation units, avoiding the problem of inefficient or ineffective regulation due to sensitivity calculation deviation, and improving the scientificity of the optimization method.

[0030] As a preferred solution, the control units whose sensitivity absolute values of the out-of-limit line rank within a preset proportion are screened out according to the system sensitivity model, specifically:

[0031] The sensitivity matrix is output based on the system sensitivity model;

[0032] For the out-of-limit line in the power grid, the sensitivity values of all units to the out-of-limit line are extracted from the sensitivity matrix, and the absolute values of each sensitivity value are calculated to obtain the sensitivity absolute values of several units to the out-of-limit line;

[0033] The sensitivity absolute values of the units to the out-of-limit line are sorted in descending order, and the control units whose sensitivity absolute values rank within a preset front proportion are screened out.

[0034] The preferred scheme avoids the waste of computing resources and the increase of strategy complexity caused by indiscriminate regulation of all units by focusing on the units that have the most significant impact on the over-limit line. At the same time, the sorting of the absolute value of the sensitivity ensures that the selected units can most directly and efficiently eliminate the over-limit of the power flow, improving the relevance of the reinforcement learning training. The agent can concentrate on learning the adjustment rules of the key units, shorten the training convergence time, and the final active adjustment strategy is more concise and efficient, reducing unnecessary unit actions and reducing the cost of grid regulation.

[0035] As a preferred scheme, the over-limit scenario is eliminated in the mirror simulation environment based on the reinforcement learning training of the regulation unit to obtain an optimal active adjustment strategy, specifically:

[0036] The active adjustment amount of the regulation unit is defined as an action space, and the target key data in the physical and operating data that affect the over-limit and regulation of the grid are included in the state space;

[0037] For the grid, an incentive function is established to eliminate over-limit and minimize generation cost;

[0038] Based on the deep deterministic policy gradient algorithm, the active adjustment amount of the regulation unit obtained from the action space and the agent state obtained from the state space are used to evaluate the action effect through the incentive function, continuously interact with the mirror simulation environment and update the network parameters until the global reward of the agent converges, and the optimal active adjustment strategy is obtained.

[0039] The preferred scheme realizes the efficiency of reinforcement learning training and the optimization of the strategy by clearly defining the action space and the state space, constructing a multi-objective incentive function, and using the deep deterministic policy gradient algorithm. The action space focuses on the active adjustment amount of the regulation unit, and the state space includes key grid data to ensure that the agent can accurately perceive the environment and output effective actions. The incentive function considers both over-limit elimination and generation cost minimization, balancing safety and economy. The deep deterministic policy gradient algorithm can efficiently learn the optimal strategy in a continuous action space and adapt to dynamic changes in the grid. The optimal active adjustment strategy obtained through training can quickly eliminate the over-limit of the power flow and reduce the generation cost, meeting the demand for safe and economic operation of the grid.

[0040] As a preferred scheme, the grid operation constraints include power flow constraints, inequality constraints, and line transmission constraints;

[0041] The power flow constraint is established based on the conservation law of power input and output in grid operation according to the active power and reactive power of the injection node;

[0042] The inequality constraint is established by explicitly defining the boundary range of unit regulation action.

[0043] The line transmission constraint is established by limiting the actual transmission power of the line to not exceed a preset safety threshold.

[0044] In the preferred solution, the power flow constraint can ensure that the regulation action conforms to the power conservation law, avoiding power imbalance; the inequality constraint can limit the unit adjustment range to prevent the unit from exceeding the safe operation limit; and the line transmission constraint can directly ensure that the line power does not exceed the safety threshold, fundamentally avoiding the risk of exceeding the limit. The introduction of these constraints ensures that the strategy generated in the reinforcement learning process always conforms to the actual operation rules of the power grid, avoiding unsafe or unfeasible regulation actions due to the pursuit of optimization goals, ensuring the practicality and safety of the optimal active adjustment strategy, and providing a guarantee for stable operation of the power grid.

[0045] As a preferred solution, the initial system sensitivity model is obtained in the following manner:

[0046] A power grid graph structure is established by taking the bus of the power grid as a node and the branch as an edge, and combining the adjacency relationship between nodes;

[0047] A network level is constructed according to a graph neural network layer, a fully connected layer, and an output layer;

[0048] The initial system sensitivity model is established based on the power grid graph structure and the network level;

[0049] The graph neural network layer is used to aggregate the adjacent edge features and neighbor node features through the node, and update the feature vector of the node by combining the node's own features, to generate node features containing neighborhood information; the fully connected layer is used to learn the high-order mapping relationship from the graph structure features to the sensitivity matrix by performing nonlinear combination and transformation on the node features extracted by the graph neural network layer; and the output layer is used to generate the sensitivity matrix required for predicting the active correction according to the high-order mapping relationship learned by the fully connected layer.

[0050] The preferred solution constructs an initial system sensitivity model based on the power grid graph structure, aggregates neighborhood features through the graph neural network layer, and learns high-order mapping through the fully connected layer, which significantly improves the model's ability to capture the characteristics of the power grid. The graph structure conforms to the topological nature of the power grid, enabling the model to naturally adapt to the spatial correlation characteristics of the power grid; the graph neural network layer effectively captures the indirect influence between nodes by aggregating the adjacent edge and neighbor node features, generating more comprehensive node features; and the fully connected layer enhances the nonlinear mapping ability from the graph structure to the sensitivity matrix, improving the sensitivity prediction accuracy. This structural design enables the initial model to more accurately learn the correlation between the power grid state and the sensitivity, providing reliable sensitivity data for subsequent selection of regulation units, and laying the foundation for the accuracy of the entire optimization method.

[0051] The application further provides a power grid active constraint optimization device based on deep reinforcement learning, comprising a data module, an environment module, a model module, a strategy module and a deployment module.

[0052] The data module is configured to acquire physical and operating data of the power grid.

[0053] The environment module is configured to establish a mirror simulation environment according to the physical and operating data, and generate a data set for different power flow operating conditions.

[0054] The model module is configured to train a preset initial system sensitivity model according to the data set, and obtain a system sensitivity model.

[0055] The strategy module is configured to filter out a control unit whose sensitivity absolute value ranking of an out-of-limit line is within a preset proportion according to the system sensitivity model, and perform reinforcement learning training on the control unit to eliminate an out-of-limit scenario in the mirror simulation environment, and obtain an optimal active adjustment strategy.

[0056] The deployment module is configured to verify the control performance of the optimal active adjustment strategy in a random environment in the mirror simulation environment, and if a preset requirement is met, deploy the power grid according to the optimal active adjustment strategy.

[0057] As a preferred scheme, the environment module comprises a condition unit, a balancing unit, a matrix unit, a graph structure unit and a comprehensive unit.

[0058] The condition unit is configured to establish a mirror simulation environment according to the physical and operating data, and generate a plurality of types of derived conditions through security verification in the mirror simulation environment.

[0059] The balancing unit is configured to obtain power balance state data under a corresponding condition by solving a power flow equation for each type of derived condition in the plurality of types of derived conditions, wherein the power flow equation is established according to grid node injection power, voltage amplitude and node admittance matrix.

[0060] The matrix unit is configured to derive and transform the power flow equation to generate sensitivity matrices under the plurality of types of derived conditions.

[0061] The graph structure unit is configured to combine the power balance state data and basic attribute data of the power grid to form graph structure data representing the state of the power grid.

[0062] The comprehensive unit is configured to take the graph structure data as input features and the sensitivity matrices under the plurality of types of derived conditions as target labels to generate the data set.

[0063] As a preferred solution, the working condition unit comprises a checking subunit and a working condition subunit;

[0064] The checking subunit is configured to construct power flow operation working conditions under different power grid topologies by performing single-element and double-element safety checks in the mirror simulation environment.

[0065] The working condition subunit is configured to simulate output fluctuations of power sources and loads in the mirror simulation environment for each constructed power flow operation working condition, and generate the several types of derived working conditions.

[0066] As a preferred solution, the matrix unit comprises a power subunit, a deviation subunit, a linear subunit, and a matrix subunit;

[0067] The power subunit is configured to perform mathematical abstract representation on the power flow equation to obtain a power balance relationship.

[0068] The deviation subunit is configured to correct the power balance relationship according to deviation amounts respectively generated by state variables and control variables, to obtain a deviation balance relationship.

[0069] The linear subunit is configured to perform Taylor series expansion on the deviation balance relationship and remove target high-order terms, to obtain a linearized approximation equation about state variable deviations and control variable deviations.

[0070] The matrix subunit is configured to organize the linearized approximation equation in combination with an association relationship between line power flow and state variable deviations, to obtain a sensitivity matrix under the several types of derived working conditions.

[0071] As a preferred solution, the strategy module comprises an output unit, a calculation unit, and a sorting unit;

[0072] The output unit is configured to output the sensitivity matrix based on the system sensitivity model.

[0073] The calculation unit is configured to, for an out-of-limit line in the power grid, extract sensitivity values of all units to the out-of-limit line from the sensitivity matrix, and calculate absolute values of each sensitivity value, to obtain sensitivity absolute values of several units to the out-of-limit line.

[0074] The sorting unit is configured to sort the sensitivity absolute values of the units to the out-of-limit line in descending order, and screen out the regulation and control units with sensitivity absolute values ranked within a preset top proportion.

[0075] As a preferred solution, the strategy module comprises a space unit, a function unit, and a strategy unit;

[0076] The space unit is configured to define an active adjustment amount of the regulating unit as an action space, and to include target key data in the physical and operation data that affects grid overrun and regulation into a state space.

[0077] The function unit is configured to establish a reward function for the grid, aiming to eliminate overrun and minimize power generation cost.

[0078] The strategy unit is configured to obtain the optimal active adjustment strategy based on a deep deterministic policy gradient algorithm, according to the active adjustment amount of the regulating unit obtained from the action space and the agent state obtained from the state space, follow grid operation constraints, evaluate action effects through the reward function, continuously interact with the mirror simulation environment and update network parameters until the global reward of the agent converges.

[0079] Preferably, the grid operation constraints include power flow constraints, inequality constraints and line transmission constraints.

[0080] The power flow constraints are established based on the conservation law of power input and output in grid operation, according to active power and reactive power of injection nodes.

[0081] The inequality constraints are established by explicitly defining the boundary range of unit regulation action.

[0082] The line transmission constraints are established by limiting the actual transmission power of the line to be less than a preset safety threshold.

[0083] Preferably, the initial system sensitivity model is obtained in the following manner:

[0084] A grid graph structure is established by taking buses of the grid as nodes and branches as edges, and combining the adjacency relationship between nodes.

[0085] A network level is constructed according to a graph neural network layer, a fully connected layer and an output layer.

[0086] The initial system sensitivity model is established based on the grid graph structure and the network level.

[0087] The graph neural network layer is configured to aggregate adjacent edge features and neighbor node features through nodes, and update its own feature vector in combination with its own features to generate node features containing neighborhood information; the fully connected layer is configured to learn high-order mapping relationships from grid graph structure features to sensitivity matrices by performing nonlinear combination and transformation on the node features extracted by the graph neural network layer; and the output layer is configured to generate a sensitivity matrix required for active power correction according to the high-order mapping relationships learned by the fully connected layer.

[0088] The application further provides a storage medium, wherein the storage medium stores a computer program, and the computer program is invoked and executed by a computer to implement the power grid active constraint optimization method based on deep reinforcement learning. BRIEF DESCRIPTION OF DRAWINGS

[0089] Figure 1 is a flowchart of a power grid active constraint optimization method based on deep reinforcement learning provided by an embodiment of the application.

[0090] Figure 2 is a structural schematic diagram of a power grid active constraint optimization device based on deep reinforcement learning provided by an embodiment of the application. DETAILED DESCRIPTION

[0091] The technical solutions in the embodiments of the application will be clearly and completely described below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only part of the embodiments of the application, but not all the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the application.

[0092] In the description of the application, unless otherwise specified, the meaning of "several" is two or more than two.

[0093] The power grid active constraint optimization method based on deep reinforcement learning provided by the embodiment of the application aims to solve the problems in the existing power grid active correction technology, i.e., the original deep reinforcement learning method has a high-dimensional action space, the decision logic is a "black box" and is difficult to be trusted by dispatchers, the regulation scheme is poor in economy, and the sensitivity calculation based on the traditional physical mechanism is low in efficiency and difficult to meet the real-time decision demand, so as to realize efficient, economic and decision-explainable power grid active safety constraint optimization.

[0094] Embodiment one:

[0095] Please refer to Figure 1 The embodiment of the application provides a power grid active constraint optimization method based on deep reinforcement learning, which comprises S1-S5, and the specific implementation steps are as follows:

[0096] S1, obtaining physical and operation data of the power grid.

[0097] The embodiment of the application is specifically S1:

[0098] The physical and operation data of the power grid comprises power grid physical structure related data and power grid operation state related data.

[0099] The power grid physical structure related data is key information representing inherent properties of the power grid, and specifically includes: basic properties of each node in the power grid, electrical parameters of each branch, and connection relationships between nodes and branches; and also includes inherent characteristic parameters of generator units, lines, and loads.

[0100] The power grid operation state related data covers dynamic operation information of the power grid under real-time and multiple scenario working conditions, and specifically includes two aspects: on the one hand, operation monitoring data under real-time and multiple working conditions, including injected active power of each node, injected reactive power of each node, voltage amplitude of each node, voltage phase angle of each node, and initial active power flow of each line; on the other hand, operation fluctuation data required by working conditions, that is, source and load output fluctuation data derived for different power flow operation working conditions.

[0101] S2, according to the physical and operation data, an image simulation environment is established, and data sets are generated for different power flow operation working conditions.

[0102] The embodiment of the application step S2 includes S2.1-S2.2, specifically:

[0103] S2.1, according to the physical and operation data, a mirror simulation environment is established using MATLAB / matpower tools, to ensure that it can accurately simulate the characteristics of generator units, lines, and loads, and system power flow fluctuation;

[0104] In the mirror simulation environment, N-1 (single element failure), N-2 (double element failure) safety checks are carried out for each operation working condition of the power grid, and power flow operation working conditions under different power grid topologies are constructed;

[0105] For each constructed power flow operation working condition, the output fluctuation of the power source and the load is simulated in the mirror simulation environment, and several types of derived working conditions are generated.

[0106] For each type of derived working condition in the several types of derived working conditions, the power balance state data under the corresponding working condition is obtained by solving the power flow equation, including node injected active power, node injected reactive power, node voltage amplitude, node voltage phase angle, and initial active power flow (i.e. line active power).

[0107] The power flow equation is established according to the node injected power, voltage amplitude, and node admittance matrix, and is specifically shown as formula (1):

[0108]

[0109] Wherein, P i , Q i , V i , δ i are the injected active power, injected reactive power, voltage amplitude and voltage phase angle of node i respectively; Nn G is the number of nodes of the power grid; ij +jB ij Yij is the admittance in the node admittance matrix corresponding to node i and node j.

[0110] The fault check of the embodiment S2.1 can ensure to cover the operation scenarios under common topology changes of the power grid, and the power supply and load fluctuation simulation can be close to the dynamic characteristics such as new energy output fluctuation and load random change in actual operation, so that the generated working conditions include different power grid topologies, contain overruns, and do not contain overruns and the like. The diversified working condition design avoids the scene limitation of model training, so that the subsequent system sensitivity model and reinforcement learning strategy can adapt to more complex actual power grid environment, and enhances the robustness and generalization ability of the optimization method.

[0111] S2.2, the power flow equation is derived and transformed to generate sensitivity matrices under several types of derived working conditions, as follows:

[0112] ①The essence of the power flow equation is to reflect the power balance constraint satisfied between the control variables u and the state variables x, so the power balance relationship is obtained by mathematically abstracting the power flow equation with the control variables u and the state variables x as the core, as shown in equation (2):

[0113] f(u,x)=0 (2)

[0114] Wherein, u represents the column vector of the control variable, x represents the column vector of the state variable, and f(·) represents the balance mapping relationship of the control variable and the state variable in the power flow equation. And the equation form is different with the selection of control variables and state variables, and the change of coordinate form.

[0115] ②In a certain stable operating state, the control variables and the state variables satisfy the power balance, at this time, equation (2) becomes equation (3):

[0116] f(u0,x0)=0 (3)

[0117] When the system operating state changes, the state variable x produces a deviation amount Δx=x-x0, and the control variable u produces a deviation amount Δu=u-u0, at this time, the deviation balance relationship is obtained by correcting the power balance relationship according to the deviation amounts of the state variable and the control variable, as shown in equation (4):

[0118] f(Δu+u0,Δx+x0)=0 (4)

[0119] Wherein, x0 and u0 are the initial values of the state variable and the control variable at the stable operating point, respectively.

[0120] ③Taylor series expansion is performed on the deviation balance relationship and the second and higher order terms are omitted to obtain a linear approximation equation about the state variable deviation and the control variable deviation, i.e., equation (5):

[0121]

[0122] Since the stable operating point satisfies f(u0, x0) = 0, equation (5) is further simplified to equation (6):

[0123]

[0124] ④In combination with the correlation between the line power flow and the state variable deviation, equation (6) is arranged to obtain a sensitivity matrix S under several types of derived working conditions, as shown in equation (7):

[0125]

[0126] The embodiment S2.2 derives the sensitivity matrix through the steps of power balance relationship correction, Taylor series expansion linearization, etc., and takes into account the calculation accuracy and efficiency. On the one hand, the derivation process based on the actual power balance data ensures that the sensitivity matrix can truly reflect the correlation between the unit output adjustment and the line power flow change; on the other hand, the linearization processing of Taylor series expansion removes the high order terms, simplifies the calculation while retaining the core influence law, so that the sensitivity matrix can not only accurately guide the regulation direction, but also is easy to be learned by the model. This accurate and simplified sensitivity matrix provides a reliable basis for subsequent screening of key regulation units, avoids the problem of inefficient or ineffective regulation due to sensitivity calculation deviation, and improves the scientificity of the optimization method.

[0127] S2.3, in combination with the power balance state data and the basic attribute data of the power grid, a graph structure data representing the state of the power grid is constructed, including: node characteristics (node type, active power injection P i , reactive power injection Q i , voltage amplitude U i , voltage phase angle δ i ) of each node, edge characteristics (admittance parameters G ij , B ij , initial active power flow P ij ) of each branch, and adjacency relationship (adjacency matrix A) between nodes. Among them, the basic attribute data of the power grid is the static inherent attribute of the power grid, specifically including the node type in the node characteristics, the admittance parameter in the edge characteristics, the adjacency relationship between nodes, etc.

[0128] Take the graph structure data representing the state of the power grid under each derived working condition as the input feature (X), and the sensitivity matrix under several derived working conditions as the target label (Y), and pack them one by one according to the principle of "one working condition-one data pair" to generate the data set required for the sensitivity model; wherein the data set includes the node features (node type, active power injection, reactive power injection, voltage amplitude, voltage phase angle) of each node in the system, the edge features (admittance parameter, initial active power flow) of each branch in the system, and the adjacency relationship between nodes.

[0129] The embodiment S2 constructs a mirror simulation environment through physical and operation data, generates derived working conditions combined with safety verification, and derives a sensitivity matrix based on a power flow equation, and finally forms a data set corresponding to the graph structure data and the sensitivity matrix. The mirror simulation environment can accurately reproduce the physical characteristics of the power grid, the derived working conditions cover various power flow operation scenarios, and the authenticity and comprehensiveness of the data set are ensured; and the graph structure data retains the topological association of the nodes and branches of the power grid, and the sensitivity matrix is directly associated with the power grid state and the control influence, so that the system sensitivity model trained subsequently can more accurately capture the dynamic characteristics of the power grid, provide high-quality data support for screening key control units and reinforcement learning training, and improve the adaptability of the optimization method to complex power grid working conditions.

[0130] S3, training the preset initial system sensitivity model according to the data set to obtain the system sensitivity model.

[0131] The embodiment of the application comprises S3.1-S3.2, and specifically comprises:

[0132] S3.1, establishing an initial system sensitivity model, specifically:

[0133] Taking the bus of the power grid as the node and the branch as the edge, and combining the adjacency relationship between nodes, a power grid graph structure is established, which can represent the topological connection characteristics and element association of the power grid;

[0134] According to the hidden layer and the output layer, a network level is constructed; wherein the hidden layer adopts a hierarchical architecture, the first layer is a graph neural network layer (GNN), and the subsequent hidden layers are all fully connected layers;

[0135] Based on the power grid graph structure and the network level, the connection logic of each level is integrated to establish an initial system sensitivity model.

[0136] The core functions of the graph neural network layer, the fully connected layer and the output layer are as follows:

[0137] ① Graph neural network layer: used for aggregating adjacent edge features and neighbor node features through nodes, and combining node features, updating the feature vector of the node through a learnable neural network, and generating new node features containing neighborhood information, providing a basis for subsequent feature processing;

[0138] ②Fully connected layer: used to learn high-order mapping relationship from power grid graph structure features to sensitivity matrix by deep nonlinear combination and transformation of node features extracted by graph neural network layer, and finally output high-dimensional feature vector to prepare for sensitivity matrix prediction of output layer; and ReLU activation function is used after each layer;

[0139] ③Output layer: used to generate sensitivity matrix required for active power correction according to high-order mapping relationship learned by fully connected layer; the dimension of this layer is set to N gen ×N line , and the generated sensitivity matrix can be directly used in active power correction control scenario. Wherein, N gen is the number of generators in the power grid, and N line is the number of lines in the power grid.

[0140] The initial system sensitivity model of this embodiment S3.1 is based on the structure of power grid graph. The model significantly improves the ability to capture the characteristics of the power grid by aggregating neighborhood features through the graph neural network layer and learning high-order mapping through the fully connected layer. The graph structure conforms to the topological nature of the power grid, enabling the model to naturally adapt to the spatial correlation characteristics of the power grid. The graph neural network layer effectively captures the indirect influence between nodes by aggregating adjacent edge and neighbor node features, generating more comprehensive node features. The fully connected layer enhances the nonlinear mapping ability from the graph structure to the sensitivity matrix, improving the prediction accuracy of the sensitivity. This structural design enables the initial model to more accurately learn the correlation between the power grid state and the sensitivity, providing reliable sensitivity data for the subsequent selection of regulation units, laying the foundation for the accuracy of the entire optimization method.

[0141] S3.2, based on the data set, the mean square error (MSE) is used as the loss function to train the initial system sensitivity model; wherein, the loss function directly compares the predicted sensitivity matrix output by the model with the true sensitivity matrix, and quantifies the difference between the two;

[0142] The training process aims to minimize this mean square error, iteratively optimize the model parameters, gradually improve the prediction accuracy of the model for the sensitivity matrix, and finally complete the training to obtain a system sensitivity model that can be used for subsequent regulation unit selection.

[0143] Wherein, the loss function is shown in formula (8):

[0144]

[0145] Wherein, L is the loss function value, N is the number of small batches of samples in the data set used to calculate the loss; S i represents the "true sensitivity matrix" corresponding to the i-th training sample, which is derived from the sensitivity matrix of the i-th derived operating condition calculated based on the above power flow equations (1)-(7). a predicted sensitivity matrix output after processing the graph structure input data of the i-th derived working condition for the initial system sensitivity model; denotes the square of the Frobenius norm of a matrix.

[0146] S4, according to the system sensitivity model, screening out the control units whose absolute values of line route sensitivity ranking are within the preset proportion, and performing reinforcement learning training on the control units in the mirror simulation environment to eliminate the out-of-limit scene, and obtaining an optimal active adjustment strategy.

[0147] The step S4 of the embodiment of the application comprises S4.1-S4.2, and specifically comprises:

[0148] S4.1, outputting a sensitivity matrix of the influence degree of unit output adjustment on line flow from the system sensitivity model;

[0149] For the out-of-limit line in the power grid, the sensitivity values of all control units on the out-of-limit line are extracted from the sensitivity matrix, that is, the element values in the "corresponding unit row-corresponding out-of-limit line column" of the matrix, and the absolute values of each sensitivity value are calculated to obtain the sensitivity absolute values of a plurality of units on the out-of-limit line; wherein the sensitivity absolute value can directly reflect the efficiency of unit adjustment in eliminating the out-of-limit of the line, and the larger the absolute value, the higher the control efficiency;

[0150] The sensitivity absolute values of the units on the out-of-limit line are sorted in descending order, and the control units whose sensitivity absolute values rank within the top 20% are screened out as the control units used to eliminate the line out-of-limit in the subsequent process.

[0151] The embodiment S4.1 focuses on the units that have the most significant impact on the out-of-limit line, avoiding the waste of computing resources and the increase of strategy complexity caused by indiscriminate control of all units. At the same time, the sorting method of the sensitivity absolute value ensures that the selected units can most directly and efficiently eliminate the power flow out-of-limit, improves the pertinence of reinforcement learning training, and the agent can concentrate on learning the adjustment rules of key units, shortens the training convergence time, and the finally generated active adjustment strategy is more concise and efficient, reduces unnecessary unit actions, and reduces the power grid control cost.

[0152] S4.2, defining the active adjustment amount of the above control units as an action space, and including the target key data affecting the power grid out-of-limit and control in the state space;

[0153] For the active correction demand of the power grid, an incentive function is established to eliminate the out-of-limit and minimize the power generation cost;

[0154] Based on the improved deep deterministic policy gradient algorithm (DDPG), according to the regulation unit active adjustment amount obtained from the action space and the agent state obtained from the state space, the experience generated by the interaction between the agent and the mirror simulation environment is stored in the experience replay pool under the premise of following the power grid operation constraint, the effect of each action is evaluated by fusing the reward function, two independent current Critic networks and corresponding target Critic networks are introduced to reduce the overestimation bias, and the training process of the Actor network and the Critic network is stabilized through the target network soft update mechanism, the network parameters are continuously updated until the global reward of the agent converges, and finally the optimal active adjustment strategy is obtained.

[0155] The power grid operation constraint includes power flow constraint, inequality constraint and line transmission constraint.

[0156] The power flow constraint is established based on the conservation law of power input and output in the power grid operation according to the active power and reactive power of the injection node.

[0157] The inequality constraint is established by explicitly defining the boundary range of unit regulation action.

[0158] The line transmission constraint is established by limiting the actual transmission power of the line to not exceed the preset safety threshold.

[0159] The following will make specific description to each calculation formula:

[0160] ① State space, as shown in formula (9):

[0161] S t ={l l ,P l ,P G,i ,P W,u ,G choice} (9)

[0162] Wherein, l l is the opening and closing condition of the line, 1 for open circuit, and 0 for otherwise; P l is the transmission power of the line c; P G,i is the active power output of the generator unit; P W,u is the active power output of the new energy unit; G choice is the regulation unit determined by the sensitivity method.

[0163] ② Action space, as shown in formula (10):

[0164] a t ={ΔP G,i} (10)

[0165] Wherein, ΔP G,i is the active adjustment amount of the generator unit.

[0166] ③Reward function, as shown in equation (11):

[0167]

[0168] where k c is the total cost penalty coefficient; c i is the generation cost coefficient of unit i; k1 is the generation adjustment coefficient; k2 is the line over-limit penalty coefficient; i is the number of adjustable units; N line is the total number of lines; and β is the direct penalty given when the line violates the set constraints.

[0169] ④Power flow constraints, as shown in equation (12):

[0170]

[0171] where P i and Q i are the active and reactive power injected into node i, V i is the voltage amplitude of node i, θ ij , G ij , and B ij are the voltage phase angle difference, conductance, and susceptance between node i and node j.

[0172] ⑤Inequality constraints, as shown in equation (13):

[0173]

[0174] where P G,i , P G,i,max , P G,i,min are the active power of synchronous generator i and its upper and lower limits, respectively; P W，u , P W，u，max， P W，u，min are the active power of new energy unit u and its upper and lower limits, respectively; ΔP G,i , ΔP G,i,max ΔP G,i,min are the active power change of synchronous generator i and its maximum upward and downward ramping power, respectively; Q G.i , Q G.i,max , Q G.i,min are the reactive power of synchronous generator i and its upper and lower limits, respectively.

[0175] ⑥Line transmission constraints, as shown in equation (14):

[0176] P 1,t ≤ 0.9P l,max (14)

[0177] where P l,max is the maximum allowable transmission power of line l.

[0178] And for the reward function, there is:

[0179]

[0180] Where P l ,P l,max is the line active transmission power and its upper limit.

[0181] ⑦The training algorithm uses an improved deep deterministic policy gradient algorithm, and the policy of the agent is represented by the Actor network, and its policy parameter is represented by θ. The cumulative expected return of the agent is shown in equation (15):

[0182]

[0183] The deterministic policy gradient of the agent can be expressed as equation (16):

[0184]

[0185] Where s represents the state space of the agent; D is the experience replay pool, which records all the experience trajectories of the agent, and each sample data is a four-tuple (s, a, r, s').

[0186] Q μ update as shown in equation (17):

[0187] L(θ)=E (s,a,r,s′)～D [(Q(s,a|θ)-y) 2 ] (17)

[0188] And the parameter y is obtained from equation (18):

[0189] y=r+γ·Q μ′ (s′,a′)| a′＝μ′(s′) (18)

[0190] Where Q μ′ represents the joint action value function of the agent in the target network, and μ' is the parameter θ i ' of the target policy network with lagging update.

[0191] ⑧Improved deep deterministic policy gradient algorithm (DDPG): To solve the problem of overestimation bias in the training process of DDPG, the AC policy is improved based on DDPG to effectively reduce the overestimation in online learning and reduce the error of each update. The specific improvement method is to introduce two independent current Critic networks and their corresponding two target Critic networks respectively for estimating the action value function, and a smaller one is selected as the target Q value during training to reduce the overestimation bias, so the Q value calculation method of the target network of formula (18) is replaced by formula (19) as shown in formula (19):

[0192]

[0193] The embodiment S4.2 realizes the efficiency of reinforcement learning training and the optimization of the strategy by clearly defining the action space and the state space, constructing a multi-objective reward function, and using a deep deterministic policy gradient algorithm. The action space focuses on regulating the active adjustment amount of the unit, and the state space includes key grid data to ensure that the agent can accurately perceive the environment and output effective actions. The reward function takes into account the elimination of over-limit and minimization of power generation cost, balancing safety and economy. The deep deterministic policy gradient algorithm can efficiently learn the optimal strategy in a continuous action space and adapt to dynamic changes in the grid. The optimal active adjustment strategy obtained through training can quickly eliminate the flow over-limit and reduce the power generation cost, balancing the safety and economic operation requirements of the grid;

[0194] In addition, the power flow constraint can ensure that the regulation action conforms to the power conservation law and avoid power imbalance; the inequality constraint can limit the unit adjustment range to prevent the unit from exceeding the safe operation limit; and the line transmission constraint can directly ensure that the line power does not exceed the safety threshold, fundamentally avoiding the risk of over-limit. The introduction of these constraints ensures that the strategy generated during reinforcement learning always conforms to the actual operation rules of the grid, avoiding unsafe or infeasible regulation actions due to the pursuit of optimization objectives, and ensuring the practicality and safety of the optimal active adjustment strategy, providing a guarantee for the stable operation of the grid.

[0195] S5, verifying the control performance of the optimal active adjustment strategy in a random environment in a mirror simulation environment, and if the preset requirements are met, deploying the grid according to the optimal active adjustment strategy.

[0196] The embodiment of the application step S5 is specifically:

[0197] In the mirror simulation environment, the control performance of the optimal active adjustment strategy is verified by the agent Actor network. The verification process needs to cover various random environments to fully test the adaptability and stability of the strategy under different working conditions. If the control performance after verification meets the preset requirements, the online hybrid deployment of the grid based on the optimal active adjustment strategy is started. The online hybrid deployment is specifically that the dispatching strategy is provided to the dispatching personnel as a decision-making reference during the early deployment stage, and after the model feasibility reaches the preset requirements, the online deployment is switched to an automatic execution scheme.

[0198] In addition, in the "online hybrid deployment phase", the state space input of the agent is derived from the wide-area measurement information obtained by the system through the wide-area measurement system and the phasor measurement unit, based on the input information, the agent outputs the control suggestions of each region, and finally realizes the active security correction control of the power grid system.

[0199] Overall, the embodiment of the application has the following beneficial effects:

[0200] The application establishes a mirror simulation environment according to physical and operation data, can transfer the high-difficulty work that needs to be completed online in the traditional way to a virtual scene without risk, can actively simulate various power flow operation conditions without risk, and can avoid online calculation conflicts caused by occupying real-time computing power of the real power grid; the generated data set will cover different scene information, and provide out-of-limit and regulation-related structured samples for subsequent model training. Training the system sensitivity model using the data set can avoid the high calculation cost of the traditional method of deriving complex power flow equations online to analyze the influence of the unit; secondly, the core regulation unit of the out-of-limit line is screened based on the sensitivity, and after screening, only a few units that are most effective in eliminating the out-of-limit can be focused on, thereby significantly reducing the problem complexity; finally, reinforcement learning training is performed on the screened units in the mirror environment, since the action space has been simplified, reinforcement learning can efficiently explore the optimal adjustment strategy in a smaller state and action space, and quickly obtain an active adjustment strategy that accurately eliminates the out-of-limit; the performance of the strategy under random scenes is verified offline in the mirror environment, which can avoid the high risk and time-consuming trial-and-error in the real power grid, and can ensure the universality and effectiveness of the strategy.

[0201] In summary, the application deeply integrates data-driven intelligent perception (GNN) and reinforcement learning decision (DRL) to create an intelligent regulation framework with a "perception-decision" closed loop. The GNN outputs clear sensitivity values, providing intuitive physical basis for agent decision-making. The operator can understand that the reason why the agent preferentially adjusts certain units is that these units have the highest regulation efficiency for the current fault, rather than being a completely black-box decision. This decision logic based on physical laws significantly enhances the trust of human operators in the decision results of the agent. At the same time, the framework explicitly considers the adjustment cost and regulation efficiency by designing a reward function. The agent learns in a simplified action space composed of a small number of key units and is guided by the reward function to preferentially select adjustment strategies with high sensitivity and low cost, resulting in a control scheme that is both efficient and economical, naturally meeting the lean operation needs of the power grid. The application discards the traditional approach of having the agent blindly explore all units and innovatively introduces a pre-trained graph neural network (GNN) sensitivity model to achieve intelligent selection of regulation resources. This model can quickly identify and output the set of effective units that are most critical for eliminating the current overrun based on the current system state using forward propagation calculation. Compared to traditional numerical iteration-based sensitivity calculation methods, the GNN perception mechanism used has significantly improved computational efficiency, overcoming the bottleneck of traditional methods that are time-consuming and difficult to meet real-time decision-making requirements. Through this step, the agent's action space is accurately compressed from a high-dimensional space encompassing all units to a low-dimensional space containing only a few efficient key units, significantly reducing the agent's exploration difficulty and training complexity while ensuring that subsequent decision-making always focuses on the most effective regulation resources, providing a high-speed and reliable decision basis for subsequent optimization control.

[0202] Embodiment two:

[0203] Please refer to Figure 2 The embodiment of the application provides a power grid active constraint optimization device based on deep reinforcement learning, which comprises a data module 10, an environment module 20, a model module 30, a policy module 40 and a deployment module 50.

[0204] The data module 10 is used to acquire physical and operating data of the power grid.

[0205] The environment module 20 is used to establish a mirror simulation environment according to the physical and operating data, and generate a data set for different power flow operating conditions.

[0206] The model module 30 is used to train a preset initial system sensitivity model according to the data set to obtain a system sensitivity model.

[0207] The strategy module 40 is configured to screen out the control units whose sensitivity absolute values of the out-of-limit lines are ranked within a preset proportion according to a system sensitivity model, and perform reinforcement learning training on the control units to eliminate the out-of-limit scenario in the mirror simulation environment, so as to obtain an optimal active adjustment strategy.

[0208] The deployment module 50 is configured to verify the control performance of the optimal active adjustment strategy in a random environment in the mirror simulation environment, and deploy the power grid according to the optimal active adjustment strategy if a preset requirement is met.

[0209] In an embodiment, the environment module 20 includes a working condition unit, a balance unit, a matrix unit, a graph structure unit and a comprehensive unit.

[0210] The working condition unit is configured to establish a mirror simulation environment according to physical and operation data, and generate a plurality of derived working conditions through security verification in the mirror simulation environment.

[0211] The balance unit is configured to obtain power balance state data under a corresponding working condition by solving a power flow equation for each of the plurality of derived working conditions; wherein the power flow equation is established according to grid node injection power, voltage amplitude and node admittance matrix.

[0212] The matrix unit is configured to derive and transform the power flow equation to generate a plurality of sensitivity matrices under the derived working conditions.

[0213] The graph structure unit is configured to combine the power balance state data and basic attribute data of the power grid to form graph structure data representing the state of the power grid.

[0214] The comprehensive unit is configured to generate a data set by taking the graph structure data as input features and the sensitivity matrices under the plurality of derived working conditions as target labels.

[0215] In an embodiment, the working condition unit includes a verification subunit and a working condition subunit.

[0216] The verification subunit is configured to construct power flow operation working conditions under different power grid topologies by performing single-element and double-element fault security verification in the mirror simulation environment.

[0217] The working condition subunit is configured to simulate power source and load output fluctuations in the mirror simulation environment to generate a plurality of derived working conditions for each constructed power flow operation working condition.

[0218] In an embodiment, the matrix unit includes a power subunit, a bias subunit, a linear subunit and a matrix subunit.

[0219] The power subunit is configured to perform mathematical abstract representation on the power flow equation to obtain a power balance relationship.

[0220] a deviation subunit configured to correct the power balance relationship according to deviations of the state variables and the control variables respectively, to obtain a deviation balance relationship;

[0221] a linearization subunit configured to perform Taylor series expansion on the deviation balance relationship and remove target high-order terms, to obtain a linearization approximation equation about deviations of the state variables and the control variables;

[0222] a matrix subunit configured to arrange the linearization approximation equation in combination with an association relationship between the line flow and the deviations of the state variables, to obtain a sensitivity matrix under several types of derived working conditions.

[0223] In an embodiment, the strategy module 40 includes an output unit, a calculation unit and a sorting unit;

[0224] The output unit is configured to output the sensitivity matrix based on the system sensitivity model;

[0225] The calculation unit is configured to, for an out-of-limit line in the power grid, extract sensitivity values of all generating units to the out-of-limit line from the sensitivity matrix, and calculate absolute values of each sensitivity value, to obtain sensitivity absolute values of the generating units to the out-of-limit line;

[0226] The sorting unit is configured to sort the sensitivity absolute values of the generating units to the out-of-limit line in descending order, and filter out a regulating generating unit ranked within a preset top proportion.

[0227] In an embodiment, the strategy module 40 includes a space unit, a function unit and a strategy unit;

[0228] The space unit is configured to define an active adjustment amount of the regulating generating unit as an action space, and include target key data affecting power grid out-of-limit and regulation in a state space;

[0229] The function unit is configured to, for the power grid, establish a reward function aiming to eliminate out-of-limit and minimize power generation cost;

[0230] The strategy unit is configured to, based on a deep deterministic policy gradient algorithm, according to the active adjustment amount of the regulating generating unit obtained from the action space and an agent state obtained from the state space, follow power grid operation constraints, evaluate action effects through the reward function, continuously interact with a mirror simulation environment and update network parameters until an agent global reward converges, to obtain an optimal active adjustment strategy.

[0231] In an embodiment, the power grid operation constraints include power flow constraints, inequality constraints and line transmission constraints;

[0232] Among them, the power flow constraint is based on the conservation law of power input and output in power grid operation, and is established according to the active power and reactive power of the injection node;

[0233] The inequality constraint is established by explicitly setting the boundary range of the unit regulation action;

[0234] The line transmission constraint is established by limiting the actual transmission power of the line to not exceed the preset safety threshold.

[0235] In one embodiment, the initial system sensitivity model is obtained in the following manner:

[0236] The power grid graph structure is established by taking the bus of the power grid as the node, taking the branch as the edge, and combining the adjacency relationship between nodes;

[0237] The network level is constructed according to the graph neural network layer, the fully connected layer and the output layer;

[0238] The initial system sensitivity model is established based on the power grid graph structure and the network level;

[0239] Among them, the graph neural network layer is used to aggregate the adjacent edge features and neighbor node features through the node, and update the feature vector of the node itself in combination with the node itself features, to generate the node features containing the neighborhood information; the fully connected layer is used to learn the high-order mapping relationship from the power grid graph structure features to the sensitivity matrix through the nonlinear combination and transformation of the node features extracted by the graph neural network layer; the output layer is used to generate the sensitivity matrix required for active power correction according to the high-order mapping relationship learned by the fully connected layer.

[0240] It should be noted that the technical concept of the second embodiment is completely consistent with that of the first embodiment, and the two maintain a high degree of synergy at the technical logic level. The specific technical details can be referred to the related description of the first embodiment, which will not be repeated here.

[0241] Embodiment three:

[0242] The embodiment of the application provides a computer readable storage medium, which comprises a stored computer program, wherein when the computer program runs, the device where the computer readable storage medium is located is controlled to execute the power grid active constraint optimization method based on deep reinforcement learning;

[0243] If the power grid active constraint optimization method based on deep reinforcement learning is implemented in the form of a software function unit and used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate form. The computer-readable medium can include any entity or device that can carry the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0244] The above is the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which are also considered within the scope of protection of the present application.

Claims

1. A power grid active power constraint optimization method based on deep reinforcement learning, characterized in that, include: Acquire physical and operational data of the power grid; A mirror simulation environment is established based on the physical and operational data, and datasets are generated for different power flow operating conditions; The system sensitivity model is obtained by training a preset initial system sensitivity model based on the dataset; Based on the system sensitivity model, control units whose absolute sensitivity values ​​of over-limit lines rank within a preset proportion are selected. In the mirror simulation environment, reinforcement learning training to eliminate over-limit scenarios is performed on the control units to obtain the optimal active power adjustment strategy. The control performance of the optimal active power adjustment strategy under random conditions is verified in the mirror simulation environment. If the preset requirements are met, the power grid is deployed according to the optimal active power adjustment strategy.

2. The power grid active power constraint optimization method based on deep reinforcement learning as described in claim 1, characterized in that, A mirror simulation environment is established based on the aforementioned physical and operational data, and datasets are generated for different power flow operating conditions, specifically: A mirror simulation environment is established based on the physical and operational data, and several types of derivative operating conditions are generated in the mirror simulation environment through security verification. For each of the aforementioned derivative operating conditions, the power balance state data under the corresponding operating condition is obtained by solving the power flow equations; wherein, the power flow equations are established based on the power injected into the grid nodes, the voltage amplitude, and the node admittance matrix; The power flow equations are derived and transformed to generate sensitivity matrices for several derivative operating conditions. By combining the power balance state data and the basic attribute data of the power grid, a graph structure data characterizing the power grid state is constructed. The dataset is generated by using the graph structure data as input features and the sensitivity matrices under the several derived working conditions as target labels.

3. The power grid active power constraint optimization method based on deep reinforcement learning as described in claim 2, characterized in that, In the mirror simulation environment, several types of derived operating conditions are generated through security verification, specifically: By performing fault safety verification of unit components and dual components in the mirror simulation environment, power flow operation conditions under different power grid topologies are constructed. For each constructed power flow operating condition, the power output fluctuations of the power source and load are simulated in the mirror simulation environment to generate the several types of derived operating conditions.

4. The power grid active power constraint optimization method based on deep reinforcement learning as described in claim 2, characterized in that, The power flow equations are derived and transformed to generate sensitivity matrices for several derived operating conditions, specifically: The power flow equations are mathematically abstracted to obtain the power balance relationship. The power balance equation is corrected based on the deviations generated by the state variables and control variables to obtain the deviation balance equation. By performing a Taylor series expansion on the aforementioned deviation balance equation and removing the higher-order terms of the objective, a linearized approximate equation for the deviations of the state variables and the control variables is obtained. By combining the correlation between line power flow and state variable deviation, the linearized approximate equations are rearranged to obtain the sensitivity matrices for the several derived operating conditions.

5. The power grid active power constraint optimization method based on deep reinforcement learning as described in claim 1, characterized in that, Based on the system sensitivity model, control units whose absolute sensitivity values ​​of out-of-limit lines rank within a preset proportion are selected, specifically: The sensitivity matrix is ​​output based on the system sensitivity model; For the over-limit lines in the power grid, the sensitivity values ​​of all generating units to the over-limit lines are extracted from the sensitivity matrix, and the absolute value of each sensitivity value is calculated to obtain the absolute sensitivity values ​​of several generating units to the over-limit lines. The absolute values ​​of the sensitivity of the generator units to the over-limit lines are sorted in descending order, and the generator units whose absolute values ​​of sensitivity rank in the top proportion of the preset range are selected.

6. The power grid active power constraint optimization method based on deep reinforcement learning as described in claim 1, characterized in that, In the mirror simulation environment, reinforcement learning training is performed on the control unit to eliminate over-limit scenarios, thereby obtaining the optimal active power adjustment strategy, specifically: The active power adjustment of the control unit is defined as the action space, and the key target data affecting grid over-limit and control in the physical and operational data are included in the state space. For the aforementioned power grid, a reward function is established with the objectives of eliminating over-limits and minimizing generation costs; Based on the deep deterministic policy gradient algorithm, according to the active power adjustment amount of the control unit obtained from the action space and the agent state obtained from the state space, following the grid operation constraints, the effect of the action is evaluated through the reward function, and the network parameters are continuously interacted with the mirror simulation environment and updated until the global reward of the agent converges, thus obtaining the optimal active power adjustment strategy.

7. The power grid active power constraint optimization method based on deep reinforcement learning as described in claim 6, characterized in that, The power grid operation constraints include power flow constraints, inequality constraints, and line transmission constraints; The power flow constraint is based on the conservation law of power input and output in power grid operation and is established according to the active power and reactive power of the injected node. The inequality constraints are established by clearly defining the boundary range of the unit's control actions; The line transmission constraint is established by limiting the actual transmission power of the line to a preset safety threshold.

8. A power grid active power constraint optimization method based on deep reinforcement learning as described in any one of claims 1-7, characterized in that, The method for obtaining the initial system sensitivity model is as follows: A power grid graph structure is established using the busbars of the power grid as nodes and the branches as edges, combined with the adjacency relationships between the nodes. The network hierarchy is constructed based on the graph neural network layer, fully connected layer, and output layer; The initial system sensitivity model is established based on the power grid diagram structure and the network hierarchy; The graph neural network layer is used to aggregate adjacent edge features and neighbor node features through nodes, and update its own feature vector by combining the node's own features to generate node features containing neighborhood information; the fully connected layer is used to learn a high-order mapping relationship from power grid diagram structure features to sensitivity matrix by performing nonlinear combination and transformation on the node features extracted by the graph neural network layer; the output layer is used to generate the sensitivity matrix required for predicting active power correction based on the high-order mapping relationship learned by the fully connected layer.

9. A power grid active power constraint optimization device based on deep reinforcement learning, characterized in that, It includes a data module, an environment module, a model module, a strategy module, and a deployment module; The data module is used to acquire physical and operational data of the power grid. The environment module is used to establish a mirror simulation environment based on the physical and operational data, and to generate datasets for different power flow operating conditions. The model module is used to train a preset initial system sensitivity model based on the dataset to obtain the system sensitivity model; The strategy module is used to select the control units whose absolute values ​​of the over-limit line sensitivity are ranked within a preset proportion according to the system sensitivity model, and to perform reinforcement learning training to eliminate over-limit scenarios based on the control units in the mirror simulation environment, so as to obtain the optimal active power adjustment strategy. The deployment module is used to verify the control performance of the optimal active power adjustment strategy under random conditions in the mirror simulation environment. If the preset requirements are met, the optimal active power adjustment strategy is deployed to the power grid.

10. The power grid active power constraint optimization device based on deep reinforcement learning as described in claim 9, characterized in that, The environment module includes a working condition unit, a balance unit, a matrix unit, a graph structure unit, and a comprehensive unit; The operating condition unit is used to establish a mirror simulation environment based on the physical and operational data, and to generate several types of derived operating conditions through safety verification in the mirror simulation environment. The balancing unit is used to obtain the power balance state data of each of the several derivative operating conditions by solving the power flow equations; wherein, the power flow equations are established based on the power injected into the grid nodes, the voltage amplitude, and the node admittance matrix. The matrix unit is used to derive and transform the power flow equation to generate sensitivity matrices under several derivative operating conditions. The graph structure unit is used to combine the power balance state data and the basic attribute data of the power grid to form graph structure data that characterizes the power grid state. The integrated unit is used to generate the dataset by using the graph structure data as input features and the sensitivity matrices under the several derived working conditions as target labels.

Images