Dual-mode constrained reinforcement learning power grid control method based on physical information map agent and monotonicity strategy

Abstract

This invention relates to the field of intelligent operation and control technology of power systems, and proposes a dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy. By constructing a panoramic proxy model based on physical information graph neural network, it extracts high-dimensional features of non-Euclidean space using complex admittance weighted topology and multi-head graph attention mechanism, and embeds AC power flow physical residual penalty, realizing high-fidelity, millisecond-level virtual simulation of power grid state with physical consistency. A two-stage Actor network structure is established. In the first stage, the economic optimization of global active power loss is dominated by distributed value assessment and reparameterization strategy. In the second stage, a monotonic safety correction network based on ReLU network and non-negative weighted projection is introduced, and a joint iterative closed-loop control module based on the proxy model is designed. Combined with full-link differentiable safety gradient feedback and continuous action physical boundary decoding, high-throughput closed-loop parameter optimization without high-consumption time-consuming mechanism simulator is achieved.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and control technology of power systems, and in particular to technologies related to real-time dispatching and voltage safety control of distribution networks based on the deep integration of physical mechanisms and data-driven approaches under high-proportion renewable energy access. Specifically, this invention relates to a dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy. Background Technology

[0002] With the widespread integration of high-proportion distributed renewable energy sources, modern distribution networks exhibit strong randomness, high volatility, and nonlinear characteristics of source-load interaction, leading to a sharp increase in the risks of system voltage exceeding limits and power flow deviations. Traditional physical mechanism modeling methods based on optimal power flow (OPF) are extremely time-consuming to solve and heavily rely on precise global grid parameters, making it difficult to meet the real-time scheduling requirements at the microsecond or millisecond level. While pure data-driven reinforcement learning methods, which have emerged in recent years, possess online rapid inference capabilities, as typical "black box" models, they not only suffer from high trial-and-error costs and low sample efficiency when interacting with the real power grid, but more importantly, they completely ignore the underlying physical constraints of the power grid, such as Kirchhoff's laws. Furthermore, conventional reinforcement learning strategies lack rigorous proof of control theory stability, making the generated control strategies highly susceptible to unexpected reverse adjustments under extreme conditions, exceeding voltage safety boundaries and triggering system oscillations. Existing technologies struggle to simultaneously balance computational efficiency, physical consistency, and operational safety under complex and highly random fluctuations. Summary of the Invention

[0003] This invention provides a dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy. The technical problem it solves is that existing real-time control and scheduling methods for distribution networks face the problems of extremely low solution efficiency of traditional mechanism models and lack of physical consistency constraints and operational safety guarantees in pure data-driven reinforcement learning models.

[0004] To address the aforementioned technical problems, this invention provides a dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy, comprising the following steps:

[0005] Step 1: Data perception and physical topology proxy model construction. The multi-source operation status of the distribution network is normalized, and a physical topology graph data object containing node features, weighted adjacency matrix and edge feature matrix is ​​constructed. The topological coupling features between nodes are extracted through a multi-head graph attention mechanism, and a physical information neural network proxy model is constructed by combining the AC power flow balance residual. The hidden features of the physical topology graph and global auxiliary information are fused into a reinforcement learning state vector, and the normalized control actions are converted into executable reactive power setting instructions through action boundary mapping.

[0006] Step 2: Construct a distributed value Critic network for risk-sensitive value assessment and a two-stage Actor network for continuous control action generation. The Stage-1 Actor network outputs a normalized economic reference action, while the Stage-2 Actor network outputs a normalized safety correction action based on local voltage deviations. The final normalized control action is generated through action fusion and limiting processing. And it is converted into actual action vectors by the action space decoupling operator. ;

[0007] Step 3: Based on the physical information neural network agent model built in Step 1, perform interactive sampling, value assessment update, economic strategy update, security correction update, and online closed-loop deployment of the two-stage Actor network.

[0008] This invention designs a Physical Information Neural Network (PINN) surrogate model that integrates a multi-head graph attention mechanism (GAT) and AC power flow residual equations. It utilizes a physical consistency loss function to constrain the training process, extracting high-dimensional hidden space features containing global topological relationships, and establishing a millisecond-level fast inference mapping from hidden states to the entire network voltage distribution. This results in a virtual interactive environment that combines physical fidelity with extreme computational efficiency. In terms of state construction and action generation, this invention uses panoramic feature vectors as reinforcement learning input and designs a two-stage Actor network structure based on Distributed Soft Actor-Commentator (DSAC-T). The first stage, the policy network (Stage-1 Actor network), dominates global economic optimization, generating a benchmark reference action aimed at minimizing system active power loss through reparameterization. The second stage introduces a monotonic safety correction network (Stage-2 Actor network), employing a stacked ReLU network that takes only local voltage deviations as input and outputs a monotonic reactive power correction amount for forced voltage regulation. Through adaptive weight switching logic and an action safety mapping layer based on device capacity, the two-stage actions are deeply integrated to synthesize a comprehensive control command that considers both global economics and absolute physical feasibility. In the adaptive optimization and closed-loop control module, this invention utilizes a high-fidelity surrogate model for high-throughput virtual trial and error. Operating cost and voltage limit violation penalties are used as a comprehensive safety loss signal. A distributed value Critic network is employed to evaluate the continuous probability distribution of action rewards and calculate the negative log-likelihood divergence, driving the algorithm to perform risk-sensitive value assessment in a strongly stochastic environment. Simultaneously, the full-link differentiability of the surrogate model is used to calculate the feedback gradient of the limit violation penalty on the correction network. After backpropagation, a non-negative weighted projection truncation is enforced, ensuring from the underlying algorithm logic that the safety correction action strictly satisfies the Lyapunov stability condition, achieving closed-loop adaptive tuning without the need for an external, time-consuming physical solver.

[0009] This invention addresses the challenges of low solution efficiency in traditional control methods under complex operating conditions and the lack of physical consistency and theoretical security guarantees in black-box reinforcement learning. It proposes a solution that integrates physical agent feedforward, distributed value assessment, and monotonic correction feedback, achieving topological robust perception, two-stage adaptive action synthesis, and theoretically stable optimization. This provides a high-performance, high-reliability, and theoretically supported intelligent control paradigm for real-time economic dispatch and second-level safety correction in complex fluctuating power grid environments. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the framework of the dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy provided in the embodiments of the present invention. Detailed Implementation

[0011] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0012] like Figure 1 As shown in this embodiment, the dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy includes:

[0013] Step 1: Data perception and physical topology proxy model construction. The multi-source operation status of the distribution network is normalized, and a physical topology graph data object containing node features, weighted adjacency matrix and edge feature matrix is ​​constructed. The topological coupling features between nodes are extracted through a multi-head graph attention mechanism, and a physical information neural network proxy model is constructed by combining the AC power flow balance residual. The hidden features of the physical topology graph and global auxiliary information are fused into a reinforcement learning state vector, and the normalized control actions are converted into executable reactive power setting instructions through action boundary mapping.

[0014] Specifically, the following steps are included:

[0015] S1: Normalization eliminates the interference of physical dimensions on gradient updates, and a physical topology graph data object is constructed based on the distribution network physical topology. The physical topology graph data object includes a node feature matrix, a weighted adjacency matrix, and an edge feature matrix. The node feature matrix describes the current operating measurements, the normalized control action at the previous time step, and node type information; the weighted adjacency matrix describes the physical connection strength between nodes; and the edge feature matrix describes branch admittance parameters and electrical distance characteristics. This processing provides a unified graph data input for subsequent spatial feature extraction.

[0016] Furthermore, the specific steps of S1 include:

[0017] S1001: Collects real-time operating status data from multiple sources in the distribution network and performs normalization processing at time steps. Obtain the entire network through a wide area measurement system The running status of each node is recorded, and the node state variables are normalized to obtain a normalized running observation set. As shown in formula (1):

[0018]

[0019] For any physical state variable to be normalized Its normalization formula is as follows (2):

[0020]

[0021] in, This indicates that node i is at time step t. Any physical state variable to be normalized, They are nodes The active power injection and reactive power injection at time step t. and They are nodes The voltage amplitude and voltage phase angle at time step t, It is the charge state of node i at time step t. This indicates that node i is at time step t. Any normalized physical state variable, that is, representing Let represent any physical state variable, where Representing nodes respectively Normalized active power injection, reactive power injection, voltage amplitude, voltage phase angle, and state of charge at time step t. express The corresponding physical lower and upper limits of the physical state variables.

[0022] S1002: Construct the weighted adjacency matrix and edge feature matrix in the physical topology graph data object. Represent the distribution network as a physical topology graph data object. A weighted adjacency matrix containing electrical distance information is constructed using branch conductance and susceptance. and edge feature matrix , The set of distribution network nodes is represented as shown in formulas (3) and (4):

[0023]

[0024]

[0025] in and These are the connection nodes. and Branch conductance and branch susceptance, Represents the weighted adjacency matrix The physical connection weight between node i and node j corresponds to this. It is the edge feature matrix Corresponding node The edge feature vectors, This represents a small normal constant to prevent the denominator from being zero. For the branch collection of the distribution network, This represents the transpose of a matrix.

[0026] S1003: Constructing the node feature matrix in the physical topology graph data object To maintain the Markov decision process characteristics required by the underlying reinforcement learning, nodes The input feature column vector at time step t Simultaneously, it integrates the current normalized operational observation set and time step extracted from S11. When applied to the node Normalized motion characteristics on the corresponding controllable device And the one-hot encoding of the physical node type of node i As shown in formula (5):

[0027]

[0028] in, This represents the node feature matrix of node i at time step t. Mid-node feature vectors Indicates node i at time step When applied to the node Normalized motion characteristics on the corresponding controllable devices. This indicates the one-hot encoding of the physical node type of node i.

[0029] S2: Based on the physical topology graph data object constructed by S1002 and S1003, the dynamic attention coefficient calculation is first performed by explicitly injecting edge feature vectors, and then the weighted aggregation of multi-attention head features and residual state generation are performed. By explicitly injecting edge feature vectors into the attention scoring function and combining residual connections to alleviate the gradient decay problem in the deep feature propagation process, adaptive weighted aggregation of strongly coupled node features in the physical topology graph data object is achieved to generate hidden features of the physical topology graph.

[0030] Furthermore, the specific steps of S2 include:

[0031] S21: Perform dynamic attention coefficient calculation for explicit injection of edge feature vectors, for the Each attention head will generate the edge features of S1102. Inject scoring function, calculate node Influenced by neighboring nodes Unnormalized correlation score of the impact and conduct The normalization function is shown in formulas (6) and (7):

[0032]

[0033]

[0034] Where k is the current attention head index, Normalization function, This represents the learnable scoring weight vector of the k-th attention head. Representing the node feature matrix Middle node and nodes The input feature vector, This indicates that at time step t, node j interacts with node... The unnormalized correlation score, Let represent the normalized attention coefficient of node j to node i at time step t. This represents the node feature mapping matrix and edge feature mapping matrix of the k-th attention head. For nodes The set of first-order adjacent nodes, This represents the concatenation operation of vector dimensions. This represents the natural exponential function, used to convert unnormalized correlation scores into positive weights. This represents the unnormalized correlation score of node m with respect to node i at time step t.

[0035] S22: Perform multi-head feature weighted aggregation and residual state generation, The weighted aggregation results of each attention head are concatenated, and residual connections are introduced to generate nodes. At time step Hidden state The set of hidden features of the physical topology graph, which is composed of the hidden features of all nodes. As shown in formulas (8) and (9):

[0036]

[0037]

[0038] in, It is a non-linear activation function. Represents the set of hidden features in the physical topology graph. This represents the high-dimensional hidden feature vector of node i after fusing topological neighborhood information at time step t. Represents a node The high-dimensional hidden feature vector after fusing topological neighborhood information at time step t, i.e. The set has N nodes. Indicates will The output vectors of each attention head are concatenated. This represents the learnable weight matrix used for residual connection mapping. For nodes The set of first-order adjacent nodes, This represents the normalized attention coefficient of node j to node i at time step t.

[0039] S3: Set of hidden features in the physical topology graph obtained based on S22 A physical information neural network surrogate model containing state prediction branches and reward prediction branches is constructed, and the AC power flow balance equation is transformed into a differentiable physical residual penalty term, so that the physical information neural network surrogate model maintains consistency with the physical laws of the power grid during the state deduction process.

[0040] Furthermore, the specific steps of S3 include:

[0041] S31: Establish an environmental dynamics feedforward network The environmental dynamics feedforward network hides the feature set in the physical topology graph output by S22. and time step Actual action vectors issued to controllable equipment in the distribution network Given the input, output the predicted state for the next time step. With instant rewards As shown in formulas (10) and (11):

[0042]

[0043]

[0044] in These represent the network parameters respectively. The decision involves a state prediction branch and an immediate reward prediction branch. This represents the operation of flattening the set of hidden features in the physical topology graph into a one-dimensional vector. Indicates time step The actual action vectors issued to the controllable equipment in the power distribution network in real time. Indicates the time step of the deduction. The predicted state vector contains the inversely normalized predicted voltage magnitude. Phase angle with predicted voltage .

[0045] S32: Construct a differentiable physical residual penalty term based on the AC power flow equation, and extract... The predicted voltage amplitude and predicted voltage phase angle are used to construct the time step based on Kirchhoff's laws. Active power imbalance residual at node i Residual of reactive power imbalance As shown in formulas (12) and (13):

[0046]

[0047]

[0048] in, Representing nodes respectively At time step The active power imbalance residual value and the reactive power imbalance residual value, Representing nodes respectively At time step The theoretical active power injection and the theoretical reactive power injection, These represent the nodes predicted by S31. and The voltage amplitude at time step t+1, These represent the nodes predicted by S31. and The voltage phase angle at time step t+1, and These are the connection nodes. and Branch conductance and branch susceptance, For nodes The set of first-order adjacent nodes.

[0049] S4: Generate historical operating trajectories through multi-condition sampling, and use the adaptive moment estimation algorithm to jointly optimize the state prediction error, reward fitting error and active and reactive physical residuals to obtain the physical information neural network surrogate model parameters for reinforcement learning training.

[0050] Furthermore, the specific steps of S4 include:

[0051] S41: Perform offline sampling and reconstruction of multi-condition operating trajectories, generating trajectory data including state transitions through Monte Carlo sampling or historical operating data sampling, forming a dataset for the physical information neural network surrogate model. A single sample record is as follows:

[0052] ,

[0053] in, , Indicates time step The actual action vectors issued to the controllable equipment in the power distribution network in real time. This represents the state at time step t+1. This represents the actual reward at time step t.

[0054] S42: Implement joint optimization driven by both physics and data. Construct a composite loss function that encompasses state prediction, reward fitting, and the physical residuals of active and reactive power. And the parameters of the physical information neural network surrogate model are updated using the adaptive moment estimation algorithm, as shown in formula (14):

[0055]

[0056] in, For network parameters The defined total loss function, These represent the traversal indices for the batch size and the current batch size, respectively. They represent the first The predicted state vector and the true state label vector of each sample. They represent the first The predicted instant reward and the actual instant reward for each sample. They represent the first The active power imbalance residual value and the reactive power imbalance residual value of each sample These represent the penalty weights corresponding to the reward fitting error, the active power residual strength, and the reactive power residual strength, respectively. Let represent the squared 2-norm of a vector.

[0057] S5: By fusing hidden features from the physical topology graph with global environmental auxiliary information, a one-dimensional panoramic state vector is constructed for input to the reinforcement learning policy network. At the same time, an action space decoupling operator for continuous physical boundary mapping is designed.

[0058] Furthermore, the specific steps of S5 include:

[0059] S51: Reinforcement learning state reconstruction by fusing global auxiliary information from the execution environment, using the physical topology graph hidden feature set output by S22. The data is flattened and spliced ​​with global environmental auxiliary information, including timestamps, photovoltaic output prediction confidence levels, and time-of-use electricity prices, to form a one-dimensional panoramic state vector for the reinforcement learning strategy network observation. :

[0060]

[0061] in, For time steps The one-dimensional panoramic state vector ultimately observed by the reinforcement learning algorithm. This indicates the timestamp corresponding to the current power grid environment, used to reflect the periodic characteristics of load and photovoltaic output. This indicates the confidence level of the current photovoltaic output forecast. This indicates the time-of-use electricity price for the current period.

[0062] S52: Design an action space decoupling operator based on continuous physical boundary mapping, where the normalized control action output by the reinforcement learning policy network is located in... The interval, action space decoupling operator projects it onto the actual capacity boundary through linear mapping, generating reactive power setting instructions for the S31 physical information neural network proxy model and controllable equipment to execute. As shown in formula (16):

[0063]

[0064] in, This represents the total number of controllable devices with continuous regulation capabilities in the power distribution network. This represents the normalized control action of the i-th controllable device at time step t, as output by the reinforcement learning policy network, with a value range of... , This indicates the reactive power setting command actually issued to the controllable equipment after denormalization. and These are the lower limit and upper limit of the reactive power that the i-th controllable device is allowed to output, respectively.

[0065] Step 2: Construct a distributed value Critic network for risk-sensitive value assessment and a two-stage Actor network for continuous control action generation. The Stage-1 Actor network outputs a normalized economic reference action, while the Stage-2 Actor network outputs a normalized safety correction action based on local voltage deviations. The final normalized control action is generated through action fusion and limiting processing. The action space decoupling operator in step S52 is then converted into the actual action vector. .

[0066] Step two includes step S6, constructing a distributed value Critic network for risk-sensitive value assessment, wherein the distributed value Critic network uses a one-dimensional panoramic state vector. and the current final normalized control action As input, it outputs the probability distribution of future cumulative returns, which is used to quantify the value assessment results under source load fluctuations, state prediction errors, and control action uncertainties.

[0067] Furthermore, S6 specifically includes the following steps:

[0068] S61: To address the problem that traditional scalar Q-values ​​are insufficient to characterize the impact of random fluctuations in the power grid, a parameter is constructed as follows: The distributed value Critic network will transform the one-dimensional panoramic state vector into the current final normalized control action pair. The probability distribution of future cumulative returns is mapped as shown in formula (17):

[0069] ,

[0070] in, The expected return prediction value fitted to the output branch of the distributed value Critic network. The predicted variance of the return distribution fitted to another independent output branch of the Critic network is used to represent the distributed value. Represents a one-dimensional panoramic state vector. Represents the actual action vector. This represents the set of learnable parameters for the distributed value Critic network. Indicates by parameters The random distribution variable that determines the future cumulative return. The mean is The variance is The Gaussian distribution.

[0071] S62: Quantification of operational randomness and risk aversion based on standard deviation, utilizing standard deviation Explicitly quantifying the randomness in power grid operation (such as sudden changes in photovoltaic output and sudden load drops) aims to provide risk-sensitive control basis for multi-agent scheduling. By introducing variance penalty in policy updates, it avoids agents from taking aggressive scheduling actions under highly uncertain conditions, thereby greatly enhancing the control robustness of the algorithm in non-stationary power grid environments.

[0072] S7: Construct a Stage-1 Actor network to generate normalized economic reference actions. First, establish a one-dimensional panoramic state vector. For the input Stage-1 Actor network, the mean and standard deviation of the continuous action distribution are obtained. Then, through reparameterization, differentiable sampling is performed from the continuous action distribution, and the sampling results are compressed to the normalized action interval [-1, 1] using the hyperbolic tangent function to form a reference action for economic optimization. .

[0073] Furthermore, the specific steps of S7 include:

[0074] S71: Construction parameters are Stage-1 Actor Network Its input is a one-dimensional panoramic state vector. The output is the mean of the continuous action distribution. with standard deviation It is used to characterize the probability distribution of continuous control actions under different operating conditions, and to provide distribution parameters for subsequent economic reference action sampling.

[0075] S72: Differentiable policy sampling and action generation are performed based on reparameterization techniques. To achieve differentiable backpropagation of policy gradients in a continuous action space, reparameterization techniques are used to generate the normalized economic reference action for the first stage. As shown in formula (18):

[0076]

[0077] in, This indicates the normalized economic reference action. and Let represent the mean and standard deviation of the continuous action distribution, respectively. This represents the parameter set of the Stage-1 Actor network. Indicates the distribution from the standard normal distribution The random noise vector obtained by sampling in the middle, Represents the identity matrix; This represents the Hadamard product, which is an element-wise multiplication of vectors. This represents the hyperbolic tangent activation function, used to map sampling actions to... The normalized action range.

[0078] S8: Construct a second-stage monotonic security correction network (Stage-2 Actor network) to address the first-stage normalized economic reference action. To address potential voltage over-limit issues, a monotonic safety correction network based on stacked ReLU networks and non-negative weight constraints is introduced. Taking local voltage deviation as input, it outputs a normalized safety correction action to perform safety correction on the first-stage normalized economic reference action.

[0079] Furthermore, the specific steps of S8 include:

[0080] S81: For any controllable node i, using its local voltage deviation as input, independently outputs a normalized safety correction action for forced voltage regulation. ;

[0081] S82: The stacked linear rectifier unit structure is adopted, and the correction amount is calculated as shown in formula (19):

[0082]

[0083] in, This indicates a normalized safety correction action. This represents the total number of stacked neural network layers or hidden nodes. This represents the connection weight parameters of the k-th layer, which are subject to forced operator constraints during network updates and always satisfy... , This represents a linear rectified function that outputs the input exactly as it is when the input is greater than 0, and outputs 0 when the input is less than or equal to 0. This represents the voltage amplitude at node i at time step t. This indicates the pre-set reference value for the safe operation voltage of the power grid (e.g., 1.05 pu). This represents the bias parameter of the k-th layer.

[0084] S9: Establish a dual-stage action fusion and adaptive control switching mechanism to coordinate the normalized economic reference action of the Stage-1 Actor network output and the safety correction action of the Stage-2 Actor network output. Maintain economic control dominance when the voltage is in the safe operating range, and enhance the safety correction effect when the voltage approaches or exceeds the safety boundary.

[0085] Furthermore, the specific steps of S9 include:

[0086] S91: Normalizing Economic Reference Actions With safety correction actions The weights are combined to synthesize the final normalized control action. As shown in formula (20):

[0087]

[0088] in, This represents the safety correction factor for time step t, used to balance economy and safety. This represents a truncation function used to ensure that the merged action does not exceed the standard output range. , This represents the normalized economic reference action output by the Stage-1 Actor network. This represents the normalized safety correction action output by the Stage-2 Actor network. This represents the final normalized control action after the two-stage fusion. Input the action space decoupling operator S52 from step one and map it to the actual action vector. .

[0089] S92: When the voltage of the controllable device access node is within the safe operating dead zone, the safety correction action output by the Stage-2 Actor network approaches zero, i.e. At this point, the final normalized control action is dominated by the economic reference action output by the Stage-1 Actor network; when the voltage of any controllable device connected to the node approaches or exceeds the safety boundary, a safety correction action is initiated. Increase, and participate in motion fusion through formula (20) to ultimately normalize the control motion. Adjusting in the direction of suppressing voltage over-limit, the corresponding final normalized control action The normalized historical action features in step S13 are formed according to the correspondence between controllable devices and physical nodes.

[0090] S10: Construct a DSAC-T-based distributed Bellman objective operator and Critic loss function. Combining maximum entropy reinforcement learning and distributed value evaluation mechanisms, define the Bellman objective distribution and use KL divergence to measure the difference between the current predicted distribution and the objective distribution, driving the distributed value Critic network and the two-stage Actor network to perform closed-loop iterative updates.

[0091] Furthermore, the specific steps of S10 include:

[0092] S101: Construct a distributed Bellman objective operator that incorporates the maximum entropy mechanism, based on the instantaneous reward derived from step one. and the next state The target value distribution of DSAC-T is constructed as shown in formula (21):

[0093]

[0094] in, This indicates that both sides of the equation are completely equal in terms of probability distribution. This represents the distributed Bellman projection operator. For instant rewards, This is a time discount factor, with values ​​between (0,1), which determines the weight of future long-term returns. The entropy regularization term is used as a reward to encourage agents to maintain randomness in their actions within the physical safety boundary and to conduct extensive exploration to avoid getting trapped in local optima. Indicates the target distribution value in the Critic network in the next state. and the final normalized control action in the next moment The distribution of future returns in the output. Represents the entropy temperature coefficient. The parameter is Stage-1 Actor network, This indicates taking the logarithm of it.

[0095] S102: Define a loss function based on KL divergence for the Critic distribution distance metric. KL divergence is used to measure the difference between the target value distribution and the current predicted value distribution. The loss function is constructed as shown in formula (22):

[0096]

[0097] in, Indicates To optimize the distribution value loss function of the variables, Represents the mathematical expectation. This represents the KL divergence operator, used to measure the true distribution of targets. Compared with the current predicted distribution The information entropy difference distance between them. Minimizing this information entropy difference distance through gradient descent allows the value distribution parameters to be driven. and Stable updates converge.

[0098] Step 3: Joint Iteration and Closed-Loop Control of DSAC-T Two-Stage Actor Based on Physical Agent

[0099] This step, based on the physical information neural network agent model constructed in Step 1, executes interactive sampling, value assessment updates, economic strategy updates, security correction updates, and online closed-loop deployment of the two-stage Actor network. Specifically, Step 2 generates the final normalized control action. Step 1 S52 will Decoded into actual action vectors Step 1 S31 Reuse State simulation and reward prediction are performed to form a complete closed-loop training and control process.

[0100] S11: Perform interactive sampling and experience storage based on the physical information neural network agent model. This step inputs the current one-dimensional panoramic state vector into the two-stage Actor network to generate the final normalized control action, and obtains the actual action vector through the action space decoding operator in step S52. Then, the agent model in step S31 is used to deduce the state and immediate reward at the next moment, forming experience samples for subsequent network updates.

[0101] Furthermore, the specific steps of S11 include:

[0102] S111: Generate the forward action of the two-stage Actor network. At time step t of the reinforcement learning interaction, the agent extracts the one-dimensional panoptic state vector generated in step S1. And input it into the two-stage Actor network in step two. The Stage-1 Actor network is based on... Generate normalized economic reference action The Stage-2 Actor network generates normalized safety correction actions based on local voltage deviations. Then, the final normalized control motion vector is generated through the motion fusion and normalization limiting mechanism in step S9. As shown in formula (23):

[0103]

[0104] in, This represents the normalized economic reference action vector output by the Stage-1 Actor network; This represents the normalized safety correction action vector output by the Stage-2 Actor network; This represents the safety correction factor for time step t; This represents a truncation function used to restrict the merged actions to... Within the normalized action range; This represents the final normalized control action vector after the two-stage fusion.

[0105] S112: Execution of action space decoding and rapid derivation of the surrogate model. The final normalized control action obtained in S111 is then processed. Input the action space decoding operator of step one S52 to obtain the actual action vector. Then, the actual motion vectors... With the hidden feature set of the physical topology graph obtained in step S22 A physical information neural network surrogate model with fixed input parameters is used to obtain the predicted state vector at time step t+1. With instant rewards As shown in formula (24):

[0106]

[0107] in, The parameter is The physical information neural network proxy model. This represents the predicted state vector at time step t+1 derived by the agent network. This represents the immediate reward estimated by the agent network. A one-dimensional panoramic state vector representing time step t.

[0108] S113: Construct and store Markov empirical transformation tuples based on the predicted state vector obtained in S112. Combined with the state reconstruction method in step S51, a one-dimensional panoramic state vector is formed at time step t+1. The interactively generated experience transformation tuples are stored in the experience replay pool. As shown in formula (25):

[0109]

[0110] in, This represents the experience replay pool. The union operator represents the set operator, indicating that a new trajectory tuple is added to the playback pool. This represents the final normalized control action vector after two-stage fusion. This represents the predicted state vector at time step t+1 derived by the agent network. This represents the instantaneous reward estimated by the agent network.

[0111] S12: Perform the distribution value Critic network update. This step samples state transition samples from the experience replay pool, combines them with the DSAC-T distributed Bellman objective operator defined in step two (S10), constructs the objective value distribution, and updates the distribution value Critic network parameters by minimizing the KL divergence. .

[0112] Furthermore, the specific steps of S12 include:

[0113] S121: Calculate the target value distribution. From the experience replay pool. Medium sampling includes Mini-batch data of transition tuples. Construct the target distribution. Let the Gaussian expectation of the target distribution of the i-th sample be . The variance is As shown in formulas (26) and (27):

[0114]

[0115]

[0116] in, This is the i-th transfer tuple sampled from the playback pool. , Let represent the expected value and variance of the target distribution corresponding to the i-th sample. The parameters of the current Critic network and the optimization variables of the hysteretic objective Critic using soft updates are given. This indicates that the target policy network depends on the next time step. The sampled action vector. This represents the temperature coefficient and time discount factor. This represents the expected and variance of the predicted return output of the target Critic network.

[0117] S122: Critic network for updating distributed value based on KL divergence. KL divergence is used to measure the difference between the target value distribution and the current Critic network's predicted value distribution, and to construct the Critic network parameters. loss function And solve using the gradient descent method. To update the parameters, as shown in formula (28):

[0118]

[0119] in, This indicates the number of mini-batch samples sampled from the experience replay pool; i represents the traversal index of the mini-batch samples. This represents the one-dimensional panoramic state vector corresponding to the i-th sample; This represents the final normalized control action vector corresponding to the i-th sample; Let represent the mean of the target value distribution corresponding to the i-th sample; This represents the standard deviation of the target value distribution corresponding to the i-th sample; The parameter is The current distribution value of the Critic network is for state-action pairs. The average of the predicted returns output; This represents the standard deviation of the predicted returns output by the Critic network, representing the current distribution value. This represents the corresponding predicted return variance; Represent the natural logarithm function; The set of learnable parameters of the current distributed value Critic network.

[0120] S13: Perform Stage-1 Actor network update. This step involves fixing the distributed value Critic network parameters. and Stage-2 Actor network parameters Under these conditions, an agent objective function incorporating expected return and policy entropy is constructed, and the Stage-1 Actor network parameters are updated using a reparameterization technique. This enables the normalized economic reference action. Improve operational economy while meeting subsequent safety correction constraints.

[0121] Furthermore, the specific steps in S13 include:

[0122] S131: Construct the agent objective function with policy entropy regularization. For the states sampled in the experience replay pool... The Stage-1 Actor network generates normalized economic reference actions based on step two, S72. The final normalized control action is obtained by fusing the action with the Stage-2 safety correction action in step S9. Construct the proxy objective function As shown in formula (29):

[0123]

[0124] in, This represents the agent objective function of the Stage-1 Actor network. This represents the joint mathematical expectation over the empirical state distribution and the standard normal noise distribution. This indicates that after reparameterization, it is affected by network parameters. and noise Determined actions decided jointly Indicates the Stage-1 Actor network parameters; Represents the mathematical expectation. This represents the experience replay pool; This represents a one-dimensional panoramic state vector sampled from the experience replay pool; Indicates the distribution from the standard normal distribution A random noise vector sampled in the middle; Represents the identity matrix; This represents the normalized economic reference action generated by the Stage-1 Actor network through reparameterization; The parameter is The expected return of the Critic network output is the value of the distribution. Represents the entropy temperature coefficient; This indicates that the Stage-1 Actor network is in The following actions generate economic references. The policy probability density; This represents the natural logarithm function.

[0125] S132: Stage-1 Actor Network Parameter Update Based on Reparameterized Gradient. The reparameterization technique is used to achieve differentiable backpropagation of the action sampling process, and gradient ascent is employed for updating. As shown in formula (30):

[0126]

[0127] in, These represent the network parameters respectively. and economic reference actions Find the gradient of the partial derivatives. This represents the Stage-1 Actor network agent objective function with respect to the parameters. The gradient; This represents the relationship between state sample S and random noise. Find the expected value; The distributed value represents the mean of future cumulative returns from the Critic network output with respect to the final normalized control action. The gradient; This represents the entropy temperature coefficient. Indicates by parameters This is for the purpose of normalizing economic reference actions.

[0128] S14: Perform Stage-2 Actor network security correction update. This step defines the voltage over-limit penalty as the security constraint loss, and calculates the security loss on the Stage-2 Actor network parameters using the differentiability property of the surrogate model from Step 1. The feedback gradient is calculated, and a nonnegative projection is performed on the weights of the stacked ReLU network after the parameter update to maintain the monotonicity constraint of the safe correction network.

[0129] Furthermore, the specific steps in S14 include:

[0130] S141: Calculation of security constraint gradients based on the differentiability of the surrogate model. Independent optimization of second-stage security correction network parameters. Define the security loss function for Stage-2 Actor networks. The feedback gradient of the over-limit penalty to the correction network can be calculated using the chain rule. As shown in formulas (31) and (32):

[0131]

[0132]

[0133] in, Let j and k be the security loss function for the Stage-2 Actor network. j and k represent the sample index and the total physical node index in the batch data, respectively. This represents the weighting coefficient of the voltage over-limit penalty term. This represents the voltage amplitude prediction of node k derived from the physical neural network surrogate model. This indicates the highest safe voltage threshold specified in the power grid regulations (e.g., 1.05 pu). This represents the Jacobian sensitivity matrix of the voltage to the input action calculated internally by the model (obtained by automatic differentiation). The parameter is Stage-2 Actor network mapping.

[0134] S142: Perform non-negative projection truncation update. Update the Stage-2 Actor network parameters according to the safety constraint loss function, and after the update, perform projection truncation on all non-negative constraint connection weights in the stacked ReLU network, as shown in Equation (33):

[0135]

[0136] in, This represents the neuron weights and connection parameters in the m-th layer of the Stage-2 Actor network. (This is achieved through...) The operation removes any negative weights generated by gradient descent, thus mathematically guaranteeing the monotonically decreasing property of the network output.

[0137] S15: Execute online closed-loop deployment and control command issuance. This step deploys the trained Stage-1 Actor network, Stage-2 Actor network, and action space decoding operator to the edge control terminal, receives the actual power grid operating status in real time, generates the final normalized control action online, and decodes it into reactive power setting commands that can be executed by the underlying devices, thereby completing real-time closed-loop control.

[0138] Furthermore, the specific steps in S15 include:

[0139] S151: Online generation of two-stage fusion actions. Real-time acquisition of distribution network measurement data, and formation of a one-dimensional panoramic state vector according to steps S1 to S5. .Will The Stage-1 Actor network generated by input training Input Stage-2 Actor network to generate Then, the final normalized control action is formed through step two, S9. .

[0140] S152: Perform action space decoding and actual instruction mapping. Input the action space decoding operator of step S52 in step one to obtain the actual reactive power setting vector. .in, This represents the actual reactive power setting vector distributed to M controllable devices. This represents the actual reactive power setting command finally issued to the i-th controllable device. As shown in formula (34):

[0141]

[0142] Where M represents the total number of controlled execution devices. Represents the fused action vector The normalized action scalar corresponds to the i-th device. This indicates the actual reactive power setting command ultimately issued to the i-th controllable device. Indicates the first The current physical reactive power regulation lower and upper limits of each hardware component.

[0143] S153: Execution control command issuance and operational safety verification. Actual motion vectors are transmitted via an industrial communication interface. The control is sent to the underlying controllable devices, and the actual operating status after execution is obtained using the measurement system. The control effect is judged based on the real-time measured voltage: if the voltage of each node is within the preset safe operating range, the online closed-loop control of the next time step continues; if the node voltage is still detected to be out of limit, the control command is not correctly executed by the device, or the actual operating status exceeds the preset safe operating range, the neural network control output is paused, and the preset underlying safety control strategy or device local protection strategy is switched until the grid operating status returns to the safe range.

[0144] As shown in Table 1, under the complex dynamic scheduling conditions of a standard test system for a distribution network with a high proportion of photovoltaic and energy storage integration, the dual-mode constraint control method based on physical information graph proxy and monotonicity strategy proposed in this invention was comprehensively compared with traditional heuristic algorithms (such as genetic algorithm GA) and traditional deep reinforcement learning algorithms (such as deep deterministic policy gradient dpole generation (DDPG) and soft actor-commentator stable algorithm (SAC)). Experimental results show that, while taking into account the overall economic efficiency of system operation, this invention, with the help of the Lyapunov monotonic stability correction network, completely solves the voltage physical limit problem commonly found in pure data-driven methods (DDPG / SAC) from the underlying mathematical logic, and improves the online scheduling response speed to the millisecond level, which is far superior to traditional heuristic solution algorithms (GA). Under extreme scenarios of drastic fluctuations in photovoltaic output and sudden load changes, this invention still maintains a voltage limit rate close to zero and extremely high dynamic convergence stability.

[0145] Table 1 Comparison Test Results

[0146]

Claims

1. A dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy, characterized in that, Includes the following steps: Step 1: Data perception and physical topology proxy model construction. The multi-source operation status of the distribution network is normalized, and a physical topology graph data object containing node features, weighted adjacency matrix and edge feature matrix is ​​constructed. The topological coupling features between nodes are extracted through a multi-head graph attention mechanism, and a physical information neural network proxy model is constructed by combining the AC power flow balance residual. The hidden features of the physical topology graph and global auxiliary information are fused into a reinforcement learning state vector, and the normalized control actions are converted into executable reactive power setting instructions through action boundary mapping. Step 2: Construct a distributed value Critic network for risk-sensitive value assessment and a two-stage Actor network for continuous control action generation. The Stage-1 Actor network outputs a normalized economic reference action, while the Stage-2 Actor network outputs a normalized safety correction action based on local voltage deviations. The final normalized control action is generated through action fusion and limiting processing. And it is converted into actual action vectors by the action space decoupling operator. ; Step 3: Based on the physical information neural network agent model built in Step 1, perform interactive sampling, value assessment update, economic strategy update, security correction update, and online closed-loop deployment of the two-stage Actor network.

2. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 1, characterized in that, Step one specifically includes the following steps: S1: Normalization is used to eliminate the interference of physical dimensions on gradient updates, and a physical topology graph data object is constructed based on the physical topology of the distribution network. The physical topology graph data object includes a node feature matrix, a weighted adjacency matrix, and an edge feature matrix. S2: Based on the physical topology graph data object, first perform dynamic attention coefficient calculation by explicitly injecting edge feature vectors, then perform multi-attention head feature weighted aggregation and residual state generation. By explicitly injecting edge feature vectors into the attention scoring function and combining residual connections to alleviate the gradient decay problem in the deep feature propagation process, adaptive weighted aggregation of strongly coupled node features in the physical topology graph data object is performed to generate hidden features of the physical topology graph. S3: Hidden Feature Set Based on Physical Topology Graph A physical information neural network proxy model containing state prediction branches and reward prediction branches is constructed, and the AC power flow balance equation is transformed into a differentiable physical residual penalty term, so that the physical information neural network proxy model can maintain consistency with the physical laws of the power grid during the state inference process. S4: Generate historical operating trajectories through multi-condition sampling, and use the adaptive moment estimation algorithm to jointly optimize the state prediction error, reward fitting error and active and reactive physical residuals to obtain the physical information neural network surrogate model parameters for reinforcement learning training. S5: By fusing hidden features from the physical topology graph with global environmental auxiliary information, a one-dimensional panoramic state vector is constructed for input to the reinforcement learning policy network. At the same time, an action space decoupling operator for continuous physical boundary mapping is designed.

3. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 2, characterized in that, Step two specifically includes the following steps: S6: Construct a distributed value Critic network for risk-sensitive value assessment, wherein the distributed value Critic network uses a one-dimensional panoramic state vector. and the current final normalized control action As input, it outputs the probability distribution of future cumulative returns, which is used to quantify the value assessment results under source load fluctuations, state prediction errors, and control action uncertainties. S7: Establish a one-dimensional panoramic state vector For the input Stage-1 Actor network, the mean and standard deviation of the continuous action distribution are obtained. Differentiable sampling is performed from the continuous action distribution through reparameterization, and the sampling results are compressed to a normalized action interval of [-1, 1] using the hyperbolic tangent function, forming a reference action for economic optimization. ; S8: Construct a Stage-2 Actor network, introduce a monotonic safety correction network based on ReLU network and non-negative weight constraints, take local voltage deviation as input, and output normalized safety correction action, which is used to adjust the first-stage normalized economic reference action. Perform security modifications; S9: Establish a two-stage action fusion and adaptive control switching mechanism to coordinate the normalized economic reference action output by the Stage-1 Actor network. The safety correction action output by the Stage-2 Actor network maintains economical control when the voltage is within the safe operating range, and enhances the safety correction effect when the voltage approaches or exceeds the safety boundary. S10: Construct a DSAC-T-based distributed Bellman objective operator and Critic loss function. Combine the maximum entropy reinforcement learning mechanism and the distributed value evaluation mechanism to define the Bellman objective distribution. Use KL divergence to measure the difference between the current predicted distribution and the objective distribution to drive the distributed value Critic network and the two-stage Actor network to perform closed-loop iterative updates.

4. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 3, characterized in that, Step three specifically includes the following steps: S11: Perform interactive sampling and experience storage based on the physical information neural network agent model, input the current one-dimensional panoramic state vector into the two-stage Actor network, and generate the final normalized control action. And obtain the actual action vector through the action space decoding operator. By using the agent model to deduce the state and immediate reward of the next moment, an empirical sample is formed for subsequent network updates. S12: Perform the distributed value Critic network update, sample state transition samples from the experience replay pool, combine the DSAC-T distributed Bellman objective operator to construct the objective value distribution, and update the distributed value Critic network parameters by minimizing the KL divergence. ; S13: Perform Stage-1 Actor network update, with fixed-distribution value Critic network parameters. Stage-2Actor network parameters Under these conditions, an agent objective function incorporating expected return and policy entropy is constructed, and the Stage-1 Actor network parameters are updated through reparameterization. ; S14: Perform Stage-2 Actor network security correction update, defining voltage over-limit penalty as security constraint loss, and calculate the security loss on Stage-2 Actor network parameters using the differentiability property of the surrogate model. The feedback gradient is obtained, and a non-negative projection is performed on the weights of the stacked ReLU network after the parameter update; S15: Execute online closed-loop deployment and control command issuance, deploy the trained Stage-1 Actor network, Stage-2 Actor network, and action space decoding operator to the edge control terminal, receive the actual power grid operating status in real time, and generate the final normalized control actions online. And decode it into actual action vectors that can be executed by the underlying device. This enables real-time closed-loop control.

5. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 2, characterized in that, The specific steps of S2 include: S21: Perform dynamic attention coefficient calculation for explicit injection of edge feature vectors, for the Each attention head will focus on edge features. Inject scoring function, calculate node Influenced by neighboring nodes Unnormalized correlation score of the impact and conduct The normalization function is shown in formulas (6) and (7): Where k is the current attention head index, Normalization function, This represents the learnable scoring weight vector of the k-th attention head. Representing the node feature matrix Middle node and nodes The input feature vector, This indicates that at time step t, node j interacts with node... The unnormalized correlation score, Let represent the normalized attention coefficient of node j to node i at time step t. This represents the node feature mapping matrix and edge feature mapping matrix of the k-th attention head. For nodes The set of first-order adjacent nodes, This represents the concatenation operation of vector dimensions. This represents the natural exponential function, used to convert unnormalized correlation scores into positive weights. This represents the unnormalized correlation score between node m and node i at time step t. S22: Perform multi-head feature weighted aggregation and residual state generation, The weighted aggregation results of each attention head are concatenated, and residual connections are introduced to generate nodes. At time step Hidden state The set of hidden features of the physical topology graph, which is composed of the hidden features of all nodes. As shown in formulas (8) and (9): in, It is a non-linear activation function. Represents the set of hidden features in the physical topology graph. This represents the high-dimensional hidden feature vector of node i after fusing topological neighborhood information at time step t. Represents a node The high-dimensional hidden feature vector after fusing topological neighborhood information at time step t, i.e. The set has N nodes. Indicates will The output vectors of each attention head are concatenated. This represents the learnable weight matrix used for residual connection mapping. For nodes The set of first-order adjacent nodes, Represents the normalized attention coefficient of node j to node i at time step t; The specific steps of S3 include: S31: Establish an environmental dynamics feedforward network The environmental dynamics feedforward network hides the feature set using a physical topology graph. and time step Actual action vectors issued to controllable equipment in the distribution network Given the input, output the predicted state for the next time step. With instant rewards As shown in formulas (10) and (11): in These represent the network parameters respectively. The decision-making state prediction branch and the immediate reward prediction branch, This represents the operation of flattening the set of hidden features in the physical topology graph into a one-dimensional vector. Indicates time step The actual action vectors issued to the controllable equipment in the power distribution network in real time. Indicates the time step of the deduction. The predicted state vector contains the inversely normalized predicted voltage magnitude. Phase angle with predicted voltage ; S32: Construct a differentiable physical residual penalty term based on the AC power flow equation, and extract... The predicted voltage amplitude and predicted voltage phase angle are used to construct the time step based on Kirchhoff's laws. Active power imbalance residual at node i Reactive power imbalance residual As shown in formulas (12) and (13): in, Representing nodes respectively At time step The active power imbalance residual value and the reactive power imbalance residual value, Representing nodes respectively At time step The theoretical active power injection and the theoretical reactive power injection, These represent the nodes predicted by S31. and The voltage amplitude at time step t+1, These represent the nodes predicted by S31. and The voltage phase angle at time step t+1, and These are the connection nodes. and Branch conductance and branch susceptance, For nodes The set of first-order adjacent nodes.

6. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 2, characterized in that, The specific steps of S4 include: S41: Perform offline sampling and reconstruction of multi-condition operating trajectories, generating trajectory data including state transitions through Monte Carlo sampling or historical operating data sampling, forming a dataset for the physical information neural network surrogate model. A single sample record is as follows: ， in, , Indicates time step The actual action vectors issued to the controllable equipment in the power distribution network in real time. This represents the state at time step t+1. This represents the actual reward at time step t; S42: Construct a composite loss function that encompasses state prediction, reward fitting, and the physical residuals of active and reactive power. And the parameters of the physical information neural network surrogate model are updated using the adaptive moment estimation algorithm, as shown in formula (14): in, For network parameters The defined total loss function, These represent the traversal indices for the batch size and the current batch size, respectively. They represent the first The predicted state vector and the true state label vector of each sample. They represent the first The predicted instant reward and the actual instant reward for each sample. They represent the first The active power imbalance residual value and the reactive power imbalance residual value of each sample These represent the penalty weights corresponding to the reward fitting error, the active power residual strength, and the reactive power residual strength, respectively. Represents the squared 2-norm of a vector; The specific steps of S5 include: S51: Reinforcement learning state reconstruction by fusing global auxiliary information from the execution environment, and hiding the physical topology graph feature set. The data is flattened and spliced ​​with global environmental auxiliary information, including timestamps, photovoltaic output prediction confidence levels, and time-of-use electricity prices, to form a one-dimensional panoramic state vector for the reinforcement learning strategy network observation. : in, For time steps The one-dimensional panoptic state vector ultimately observed by the reinforcement learning policy network. This represents the timestamp corresponding to the current power grid environment. This indicates the confidence level of the current photovoltaic output forecast. This indicates the time-of-use electricity price for the current period; S52: Design an action space decoupling operator based on continuous physical boundary mapping, where the normalized control action output by the reinforcement learning policy network is located in... The interval, action space decoupling operator projects it onto the actual capacity boundary through linear mapping, generating physical information neural network proxy model and reactive power setting instructions executed by controllable equipment. As shown in formula (16): in, This represents the total number of controllable devices with continuous regulation capabilities in the power distribution network. This represents the final normalized control action of the i-th controllable device at time step t, as output by the reinforcement learning policy network. This indicates the reactive power setting command actually issued to the controllable equipment after denormalization. and These are the lower limit and upper limit of the reactive power that the i-th controllable device is allowed to output, respectively.

7. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 3, characterized in that, The specific steps of S6 include: S61: Construction parameters are The distributed value Critic network links the one-dimensional panoramic state vector with the current final normalized control action pair. The probability distribution of future cumulative returns is mapped as shown in formula (17): ， in, The expected return prediction value fitted to the output branch of the distributed value Critic network. The predicted variance of the return distribution fitted to another independent output branch of the Critic network is used to represent the distributed value. Represents a one-dimensional panoramic state vector. Represents the actual action vector. This represents the set of learnable parameters for the distributed value Critic network. Indicates by parameters The random distribution variable that determines the future cumulative return. The mean is The variance is Gaussian distribution; S62: Quantification of operational randomness and risk aversion based on standard deviation, utilizing standard deviation The randomness in power grid operation is explicitly quantified, and variance penalty is introduced into the strategy update. The specific steps of S7 include: S71: Construction parameters are Stage-1 Actor Network Its input is a one-dimensional panoramic state vector. The output is the mean of the continuous action distribution. with standard deviation It is used to characterize the probability distribution of continuous control actions under different operating conditions; S72: Use reparameterization to generate the normalized economic reference action for the first stage. As shown in formula (18): in, This indicates the normalized economic reference action. and Let represent the mean and standard deviation of the continuous action distribution, respectively. This represents the parameter set of the Stage-1 Actor network. Indicates the distribution from the standard normal distribution The random noise vector obtained by sampling in the middle, Represents the identity matrix; This represents the Hadamard product, which is an element-wise multiplication of vectors. This represents the hyperbolic tangent activation function.

8. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 3, characterized in that, The specific steps of S8 include: S81: For any controllable node i, using its local voltage deviation as input, independently outputs a normalized safety correction action for forced voltage regulation. ; S82: The stacked linear rectifier unit structure is adopted, and the correction amount is calculated as shown in formula (19): in, This indicates a normalized safety correction action. This represents the total number of layers in a stacked neural network. This represents the connection weight parameters of the k-th layer, which are subject to forced operator constraints during network updates. This represents a linear rectified function that outputs the input exactly as it is when the input is greater than 0, and outputs 0 when the input is less than or equal to 0. This represents the voltage amplitude at node i at time step t. This represents the pre-set reference value for the safe operation voltage of the power grid. This represents the bias parameter of the k-th layer; The specific steps of S9 include: S91: Normalizing Economic Reference Actions With safety correction actions The weights are combined to synthesize the final normalized control action. As shown in formula (20): in, This represents the safety correction factor for time step t. This represents a truncation function used to ensure that the merged action does not exceed the standard output range. , This represents the normalized economic reference action output by the Stage-1 Actor network. This represents the normalized safety correction action output by the Stage-2 Actor network. This represents the final normalized control action after the two-stage fusion; S92: When the voltage of the controllable device access node is within the safe operating dead zone, the final normalized control action is dominated by the economic reference action output by the Stage-1 Actor network; when the voltage of any controllable device access node approaches or exceeds the safety boundary, a safety correction action is initiated. Increase, and participate in motion fusion through formula (20) to ultimately normalize the control motion. Adjusting in the direction of suppressing voltage over-limit, the corresponding final normalized control action Normalized historical action characteristics are formed based on the correspondence between controllable devices and physical nodes; The specific steps of S10 include: S101: Construct a distributed Bellman objective operator that integrates the maximum entropy mechanism, based on the derived instantaneous reward. and the next state The target value distribution of DSAC-T is constructed as shown in formula (21): in, This indicates that both sides of the equation are completely equal in terms of probability distribution. This represents the distributed Bellman projection operator. For instant rewards, As a time discount factor, As a reward for entropy regularization, Indicates the target distribution value in the Critic network in the next state. and the final normalized control action in the next moment The distribution of future returns in the following output, Represents the entropy temperature coefficient. The parameter is Stage-1 Actor network, This indicates taking its logarithm; S102: Define a loss function based on KL divergence for the Critic distribution distance metric. KL divergence is used to measure the difference between the target value distribution and the current predicted value distribution. The loss function is constructed as shown in formula (22): in, Indicates To optimize the distribution value loss function of the variables, Represents the mathematical expectation. This represents the KL divergence operator, used to measure the true distribution of targets. Compared with the current predicted distribution The information entropy difference distance between them.

9. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 4, characterized in that, The specific steps of S11 include: S111: Generate forward actions using the two-stage Actor network. At time step t of the reinforcement learning interaction, extract a one-dimensional panoptic state vector. And input it into a two-stage Actor network, the Stage-1 Actor network according to Generate normalized economic reference action The Stage-2 Actor network generates normalized safety correction actions based on local voltage deviations. Then, the final normalized control motion vector is generated through motion fusion and normalization limiting mechanisms. As shown in formula (23): in, This represents the normalized economic reference action vector output by the Stage-1 Actor network; This represents the normalized safety correction action vector output by the Stage-2 Actor network; This represents the safety correction factor for time step t; This represents a truncation function used to restrict the merged actions to... Within the normalized action range; This represents the final normalized control action vector after the two-stage fusion; S112: Decode the action space and rapidly extrapolate the proxy model to obtain the final normalized control action from S111. Input the action space decoding operator to obtain the actual action vector. , to the actual action vector With physical topology graph hidden feature set A physical information neural network surrogate model with fixed input parameters is used to obtain the predicted state vector at time step t+1. With instant rewards As shown in formula (24): in, The parameter is The physical information neural network proxy model, This represents the predicted state vector at time step t+1 derived by the agent network. This represents the immediate reward estimated by the agent network. A one-dimensional panoramic state vector representing time step t; S113: Predicted state vector obtained from S112 By combining the state reconstruction method, a one-dimensional panoramic state vector is formed at time step t+1. The interactively generated experience transformation tuples are stored in the experience replay pool. As shown in formula (25): in, This represents the experience replay pool. The union operator represents the set operator, indicating that a new trajectory tuple is added to the playback pool. This represents the final normalized control action vector after two-stage fusion. This represents the predicted state vector at time step t+1 derived by the agent network. This represents the immediate reward estimated by the agent network; The specific steps in S12 include: S121: Calculate the target value distribution from the experience replay pool. Medium sampling includes Construct the target distribution from a small batch of transition tuples. Let the Gaussian expectation of the target distribution of the i-th sample be . The variance is As shown in formulas (26) and (27): in, For the i-th transfer tuple sampled from the playback pool, , Let represent the expected value and variance of the target distribution corresponding to the i-th sample. Let be the current distributed value Critic network parameters and the optimization variables for the hysteretic objective Critic using soft updates. This indicates that the target policy network depends on the next time step. The sampled action vector, This represents the temperature coefficient and time discount factor. This represents the expected value and variance of the predicted return output by the target Critic network; S122: KL divergence is used to measure the difference between the target value distribution and the value distribution predicted by the current value Critic network, and to construct the distribution value Critic network parameters. loss function And solve using the gradient descent method. To update the parameters, as shown in formula (28): in, This indicates the number of mini-batch samples sampled from the experience replay pool; i represents the traversal index of the mini-batch samples. This represents the one-dimensional panoramic state vector corresponding to the i-th sample; This represents the final normalized control action vector corresponding to the i-th sample; Let represent the mean of the target value distribution corresponding to the i-th sample; This represents the standard deviation of the target value distribution corresponding to the i-th sample; The parameter is The current distribution value of the Critic network is for state-action pairs. The average of the predicted returns output; This represents the standard deviation of the predicted returns output by the Critic network, representing the current distribution value. This represents the corresponding predicted return variance; Represents the natural logarithm function; This represents the set of learnable parameters for the current distributed value Critic network.

10. The dual-mode constrained reinforcement learning power grid control method based on physical information graph proxy and monotonicity strategy according to claim 4, characterized in that, The specific steps in S13 include: S131: Constructing the agent objective function with policy entropy regularization As shown in formula (29): in, This represents the agent objective function of the Stage-1 Actor network. This represents the joint mathematical expectation over the empirical state distribution and the standard normal noise distribution. This indicates that after reparameterization, it is affected by network parameters. and noise Determined actions decided jointly Indicates the Stage-1 Actor network parameters; Represents the mathematical expectation. This represents the experience replay pool; This represents a one-dimensional panoramic state vector sampled from the experience replay pool; Indicates the distribution from the standard normal distribution A random noise vector sampled in the middle; Represents the identity matrix; This represents the normalized economic reference action generated by the Stage-1 Actor network through reparameterization; The parameter is The expected return of the Critic network output is the value of the distribution. Represents the entropy temperature coefficient; This indicates that the Stage-1 Actor network is in The following actions generate economic references. The policy probability density; Represents the natural logarithm function; S132: Differentiable backpropagation of the action sampling process is achieved through reparameterization, and gradient ascent is used for updating. As shown in formula (30): in, These represent the network parameters respectively. and economic reference actions Find the gradient of the partial derivatives. This represents the Stage-1 Actor network agent objective function with respect to the parameters. The gradient; This represents the relationship between state sample S and random noise. Find the expected value; The distributed value represents the mean of future cumulative returns from the Critic network output with respect to the final normalized control action. The gradient; This represents the entropy temperature coefficient. Indicates by parameters For the purpose of normalizing economic reference actions; The specific steps in S14 include: S141: Define the security loss function for Stage-2 Actor networks The feedback gradient of the over-limit penalty to the correction network is calculated using the chain rule. As shown in formulas (31) and (32): in, Let j and k be the security loss function for the Stage-2 Actor network, where j and k represent the sample index and the total physical node index in the batch data, respectively. This represents the weighting coefficient of the voltage over-limit penalty term. This represents the voltage magnitude prediction of node k derived from the physical neural network surrogate model. This indicates the highest safe voltage threshold specified in the power grid regulations. This represents the Jacobian sensitivity matrix of the voltage to the input action, calculated internally by the model. The parameter is Stage-2 Actor network mapping; S142: Perform non-negative projection truncation update, update the Stage-2 Actor network parameters according to the safety constraint loss function, and perform projection truncation on all non-negative constraint connection weights in the stacked ReLU network after the update, as shown in formula (33): in, This represents the neuron weights and connection parameters in the m-th layer of the Stage-2 Actor network.

Images