A power information system self-healing decision optimization method based on reinforcement learning feedback

Abstract

This invention discloses a self-healing decision optimization method for power information systems based on reinforcement learning feedback. The method includes: acquiring operational data to construct a dynamic graph structured state representation; formalizing the self-healing objective into a temporal logic reduction based on signal temporal logic; predicting the quantitative robustness of the state trajectory to the reduction, and generating a dense reward signal in the form of a potential function difference to reshape the feedback reward; finally, training the agent based on the reshaped reward, the graph network foundation, and the course learning mechanism, outputting an optimal decision sequence including topology reconstruction and resource scheduling. This invention eliminates the reward sparsity problem of traditional models, guides the agent to find the optimal solution under strict physical temporal constraints, effectively avoids the risk of cascading failures caused by trial and error, and significantly improves the convergence speed and online decision reliability of self-healing in high-dimensional complex power grids.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for power systems, and in particular to a self-healing decision optimization method for power information systems based on reinforcement learning feedback. Background Technology

[0002] With the advancement of new power systems, the power physical network and information network are deeply integrated, evolving into a highly complex power information system. Fault self-healing, as a core mechanism to ensure the high reliability of power information systems, requires rapid isolation of risks and restoration of power supply and network communication after a fault. Traditional self-healing strategies often rely on expert rules or heuristic optimization algorithms, which often exhibit limitations in solution efficiency and generalization ability when dealing with the massive operational data, multidimensional control variables, and dynamic topology changes of modern power grids. In recent years, deep reinforcement learning, with its outstanding performance in handling high-dimensional sequence decision-making problems, has been gradually introduced into the fields of self-healing control, resource scheduling, and topology reconstruction in power information systems, providing a highly promising solution for achieving intelligent and automated autonomous operation and maintenance decision-making.

[0003] However, existing reinforcement learning-based self-healing decision-making techniques still have some shortcomings in practical applications. First, existing methods mostly use one-dimensional vectors to represent the system state, severing the inherent spatial graph structure relationship between bus nodes and branches, making it difficult for the agent to effectively perceive the dynamic topological evolution caused by the fault. Second, because power self-healing is often a long sequence process involving multiple stringent temporal constraints such as safety and stability, traditional reinforcement learning generally faces serious problems of reward sparsity and delayed feedback. This mechanism of only providing evaluation at the end of the task not only makes credit allocation in intermediate decision steps extremely difficult and model convergence extremely slow, but also makes it difficult for conventional heuristic rewards to rigorously formalize the complex power time series specifications mathematically, thus making the agent prone to outputting illegal actions that trigger cascading faults during trial and error exploration. In addition, when faced with extremely complex fault scenarios, traditional models often lack training guidance mechanisms, further exacerbating the trial and error costs and global optimization difficulties in high-dimensional action spaces. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a self-healing decision optimization method for power information systems based on reinforcement learning feedback, to solve the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a self-healing decision optimization method for power information systems based on reinforcement learning feedback, comprising:

[0007] Acquire real-time operational data of the power information system and construct it into a graph-structured state representation that includes node features, edge features, and topological relationships;

[0008] Based on signal timing logic, the pre-defined self-healing objectives of the power information system in terms of security, stability and resilience are formalized into a set of logic specifications containing timing constraints.

[0009] Based on the graph-structured state representation, the degree to which the future state trajectory of the power information system satisfies the logical specification is calculated or predicted, and quantified into one or more quantitative robustness values.

[0010] Based on the change in the quantitative robustness value before and after the state of the power information system changes due to decision-making, a dense reward signal is generated, and the feedback reward of the reinforcement learning agent is reshaped using the signal.

[0011] The reinforcement learning agent is trained through continuous interaction with the simulated environment based on the reshaped feedback reward, and learns and outputs the optimal decision sequence for fault self-healing in power information systems.

[0012] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0013] 1. This invention abandons the traditional one-dimensional state vector and constructs a dynamic graph structured state representation that includes node features, edge features, and switch on / off states. Combined with a shared underlying graph neural network encoder, this enables the agent to adaptively perceive topology changes caused by fault isolation or route reconstruction, accurately capture the spatial cascading effects of fault propagation, and thus improve the model's feature extraction and generalization capabilities in complex dynamic power grid environments.

[0014] 2. This invention, based on signal-sequence logic, transforms the abstract safety, stability, and recoverability specifications in long-sequence self-healing tasks into continuously differentiable quantitative robustness, and generates a dense reshaping reward based on potential function difference. This mechanism distributes the evaluation, originally given only at the end of the task, to each intermediate decision step, laying a clear value gradient descent path for the agent, avoiding ineffective blind trial and error in high-dimensional action spaces, and accelerating network convergence.

[0015] 3. Furthermore, this invention transforms the stringent power industry dispatching and operation procedures into mathematical constraints understandable to the intelligent agent, and introduces a course learning mechanism with quantified trigger conditions during the training phase to guide the model in safe optimization. This ensures that the joint actions output by the deployed model are strictly limited by the physical safety margin of the equipment, eliminating potentially fatal hidden dangers that could trigger cascading collapses, and meeting the reliability requirements of critical power infrastructure. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0017] Figure 1 This is a flowchart illustrating the overall process of a self-healing decision optimization method for power information systems based on reinforcement learning feedback, as described in one embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0019] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0020] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0021] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0022] Example 1

[0023] Reference Figure 1 This is the first embodiment of the present invention, which provides a self-healing decision optimization method for power information systems based on reinforcement learning feedback, including:

[0024] S1. Obtain real-time operation data of the power information system and construct it into a graph-structured state representation that includes node features, edge features, and topological relationships.

[0025] It should be noted that traditional reinforcement learning methods typically concatenate the states of a power system into a one-dimensional vector. This approach, when used for feature dimensionality reduction or fully connected network processing, severs the inherent spatial topological relationships of the power system, preventing the agent from perceiving dynamic topological changes caused by fault isolation or route reconstruction. To address this technical problem, this invention abandons the traditional one-dimensional state representation method and integrates the physical connections and real-time operational status of the power information system using graph theory methods, constructing a dynamic graph-structured state suitable for graph neural network (GNN) processing. .

[0026] Furthermore, in this embodiment, at a specific time step, the structured state representation of the dynamic graph can be defined as a tuple. For this tuple, we first construct the node features.

[0027] Specifically, the energy aggregation point, i.e., the bus, in the power information system is defined as the set of nodes in the graph. Let the total number of nodes be . For any node It acquires real-time operating data through SCADA (Supervisory Control and Data Acquisition) or PMU (Phasor Measurement Unit), extracting bus voltage, phase angle, load information, and generator output as node characteristics. Among these, the node... At time step eigenvectors The mathematical expression can be:

[0028]

[0029] in, Indicates bus node The real-time voltage amplitude; Indicates bus node The real-time voltage phase angle is used to characterize the power angle stability of the power information system; and These respectively represent connections to the nodes. The active and reactive power output of the generator set (if there is no generator at this node, the value is 0). and Representing nodes respectively The active and reactive loads consumed at each node are then concatenated. This process involves piecing together the feature vectors of all nodes to construct the power information system. The node feature matrix at time t. T represents the transpose operation. This feature matrix is ​​used to endow artificial intelligence with the ability to perceive local power supply and demand balance and the risk of node voltage exceeding limits.

[0030] Furthermore, after constructing node features, edge features are constructed. Transmission lines and transformers, which perform the functions of power transmission and network routing, are defined as the set of edges in the graph. For connected nodes and nodes branch road Branch power flow, switch on / off status, and impedance parameters are extracted as edge features. At time step eigenvectors The mathematical expression is:

[0031]

[0032] in, and Representing branch roads The active and reactive power transmission volumes on the network; and These represent the inherent resistance and reactance of the branch, respectively, and are used to reflect the network transmission limits and loss characteristics of the physical line. This represents the real-time on / off status of the circuit breakers or disconnectors at both ends of a branch, with 1 indicating closed (connected) and 0 indicating open. Similarly, the features of all branches can form an edge feature matrix. By explicitly incorporating the switch states representing the results of control commands into the feature space, the reinforcement learning agent can intuitively evaluate the specific physical impact of opening or closing a line on the power flow transfer of adjacent branches.

[0033] Furthermore, based on the node and edge features defined above, a dynamic adjacency matrix is ​​constructed to characterize spatial connectivity relationships. To enable the artificial intelligence model to respond in real time to fault resolution or self-healing reconfiguration in the power information system, this embodiment couples the on / off state of the switch with the topology matrix. The update rule for the adjacency matrix elements is defined as follows:

[0034]

[0035] It is important to emphasize that in this formula, the corresponding node in the graph structure is connected by an edge only when the physical branch exists and the switch is closed.

[0036] It should be noted that, through the above processing, this invention can transform highly nonlinear, low-level sensor data of power information systems into a standardized, spatiotemporally correlated graph-structured state representation. This lays a data foundation for subsequent reinforcement learning agents to perceive the state space, ensuring that even in extreme scenarios where equipment failures or self-healing actions in the power information system drastically alter the network topology, the graph structure can adaptively reflect topology changes, enabling artificial intelligence to accurately capture the spatial cascading effects of fault propagation.

[0037] S2. Based on signal timing logic, the preset self-healing objectives of the power information system in terms of security, stability and resilience are formalized into a set of logic specifications containing timing constraints.

[0038] It should be noted that when a power information system executes self-healing decisions, focusing solely on the final recovery result is insufficient; the system must meet stringent physical constraints and timeliness requirements throughout the entire recovery trajectory. Therefore, this embodiment introduces Signal Temporal Logic (STL) as a translator between expert knowledge and reinforcement learning algorithms, used to transform the fuzzy self-healing objective into a mathematical reduction for computer computation, denoted as... .

[0039] Furthermore, we define atomic predicates describing the state of a power system, denoted as . For a certain signal in the system state trajectory (For example, the voltage signal at a node), this predicate can be defined as Based on this, a logic reduction is constructed using time-domain operators in signal-sequential logic. The operators used in this embodiment include the following two types:

[0040] start operator : Indicates a time period Within this framework, the regulations must be met at all times;

[0041] Final operator : Indicates a time period Within this period, the specification must be satisfied at least once.

[0042] Furthermore, in this invention, the system self-healing objective is decomposed into three specific logical specifications. , , This is to ensure the comprehensiveness and compliance of the system's self-healing process.

[0043] Specifically, regarding security specifications This specification is used to constrain the operating parameters of the power information system to not exceed a safety threshold within a preset time window. In this embodiment, node voltage is used as the reference. Taking the example of not exceeding the limit, its expression can be:

[0044]

[0045] in, and These are the upper and lower limits of the allowable bus voltage (e.g., 1.05 and 0.95 per unit). This indicates the total duration of the self-healing assessment. Represents logical AND operations.

[0046] It should be noted that the definition of the safety specification is to require that the system must not sacrifice voltage safety in pursuit of recovery speed throughout the entire process of self-healing decision execution, so as to prevent the risk of voltage collapse or equipment damage during the self-healing process.

[0047] Specifically, regarding stability specifications This specification defines how key indicators (such as system frequency or power angle balance) should return to a stable range within a specified time after a fault is cleared. In this embodiment, system frequency is used as the reference value. For example, the expression for recovery is:

[0048]

[0049] in, Indicates the maximum allowed recovery time; Indicates the rated frequency (50Hz); Indicates the allowable steady-state deviation; This indicates the observation window that needs to be maintained after reaching steady state.

[0050] It should be noted that, because traditional reward functions are difficult to describe the dynamic characteristic of maintaining stability after a certain point in time, the definition of this stability specification clarifies the deadline for stability recovery and provides a clear temporal guide for reinforcement learning.

[0051] Specifically, regarding the restorative statute This specification characterizes the maximum recovery of a lost power load within a specified time, and its expression is:

[0052]

[0053] in, This represents the amount of load recovered by node i after self-healing; This indicates the total load demand of the system before the failure. This represents the preset minimum load recovery ratio, and its value ranges from [0,1].

[0054] It should be noted that the purpose of restorative specification is to refine the goal of restoring as much load as possible into hard indicators with time cutoffs and quantity requirements, so as to ensure that the decision sequence output by the agent has socio-economic benefits.

[0055] Furthermore, by integrating the aforementioned multidimensional objectives through logic and operations, a global self-healing logic specification can be formed. .

[0056] It should be noted that, through the above processing, this invention can transform power system operation criteria into a mathematical language understandable by reinforcement learning. This logical specification not only serves as a standard for evaluating the quality of decisions, but also, compared to traditional rewards based on heuristic functions, the logical specification generated based on signal-time logic has extremely strong interpretability and rigorous physical boundaries. This allows the self-healing decision sequences generated by the reinforcement learning agent to strictly adhere to the power industry's dispatching and operation procedures, thus solving the problem of unreliability in artificial intelligence decision-making applications in critical power infrastructure.

[0057] S3. Based on graph-structured state representation, calculate or predict the degree to which the future state trajectory of the power information system satisfies the logic specification, and quantify it into one or more quantitative robustness values.

[0058] Furthermore, traditional reinforcement learning, if Boolean logic (i.e., 1 for compliance and 0 for non-compliance) is directly used as feedback, will face extremely flat gradients and severe reward sparsity problems, causing the neural network to be unable to determine the direction of evolution. To address this issue, this embodiment introduces the concept of quantitative robustness based on signal-time logic. Due to the characteristics of robustness, it can not only determine whether the system meets the requirement, but also measure the margin of compliance or the severity of violation, thereby transforming discrete logical judgments into continuously differentiable mathematical evaluations.

[0059] Furthermore, in a real-world physical scenario, assuming we are at the current moment... Since the agent has not yet executed a complete decision sequence, it cannot directly obtain the future real state trajectory to calculate the true robustness. Therefore, this invention provides a predictive model built using deep neural networks to estimate robustness before decision-making.

[0060] Specifically, for the construction and application of the above prediction model, this embodiment designs it as a parallel output module of the actor-critic framework in reinforcement learning. Its forward inference and computation process includes the following steps:

[0061] (i) The graph-structured state constructed in S1 is input into a graph neural network encoder. This graph neural network encoder fuses local node features with the global topological structure through graph convolution operations. In this embodiment, the graph structured state constructed in S1 is used as an example. Taking layered graph convolution calculation as an example, its mathematical expression is:

[0062]

[0063] in, This represents an adjacency matrix with self-loops; for The degree matrix; For the first Hidden features of the layer (initial input) That is, an aggregated representation of node feature matrices combined with edge features). The weight matrix is ​​a learnable matrix; It is the ReLU nonlinear activation function.

[0064] Then, after multi-layer graph convolution and global pooling operations, the graph neural network encoder outputs a low-dimensional embedding vector that represents the real-time operation and topological status of the global power information system, denoted as . .

[0065] It should be noted that, in order to meet the stringent requirements of real-time decision-making in power systems for computational timeliness and to promote the generalization ability of feature representation learning, this embodiment constructs a joint neural network architecture with a shared bottom layer and multiple head outputs.

[0066] Specifically, the graph neural network encoder described above serves as the shared underlying feature extractor for the entire reinforcement learning agent. The prediction model, along with the subsequent actor and critic networks, are all parallel, independent output layers of this joint architecture. The low-dimensional embedding vectors output by the graph neural network encoder are fed into these three modules simultaneously and in parallel. During prediction model training, the gradients generated by these three modules are backpropagated to the graph neural network encoder, forcing it to learn a general power grid state representation that contains both decision-making value and logical robustness.

[0067] (ii) The aforementioned low-dimensional embedding vector is fed into the parallel prediction model module. This module can be composed of a multilayer perceptron (MLP) to evaluate the ease with which the future state trajectory conforms to the global self-healing logic reduction in S2 under the current system state. The prediction model outputs the current prediction quantitative robustness value. for:

[0068]

[0069] in, The parameter is The predictive neural network.

[0070] It should be noted that this quantitative robustness value is equivalent to a safety radar. If the quantitative robustness value is greater than 0, it indicates that the model predicts that the system has a high degree of confidence in successfully self-healing under the current situation and does not violate physical timing constraints. The larger the value, the higher the safety margin (for example, the voltage distance is further away from the limit boundary). If the quantitative robustness value is less than 0, it indicates that the system is sliding towards the edge of violation. The smaller the value, the higher the risk of triggering cascading failures.

[0071] Furthermore, to ensure that the prediction model can accurately estimate the complex dynamic evolution of the power system, we also need to train it. In this embodiment, a post-hoc experience replay mechanism is used to supervise the training of the prediction model.

[0072] Specifically, when a reinforcement learning agent completes a full round, i.e., a real interaction trajectory... Then, the true robustness value of the trajectory is calculated ex-post using the signal timing logic rules defined in S2, denoted as . The calculation of true robustness follows the continuous semantic rules of signal temporal logic. For example, for the time-to-earth operator, the minimum value within the trajectory segment is used. For the final operator, the maximum value is used. Subsequently, the mean squared error (MSE) loss function for the prediction model was constructed:

[0073]

[0074] in, This is the training batch size.

[0075] Furthermore, the parameters are optimized using the gradient descent algorithm. This minimizes the error between the predicted quantitative robustness value and the actual robustness value.

[0076] It should be noted that, through the above processing, the present invention can compress the temporal constraints of multiple future steps into a continuous scalar value (prediction robustness) at the current moment, thereby giving the artificial intelligence model the ability to predict the future without running physical simulation.

[0077] S4. Based on the change in the quantitative robustness value before and after the state of the power information system changes due to decision-making, a dense reward signal is generated, and the signal is used to reshape the feedback reward of the reinforcement learning agent.

[0078] It should be noted that in the self-healing process of complex power information systems, agents often need to execute long-sequence decisions consisting of dozens of topology reconfiguration and resource scheduling actions (e.g., first disconnecting the circuit breaker adjacent to the fault point, then closing the tie switch, and finally adjusting the generator output). Traditional reinforcement learning typically only provides a sparse global reward when the entire sequence is completed, the system restores power, or a cascading collapse is triggered. This delayed feedback mechanism leads to a serious credit allocation problem, where the agent cannot determine which action in the long-sequence decision-making process led to the final success or failure, resulting in a massive amount of invalid trial and error, and even causing the model to fail to converge. To solve this problem, this embodiment introduces a reward reshaping mechanism based on a potential function, transforming the quantitative robustness value predicted in S3 into a dense reward guiding the agent.

[0079] Furthermore, the process of constructing intensive reward signals and reshaping feedback rewards is as follows:

[0080] First, define the basic reward signal from the system environment feedback. This signal is used to reflect the execution result of hard tasks at the underlying level of the power information system. It is usually triggered only at the end of a round, and its mathematical expression can be set as follows:

[0081]

[0082] Then, using the prediction model in S3, the state of the power information system is transformed into potential energy for assessing safety margins and target achievement. Specifically, we can assume the current time step is... The system is in a state. (That is, the current graph structured state), at this point the quantitative robustness value output by the prediction model is defined as the first potential function value, denoted as... Then, when the agent executes a self-healing decision... When switching the state of a switch, the physical environment of the system is driven by this action to transition from the current state to the new state at the next moment. Then, the prediction model is invoked again, and the quantitative robustness value output under the new state is used. Defined as the value of the second potential function, denoted as .

[0083] Finally, based on the potential function reward reshaping theory in reinforcement learning, in order to ensure that the optimal policy learned by the agent remains consistent with the original goal after the reward function is changed, and to prevent score manipulation, i.e., the agent repeatedly jumping between two states to cheat for rewards, the proposed solution of this invention designs the additional reward term as a temporal difference form of the potential function.

[0084] Specifically, the second potential function value, after being attenuated by a discount factor, is subtracted from the first potential function value, and this difference is used as an additional term to reshape the basic reward signal. ,get:

[0085]

[0086] in, A discount factor is used in reinforcement learning algorithms to weigh the value of future states.

[0087] Ultimately, this results in a reshaped, intensive feedback reward system. :

[0088]

[0089] It should be noted that in the above formula, if This means that the intelligent agent executes the action. Subsequently, the quantitative robustness of the system state improves (e.g., by removing some non-critical loads, the bus voltage is moved further away from the collapse edge, increasing the system safety margin). At this point, the reshaping reward is positive, and the agent receives immediate praise for the current step, encouraging it to continue exploring in the direction of improving robustness; if This means action This leads to a decrease in the system's quantitative robustness (e.g., an incorrect loop-closing operation causes a surge in short-circuit current, violating the trend of safety specifications). In this case, the reshaping reward becomes negative, and the agent is immediately penalized, thus learning to avoid such high-risk actions before triggering a real physical catastrophe. Through the above processing, this invention distributes the delayed evaluation, which originally only existed at the end of the task, to every intermediate decision-making step. This changes the inefficient exploration of traditional reinforcement learning in power self-healing tasks, paving a value gradient descent path with rigorous physical constraints for the agent, and improving the convergence speed and optimization accuracy of artificial intelligence models in the vast power state space.

[0090] S5. The reinforcement learning agent learns and outputs the optimal decision sequence for fault self-healing in power information systems by continuously interacting with the simulated environment based on the reshaped feedback reward.

[0091] Furthermore, when facing the high-dimensional, nonlinear physical environment of power information systems, which includes a mixed action space (containing both discrete actions of switching states and continuous actions of generator output), traditional reinforcement learning algorithms often suffer from convergence difficulties or are prone to getting trapped in local optima. Therefore, this embodiment uses the aforementioned actor-critic framework as the brain of the reinforcement learning agent, supplemented by a curriculum learning strategy, to guide the agent to extract safe and efficient self-healing knowledge from massive amounts of trial and error.

[0092] Furthermore, the process of interacting and training with a power simulation environment (such as an interactive power environment built on open-source Grid2Op or simulation software) is as follows:

[0093] At time step The agent receives a low-dimensional embedding vector representing the state of the global power information system, output by the graph neural network encoder in S3. This vector, as a high-level abstraction of the environment state, is simultaneously fed into both core neural network components of the agent:

[0094] First, the actor network outputs the probability distribution of decision actions based on low-dimensional embedding vectors; this process is analogous to the execution strategy of a power dispatcher. The actor network uses parameters... Perform parameterization and output the policy function. In this embodiment, since self-healing actions include different types, the actor network can be designed as a multi-head output structure. One branch outputs a discrete probability distribution to select topology reconfiguration operations (i.e., switching the on / off state of a specified circuit breaker or disconnector); the other branch outputs a continuous probability distribution (such as the mean and variance of a Gaussian distribution) to select resource scheduling operations (i.e., adjusting the specific values ​​of controllable generator output or load shedding). Furthermore, to ensure that the abstract mathematical distribution output by the actor network can be effectively transformed into the underlying physical execution instructions of the power information system, this embodiment also sets up a clear action mapping mechanism. Specifically, for topology reconfiguration operations, the Argmax function (inference phase) or a probability sampling mechanism (training phase) is used to transform the discrete probability distribution into a specific One-Hot code, thereby mapping it to a hard control instruction for a specified circuit breaker to close (1) or open (0). For resource scheduling operations, the actor network outputs the mean and standard deviation of a Gaussian distribution, samples specific continuous values ​​from it using reparameterization techniques, and scales and truncates these values ​​using an activation function (e.g., Tanh) within the allowable upper and lower limits of the physical capacity of each generator set or load node, ensuring that the issued active / reactive power adjustments absolutely meet the physical safety margin requirements of the equipment. Secondly, the commentator network, based on... Assessing the value of the current state is a process akin to a power dispatcher's global intuition, used to predict the total future benefits that the current grid situation can yield. Finally, the commentator network uses parameters... Perform parameterization and output the state value function. .

[0095] Furthermore, to prevent the agent from becoming completely bewildered and causing training failure when faced with extremely complex cascading faults in the initial stage, this invention introduces a course learning strategy in the environmental interaction phase. This strategy dynamically provides the agent with fault scenarios for interaction in a preset order of increasing difficulty.

[0096] Specifically, the training process is divided into multiple stages:

[0097] In the first stage (simple scenario), only a single line tripping fault is injected into the environment, and the agent only needs to learn how to close the tie switch to meet the basic network topology connectivity restoration.

[0098] In the second stage (medium scenario), a complex fault involving power flow exceeding the limit is injected. The agent needs to learn to coordinate switching actions and generator output, not only to restore power supply, but also to meet the safety specifications in S2 (such as voltage not exceeding the limit).

[0099] In the third stage (complex scenario), an extreme fault that causes a large-scale power outage is injected. Under the pressure of the time dimension, the agent must output a long sequence of combined actions that satisfy connectivity and security, and strictly conform to the stability and recovery restrictions.

[0100] Furthermore, in order to ensure the automatic advancement and repeatability of the course learning strategy at the algorithm level, this embodiment sets quantitative stage leap trigger conditions.

[0101] Specifically, during the training process, the system continuously calculates the agent's most recent... One interactive round (e.g., setting) The overall performance within ) . If and only if the agent in this The algorithm will automatically unlock and enter the next difficulty training phase only when the average success rate of successfully completing the self-healing task within a round and the trajectory always strictly meets the safety, stability and recovery logic reduction exceeds a preset threshold (e.g., 90%).

[0102] It should be noted that the above partitioning process can reduce the trial and error cost in high-dimensional action space, thereby preventing the model from blindly exploring in the early stages and improving optimization efficiency.

[0103] Furthermore, after the agent completes a batch of interactions with the simulated environment, interaction trajectory data is collected. .in, This is used to generate a dense reshaping feedback reward in S4, which contains quantitative robustness of the signal-sequential logic. Subsequently, the dominance function is calculated. Used to measure the execution of the current action Compared to the current state How much better is the average expected performance? Its mathematical expression is:

[0104]

[0105] Furthermore, gradient updates are performed on the commentator network and the actor network respectively.

[0106] Specifically, for commenter networks, the goal is to fit the true cumulative return as accurately as possible; therefore, mean squared error can be used to construct the loss function. :

[0107]

[0108] in, For the sample size, The target network used for stable training.

[0109] Specifically, for actor networks, in order to encourage agents to output actions that yield higher advantage functions while ensuring smooth policy updates, a policy gradient loss function with entropy regularization is adopted. :

[0110]

[0111] in, Shannon entropy is used to encourage agents to maintain diversity of exploration in the early stages of training, preventing premature convergence to local unsolvable solutions that lead to suboptimal solutions (e.g., only cutting load without attempting network reconstruction). is the entropy regularization coefficient.

[0112] Furthermore, the loss function is continuously minimized using a backpropagation algorithm (e.g., the Adam optimizer), and iteratively updated. and The network parameters are iteratively updated by combining the prediction model loss function propagated by the graph neural network encoder with multi-task joint backpropagation until the network converges, thus completing the offline training process.

[0113] Furthermore, after completing the offline training described above, the process moves into the online inference and actual deployment phase.

[0114] Specifically, the convergent actor network and graph neural network encoder are solidified and deployed to the actual power information system control center or edge gateway. When the system detects a fault in the actual physical power grid, the model constructs a graph-structured state based on real-time sensor data and performs forward inference, directly outputting a series of optimal decision sequences with a millisecond-level time delay. Each action in this sequence (whether it is topology reconfiguration to switch a specified circuit breaker or resource scheduling to adjust controllable generator units) has already internalized complex physical timing specifications during the training phase, thus enabling the power information system to achieve rapid and secure self-healing without the need for time-consuming large-scale online simulations.

[0115] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A self-healing decision optimization method for power information systems based on reinforcement learning feedback, characterized in that, include: Acquire real-time operational data of the power information system and construct it into a graph-structured state representation that includes node features, edge features, and topological relationships; Based on signal timing logic, the pre-defined self-healing objectives of the power information system in terms of security, stability and resilience are formalized into a set of logic specifications containing timing constraints. Based on the graph-structured state representation, the degree to which the future state trajectory of the power information system satisfies the logical specification is calculated or predicted, and quantified into one or more quantitative robustness values. Based on the change in the quantitative robustness value before and after the state of the power information system changes due to decision-making, a dense reward signal is generated, and the feedback reward of the reinforcement learning agent is reshaped using the signal. The reinforcement learning agent is trained through continuous interaction with the simulated environment based on the reshaped feedback reward, and learns and outputs the optimal decision sequence for fault self-healing in power information systems.

2. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 1, characterized in that, Constructing the graph-structured state representation includes: In the power information system, the busbars are defined as nodes of the graph, and the transmission lines and transformers are defined as edges of the graph. Bus voltage, phase angle, load information, and generator output are used as the node characteristics; The branch power flow, switch on / off state, and impedance parameters are used as the side features.

3. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 1, characterized in that, The logical specification includes at least one of the following types: A security protocol used to constrain the operating parameters of a power information system from exceeding a safety threshold within a preset time window; Stability specifications are used to define the recovery of key indicators to a stable range within a specified time after fault clearance; A restorative specification used to characterize the restoration of a lost power load within a specified time.

4. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 1, characterized in that, The degree to which the calculation or prediction of the future state trajectory of the power information system satisfies the logical specification, and quantifies it into one or more quantitative robustness values, including: Using a predictive model, based on the current graph structured state representation, the difference between the future state trajectory and the target trajectory defined by the logical reduction under decision-driven conditions is estimated, and the quantification result of the difference is used as the quantitative robustness value.

5. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 4, characterized in that, Generate a dense reward signal, including: The quantitative robustness value output by the prediction model in the current state is defined as the first potential function value, and the quantitative robustness value output by the prediction model in the next state after the decision is made is defined as the second potential function value. The difference between the second potential function value (after being attenuated by a discount factor) and the first potential function value is used as a term of the basic reward signal to form the dense reward signal.

6. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 1, characterized in that, Before the reinforcement learning agent is trained, a graph neural network encoder is used to learn features from the graph structured state representation to generate a low-dimensional embedding vector representing the state of the global power information system.

7. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 6, characterized in that, The reinforcement learning agent adopts an actor-critic framework, whereby the actor outputs the probability distribution of decision actions based on the low-dimensional embedding vector, and the critic evaluates the value of the current state based on the low-dimensional embedding vector.

8. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 4 or 7, characterized in that, The prediction model is used as a parallel output module of the actor-critic framework, and the training objective is to minimize the error between its predicted quantitative robustness value and the true robustness value calculated ex post in the actual interaction trajectory.

9. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 1, characterized in that, The training employs a course-based learning strategy, which provides fault scenarios to the agent in a preset order of increasing difficulty, gradually transitioning from simple scenarios that only require basic topology recovery to complex scenarios that simultaneously require compliance with the security, stability, and recoverability specifications.

10. The self-healing decision optimization method for power information systems based on reinforcement learning feedback as described in claim 1, characterized in that, The optimal decision sequence includes at least one of the following operations: Topology reconfiguration operation that switches the on / off state of a specified circuit breaker or disconnector; Resource scheduling operations that adjust the output of controllable generator sets or the load that can be cut off.

Images