A power distribution network dynamic topology identification and anomaly detection method based on reinforcement learning

Abstract

This invention discloses a method for dynamic topology identification and anomaly detection in distribution networks based on reinforcement learning. The method employs a spatiotemporal decoupling strategy, utilizes a graph neural network to electrically partition the power grid, and activates decisions in an event-driven manner. A hierarchical reinforcement learning framework is adopted, where a meta-controller performs global analysis and task allocation, and a sub-controller generates candidate actions containing topology hypotheses and anomaly declarations within the target sub-region. Power flow calculation is performed based on the network state defined by the candidate actions, and a reward signal is generated based on the calculation results to update the strategies of the meta-controller and sub-controllers. This invention solves the curse of dimensionality problem in large-scale networks through a spatiotemporal decoupling strategy and achieves strong coupling between topology identification and data anomaly diagnosis through hierarchical collaborative decision-making, improving robustness in complex scenarios and balancing model learning efficiency and decision security.

Description

Technical Field

[0001] This invention relates to the field of power system automation technology under reinforcement learning, and in particular to a method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning. Background Technology

[0002] With the high proportion of distributed power sources and electric vehicle charging facilities connected to the distribution network, the dynamics, complexity, and uncertainty of its operating conditions are increasing. To ensure the safe and stable operation of the power grid and optimize resource scheduling, rapid and accurate identification of network topology caused by frequent switching operations has become a key technical aspect. Currently, distribution network topology identification technology is developing along two main paths: one is model-driven methods based on traditional electrical theory, such as using residual analysis of power flow calculations or state estimation to infer switch states; the other is data-driven methods represented by artificial intelligence, using machine learning models, such as support vector machines, decision trees, or deep neural networks, to learn the mapping relationship between topology and data features from massive amounts of measurement data. In recent years, reinforcement learning, due to its advantages in sequential decision-making problems, has also been introduced into the field of topology identification. Intelligent agents learn the optimal identification strategy through trial and error with the power grid environment.

[0003] However, existing technologies still have significant shortcomings in addressing the challenges of modern distribution networks. First, model-driven methods are highly sensitive to the quality of measurement data; noise, missing data, or malicious attacks can easily lead to misidentification, and the solution process of their physical models struggles to effectively distinguish between topology errors and data anomalies—two concurrent faults. Second, traditional data-driven methods, including some rudimentary reinforcement learning applications, often face the "curse of dimensionality" when dealing with large-scale distribution networks. This means that the increase in the number of switches leads to an exponential growth in the decision space, making model training difficult, convergence slow, and identification inefficient. More critically, existing technologies typically treat topology identification and data anomaly detection as two independent tasks, ignoring their inherent strong coupling. Incorrect topological assumptions can lead to data being misclassified as anomaly, and vice versa. This separated processing logic not only reduces the overall robustness of the system but also creates redundancy in computational resources and decision delays, making it difficult to meet the autonomous and real-time operational requirements of future distribution networks. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a reinforcement learning-based method for dynamic topology identification and anomaly detection in power distribution networks to address the problems mentioned in the background section.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a method for dynamic topology identification and anomaly detection in distribution networks based on reinforcement learning, comprising:

[0007] The power distribution network is spatially decoupled and divided into multiple sub-regions. Real-time measurement data of the power distribution network is temporally decoupled and monitored. When a preset event triggering condition is met, a hierarchical reinforcement learning decision-making process is activated.

[0008] The meta-controller in the hierarchical reinforcement learning decision-making process selects a target sub-region from the multiple sub-regions based on the global state information of the distribution network and assigns a task target to it.

[0009] Through the sub-controller corresponding to the target sub-region in the hierarchical reinforcement learning decision-making process, a candidate action is generated based on the local measurement data within the target sub-region and the task objective;

[0010] Power flow calculation is performed based on the network state defined by the candidate actions, and a reward signal is generated based on the result of the power flow calculation. The decision-making strategies of the meta-controller and the sub-controller are then updated using the reward signal.

[0011] As a preferred embodiment of the reinforcement learning-based power distribution network dynamic topology identification and anomaly detection method described in this invention, the spatial decoupling includes:

[0012] The power distribution network is represented as a graph structure, where nodes represent electrical components and edges represent connections.

[0013] Using a graph neural network model, an embedding vector encoding its own features and neighborhood topology information is generated for each node in the graph structure.

[0014] A clustering algorithm is applied to the set of embedding vectors generated for all nodes to divide the nodes into different sub-regions.

[0015] As a preferred embodiment of the reinforcement learning-based power distribution network dynamic topology identification and anomaly detection method of the present invention, the preset event triggering condition includes at least one of the following:

[0016] The rate of change of the node's measurement value exceeds the first threshold;

[0017] Alternatively, the reconstruction error generated by reconstructing the real-time measurement data by a pre-trained generative model exceeds the second threshold.

[0018] As a preferred embodiment of the reinforcement learning-based power distribution network dynamic topology identification and anomaly detection method described in this invention, the meta-controller is a sequence containing historical state information, and the meta-controller outputs the decision to select the target sub-region and assign the task target through a network structure containing a recurrent neural network layer and an attention mechanism layer.

[0019] As a preferred embodiment of the reinforcement learning-based power distribution network dynamic topology identification and anomaly detection method of the present invention, the task objectives are selected from a preset set, which includes topology identification tasks and data anomaly diagnosis tasks.

[0020] As a preferred embodiment of the reinforcement learning-based power distribution network dynamic topology identification and anomaly detection method described in this invention, the sub-controller is based on a graph attention network model to process the graph structured data of the target sub-region and generate the candidate actions; the candidate actions are expressed as a probability distribution of the closed state of all unknown switches in the target sub-region.

[0021] As a preferred embodiment of the reinforcement learning-based distribution network dynamic topology identification and anomaly detection method of the present invention, the candidate actions generated by the sub-controller further include an anomaly declaration action; and the reward signal is configured according to whether the sub-controller executes the anomaly declaration action and whether the declaration action is consistent with the actual data state, for rewarding correct anomaly declaration actions and penalizing incorrect anomaly declaration actions.

[0022] As a preferred embodiment of the reinforcement learning-based distribution network dynamic topology identification and anomaly detection method of the present invention, wherein: a reward signal is generated based on the result of the power flow calculation, including:

[0023] If the power flow calculation converges, calculate a consistency metric between the theoretical measurement value obtained from the power flow calculation and the real-time measurement data.

[0024] The reward signal is generated based on the consistency metric, wherein the higher the consistency metric, the higher the value of the reward signal.

[0025] As a preferred embodiment of the reinforcement learning-based distribution network dynamic topology identification and anomaly detection method of the present invention, wherein: a reward signal is generated based on the result of the power flow calculation, and the method further includes:

[0026] If the power flow calculation fails to converge, a preset negative reward signal is generated.

[0027] As a preferred embodiment of the reinforcement learning-based distribution network dynamic topology identification and anomaly detection method of the present invention, the training process of the meta-controller and sub-controller includes:

[0028] An offline pre-training phase is provided, in which the sub-controller is pre-trained using an offline reinforcement learning algorithm on a static dataset containing historical running data and simulation data.

[0029] In addition, there is an online update phase in which the reward signal is used to jointly update the strategy of the meta-controller and the pre-trained sub-controller through real-time interaction with the distribution network.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] 1. By decoupling the electrical characteristic space based on graph neural networks, the complex problem of topology identification across the entire network is decomposed into local decision-making problems within finite sub-regions; combined with an event-driven mechanism, the decision-making process is activated only when necessary. This significantly reduces the decision space and computational load of the reinforcement learning agent, and substantially improves the efficiency and scalability of topology identification and anomaly detection in large-scale distribution networks.

[0032] 2. A hierarchical reinforcement learning framework was constructed, in which the meta-controller is responsible for macro-level task allocation (topology identification or data anomaly diagnosis), and the sub-controllers are responsible for executing specific actions locally (including topology guessing and anomaly declaration). This coupling of two isolated tasks enables the agent to learn to weigh and jointly judge between them, effectively avoiding misjudgments of another state (data quality) due to a single fault (topology error), thereby enhancing the overall robustness and accuracy of the method under complex fault scenarios.

[0033] 3. Furthermore, this invention utilizes offline reinforcement learning algorithms to safely and efficiently pre-train sub-controllers from static data, enabling them to possess basic recognition capabilities. During online interaction with the real power grid, policy fine-tuning and meta-controller training are performed through online algorithms. This significantly shortens the exploration time and convergence cycle of online learning, while relying on a reward mechanism to ensure the safety of the online exploration process and avoid physical impacts on the power grid. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0035] Figure 1 This is a flowchart illustrating the overall process of a reinforcement learning-based dynamic topology identification and anomaly detection method for power distribution networks according to an embodiment of the present invention.

[0036] Figure 2 This is a schematic diagram of the hierarchical reinforcement learning decision framework structure of the power distribution network dynamic topology identification and anomaly detection method based on reinforcement learning according to an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of the two-stage training process of the reinforcement learning-based distribution network dynamic topology identification and anomaly detection method according to an embodiment of the present invention.

[0038] Figure 4 This is a comparison chart of the convergence and topology identification accuracy of hierarchical reinforcement learning and standard reinforcement learning in a reinforcement learning-based distribution network dynamic topology identification and anomaly detection method according to an embodiment of the present invention. Detailed Implementation

[0039] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0040] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0041] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0042] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0043] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0044] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0045] Example 1

[0046] Reference Figure 1 This is the first embodiment of the present invention, which provides a method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning, including:

[0047] S1. Spatially decouple the distribution network and divide it into multiple sub-regions. Perform time-decoupled monitoring on the real-time measurement data of the distribution network. When the preset event triggering conditions are met, activate a hierarchical reinforcement learning decision-making process.

[0048] It should be noted that, in order to overcome the limitations of traditional geographical partitioning or simple connectivity partitioning and to achieve an intelligent sub-region division based on electrical characteristics with "high cohesion and low coupling", this invention uses a graph neural network-based approach to decouple the electrical sensing space.

[0049] Specifically, the entire power distribution network is abstracted into a graph structure. Among them, the node set This represents electrical components in the power grid, such as substation buses, load nodes, distributed generation (DER) connection points, switches, etc. Edge set This represents the physical connections between components, such as power transmission lines, cables, and transformers. Then, for each node... Construct an initial feature vector This initial feature vector encodes the static and quasi-dynamic attributes of the nodes, including, for example, the normalized node type (e.g., represented using one-hot encoding), rated voltage level, historical active / reactive power statistics (mean, variance), and node degree (connectivity). For each edge... Construct an initial feature vector The initial feature vector encodes the physical properties of the connection, such as line type, per-unit resistance, reactance, and rated capacity.

[0050] Furthermore, to capture deep topological and feature information of each node within its electrical environment, this invention employs a graph neural network model, such as a graph sampling and aggregation network, to learn the graph structure. This graph neural network model updates the representation of each node by iteratively aggregating neighborhood information. Specifically, in the... The update rules for a layer can be described as follows:

[0051] First, for each node From its first-order neighbor set A portion of the nodes are sampled and placed in the first... eigenvectors of the layer A neighborhood feature vector is obtained by aggregating the features using an aggregation function (such as mean, max pooling, or LSTM). Then, the nodes Its own old feature vector Its aggregated neighborhood feature vector The data is spliced ​​together using a non-linear activation function. Transform the fully connected layer (such as ReLU) to obtain the nodes. In the New embedding vector of the layer The process can be represented as follows:

[0052]

[0053] in, For nodes In the Layer embedding vectors, This is the learnable weight matrix for this layer. After multiple iterations, the final embedding vector for each node is... This allows for the effective encoding of structural information and electrical characteristics within its higher-order neighborhood.

[0054] Furthermore, the final embedding vectors of all nodes are used to construct a feature matrix. ,in, Let be the dimension of the embedding vector. Then, a spectral clustering algorithm is applied to this embedding space to partition the nodes into... Sub-regions. It should be explained that spectral clustering can effectively identify non-convex cluster structures, making it very suitable for handling high-dimensional complex distributions generated by GNNs. The specific process is as follows:

[0055] S101. Construct a similarity matrix between nodes based on embedding vectors. The elements of the matrix It can be calculated using the Gaussian kernel function:

[0056]

[0057] in, It is a node and The squared Euclidean distance of the embedding vectors. This is the kernel width parameter, used to control the impact of distance on similarity. Its value can be determined through statistical analysis of the distances between samples in the dataset, for example, by setting it to the median of the distances between all sample pairs.

[0058] S102, Based on the similarity matrix Calculate the Laplace matrix .

[0059] S103, Calculation The former The eigenvectors corresponding to the smallest eigenvalues ​​are then stacked column-wise into a new matrix.

[0060] S104. Treat each row of the new matrix as a new data point and apply the K-Means algorithm to it to obtain... A cluster of nodes, i.e. An electrically highly cohesive subregion.

[0061] It should be noted that this division result can group areas with the same feeder or close electrical connection together, while weak connection boundaries such as tie switches naturally become the division boundaries of sub-regions.

[0062] Furthermore, to reduce unnecessary computation, the identification process is only initiated when significant changes occur in the system, thus achieving efficient utilization of computing resources. We continuously monitor the real-time measurement data stream of the power grid by deploying a pre-trained generative model based on a variational autoencoder (VAE).

[0063] Specifically, this variational autoencoder consists of an encoder and a decoder Composition. The encoder converts high-dimensional real-time measurement data vectors. The input (including voltage magnitudes of all key buses, line start-up currents, etc.) is mapped to a low-dimensional latent spatial distribution, from which the decoder samples and attempts to reconstruct the original input. It is important to note that this VAE is trained only on datasets labeled as "normal operating conditions" in the historical database. Furthermore, the training objective is to minimize the reconstruction error and the KL divergence between the latent distribution and the standard normal distribution, in order to learn the inherent distributional patterns of the power grid's normal operating modes. When the model is running online, for each new real-time measurement data vector, its reconstruction error is calculated as an anomaly score. :

[0064]

[0065] It should be noted that the physical meaning of this generative model is that when the power grid undergoes topological changes or data anomalies, real-time measurement data will deviate from the "normal" data manifold learned by the model. Therefore, the model cannot effectively reconstruct this data, resulting in reconstruction errors. Significantly increased.

[0066] Furthermore, an event is considered triggered and the subsequent hierarchical reinforcement learning decision-making process is activated when any of the following conditions are met:

[0067] First condition (mutation detection): The short-term rate of change of the measured value of any critical node exceeds a preset engineering threshold (first threshold), for example, the rate of change of voltage exceeds 5% of the rated voltage. This condition can quickly detect violent disturbances caused by switching actions, etc.

[0068] Second condition (mode drift detection): Real-time reconstruction error calculated by the generative model. Exceeding a threshold (the second threshold) set through statistical analysis (e.g., the 99th percentile of the reconstruction error distribution based on a normal dataset). This condition is more sensitive to small, coordinated changes that deviate from the normal pattern, even if they are not drastic, such as slow topology switching or covert data attacks.

[0069] The third condition (periodic inspection): To prevent any slow changes that may be missed, set a periodic trigger, for example, to force the decision-making process to be activated once every 30 minutes.

[0070] It should be noted that, through the above decoupling process, the present invention can transform the original problem into: when a specific event occurs, within a limited sub-region that is intelligently identified, a target-specific identification or diagnostic task is performed, thereby reducing the complexity of the problem.

[0071] S2. Through the meta-controller in the hierarchical reinforcement learning decision-making process, based on the global state information of the distribution network, a target sub-region is selected from multiple sub-regions and a task objective is assigned to it.

[0072] It should be noted that when the event triggering mechanism of step S1 is activated, the meta controller will be responsible for analyzing the status of the entire distribution network from a macro perspective, deciding which of the most suspicious sub-regions to prioritize the allocation of limited computing and identification resources, and clarifying the specific tasks that need to be performed in that region.

[0073] Furthermore, in order to make globally optimal decisions, the meta-controller needs a global state representation that comprehensively reflects the current and historical dynamics of the power grid. This invention constructs a temporal state sequence containing multi-dimensional information. As the input to the meta-controller, where, Is The instantaneous global state vector at time t.

[0074] Specifically, in this embodiment, the instantaneous global state vector It includes global electrical overview information, event triggering information, and sub-region summary information. The global electrical overview information refers to the macroscopic operating indicators of the entire distribution network, such as total active / reactive load, average voltage deviation, and network loss rate. The event triggering information refers to the quantized values ​​of the event triggering signals from step S1, particularly the reconstruction error of the generative model. This is used to directly reflect the severity of the current system state deviating from the "normal" mode. Sub-region summary information refers to the information allocated in step S1. Each sub-region is a sub-region Generate a summary vector It is important to emphasize that this summary vector can be obtained by taking the final embedding vector of all nodes within that sub-region. The result is obtained by average pooling, i.e. ,in, It is a sub-region The number of nodes within the region. This summary vector allows the electrical characteristics and topology of each sub-region to be compressed into a fixed-dimensional vector for processing by the meta-controller.

[0075] It should be noted that by piecing together the above three parts of information, one can construct... Instantaneous global state vector at time t Furthermore, the present invention employs a length of... state sequence As input, rather than using the current time. The goal is to enable the meta-controller to capture the dynamic evolution of events, meaning that a gradually increasing reconstruction error and a sudden, instantaneous increase in reconstruction error may point to different types of problem root causes. Therefore, by analyzing timing information, the meta-controller can make forward-looking and accurate judgments.

[0076] Furthermore, in order to effectively process the above-mentioned input state sequence and output a combined action, the meta-controller designed in this invention adopts a deep network structure that includes a recurrent neural network (RNN) layer and an attention mechanism layer.

[0077] Specifically, in the recurrent neural network layer, the state sequence is... The input is fed into an RNN layer, which can be a Long Short-Term Memory (LSTM) network or a Gated Recurrent Unit (GRU). In this embodiment, a GRU is used as an example. It should be explained that the GRU primarily learns long-term dependencies in the state sequence by updating and resetting gates. After processing the entire state sequence, the GRU layer will exit at the last time step. Output a final hidden state vector. The hidden state vector can be viewed as a condensed representation of the entire historical state sequence, encoding key temporal information about how the event evolved to the current state.

[0078] Specifically, in the attention mechanism layer, after obtaining the hidden state vector representing the global temporal context... Next, the meta-controller needs to decide which sub-region to focus on. An attention mechanism is used here to calculate the correlation between each sub-region and the current global context; the calculation process is as follows:

[0079] S201, The hidden vector output by the gated loop unit. As a query, Summary vector of each subregion As keys and values.

[0080] S202. Calculate the attention score between the query and each key. A common calculation method is dot product attention, which can be expressed as:

[0081]

[0082] in, It is a sub-region Attention score and It is a learnable weight matrix used to project queries and keys into the same space. It is the dimension of the key vector, used as a scaling factor to prevent the gradient from becoming too small.

[0083] S203. Normalize all scores using the Softmax function to obtain the attention weights for each sub-region. :

[0084]

[0085] It should be noted that attention weights are used here. The physical meaning is that it represents the current global state. Below, the meta-controller considers the sub-region It refers to the probability or importance of the location of the root cause of the event. The higher the value, the more "suspicious" the sub-region is.

[0086] Furthermore, the motion of the meta-controller is defined as a binary tuple. , representing the selected target sub-region and the assigned task objective, respectively.

[0087] Furthermore, the selection of this target sub-region can be directly based on the attention weight distribution. During decision-making, the meta-controller selects the sub-region with the highest attention weight. ,Right now:

[0088]

[0089] It should be noted that this approach makes the decision-making process highly interpretable, because we can understand why the meta-controller makes this choice by visualizing the attention weights.

[0090] Furthermore, after identifying the target sub-region Afterwards, the meta-controller also needs to decide what task objective to assign. This task objective... Select from a preset set, which includes topology identification tasks and data anomaly diagnosis tasks.

[0091] Specifically, in order to make this decision, the global context vector is... and the selected target sub-region The summary vectors are concatenated and then fed into a small multilayer perceptron (MLP) classifier. The output layer of this classifier uses the Softmax activation function to output a probability distribution over the task set. If This indicates that a drastic change has occurred in the system (such as the first condition in step S1 being triggered), and the sub-region If the voltage drops suddenly within the area, the MLP classifier will output a "topology identification task" with high probability to check if any switches have tripped in that area. Conversely, if... If the system state deviates from normal over a long period of time (e.g., the second condition in step S1 is continuously triggered), a "data anomaly diagnosis task" will be output with high probability to check whether the measurement equipment in that area is faulty or under attack. Based on this, the meta-controller will select the task with the highest probability as the target of this task. .

[0092] It should be noted that, through the above design, the meta controller can locate a fuzzy, network-wide "abnormal event" to a specific "specific problem in a certain sub-region", thereby greatly reducing the search space and providing guidance for the subsequent operation of the sub-controller in a local area.

[0093] S3. Through the sub-controller corresponding to the target sub-region in the hierarchical reinforcement learning decision-making process, a candidate action is generated based on the local measurement data and task objective within the target sub-region.

[0094] It should be noted that upon receiving the target sub-region specified by the meta controller... and mission objectives Then, the corresponding sub-controller will be activated. The sub-controller's task is to focus on the current region (local region) and generate a specific, executable candidate action. This candidate action is designed to resolve tasks assigned by the meta controller.

[0095] Furthermore, since the sub-controller's decision-making is based on its local state... Therefore, this state must be represented in a format suitable for graph neural network processing to characterize the detailed information of the target sub-region.

[0096] Specifically, the network diagram structure constructed from step S1 In the process, the target sub-region selected by the meta-controller is extracted. Corresponding subgraph .in, and These are the sets of nodes and edges within the sub-region, respectively.

[0097] Furthermore, to enable the sub-controller to perceive its real-time operating status, we can construct an information-rich node feature vector for it. For the subgraph... Each node in Its input feature vector It is composed of the following three parts:

[0098] Static / quasi-static characteristics: Inherited from the initial node characteristics of step S1, including node type, rated voltage, historical load statistics, etc.

[0099] Real-time measurement data: This includes the real-time, normalized voltage amplitude, phase angle (if applicable), and active / reactive power injection for this node. It should be noted that if some measurements are missing, preset values ​​or simple estimates will be used to fill the gaps, accompanied by a "missing" flag.

[0100] Task target encoding: The task target issued by the meta controller. (For example, the aforementioned "topology identification task" or "data anomaly diagnosis task") performs one-hot encoding to obtain a task vector. Copy this vector and append it to the feature vector of each node within this sub-region.

[0101] It should be noted that assigning the task objective vector to each node has the physical meaning of ensuring that the entire network's computational process "knows" what the current core task is when the sub-controller performs subsequent graph information propagation and aggregation. This enables the model to learn differentiated feature extraction and decision-making patterns for different tasks. For example, in the "topology identification" task, the model will pay more attention to the relationship between voltage, current, and connectivity, while in the "data anomaly diagnosis" task, it will pay more attention to the statistical consistency of a certain measurement value with other relevant measurements.

[0102] Furthermore, to handle the aforementioned subgraph data with complex topology and rich node features, this invention employs a Graph Attention Network (GAT) as the sub-controller model. It should be explained that GAT adaptively learns the importance of neighboring nodes to the central node, making it highly suitable for capturing critical electrical connections in heterogeneous graphs such as power grids. In this embodiment, the graph attention network employs a graph attention layer and a multi-head attention structure.

[0103] Furthermore, in the graph attention layer, GAT updates the node representation by stacking multiple graph attention layers. In each layer, for a node... The update process of its feature vector is as follows:

[0104] S301. Perform a linear transformation on the feature vectors of all nodes, resulting in a learnable weight matrix. accomplish: .

[0105] S302, Computation Node With all its neighboring nodes Attention coefficient between :

[0106]

[0107] in, This represents vector concatenation. It is a weight vector of a learnable single-layer feedforward neural network. It is a modified linear unit activation function with leakage. Measured the nodes Information for nodes The importance of.

[0108] S303. To facilitate comparisons between different nodes, the Softmax function is used to normalize the attention coefficients of a node and all its neighbors, resulting in the final attention weights. :

[0109]

[0110] S304. The features of neighboring nodes are aggregated according to their attention weights to form nodes. New feature vector :

[0111]

[0112] It should be noted that in power systems, the state (such as voltage) of a node is often affected to varying degrees by its neighbors. For example, the influence from main power supply lines is usually greater than that from end-point branches with small loads. The attention weights of GAT (Graphical Attention Array) can automatically learn this physical "electrical distance" and influence weights through training, rather than treating all neighbors equally like traditional Graph Convolutional Networks (GCNs), thus achieving more refined feature extraction.

[0113] Furthermore, to make the learning process more stable and capture richer information, this invention employs a multi-head attention mechanism, i.e., independent execution. The above attention process (each called a "head") is then used to obtain... The new feature vectors are concatenated or their mean is calculated to serve as the final output of this layer.

[0114] It should be noted that after multi-layer GAT processing, the final output embedding vector of each node in the subgraph not only includes its own real-time measurement and task information, but also deeply integrates its neighborhood context information in the sub-region topology, which is weighted according to electrical correlation.

[0115] Furthermore, due to the candidate actions of the sub-controller It is a composite action, so it must be able to handle both "topology identification" and "data anomaly diagnosis" tasks at the same time.

[0116] Specifically, a topology identification action is generated. The goal of this action is to identify all switches within a sub-region whose state is unknown (or suspicious). A state judgment is given. First, for each switch to be identified... (It connects the nodes) and nodes ), and the final embedding vectors of its two endpoints. and By splicing the components together, the characteristic representation of the switch is formed: Then, the switch feature vector The input is a standalone, small multilayer perceptron (MLP), referred to as the "topological action head". In this embodiment, the output layer of the MLP has a dimension of 2, and a softmax activation function is applied to output a probability distribution:

[0117]

[0118] It should be noted that this probability distribution represents the model's response to the switch. The confidence level for being in a closed or open state. Ultimately, the topology identification process is this. The set of probability distributions of a switch In actual implementation, sampling can be performed based on this distribution to obtain a specific candidate topology (e.g., selecting the state with the highest probability for each switch).

[0119] Specifically, an anomaly declaration action is generated. The goal of this action is to determine whether there are anomalies in the measurement data of the current sub-region that cannot be explained by any reasonable topology. First, a readout function is used to generate the final embedding vectors of all nodes in the subgraph. They are aggregated into a graph-level embedding vector representing the state of the entire subgraph. In this embodiment, the readout function is average pooling:

[0120]

[0121] Then, the graph-level embedding vector The input is fed into another independent MLP, called the "abnormal action head". In this embodiment, the output layer of this MLP has a dimension of 1 and applies a sigmoid activation function to output a single probability value. :

[0122]

[0123] It should be noted that this probability value This represents the probability that the sub-controller believes there is a data anomaly in the current sub-region. The anomaly declaration action is either "declare an anomaly" or "do not declare an anomaly," and the decision is based on... For example, if If (an adjustable decision threshold) is reached, then the "declare exception" action is executed.

[0124] Furthermore, after receiving a task, the sub-controller performs in-depth processing of local information through the GAT network and generates a composite candidate action in parallel using two decoupled "action heads". This action involves both a specific guess about the network topology and an overall assessment of the data quality.

[0125] S4. Perform power flow calculation based on the network state defined by the candidate actions, and generate a reward signal based on the result of the power flow calculation. Use the reward signal to update the decision-making strategies of the meta-controller and sub-controller.

[0126] It should be noted that this step aims to transform the composite candidate action generated by the sub-controller in step S3 into a physically verifiable grid state, and to evaluate the merits of the action through power flow calculation.

[0127] Furthermore, since the sub-controller in step S3 is for each switch to be identified The output action is a probability distribution To perform power flow calculations, a defined topology is required. Therefore, this invention employs a maximum probability selection strategy, choosing the state with the highest confidence for each switch to construct a candidate topology. That is, if Then the switch will be included in the candidate topology. Set to the closed state, and vice versa. Simultaneously, based on this candidate topology... By combining the static line parameters (resistance, reactance) and node parameters (real-time measured load power, generator output, etc.) of the power grid, a power grid model for power flow calculation is constructed.

[0128] Furthermore, a standard power flow calculation is performed on this candidate power grid model. It is important to emphasize that this power flow calculation can employ the robust and fast-converging Newton-Raphson method or its decoupled variant. It should be explained that the purpose of the power flow calculation is to solve for the voltage distribution and power flow of the entire network, given the candidate topology and node injected power. Based on this, the result of the power flow calculation will be one of the following two cases:

[0129] Scenario 1: Calculation convergence. This indicates that, under the assumed topology, the currently measured power injection can maintain a physically stable power grid operating state.

[0130] Scenario 2: Computation does not converge. This usually means that the assumed topology is incorrect, leading to a physically unsolvable ill-conditioned problem, such as the presence of islanded loads (load areas without power connections) or unreasonable power disconnection.

[0131] In addition, due to reward signals The design of the reward function is crucial for the success of reinforcement learning; it must accurately reflect the "goodness" or "badness" of actions. Therefore, this invention designs a composite reward function that includes topology recognition rewards and anomaly diagnosis rewards. This composite reward function aims to simultaneously guide the agent to learn topology recognition and anomaly diagnosis.

[0132] Specifically, the reward for topology identification is recorded as... For scenario one above, we need to further evaluate the degree of agreement between this "physically possible" topology and reality. This process can be achieved by calculating a consistency metric between theoretical measurements derived from power flow calculations and real-time measurement data, and the steps are as follows:

[0133] S401.1 Extract the theoretical values ​​corresponding to the actual measurement points from the converged power flow calculation results, such as the voltage amplitude of each bus and the current amplitude of the line, to form a theoretical measurement vector.

[0134] S401.2, the theoretical measurement vector ( ) and real-time measurement vectors obtained from the power grid ( The consistency measure is obtained by comparing the values ​​of the two variables. In this embodiment, the consistency measure is the normalized mean squared error (MSE).

[0135] S401.3. Generate reward values ​​based on this consistency metric. To ensure the reward signal is smooth and bounded, a negative exponential function can be used, resulting in:

[0136]

[0137] in, It is a positive hyperparameter used to adjust the sensitivity of rewards to error. The smaller the MSE, the higher the consistency metric. The closer the value is to 1 (higher reward); conversely, it tends to 0.

[0138] Specifically, regarding scenario two above, the candidate topology is directly determined to be seriously flawed, and a significant negative reward signal is given:

[0139]

[0140] in, It is a pre-defined negative constant (e.g., -10).

[0141] It should be noted that, through topology recognition rewards, this invention can provide clear physical boundaries for the agent's exploration, while also preventing the agent from learning invalid topology combinations that would lead to the physical collapse of the power grid.

[0142] Specifically, the reward for abnormal diagnosis will be recorded as... This reward is used to evaluate the accuracy of the "anomaly declaration action" in step S3. Since we know whether the data has been injected with anomalies in the training environment, a valid criterion for online operation is: if the sub-controller tries multiple topology hypotheses it considers most probable, but the MSE of the power flow calculation remains high (i.e., all reasonable explanations fail), then the current data state can be determined to be "actually anomaly" with high probability. Based on this criterion, we can configure the reward according to the confusion matrix as follows:

[0143] Correct Declaration: When the sub-controller performs the "Declare Exception" action and the data status is "Actual Exception", the reward is... (A large positive number, such as +10).

[0144] Error declaration: When the sub-controller performs an "exception declaration" action, but the data status is "actually normal," the reward is... (A large negative number, such as -5).

[0145] Missed report: When the sub-controller does not declare an exception, but the data status is "actually abnormal". Reward is... (The largest negative number, such as -15).

[0146] Correctly ignored: When the sub-controller does not declare an exception and the data status is "actually normal". Reward: (A small positive number or 0, such as +0.5).

[0147] It should be noted that the above configuration is based on the known actual data state labels in the training environment. In online application environments without real labels, the system can approximate the actual data state using a proxy mechanism. That is, when a sub-controller, within the current decision-making cycle, tries several (e.g., 5) candidate topologies with the highest confidence according to its strategy, but the consistency metric of the power flow calculation results corresponding to all attempts is lower than a preset, extremely low threshold (e.g., 0.1), the system determines the current state as actually abnormal. Conversely, if even one attempt causes the consistency metric to exceed the threshold, the current state is considered actually normal. Based on this proxy judgment result, the system can calculate the reward and update the strategy in the online environment.

[0148] In addition, based on the above configuration rewards, it is also necessary to set This setting is intended to make the model more "vigilant" during training, preferring to give false positives rather than miss genuine data anomalies, thus adhering to the principle of prioritizing power system safety.

[0149] Furthermore, the final reward for the sub-controller is the weighted sum of the two rewards mentioned above:

[0150]

[0151] in, and This is a weighting coefficient, with a value ranging from [0,1], used to balance the importance of the two subtasks. In scenarios where topology recognition is the primary objective, A larger value can be chosen, such as 0.7; in scenarios with poor data quality, A larger value can be taken, such as 0.6.

[0152] Furthermore, in this invention, after the meta-controller makes a macro-level decision, the reward it receives is directly equal to the reward received by the sub-controller it selected after executing the task. That is:

[0153]

[0154] It should be noted that by adopting this direct reward transmission mechanism, the meta-controller can learn which sub-region to select and what tasks to assign to it will bring the highest final reward, thus enabling it to perform optimal "task assignment".

[0155] Furthermore, the generated reward signal is used to update the neural network parameters of the meta-controller and sub-controllers through a reinforcement learning algorithm. This update process consists of two parts: offline pre-training and online joint updating.

[0156] Furthermore, to accelerate the convergence speed of online learning and improve sample efficiency, the sub-controller is first pre-trained offline. The offline pre-training process is as follows:

[0157] First, a large-scale static dataset is constructed. This dataset is derived from historical operational data and numerous case studies generated by power system simulation software (such as PSASP and PSCAD). Each data sample is a tuple containing the local state, the action performed, the reward obtained, and the subsequent state under a specific task. Then, an advanced offline reinforcement learning algorithm, such as conservative Q-learning, is employed. Regardless of the offline reinforcement learning algorithm used, the aforementioned dataset is required. This algorithm iteratively updates the weight parameters of the GAT-based sub-controller network defined in step S3, enabling it to possess a basic ability to identify common topologies and anomaly patterns before interacting with the real environment.

[0158] Furthermore, in a real or high-fidelity simulated power grid environment, the pre-trained sub-controller is fine-tuned, and the meta-controller is trained from scratch to achieve collaborative optimization between the two. In this process, the present invention employs an online Actor-Critic algorithm suitable for hierarchical and continuous hybrid action spaces, such as hierarchical near-end policy optimization. The process is as follows:

[0159] At every decision moment Meta controller output The corresponding sub-controller output Environmental (current flow calculation) return rewards and new status Complete empirical tuples and The data is stored in their respective experience replay pools. After the headphones are activated, a batch of data is sampled from the experience replay pool of the meta-controller, the advantage function is calculated, and the parameters of the meta-controller's policy network and value network are updated according to the objective function of the PPO algorithm (including policy ratio pruning and value function loss). Finally, the same sampling method is used to sample from the experience replay pool of the sub-controllers and update (fine-tune) the parameters of the activated sub-controllers using the PPO algorithm.

[0160] It should be noted that, through the above optimizations, this invention enables the entire decision-making system to adapt to new operating conditions and dynamic characteristics in real power grids that have not appeared in the offline dataset. Specifically, the meta-controller, through online trial and error, learns how to decompose problems and optimally delegate them to sub-controllers that already possess basic capabilities. The sub-controllers, guided by real-world feedback from power grid data, gradually improve their performance.

[0161] Example 2

[0162] Reference Figures 2 to 4 This is the second embodiment of the present invention. Based on embodiment 1, this embodiment further defines the specific structure and training framework of the meta-controller involved.

[0163] Furthermore, refer to Figure 2 The network structure of the meta-controller is as follows: its recurrent neural network layer uses a 2-layer gated recurrent unit (GRU) network with a hidden layer dimension of 128. The subsequent attention mechanism layer is used to generate the linear projection matrix for the query and key. and All dimensions are 128×64. The final MLP classifier used for decision-making tasks contains two hidden layers, each with 64 neurons, using the ReLU activation function. The output layer has a dimension of 2 (corresponding to the two tasks of topology identification and data anomaly diagnosis), using the Softmax activation function.

[0164] Specifically, the network structure of the sub-controller k corresponding to sub-region k is as follows: its core graph attention network uses three stacked graph attention layers and applies a four-head multi-head attention mechanism. The output feature dimension of each head is 32, and after concatenation, a 128-dimensional node embedding vector is obtained. The "topology action head" MLP used to generate topology recognition actions contains one hidden layer, 32 neurons, and ReLU activation. The "abnormal action head" MLP used to generate abnormal declaration actions also contains one hidden layer, 32 neurons, and ReLU activation.

[0165] Furthermore, refer to Figure 3 In the offline pre-training phase, the sub-controller is trained. The offline reinforcement learning algorithm used is Conservative Q-Learning (CQL). ​​Its core idea is to add a regularization term to the standard Q-learning objective function to penalize the Q-values ​​of state-action pairs not seen in the dataset, while minimizing the Q-values ​​of actions already present in the dataset. This updates the sub-controller network parameters. The goal is to minimize the following loss function :

[0166]

[0167] Where D is the offline dataset, It is the conservatism coefficient, a hyperparameter with a typical value range between [0.5, 5.0], used to balance the conservatism penalty and Bellman error. This indicates that the average value is calculated for all (state s, action a) data pairs sampled from the offline dataset D. This represents the average of all complete transition tuples (state s, action a, reward r, next state s') sampled from the offline dataset D. r represents the immediate reward obtained from the environment after performing action a in state s. It is a discount factor, with a value of [0,1], which determines the importance of future rewards relative to current rewards. The closer the value is to 1, the more forward-looking the intelligent agent is. This represents the Q-value (value) estimate of the data network for each (state s, action a) that actually occurred in the dataset. Indicates the target network. Indicates a possible action.

[0168] Furthermore, in the online joint update phase, joint policy updates are performed on the meta-controller and the pre-trained sub-controllers. The online Actor-Critic algorithm used is Proximal Policy Optimization (PPO). Both the meta-controller and the sub-controller have their own policy networks (Actors) and value networks (Critics). Taking the update of the meta-controller's policy network parameters as an example, its objective function is:

[0169]

[0170] in, It is the importance sampling ratio. These are the learnable parameters of the policy network π. It evaluates how well action a is relative to the average action in state s at time step t, and is usually calculated using generalized advantage estimation (GAE). This indicates that the average value is calculated for all time steps t collected in a batch. It is the importance sampling ratio, used to measure the difference between the new and old strategies. If the importance sampling ratio is greater than 1, it means that the new strategy is more inclined to take action a; if the importance sampling ratio is less than or equal to 1, the opposite is true. It is a clipping function. This is the pruning factor, typically set to 0.1 or 0.2. By maximizing this objective function, stable updates to the meta-controller's policy are achieved. The update process for the sub-controllers is similar.

[0171] For details, please refer to Figure 4 In this embodiment, we also simulated a complex power distribution network containing 10 sub-regions. The control group used a single controller to directly handle the global state; the experimental group (this scheme) used a meta-controller to select sub-regions, and the sub-controllers performed local identification tasks. Figure 4 As shown on the left, our proposed solution (solid black line) reached reward saturation around episode 30, while the traditional tiling algorithm (dashed gray line) did not reach complete plateau even after 100 episodes. Calculations show that the convergence speed of the decision task is improved by approximately 60%. Figure 4 The right side shows that the final accuracy of this scheme stabilizes at around 95%, while the traditional method only achieves around 85%. This is because the hierarchical architecture reduces the search space of the sub-controllers through the meta-controller, thereby improving the identification accuracy under complex topologies by approximately 11.7%. In summary, through simulation and specific controller structures, the effectiveness of spatial decoupling and hierarchical decision-making in the scheme of this invention can be verified.

[0172] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0173] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0177] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for dynamic topology identification and anomaly detection in distribution networks based on reinforcement learning, characterized in that, include: The power distribution network is spatially decoupled and divided into multiple sub-regions. Real-time measurement data of the power distribution network is temporally decoupled and monitored. When a preset event triggering condition is met, a hierarchical reinforcement learning decision-making process is activated. The meta-controller in the hierarchical reinforcement learning decision-making process selects a target sub-region from the multiple sub-regions based on the global state information of the distribution network and assigns a task target to it. Through the sub-controller corresponding to the target sub-region in the hierarchical reinforcement learning decision-making process, a candidate action is generated based on the local measurement data within the target sub-region and the task objective; Power flow calculation is performed based on the network state defined by the candidate actions, and a reward signal is generated based on the result of the power flow calculation. The decision-making strategies of the meta-controller and the sub-controller are then updated using the reward signal.

2. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1, characterized in that, The aforementioned spatial decoupling includes: The power distribution network is represented as a graph structure, where nodes represent electrical components and edges represent connections. Using a graph neural network model, an embedding vector encoding its own features and neighborhood topology information is generated for each node in the graph structure. A clustering algorithm is applied to the set of embedding vectors generated for all nodes to divide the nodes into different sub-regions.

3. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1, characterized in that, The preset event triggering conditions include at least one of the following: The rate of change of the node's measurement value exceeds the first threshold; Alternatively, the reconstruction error generated by reconstructing the real-time measurement data by a pre-trained generative model exceeds the second threshold.

4. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1, characterized in that, The meta-controller is a sequence containing historical state information. The meta-controller outputs decisions on selecting the target sub-region and assigning the task target through a network structure containing recurrent neural network layers and attention mechanism layers.

5. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1 or 4, characterized in that, The task objectives are selected from a preset set, which includes topology identification tasks and data anomaly diagnosis tasks.

6. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1, characterized in that, The sub-controller is based on a graph attention network model to process the graph structured data of the target sub-region and generate the candidate actions; the candidate actions are represented by a probability distribution of the closed state of all unknown switches in the target sub-region.

7. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 6, characterized in that, The candidate actions generated by the sub-controller also include an exception declaration action; and the reward signal is configured based on whether the sub-controller executes the exception declaration action and whether the declaration action matches the actual data state, and is used to reward correct exception declaration actions and punish incorrect exception declaration actions.

8. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1, characterized in that, A reward signal is generated based on the result of the power flow calculation, including: If the power flow calculation converges, calculate the consistency metric between the theoretical measurement value obtained from the power flow calculation and the real-time measurement data. The reward signal is generated based on the consistency metric, wherein the higher the consistency metric, the higher the value of the reward signal.

9. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 8, characterized in that, Generating a reward signal based on the power flow calculation results also includes: If the power flow calculation fails to converge, a preset negative reward signal is generated.

10. The method for dynamic topology identification and anomaly detection of distribution networks based on reinforcement learning as described in claim 1, characterized in that, The training process for the meta-controller and sub-controllers includes: An offline pre-training phase is provided, in which the sub-controller is pre-trained using an offline reinforcement learning algorithm on a static dataset containing historical running data and simulation data. In addition, there is an online update phase in which the reward signal is used to jointly update the strategy of the meta-controller and the pre-trained sub-controller through real-time interaction with the distribution network.

Images