A Power Data Quality Control Method Based on Adaptive Routing and Graph Snapshots

Abstract

This invention relates to a power data quality control method based on adaptive routing and graph snapshots, belonging to the field of power big data processing and smart grid operation technology. This method achieves intelligent scheduling and backpressure protection for high-concurrency data streams by constructing a dynamic mapping model between features and loads; it utilizes memory graph snapshots to directly perform topology consistency checks in streaming scenarios, avoiding database I / O bottlenecks; it adopts a layered processing architecture combining compliance filtering, statistical anomaly detection, and topology verification to achieve multi-dimensional quality checks while ensuring system throughput; and it introduces an automatic relaxation mechanism for rule thresholds based on manual feedback and an incremental model learning mechanism to achieve continuous self-optimization of the quality control system. This invention improves the real-time performance, accuracy, and system stability of data quality control, promotes the evolution of power data governance towards intelligence and adaptability, and enhances the data quality assurance capabilities across the entire power grid chain.

Description

Technical Field

[0001] This invention belongs to the field of power big data processing and smart grid operation technology, and relates to a power data quality control method based on adaptive routing and graph snapshots. Background Technology

[0002] With the construction of new power systems, the measurement data on the distribution and consumption side has experienced explosive growth, reaching trillions of data points and characterized by high concurrency and multi-source heterogeneity. Data quality control is fundamental to ensuring the normal operation of power grid services, encompassing multiple dimensions such as data integrity, standardization, and archive topology consistency. Existing data governance methods mainly rely on traditional ETL tools or batch processing systems based on static rules, exhibiting significant lag. Faced with millions of concurrent access requests per second, traditional architectures lack real-time perception and adaptive adjustment capabilities for system load, easily leading to message backlog or even system crashes. Furthermore, for complex archive topology verification, existing technologies often rely on frequent access to relational or graph databases, resulting in huge I / O overhead, making it difficult to meet the millisecond-level real-time requirements of streaming computing scenarios and failing to achieve closed-loop real-time control of data quality.

[0003] Patent CN114881164A discloses a method and device for automatic topology verification of power distribution networks based on graph databases. Specifically, it discloses a method for topology verification using graph technology. This method first acquires power distribution network topology information stored in the graph database, performs a depth-first traversal starting from the vertices, and merges or deletes nodes based on the closure information of switching quantities to simplify the topology. Next, the method uses a hierarchical clustering algorithm to partition the power supply area, grouping node data within the same area together. Finally, by calculating the discrete Freming distance between node voltage curves, the correlation between different nodes within the power supply area is detected, thereby identifying outliers and verifying the power distribution network topology. While this method solves the problem of low efficiency in manual identification by utilizing graph databases, its core relies on the persistent storage and query operations of the graph database. When facing massive real-time measurement data streams at the enterprise level, frequent database I / O interactions will become a serious performance bottleneck. Furthermore, the depth-first traversal and hierarchical clustering algorithms it employs have high computational complexity, making them more suitable for offline or near real-time periodic analysis. They lack adaptive backpressure routing mechanisms and memory-level streaming computing capabilities for high-concurrency scenarios, making it difficult to achieve millisecond-level real-time verification while ensuring system stability.

[0004] In summary, traditional power data quality control primarily relies on static rule-based ETL tools or batch processing systems to perform integrity, standardization, and topology consistency checks on data through periodic tasks. Its drawbacks include: a lagging processing model, inability to handle high-concurrency real-time data streams, and susceptibility to message backlog and system crashes; heavy reliance on database queries for topology verification, resulting in high I / O overhead and difficulty meeting millisecond-level real-time requirements; a lack of system load awareness and adaptive scheduling capabilities, failing to guarantee core data quality under high pressure; and rigid rules and models, limiting automatic optimization based on business feedback and thus limiting the level of intelligence. A new power data quality control method is urgently needed to address these issues. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a power data quality control method based on adaptive routing and graph snapshots. This method addresses the technical problems of existing power data quality control methods, such as lack of load adaptability when facing massive high-concurrency data access, and insufficient real-time performance due to excessive reliance on database I / O in traditional topology consistency checks. It achieves real-time, intelligent, and closed-loop control of data quality. This method constructs a load-aware adaptive backpressure routing mechanism to dynamically balance system throughput and verification depth, ensuring real-time processing of core data. Simultaneously, it introduces streaming memory graph snapshot technology to achieve millisecond-level complex topology verification without relying on external database queries. Furthermore, it combines incremental learning to achieve automatic rule evolution. This invention effectively improves the concurrent processing capability and intelligent control level of the power measurement data center, ensuring the integrity, accuracy, and timeliness of data across the entire power grid chain.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A power data quality control method based on adaptive routing and graph snapshots is proposed. By constructing a dynamic mapping model between features and loads, it achieves intelligent scheduling and backpressure protection for high-concurrency data streams. It utilizes memory graph snapshots to directly perform topology consistency verification in streaming scenarios, avoiding database I / O bottlenecks. It adopts a layered processing architecture that combines compliance filtering, statistical anomaly detection, and topology verification to achieve multi-dimensional quality verification while ensuring system throughput. Furthermore, it introduces an automatic relaxation mechanism for rule thresholds based on manual feedback and an incremental learning mechanism for the model to achieve continuous self-optimization of the quality control system.

[0007] The method specifically includes the following steps: S1: Multi-dimensional feature perception and dynamic back pressure routing: By perceiving the service characteristics of access data and the real-time load status of the system, a dynamic mapping model of "feature-load" is established to realize intelligent scheduling and back pressure protection of access traffic. S2: Multi-granularity hierarchical streaming verification pipeline: Construct a streaming processing pipeline that integrates ultra-fast compliance checks, statistical anomaly analysis, and streaming topology verification. The verification logic is hierarchically layered and processed in parallel in memory, solving the I / O bottleneck of traditional database queries and meeting high concurrency processing requirements. S3: Comprehensive Anomaly Adjudication and Hierarchical Closed-Loop Evolution: The scattered verification results produced by the pipeline in step S2 are summarized and judged to solve the problem of false alarms caused by a single method; at the same time, the verification rule parameters and model parameters in step S2 are automatically corrected in reverse by using the processing feedback of business personnel, so as to realize the self-evolution of the system.

[0008] Furthermore, step S1 specifically includes the following steps: S11: Data Access and Feature Vector Construction: Multi-source measurement data from metering automation, scheduling OCS, and IoT platforms are accessed via message queues; raw messages are parsed at the Source end of the stream processing engine to construct feature vectors reflecting data attributes. ; Eigenvector The definition is as follows:

[0009] in, Indicates the source system of the data (such as main network scheduling, distribution network scheduling, metering automation); Indicates the type of equipment and service (such as transformer voltage, line current, and meter freeze code). Data assets are classified into core categories based on their security level and business importance. Important categories and general class ; S12: System Backpressure State Quantization: Establish a full-link performance monitoring probe to collect real-time operating metrics of the computing cluster and construct a system backpressure state vector. This is used to quantify the current system's processing load;

[0010] in, This represents the average CPU utilization of the computing node. This represents the average memory usage of the computing nodes. This is the consumption backlog of the message queue (the difference between the production offset and the consumption offset). S13: Construction of Dynamic Routing Decision Matrix: Based on Eigenvectors and back pressure state vector Construct a dynamic routing decision function to determine the verification path through which data flows; set a set of system load thresholds. ,in Set the CPU alarm threshold (e.g., 80%). The backlog alarm threshold (e.g., 100,000 entries). The routing decision logic is as follows: when the system is in a stable state, i.e. At that time, execute the full route:

[0011] That is, all data undergoes rapid compliance checks in sequence. L 1. Statistical anomaly analysis L Layer 2 and streaming topology verification L 3 floors; among which, For full routing, For the process L The path on level 1, For the process L 2-level path, For the process L A 3-level path; When the system is under high pressure, that is When this occurs, backpressure protection routing is triggered:

[0012] in, For inclusion L 1. L 2. L 3. End-to-end verification; For a lightweight approach, only rapid compliance checks are performed synchronously. Simultaneously, the system generates a low-priority asynchronous task, sending the device ID and measurement values ​​of this task data to a separate message queue. A background thread consumes this queue, asynchronously updating only the time window status of the device corresponding to the statistical anomaly assessment and the latest readings of the corresponding nodes in the streaming topology verification memory graph snapshot, without triggering any verification calculations. This approach aims to ensure throughput under high system load while maintaining the freshness of the underlying state for rapid recovery of full verification when the load decreases.

[0013] Furthermore, in step S2, the rapid compliance check is... L Layer 1 specifically includes: implementing line-rate filtering using stateless operators to intercept basic data containing fatal errors and general violations; firstly, preloading the basic rule set on the Source side of the stream computing engine (such as Flink). For each record in the input data stream, perform rule matching and processing logic: Formatting errors (such as missing key fields): The data will be discarded and will not proceed to the next step. General violations (such as minor data delays): Compliance flag. Allow entry L Continue processing on layer 2; Among them, rule set include: Integrity rules: Check that critical fields (such as device ID, timestamp, and reading) are not empty; Normative rules: Data type and numerical range check (e.g., voltage) ); Timeliness rule: Calculate data latency ,like (e.g., 15 minutes) are then marked as timeout data; where, This is the preset threshold for determining data latency.

[0014] Furthermore, in step S2, the statistical anomaly assessment is... L The second layer specifically includes: using stateful operators to mine abnormal patterns in the data in the time and group dimensions based on a time sliding window; Window building: for each device Maintain a length of Time sliding window data sequence ; Vertical trend verification: The theoretical value at the current moment is predicted using a streaming exponential smoothing algorithm. :

[0015] in, The smoothing coefficient, ranging from 0 to 1, represents the degree of importance placed on recent data. A higher value indicates a more sensitive system, more easily affected by fluctuations in the latest data; a lower value indicates a more stable system, more inclined to rely on historical trends. Calculate anomaly scores :

[0016] in, Set the standard deviation within the window; The preset threshold for judging trend anomalies (usually set to 3, representing 3 times the standard deviation) is used if This was determined to be an abnormal numerical mutation. Horizontal group verification: For a set of devices G within the same distribution area, a sliding window-based streaming group anomaly detection model is used to calculate the outlier score of each device's measurement value within the group. This model is implemented through a streaming modification of the classic isolated forest algorithm, specifically including: 1) Incremental tree structure: The model maintains a fixed number of isolation trees (iTree); when a new data point arrives, the sample count and boundary information of the nodes traversed along the path of each tree are updated without rebuilding the entire tree, thus enabling continuous learning of the data distribution. 2) Time decay window: A time sliding window is associated with each tree, and only data points within the window are retained for path length calculation; old data points are removed as the window slides, ensuring that the model focuses on recent data distribution; 3) Flow cytometry outlier calculation: equipment At any moment outlier score The path length is calculated based on the average path length it traverses through each isolation tree. The path length is normalized by the data distribution within the current window; the closer the score is to 1, the more abnormal the device is in the current group.

[0017] Furthermore, in step S2, the streaming topology verification is... L The three layers specifically include: using memory graph snapshots to implement "archive topology integrity comparison" and "consistency verification"; Constructing a memory graph snapshot specifically includes mapping the power grid topology to a graph structure in memory. ,in For device nodes, It serves as the electrical connection edge; it receives file change messages in real time via broadcast stream and updates them dynamically. This ensures the real-time performance of the verification benchmark. Energy balance verification: When data flows through, it does not query the external database, but directly checks the memory graph. Search for the current node Parent-child relationship nodes; Construct a real-time energy balance equation for verification:

[0018] in, This represents the absolute value of the real-time power balance deviation. Total power consumption. The sum of the power of all sub-meters. For the current node v i The set of electrical topology child nodes, For theoretical line loss (typical loss values ​​or loss curves can be read from the line / transformer standard parameter table synchronized with the connected power grid asset management system); set a threshold. ,like If so, it is determined to be a topological anomaly (such as an error in the household-transformer relationship) or a metering anomaly.

[0019] Furthermore, step S3 specifically includes the following steps: S31: Weighted decision of multi-source abnormal signals: L Level 1 compliance check L 2-layer statistical anomaly analysis LThe outputs of the 3-layer flow topology verification are weighted and fused to calculate a comprehensive score. This will determine whether to block abnormal data. S32: Tiered Disposal Strategy: Based on the calculated... Classify and process the data; S33: Hierarchical closed-loop evolution: Establish a "feedback-optimization" closed loop; The system collects feedback from maintenance personnel on alarms (whether it is "confirmed fault" or "false alarm"), and executes differentiated automatic optimization strategies for the different levels of verification logic in step S2. S34: Uses Shadow Mode for security verification.

[0020] Furthermore, step S31 specifically includes: defining a validation vector for a single data entry. ,in This is a compliance flag (from layer L1). It takes a value of 1 for non-fatal general violations (such as delays) and 0 otherwise. The outlier is calculated (from the L2 layer), and the value ranges from [0, 1]. The larger the value, the more the data deviates from the historical pattern (for example, 0.9 represents extreme anomaly, and 0.1 represents normal fluctuation). This is used to mark topological anomalies (from L3 layer); if the energy hit is unbalanced or inconsistent across sources, the value is 1, otherwise it is 0. To highlight the importance of different verification dimensions (topological errors are generally more serious than statistical fluctuations), a weighted summation formula is used to calculate the overall score. :

[0021] in, These are the weighting coefficients, and Generally speaking, the system default settings (Topology weight) is the highest because topology errors usually mean serious problems with physical connections or files; when data is routed via backpressure protection. During processing, the default and (That is, by default, all non-L1 anomalies are considered normal), and at this time, the risk confidence level depends only on the L1 layer compliance check results.

[0022] Furthermore, step S32 specifically includes: Confirmed Interception: When If the value exceeds the set upper limit threshold (e.g., 0.8), it is judged as a serious error, the system automatically blocks the data from entering the database, and generates a defect work order; Suspected warning: When When the value is in the middle range (e.g., 0.5 to 0.8), it is judged as suspicious data. The data is allowed to be entered into the database but is marked with "suspected" and prompts for manual review. Normal release: When Data below the lower threshold (e.g., 0.5) is considered normal.

[0023] Furthermore, in step S33, the differentiated automatic optimization strategy specifically includes: Strategy 1: Targeting L 1 or L Three layers of false alarms, using statistical relaxation of rule thresholds; When business feedback shows L The Layer 3 "line loss anomaly" is a false alarm (e.g., the system sets the line loss threshold to 5%, but in actual operation, a certain line has been consistently at 5.2% and the service is confirmed to be normal). The following adjustments are made: The system extracts a set of historical data samples marked as "false alarms," ​​and then calculates the statistical characteristic value of this sample set (e.g., the maximum value). or mean Finally, the parameters are automatically corrected, and the parameters in step S2 are adjusted accordingly. L The decision threshold for Layer 3 is updated from the original value (5%) to the calculated statistical boundary value (e.g., 5.5%), thus ensuring compatibility with such operating conditions in future verifications; Strategy Two: Targeting L Two layers of false positives are addressed by incremental correction training of the model. When business feedback shows L The "numerical mutation" in Layer 2 is a false alarm (e.g., a user's normal "tiered electricity consumption" is misjudged by the model as an "abnormal data jump"). The following strategy is adopted: First, the false alarm data and its context are marked as "positive samples" (i.e. normal data); then online incremental learning is performed; specifically, the data samples that are confirmed as "false alarms" and their context (group data in the same time window) are treated as normal samples and input into the streaming group anomaly detection model based on sliding window. The model's update mechanism is triggered: the sample traverses the paths of each isolation tree and updates the statistical information of the nodes along the way (such as the number of samples and the range of values). This process is equivalent to allowing the model to "see" and learn this new normal pattern, thereby generating lower outlier scores for data points with similar characteristics in future calculations, and enabling the model to adapt to data distribution drift.

[0024] Furthermore, step S34 specifically includes: Parallel idle run: The newly generated threshold or model version runs in the background as a "shadow version", processing real-time data without issuing alarms, and monitoring for errors or timeouts. Performance comparison: The system periodically backtests the shadow version using the accumulated "historical tagged feedback sample set" and calculates its precision and recall on known samples; Automatic switching: The system will automatically perform hot switching only when the shadow version is running without any abnormalities and the backtest accuracy is better than the main version.

[0025] The beneficial effects of this invention are as follows: The core of this invention proposes a load-aware dynamic backpressure routing mechanism, streaming memory graph snapshot technology, a multi-layer parallel verification pipeline, and a feedback-based closed-loop evolution mechanism, achieving real-time, intelligent, and closed-loop management of data quality. Specifically, this invention constructs a load-aware dynamic backpressure routing mechanism, enabling the system to intelligently schedule data streams based on real-time pressure, ensuring throughput and stability under high concurrency; utilizes streaming memory graph snapshot technology to directly complete complex topology verification in memory, avoiding database I / O bottlenecks and achieving millisecond-level response; designs a layered streaming verification pipeline to process compliance, statistical anomalies, and topology verification in parallel, significantly improving the coverage depth and processing efficiency of data quality inspection; and introduces a feedback-based closed-loop evolution mechanism, which can automatically optimize rule thresholds and detection models based on business confirmation results, enabling the system to have continuous self-learning and adaptive capabilities. This invention effectively improves the real-time performance, accuracy, and overall robustness of power data quality management, providing reliable technical support for high-quality data supply for smart grids.

[0026] This invention is primarily applied to real-time quality control of power big data, smart grid operation optimization, and high-concurrency measurement data processing. By constructing a load-aware adaptive backpressure routing mechanism and streaming memory graph snapshot technology, it achieves real-time quality verification and closed-loop control of massive amounts of power data, improving the real-time performance, accuracy, and system stability of data quality control, promoting the evolution of power data governance towards intelligence and adaptability, and enhancing the data quality assurance capabilities across the entire power grid chain.

[0027] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0028] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein: Figure 1 The flowchart of the power data quality control method based on adaptive routing and graph snapshot provided by the present invention is shown. Detailed Implementation

[0029] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0030] Please see Figure 1 This invention provides a power data quality control method based on adaptive routing and graph snapshots, specifically including the following steps: Step 1: Multidimensional Feature Perception and Dynamic Backpressure Routing This step serves as the system's entry point. By sensing the service characteristics of the access data and the system's real-time load status, a dynamic "feature-load" mapping model is established to achieve intelligent scheduling and backpressure protection of access traffic. The specific implementation steps are as follows: (1) Data access and feature vector construction Multi-source measurement data from metering automation, scheduling OCS, and IoT platforms is received via message queues. The raw messages are parsed at the Source end of the stream processing engine to construct feature vectors reflecting data attributes. eigenvectors The definition is as follows:

[0031] in, Indicates the data source system (such as main network scheduling, distribution network scheduling, metering automation); Indicates the type of equipment and service (such as transformer voltage, line current, and meter freeze code). Data assets are classified into core categories based on their security level and business importance. ), important categories ( ) and general class ( ).

[0032] (2) Quantification of system back pressure state Establish a full-link performance monitoring probe to collect real-time operational metrics of the computing cluster and construct a system backpressure state vector. Used to quantify the current system's processing load:

[0033] in, This represents the average CPU utilization of the computing node. This represents the average memory usage of the computing nodes. This represents the consumption backlog of the message queue (the difference between the production offset and the consumption offset).

[0034] (3) Construction of dynamic routing decision matrix Based on feature vectors and back pressure state vector A dynamic routing decision function is constructed to determine the verification path through which data flows. A set of system load thresholds is set. ,in Set the CPU alarm threshold (e.g., 80%). The backlog alarm threshold (e.g., 100,000 entries).

[0035] The routing decision logic is as follows: when the system is in a stable state ( When ), execute the full route:

[0036] That is, all data undergoes compliance checks in sequence. L 1st layer), statistical analysis ( L 2-layer) and topology verification ( L (3 floors).

[0037] When the system is under high pressure ( When this occurs, backpressure protection routing is triggered:

[0038] in, For inclusion L 1. L 2. L 3. End-to-end verification; For lightweight paths, execution is only synchronous. L A single, ultra-fast compliance check is performed. Simultaneously, the system generates a low-priority asynchronous task that sends the device ID and measurement value of the data to a separate message queue. A background thread consumes this queue, updating only asynchronously. L The time window status of the corresponding devices at Layer 2 and L The latest reading of the corresponding node in the 3-layer memory graph snapshot does not trigger any verification calculations. This is intended to ensure throughput under high system pressure, while maintaining the freshness of the underlying state so that full verification can be quickly restored when the pressure drops.

[0039] Step 2: Multi-granularity stratified flow calibrated pipeline This step is the core of quality control, aiming to build a streaming pipeline that integrates rapid compliance checks, statistical anomaly analysis, and topology consistency verification. By layering the verification logic and processing it in parallel in memory, it solves the I / O bottleneck of traditional database queries and meets high-concurrency processing requirements. The specific implementation steps are as follows: (1) L Layer 1: Ultra-fast compliance checks based on stateless operators L Layer 1 is primarily responsible for intercepting erroneous and obviously non-compliant basic data, employing stateless operators to achieve line-speed filtering. First, a basic rule set is preloaded on the Source side of the stream processing engine (such as Flink). For each record in the input data stream, perform rule matching and processing logic: Fatal errors (such as missing critical fields): The data is discarded and not processed further.

[0040] Minor violations (such as minor data delays): flagged This allows you to proceed to the next level for further processing.

[0041] The rule set includes: Integrity rules: Check that critical fields (such as device ID, timestamp, and display value) are not empty.

[0042] Normative rules: Data type and numerical range check (e.g., voltage) ).

[0043] Timeliness rule: Calculate data latency ,like If the timeout is 15 minutes, it is marked as timeout data.

[0044] (2) L Layer 2: Statistical Anomaly Analysis Based on Sliding Window L Layer 2 utilizes stateful operators to mine anomalous patterns in the data across the time and population dimensions using a time-sliding window.

[0045] Window building: for each device Maintain a length of Time sliding window data sequence .

[0046] Vertical trend verification: The theoretical value at the current moment is predicted using a streaming exponential smoothing algorithm. :

[0047] Calculate anomaly scores :

[0048] in, The standard deviation within the window; This is a preset threshold for judging trend anomalies (usually set to 3, representing 3 times the standard deviation). If... It was determined to be an abnormal numerical mutation.

[0049] Horizontal group verification: For a set of devices G within the same distribution area, a sliding window-based streaming group anomaly detection model is used to calculate the outlier score of each device's measurement value within the group. This model is implemented through a streaming modification of the classic isolated forest algorithm, specifically including: 1) Incremental tree structure: The model maintains a fixed number of isolation trees (iTrees). When a new data point arrives, the sample count and boundary information of the nodes traversed along the path of each tree are updated without rebuilding the entire tree, thus enabling continuous learning of the data distribution.

[0050] 2) Time decay window: A time sliding window is associated with each tree, and only data points within the window are retained for path length calculation. Old data points are removed as the window slides, ensuring that the model focuses on recent data distribution.

[0051] 3) Flow cytometry outlier calculation: equipment At any moment outlier score The score is calculated based on the average path length traversing each isolation tree, with the path length normalized by the data distribution within the current window. The closer the score is to 1, the more anomalous the device is within the current population.

[0052] (3) L Layer 3: Streaming Topology Verification Based on Memory Graph Snapshots L Layer 3 utilizes in-memory computing technology to solve the I / O performance problem of complex topology verification, realizing "archive topology integrity comparison" and "consistency verification".

[0053] Memory Graph Snapshot Construction: Mapping power grid topology relationships to a graph structure in memory ,in For device nodes, This is the electrical connection side. It receives file change messages in real time via broadcast stream and updates dynamically. This ensures the real-time nature of the verification benchmark.

[0054] Energy balance verification: When data flows through, it does not query the external database, but directly checks the memory graph. Search for the current node Parent-child relationship nodes.

[0055] Construct a real-time energy balance equation for verification:

[0056] in, Total power consumption. The sum of the power of all sub-meters. The theoretical line loss (which can be read from the line / transformer standard parameter table synchronized with the connected power grid asset management system, specifying typical loss values ​​or loss curves). Set a threshold. ,like If so, it is determined to be a topological anomaly (such as an error in the household-transformer relationship) or a metering anomaly.

[0057] Step 3: Anomaly Comprehensive Judgment and Hierarchical Closed-Loop Evolution The core task of this step is to summarize and judge the scattered verification results produced by the pipeline in step 2, solving the problem of false alarms caused by a single method; at the same time, by utilizing the processing feedback from business personnel, the verification rule parameters and model parameters in step 2 are automatically corrected in reverse, realizing the self-evolution of the system. (1) Weighted decision of multi-source abnormal signals Will L Level 1 rapid compliance check L 2-layer statistical anomaly analysis L The output results of the 3-layer streaming topology verification are weighted and fused to calculate a final "risk confidence level", which determines whether to block the data.

[0058] Define a check vector for a single data entry. .in This is a compliance flag (from layer L1). It takes a value of 1 for non-fatal general violations (such as delays) and 0 otherwise. The outlier is calculated (from the L2 layer), and the value ranges from [0, 1]. The larger the value, the more the data deviates from the historical pattern (for example, 0.9 represents extreme anomaly, and 0.1 represents normal fluctuation). This is a topological anomaly marker (from layer L3). It takes a value of 1 if the energy level is unbalanced or inconsistent across sources, otherwise it takes a value of 0.

[0059] To highlight the importance of different verification dimensions (topological errors are generally more serious than statistical fluctuations), a weighted summation formula is used to calculate the overall score. :

[0060] in, These are the weighting coefficients, and Generally speaking, the system default settings (Topology weight) is the largest because topology errors usually mean serious problems with physical connections or archives.

[0061] When data is routed via backpressure protection During processing, the default and (That is, by default, all non-L1 anomalies are considered normal), at this point the risk confidence level depends only on L 1. Compliance check results.

[0062] (2) Tiered handling strategy According to the calculation Classify and process the data: Confirmed Interception: When If the value exceeds a set threshold (e.g., 0.8), it is considered a serious error, the system automatically blocks the data from being entered into the database, and generates a defect work order.

[0063] Suspected warning: When When the value is in the middle range (e.g., 0.5 to 0.8), it is considered suspicious data. The data is allowed to be entered into the database but is marked with a "suspected" label and prompts for manual review.

[0064] Normal release: When Data below a threshold (e.g., 0.5) is considered normal.

[0065] (3) Hierarchical closed-loop evolution This step establishes a "feedback-optimization" closed loop. The system collects feedback from maintenance personnel on alarms (whether it is a "confirmed fault" or a "false alarm"), and executes differentiated automatic optimization strategies for different levels of verification logic in step 2.

[0066] 1) Strategy 1: Statistical relaxation of rule thresholds (for...) L 1 / L 3-layer false alarm) When business feedback shows L The Layer 3 "line loss anomaly" is a false alarm (e.g., the system sets the line loss threshold to 5%, but in actual operation, a certain line consistently maintains a line loss of 5.2% while services are confirmed to be normal). The following adjustments will be made: The system extracts a set of historical data samples marked as "false alarms" and then calculates the statistical characteristic values ​​(such as the maximum value) of this sample set. or mean Finally, the parameters are automatically corrected, adjusting the values ​​from step 1. L The decision threshold for Layer 3 is updated from the original value (5%) to the calculated statistical boundary value (e.g., 5.5%), thus ensuring compatibility with such operating conditions in future verifications.

[0067] 2) Strategy Two: Incremental Correction Training of the Model (for...) L (2-layer false alarm) When business feedback shows L The "numerical abrupt change" at level 2 is a false alarm (e.g., a user's normal "tiered electricity consumption" is misjudged by the model as an "abnormal data jump"). The following strategy will be implemented: First, the false alarm data and its context are labeled as "positive samples" (i.e., normal data). Then, online incremental learning is performed. Specifically, the data sample identified as a "false alarm" and its context (group data within the same time window) are treated as normal samples and input into the aforementioned sliding window-based streaming group anomaly detection model.

[0068] The model's update mechanism is triggered: the sample traverses the paths of each isolation tree and updates the statistical information of the nodes along the way (such as the number of samples and the range of values). This process is equivalent to allowing the model to "see" and learn this new normal pattern, thereby generating lower outlier scores for data points with similar characteristics in future calculations, and enabling the model to adapt to data distribution drift.

[0069] (4) Shadow Mode Security Verification To prevent the automatically adjusted threshold from being too wide or the model from failing, the new strategy does not directly cover the production environment.

[0070] Parallel idle run: The newly generated threshold or model version runs in the background as a "shadow version", processing real-time data without issuing alarms, and monitoring for errors or timeouts.

[0071] Performance comparison: The system periodically backtests the shadow version using the accumulated "historical labeled feedback sample set" and calculates its precision and recall on known samples.

[0072] Automatic switching: The system will automatically perform hot switching only when the shadow version is running without any abnormalities and the backtest accuracy is better than the main version.

[0073] Application Examples: Taking a power company's measurement data center as an example, a quality control system based on the Flink stream processing engine is deployed. It receives real-time measurement data streams from metering automation, scheduling OMS, and IoT platforms via Kafka, achieving a peak throughput of 800,000 messages per second. The system collects CPU utilization, memory usage, and message backlog in real time. When CPU usage exceeds 80% or the backlog exceeds 100,000 messages, backpressure routing is automatically triggered. L1-L3 full-link verification is maintained for core data (such as substation voltage and gate power), while for general data, only L1 compliance quick checks are performed and the status is updated asynchronously, ensuring that system throughput does not decrease. In the verification pipeline, the L1 layer performs field non-empty, numerical range, and timeout checks based on a pre-loaded rule set; the L2 layer uses sliding window exponential smoothing and streaming isolated forest algorithms to detect mutations and outliers; the L3 layer builds and dynamically updates a snapshot of the power grid topology in memory, directly calculating bus power balance and line loss deviation, achieving millisecond-level topology consistency verification. All verification results are weighted and fused to generate a risk confidence score. Data with a confidence score higher than 0.8 is automatically intercepted and a work order is generated. Data with a confidence score lower than 0.5 is... Data from version 0.8 is tagged and stored in the database, and prompts for manual review. The system collects daily feedback from operations and maintenance personnel on alarms, automatically relaxes the line loss threshold using false alarm samples, or incrementally trains the anomaly detection model, and then hot-swaps it to the production environment after verification in shadow mode.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A power data quality control method based on adaptive routing and graph snapshots, characterized in that, The method specifically includes the following steps: S1: Multidimensional feature perception and dynamic backpressure routing: By perceiving the service characteristics of access data and the real-time load status of the system, a dynamic mapping model of "feature-load" is established. S2: Multi-granularity hierarchical streaming verification pipeline: Construct a streaming processing pipeline that integrates ultra-fast compliance checks, statistical anomaly analysis, and streaming topology verification, and hierarchically process the verification logic in parallel in memory; S3: Comprehensive Anomaly Adjudication and Hierarchical Closed-Loop Evolution: The scattered verification results produced by the pipeline in step S2 are summarized and judged; at the same time, the verification rule parameters and model parameters in step S2 are automatically corrected in reverse by using the processing feedback of business personnel.

2. The power data quality control method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11: Data Access and Feature Vector Construction: Multi-source measurement data from metering automation, scheduling OCS, and IoT platforms are accessed via message queues; raw messages are parsed at the Source end of the stream processing engine to construct feature vectors reflecting data attributes. ; Eigenvector The definition is as follows: in, Indicates the source system of the data; Indicates the type of equipment and service; Data assets are classified into core categories based on their security level and business importance. Important categories and general class ; S12: System Backpressure State Quantization: Establish a full-link performance monitoring probe to collect real-time operating metrics of the computing cluster and construct a system backpressure state vector. This is used to quantify the current system's processing load; in, This represents the average CPU utilization of the computing node. This represents the average memory usage of the computing nodes. The backlog of messages consumed; S13: Construction of Dynamic Routing Decision Matrix: Based on Eigenvectors and back pressure state vector Construct a dynamic routing decision function to determine the verification path through which data flows; set a set of system load thresholds. ,in CPU alarm threshold, The backlog alarm threshold; The routing decision logic is as follows: when the system is in a stable state, i.e. At that time, execute the full route: That is, all data undergoes rapid compliance checks in sequence. L 1. Statistical anomaly analysis L Layer 2 and streaming topology verification L 3 floors; among which, For full routing, For the process L The path on level 1, For the process L 2-level path, For the process L A 3-level path; When the system is under high pressure, that is When this occurs, backpressure protection routing is triggered: in, For inclusion L 1. L 2. L 3. End-to-end verification; For a lightweight path, only synchronous ultra-fast compliance checks are performed; at the same time, the system generates a low-priority asynchronous task, which sends the device ID and measurement value of the task data to a separate message queue; the background thread consumes this queue and only asynchronously updates the time window status of the corresponding device for statistical anomaly judgment and the latest reading of the corresponding node in the streaming topology verification memory graph snapshot, without triggering any verification calculations.

3. The power data quality control method according to claim 2, characterized in that, In step S2, the rapid compliance check is... L Layer 1 specifically includes: implementing line-rate filtering using stateless operators to intercept basic data containing fatal errors and general violations; firstly, preloading the basic rule set on the Source side of the stream computing engine. For each record in the input data stream, perform rule matching and processing logic: Formatting error: The data will be discarded and stored in the database without proceeding to the next step. General violation: Compliance mark Allow entry L Continue processing on layer 2; Among them, rule set include: Integrity rules: Check key fields for not being null; Normative rules: Data type and numerical range checks; Timeliness rule: Calculate data latency ,like If the data is not found, it is marked as timed out; where, This is the preset threshold for determining data latency.

4. The power data quality control method according to claim 2, characterized in that, In step S2, the statistical anomaly assessment is... L The second layer specifically includes: using stateful operators to mine abnormal patterns in the data in the time and population dimensions based on a time sliding window; Window building: for each device Maintain a length of Time sliding window data sequence ; Vertical trend verification: The theoretical value at the current moment is predicted using a streaming exponential smoothing algorithm. : in, For smoothing coefficients; Calculate anomaly scores : in, Set the standard deviation within the window; The preset threshold for determining trend anomalies is set if... This was determined to be an abnormal numerical mutation. Horizontal group verification: For a set of devices G within the same distribution area, a sliding window-based streaming group anomaly detection model is used to calculate the outlier score of each device's measurement value within the group. This model is implemented through a streaming modification of the classic isolated forest algorithm, specifically including: 1) Incremental tree structure: The model maintains a fixed number of isolation trees; when a new data point arrives, the sample count and boundary information of the nodes traversed along the path of each tree are updated without rebuilding the entire tree, thus enabling continuous learning of the data distribution. 2) Time decay window: A time sliding window is associated with each tree, and only data points within the window are retained for path length calculation; old data points are removed as the window slides, ensuring that the model focuses on recent data distribution; 3) Flow cytometry outlier calculation: equipment At any moment outlier score The path length is calculated based on the average path length it traverses through each isolation tree. The path length is normalized by the data distribution within the current window; the closer the score is to 1, the more abnormal the device is in the current group.

5. The power data quality control method according to claim 2, characterized in that, In step S2, the streaming topology verification is... L The three layers specifically include: using memory graph snapshots to implement "archive topology integrity comparison" and "consistency verification"; Constructing a memory graph snapshot specifically includes mapping the power grid topology to a graph structure in memory. ,in For device nodes, For electrical connection; receive file change messages in real time via broadcast stream and update dynamically. This ensures the real-time performance of the verification benchmark. Energy balance verification: When data flows through, it does not query the external database, but directly checks the memory graph. Search for the current node Parent-child relationship nodes; Construct a real-time energy balance equation for verification: in, This represents the absolute value of the real-time power balance deviation. Total power consumption. The sum of the power of all sub-meters. For the current node v i The set of all electrical topology sub-associated nodes, Theoretical line loss; set a threshold. ,like If so, it is determined to be a topological anomaly or a measurement anomaly.

6. The power data quality control method according to claim 2, characterized in that, Step S3 specifically includes the following steps: S31: Will L Level 1 compliance check L 2-layer statistical anomaly analysis L The outputs of the 3-layer flow topology verification are weighted and fused to calculate a comprehensive score. This will determine whether to block abnormal data. S32: Based on the calculated Classify and process the data; S33: Establish a "feedback-optimization" closed loop; the system collects feedback from maintenance personnel on alarms and executes differentiated automatic optimization strategies for different levels of verification logic in step S2. S34: Use shadow mode for security verification.

7. The power data quality control method according to claim 6, characterized in that, Step S31 specifically includes: defining the check vector for a single data entry. ,in This is a compliance flag; it is set to 1 for non-fatal general violations and 0 otherwise. To determine the outlier, the value ranges from [0, 1]. A larger value indicates that the data deviates more from historical patterns. This is used to mark topological anomalies; if the energy is unbalanced or inconsistent across sources, the value is 1, otherwise it is 0. The overall score is calculated using a weighted summation formula. : in, These are the weighting coefficients, and System default settings Maximum, when data is routed via backpressure protection During processing, the default and At this point, the risk confidence level depends solely on the results of the L1 layer compliance check.

8. The power data quality control method according to claim 7, characterized in that, Step S32 specifically includes: Confirmed Interception: When If the value exceeds the set upper limit threshold, it is judged as a serious error, the system automatically blocks the data from entering the database, and generates a defect work order; Suspected warning: When When the data is in the middle range, it is considered suspicious data. The data is allowed to be entered into the database but is marked with a "suspected" label and prompts for manual review. Normal release: When Data below the lower threshold is considered normal.

9. The power data quality control method according to claim 7, characterized in that, In step S33, the differentiated automatic optimization strategy specifically includes: Strategy 1: Targeting L 1 or L Three layers of false alarms, using statistical relaxation of rule thresholds; When business feedback shows L The "line loss anomaly" at layer 3 is a false alarm, and the following adjustments are made: The system extracts a set of historical data samples marked as "false alarms," ​​then calculates the statistical characteristic value of this sample set; finally, the parameters are automatically corrected, adjusting the steps in S2... L The decision threshold for the three layers is updated from the original value to the calculated statistical boundary value, thus ensuring compatibility with such operating conditions in future verifications; Strategy Two: Targeting L Two layers of false positives are addressed by incremental correction training of the model. When business feedback shows L The "numerical mutation" in layer 2 is a false alarm. The following strategy is adopted: First, the false alarm data and its context are marked as "positive samples"; then online incremental learning is performed. Specifically, the data samples that are confirmed as "false alarms" and their contexts are treated as normal samples and input into the streaming population anomaly detection model based on sliding window. The model's update mechanism is triggered: the sample will traverse the paths of each isolation tree and update the statistical information of the nodes it passes through.

10. The power data quality control method according to claim 6, characterized in that, Step S34 specifically includes: Parallel idle run: The newly generated threshold or model version runs in the background as a "shadow version", processing real-time data without issuing alarms, and monitoring for errors or timeouts. Performance comparison: The system periodically backtests the shadow version using the accumulated "historical labeled feedback sample set" and calculates its precision and recall on known samples; Automatic switching: The system will automatically perform hot switching only when the shadow version is running without any abnormalities and the backtest accuracy is better than the main version.

Images