A method, apparatus, and related equipment for instant acquisition and calibration of power metering data based on streaming computing.

By performing pre-verification and multi-level verification of power metering data before it is stored in the database through streaming computing, and by using shadow data stream parallel verification and optimization technology, the real-time performance and parameter optimization problems of the power metering data verification system are solved, and efficient data verification and automatic parameter adjustment are realized.

CN123116092APending Publication Date: 2026-11-17GUIZHOU POWER GRID CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611387006.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-09-08
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

The existing power metering data verification system cannot perform real-time pre-database verification, and the verification parameters cannot be adaptively optimized online, resulting in problems such as high delay in anomaly detection, incorrect verification parameters, and production shutdowns during the debugging process.

Method used

A streaming computing-based approach is used to perform pre-verification and multi-level verification on the power metering data stream before it is stored in the database. Multiple sets of candidate verification parameters are executed in parallel through shadow data streams to intercept invalid data in real time. The optimal verification parameters are obtained by optimizing based on multiple judgment results and reference labels.

Benefits of technology

It enables real-time verification of power metering data before it is entered into the database and online adaptive optimization of verification parameters, which significantly improves the real-time performance, accuracy and automation level of data verification and parameter adjustment, avoiding conflicts between anomaly detection delays and uninterrupted production.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123116092A_ABST
    Figure CN123116092A_ABST
Patent Text Reader

Abstract

This invention relates to the field of data processing technology and provides a method, apparatus, and related equipment for real-time acquisition and calibration of power metering data based on streaming computing. The method includes: performing pre-verification on the acquired data stream before data storage; performing multi-level verification on the first processed data stream that passes the pre-verification; using the data that passes the multi-level verification as the second processed data stream and writing it into a data archive; when preset debugging trigger conditions are met, copying the first processed data stream into a production data stream and a shadow data stream without interrupting the multi-level verification and data archive writing; optimizing candidate verification parameters based on multiple sets of judgment results and their corresponding reference labels to obtain the optimal verification parameters; updating the optimal verification parameters to the verification parameters used in the pre-verification and multi-level verification, and subsequently performing verification according to the updated verification parameters. This effectively improves the real-time performance, accuracy, and automation level of data verification and parameter adjustment.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus and related equipment for instant acquisition and calibration of power metering data based on streaming computing. Background Technology

[0002] Currently, the verification of electricity metering data typically employs a batch processing model. Specifically, after the acquisition terminal uploads raw data such as voltage, current, and meter readings, it is first stored uniformly in a raw database. Then, offline tasks execute predefined verification rules at fixed intervals, such as primary key missing detection, meter reading reversal judgment, and power consumption jump threshold analysis. Abnormal data is only identified after being stored in the database, requiring manual backtracking and re-initiating of data collection during processing. With the application of cloud-edge collaborative architecture on the distribution transformer area side, some systems perform local calculations locally on the edge side terminals and then upload the calculation results to the cloud platform. The cloud platform verifies the edge-up results and locates abnormal areas by setting thresholds, comparing horizontal distribution transformer areas, and comparing vertical historical data. Regarding the determination of verification parameters, current practices mainly rely on manual experience to set uniform thresholds, or first trial calculations of different parameter values ​​in an offline or test environment, then manually comparing the number of anomaly detections under each value to select the appropriate parameters before redeploying online.

[0003] Batch processing mode results in high latency for anomaly detection, failure to intercept invalid data before data entry, and the need for system downtime and redeployment for rule expansion; cloud-edge data is unidirectional, edge criteria are static, and they are disconnected from the cloud; the verification process is unidirectional and serial, and data errors in the archive baseline parameters are misjudged as data anomalies, repeatedly triggering invalid data collection; during debugging and optimization, there is a conflict between the inability to stop production and the ability to pause debugging, parallel subtask breakpoints are not reproducible, traffic segmentation samples are not comparable, and there is a lack of automatic labeling and cost differentiation criteria. As a result, the existing system struggles to achieve real-time verification before data entry and continuous optimization of verification parameters while ensuring no data loss and no business interruption. Summary of the Invention

[0004] This invention provides a method, apparatus, and related equipment for real-time verification of power metering data based on streaming computing, which solves the problem of not being able to achieve real-time verification of collected data before it is stored in the database and the problem of online adaptive optimization of verification parameters.

[0005] The first aspect of this invention provides a method for instant acquisition and calibration of power metering data based on streaming computing, comprising: Perform a pre-verification on the obtained raw power metering data stream before it is stored in the database, and use the data that passes the pre-verification as the first data stream to be processed. Perform multi-level verification on the first processed data stream, and use the data that passes the multi-level verification as the second processed data stream and write it into the data archive. When the preset debugging trigger conditions are met, without interrupting the multi-level verification and writing to the data archive, the first processed data stream is copied into a production data stream and a shadow data stream; the production data stream continues to perform the multi-level verification according to the currently effective verification parameters and is entered into the database; the shadow data stream performs the multi-level verification on the same data record with multiple sets of candidate verification parameters, obtains multiple sets of judgment results, and does not enter into the database. Based on the multiple sets of judgment results and their corresponding reference labels, the candidate verification parameters are optimized to obtain the optimal verification parameters. The optimal verification parameters are updated to the verification parameters used in the pre-verification before data entry and the multi-level verification, so that subsequent power metering data streams are verified according to the updated verification parameters.

[0006] Optionally, the step of performing pre-verification on the obtained raw power metering data stream before data entry, and using the pre-verified data as the first processed data stream, includes: Perform data cleaning on the original electricity metering data stream to filter out data with out-of-bounds values, files that cannot be parsed, or data with incorrect formats; Perform primary key integrity verification and numerical range checks on the cleaned data; Data that passes the primary key integrity verification and numerical range check will be used as the first processed data stream, while data that fails will be recorded in the exception log and will not be included in subsequent verification.

[0007] Optionally, the multi-level verification includes data integrity verification, business logic verification, and cross-source association verification; wherein: The data integrity check is used to intercept data with incorrect message format, missing fields, or overflowing values. The business logic verification is used to detect reverse meter readings, sudden changes in power consumption, or discrepancies in total and time-of-use power consumption based on historical status data. The cross-source association verification is used to review and attribute the judgment results of the business logic verification based on user profile information and historical frozen data.

[0008] Optionally, the step of performing multi-level verification on the first processed data stream, and using the data that passes the multi-level verification as the second processed data stream and writing it into the data archive includes: Perform the data integrity check on the first processed data stream, and use the data that passes the data integrity check to perform the business logic check. The cross-source association verification will be performed on the data that has passed the business logic verification. The data that passes the cross-source association verification is used as the second processed data stream and written into the data archive.

[0009] Optionally, the shadow data stream performs the multi-level verification on the same data record using multiple sets of candidate verification parameters, obtaining multiple sets of judgment results that are not entered into the database, including: Each data record in the shadow data stream is copied into multiple copies, the same number as the number of candidate verification parameter groups; The multiple copies are respectively input into multiple parallel verification branches loaded with different candidate verification parameters, and each verification branch independently executes the multi-level verification; The judgment results of each of the verification branches are collected to form a judgment vector corresponding to the same data record, and the judgment vector is written into the debugging result set instead of the data archive.

[0010] Optionally, the multiple sets of candidate verification parameters are carried through a debugging control record; The debugging control record is injected into a broadcast stream with the same verification rules, and sent down to each parallel subtask of the shadow data stream along with the data stream; The debugging control record also carries the target pause stage identifier and the effective water level line; when the shadow data stream performs the multi-level verification to the target pause stage and the local water level line advances to the effective water level line, each of the parallel subtasks pauses the execution of the verification after the target pause stage and outputs the intermediate result of the target pause stage.

[0011] Optionally, the step of optimizing the candidate verification parameters based on the multiple sets of judgment results and their corresponding reference labels to obtain the optimal verification parameters includes: The judgment results under each candidate verification parameter are compared with their corresponding reference labels to determine the missed and false alarm data under each candidate verification parameter. Based on the preset cost weights for missed detections and false positives, calculate the cost function value for each candidate verification parameter. The candidate verification parameter with the smallest cost function value is selected as the optimal verification parameter for the data cluster; wherein the data cluster is divided according to at least one of data category, user type or measurement method.

[0012] Optionally, the cost function value is: in: Indicates candidate validation parameters; Indicates candidate verification parameters The corresponding cost function value; This represents the underreporting cost weight; This represents the false positive cost weight; Indicates candidate verification parameters The amount of data missed; Indicates candidate verification parameters The false alarm data volume; the ratio of the false alarm cost weight to the false alarm cost weight is determined according to the data purpose corresponding to the data cluster.

[0013] A second aspect of the present invention provides a device for instant acquisition and calibration of power metering data based on streaming computing, comprising: The data pre-verification module is used to perform pre-verification on the acquired raw power metering data stream before it is stored in the database, and the data that passes the pre-verification is used as the first data stream to be processed. A multi-level verification module is used to perform multi-level verification on the first processed data stream, and to use the data that passes the multi-level verification as the second processed data stream and write it into the data archive. The first data processing module is used to copy the first processed data stream into a production data stream and a shadow data stream when a preset debugging trigger condition is met, without interrupting the multi-level verification and writing to the data archive. The production data stream continues to perform the multi-level verification according to the currently effective verification parameters and is stored in the database. The shadow data stream performs the multi-level verification on the same data record with multiple sets of candidate verification parameters, obtains multiple sets of judgment results, and does not store them in the database. The optimal verification parameter determination module is used to optimize the candidate verification parameters based on the multiple sets of judgment results and their corresponding reference labels to obtain the optimal verification parameters. The verification parameter update module is used to update the optimal verification parameters to the verification parameters used in the pre-verification before data entry and the multi-level verification, so that the subsequent power metering data stream can be verified according to the updated verification parameters.

[0014] A third aspect of the present invention provides an electronic device, comprising: Memory, used to store computer programs; The processor is used to execute computer programs to implement the above-mentioned method for real-time acquisition and calibration of power metering data based on streaming computing.

[0015] As can be seen from the above technical solutions, the present invention has the following advantages: This invention performs quality checks on the original power metering data stream before it is written to the data archive through pre-verification and multi-level verification. This allows for the immediate interception of invalid data, preventing abnormal meter codes from entering the business database, ensuring the accuracy of the data, and significantly reducing the anomaly detection delay, thus enabling real-time verification. When the debugging trigger condition is met, without interrupting multi-level verification and normal data entry, the first processed data stream is copied into a production data stream and a shadow data stream. The production data stream continues to perform verification and data entry according to the currently effective verification parameters, while the shadow data stream performs multi-level verification in parallel on the same data record using multiple sets of candidate verification parameters, obtaining multiple sets of judgment results without data entry. This method realizes the verification logic in real-time... Online debugging and parameter evaluation resolve the conflict between the unstoppable nature of production and the need for paused debugging. Furthermore, because the same data record is processed by multiple sets of candidate parameters, the comparability of conclusions from different parameter versions is ensured. Based on multiple sets of judgment results and their corresponding reference labels, optimization is performed to obtain the optimal verification parameters. These optimal verification parameters are then updated in the pre-verification and multi-level verification parameters used before data entry, ensuring that subsequent power metering data streams are verified according to the updated parameters. While ensuring no data loss and no business interruption, real-time verification of power metering data before data entry and online adaptive optimization of verification parameters are achieved, significantly improving the real-time performance, accuracy, and automation level of parameter adjustment in data verification. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for instant acquisition and calibration of power metering data based on streaming computing in this invention. Figure 2 This is a schematic diagram of the pre-verification process before warehousing in this invention; Figure 3 This is a flowchart illustrating the specific execution process of each stage of the multi-level verification in this invention; Figure 4 This is a schematic diagram of the process by which the shadow data stream completes multi-version parallel verification and evaluation in this invention; Figure 5 This is a schematic diagram of the process for optimizing each candidate verification parameter in this invention. Detailed Implementation

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “corresponding to,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] The implementation method in this embodiment can be implemented in a system, on a server, or on a terminal; no specific limitation is made. The method in this application will be described below from the perspective of system implementation. Please refer to... Figure 1 The present application provides a method for instant acquisition and calibration of power metering data based on streaming computing, which includes the following steps: S1. Perform pre-verification on the obtained raw power metering data stream before data entry, and use the data that passes the pre-verification as the first processed data stream; In this embodiment, the raw power metering data stream originates from acquisition terminals such as smart meters, power quality sensors, environmental sensors, photovoltaic inverters, and energy storage systems. This data includes voltage, current, active / reactive power, total / time-of-use meter readings, demand, power quality indicators, and equipment status parameters; the data is generated in real-time as a continuous stream. Pre-verification before data entry is performed by the edge computing layer to complete initial quality filtering before data is uploaded to the cloud. The edge computing layer loads local lightweight verification rules, including basic rules and dynamically loaded rules. Dynamically loaded rules are hot-updated by receiving verification parameters from the cloud, ensuring consistency between edge-side criteria and cloud-side criteria. Please refer to [link to relevant documentation]. Figure 2 Pre-verification before warehousing includes the following steps: S11. Perform data cleaning on the raw power metering data stream, filtering out data with out-of-bounds values, files that cannot be parsed, or data with incorrect formats; S12. Perform primary key integrity verification and numerical range check on the cleaned data; S13. Data that passes primary key integrity verification and numerical range check is used as the first data stream for processing. Data that fails is recorded in the exception log and is not included in subsequent verification.

[0019] Specifically, the edge computing layer parses the raw data packets, discarding data that cannot be parsed, has an invalid format, or whose values ​​significantly exceed physical possibilities, retaining only data with a correct format and reasonable values ​​for subsequent processing. The primary key is a combination of the meter ID and the collection timestamp. Primary key integrity verification checks whether the meter ID exists in the database, whether the timestamp is valid and not earlier than the last reported time, and whether there is duplicate frozen data from the same meter at the same time. Value range checks, combined with equipment range, metering method, and dynamic parameters, determine whether the data is within a reasonable operating range. For example, whether the current value of industrial users connected via current transformers is within a reasonable multiple of the rated current on the primary side of the current transformer, and whether the daily frozen meter code is negative or exceeds a reasonable cumulative range. The edge computing layer marks the data that passes the above checks as the first processed data stream and uploads it to the cloud. Data that fails is cached in the local exception log, recording the original packet, failure reason identifier, and timestamp. In normal operating mode, data that fails is only cached in the log and not uploaded; in debug mode, the edge computing layer can switch to a marked pass-through mode, adding a judgment mark to the data that should be intercepted before uploading it for the cloud shadow link to evaluate candidate parameters.

[0020] S2. Perform multi-level verification on the first processed data stream, and use the data that passes the multi-level verification as the second processed data stream and write it into the data archive. Multi-level verification is executed by the cloud computing layer, relying on the Flink stream computing engine to build an on-demand rule engine. Verification rules are dynamically loaded through the BroadcastState mechanism, enabling real-time rule updates without service restarts. Multi-level verification includes data integrity verification, business logic verification, and cross-source correlation verification. Specifically: data integrity verification intercepts data with incorrect message format, missing fields, or overflowing values; business logic verification detects reverse meter readings, sudden changes in power consumption, or discrepancies between total and time-of-use power consumption based on historical status data; and cross-source correlation verification reviews and attributes the results of business logic verification based on user profile information and historical frozen data. These three verification stages have a bidirectional dependency: cross-source correlation verification generates the transition threshold used by business logic verification based on user profiles. and total score difference threshold This data is then written back to the business logic validation; simultaneously, the result of the business logic validation determines whether to perform a complete cross-source association validation on the corresponding data. Please refer to [link to relevant documentation]. Figure 3 The specific execution process for each stage is as follows: S21. Perform data integrity verification on the first processed data stream, and perform business logic verification on the data that passes the data integrity verification; Data integrity verification performs secondary checks on the first-processed data uploaded from the edge computing layer, intercepting data with incorrect message format, missing fields, or overflowing values. For example, it checks whether the meter ID is a 12-digit number, whether the timestamp is in a valid format, whether the active power value exceeds the device's range or a reasonable range, and whether the primary key is duplicated. Data that passes the data integrity verification proceeds to business logic verification; data that fails is recorded in the exception log, terminating subsequent verification.

[0021] S22. Perform cross-source correlation verification on the data that has passed the business logic verification; Business logic validation relies on state management to track historical data. It detects anomalies such as reversed meter readings, sudden changes in battery levels, and discrepancies between total and time-of-use battery usage by comparing current data with historical meter codes and frozen values ​​stored in the state database. The business logic validation uses a jump threshold. and total score difference threshold Cross-source correlation verification is generated and written back in real time based on user profile information. For example, cross-source correlation verification reads the current transformer ratio, user type, and applied capacity from the user profile, calculates the user's load baseline and reasonable fluctuation range, and writes the results back as the threshold for business logic verification. Therefore, different user types correspond to different threshold values. The judgment result of business logic verification determines the execution path of cross-source correlation verification: when the business logic verification determines the data to be abnormal or critical, a complete cross-source correlation verification is triggered; when the business logic verification determines the data to be normal, only a lightweight correlation verification is performed, that is, only the existence of the profile and the user status are checked, in order to reduce the overhead of accessing the status database.

[0022] S23. The data that has passed the cross-source association verification is used as the second processing data stream and written to the data archive.

[0023] Cross-source correlation verification utilizes historical meter data and user profile information stored in the status database to review and attribute the judgment results of business logic verification. When business logic verification continuously determines abnormal electricity consumption fluctuations, cross-source correlation verification further compares the user's measured electricity consumption level with the current transformer ratio and applied capacity calculation values ​​registered in the profile. If a long-term systematic deviation is found and the deviation is an integer multiple, it is determined that the profile baseline parameters are incorrect, causing a threshold calculation deviation. An profile error anomaly is output, triggering the profile verification process instead of a re-collection process. If the deviation is not an integer multiple and is accompanied by a user capacity increase work order, it is determined that the electricity consumption behavior has changed genuinely, triggering the threshold update process. Only when neither of the above two situations is met is the collected data deemed abnormal, triggering a re-collection process. For data that cross-source correlation verification confirms does not belong to the above-mentioned profile errors or collection anomalies, i.e., data that passes cross-source correlation verification, it is marked as a second processing data stream and written into the data archive for subsequent billing settlement, load analysis, and other business uses. Data that fails cross-source correlation verification is recorded in the anomaly log and enters the corresponding processing process according to the attribution results.

[0024] S3. When the preset debugging trigger conditions are met, without interrupting the multi-level verification and writing to the data archive, the first processing data stream is copied into a production data stream and a shadow data stream; the production data stream continues to perform multi-level verification according to the currently effective verification parameters and enters the database; the shadow data stream performs multi-level verification on the same data record with multiple sets of candidate verification parameters, obtains multiple sets of judgment results and does not enter the database. The preset debugging trigger conditions include: maintenance personnel discovering abnormal fluctuations in the anomaly detection rate, the need to add or modify verification rules, the need to optimize verification parameters, or the need to locate errors in a certain verification process. When any of the above conditions are met, the debugging control module issues a debugging command, and the system enters debugging mode. In debugging mode, the cloud computing layer uses Flink's bypass output mechanism to copy the first processed data stream that has passed edge pre-verification into two independent links: a production data stream and a shadow data stream. The production data stream continues to load the currently effective verification parameters and executes data integrity verification, business logic verification, and cross-source correlation verification in sequence according to the normal process. Data that passes verification is written to the data archive. Its operation process is completely consistent with that before debugging and is not affected by debugging. The shadow data stream enters a dedicated debugging evaluation process, loads multiple sets of candidate verification parameters, and performs parallel verification on the same data record. Its judgment result is only written to the debugging result set, not to the data archive, and does not participate in billing business.

[0025] To ensure that debugging activities on the shadow link do not interfere with the production link, the two links are isolated in terms of resources and allocated independent operator slot sharing groups. The shadow link's buffer is set as a bounded buffer. When debugging is suspended and the shadow-side buffer reaches its limit, only the oldest record on the shadow link is discarded, and this backpressure is not propagated upstream, thus ensuring that the throughput and end-to-end latency of the production link are not affected. Please refer to [link / reference]. Figure 4 Based on the above resource isolation, the shadow data stream completes multi-version parallel verification and evaluation according to the following steps: S31. Copy each data record in the shadow data stream into multiple copies, the same number as the number of candidate check parameter groups; After the shadow data stream enters the debugging and evaluation module, a full copy operation is performed on each data record. Let the number of candidate verification parameter groups be... Then each record will be copied as Each copy is an identical replica. Each replica retains all fields of the original data record and includes a debug batch number identifier to indicate its corresponding debug batch. For example, in this embodiment, three sets of candidate transition thresholds are set. If the percentages are 5%, 8%, and 10% respectively, then each data record is copied into three copies, corresponding to the three sets of candidate parameters mentioned above. The copying operation is completed at the input end of the shadow link and has no impact on the production link.

[0026] S32. Input multiple copies into multiple parallel verification branches loaded with different candidate verification parameters, and each verification branch independently performs multi-level verification; Setting in shadow link The system employs several parallel verification branches, each pre-loaded with a set of candidate verification parameters. These candidate parameters include a transition threshold. Total score difference threshold And other rule parameters that need to be debugged. For example, the first branch loads... =5% of candidate parameters, second branch loading =8%, third branch loading =10%. Generated in S31 Each copy is routed to its corresponding verification branch. Each verification branch independently executes the complete multi-level verification process, namely, performing data integrity verification, business logic verification, and cross-source correlation verification in sequence. The computing resources and state storage of each branch are isolated from each other and do not interfere with each other. The judgment results of each verification branch are only output to the aggregation module and are not written to the data archive.

[0027] S33. Collect the judgment results of each verification branch to form a judgment vector corresponding to the same data record, and write the judgment vector into the debugging result set instead of the data archive.

[0028] For the same original data record, A copy The result after processing each verification branch Each judgment result is merged in the aggregation module according to the branch order into a result of length [length missing]. The decision vector. Each element in the decision vector represents the verification conclusion of the data record under the corresponding candidate verification parameters. For example, [abnormal, abnormal, normal] indicates that the same data record... =5% and =8% was judged as abnormal, =10% is considered normal. The judgment vector is appended with the original data record identifier, debugging batch number and timestamp, and written to the debugging result set. The debugging result set is stored in a separate database or in-memory table, physically isolated from the data archive, and is only used for subsequent label comparison, cost function calculation and parameter optimization, and does not participate in any billing settlement or load analysis business.

[0029] In this embodiment, multiple sets of candidate verification parameters are carried through a debug control record. The debug control record is injected into a broadcast stream identical to the verification rules and distributed along with the data stream to each parallel subtask of the shadow data stream. The debug control record contains at least the following fields: debug batch number, target pause stage identifier, candidate parameter set, and effective water level value. The candidate parameter set is the one used in steps S31 to S33. The specific values ​​of the candidate verification parameters for the group; the target pause stage identifier is used to specify after which verification stage the process is suspended; the effective water level is a logical timestamp used to control the precise effective time of the breakpoint.

[0030] The debug control log also carries the target pause stage identifier and the effective water level. When each parallel subtask in the shadow data stream performs multi-level verification up to the target pause stage and its local water level advances to the effective water level, it pauses the verification execution after the target pause stage and outputs the intermediate result of the target pause stage. Specifically, after receiving the debug control log, each parallel subtask in the shadow data stream continues to execute the verification process until two conditions are met: first, the subtask has executed up to the verification stage specified by the target pause stage identifier; second, the subtask's local water level has advanced to the effective water level specified in the debug control log. When both conditions are met simultaneously, the subtask is suspended after the target stage, outputs the intermediate judgment result of that stage, and pauses the execution of all verification stages after the target pause stage. Since all parallel subtasks use the same effective water level as the alignment reference, all subtasks pause at the same logical point in time and the same data boundary, and the intercepted intermediate result sets are completely consistent. Debugging is initiated repeatedly with the same debug batch number, and the intermediate result sets intercepted in the two debugging sessions are identical, demonstrating the reproducibility of the debugging conclusions.

[0031] The aforementioned breakpoint mechanism only applies to the shadow link; the production link always runs at full speed and is unaffected. By moving the breakpoint step by step, intermediate results can be captured sequentially after data integrity verification, after each sub-rule of business logic verification, and after cross-source correlation verification. Differences can be compared and determined at each level to accurately locate the verification step where the error occurred. This mechanism also applies to the edge computing layer: the debugging control module can send debugging control commands to the edge computing layer through the rule distribution channel, causing each step of the edge-side pre-verification to pause and output results after completion, thereby locating the specific step on the edge side where the error occurred.

[0032] S4. Based on multiple sets of judgment results and their corresponding reference labels, optimize the candidate verification parameters to obtain the optimal verification parameters; In this embodiment, the reference label refers to the baseline information used to determine whether the verification result of a certain data record is a correct judgment, a missed report, or a false alarm. The sources of the reference labels include automatic business event comparison labels, T+1 full recalculation reference labels, and manual on-site verification backfill labels. Automatic business event comparison labels are obtained from the marketing business system, including meter replacement work orders, planned power outage records, equipment maintenance records, and user capacity increase change records. The verification judgment results are associated with the time windows of these business events. If the occurrence time of a certain anomaly judgment falls within the time window of the corresponding business event, and the anomaly type matches the expected data performance of that event, it is automatically marked as a false alarm. The T+1 full recalculation reference label uses the results of a complete verification of the same batch of data performed on the next day in a batch processing manner as a reference set, comparing it with the real-time judgment results one by one. Differences are marked as pending review. Manual on-site verification backfill labels are filled in after the customer manager's on-site verification conclusions, user appeal processing results, and refund / refund electricity records have been entered. Because the third type of label has a delay of several days, and the parameter version may have been iterated multiple times during the debugging period, the system archives a triplet record of parameter version number, judgment result and label for each verification judgment result. Delayed labels are back-collected to the corresponding version according to the parameter version number of the judgment result associated with them, so as to avoid label mismatch.

[0033] For pre-verification before data entry, the candidate parameter set is independent of the multi-level verification candidate parameter set. Candidate pre-verification parameters include upper and lower limit thresholds in numerical range checks, required field combinations in primary key integrity verification, and outlier removal thresholds in data cleaning. The second shadow data stream performs pre-verification on each group of candidate pre-verification parameters, generating corresponding pre-verification judgment results. The reference labels for the pre-verification judgment results are also derived from automatic comparison of business events, T+1 full recalculation reference, and manual on-site verification backfilling; their labeling objects are the interception / release conclusions of the pre-verification stage. Following the same method as multi-level verification parameter optimization, the pre-verification judgment results under each group of candidate pre-verification parameters are compared with the reference labels, and missed and false alarm data are statistically analyzed. Based on the preset missed and false alarm cost weights, the cost function value is calculated, and the candidate pre-verification parameter with the smallest cost function value is selected as the optimal pre-verification parameter for that data cluster.

[0034] Please see Figure 5 After obtaining multiple sets of judgment results and their corresponding reference labels, the following steps are used to optimize each candidate verification parameter: S41. Compare the judgment results under each candidate verification parameter with their corresponding reference labels to determine the missed data and false alarm data under each candidate verification parameter; For each set of candidate verification parameters, all judgment results generated within the debugging batch are compared with the corresponding reference labels one by one. If the reference label of a data record indicates that the record has a real anomaly, but the judgment result under the candidate verification parameter is normal, then the record is counted as a missed data under that candidate verification parameter; if the reference label of a data record indicates that the record does not have an anomaly, but the judgment result under the candidate verification parameter is an anomaly, then the record is counted as a false alarm under that candidate verification parameter. Records whose judgment results are consistent with the reference labels are not counted as missed or false alarm data. Through the above comparison, the number of missed data and false alarm data corresponding to each set of candidate verification parameters are statistically obtained.

[0035] S42. Based on the preset false positive and false negative cost weights, calculate the cost function value for each candidate verification parameter; the cost function value is: in: Indicates candidate validation parameters; Indicates candidate verification parameters The corresponding cost function value; Indicates the cost weight of underreporting; Indicates the cost weight of false alarms; Indicates candidate verification parameters The amount of data missed; Indicates candidate verification parameters The amount of false alarm data is determined by the ratio of the cost weight for missed alarms to the cost weight for false alarms, based on the intended use of the data clusters. For data directly involved in electricity billing, including daily and monthly frozen meter codes, missed alarms will result in incorrect bills and refunds / refunds, with costs far exceeding those for false alarms. Therefore, the ratio of the cost weight for missed alarms to the cost weight for false alarms is set between 5 and 20, preferably 10, to strongly suppress missed alarms. For data used only for load analysis, including curve data, the costs of missed alarms and false alarms are comparable. The ratio of the cost weight for missed alarms to the cost weight for false alarms is set between 0.5 and 2, preferably 1, to treat missed alarms and false alarms equally.

[0036] S43. Select the candidate verification parameter with the smallest cost function value as the optimal verification parameter for the data cluster; wherein, the data cluster is divided according to at least one of data category, user type or measurement method.

[0037] Parameter optimization is performed independently based on data clusters. The dimensions for data clustering include: data category (curve data, daily frozen table data, monthly frozen table data); user type (residential users, general industrial and commercial users, large industrial users); and metering method (direct access, access via instrument transformer). In practice, clustering can be performed based on at least one of the above dimensions, such as a combination of data category and user type, or a combination of data category and metering method.

[0038] Each data cluster independently executes the statistical and calculation processes S41 and S42, obtaining the missed data volume, false alarm data volume, and cost function value of each candidate verification parameter for that cluster. The candidate verification parameter with the smallest cost function value is selected as the optimal verification parameter for that cluster. Different clusters may select different optimal verification parameters, ultimately forming a parameter matrix rather than a single parameter value. For example, within the same candidate threshold group, the optimal value is 8% for the daily frozen table code cluster, 10% for the curve data cluster, and the optimal value for the large industrial user cluster differs from that for the residential user cluster. By optimizing independently for each cluster, the problem of a uniform threshold across the entire network being too lenient for some groups and too strict for others is avoided, ensuring that the verification parameters match the actual data characteristics.

[0039] S5. Update the optimal verification parameters to the verification parameters used in the pre-verification and multi-level verification before data entry, so that the subsequent power metering data stream can be verified according to the updated verification parameters.

[0040] Specifically, the optimal multi-level verification parameters and optimal pre-verification parameters obtained from S4 optimization are progressively updated to the corresponding stages through a three-level gray-scale write-back process. The optimal multi-level verification parameters update the verification parameters used for multi-level verification in the cloud through the rule distribution channel; the optimal pre-verification parameters update the verification parameters used for pre-verification before edge database entry through the same rule distribution channel. The edge computing layer completes hot updates through a dynamic parameter loading mechanism. The cloud computing layer and the edge computing layer each record the version number and effective time of the currently effective parameters and support one-click rollback. The optimal verification parameters are run on a complete debugging batch on the shadow link to observe the comparison between their judgment results and reference labels, confirming that their cost function value is better than the currently effective parameters. The judgment results of the shadow verification stage are only written to the debugging result set, not to the data archive, and do not participate in any billing services. The optimal verification parameters that pass the shadow verification are distributed to a small number of selected areas to officially take effect. During distribution, the system assigns a globally unique parameter version number to this parameter update and distributes it to both the cloud computing layer and the edge computing layer simultaneously through the rule distribution channel. The trial operation involves continuous observation for no less than seven data collection cycles. During this period, the shadow judgment results of the original parameters are retained as a reference to continuously compare the cost function performance of the new and original parameters. When the cost function value of the optimal verification parameter is consistently better than that of the original parameter during the trial operation, the system distributes the parameter to the entire network through the same rule distribution channel, updating the verification parameters used for multi-level verification in the cloud and pre-verification before edge data entry. Subsequent power metering data streams will then be verified according to the updated verification parameters. The edge computing layer completes hot updates through a dynamic parameter loading mechanism, without requiring a restart or interruption of the data collection service.

[0041] Regarding version management, both the cloud computing layer and the edge computing layer record the version number and effective time of the currently effective parameters. If an updated parameter is found to have degraded the false alarm rate or false negative rate, a one-click rollback to the previous version is possible based on the version number. The rollback command is sent through the same broadcast channel, and the rollback is synchronized between the cloud and the edge to avoid conflicting criteria due to inconsistent parameter versions. For parameter updates involving data categories directly related to electricity billing, double confirmation and approval records must be maintained.

[0042] In debug mode, if the edge computing layer switches to the marker pass-through mode to eliminate blind spots in the optimization observation, the edge computing layer automatically reverts to interception mode after the debug batch ends and the optimal verification parameters are updated. It then performs pre-verification before data entry according to the updated verification parameters. Thus, the system forms a two-way parameter closed loop: edge pre-verification, cloud-based multi-level verification, shadow link optimization, parameter write-back and distribution, and edge hot update. Verification parameters can be continuously optimized and take effect synchronously based on actual data performance.

[0043] Furthermore, an embodiment of the power metering data acquisition and calibration device based on streaming computing provided by the present invention includes the following: The data pre-verification module is used to perform pre-verification on the acquired raw power metering data stream before it is stored in the database, and the data that passes the pre-verification is used as the first data stream to be processed. The multi-level verification module is used to perform multi-level verification on the first processed data stream, and the data that passes the multi-level verification is used as the second processed data stream and written into the data archive. The first processing data copying module is used to copy the first processing data stream into a production data stream and a shadow data stream when the preset debugging trigger conditions are met, without interrupting the multi-level verification and writing to the data archive. The production data stream continues to perform the multi-level verification according to the currently effective verification parameters and is stored in the database. The shadow data stream performs multi-level verification on the same data record with multiple sets of candidate verification parameters, obtains multiple sets of judgment results, and does not store them in the database. The optimal verification parameter determination module is used to optimize candidate verification parameters based on multiple sets of judgment results and their corresponding reference labels to obtain the optimal verification parameters. The verification parameter update module is used to update the optimal verification parameters to the verification parameters used in the pre-verification and multi-level verification before data entry, so that the subsequent power metering data stream can be verified according to the updated verification parameters.

[0044] For specific limitations regarding the device, please refer to the limitations of the method above, which will not be repeated here. Each module in the above device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0045] Furthermore, the present invention provides an electronic device comprising: Memory, used to store computer programs; A processor for executing computer programs to implement the streaming computing-based method for on-demand calibration of electricity metering data, as described above.

[0046] It is understood that those skilled in the art can combine various implementation methods in the above embodiments under the guidance of the above examples to obtain technical solutions with multiple implementation methods.

[0047] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for immediate acquisition and calibration of power metering data based on streaming computing, characterized in that, include: Perform a pre-verification on the obtained raw power metering data stream before it is stored in the database, and use the data that passes the pre-verification as the first data stream to be processed. Perform multi-level verification on the first processed data stream, and use the data that passes the multi-level verification as the second processed data stream and write it into the data archive. When the preset debugging trigger conditions are met, without interrupting the multi-level verification and writing to the data archive, the first processed data stream is copied into a production data stream and a shadow data stream; the production data stream continues to perform the multi-level verification according to the currently effective verification parameters and is then entered into the database. The shadow data stream performs the multi-level verification on the same data record with multiple sets of candidate verification parameters, obtains multiple sets of judgment results, and does not enter the database. Based on the multiple sets of judgment results and their corresponding reference labels, the candidate verification parameters are optimized to obtain the optimal verification parameters. The optimal verification parameters are updated to the verification parameters used in the pre-verification before data entry and the multi-level verification, so that subsequent power metering data streams are verified according to the updated verification parameters.

2. The method for immediate acquisition and calibration of power metering data based on streaming computing as described in claim 1, characterized in that, The step of performing a pre-verification on the acquired raw power metering data stream before data entry, and using the pre-verified data as the first processed data stream, includes: Perform data cleaning on the original electricity metering data stream to filter out data with out-of-bounds values, files that cannot be parsed, or data with incorrect formats; Perform primary key integrity verification and numerical range checks on the cleaned data; Data that passes the primary key integrity verification and numerical range check will be used as the first processed data stream, while data that fails will be recorded in the exception log and will not be included in subsequent verification.

3. The method for immediate acquisition and calibration of power metering data based on streaming computing as described in claim 1, characterized in that, The multi-level verification includes data integrity verification, business logic verification, and cross-source correlation verification; wherein: The data integrity check is used to intercept data with incorrect message format, missing fields, or overflowing values. The business logic verification is used to detect reverse meter readings, sudden changes in power consumption, or discrepancies in total and time-of-use power consumption based on historical status data. The cross-source association verification is used to review and attribute the judgment results of the business logic verification based on user profile information and historical frozen data.

4. The method for immediate acquisition and calibration of power metering data based on streaming computing according to claim 3, characterized in that, The step of performing multi-level verification on the first processed data stream, and using the data that passes the multi-level verification as the second processed data stream and writing it into the data archive, includes: Perform the data integrity check on the first processed data stream, and use the data that passes the data integrity check to perform the business logic check. The cross-source association verification will be performed on the data that has passed the business logic verification. The data that passes the cross-source association verification is used as the second processed data stream and written into the data archive.

5. The method for immediate acquisition and calibration of power metering data based on streaming computation according to claim 1, characterized in that, The shadow data stream performs the multi-level verification on the same data record using multiple sets of candidate verification parameters, obtaining multiple sets of judgment results that are not entered into the database, including: Each data record in the shadow data stream is copied into multiple copies, the same number as the number of candidate verification parameter groups; The multiple copies are respectively input into multiple parallel verification branches loaded with different candidate verification parameters, and each verification branch independently executes the multi-level verification; The judgment results of each of the verification branches are collected to form a judgment vector corresponding to the same data record, and the judgment vector is written into the debugging result set instead of the data archive.

6. The method for immediate acquisition and calibration of power metering data based on streaming computing according to claim 5, characterized in that, The multiple sets of candidate verification parameters are carried through debugging control records; The debugging control record is injected into a broadcast stream with the same verification rules, and sent down to each parallel subtask of the shadow data stream along with the data stream; The debugging control record also carries the target pause stage identifier and the effective water level line; when the shadow data stream performs the multi-level verification to the target pause stage and the local water level line advances to the effective water level line, each of the parallel subtasks pauses the execution of the verification after the target pause stage and outputs the intermediate result of the target pause stage.

7. The method for immediate acquisition and calibration of power metering data based on streaming computing as described in claim 5, characterized in that, The process of optimizing candidate verification parameters based on the multiple sets of judgment results and their corresponding reference labels to obtain optimal verification parameters includes: The judgment results under each candidate verification parameter are compared with their corresponding reference labels to determine the missed and false alarm data under each candidate verification parameter. Based on the preset cost weights for missed detections and false positives, calculate the cost function value for each candidate verification parameter. The candidate verification parameter with the smallest cost function value is selected as the optimal verification parameter for the data cluster; wherein the data cluster is divided according to at least one of data category, user type or measurement method.

8. The method for immediate acquisition and calibration of power metering data based on streaming computing according to claim 7, characterized in that, The cost function value is: in: Indicates candidate validation parameters; Indicates candidate verification parameters The corresponding cost function value; This represents the underreporting cost weight; This represents the false positive cost weight; Indicates candidate verification parameters The amount of data missed; Indicates candidate verification parameters The false alarm data volume; the ratio of the false alarm cost weight to the false alarm cost weight is determined according to the data purpose corresponding to the data cluster.

9. A device for instant acquisition and calibration of power metering data based on streaming computing, characterized in that, The method for immediate acquisition and calibration of power metering data based on streaming computing as described in any one of claims 1 to 8 includes: The data pre-verification module is used to perform pre-verification on the acquired raw power metering data stream before it is stored in the database, and the data that passes the pre-verification is used as the first data stream to be processed. A multi-level verification module is used to perform multi-level verification on the first processed data stream, and to use the data that passes the multi-level verification as the second processed data stream and write it into the data archive. The first data processing module is used to copy the first processed data stream into a production data stream and a shadow data stream when a preset debugging trigger condition is met, without interrupting the multi-level verification and writing to the data archive. The production data stream continues to perform the multi-level verification according to the currently effective verification parameters and is stored in the database. The shadow data stream performs the multi-level verification on the same data record with multiple sets of candidate verification parameters, obtains multiple sets of judgment results, and does not store them in the database. The optimal verification parameter determination module is used to optimize the candidate verification parameters based on the multiple sets of judgment results and their corresponding reference labels to obtain the optimal verification parameters. The verification parameter update module is used to update the optimal verification parameters to the verification parameters used in the pre-verification before data entry and the multi-level verification, so that the subsequent power metering data stream can be verified according to the updated verification parameters.

10. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program to implement the streaming computing-based method for instant acquisition and calibration of power metering data as described in any one of claims 1 to 8.