An energy internet multi-source heterogeneous data anomaly treatment method

Abstract

The application discloses an energy internet multi-source heterogeneous data anomaly treatment method, relates to the technical field of energy internet data processing, and comprises the following steps: processing multi-source heterogeneous data to form a multi-modal data set to be treated; constructing a space-time semantic correlation graph, fusing a horizontal reference benchmark and a vertical reference benchmark to calculate cross-source collaborative anomaly confidence, marking abnormal data and abnormal types; matching a treatment operation sequence to repair and calculate an offset influence factor; dynamically adjusting an anomaly detection threshold according to the offset influence factor, recording an execution log and updating a treatment strategy library priority weight; and circularly executing until the number of abnormal marks is lower than an abnormal mark number threshold, and outputting a clean data set. The energy internet multi-source heterogeneous data anomaly treatment method realizes cross-source collaborative anomaly detection, interactive feedback of treatment operation and criterion updating, improves the pertinence of abnormal data repair, and realizes adaptive adjustment of treatment strategies through a dynamic threshold evolution mechanism.

Description

Technical Field

[0001] This invention relates to the field of energy internet data processing technology, specifically to a method for managing anomalies in multi-source heterogeneous data in the energy internet. Background Technology

[0002] As a complex mega-system that deeply integrates power systems, communication networks, the Internet of Things, and information systems, the Energy Internet's data sources cover a variety of heterogeneous data sources, including smart meters, sensor networks, SCADA systems, meteorological monitoring, and user-side terminals. The data types include multimodal forms such as numerical measurement data, text alarm logs, time-series load curves, and semi-structured configuration files. The heterogeneous nature of the multi-source data makes it easy for anomalies such as missing data, conflicts, delays, jumps, and semantic inconsistencies to occur in the collection, transmission, storage, and processing stages, which seriously affects the decision-making accuracy of core applications such as upper-level situational awareness, fault diagnosis, and optimization and control. Most existing data anomaly management methods rely on threshold detection or statistical models based on a single data source for anomaly identification, followed by independent data repair methods for correction. The detection and repair phases are disconnected, and the correction process fails to effectively utilize the anomaly feature information generated during the detection phase. This results in a lack of consistency between the repaired data and the detection criteria, with the repaired value potentially falling on the opposite side of the detection threshold, thus introducing new anomalies or causing missed anomalies. Furthermore, existing methods lack closed-loop feedback and adaptive evaluation mechanisms for management effectiveness. Once the management rules are set, they remain fixed and difficult to adapt to the dynamic changes in the operating conditions of the energy internet. As a result, management performance deteriorates significantly in scenarios such as system topology changes, load characteristic migration, or fluctuations in renewable energy output. Summary of the Invention

[0003] The purpose of this invention is to provide a method for managing anomalies in multi-source heterogeneous data in the energy internet, in order to solve the problems of insufficient detection accuracy, fragmented management process, and lack of adaptive strategy in the existing technologies for managing anomalies in multi-source heterogeneous data in the energy internet as mentioned in the background. To achieve the above objectives, the present invention provides the following technical solution: a method for managing anomalies in multi-source heterogeneous data in the energy internet, comprising the following steps: S1. Standardize the format, align the timestamps, and mark missing data for multi-source heterogeneous data to form a multimodal dataset to be governed with a unified time benchmark; S2. Construct a spatiotemporal semantic association graph based on the physical connection, business logic, and temporal coupling relationship between data sources. Extract horizontal and vertical reference benchmarks from the associated nodes and the historical windows of the same node, respectively, and determine the semantic consistency score. Calculate the cross-source collaborative anomaly confidence based on the horizontal deviation, vertical deviation, and semantic consistency score. When the confidence score exceeds the preset anomaly detection threshold, mark the abnormal data, and determine the dominant anomaly type among source-end anomaly, transmission anomaly, and semantic anomaly based on the confidence contributions of the three. S3. Match and execute governance operation sequences from the governance strategy library based on the abnormal data and the dominant anomaly type to replace the abnormal data with corrected data, forming an updated multimodal dataset to be governed; calculate the offset influence factor based on the feature changes of the data segments before and after governance, the influence of the operation parameters of the governance operation sequence, and the cross-source collaborative anomaly confidence before governance. S4. Update the anomaly detection threshold based on the offset impact factor, record the execution log of the governance operation, and update the priority weight of the governance operation sequence in the governance strategy library based on the governance effect score. S5. Based on the updated dataset, repeat steps S2 to S4. When the number of anomaly markers for a consecutive preset positive integer number of data periods is lower than the preset anomaly marker count threshold, terminate the loop and output a clean dataset. Furthermore, the method for constructing the spatiotemporal semantic association graph is as follows: data records with independent semantic meanings in each data source are used as graph nodes. The attributes of the nodes include data source identifier, data type label, collection timestamp, and physical dimension; a first type of edge is established based on the device topology connection relationship, a second type of edge is established based on the business logic dependency relationship, and a third type of edge is established based on the temporal synchronization relationship. The temporal synchronization relationship refers to the temporal association between data collected from different data sources at the same time. Furthermore, the cross-source collaborative anomaly confidence score is calculated using the following combined algorithm: in, Indicates the first Data nodes at The confidence level of cross-source collaboration anomalies at any given moment. This represents a lateral deviation measure between the lateral reference baseline and the data to be tested. This is the normalized constant for the lateral deviation. This represents a measure of the longitudinal deviation between the longitudinal reference baseline and the data to be tested. This is the normalization constant for longitudinal deviation; when and When any item in the normalization bias is greater than 1, the normalization biases greater than 1 are respectively set to 1, and the processed values ​​are... and Substitute into the above formula to calculate the confidence level of cross-source collaborative anomalies; A semantic consistency score is assigned between the data to be detected and the associated nodes in the spatiotemporal semantic association graph. , , The weighting coefficients are non-negative and satisfy the following conditions: + + =1. Furthermore, in the calculation of the cross-source collaborative anomaly confidence, when the number of available horizontal reference benchmarks for the data to be detected is less than a preset threshold, the weighting coefficient corresponding to the vertical reference benchmark is increased. And reduce the weighting coefficient corresponding to the lateral reference datum. When the historical data integrity rate of the longitudinal reference benchmark of the data to be tested is lower than the preset integrity rate threshold, the weight coefficient corresponding to the lateral reference benchmark is increased. And reduce the weighting coefficient corresponding to the longitudinal reference benchmark. When both of the above situations occur simultaneously within the same data period, and Maintain the value before the adjustment in this cycle; if only one of the conditions is met... remain unchanged. and Adjust the weights in the opposite direction with the same adjustment amount, and the three weights after adjustment are all non-negative and their sum is 1. Furthermore, the atomic governance operations include missing value imputation, limit smoothing, temporal realignment, semantic translation, and deduplication. The governance operation sequence is a combination of the atomic governance operations, which is determined by matching from the governance strategy library based on the dominant anomaly type and the characteristic parameters of the anomaly data. Specifically, source anomalies are preferentially matched with the combination of limit smoothing and temporal realignment. Transmission anomalies are matched with at least one of temporal realignment, missing value imputation, and deduplication based on time offset, missing data markers, and duplicate reporting features. Semantic anomalies are preferentially matched with semantic translation. Furthermore, the method for calculating the offset impact factor is as follows: calculate the quality feature vectors for the abnormal data segments before and after the governance operation, wherein the quality feature vectors include the mean, variance, range, and temporal rate of change; calculate the offset impact factor based on the feature changes before and after governance, the impact of the operation parameters of the governance operation sequence, and the cross-source collaborative anomaly confidence level before the governance operation. Furthermore, the dynamic adjustment method of the preset anomaly detection threshold is as follows: when the offset influence factor is greater than the first threshold, the preset anomaly detection threshold is increased by one step; when the offset influence factor is less than the second threshold, the preset anomaly detection threshold is decreased by one step; the first threshold is greater than the second threshold; when the offset influence factor is greater than or equal to the second threshold and less than or equal to the first threshold, the current anomaly detection threshold remains unchanged. Furthermore, the execution log in S4 includes governance execution time, governance object identifier, governance operation sequence identifier, comparison of data features before and after governance, and governance effect score; the governance effect score is calculated and generated based on the normalized horizontal deviation, normalized vertical deviation, and semantic consistency score corresponding to the corrected data. Furthermore, the governance strategy library is updated as follows: for each governance operation, the utility value of the governance operation sequence is calculated based on the governance effect score, and the utility value is used as a feedback signal to update the priority weight of the governance operation sequence under the corresponding anomaly type in a weighted moving average manner. Furthermore, the preset threshold for the number of anomaly markers in S5 is dynamically set based on the total amount of data and the number of data types in the multimodal dataset to be managed. Compared with the prior art, the beneficial effects of the present invention are: This energy internet multi-source heterogeneous data anomaly governance method achieves cross-source collaborative anomaly detection by constructing a spatiotemporal semantic association graph, improves the targeting of anomaly data repair through interactive feedback between governance operations and criterion updates, and achieves adaptive adjustment of governance strategies through a dynamic threshold evolution mechanism. 1. Furthermore, by constructing a spatiotemporal semantic association graph, the detection is integrated with horizontal multi-source verification and vertical historical verification. Compared with the single data source threshold judgment method, the false alarm rate and false negative rate are significantly reduced. 2. Furthermore, a closed-loop iterative architecture of anomaly detection, governance execution, and criterion update is adopted, which uses the feedback information of governance operations to dynamically adjust the detection threshold, thus solving the problem of the separation of detection and correction information in the traditional serial architecture. 3. Furthermore, the perturbation of the detection criteria by the offset influence factor is quantified, and the threshold is adaptively evolved, so that the governance strategy can be adjusted with the changes in the system's operating state, thereby improving the adaptability of the method under different operating conditions. 4. Furthermore, by dynamically updating the priority weights of the governance strategy library, governance experience can be accumulated and reused. As the operating cycle increases, it helps to improve the efficiency and accuracy of governance of similar anomalies in the future. Attached Figure Description

[0004] Figure 1This is a schematic diagram of the overall process of the energy internet multi-source heterogeneous data anomaly management method of the present invention; Figure 2 This is a schematic diagram of the spatiotemporal semantic association graph of the present invention; Figure 3 This is a schematic diagram illustrating the principle of cross-source collaborative anomaly confidence calculation in this invention. Figure 4 This is a schematic diagram of the closed-loop feedback of the governance operation and threshold dynamic adjustment of the present invention. Detailed Implementation

[0005] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. This invention provides the following technical solution: a method for managing anomalies in multi-source heterogeneous data in the energy internet. Its core lies in achieving cross-source collaborative detection through spatiotemporal semantic association graphs and achieving threshold adaptive evolution through governance feedback linkage. This solves the problems of fragmented detection and repair and lack of closed-loop feedback in existing methods, and is suitable for efficient governance of multi-source heterogeneous data in the energy internet. Example 1: To better understand the above technical solution, the following will provide a detailed description of the technical solution in conjunction with the accompanying drawings and specific implementation methods. (Refer to...) Figure 1 As shown in the figure, this is a schematic diagram of the overall process of the energy internet multi-source heterogeneous data anomaly governance method. The energy internet multi-source heterogeneous data anomaly governance method includes the following steps: S1. Perform format normalization, timestamp alignment, and missing data marking on multi-source heterogeneous data to form a multimodal dataset to be addressed with a unified time base. For numerical measurement data, retain the numerical sequence after format normalization and unit unification. For text-based alarm data, calculate the alarm field missing ratio, alarm code mismatch ratio, and alarm semantic constraint violation ratio, respectively. Limit the arithmetic mean of these three ratios to 0-1, and use 1 minus the arithmetic mean as the text quality value. For semi-structured configuration data, calculate the field missing ratio, field type inconsistency ratio, and configuration constraint violation ratio, respectively. Limit the arithmetic mean of these three ratios to 0-1, and use 1 minus the arithmetic mean as the text quality value. The result of subtracting the arithmetic mean from 1 is used as the configuration quality value; this converts non-numerical data into a numerical quality sequence for deviation measurement and comparison of features before and after governance. This numerical quality sequence is used only for anomaly detection and governance effect calculation, while the original text-based alarm data and semi-structured configuration data remain unchanged. Specifically, the format standardization process includes: uniformly converting numerical data to IEEE 754 double-precision floating-point numbers, uniformly converting units to the International System of Units (SI), using NTP-synchronized Unix timestamps for timestamp alignment, performing time-series compensation based on the timestamps in the data packets using linear interpolation for data transmission delay, and using bitmaps for missing marker handling. The bitmap method records the valid status of each data point, marking valid data as 1 and missing data as 0, forming a multimodal dataset to be addressed with a unified time reference. S1 serves as the data entry layer and basic support layer of this invention. Through format standardization and unit unification, it enables multimodal data from different data sources such as smart meters, SCADA systems, and meteorological monitoring to have an interoperable data format, providing structurally consistent node attribute data for the subsequent construction of the spatiotemporal semantic association graph in S2. NTP timestamp alignment and timing compensation address the timing misalignment problem caused by asynchronous acquisition clocks and varying transmission delays in multi-source data, ensuring timing synchronization in S2. The establishment of the third type of edge has an accurate time base, enabling the extraction of the horizontal reference benchmark to have true contemporaneous comparability. By recording the valid / missing status of each data point through a bitmap, the data quality information is decoupled from the data itself. This provides a preliminary judgment basis for the targeted matching of the governance operation sequence in S3, especially the triggering of missing value imputation operations, and avoids the pollution of statistical results by missing values ​​in subsequent confidence calculation and feature extraction. S1 is the data prerequisite to ensure the effective operation of the entire closed-loop governance process. Its output of a high-quality dataset to be governed directly affects the reliability of subsequent cross-source collaborative anomaly detection and the accuracy of matching the governance strategy library. S2. Construct a spatiotemporal semantic association graph based on the physical connections, business logic, and temporal coupling relationships between data sources. The structure of the spatiotemporal semantic association graph is as follows: Figure 2As shown, the specific method for constructing the spatiotemporal semantic association graph is as follows: data records with independent semantic meaning in each data source are used as graph nodes. The attributes of the nodes include data source identifier, data type label, collection timestamp, and physical dimension. In this embodiment, data records with independent semantic meaning refer to logical data record units determined according to data source identifier and data type label, rather than just a data value corresponding to a single collection time. Each logical data record unit corresponds to a graph node and is associated with the time-series data sequence formed by the logical data record unit at consecutive collection times. Therefore, extracting data from the historical time-series window of the same node means extracting historical data before the current time from the time-series data sequence associated with the graph node. The edge establishment method is as follows: the first type of edge is established according to the equipment topology connection relationship. For example, a transformer relationship edge is established between the high-voltage side node and the low-voltage side node of the transformer, and a connection relationship edge is established between the bus node and the feeder node. The second type of edge is established according to the business logic dependency relationship. For example, according to the power balance constraint, a power balance relationship edge is established between the active power of the high-voltage side of the transformer and the sum of the active power of each feeder on the low-voltage side. The constraint expression is: ,in This indicates the active power input to the high-voltage side of the transformer. Indicates the first The active power of the low-voltage side feeder Indicates the low-voltage side feeder number. Indicates the number of low-voltage side feeders. This represents power loss; a third type of edge is established based on the time-series synchronization relationship, which refers to the time correlation between data collected from different data sources at the same time. For example, within the same time collection period, a time-series synchronization edge is established between the electricity consumption data reported by the smart meter and the line current data collected by the SCADA system. Horizontal and vertical reference benchmarks are extracted from the associated nodes and the historical windows of the same node, respectively, to determine the semantic consistency score. The cross-source collaborative anomaly confidence is calculated based on the horizontal deviation, vertical deviation, and semantic consistency score. Based on the spatiotemporal semantic correlation graph, for any data node to be detected, data of the same type is extracted from the associated nodes as a horizontal reference benchmark, and data is extracted from the historical time-series window of the same node as a vertical reference benchmark. The horizontal and vertical reference benchmarks are then fused to calculate the cross-source collaborative anomaly confidence of the data node. The calculation of the cross-source collaborative anomaly confidence and the anomaly type labeling process are as follows: Figure 3 As shown, specifically, the confidence level of cross-source collaborative anomalies is calculated using the following combined algorithm: in, Indicates the first Data nodes at The cross-source collaborative anomaly confidence level at time point [0, 1], with a larger value indicating a higher likelihood of data anomalies. This represents the lateral deviation metric between the lateral reference baseline and the data to be detected. For numerical measurement data, it is calculated by weighting the absolute values ​​of the differences between the data to be detected and the values ​​of each lateral reference baseline. For text-based alarm data and semi-structured configuration data, it is calculated by weighting the differences between the numerical quality values ​​of the data to be detected and the numerical quality values ​​of each lateral reference baseline. The weights are determined based on the type of the associated edges. The lateral deviation normalization constant is set based on the 95th quantile of the lateral deviation under normal conditions in historical data; when the 95th quantile is 0, the preset minimum positive number is determined as... , This represents the longitudinal deviation metric between the longitudinal reference baseline and the data to be tested. For numerical measurement data, it is calculated by weighting the absolute values ​​of the differences between the data to be tested and the historical values ​​in the historical time series window. For text-based alarm data and semi-structured configuration data, it is calculated by weighting the differences between the numerical quality value corresponding to the data to be tested and the numerical quality values ​​in the historical time series window. More recent historical values ​​have greater weights, and an exponential decay weighting is used. The longitudinal deviation normalization constant is set based on the 95th quantile of the longitudinal deviation under normal conditions in historical data; when the 95th quantile is 0, the preset minimum positive number is determined as... , The semantic consistency score between the data to be detected and the associated nodes in the spatiotemporal semantic association graph is calculated, with a value range of [0,1]. For quantifiable business logic constraints, the ratio of the absolute amount by which the data to be detected exceeds the allowable tolerance of the business logic constraint to the preset maximum allowable exceedance is limited to 0 to 1, and is used as the normalized violation amount of the business logic constraint. For non-quantifiable business logic constraints, the normalized violation amount is set to 0 when the constraint is satisfied and set to 1 when the constraint is not satisfied. The arithmetic mean of the normalized violations of each business logic constraint is determined as the comprehensive violation amount, and the result of subtracting the comprehensive violation amount from 1 is used as the score. When the data to be tested meets all business logic constraints, , , , The weighting coefficients are non-negative and satisfy the following conditions: + + =1; when and When any item in the normalization bias is greater than 1, the normalization biases greater than 1 are respectively set to 1, and the processed values ​​are... and Substituting into the above formula, calculate the confidence level of cross-source collaboration anomalies, thereby making the confidence level of cross-source collaboration anomalies... The value range is maintained between 0 and 1. The larger the horizontal and vertical deviations, or the smaller the semantic consistency score, the higher the confidence level of cross-source collaborative anomaly. For data points marked as missing (0), since there are no detectable data values ​​that can be directly used for calculating horizontal and vertical deviations, their cross-source collaborative anomaly confidence level is set to 1, and their anomaly type is marked as transmission anomaly. For data points marked as valid (1), the cross-source collaborative anomaly confidence level is calculated according to the above combination formula. In the process of calculating the cross-source collaborative anomaly confidence level, when only the number of available horizontal reference benchmarks is less than the preset threshold, it is calculated using 0.1 and the current weight coefficient. The smaller value in the middle is used as the first actual adjustment amount, and the weighting coefficient is... Reduce the first actual adjustment amount and the weighting coefficient Increase the same initial actual adjustment amount; when only the historical data completeness rate of the longitudinal reference benchmark is lower than the preset completeness rate threshold, adjust it by 0.1 with the current weighting coefficient. The smaller value in the middle is used as the second actual adjustment amount, and the weighting coefficient is... Reduce the second actual adjustment amount and the weighting coefficient Increase the same second actual adjustment amount; when the number of available horizontal reference benchmarks is less than a preset number threshold, and the historical data integrity rate of the vertical reference benchmark is lower than a preset integrity rate threshold, maintain the weighting coefficient. and Unchanged; the weighting coefficients remain unchanged during the above adjustments. Keep it unchanged, and make the adjusted weighting coefficients , and All three are non-negative and their sum is 1. When the cross-source collaborative anomaly confidence exceeds the preset anomaly detection threshold, the abnormal data is marked, and the dominant anomaly type among source-end anomaly, transmission anomaly, and semantic anomaly is determined based on the confidence contributions of the three. Multi-label annotation is allowed first, and then a governance sequence is matched according to the dominant anomaly type. When a unique classification cannot be achieved, the anomaly type is determined by the deviation dimension with the largest anomaly confidence contribution. The weighted result corresponding to the horizontal normalization deviation is recorded as the horizontal contribution, and the weighted result corresponding to the vertical normalization deviation is recorded as the vertical contribution. This is denoted as semantic contribution. A transmission anomaly is identified when at least one of the following exists: timestamp offset, missing marker, or duplicate reporting, and the vertical contribution reaches the transmission anomaly threshold. A semantic anomaly is also identified when the semantic contribution reaches the semantic anomaly threshold. If the transmission anomaly criteria are not met, but at least one of the horizontal or vertical contributions reaches the corresponding source-end anomaly threshold, a source-end anomaly is identified. When two or more anomaly types are identified simultaneously, the anomaly type with the highest degree of exceedance relative to its threshold is determined as the dominant anomaly type. S2 is the core detection layer of this invention. By establishing three types of edges—topology connection edges, business logic edges, and time-series synchronization edges—it organizes the originally isolated and scattered multi-source heterogeneous data into a graph structure with rich semantic relationships. This graph enables subsequent anomaly detection to move beyond a single data source or a single point in time, providing dual detection capabilities of cross-device horizontal comparison and cross-time vertical tracing. This provides associated node information support for the accurate matching of governance operation sequences in S3. Through a combined algorithm, the detection results from three dimensions—horizontal reference benchmark (comparison of data of the same type during the same period), vertical reference benchmark (comparison of data of the same period during the same period in history), and semantic consistency (verification of business logic constraints)—are weighted and fused to form a comprehensive anomaly confidence score. Compared with traditional single-threshold or single-dimensional detection methods, this algorithm can effectively distinguish between real power grid disturbances and data acquisition failures, helping to reduce false alarms and false negatives. When the reliability of the horizontal or vertical reference benchmark is insufficient due to data loss, it adaptively adjusts... , The weighting coefficients ensure that confidence calculation maintains reasonable detection performance under different data quality conditions, enhancing the robustness of the method in extreme scenarios such as data sparsity or communication interruption. By initially locating the root causes of anomalies (source anomalies / transmission anomalies / semantic anomalies), pre-classification labels are provided for the differentiated matching of governance operation sequences in S3, enabling subsequent governance to be targeted. Different types of anomalies are matched with different combinations of atomic governance operations, rather than using a uniform repair strategy for all anomalies, thereby improving the targeting and repair quality of governance. S3. Based on the abnormal data and the dominant anomaly type, match and execute governance operation sequences from the governance strategy library to replace the abnormal data with corrected data, forming an updated multimodal dataset to be governed. The initial construction method of the governance strategy library is as follows: based on historical governance experience data and expert rules, 3-5 candidate governance operation sequences are pre-set for each anomaly type. When there is a historical governance record for the corresponding anomaly type, normalization is performed according to the historical governance success rate of each candidate governance operation sequence, and the normalization result is set as the initial priority weight of the corresponding candidate governance operation sequence. When there is no historical governance record for the corresponding anomaly type, the initial priority weight of each candidate governance operation sequence is set equally. As the method of this invention runs, the priority weight of each sequence is continuously updated and iterated according to the actual governance effect, realizing the continuous optimization of the governance strategy library. The governance operation sequence includes at least one atomic governance operation. Specifically, atomic governance operations include missing value imputation operation, limit smoothing operation, temporal realignment operation, semantic translation operation, and deduplication operation. Specifically, the missing value imputation operation is as follows: For data points marked as missing, an imputation method based on spatiotemporal correlation is used, prioritizing the use of the weighted average of the horizontal reference benchmark for imputation. If the horizontal reference benchmark is insufficient, historical trend extrapolation of the vertical reference benchmark is used for imputation. The out-of-limit smoothing operation is as follows: For data points exceeding the normal range, a Hampel filter is used to identify outliers and replace them with local medians, with the window width set to three acquisition cycles before and after. The time-series realignment operation is as follows: For data points with time offsets, cubic spline interpolation is used to remap the data points onto the standard time grid to eliminate acquisition time deviations. The semantic translation operation is as follows: For semantically inconsistent data, data conversion is performed based on the semantic constraints of the associated nodes, such as uniformly converting data of different dimensions or standardizing and parsing alarm texts of different encoding formats. The deduplication operation is as follows: For cases where the same data record is repeatedly reported by two or more data sources, the cross-source collaborative anomaly confidence of each duplicate record is calculated, the record with the lowest cross-source collaborative anomaly confidence is retained, and the remaining duplicate records are deleted. The governance operation sequence is a combination of the atomic governance operations. This combination is determined by matching from the governance strategy library based on the dominant anomaly type and the characteristic parameters of the anomaly data. Specifically, the matching rules are as follows: source-end anomalies are preferentially matched with a combination of over-limit smoothing and time-series realignment operations; transmission anomalies are matched based on time offset, missing value marking, and duplicate reporting features, matching at least one of time-series realignment, missing value imputation, and duplicate deduplication operations; semantic anomalies are preferentially matched with semantic translation operations. For example, if a data point is marked as a source-end anomaly and its value deviates from the normal range by more than 40%, the governance operation sequence matched by the governance strategy library is: first, perform over-limit smoothing, expanding the window width to 5 collection cycles before and after, then perform time-series realignment; execute the governance operation sequence to repair the anomaly data and generate corrected data. Simultaneously, calculate the offset influence factor based on the characteristic changes of the data segments before and after governance, the influence of the operation parameters of the governance operation sequence, and the cross-source collaborative anomaly confidence level before governance. Specifically, calculate the quality feature vector for the data segments of the governance object before and after the governance operation. and The quality feature vector includes the mean. ,variance Range and time series change rate The time-series change rate is the arithmetic mean of the absolute values ​​of the differences between adjacent data values ​​within the data segment of the governance object; when the abnormal data is numerical measurement data, the quality feature vector is calculated based on the numerical sequence before and after governance; when the abnormal data is text-based alarm data or semi-structured configuration data, the quality feature vector is calculated based on the numerical quality sequence generated by S1 before governance. After treatment, following the same text quality values ​​or configuration quality value generation rules as in S1, a new numerical quality sequence is generated based on the treated data, and the numerical quality is calculated based on the regenerated numerical quality sequence. For the first eigenvector of quality For each feature, the relative change of the feature is calculated according to the following formula: in, Indicates the first The relative change of a feature. The first element in the quality eigenvector before treatment eigenvalues, The first element in the quality eigenvector after treatment eigenvalues, These correspond to the mean, variance, range, and rate of change over time, respectively. This represents a preset positive number used to prevent the denominator from being zero; the change in features before and after the treatment is calculated based on the feature weights corresponding to each feature. in, This indicates the change in characteristics before and after the treatment. Indicates the first The non-negative feature weights corresponding to each feature, and satisfying the following conditions: ; The first in the governance operation sequence The parameters of each atomic governance operation are normalized to: The influence of the operating parameters is calculated according to the following formula: in, This indicates the impact of the operation parameters on the governance operation sequence. This indicates the number of atomic governance operations contained in the governance operation sequence. Indicates the first Normalization parameters for individual atom governance operations Indicates the first The preset non-negative impact coefficient of each atom governance operation, and satisfies =1; The normalization parameters include the imputation window length for missing value imputation, the smoothing window width for out-of-bounds smoothing, the time correction amount for temporal realignment, the number of field transformations for semantic translation, and the number of duplicate records for deduplication. The ratio of each parameter to its corresponding preset maximum value is limited to between 0 and 1; the offset influence factor is calculated based on the feature changes before and after governance, the influence of operation parameters, and the cross-source collaborative anomaly confidence before governance. in, Indicates the offset influence factor. This indicates the cross-source collaborative anomaly confidence level of the corresponding abnormal data before the governance operation is executed. This represents the distribution coefficient between the characteristic change and the influence of the operating parameters, and 0 ≤ ≤1. S3 is the core execution layer and feedback acquisition layer of this invention. Using the anomaly type labels output by S2 (source anomaly / transmission anomaly / semantic anomaly), it matches differentiated governance operation sequences from the governance strategy library to achieve tailored solutions. Source anomalies emphasize smoothing and realignment, transmission anomalies emphasize realignment and interpolation, and semantic anomalies emphasize semantic translation. Compared to traditional methods that use a uniform repair strategy for all anomalies, this invention's differentiated matching mechanism significantly improves the targeting and quality of governance. Furthermore, the modular design of atomic governance operations allows for dynamic adjustment of operation parameters based on the characteristic parameters of the anomaly data, further enhancing the flexibility of governance. In traditional methods, the detection and repair phases are independent; repaired data may fall on the other side of the detection threshold, introducing new anomalies or causing missed detections. This step introduces an offset influence factor. By comparing the quality feature vectors of the anomaly data segments before and after governance—mean, variance, range, and temporal rate of change—the combination of these four factors can comprehensively characterize the anomaly from four perspectives: central location, overall dispersion, extreme bias, and temporal stationarity. This paper describes the multidimensional impact of governance operations on data distribution, avoiding the shortcomings of single indicators in fully reflecting the disturbances caused by governance operations. It quantifies the degree to which governance operations change the characteristics of data distribution. This factor makes the side effects of governance operations explicit, providing a quantitative basis for the adaptive adjustment of the detection threshold in S4. This transforms detection and governance from a sequential and isolated process into an interactive linkage. The calculation result of the offset influence factor directly drives the dynamic adjustment of the anomaly detection threshold in S4. The larger the offset influence factor, the more drastic the change in data distribution caused by the governance operation, and the more necessary it is to raise the detection threshold to reduce the risk of false alarms. The offset influence factor is the bridge variable of the entire closed-loop feedback link, quantifying the execution effect of the governance layer into an adjustment signal for the detection layer. It is the core hub for realizing the adaptive evolution of the governance strategy. S3 inherits the anomaly labeling and type from S2, realizes data repair through differentiated governance, and feeds back the governance effect to the detection end through the quantitative calculation of the offset influence factor. It is a key link in the closed-loop architecture of detection, governance, and feedback of this invention, directly supporting the adaptive adjustment of the threshold in S4 and the continuous optimization of the strategy library. S4. Update the anomaly detection threshold based on the offset impact factor, record the execution log of the governance operation, and update the priority weight of the governance operation sequence in the governance strategy library based on the governance effect score. Specifically, dynamically adjust the value of the preset anomaly detection threshold based on the offset impact factor to generate an updated anomaly detection threshold, and replace the preset anomaly detection threshold in S2 with the updated anomaly detection threshold for anomaly detection in subsequent data periods. The closed-loop feedback relationship between the governance operation, the anomaly detection threshold adjustment, and the update of the priority weight of the governance strategy library is as follows: Figure 4As shown, specifically, the first threshold is greater than the second threshold. When the offset influence factor is greater than the first threshold, the sum of the current anomaly detection threshold and the preset threshold adjustment step size is used as a candidate upward threshold. A threshold upper limit greater than 0 and less than 1 is preset. When the candidate upward threshold is not greater than the threshold upper limit, the candidate upward threshold is determined as the updated anomaly detection threshold. When the candidate upward threshold is greater than the threshold upper limit, the threshold upper limit is determined as the updated anomaly detection threshold. In this embodiment, the threshold upper limit is set to 0.99. When the offset influence factor is less than the second threshold, the difference between the current anomaly detection threshold and the preset threshold adjustment step size is used as a candidate downward threshold. When the candidate downward threshold is not less than 0, the candidate downward threshold is determined as the updated anomaly detection threshold. When the candidate downward threshold is less than 0, the updated anomaly detection threshold is determined as 0. When the offset impact factor is greater than or equal to the second threshold and less than or equal to the first threshold, the current anomaly detection threshold remains unchanged and is determined as the updated anomaly detection threshold. The updated anomaly detection threshold is used for anomaly detection in the next data cycle. For example, if the offset impact factor calculated after a governance operation is 0.28, which is greater than the first threshold of 0.20, the anomaly detection threshold is increased from the current 0.75 to 0.80 to reduce the sensitivity of subsequent detection cycles and avoid normal data being misjudged as anomalies due to changes in the statistical characteristics of the corrected data. Conversely, if the offset impact factor after a governance operation is 0.05, which is less than the second threshold of 0.08, the threshold is decreased to 0.70 to increase detection sensitivity and ensure that abnormal data is not missed. The system checks and replaces the preset anomaly detection threshold in step S2 with the updated anomaly detection threshold for subsequent data cycles. Simultaneously, it records the execution log of this governance operation and updates the priority weight of the corresponding governance operation sequence in the governance strategy library based on the execution log. Specifically, the execution log includes the governance execution time, governance object identifier, governance operation sequence identifier, comparison of data features before and after governance, and governance effect score. The governance effect score is calculated based on the normalized horizontal bias, normalized vertical bias, and semantic consistency score corresponding to the corrected data, using the following formula: in, This represents the score for governance effectiveness, with a value ranging from 0 to 1; This indicates the normalized lateral bias corresponding to the corrected data. This indicates the normalized longitudinal bias corresponding to the corrected data. This represents the semantic consistency score corresponding to the corrected data. The higher the governance effectiveness score, the higher the degree of matching between the corrected data and the related node data and business logic constraints. The governance strategy library is updated as follows: for each governance operation, the utility value of that governance operation sequence is calculated based on the governance effectiveness score. Utility value = Governance Effectiveness Score × (1 - 0.1 × Normalized Value of Governance Operation Execution Time), where the normalized value of governance operation execution time refers to the ratio of the actual execution time of the governance operation to the preset maximum allowed execution time, and its value range is limited to [0,1]. When the actual execution time exceeds the maximum allowed execution time, the normalized value is set to 1. The utility value is used as a feedback signal to update the priority weight of the governance operation sequence under the corresponding anomaly type according to the weighted moving average method. ,in, For the updated weights, For the previous round of old weights, This represents the utility value calculated in this governance operation sequence. Through multiple iterations, governance operation sequences with relatively high priority are accumulated for handling various anomaly types. When encountering similar anomalies in the future, they can be matched first to improve governance efficiency. The priority weight is used to sort candidate governance operation sequences under the same anomaly type and is not used as a probability value. The sum of the priority weights of each candidate governance operation sequence is not required to be 1. S4 is the closed-loop feedback layer and knowledge accumulation layer of this invention. By comparing the offset influence factor with a preset threshold, it dynamically adjusts or lowers the anomaly detection threshold, achieving online adaptive adjustment of the detection criteria. This mechanism solves the fundamental defect of traditional methods where the detection threshold is fixed once set and cannot adapt to changes in operating conditions. When the governance operation drastically changes the data distribution, the offset influence factor > the first threshold, and the threshold is raised to reduce the false alarm rate. When the governance operation hardly changes the data distribution, the offset influence factor < the second threshold, and the threshold is lowered to improve detection sensitivity. The threshold evolves continuously with the governance feedback, enabling the detection system to adaptively track the dynamic migration of the energy internet's operating status, such as changes in load characteristics, fluctuations in new energy output, and network topology adjustments, ensuring optimal detection performance at different stages. The execution log fully records the entire process information of each governance operation, including execution time, governance object, operation sequence identifier, data feature comparison, and governance effect score, and calculates the utility value based on the governance effect score. The priority weights of each governance operation sequence under the corresponding anomaly type are then updated using a weighted moving average method. As the running cycle increases, the weights of operation sequences with excellent governance effects and high execution efficiency gradually increase, while the weights of sequences with poor effects gradually decrease. This mechanism enables the governance strategy library to have self-iterative optimization capabilities, realizing the quantitative accumulation and reuse of governance experience. When encountering similar anomalies in the future, high-weight operation sequences are prioritized for matching. Governance efficiency and accuracy continue to improve with the running time, rather than relying on fixed expert rules in the long term as in traditional methods. S4 takes over the offset influence factor and execution log output by S3 and injects governance feedback into the detection front end through dynamic threshold adjustment, forming a closed loop. By updating the strategy library weights, governance experience is precipitated into long-term knowledge and continuously evolves. Together, these two constitute the core innovative feature of this invention that distinguishes it from existing technologies. The closed-loop adaptive governance architecture of detection, governance, feedback, and learning enables the entire system to continuously optimize itself during use and update detection criteria and governance strategies with governance feedback. S5. Based on the updated dataset, S2 to S4 are executed iteratively. The loop terminates when the number of anomaly markers for a consecutive preset positive integer number of data periods is lower than the preset anomaly marker count threshold, and a clean dataset is output. The anomaly marker count is the number of data records marked as anomaly by S2 in the current data period. The total data volume is the total number of data records participating in S2 anomaly detection in the current data period. The number of data types is the number of data types participating in S2 anomaly detection in the current data period. The preset anomaly marker count threshold is dynamically set according to the total data volume and the number of data types in the current data period. First, the initial calculated value of the anomaly marker count threshold is calculated according to the following formula: Initial calculated value of anomaly marker count threshold = Total data volume × Baseline ratio × Data type complexity coefficient. Wherein, the baseline ratio is set to 0.5%. The data type complexity coefficient is determined according to the number of data types contained in the multimodal dataset to be treated. When the number of data types is 1 to 3, the data type complexity coefficient is 0.8; when the number of data types is 4 to 6, the data type complexity coefficient is 1.0; when the number of data types is greater than 6, the data type complexity coefficient is 1.2. The initial calculated value of the anomaly marker quantity threshold is rounded up, and the larger value between the rounded up result and 1 is determined as the anomaly marker quantity threshold. S5 is the iteration termination control layer and final output layer of the technical solution of this invention. By setting the number of anomaly markers below an anomaly marker count threshold within a consecutive preset positive integer number of data periods as the iteration termination condition, the governance process has an automatic convergence mechanism. When the data quality is continuously stable at an acceptable level, it actively exits the loop, avoiding the waste of computational resources caused by invalid iterations. This allows the governance process to stop looping when the number of anomaly markers continuously meets the preset governance termination condition, reducing redundant calculations after the preset data quality requirements are met. The anomaly marker count threshold is dynamically set according to the total amount of data and the complexity of the data type. Threshold = Total amount of data × Benchmark The 0.5% multiplier of the data type complexity coefficient enables the convergence criterion to adapt to energy internet application scenarios of varying scales and complexities. This is more universal and reasonable than a fixed threshold, and indirectly compensates for the lack of a clear iteration termination strategy in traditional methods. After multiple rounds of closed-loop iteration involving detection, governance, and feedback, the final clean dataset meets the preset quality standards in dimensions such as the number of anomaly markers, data consistency, and satisfaction of business logic constraints. It can directly serve the upper-layer core applications. When the number of available horizontal reference benchmarks is insufficient or the historical data integrity rate of the vertical reference benchmark is low, the weight coefficient is dynamically adjusted. and This allows the confidence calculation to maintain reasonable detection performance even under sparse data conditions. The adjustment mechanism is the engineering implementation of the weight coefficient definition in step S2, which further enhances the robustness of the method in practical engineering scenarios such as unstable data acquisition and communication interruption. S5, with its triple mechanism of iterative loop, adaptive threshold, and dynamic weight adjustment, ensures the autonomous convergence of the governance process and high-quality data output. It is the final transformation link from technical solution to engineering application value of this invention. The entire governance process begins with the input of raw heterogeneous data in step S1 and ends with the output of clean dataset in step S5, forming a complete technical solution with clear input and output boundaries. The contents not described in detail in this specification are existing technologies known to those skilled in the art. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for managing anomalies in multi-source heterogeneous data in the energy internet, characterized in that, Includes the following steps: S1. Standardize the format, align the timestamps, and mark missing data for multi-source heterogeneous data to form a multimodal dataset to be governed with a unified time benchmark; S2. Construct a spatiotemporal semantic association graph based on the physical connection, business logic, and temporal coupling relationship between data sources. Extract horizontal and vertical reference benchmarks from the associated nodes and the historical windows of the same node, respectively, and determine the semantic consistency score. Calculate the cross-source collaborative anomaly confidence based on the horizontal deviation, vertical deviation, and semantic consistency score. When the confidence score exceeds the preset anomaly detection threshold, mark the abnormal data, and determine the dominant anomaly type among source-end anomaly, transmission anomaly, and semantic anomaly based on the confidence contributions of the three. S3. Match and execute governance operation sequences from the governance strategy library based on the abnormal data and the dominant anomaly type to replace the abnormal data with corrected data, forming an updated multimodal dataset to be governed; calculate the offset influence factor based on the feature changes of the data segments before and after governance, the influence of the operation parameters of the governance operation sequence, and the cross-source collaborative anomaly confidence before governance. S4. Update the anomaly detection threshold based on the offset impact factor, record the execution log of the governance operation, and update the priority weight of the governance operation sequence in the governance strategy library based on the governance effect score. S5. Based on the updated dataset, repeat steps S2 to S4. When the number of anomaly markers for a consecutive preset positive integer number of data periods is lower than the preset anomaly marker count threshold, terminate the loop and output a clean dataset.

2. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The method for constructing the spatiotemporal semantic association graph is as follows: data records with independent semantic meanings in each data source are used as graph nodes. The attributes of the nodes include data source identifier, data type label, collection timestamp, and physical dimension. First-type edges are established based on device topology connection relationships, second-type edges are established based on business logic dependency relationships, and third-type edges are established based on temporal synchronization relationships. The temporal synchronization relationship refers to the temporal association between data collected from different data sources at the same time.

3. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The cross-source collaborative anomaly confidence score is calculated using the following combined algorithm: in, Indicates the first Data nodes at The confidence level of cross-source collaboration anomalies at any given moment. This represents a lateral deviation measure between the lateral reference baseline and the data to be tested. This is the normalized constant for the lateral deviation. This represents a measure of the longitudinal deviation between the longitudinal reference baseline and the data to be tested. This is the normalization constant for longitudinal deviation; when and When any item in the normalization bias is greater than 1, the normalization biases greater than 1 are respectively set to 1, and the processed values ​​are... and Substitute into the above formula to calculate the confidence level of cross-source collaborative anomalies; A semantic consistency score is assigned between the data to be detected and the associated nodes in the spatiotemporal semantic association graph. , , The weighting coefficients are non-negative and satisfy the following conditions: + + =1.

4. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 3, characterized in that: In the calculation of the cross-source collaborative anomaly confidence, when the number of available horizontal reference benchmarks for the data to be detected is less than a preset threshold, the weighting coefficient corresponding to the vertical reference benchmark is increased. And reduce the weighting coefficient corresponding to the lateral reference datum. When the historical data integrity rate of the longitudinal reference benchmark of the data to be tested is lower than the preset integrity rate threshold, the weight coefficient corresponding to the lateral reference benchmark is increased. And reduce the weighting coefficient corresponding to the longitudinal reference benchmark. When both of the above situations occur simultaneously within the same data period, and Maintain the value before the adjustment in this cycle; if only one of the conditions is met... remain unchanged. and Adjust the weights in the opposite direction with the same adjustment amount, and the three weights after adjustment are all non-negative and their sum is 1.

5. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The atomic governance operations include missing value imputation, out-of-bounds smoothing, temporal realignment, semantic translation, and deduplication. The governance operation sequence is a combination of the atomic governance operations, which is determined by matching from the governance strategy library based on the dominant anomaly type and the characteristic parameters of the anomaly data. Specifically, source anomalies are preferentially matched with the combination of out-of-bounds smoothing and temporal realignment. Transmission anomalies are matched with at least one of temporal realignment, missing value imputation, and deduplication based on time offset, missing data markers, and duplicate reporting features. Semantic anomalies are preferentially matched with semantic translation.

6. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The method for calculating the offset impact factor is as follows: calculate the quality feature vectors for the abnormal data segments before and after the governance operation, wherein the quality feature vectors include the mean, variance, range and time series change rate; calculate the offset impact factor based on the feature changes before and after governance, the impact of the operation parameters of the governance operation sequence and the cross-source collaborative anomaly confidence before the governance operation.

7. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The dynamic adjustment method of the preset anomaly detection threshold is as follows: when the offset influence factor is greater than the first threshold, the preset anomaly detection threshold is increased by one step; when the offset influence factor is less than the second threshold, the preset anomaly detection threshold is decreased by one step; the first threshold is greater than the second threshold; when the offset influence factor is greater than or equal to the second threshold and less than or equal to the first threshold, the current anomaly detection threshold remains unchanged.

8. The method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The execution log in S4 includes governance execution time, governance object identifier, governance operation sequence identifier, comparison of data features before and after governance, and governance effect score; the governance effect score is calculated and generated based on the normalized horizontal bias, normalized vertical bias, and semantic consistency score corresponding to the corrected data.

9. A method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 8, characterized in that: The governance strategy library is updated as follows: for each governance operation, the utility value of the governance operation sequence is calculated based on the governance effect score, and the utility value is used as a feedback signal to update the priority weight of the governance operation sequence under the corresponding anomaly type in a weighted moving average manner.

10. A method for managing anomalies in multi-source heterogeneous data in the energy internet according to claim 1, characterized in that: The threshold for the number of anomaly markers preset in S5 is dynamically set based on the total amount of data and the number of data types in the multimodal dataset to be managed.

Images