Power data quality optimization system and method

Abstract

The invention discloses an electric power data quality optimization system and method, and relates to the field of electric power data governance, and the system comprises an edge optimization unit which is used for obtaining edge optimization data; the central optimization unit is used for performing spatio-temporal data completion operation, depth anomaly detection operation and governance tracking operation on the basis of the compressed data so as to obtain central optimization data; the cloud optimization unit is used for carrying out optimization according to the optimized central data and the digital twin simulation data so as to obtain cloud optimization data; and the comprehensive optimization unit is used for carrying out secondary optimization on the edge optimization data, the center optimization data and the cloud optimization data on the basis of the edge optimization data, the optimized center data and the cloud optimization data.

Description

Technical Field

[0001] This specification relates to the field of power data governance. More specifically, the present application relates to a power data quality optimization system and method. Background Art

[0002] In modern power systems, data quality directly affects the operating efficiency of the power grid, intelligent dispatching, and equipment health management. However, limited by factors such as sensor errors, communication interference, and equipment failures, power data often has problems such as missing, noisy, redundant, and abnormal data, affecting the reliability and application value of the data.

[0003] Current power data processing methods mainly rely on manual rules and static algorithms, but have the following deficiencies: 1. The data cleaning method is single and it is difficult to remove complex noise. 2. The data compression strategy is not intelligent and is prone to data loss or redundant storage. 3. The data completion accuracy is limited, ignoring spatio-temporal correlation and affecting the completion quality. 4. The accuracy of anomaly detection is low and it is difficult to identify sudden events or non-linear anomalies. 5. There is a lack of data governance tracking ability and it is difficult to analyze the root cause of data anomalies.

[0004] With the development of smart grids, the demand for high-quality data in applications such as intelligent dispatching, fault diagnosis, new energy access, and power market transactions is increasing continuously.

[0005] Therefore, there is an urgent need for an intelligent and systematic data quality optimization solution that can improve power data quality in aspects such as data cleaning, compression, completion, anomaly detection, and governance tracking, and ensure the safety and stability of power systems. Summary of the Invention

[0006] A series of simplified concepts are introduced in the Summary of the Invention section, which will be further elaborated in the Detailed Description section. The Summary of the Invention section of the present application does not mean to attempt to define the key features and essential technical features of the claimed technical solution, nor does it mean to attempt to determine the protection scope of the claimed technical solution.

[0007] In a first aspect, the present application proposes a power data quality optimization system, including:

[0008] An edge optimization unit, which is used to clean the sensor terminal data according to the upper confidence bound exploration strategy to obtain the cleaned data, and the edge optimization unit is also used to perform data compression based on the cleaned data to obtain edge optimized data;

[0009] A central optimization unit, which is used to perform spatio-temporal data completion operation, deep anomaly detection operation, and governance tracking operation based on the compressed data to obtain central optimized data;

[0010] A cloud optimization unit, which is used to optimize according to the optimized central data and digital twin simulation data to obtain cloud optimized data;

[0011] A comprehensive optimization unit, which is used to perform secondary optimization on the edge optimized data, the central optimized data, and the cloud optimized data based on the edge optimized data, the optimized central data, and the cloud optimized data.

[0012] In a second aspect, the present application proposes a method for optimizing the quality of power data, which is used for the power data quality optimization system in the first aspect, including:

[0013] The above-mentioned edge optimization unit performs a cleaning operation on the sensor terminal data according to the above-mentioned upper confidence bound exploration strategy to obtain the cleaned data;

[0014] The above-mentioned edge optimization unit compresses the data based on the above-mentioned cleaned data to obtain edge optimized data;

[0015] The above-mentioned central optimization unit performs spatio-temporal data completion operation, deep anomaly detection operation, and governance tracking operation on the above-mentioned compressed data to obtain central optimized data;

[0016] The above-mentioned cloud optimization unit optimizes according to the above-mentioned optimized central data and digital twin simulation data to obtain cloud optimized data;

[0017] The above-mentioned comprehensive optimization unit performs secondary optimization on the above-mentioned edge optimized data, the central optimized data, and the cloud optimized data based on the above-mentioned edge optimized data, the above-mentioned optimized central data, and the above-mentioned cloud optimized data.

[0018] In a feasible implementation manner, the above-mentioned edge optimization unit performs a cleaning operation on the sensor terminal data according to the above-mentioned upper confidence bound exploration strategy to obtain the cleaned data, including:

[0019] Construct a cleaning method set A, where the above-mentioned cleaning method set A includes multiple cleaning actions a;

[0020] Define an improved upper confidence bound decision function U(a) corresponding to each cleaning action a, where the above-mentioned improved upper confidence bound decision function includes a quality gain term, an energy consumption cost term, and an improved upper confidence bound term;

[0021] Extract features from the sensor terminal data to obtain a multi-dimensional feature vector F;

[0022] Calculate the adaptability score for the above-mentioned cleaning method set A to obtain a filtered cleaning method set A that meets the adaptability conditions cand ;

[0023] Based on the above-improved upper confidence bound decision function U(a), the above set of screening and cleaning methods A cand is selected to obtain the optimized action a*;

[0024] Through the above edge optimization unit, the above sensor terminal data is cleaned based on the above optimized action a* to obtain the above cleaned data.

[0025] In a feasible implementation manner, the above edge optimization unit performs data compression on the above cleaned data to obtain edge optimized data, including:

[0026] Determine the target compression ratio based on the number of single-sampling data points, the sampling frequency, and the Shannon entropy of the cleaned data;

[0027] Obtain the periodic characteristics of the above cleaned data;

[0028] Determine the adaptive coding strategy according to the above periodic characteristics;

[0029] Through the above edge optimization unit, the cleaned data is compressed according to the above target compression ratio and the above adaptive coding strategy to obtain edge optimized data.

[0030] In a feasible implementation manner, the above central optimization unit includes a data completion module, an anomaly repair module, and a governance tracking module

[0031] The above central optimization unit performs spatio-temporal data completion operations, deep anomaly detection operations, and governance tracking operations on the above compressed data to obtain central optimized data, including:

[0032] Through the above data completion module, spatio-temporal data completion operations are performed on the above compressed data to obtain the completed data;

[0033] Through the above anomaly repair module, deep anomaly detection operations are performed on the completed data to obtain anomaly repaired data;

[0034] Through the above governance tracking module, governance tracking operations are performed on the anomaly repaired data to obtain central optimized data.

[0035] In a feasible implementation manner, the above spatio-temporal data completion operations are performed on the above compressed data through the above data completion module to obtain the completed data, including:

[0036] Construct a spatio-temporal tensor based on the time step, spatial nodes, and feature channels;

[0037] Construct an objective function based on the data fidelity term, spatio-temporal smoothing term, and low-rank constraint term;

[0038] Iteratively calculate the optimization problem formed by the above objective function based on the alternating direction multiplier method until convergence or reaching a preset number of iterations to obtain the above-completed data.

[0039] In a feasible implementation manner, the above-mentioned deep anomaly detection operation is performed on the completed data through the above anomaly repair module to obtain anomaly repair data, including:

[0040] Extract the time feature information, spatial feature information, and frequency domain feature information of the completed data;

[0041] Determine the weight information corresponding to each feature information according to the historical verification set;

[0042] Calculate the data anomaly score according to the above time feature information, the above spatial feature information, the above frequency domain feature information, and the weight information corresponding to each feature information;

[0043] In the case where the above data anomaly score is less than or equal to a preset threshold, use the above-completed data group as the above anomaly repair data;

[0044] In the case where the above data anomaly score is greater than the preset threshold, re-complete the above-completed data.

[0045] In a feasible implementation manner, the above-mentioned governance tracking operation is performed on the anomaly repair data through the above governance tracking module to obtain central optimization data, including:

[0046] Construct an anomaly data stream D based on the above anomaly repair data, where the above anomaly data stream includes the original data, cleaned data, compressed data, repaired data, operation type, processing parameter information, and quality change information of the anomaly repair data;

[0047] Construct the node information of the traceability map based on the above original data, the above cleaned data, the above compressed data, and the above repaired data of the anomaly repair data;

[0048] Construct the edge information of the traceability map according to the above operation type, the above processing parameter information, and the above quality change information of the anomaly repair data;

[0049] Construct the above traceability map according to the above node information and the above edge information to form central optimization data.

[0050] In a feasible implementation manner, the above-mentioned cloud optimization unit optimizes according to the above optimized central data and digital twin simulation data to obtain cloud optimization data, including:

[0051] Determine a correction coefficient based on the above optimized central data, the above digital twin simulation data, and the device health status;

[0052] Optimize the above optimized central data based on the above correction coefficient and the above digital twin simulation data to obtain cloud-optimized data.

[0053] In a feasible implementation manner, the above-mentioned secondary optimization of the above edge optimization data, the above optimized central data, and the above cloud-optimized data by the above comprehensive optimization unit based on the above edge optimization data, the above optimized central data, and the above cloud-optimized data includes:

[0054] Perform dynamic time warping and spatial topology mapping on the above edge optimization data, the above optimized central data, and the above cloud-optimized data to perform secondary optimization on the above edge optimization data, the above central optimization data, and the above cloud-optimized data.

[0055] In summary, traditional power data processing methods often have problems of isolated processing in data cleaning, compression, complementation, etc., and cannot form an overall optimization strategy. This embodiment adopts a multi-level optimization architecture (edge-center-cloud-comprehensive) to improve data integrity, accuracy, and consistency, thereby improving data quality. Traditional methods rely on manual rules for data correction, with insufficient flexibility. This embodiment adopts intelligent algorithms such as the upper confidence bound exploration strategy (UCB), low-rank matrix completion, deep anomaly detection, digital twin optimization, etc. to achieve adaptive data optimization, reduce manual intervention, and improve data governance efficiency. Edge optimization of data reduces the amount of data transmission and the demand for communication bandwidth. At the same time, the central and cloud optimization units further screen and optimize the data, avoiding waste of computing resources and improving computing efficiency. Through the governance tracking module, this embodiment can track the entire life cycle of the data, form a complete traceability map, facilitate anomaly analysis, data recovery, and quality supervision, and enhance the reliability and transparency of the system. The method provided in this embodiment can be applied to multiple scenarios such as smart grids, distribution automation, power transmission and transformation monitoring, load forecasting, etc. It adapts to the data quality requirements of different environments through a multi-level optimization strategy, improving the stability and security of power data. The power data quality optimization system and method proposed in this application, other advantages, objectives, and features of this application will be partially reflected by the following description, and partially will be understood by those skilled in the art through the research and practice of this application. Description of the Drawings

[0056] By reading the detailed description of the preferred implementation manners below, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred implementation manners and are not considered to limit this specification. Moreover, throughout the drawings, the same reference symbols are used to represent the same components. In the drawings:

[0057] Figure 1 It is a structural schematic diagram of a power data quality optimization system provided by an embodiment of the present application;

[0058] Figure 2 It is a flowchart schematic diagram of the power data quality optimization method of the power data quality optimization system provided by an embodiment of the present application. Detailed implementation manners

[0059] Terms such as "first", "second", "third", "fourth", etc. (if any) in the specification, claims and above-mentioned drawings of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily limit to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices. Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments.

[0060] Please refer to Figure 1 , which is a structural schematic diagram of the power data quality optimization system 10 provided by an embodiment of the present application, and specifically may include:

[0061] An edge optimization unit 101, which is used to clean the sensor terminal data according to the upper confidence bound exploration strategy to obtain the cleaned data, and the edge optimization unit is also used to perform data compression based on the cleaned data to obtain edge optimization data;

[0062] A central optimization unit 102, which is used to perform spatio-temporal data completion operation, deep anomaly detection operation and governance tracking operation based on the compressed data to obtain central optimization data;

[0063] A cloud optimization unit 103, which is used to perform optimization according to the optimized central data and digital twin simulation data to obtain cloud optimization data;

[0064] A comprehensive optimization unit 104, which is used to perform secondary optimization on the edge optimization data, the central optimization data and the cloud optimization data based on the edge optimization data, the optimized central data and the cloud optimization data.

[0065] Exemplarily, this embodiment provides a power data quality optimization system (10), which includes an edge optimization unit (101), a central optimization unit (102), a cloud optimization unit (103), and a comprehensive optimization unit (104), which play roles in different stages of power data collection, processing, optimization, and governance respectively. This system can improve the quality, reliability, and traceability of power data through a multi-level optimization strategy, providing high-quality data support for the stable operation and intelligent decision-making of the power system.

[0066] The edge optimization unit (101) is located at the data collection end and mainly performs preliminary optimization on sensor terminal data. The optimization process includes data cleaning and data compression. Data cleaning can use the Upper Confidence Bound (UCB) exploration strategy to clean the sensor data, screen out abnormal and noisy data, and retain high-quality data. After data cleaning, data compression can calculate the target compression ratio based on the periodic characteristics of the data and Shannon Entropy, and use an adaptive coding strategy for compression, thereby reducing data transmission costs, reducing bandwidth occupancy, and improving transmission efficiency.

[0067] The central optimization unit (102) mainly performs deeper optimization on the data optimized by the edge, including spatio-temporal data completion operations, deep anomaly detection operations, and governance tracking operations. The spatio-temporal data completion operation constructs a spatio-temporal tensor model, constructs an objective function based on the data fidelity term, spatio-temporal smoothing term, and low-rank constraint term, and uses the Alternating Direction Method of Multipliers (ADMM) for optimization and solution to complete the missing part of the data and improve data integrity. The deep anomaly detection operation calculates the data anomaly score by combining time, space, and frequency domain feature information, determines the feature weights through the historical validation set, and performs anomaly repair based on a preset threshold. The governance tracking operation constructs an abnormal data stream D, which includes information such as original data, cleaned data, compressed data, and repaired data, and establishes a traceability map to provide traceability analysis.

[0068] The cloud optimization unit (103) further optimizes based on the data optimized by the center and the digital twin simulation data, constructs a state correction coefficient based on the device health, and uses the digital twin simulation model to adjust the optimized data to improve the authenticity and prediction ability of the data.

[0069] The comprehensive optimization unit (104) performs secondary optimization on the edge optimization data, central optimization data, and cloud optimization data. The main means include Dynamic Time Warping (DTW) and spatial topology mapping. Dynamic Time Warping (DTW) improves time consistency by aligning time series data. Spatial topology mapping uses topology learning technology to optimize the structure of data in the spatial dimension and improve the global consistency of data.

[0070] In summary, traditional power data processing methods often have the problem of isolated processing in data cleaning, compression, complementation, etc., and cannot form an overall optimization strategy. This embodiment adopts a multi-level optimization architecture (edge - central - cloud - comprehensive) to improve data integrity, accuracy, and consistency, thereby improving data quality. Traditional methods rely on manual rules for data correction and lack flexibility. This embodiment adopts intelligent algorithms such as the Upper Confidence Bound (UCB) exploration strategy, low-rank matrix completion, deep anomaly detection, digital twin optimization, etc. to achieve adaptive data optimization, reduce manual intervention, and improve data governance efficiency. Edge optimization of data reduces the amount of data transmission and the demand for communication bandwidth. At the same time, the central and cloud optimization units further screen and optimize the data, avoiding waste of computing resources and improving computing efficiency. Through the governance tracking module, this embodiment can track the entire life cycle of data, form a complete traceability map, facilitate anomaly analysis, data recovery, and quality supervision, and enhance the reliability and transparency of the system. The method provided in this embodiment can be applied to multiple scenarios such as smart grids, distribution automation, power transmission and transformation monitoring, load forecasting, etc. By adopting a multi-level optimization strategy, it can adapt to the data quality requirements of different environments and improve the stability and security of power data.

[0071] In a second aspect, the present application proposes a power data quality optimization method for the power data quality optimization system in the first aspect, including:

[0072] S210. The edge optimization unit cleans the sensor terminal data according to the Upper Confidence Bound (UCB) exploration strategy to obtain the cleaned data;

[0073] Exemplarily, the edge optimization unit (101) is located at the data acquisition end and mainly performs preliminary optimization on the sensor terminal data. The optimization process includes data cleaning and data compression. Data cleaning can use the Upper Confidence Bound (UCB) exploration strategy to clean the sensor data, screen out abnormal and noisy data, and retain high-quality data.

[0074] S220. The edge optimization unit compresses the data based on the cleaned data to obtain edge optimization data;

[0075] Exemplarily, after data cleaning, data compression can calculate the target compression ratio based on the periodic characteristics of the data and Shannon Entropy, and adopt an adaptive coding strategy for compression, thereby reducing data transmission costs, reducing bandwidth occupancy, and improving transmission efficiency.

[0076] S230. Through the above-mentioned central optimization unit, perform spatio-temporal data completion operations, deep anomaly detection operations, and governance tracking operations on the above-mentioned compressed data to obtain central optimization data;

[0077] Exemplarily, the central optimization unit (102) mainly performs deeper optimization on the data optimized by the edge, including spatio-temporal data completion operations, deep anomaly detection operations, and governance tracking operations. The spatio-temporal data completion operation constructs a spatio-temporal tensor model, constructs an objective function based on the data fidelity term, spatio-temporal smoothing term, and low-rank constraint term, and uses the Alternating Direction Method of Multipliers (ADMM) for optimization and solution to complete the missing part of the data and improve data integrity. The deep anomaly detection operation calculates the data anomaly score by combining time, space, and frequency domain feature information, determines the feature weights through the historical validation set, and performs anomaly repair based on a preset threshold. The governance tracking operation constructs an abnormal data stream D, including information such as raw data, cleaned data, compressed data, and repaired data, and establishes a traceability map to provide traceability analysis.

[0078] S240. Through the above-mentioned cloud optimization unit, optimize according to the above-mentioned optimized central data and digital twin simulation data to obtain cloud optimization data;

[0079] Exemplarily, the cloud optimization unit (103) further optimizes based on the data optimized by the center and the digital twin simulation data, constructs a state correction coefficient based on the device health, and uses the digital twin simulation model to adjust the optimized data to improve the authenticity and prediction ability of the data.

[0080] S250. According to the above-mentioned comprehensive optimization unit, perform secondary optimization on the above-mentioned edge optimization data, the above-mentioned optimized central data, and the above-mentioned cloud optimization data based on the above-mentioned edge optimization data, the above-mentioned optimized central data, and the above-mentioned cloud optimization data.

[0081] The comprehensive optimization unit (104) performs secondary optimization on the edge optimization data, central optimization data, and cloud optimization data. The main means include Dynamic Time Warping (DTW) and spatial topology mapping. Dynamic Time Warping (DTW) improves time consistency by aligning time series data. Spatial topology mapping uses topology learning technology to optimize the structure of data in the spatial dimension and improve the global consistency of the data.

[0082] In summary, traditional power data processing methods often have problems of isolated processing in data cleaning, compression, filling, etc., and cannot form an overall optimization strategy. This embodiment adopts a multi-level optimization architecture (edge-center-cloud-integration) to improve data integrity, accuracy, and consistency, thereby improving data quality. Traditional methods rely on manual rules for data correction, with insufficient flexibility. This embodiment adopts intelligent algorithms such as the upper confidence bound exploration strategy (UCB), low-rank matrix completion, deep anomaly detection, and digital twin optimization to achieve adaptive data optimization, reduce manual intervention, and improve data governance efficiency. Edge optimization of data reduces the amount of data transmission and the demand for communication bandwidth. At the same time, the central and cloud optimization units further screen and optimize the data, avoiding waste of computing resources and improving computing efficiency. Through the governance tracking module, this embodiment can track the entire life cycle of data, form a complete traceability map, facilitate anomaly analysis, data recovery, and quality supervision, and enhance the reliability and transparency of the system. The method provided by this embodiment can be applied to multiple scenarios such as smart grids, distribution automation, power transmission and transformation monitoring, and load forecasting, and can adapt to the data quality requirements of different environments through a multi-level optimization strategy, improving the stability and security of power data.

[0083] In a feasible implementation manner, the above-mentioned edge optimization unit performs a cleaning operation on the sensor terminal data according to the above-mentioned upper confidence bound exploration strategy to obtain the cleaned data, including:

[0084] Construct a cleaning method set A, where the cleaning method set A includes multiple cleaning actions a;

[0085] Define an improved upper confidence bound decision function U(a) corresponding to each cleaning action a, where the improved upper confidence bound decision function includes a quality benefit term, an energy consumption cost term, and an improved upper confidence bound term;

[0086] Extract features from the sensor terminal data to obtain a multi-dimensional feature vector F;

[0087] Calculate the adaptability score for the above-mentioned cleaning method set A to obtain a screened cleaning method set A that meets the adaptability conditions cand ;

[0088] Based on the above-mentioned improved upper confidence bound decision function U(a), perform selection on the above-mentioned screened cleaning method set A cand to obtain an optimized action a*;

[0089] Through the above-mentioned edge optimization unit, perform a cleaning operation on the above-mentioned sensor terminal data based on the above-mentioned optimized action a* to obtain the above-mentioned cleaned data.

[0090] Exemplarily define the cleaning method set A, where:

[0091] A = {a1, a2, …, a k}

[0092] a1, a2, …, a k represent the elements of the set of optional cleaning methods, such as: Kalman filter (suitable for dynamic noise), median filter (suitable for impulse noise), linear interpolation (suitable for data loss), and wavelet denoising (suitable for high-frequency noise). The utility of each cleaning action a is jointly determined by the data quality improvement ΔQ and the energy consumption cost E.

[0093] This embodiment adopts a dual-objective optimization UCB mechanism to balance data quality improvement and computational resource consumption and optimize the data cleaning process. The goal of the cleaning strategy is to select the optimal cleaning method a * :

[0094]

[0095] where: The first term is the quality benefit term, which is used to measure the average data quality improvement degree of the cleaning method a in historical calls. The second term is the energy consumption cost term, which is used to measure the computational resource consumption of the cleaning method a and is controlled by the penalty coefficient λ. The third term is the improved UCB exploration term, which is used to dynamically balance exploration and exploitation. It is used to encourage the use of cleaning methods with fewer calls and avoid premature convergence to local optima. It is used to combine energy consumption stability and avoid selecting cleaning methods with large computational fluctuations.

[0096] The data quality improvement degree ΔQ is used to measure the data quality improvement after the i-th use of the action a and can be calculated based on the following formula:

[0097]

[0098] where: X raw is the original data, and X clean is the data after cleaning. The energy consumption E a can measure the computational load through a power monitoring chip and reflect the computational resource consumption of the cleaning method. The historical call count n a records the historical usage frequency of the cleaning method a. The energy consumption penalty coefficient λ is used to control the trade-off between quality improvement and energy consumption (it can take values from 0.3 to 0.77). The exploration intensity coefficient ∈ allows for dynamic adjustment (the initial value can be 1.0). The energy consumption stability parameter σ E / μ E is used to measure the energy consumption stability of the cleaning method and avoid selecting methods with large computational overhead fluctuations.

[0099] Extract features from the sensor terminal data to obtain a multi-dimensional feature vector F = [f time , f freq , f topo . The multi-dimensional feature vector F includes time-domain features, frequency-domain features, and topological features. Time-domain feature extraction may include the mean (μ x ), variance , kurtosis, skewness, etc. Frequency-domain features can be calculated by the Fast Fourier Transform (FFT) for the main frequency components of the signal FFT(x(t)): Topological features can be calculated using the PageRank algorithm to measure the influence of data in the network.

[0100] Calculate the applicability score of cleaning method a:

[0101] Applicability(a) = Sigmoid(w T F + b)

[0102] where F is the feature vector, containing data features such as time-domain, frequency-domain, and topological information. w is the weight vector, representing the degree of influence of different features on applicability. b is the bias term, used to adjust the reference value of the calculation result. w T F is the dot product of the feature vector and the weight vector, calculating the correlation between data features and the cleaning method. Sigmoid function: This function is used to normalize the applicability score so that its value is between (0, 1), facilitating comparison and screening.

[0103] Screen the set of candidate cleaning methods:

[0104]

[0105] This formula is used to screen out the set of candidate methods A from all cleaning methods cand Only retain the methods with higher applicability. τ is the threshold, calculate the highest applicability score among all methods, and take 60% of it as the screening threshold.

[0106] Output the set of candidate cleaning methods According to the set of candidate cleaning methods A cand Improve the UCB objective function and select the optimal cleaning method based on the UCB formula: Thus, determine the optimal cleaning method a * , and execute a step by step * Clean the sensor data to remove noise and outliers, thereby obtaining the cleaned data.

[0107] In a feasible implementation, the above-mentioned edge optimization unit performs data compression on the above-mentioned cleaned data to obtain edge-optimized data, including:

[0108] Determine the target compression ratio based on the number of data points per single sampling, the sampling frequency, and the Shannon entropy of the data after cleaning;

[0109] Obtain the periodic characteristics of the data after the above cleaning;

[0110] Determine the adaptive coding strategy according to the above periodic characteristics;

[0111] The edge optimization unit compresses the data after cleaning according to the above target compression ratio and the above adaptive coding strategy to obtain edge-optimized data.

[0112] Exemplarily, calculate a reasonable compression ratio to ensure that while reducing the data storage requirement, key information is retained as much as possible. The compression ratio can be calculated by the following formula:

[0113]

[0114] where N * is the number of data points per single sampling (for example, a power sensor samples 1024 points per cycle). f * is the sampling frequency (for example, 10 kHz). H Shan (X) is the Shannon entropy of the data after cleaning, which is used to reflect the information redundancy.

[0115] Use the pattern recognition method to calculate the periodic characteristics of the data:

[0116]

[0117] where the shift function is used to shift the signal X clean . Select the time lag τ corresponding to the minimum error as the periodic characteristic.

[0118] The entropy value can be calculated by the following formula:

[0119]

[0120] This formula estimates the data distribution based on the histogram and calculates the information entropy of the data (i.e., the uncertainty of the data). As shown in Table 1, in the case of data with high periodicity, differential coding and run-length coding can be used for efficient compression. In the case of burst noise data, more complex LZ77 compression or Huffman coding is required. In the case of mixed-mode data, wavelet transform and quantization compression can be used.

[0121]

[0122] Table 1

[0123] According to the target compression ratio, select an appropriate encoding strategy and compress the data using an adaptive compression strategy (such as differential encoding, wavelet transform). Calculate the Shannon entropy of the optimized data to ensure that the information loss is within an acceptable range.

[0124] In a feasible implementation, the above-mentioned central optimization unit includes a data completion module, an anomaly repair module, and a governance tracking module

[0125] The above-mentioned central optimization unit performs spatio-temporal data completion operations, deep anomaly detection operations, and governance tracking operations based on the above-mentioned compressed data to obtain central optimized data, including:

[0126] The above-mentioned data completion module performs spatio-temporal data completion operations based on the above-mentioned compressed data to obtain completed data;

[0127] The above-mentioned anomaly repair module performs deep anomaly detection operations based on the completed data to obtain anomaly repair data;

[0128] The above-mentioned governance tracking module performs governance tracking operations based on the anomaly repair data to obtain central optimized data.

[0129] In a feasible implementation, the above-mentioned data completion module performs spatio-temporal data completion operations based on the above-mentioned compressed data to obtain completed data, including:

[0130] Construct a spatio-temporal tensor based on the time step, spatial node, and feature channel;

[0131] Construct an objective function based on the data fidelity term, spatio-temporal smoothness term, and low-rank constraint term;

[0132] Based on the alternating direction multiplier method, perform iterative calculations on the optimization problem formed by the above-mentioned objective function until convergence or a preset number of iterations is reached to obtain the above-mentioned completed data.

[0133] Exemplarily, represent the compressed data as a spatio-temporal tensor: Z ∈ R T′×S×C , where T′ is the time step, S is the spatial node (such as different sensors), and C is the feature channel (such as voltage, current, power, etc.)

[0134] Establish an optimization objective function, and the optimization problem is as follows:

[0135]

[0136] Among them, is the data fidelity term, which is used to constrain that the completed data Z at the known position Ω should be consistent with the observed data X to ensure the accuracy of data completion. P Ω is the projection operator, X is the observed data, and Z is the data to be completed, is the Frobenius norm.

[0137] The spatio-temporal smoothing term (total variation regularization, TV) is calculated based on the following formula:

[0138]

[0139] The spatio-temporal smoothing term is used to avoid drastic changes in adjacent spatio-temporal points and ensure the local smoothness of the data. The first term of Z t,s represents the completed data value at time t and spatial position s. This term represents the difference between adjacent time steps, aiming to avoid drastic fluctuations in the data in the time dimension and make the data smoother. The second term This term represents the difference between adjacent spatial positions, aiming to make the adjacent sensor data have a certain smoothness in the spatial dimension and avoid sudden changes.

[0140] The low-rank constraint term is calculated based on the following formula (Tucker decomposition):

[0141]

[0142] Among them, the low-rank constraint term is used to force the completed data to maintain a low-rank structure, utilize the global spatio-temporal correlation, and improve the completion quality. Z (k) is the unfolding matrix of the k-th mode (pattern), representing the unfolding structure of the data in different dimensions. γ k is the regularization coefficient, controlling the low-rank constraint strength on each mode. k = 1, 2, 3: k = 1 represents the low-rank constraint in the time dimension (temporal pattern). k = 2 represents the low-rank constraint in the spatial dimension (sensor topology pattern). k = 3 represents the low-rank constraint in the feature channel (multivariate feature pattern).

[0143] The alternating direction method of multipliers (ADMM) is used for solving. Specifically:

[0144] 1. Update the completed data:

[0145]

[0146] Among them, Z k+1 is the completed data matrix at the (k + 1)-th iteration, representing the current updated data estimate. P Ω is the projection operator, representing the position of the known data, that is, the completed data Z must be consistent with the observed data X at the known data points Ω. X is the original observed data (including missing values). ρ is the penalty coefficient, used to control the balance of the optimization objective and ensure the smoothness and low-rankness of the data. Y k is the auxiliary variable matrix, used to decompose the optimization problem in the ADMM framework. Λ kis the Lagrange multiplier matrix, which controls the update of the constraint term to ensure the consistency of the constraints between the completed data and the original data.

[0147] 2. Projective correction of TV and Tucker low-rank terms:

[0148]

[0149] where Y k+1 is the optimized data matrix, which is the data after TV regularization and Tucker low-rank constraint. Prox is the proximal operator, which is used to handle non-smooth regularization terms and perform TV smoothing and low-rank projection operations. λ1 is the TV regularization parameter, which controls the spatio-temporal smoothness and ensures that the data changes smoothly at adjacent time steps and spatial positions. λ2 is the Tucker low-rank constraint parameter, which controls the global structure of the data, making the completed data maintain the low-rank property and improving the completion accuracy. ||·|| Tucker is the low-rank constraint term of Tucker decomposition, representing the low-rank property of the data matrix in different dimensions.

[0150] 3. Update of Lagrange multipliers:

[0151] Λ k+1 = Λ k + Z k+1 - Y k+1

[0152] where Λ k+1 is the updated Lagrange multiplier matrix, which is used to adjust the constraints during the optimization process. Z k+1 - Y k+1 represents the error between the completed data and the proximal projection data in the current iteration. The smaller the error, the closer the optimization is to convergence.

[0153] Calculate the relative error:

[0154]

[0155] where ||Z k+1 - Z k || F is the change amount (Frobenius norm) between the completed data in the current step and the previous step. ||Z k || F is the normalization denominator, ensuring that the convergence determination is not affected by the data scale. 10 -4 is the set convergence threshold. When the relative error is less than 10 -4 , it is considered that the optimization process has converged. If the threshold is reached or the number of iterations reaches the upper limit, the calculation is stopped.

[0156] In a feasible implementation manner, the above-mentioned deep anomaly detection operation is performed on the complemented data by the above-mentioned anomaly repair module to obtain anomaly repair data, including:

[0157] Extract the time feature information, spatial feature information, and frequency domain feature information of the complemented data;

[0158] Determine the weight information corresponding to each feature information according to the historical verification set;

[0159] Calculate the data anomaly score according to the above-mentioned time feature information, the above-mentioned spatial feature information, the above-mentioned frequency domain feature information, and the weight information corresponding to each feature information;

[0160] In the case where the above data anomaly score is less than or equal to the preset threshold, the above complemented data group is used as the above anomaly repair data;

[0161] In the case where the above data anomaly score is greater than the preset threshold, the above complemented data is complemented again.

[0162] Exemplarily, after obtaining the preliminarily complemented data, start performing multi-angle deep anomaly detection on it, and extract the time feature information, spatial feature information, and frequency domain feature information.

[0163] The time feature information may include the rate of temperature change (differential feature) between adjacent time periods of each sensor, periodic features (24-hour temperature curve of a day, weekday / weekend pattern, etc.), and historical comparison features (comparison with the average temperature at the same time period of the previous few days or last week).

[0164] The spatial feature information may include the temperature difference or correlation coefficient between adjacent sensors at the same moment. Sensors on the same floor or the same production line usually have similar temperature change trends. If the difference between a certain sensor and most nearby sensors suddenly increases significantly, it may be abnormal.

[0165] The frequency domain feature information may include using Fourier transform (FFT) or wavelet transform to extract the spectral features in the data, and judge whether there are fixed frequency components in the temperature data (such as the features brought by the periodic fluctuations of the equipment). Observe the energy distribution in different frequency bands to distinguish between regular periodic fluctuations and sudden noise spikes.

[0166] The historical verification set refers to temperature data that has been confirmed as normal in the past or has been successfully repaired and verified. Through model training or statistical analysis, different importance levels (weights) can be assigned to the above-mentioned time, space, and frequency domain features. The time feature information, space feature information, frequency domain feature information, and the weight information of each feature are weighted and fused to calculate the data anomaly score. If the data anomaly score is less than or equal to the preset threshold, it indicates that the completed data basically meets the rationality requirements and is considered "normal" or "acceptable repair result". At this time, this batch of completed data will be directly used as "anomaly repair data", that is, the final output or stored result.

[0167] If the data anomaly score is greater than the preset threshold, it indicates that the current completion result still significantly deviates from the reasonable range of historical or surrounding sensors, and there may still be large anomalies. At this time, it is necessary to repair or complete the data again.

[0168] In a feasible implementation, the above-mentioned governance tracking module performs governance tracking operations based on the anomaly repair data to obtain the central optimization data, including:

[0169] Construct an anomaly data stream D based on the above-mentioned anomaly repair data. Among them, the above-mentioned anomaly data stream includes the original data, cleaned data, compressed data, repaired data, operation type, processing parameter information, and quality change information of the anomaly repair data;

[0170] Construct the node information of the traceability map based on the above-mentioned original data, above-mentioned cleaned data, above-mentioned compressed data, and above-mentioned repaired data of the above-mentioned anomaly repair data;

[0171] Construct the edge information of the traceability map according to the above-mentioned operation type, above-mentioned processing parameter information, and above-mentioned quality change information of the above-mentioned anomaly repair data;

[0172] Construct the above-mentioned traceability map according to the above-mentioned node information and above-mentioned edge information to form the central optimization data.

[0173] Exemplarily, construct the nodes in the map according to the four main data forms (original data, cleaned data, compressed data, repaired data) in the anomaly repair data. In the figure, it can be marked as: the original data node v raw 、the cleaned data node v clean 、the compressed data node v compressed and the repaired data node v repaired .

[0174] Construct the edges in the map according to the operation type, processing parameter information, and quality change information in the anomaly repair data. In the traceability map, each operation can be regarded as a transformation edge from the data in the previous stage to the data in the next stage.

[0175] Edge attributes can include operation types. For example, an edge from v raw to v clean can be labeled as "Clean", an edge from v clean to v compressed can be labeled as "Compress", and an edge from v compressed to v repaired can be labeled as "Repair".

[0176] Edge attributes can also include processing parameter information. Parameters corresponding to specific operation types, such as: interpolation algorithms used in the cleaning operation (linear interpolation, polynomial interpolation, deep learning-based prediction, etc.). Compression ratio, block size, retention accuracy, etc. in the compression operation. Anomaly detection model type, threshold setting, correction strategy, etc. in the repair operation

[0177] Edge attributes can also include quality change information. The impact of different operations on data quality can be recorded as the amount of quality change. For example: the cleaning operation reduces the noise rate and the missing rate. Changes in distortion and storage occupancy rate brought about by the compression operation. Improvements in accuracy or consistency by the repair operation, etc.

[0178] In the graph, it can also be quantified as the edge weight w ij , as shown in the formula:

[0179]

[0180] Characterizes the ratio of quality gain to energy consumption cost, used to measure the comprehensive value of cost performance of different operations.

[0181] The traceability graph is not fixed once constructed, but often incrementally updated over time. For example, when at a subsequent moment, new suspected anomalies are found and repaired for data that has already been compressed, a new subgraph will be generated, which is incorporated into the original traceability graph to form Gt+1. This process facilitates the continuous improvement of data governance tracking information and realizes more efficient full-life cycle management.

[0182] After the construction of nodes and edges is completed, a "traceability map" is obtained, which fully records the whole process of how the original data evolves into the cleaned data, then to the compressed data, and finally to the repaired data. All processing steps, operation parameters, quality benefits, etc. are clearly visible in the map. If it is finally analyzed that the result is abnormal, it is possible to trace back along the traceability map to find out which processing parameter or algorithm in which step caused the deviation. By comparing the quality changes and energy consumption of different operation edges, identify in which stages more optimal algorithms or parameter adjustments can be made, so as to minimize the computing and storage overhead while ensuring data quality. Through the construction and update of the above traceability map and its meta-information, we obtain the core data basis for center optimization, that is, center optimization data. This data not only includes the final repair result itself, but also contains the detailed evolution process of each stage, providing a decision-making basis for subsequent refined management, data quality improvement, and resource cost optimization.

[0183] In a feasible implementation manner, the above-mentioned cloud optimization unit optimizes according to the above-mentioned optimized center data and digital twin simulation data to obtain cloud optimization data, including:

[0184] Determine a correction coefficient according to the above-mentioned optimized center data, the above-mentioned digital twin simulation data, and the device health status;

[0185] Optimize the above-mentioned optimized center data based on the above-mentioned correction coefficient and the above-mentioned digital twin simulation data to obtain cloud optimization data.

[0186] Exemplarily, the device health status (Health Index) can be derived from the fusion of multiple indicators, such as: the usage duration of device components, maintenance records, failure rates. Vibration, temperature, current anomalies detected by sensors. Numerical values calculated by mechanism models for device wear and fatigue. It can be represented by a score within an interval (such as 0, 10 or 0, 100), where the lower the value, the less healthy the device condition and the closer it is to failure.

[0187] When the device health status is good, its actual performance is relatively close to the "ideal performance" preset by the digital twin simulation model; when the device health status is poor, the actual performance often deviates significantly from the ideal model, and greater corrections need to be made to the simulation results. The result of the twin model is adjusted from the "ideal value" to a value closer to reality through the correction coefficient γ.

[0188] The device health status is φ health ∈[0, 1]; 1 indicates excellent health status, and 0 indicates extremely poor or close to scrapping. γ = 1 - k·(1 - φ health ), where k is an adjustment factor (such as when k = 1, γ = φ health )

[0189] Based on the correction coefficient and digital twin simulation data, the optimized central data is further optimized. The digital twin model can perform real-time or offline simulations in the cloud and output a set of simulation parameters / predicted values (such as equipment energy consumption, production output, failure rate prediction, etc.) according to external environmental changes and historical operation data. Under ideal conditions (no wear on the equipment and no attenuation of performance standards), these twin simulation data can often give the theoretical optimal operating point or optimal scheduling plan of the system.

[0190] According to γ obtained in the previous step, the output result of the twin model is adjusted accordingly. Exemplarily:

[0191] AdjustedTwin(p i ) = p i ×γ

[0192] p i refers to a certain parameter given by the twin model (such as theoretical production capacity, energy consumption index, recommended rotation speed, etc.). AdjustedTwin(p i ) is the corrected result considering the equipment health. In a more complex scenario, the correction coefficient can also be set separately according to the health of different equipment to correct the twin simulation data in multiple dimensions. The missing actual values can be optimized according to the corrected twin model.

[0193] In a feasible implementation manner, the above-mentioned secondary optimization of the edge optimization data, the central optimization data, and the cloud optimization data by the above-mentioned comprehensive optimization unit based on the above-mentioned edge optimization data, the above-mentioned optimized central data, and the above-mentioned cloud optimization data includes:

[0194] Performing dynamic time warping and spatial topology mapping on the edge optimization data, the optimized central data, and the cloud optimization data to perform secondary optimization on the edge optimization data, the central optimization data, and the cloud optimization data.

[0195] Exemplarily, since the edge optimization data, the central optimization data, and the cloud optimization data may have different sampling times or time offsets due to external interference, dynamic time warping (DTW) is used for time alignment to ensure that the time scales of data in different stages are consistent.

[0196]

[0197] Where: X and Y respectively represent the time series of data from different sources. d(X i ,Y j) Represents the distance metric between data points (such as Euclidean distance). Through dynamic programming, find the optimal time alignment path to align data from different sources on the time axis.

[0198] Since the data may come from different sensors, regions or devices, there may be problems with topological inconsistencies in their spatial distribution. Use spatial topology mapping to adjust the spatial relationship of the data.

[0199] Suppose the data comes from multiple sensor nodes, and define the sensor network topology:

[0200] G = (V, E)

[0201] Where: V is the set of sensors, and each sensor is a vertex. E is the connection relationship between sensors, defining the similarity between sensors (such as Euclidean distance, correlation, etc.).

[0202] Adopt a method based on Pearson correlation coefficient to calculate the similarity of adjacent sensor data:

[0203]

[0204] S i,j Represents the similarity between sensors i and j.

[0205] Calculate important nodes based on PageRank to ensure that the weights of key sensor data are higher:

[0206]

[0207] Topology learning: If the data correlation between sensors is high, then enhance their weights. If there are deviations in the data of some sensors, then adjust their data structure to make it more conform to the topology of the global network.

[0208] Combine edge-optimized data, center-optimized data, and cloud-optimized data to ensure that the final data is optimal in terms of time, space, and global dimensions.

[0209] Let X edge , X center , X cloud Represent three data sources respectively, and calculate the weighted average:

[0210] X final = w1 × X edge + w2 × X center + w3 × X cloud

[0211] Weight assignment: If the edge optimization data has high accuracy, assign a higher weight w1. If the center optimization data is repaired through anomaly detection and the data quality is improved, increase w2. If the cloud optimization data is optimized based on simulation, adjust w3 according to its credibility.

[0212] Optimize data fusion through the least squares error (LeastSquares): min∑ i (X final,i -X original,i ) 2 , to ensure that the final fused data is closest to the real data.

[0213] The above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An electric power data quality optimization system, characterized in that, Including: An edge optimization unit, which is used to clean the sensor terminal data according to the upper confidence bound exploration strategy to obtain the cleaned data, and the edge optimization unit is also used to perform data compression based on the cleaned data to obtain edge optimized data; A central optimization unit, which is used to perform spatio-temporal data completion operation, deep anomaly detection operation and governance tracking operation based on the compressed data to obtain central optimized data; A cloud optimization unit, which is used to optimize according to the optimized central data and digital twin simulation data to obtain cloud optimized data; A comprehensive optimization unit, which is used to perform secondary optimization on the edge optimized data, the central optimized data and the cloud optimized data based on the edge optimized data, the optimized central data and the cloud optimized data.

2. A power data quality optimization method, used for the power data quality optimization system described in claim 1, characterized in that, Including: The edge optimization unit cleans the sensor terminal data according to the upper confidence bound exploration strategy to obtain the cleaned data; The edge optimization unit performs data compression based on the cleaned data to obtain edge optimized data; The central optimization unit performs spatio-temporal data completion operation, deep anomaly detection operation and governance tracking operation based on the compressed data to obtain central optimized data; The cloud optimization unit optimizes according to the optimized central data and digital twin simulation data to obtain cloud optimized data; The comprehensive optimization unit performs secondary optimization on the edge optimized data, the central optimized data and the cloud optimized data based on the edge optimized data, the optimized central data and the cloud optimized data.

3. The power data quality optimization method according to claim 2, wherein The step of the edge optimization unit cleaning the sensor terminal data according to the upper confidence bound exploration strategy to obtain the cleaned data includes: Construct a cleaning method set A, where the cleaning method set A includes multiple cleaning actions a; Define an improved upper confidence bound decision function U(a) corresponding to each cleaning action a, where the improved upper confidence bound decision function includes a quality gain term, an energy consumption cost term and an improved upper confidence bound term; Extract features from the sensor terminal data to obtain a multi-dimensional feature vector F; Calculate the adaptability score for the set A of cleaning methods to obtain the filtered set A of cleaning methods that meet the adaptability conditions cand ; Based on the improved upper confidence bound decision function U(a), select from the set of screening and cleaning methods A cand to obtain the optimized action a*; The edge optimization unit performs a cleaning operation on the sensor terminal data based on the optimization action a* to obtain the cleaned data.

4. The power data quality optimization method according to claim 2, characterized in that The step of the edge optimization unit performing data compression based on the cleaned data to obtain edge optimized data includes: Determine the target compression ratio based on the number of single-sampling data points, the sampling frequency and the Shannon entropy of the cleaned data; Obtain the periodic characteristics of the cleaned data; Determine an adaptive coding strategy according to the periodic characteristics; The edge optimization unit performs data compression on the cleaned data according to the target compression ratio and the adaptive coding strategy to obtain edge optimized data.

5. The power data quality optimization method according to claim 2, wherein, The central optimization unit includes a data completion module, an anomaly repair module and a governance tracking module The above-mentioned central optimization unit performs spatio-temporal data completion operations, deep anomaly detection operations, and governance tracking operations based on the compressed data to obtain central optimization data, including: The data completion module performs spatio-temporal data completion operations based on the compressed data to obtain the completed data; The anomaly repair module performs deep anomaly detection operations based on the completed data to obtain anomaly repair data; The governance tracking module performs governance tracking operations based on the anomaly repair data to obtain central optimization data.

6. The power data quality optimization method according to claim 5, characterized in that The above-mentioned data completion module performs spatio-temporal data completion operations based on the compressed data to obtain the completed data, including: Construct a spatio-temporal tensor based on the time step, spatial node, and feature channel; Construct an objective function based on the data fidelity term, spatio-temporal smoothing term, and low-rank constraint term; Perform iterative calculations on the optimization problem formed by the objective function based on the alternating direction multiplier method until convergence or a preset number of iterations is reached to obtain the completed data.

7. The power data quality optimization method according to claim 5, characterized in that The above-mentioned anomaly repair module performs deep anomaly detection operations based on the completed data to obtain anomaly repair data, including: Extract the time feature information, spatial feature information, and frequency domain feature information of the completed data; Determine the weight information corresponding to each feature information according to the historical verification set; Calculate the data anomaly score according to the time feature information, the spatial feature information, the frequency domain feature information, and the weight information corresponding to each feature information; In the case where the data anomaly score is less than or equal to the preset threshold, use the completed data group as the anomaly repair data; In the case where the data anomaly score is greater than the preset threshold, complete the completed data again.

8. The power data quality optimization method according to claim 5, wherein, The above-mentioned governance tracking module performs governance tracking operations based on the anomaly repair data to obtain central optimization data, including: Construct an anomaly data stream D based on the anomaly repair data, where the anomaly data stream includes the original data, cleaned data, compressed data, repaired data, operation type, processing parameter information, and quality change information of the anomaly repair data; Construct the node information of the traceability map based on the original data, the cleaned data, the compressed data, and the repaired data of the anomaly repair data; Construct the edge information of the traceability map according to the operation type, the processing parameter information, and the quality change information of the anomaly repair data; Construct the traceability map according to the node information and the edge information to form central optimization data.

9. The power data quality optimization method according to claim 5, characterized in that The above-mentioned cloud optimization unit performs optimization according to the optimized central data and digital twin simulation data to obtain cloud optimization data, including: Determine a correction coefficient according to the optimized central data, the digital twin simulation data, and the device health; Optimize the optimized central data based on the correction coefficient and the digital twin simulation data to obtain cloud optimization data.

10. The power data quality optimization method according to claim 2, characterized in that, The secondary optimization of the edge optimization data, the central optimization data, and the cloud optimization data based on the edge optimization data, the optimized central data, and the cloud optimization data by the comprehensive optimization unit includes: Performing dynamic time warping and spatial topology mapping on the edge optimization data, the optimized central data, and the cloud optimization data to perform secondary optimization on the edge optimization data, the central optimization data, and the cloud optimization data.

Images