Power grid data restoration method and system based on Huber norm tensor decomposition

Abstract

The invention provides a power grid data restoration method and system based on Huber norm tensor decomposition, and belongs to the technical field of intelligent power grids and high-dimensional data processing. The method comprises the following steps: constructing multi-source and multi-dimensional power grid load data into a high-dimensional tensor structure; a Huber norm-based tensor decomposition model is adopted as a core reconstruction engine, through differential punishment of different residual errors, random noise is effectively suppressed, key abnormal modes such as electricity stealing are reserved, and excessive smoothing of abnormal features by a traditional method is avoided; efficiently solving the low-rank factor matrix of the tensor by using a stochastic optimization algorithm; and finally, realizing accurate filling of massive missing values in the original data through the reconstructed factor matrix. According to the method, the restoration precision and robustness of the power grid data containing noise and abnormal values can be remarkably improved, a high-quality data basis is provided for subsequent advanced analysis tasks such as electricity larceny detection and load prediction, and the method has important practical application value.

Description

Technical Field

[0001] This invention belongs to the field of smart grid and high-dimensional data processing technology, specifically relating to a method for repairing missing data under outlier interference in massive smart meter data. Background Technology

[0002] With the comprehensive advancement of smart grid construction, smart meters have been widely deployed on the user side, generating massive amounts of high-dimensional electricity consumption data that record multi-dimensional information such as users, time, and electricity consumption. Against this backdrop, ensuring the integrity and accuracy of grid data to provide a high-quality data foundation for upper-level applications such as electricity theft detection, accurate load forecasting, and grid stability assessment has become one of the core challenges in smart grid big data analytics.

[0003] However, in actual operation, smart meter data commonly suffers from problems such as data loss and outlier interference due to communication interruptions, equipment failures, or human tampering. In particular, fraudulent activities such as electricity theft can create strong, non-Gaussian distributed anomalies in the data. These problems not only distort actual electricity consumption patterns but also severely impact the accuracy of downstream electricity theft detection models and the reliability of load forecasting. Therefore, there is an urgent need for a data repair technology that can simultaneously handle data loss and resist outlier interference to improve the overall quality and availability of power grid data.

[0004] Traditional power grid data restoration methods mainly include statistical interpolation and low-rank decomposition-based modeling. Simple interpolation methods ignore the complex correlations between users and over time, resulting in limited restoration accuracy. While low-rank decomposition methods can utilize the multidimensional structure of the data, they are typically optimized based on the L2 norm (least squares). This assumption makes the model robust to random noise, but it is extremely sensitive to strong outliers such as electricity theft. During the restoration process, these methods tend to over-smooth, often erasing key anomalous electricity theft patterns as noise, thus destroying crucial evidence for subsequent detection tasks.

[0005] In recent years, deep learning methods have also been used for data repair, but many models still fail to fundamentally solve the problem of outlier interference. Some graph neural network-based methods also risk over-smoothing during feature propagation, which may lead to the loss of personalized electricity consumption characteristics for different users. In addition, most methods separate data repair from downstream detection tasks, failing to achieve synergistic gains between repair and detection.

[0006] In summary, the problem of smart grid data restoration has evolved from simple missing value filling to a complex issue requiring simultaneous high-dimensional spatiotemporal structure modeling and anomaly pattern preservation. To improve the effectiveness of restoration results for downstream tasks, it is necessary to introduce a new robust data restoration framework. This framework should achieve high-fidelity restoration of large-scale power grid data through differentiated processing of outliers, thereby providing solid data support for building a safe and efficient modern power system. Summary of the Invention

[0007] The technical problem to be solved by this invention is to propose a power grid data repair method and system based on Huber norm tensor decomposition, which is used to improve the integrity and accuracy of massive user electricity consumption data of smart grids under the presence of outlier interference. It solves the problem that existing tensor decomposition methods based on conventional L2 norm are sensitive to abnormal patterns such as electricity theft and are prone to feature smoothing, and provides high-quality data support for advanced analysis tasks such as electricity theft detection, user load forecasting and power grid status assessment.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] First, this invention proposes a power grid data repair method based on Huber norm tensor decomposition, comprising the following steps:

[0010] S1. Obtain the multidimensional smart meter dataset with missing values ​​and construct it as a three-dimensional observation tensor:

[0011] ,

[0012] Where U represents the user ID, H represents the intraday time point, and W represents consecutive dates; at the same time, an index set Λ of known observation data points is defined.

[0013] S2. The data repair problem is modeled as a low-rank tensor approximation solution, and a robust reconstruction objective function based on the Huber loss function is constructed to replace the traditional least squares loss. This objective function is defined as:

[0014] ,

[0015] in, The Huber loss function is used to evaluate the observations. With reconstructed values A quadratic penalty is applied to small residuals, and a linear penalty is applied to large residuals. This module, by differentiating between large and small residuals, endows the model with robustness to abnormal signals such as electricity theft, avoiding the over-smoothing problem of abnormal patterns in traditional methods. P, Q, and D are the low-rank factor matrices to be solved, and λ is the regularization coefficient.

[0016] S3. Using a stochastic optimization algorithm (Adam optimizer), with the aim of minimizing the objective function defined in step S2, the optimal factor matrices P, Q, and D are solved through iterative updates.

[0017] S4. Using the optimal factor matrix obtained in S3, reconstruct it using the CP decomposition formula. Generate a complete, repaired data tensor The result is the accurate filling of missing values ​​in the original data.

[0018] Meanwhile, this invention proposes a power grid data repair system, comprising:

[0019] The data construction module acquires multidimensional power grid load data containing observed and missing values, and constructs the data into a three-dimensional observation tensor, where the three dimensions of the three-dimensional observation tensor correspond to user ID, intraday time point, and consecutive dates, respectively. The objective function construction module constructs a low-rank tensor reconstruction objective function with the Huber loss function as the core loss term. The Huber loss function applies differential penalties to the residuals between observed and reconstructed values: a quadratic penalty is applied to residuals less than a preset threshold, and a linear penalty is applied to residuals greater than or equal to the preset threshold. The objective function solves for the low-rank factor matrix of the three-dimensional observation tensor by minimizing the core loss term. The optimization solution module iteratively optimizes the low-rank tensor reconstruction objective function using a stochastic optimization algorithm to obtain the optimal low-rank factor matrix. The data reconstruction module uses the optimal low-rank factor matrix to generate a complete power grid data tensor using a tensor decomposition reconstruction formula to fill in the missing values ​​in the three-dimensional observation tensor.

[0020] Furthermore, the present invention proposes an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed, it performs the steps described in the method of the present invention.

[0021] Meanwhile, the present invention proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method of the present invention.

[0022] Finally, the present invention proposes a computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method proposed in this invention.

[0023] The technical advantages of this invention compared to existing technologies are as follows:

[0024] This invention effectively solves the oversmoothing problem caused by the sensitivity to outliers in traditional L2-norm-based low-rank decomposition methods when processing data containing outliers by introducing a Huber norm tensor decomposition mechanism. This method suppresses random noise while preserving key anomalous structural features such as electricity theft, maintaining low repair errors under varying data missing rates. Compared to traditional interpolation methods, the repaired data provided by this invention can offer higher-quality input for downstream tasks such as electricity theft detection, significantly improving the accuracy of detection models and demonstrating good applicability and industrial applicability. Attached Figure Description

[0025] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] Example 1: In a preferred embodiment of the present invention, large-scale user electricity consumption data collected by smart meters in the power system is first preprocessed and organized into a three-dimensional tensor, wherein the user ID, intraday time point, and consecutive dates constitute the three dimensions of the tensor. Due to reasons such as abnormal meter communication, equipment failure, or human interference, some data points are missing in the tensor.

[0028] To achieve effective repair of missing values, this invention proposes a low-rank tensor decomposition method based on Huber norm constraints. For example... Figure 1 As shown, this embodiment provides a power grid data repair method based on Huber norm tensor decomposition. This method is applied to electronic devices and includes the following steps:

[0029] Specifically, this invention first maps the observed power data into tensors. Specifically:

[0030] U represents the user ID dimension. For example, if there are 1000 smart meter users in a certain power supply area, then U=1000.

[0031] H represents the dimension of time points within a day. For example, if sampling is done every 15 minutes, there are a total of 96 time points in a day, then H=96.

[0032] W represents the continuous date dimension. For example, if data is observed continuously for 30 days, then W=30.

[0033] To balance noise suppression and anomaly preservation, this invention introduces the Huber loss function, which takes the form of: a squared penalty when the residual magnitude is less than a threshold δ, and a linear penalty when the residual exceeds the threshold. This loss function can filter out random noise while avoiding over-smoothing of key anomaly patterns such as electricity theft. Based on this, this invention uses a three-factor low-rank decomposition model to model the tensor. Specifically, each element in the original tensor... It can be approximated by the product of the user factor matrix P, the time factor matrix Q, and the date factor matrix D, that is:

[0034]

[0035] To avoid excessive penalty from traditional squared loss (L2 norm) when outliers exist, this invention introduces the Huber norm as the residual metric. Let the residual be...

[0036]

[0037] The Huber loss is defined as follows:

[0038]

[0039] Where δ is the segmentation threshold. This design ensures a balance between quadratic penalty for small residuals and linear penalty for large residuals, enabling the algorithm to both suppress noise and preserve anomalous features. Therefore, the optimization objective of this invention is:

[0040]

[0041] Where λ is the regularization parameter, used to suppress overfitting.

[0042] In the solution process, this invention employs a gradient-based first-order optimization method, combined with the Adam algorithm for adaptive updates. First, for each residual... According to the piecewise definition of Huber loss, its gradient is:

[0043]

[0044] Furthermore, for the factor matrix parameters, for example Its gradient can be expressed as:

[0045]

[0046] The gradient forms of the other matrices Q and D are similar.

[0047] After obtaining the gradient, the Adam algorithm is used to update the parameters. For any parameter... In the t-th iteration, the current gradient is first calculated. Then, first-order moment estimation and second-order moment estimation are performed:

[0048]

[0049] in , These are the attenuation coefficients, typically taken to be close to 0.9 and 0.999. Then, deviation correction is performed:

[0050]

[0051] The final update rules are as follows:

[0052]

[0053] Where η is the learning rate and ϵ is a local constant.

[0054] Through continuous iteration, the Adam optimizer can dynamically adjust the step size across different parameter dimensions, accelerating the optimization process while maintaining convergence stability. The entire process continues until the objective function converges to a preset threshold, or the maximum number of iterations is reached. The factor matrices P, Q, and D obtained after convergence are used to calculate the reconstructed tensor. The missing elements are filled with reconstructed values ​​to obtain complete and robust grid load data.

[0055] The results of this method demonstrate that the present invention can not only effectively repair missing values ​​in large-scale power data, but also retain local anomalies such as electricity theft while maintaining the overall trend, ensuring the integrity and practicality of the repair results, and providing reliable data support for electricity theft detection, load forecasting and energy efficiency analysis.

[0056] Example 2: In another embodiment, the present invention also provides a power grid data repair system. This system includes a data tensor quantization module, an objective function construction module, an iterative optimization module, and a data reconstruction module. These modules work together to achieve the aforementioned repair method based on Huber norm tensor decomposition. This system can run on general-purpose computing platforms or dedicated power data processing equipment, and has strong scalability and engineering application value.

[0057] This embodiment selects a real electricity consumption dataset provided by the State Grid Corporation of China (SGCC) to verify the effectiveness of the proposed tensor decomposition-based electricity consumption data recovery method based on the Huber norm. This dataset covers daily electricity consumption records of 42,372 users from January 1, 2014 to October 31, 2016, with each user corresponding to a time series of 1035 time steps. Among all users, 3615 users were marked as anomalous, accounting for 8.53%. Because this dataset is representative in terms of scale, time span, and user diversity, and has a high proportion of anomalous samples, it can provide sufficient experimental verification and practical reference value for missing data recovery and subsequent anomaly detection tasks.

[0058] To evaluate the recovery effect, the experiment used two mainstream evaluation metrics: root mean squared error (RMSE) and mean absolute error (MAE). Their definitions are as follows:

[0059] ,

[0060] ,

[0061] Where Ω represents the masked missing position, Reconstructing values ​​for the model, This is the actual value.

[0062] In the comparative experiments, four mainstream interpolation methods were used as the benchmark model: mean imputation, zero imputation, linear interpolation, and K-nearest neighbor interpolation based on Euclidean distance (KNN). These methods cover typical interpolation techniques ranging from simple statistical completion to structure-aware interpolation, and are used to comprehensively verify the superiority of the MMA method.

[0063] To verify the robustness of the models, experiments were conducted to compare the models at data missing rates of 30%, 40%, and 50%, and the RMSE and MAE of each method were recorded. The results are shown in Table 1:

[0064] Table 1

[0065]

[0066] In summary, the experimental results fully demonstrate that the tensor decomposition method proposed in this invention has significantly better recovery accuracy and stability than traditional interpolation models in the task of recovering missing user electricity consumption data in smart grids, and can provide more reliable basic data support for subsequent anomaly detection and behavior modeling tasks.

[0067] Example 3: This example constructs a third-order tensor with the following dimensions: 3 users (User1, User2, User3) × 7 days a week × 2 weeks (Week1, Week2). Unit: kWh. Some observations are randomly set to missing (missing rate approximately 20%), and two "anomalies" are artificially retained and inserted to simulate extreme values ​​of electricity theft / tripping or meter reading errors (one anomaly is an abnormally low value, and the other is an abnormally high value).

[0068] The algorithm flow is as follows:

[0069] 1. Tensor Decomposition and Loss Construction: Decompose the tensor into low-rank factor matrices. , , Construct the Huber loss function The threshold is δ=2.0, and an L2 regularization term λ=0.001 is added.

[0070] 2. Optimization Iteration: Calculate the residuals for the observations at the mask positions. For example, the residual for User1Week2 on day 1 is r=11.540−11.969=−0.429. Use the Adam optimizer to iteratively update the factor matrix. When the residual is less than δ, update it according to the quadratic gradient; when it is greater than δ, update it according to the linear gradient. Iterate until convergence.

[0071] 3. Tensor Reconstruction and Missing Filling: The final reconstruction yields a complete tensor. For example, the missing values ​​of User2 and Week1 on day 3 are filled to 19.249 kWh from the original missing values. The outlier of User3 on day 5 of Week2, 45 kWh, is reduced to 44.717 kWh during reconstruction. This shows that the algorithm both repairs missing values ​​and preserves the outlier structure.

[0072] 4. Results evaluation: The RMSE for missing locations is approximately 2.96 kWh, and the RMSE for observed locations is approximately 2.48 kWh, which verifies the algorithm's repair accuracy and anomaly preservation ability on small-scale numerical examples.

[0073] Example 4: This example proposes an electronic system, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method steps of the present invention.

[0074] Example 5: This example proposes a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the method described in this invention, which will not be repeated here.

[0075] Example 6: This example proposes a computer program product, including a computer program / instructions, which, when executed by a processor, implements the steps of the method proposed in this invention, and will not be described again here.

[0076] It should be noted that the processing flow of embodiments 2-6 corresponds to the specific steps of the method provided in the embodiments of the present invention, and has the corresponding functional modules and beneficial effects of the method. Technical details not described in detail in this embodiment can be found in the method provided in the embodiments of the present invention.

[0077] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0078] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0079] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A power grid data repair method based on Huber norm tensor decomposition, characterized in that, include: S1. Receive multidimensional power grid load data containing observed values ​​and missing values, and construct it into a three-dimensional observation tensor, wherein the three dimensions of the three-dimensional observation tensor correspond to user number, intraday time point and consecutive dates, respectively. S2. Construct a low-rank tensor reconstruction objective function with the Huber loss function as the core loss term. The Huber loss function is used to apply differential penalties to the residuals between the observed values ​​and the reconstructed values: a quadratic penalty is applied to residuals less than a preset threshold, and a linear penalty is applied to residuals greater than or equal to the preset threshold. The objective function is used to solve for the low-rank factor matrix of the three-dimensional observation tensor by minimizing the core loss term. S3. The objective function is iteratively optimized using a stochastic optimization algorithm to obtain the optimal low-rank factor matrix; S4. Using the optimal low-rank factor matrix, a complete power grid data tensor is generated through the tensor decomposition and reconstruction formula to fill in the missing values ​​in the three-dimensional observation tensor.

2. The method according to claim 1, characterized in that, The objective function is defined as follows: ， in, The Huber loss function is used to evaluate the observations. With reconstructed values A quadratic penalty is applied to small residuals and a linear penalty is applied to large residuals. P, Q, and D are low-rank factor matrices to be solved, λ is the regularization coefficient, u represents the user ID, h represents the intraday time point, w represents consecutive dates, and Λ is the index set of known observation data points.

3. The method according to claim 1, characterized in that, The Huber loss function is defined as follows: Where r is the residual and δ is the segmented threshold.

4. The method according to claim 1 or 2, characterized in that, In step S4, the tensor decomposition and reconstruction formula is: ， Where p is the user factor matrix, q is the time factor matrix, d is the date factor matrix, and the subscripts are... u The user ID is represented by h, the time point within the day is represented by w, the consecutive dates are represented by r, and the residual is represented by r.

5. The method according to claim 1, characterized in that, The low-rank tensor reconstruction is achieved based on the CP decomposition model.

6. The method according to claim 1, characterized in that, The stochastic optimization algorithm is the Adam optimization algorithm.

7. A power grid data repair system, characterized in that, include: The data construction module acquires multidimensional power grid load data containing observed and missing values, and constructs the data into a three-dimensional observation tensor, where the three dimensions of the three-dimensional observation tensor correspond to user ID, intraday time point, and consecutive dates, respectively. The objective function construction module constructs a low-rank tensor reconstruction objective function with the Huber loss function as the core loss term. The Huber loss function applies differential penalties to the residuals between observed and reconstructed values: a quadratic penalty is applied to residuals less than a preset threshold, and a linear penalty is applied to residuals greater than or equal to the preset threshold. The objective function solves for the low-rank factor matrix of the three-dimensional observation tensor by minimizing the core loss term. The optimization solution module iteratively optimizes the low-rank tensor reconstruction objective function using a stochastic optimization algorithm to obtain the optimal low-rank factor matrix. The data reconstruction module uses the optimal low-rank factor matrix to generate a complete power grid data tensor using a tensor decomposition reconstruction formula to fill in the missing values ​​in the three-dimensional observation tensor.

8. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor is configured to execute the computer program, it implements the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 5.

Images