A data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm

Abstract

The present application relates to a kind of data cleaning method combined with space-time clustering and iterative threshold shrinkage algorithm, belong to data cleaning and preprocessing field, solve the problem of existing space-time power data cleaning inaccuracy, poor adaptability and low computing efficiency of abnormal detection.It includes steps S1, obtaining the original power data to be cleaned in low-voltage camp distribution fusion scenario;Original power data includes time layer and spatial layer data;Step S2, using clustering algorithm, time layer data is subdivided into mutation layer and stable layer data, and spatial layer data is subdivided into high-density layer and low-density layer data;Step S3, using iterative threshold shrinkage algorithm, respectively, mutation layer and stable layer, and high-density layer and low-density layer data are cleaned, to obtain cleaned each layered data;Step S4, calculate the residual error to obtain final residual error;If final residual error is less than preset residual error threshold, then output clean data;Otherwise, optimization control parameter returns step S2.Realize efficient and accurate power data cleaning.

Description

Technical Field

[0001] This invention relates to the field of data cleaning and preprocessing technology, and in particular to a data cleaning method that combines spatiotemporal clustering and iterative threshold shrinkage algorithm. Background Technology

[0002] With the accelerated construction of new power systems, the intelligence and digitalization of low-voltage distribution networks are continuously improving. "Low-voltage operation and distribution integration" has become a development strategy for digital power grids and an important development path for improving data quality on both the power consumption and distribution sides of new power systems. However, in the practical application of operation and distribution integration data governance, problems such as abnormal meter data collection and storage, inconsistent data formats, and missing or redundant data exist due to factors such as equipment failure, transmission errors, and environmental interference. Therefore, it is necessary to clean the power data under the low-voltage operation and distribution integration environment.

[0003] Traditional data cleaning methods mainly include statistical thresholding, rule-based methods, and machine learning-based methods. Thresholding is simple and efficient, but it is poorly adaptable to dynamically changing spatiotemporal data. Rule-based methods rely on prior knowledge and struggle to handle complex and ever-changing anomaly patterns. While machine learning methods can automatically learn data features, they are computationally complex and heavily reliant on labeled data. Furthermore, most existing methods do not adequately consider the hierarchical nature of spatiotemporal data and the dynamic changes in data quality, resulting in unsatisfactory cleaning effects. Summary of the Invention

[0004] Based on the above analysis, the embodiments of the present invention aim to provide a data cleaning method that combines spatiotemporal clustering and iterative threshold shrinkage algorithm. Through hierarchical processing and dynamic threshold optimization, it solves the technical problems of inaccurate anomaly detection, poor adaptability and low computational efficiency in spatiotemporal power data cleaning of existing methods.

[0005] This invention provides a data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm, comprising the following steps:

[0006] Step S1: Obtain the raw power data to be cleaned in the low-voltage operation and distribution integration scenario; wherein, the raw power data includes time layer data and spatial layer data;

[0007] Step S2: Use a clustering algorithm to subdivide the time layer data into abrupt change layer and a stationary layer, and subdivide the spatial layer data into a high-density layer and a low-density layer.

[0008] Step S3: Use the iterative threshold shrinkage algorithm to clean the data of the mutation layer and the stable layer, as well as the data of the high-density layer and the low-density layer, to obtain the cleaned data of each layer.

[0009] Step S4: Calculate the hierarchical residuals of the cleaned mutation layer, stable layer, high-density layer, and low-density layer data respectively to obtain the final residuals; if the final residuals are less than the preset residual threshold, output clean data; otherwise, optimize the clustering and cleaning control parameters and return to step S2 for iteration.

[0010] Further, step S2 includes:

[0011] Extract the temporal features of the time-layer data, and subdivide the time-layer data into abrupt change layer and stationary layer data based on the K-means clustering algorithm;

[0012] The spatial features of the spatial layer data are extracted, and the spatial layer data is further subdivided into high-density layer and low-density layer data based on the DBSCAN clustering algorithm.

[0013] Furthermore, based on the K-means clustering algorithm, the time-layer data is further subdivided into abrupt change layer and a stationary layer, including:

[0014] Extracting wavelet energy from time-layer data for each time window With local outliers , obtained the The feature vectors of each time window are ;

[0015] The feature vectors of all time windows constitute the input dataset for the K-means clustering algorithm. ;in, Number of time windows;

[0016] Random initialization Cluster centers ;

[0017] Calculate the Euclidean distance from the feature vector of each time window to any cluster center;

[0018] The feature vector of each time window is assigned to the cluster corresponding to the cluster center with the nearest Euclidean distance;

[0019] Once the feature vectors for all time windows have been assigned, the mean of all feature vectors within each cluster is used as the new cluster center.

[0020] For multiple randomly initialized Value, calculate each The value corresponds to the DB index, and the number of clusters with the smallest DB index is selected. As the optimal clustering number ;

[0021] extract The wavelet energy and LOF of each cluster center are used to form the wavelet energy set of the cluster centers and the LOF value set of the center points;

[0022] Based on the cluster center wavelet energy set and the center point LOF value set, respectively, the wavelet energy threshold and LOF threshold are obtained using the median principle;

[0023] For each cluster center, if the wavelet energy is greater than the wavelet energy threshold and the LOF value is greater than the LOF threshold, the cluster is determined to be a mutation layer; if the wavelet energy is less than the wavelet energy threshold and the LOF value is less than the LOF threshold, the cluster is determined to be a stationary layer.

[0024] Based on the defined mutation layer reference point and stationary layer reference point, a cluster center that does not meet the conditions of either mutation layer or stationary layer is determined to be a mutation layer if the distance from the mutation layer reference point to the cluster center is less than the distance to the stationary layer reference point; otherwise, it is a stationary layer.

[0025] Furthermore, the reference point of the mutation layer is a point composed of the maximum wavelet energy and the maximum LOF value of all cluster centers, as shown below:

[0026] ;

[0027] The reference point for the stable layer is a point constructed based on the minimum wavelet energy and the minimum LOF value of all cluster centers, as shown below:

[0028] ;

[0029] in, Wavelet energy value for each cluster center; The LOF value for each cluster center. Further, based on the DBSCAN clustering algorithm, the spatial layer data is subdivided into high-density and low-density layers, including:

[0030] The spatial layer data includes the location data of multiple devices;

[0031] Calculate the Haversine distance between any two device points based on location data;

[0032] Calculate the kernel density estimate of the device points based on the Haversine distance between them;

[0033] The location data of each device and the kernel density estimate constitute the composite spatial feature vector of each device;

[0034] Set neighborhood radius and minimum points DBSCAN clustering is performed based on the composite spatial feature vector of each device to obtain the labeling results of whether the device points are core points, boundary points or noise points;

[0035] The connected set consisting of devices marked as core points and their associated boundary points is determined as high-density layer devices; the data corresponding to high-density layer devices is high-density layer data.

[0036] The set of devices marked as noise points is identified as low-density layer devices; the data corresponding to low-density layer devices is low-density layer data.

[0037] Furthermore, the iterative threshold shrinkage algorithm is used to clean the data of the mutation layer and the stationary layer respectively, to obtain the cleaned mutation layer and stationary layer data, including:

[0038] Strong regularization constraints are applied to the abrupt change layer, while weak regularization constraints are applied to the stable layer.

[0039] A first objective function for cleaning the data of the mutation layer and the stationary layer is constructed; iterative cleaning is performed using an iterative threshold shrinkage algorithm until the first objective function converges, and the cleaned mutation layer and stationary layer data are obtained.

[0040] Furthermore, an iterative threshold shrinkage algorithm is used to clean the high-density layer and low-density layer data respectively, resulting in cleaned high-density layer and low-density layer data, including:

[0041] Strong regularization constraints are applied to high-density layers, while weak regularization constraints are applied to low-density layers.

[0042] A second objective function for cleaning high-density and low-density layer data is constructed; iterative cleaning is performed using an iterative threshold shrinkage algorithm until the second objective function converges, resulting in cleaned high-density and low-density layer data.

[0043] Furthermore, the layering residuals of the cleaned abrupt change layer, stable layer, high-density layer, and low-density layer were calculated separately. , , and ;

[0044] Obtain the final residual ;

[0045] like Output clean data for the cleaned mutation layer, stationary layer, high-density layer, and low-density layer; among which This is the preset residual threshold.

[0046] Furthermore, the clustering and cleaning control parameters include: the number of clusters in K-means clustering and the neighborhood radius in DBSCAN clustering. With minimum number of points And the strong regularization coefficient and weak regularization coefficient of the iterative threshold shrinkage algorithm.

[0047] Furthermore, the time-layer data includes voltage, current, or active power data;

[0048] The spatial layer data includes the location data of electricity meters and distribution transformers, or the power grid topology diagram.

[0049] To achieve the above objectives, this invention provides a data cleaning method that combines spatiotemporal clustering with an iterative threshold shrinkage algorithm.

[0050] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0051] 1. This invention solves the problems of poor adaptability of traditional threshold and rule-based methods to dynamic spatiotemporal data, and the "over-cleaning" and "under-cleaning" of spatiotemporal data due to machine learning methods not fully considering the spatiotemporal hierarchy. It achieves precise and adaptive cleaning by using spatiotemporal clustering to create hierarchical layers (temporally divided into abrupt / stable layers, spatially divided into high / low density layers), thus identifying the inherent structure of the data. Based on this, differentiated ISTA cleaning strategies are implemented for different layers (strong regularization for abrupt / high density layers, and weak regularization for stable / low density layers), achieving a "targeted approach." This avoids the problems of traditional methods' "one-size-fits-all" approach, which erases normal subtle features (over-cleaning) or fails to handle hidden anomalies (under-cleaning), significantly improving the accuracy of cleaning.

[0052] 2. This invention addresses the shortcomings of existing methods in handling dynamic changes in data quality and their lack of self-optimization capabilities. It constructs a closed-loop governance framework of "clustering-cleaning-hierarchical residual feedback," enhancing the method's adaptability and robustness. It innovatively introduces a hierarchical residual feedback mechanism. By calculating the residuals after cleaning at each layer, the cleaning effect is dynamically assessed. If the effect is unsatisfactory (residuals not converged), the clustering and cleaning control parameters (such as the number of clusters, neighborhood radius, regularization coefficient, etc.) are automatically optimized, and a new round of iteration is initiated. This enables the method to possess self-verification and continuous optimization capabilities, adapting to changes in data distribution and ensuring stable long-term application results, achieving a leap from "static cleaning" to "dynamic governance."

[0053] 3. This invention addresses the issue that existing machine learning methods suffer from high computational complexity, making it difficult to meet the efficiency requirements of large-scale data cleaning. This invention significantly improves the efficiency of large-scale spatiotemporal data processing while maintaining cleaning accuracy. The solution combines hierarchical processing with the efficient ISTA algorithm. First, hierarchical clustering decomposes the complex problem, reducing the data load and simplifying the problem. Second, the ISTA algorithm itself is an efficient iterative algorithm for solving L1 regularization problems, with low computational complexity and good convergence. The combination of these two approaches enables this solution to significantly reduce computation time while maintaining high cleaning accuracy when dealing with large-scale, multi-source, and heterogeneous spatiotemporal data in low-voltage distribution and operation scenarios. This provides feasible high-quality data support for real-time applications such as distribution network state estimation and demand response.

[0054] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0055] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0056] Figure 1 This is a flowchart of a data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm in an embodiment of the present invention;

[0057] Figure 2 This is a block diagram of a clustering hierarchical model that considers spatiotemporal data differences in an embodiment of the present invention;

[0058] Figure 3 This is a block diagram of the hierarchical residual feedback mechanism cleaning strategy in an embodiment of the present invention;

[0059] Figure 4 This is a schematic diagram comparing the data cleaning performance and cleaning efficiency of each model in the embodiments of the present invention. Detailed Implementation

[0060] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0061] To address the aforementioned technical challenges and ensure the accuracy and reliability of data governance, it is necessary to consider the temporal differences and spatial regionalities of data. Therefore, a hierarchical power data cleaning method for low-voltage integrated operation and distribution scenarios is urgently needed. This method should accurately identify and repair spatiotemporal data anomalies while maintaining computational data cleaning efficiency, providing high-quality data support for low-voltage integrated operation and distribution grid state estimation, demand response, and other business operations. This invention is applicable to anomaly detection and data quality improvement of large-scale low-voltage integrated operation and distribution spatiotemporal datasets.

[0062] With the continuous advancement of the construction of new power systems, the level of intelligence and digitalization of low-voltage distribution networks is being significantly improved. The integration of low-voltage operation and distribution has now become a development strategy for digital power grids and an important development direction for the governance and improvement of data quality on the power consumption side and distribution side of new power systems.

[0063] Against this backdrop, the present invention provides a hierarchical data cleaning method that combines spatiotemporal clustering analysis with ISTA (Iterative Shrinkage-Thresholding Algorithm). Through hierarchical processing and dynamic threshold optimization, it solves the problems of inaccurate anomaly detection, poor adaptability and low computational efficiency in spatiotemporal power data cleaning.

[0064] A specific embodiment of the present invention discloses a data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm, such as... Figure 1 As shown, it includes the following steps:

[0065] Step S1: Obtain the raw power data to be cleaned in the low-voltage operation and distribution integration scenario; wherein, the raw power data includes time layer data and spatial layer data;

[0066] Step S2: Use a clustering algorithm to subdivide the time layer data into abrupt change layer and a stationary layer, and subdivide the spatial layer data into a high-density layer and a low-density layer.

[0067] Step S3: Use the iterative threshold shrinkage algorithm to clean the data of the mutation layer and the stable layer, as well as the data of the high-density layer and the low-density layer, to obtain the cleaned data of each layer.

[0068] Step S4: Calculate the hierarchical residuals of the cleaned mutation layer, stable layer, high-density layer, and low-density layer data respectively to obtain the final residuals; if the final residuals are less than the preset residual threshold, output clean data; otherwise, optimize the clustering and cleaning control parameters and return to step S2 for iteration.

[0069] Step S1, specifically.

[0070] The low-voltage operation integration scenario refers to the deep integration of power data from the power consumption side (such as voltage, current, or active power) with power data from the power distribution network operation side (such as meter location, transformer location, or power grid topology diagram) in a low-voltage distribution network scenario. This enables data interoperability and business collaboration, supporting the application environment of digital governance such as distribution network status monitoring, fault diagnosis, and demand response.

[0071] Acquire the raw power data to be cleaned in the low-voltage operation convergence scenario, including time-layer data and spatial-layer data, as well as the timestamp of the raw power data collection.

[0072] The time-layer data includes voltage, current, or active power data;

[0073] The spatial layer data includes the location data of electricity meters and distribution transformers, or the power grid topology diagram.

[0074] Step S1 aims to acquire and define the raw power data that possesses both time-series and spatial distribution characteristics, providing data input for subsequent spatiotemporal clustering and hierarchical cleaning. Acquiring the raw power data and clarifying its dual dimensions of "time" and "space" are prerequisites for implementing subsequent spatiotemporal clustering and hierarchical cleaning steps.

[0075] Step S2 includes steps S21-S22.

[0076] A flowchart of a clustering hierarchical model considering spatiotemporal data variability is shown below. Figure 2 As shown.

[0077] Clustering essentially involves dividing a set of data objects into several groups based on their commonalities, resulting in more commonalities within each group and greater differences between different groups.

[0078] Using voltage measurement data collected every 15 minutes from 300 households with location tags in a certain area of ​​Shanghai as an example, a multi-index data evaluation was conducted.

[0079] Taking the collected voltage data as an example, the definition of clustering is as follows: Collected voltage a time window Dimension Dataset :

[0080] ;

[0081] in, They are respectively Voltage data for each time window.

[0082] gather Clustering is the process of breaking down data into smaller, more similar groups based on their similarity. Cluster , Indicates the first There are several clusters. The clustering result is represented as follows:

[0083] ;

[0084] Step S2 includes:

[0085] Extract the temporal features of the time-layer data, and subdivide the time-layer data into abrupt change layer and stationary layer data based on the K-means clustering algorithm;

[0086] Spatial features of the spatial layer data are extracted, and the spatial layer data is further subdivided into high-density and low-density layers based on the DBSCAN clustering algorithm. For example... Figure 2 As shown.

[0087] K-means clustering is performed on practical data, such as voltage, current, or active power. The practical data used in this paper is voltage data, but this method is also applicable to current or active power.

[0088] like Figure 1 As shown, this invention first uses K-means and DBSCAN algorithms to cluster and stratify spatiotemporal data, dividing the time dimension data into a mutation layer with frequent anomalies and a stable layer with normal fluctuations, and dividing the spatial dimension data into a high-density layer with dense equipment and a low-density layer with sparse edges.

[0089] The acquired raw power data is classified into time and space layers. For example, voltage data belongs to time data, while the latitude and longitude coordinates of the meter equipment belong to space data.

[0090] Step S21: Based on the K-means clustering algorithm, the time layer data is subdivided into a mutation layer and a stationary layer.

[0091] The time-layer data is further subdivided into a mutation layer and a stationary layer based on the K-means clustering algorithm, including:

[0092] Extracting wavelet energy from time-layer data for each time window With local outliers , obtained the The feature vectors of each time window are ;

[0093] The feature vectors of all time windows constitute the input dataset for the K-means clustering algorithm. ;in, Number of time windows;

[0094] Random initialization Cluster centers ;

[0095] Calculate the Euclidean distance from the feature vector of each time window to any cluster center;

[0096] The feature vector of each time window is assigned to the cluster corresponding to the cluster center with the nearest Euclidean distance;

[0097] Once the feature vectors for all time windows have been assigned, the mean of all feature vectors within each cluster is used as the new cluster center.

[0098] For multiple randomly initialized Value, calculate each The value corresponds to the DB index, and the number of clusters with the smallest DB index is selected. As the optimal clustering number ;

[0099] extract The wavelet energy and LOF of each cluster center are used to form the wavelet energy set of the cluster centers and the LOF value set of the center points;

[0100] Based on the cluster center wavelet energy set and the center point LOF value set, respectively, the wavelet energy threshold and LOF threshold are obtained using the median principle;

[0101] For each cluster center, if the wavelet energy is greater than the wavelet energy threshold and the LOF value is greater than the LOF threshold, the cluster is determined to be a mutation layer; if the wavelet energy is less than the wavelet energy threshold and the LOF value is less than the LOF threshold, the cluster is determined to be a stationary layer.

[0102] Based on the defined mutation layer reference point and stationary layer reference point, a cluster center that does not meet the conditions of either mutation layer or stationary layer is determined to be a mutation layer if the distance from the mutation layer reference point to the cluster center is less than the distance to the stationary layer reference point; otherwise, it is a stationary layer.

[0103] For time-series data, we extract the temporal features of the data and divide the time-series data into a mutation layer with frequent abnormal behavior and a stable layer with normal fluctuations based on the K-means clustering method; this lays the foundation for achieving differentiated cleaning of time-dimensional data.

[0104] This invention utilizes the K-means clustering algorithm to divide time series data into abrupt change layers and stationary layers based on the behavioral pattern characteristics of data from different time periods. This process does not directly apply to the original time series electricity data; instead, it divides the data in the feature space based on feature vectors extracted from the original time series electricity data to obtain the abrupt change layer and stationary layer data.

[0105] The specific steps are as follows:

[0106] (1) Feature vector construction: For each time layer data to be analyzed, extract the time series features that can characterize volatility and anomalies, and construct the feature vector.

[0107] This invention uses wavelet energy With Local Outlier Factor (LOF) As a feature, then the first The feature vectors of each time window are .

[0108] The feature vectors of all time windows in the time-layer data constitute the input dataset for the K-means clustering algorithm, as shown below:

[0109] ;

[0110] in, They are respectively Feature vectors of time-layer data within a time window.

[0111] (2) Clustering iteration and center update:

[0112] a) Random initialization Cluster centers ;

[0113] b) Calculate the Euclidean distance between any two feature vectors. The smaller the Euclidean distance, the higher the similarity between the two feature vectors. The Euclidean distance is used as a similarity evaluation index to measure the overall difference across different time periods. For any two feature vectors... and The Euclidean distance is calculated as follows:

[0114] ;

[0115] The smaller the Euclidean distance value, the more similar the time-layer data of the two time windows are.

[0116] Euclidean distance is used as the similarity evaluation index, and objects that are close in distance, i.e. have high similarity, are grouped together.

[0117] c) Each feature vector Assigned to the cluster center with the closest Euclidean distance. The corresponding cluster middle.

[0118] d) Recalculate each cluster Cluster center, new center It is the mean of all eigenvectors within this cluster.

[0119] e) Clustering effectiveness evaluation and determination of the optimal number of clusters: To determine the optimal number of clusters... The effectiveness was evaluated using the Davies-Bouldin (DB) index.

[0120] The DB index evaluates clustering quality by measuring the ratio of intra-class divergence to inter-class distance, as shown below:

[0121] ;

[0122] in, Cluster All samples were sent to its center. The average distance is used to measure the divergence within a cluster (i.e., a class); Cluster All samples to cluster Average distance from the center; For clusters and The Euclidean distance between the centers. The number of clusters that minimizes the DB value. Selected as the best number of clusters .

[0123] For example, It can take 2, 3, 4... values. ;like The minimum DB value indicates the optimal number of clusters. .

[0124] (3) Division of mutation layer and stationary layer: Based on the optimal clustering results, the final obtained Each cluster is then identified.

[0125] The median principle is used to define the reference points for the mutation layer and the stationary layer.

[0126] The reference points for the mutation layer are points formed by the maximum wavelet energy and the maximum LOF value of all cluster centers, as shown below:

[0127] ;

[0128] The reference point for the stable layer is a point constructed based on the minimum wavelet energy and the minimum LOF value of all cluster centers, as shown below:

[0129] ;

[0130] in, Wavelet energy value for each cluster center; The LOF value for each cluster center.

[0131] The K-means algorithm is a partition-based clustering algorithm. Its implementation mainly involves processing a set of data with... The dataset of tuples or records is divided into groups, assuming the number of groups is... Grouping is used as the basis for clustering. Optimal number of clusters. The value represents the number of clusters after the final clustering, which is only known after performing K-means clustering on the specific data. What is the value?

[0132] Each group must contain at least one data record and the record contained therein must not appear in other groups.

[0133] Step S22: Based on the DBSCAN clustering algorithm, the spatial layer data is subdivided into high-density layer data and low-density layer data.

[0134] For spatial layer data, spatial features are extracted, and the spatial layer data is divided into a high-density layer and a sparsely distributed low-density layer based on the DBSCAN clustering method, which represents the distribution relationship between electricity meter equipment, distribution transformer equipment, or power grid topology.

[0135] The spatial layer data is further subdivided into high-density and low-density layers based on the DBSCAN clustering algorithm, including:

[0136] The spatial layer data includes the location data of multiple devices;

[0137] Calculate the Haversine distance between any two device points based on location data;

[0138] Calculate the kernel density estimate of the device points based on the Haversine distance between them;

[0139] The location data of each device and the kernel density estimate constitute the composite spatial feature vector of each device;

[0140] Set neighborhood radius and minimum points DBSCAN clustering is performed based on the composite spatial feature vector of each device to obtain the labeling results of whether the device points are core points, boundary points or noise points;

[0141] The connected set consisting of devices marked as core points and their associated boundary points is determined as high-density layer devices; the data corresponding to high-density layer devices is high-density layer data.

[0142] The set of devices marked as noise points is identified as low-density layer devices; the data corresponding to low-density layer devices is low-density layer data.

[0143] Perform DBSCAN clustering on spatial data, such as device location coordinates.

[0144] The DBSCAN algorithm divides the location coordinate data of multiple devices in the spatial layer dataset into three categories: core points, boundary points, and noise points.

[0145] To achieve differentiated cleaning of spatial data, this invention first needs to divide the data into high-density and low-density layers based on the spatial distribution density of power equipment (such as smart meters).

[0146] Traditional DBSCAN algorithms rely solely on geometric distances of geographic coordinates, making it difficult to accurately reflect the complex density distribution caused by equipment cluster deployment and topological connections in power distribution networks. Therefore, this invention proposes an improved DBSCAN clustering method. Its core lies in constructing a composite spatial feature space that integrates geographic distance and kernel density estimation (KDE), thereby enabling more accurate density stratification of spatial data. Figure 2 As shown.

[0147] (1) Spatial layer data includes location data of multiple devices. Based on the location data, the Haversine distance and kernel density estimate between any two device points are calculated:

[0148] Haversine geographic distance directly quantifies the physical proximity between device points. For any two device points... and Haversine distance between the two device points The calculation is as follows:

[0149] ;

[0150] in, For the Earth's radius, For equipment points Its longitude and latitude location; For equipment points Its longitude and latitude location;

[0151] Kernel density estimates characterize the global distribution density at the location of equipment points, and are used to correct for the limitations of pure distance metrics. For equipment points... KDE value The calculation is as follows:

[0152] ;

[0153] in, For kernel functions (such as Gaussian kernels). For bandwidth parameters, This represents the total number of equipment points. The higher the value, the higher the equipment point. The denser the surrounding equipment.

[0154] (2) DBSCAN clustering based on composite spatial feature vectors:

[0155] Each device point is represented as a composite spatial feature vector, as shown below:

[0156] ;

[0157] in For equipment points . composite spatial eigenvectors.

[0158] Perform DBSCAN clustering in this three-dimensional feature space:

[0159] Weighted Euclidean distance is used as the distance metric for DBSCAN. For two points... and The Euclidean distance is calculated as follows:

[0160] ;

[0161] in, is a weighting coefficient used to balance the contribution of geographic distance and density KDE features in clustering.

[0162] Given neighborhood radius and minimum points For a device point If in its Neighborhood Contains at least If there are 1 point, then It was identified as the core point.

[0163] Neighborhood is represented in the data space by points. Centered on, with radius as The set of all points within the region.

[0164] Starting from any core point, identify all points whose density is reachable to form a cluster. Points that cannot be assigned to any cluster are marked as noise points.

[0165] This represents the minimum number of sample points required to form a cluster, or in other words, the minimum number of points within the neighborhood required to determine a point as a core point. This paper uses the distribution location and concentration of electricity meters in a certain region as an example to perform spatial layer data clustering.

[0166] For spatial DBSCAN clustering, the specific hierarchical method is as follows:

[0167] For a non-core point sample b, if b is located at a core point... of Within the neighborhood, sample b is called the boundary point.

[0168] For a sample n that is not a core point, if n is not in any core point... of Within the domain, sample n is called a noise point.

[0169] The set of all non-noise points forms a high-density region, and the set of noise points forms a low-density region.

[0170] (3) High-density layer A collection of connected regions consisting of all core points and their associated boundary points. In the composite feature space, these regions represent areas with dense device deployment and tight topological connections.

[0171] low-density layer The set of all devices marked as noise points. These points are relatively isolated in the composite feature space, representing sparsely distributed devices located at the network edge or in remote areas.

[0172] The purpose of step S2 is to intelligently divide the original power data into sub-layers with different anomaly characteristics and distribution features through spatiotemporal clustering (the time layer data is subdivided into abrupt change layer and stable layer data, and the spatial layer data is subdivided into high-density layer and low-density layer data), providing a precise data structure foundation for the subsequent implementation of differentiated adaptive cleaning strategies.

[0173] Step S3, specifically.

[0174] The iterative threshold shrinkage algorithm is used to clean the data of the mutation layer and the stationary layer respectively, resulting in cleaned mutation layer and stationary layer data, including:

[0175] Strong regularization constraints are applied to the abrupt change layer, while weak regularization constraints are applied to the stable layer.

[0176] A first objective function for cleaning the data of the mutation layer and the stationary layer is constructed; iterative cleaning is performed using an iterative threshold shrinkage algorithm until the first objective function converges, and the cleaned mutation layer and stationary layer data are obtained.

[0177] The high-density layer and low-density layer data are cleaned using an iterative threshold shrinkage algorithm, resulting in cleaned high-density layer and low-density layer data, including:

[0178] Strong regularization constraints are applied to high-density layers, while weak regularization constraints are applied to low-density layers.

[0179] A second objective function for cleaning high-density and low-density layer data is constructed; iterative cleaning is performed using an iterative threshold shrinkage algorithm until the second objective function converges, resulting in cleaned high-density and low-density layer data.

[0180] Differentiated ISTA cleaning methods are constructed for different data layer characteristics. Strong regularization constraints are applied to the mutation layer and high-density layer to effectively filter out outliers, while weak regularization constraints are applied to the stable layer and low-density layer to retain subtle features.

[0181] (1) K-means time layer data clustering: high fluctuation means that voltage data changes more, so a high fluctuation mutation layer means frequent abnormalities; low fluctuation corresponds to normal fluctuation.

[0182] The time series data, which has already been divided into periods of high anomaly incidence and periods of normal fluctuation using the K-means clustering method, are denoted as mutation layers, respectively. and the stable layer .

[0183] Let the time series data be a vector. Data after cleaning ,in, This represents the total number of time windows. It represents the set of real numbers. Refers to raw power data It is a collection A vector of real numbers (i.e., the entire time series).

[0184] Similarly, This refers to the data after cleaning. It is also A real vector of dimension 1.

[0185] The first objective function for hierarchical regularization is constructed as follows:

[0186] ;

[0187] in, This is the time-layer data after cleaning; This is the original time-layer data; Represents the data fit term, measuring and Differences between them; This is the time-strength regularization coefficient; For the first Data values ​​after cleaning of the mutation layer data at each time point; For the first Data values ​​of the stable layer at each time point after cleaning; The time-based weak regularization coefficient; Strong L1 regularization is applied to the mutation layer, and weak L1 regularization is applied to the stationary layer.

[0188] The iterative update of the time-layer data using the ISTA iterative threshold shrinkage algorithm consists of two steps: the gradient calculation step and the hierarchical threshold shrinkage step.

[0189] The gradient step results are calculated as follows:

[0190] ;

[0191] in, For the first The intermediate result vector of time-layer data before threshold shrinkage is performed after each ISTA iteration; For the first The estimated value of the cleaned time layer data vector obtained after the next iteration; The iteration step size controls the magnitude of each iteration update; This represents the gradient operator.

[0192] In ISTA, the `sign` function ensures that the parameters after shrinkage maintain the direction of the original gradient, while the thresholding operation controls the magnitude of shrinkage by subtracting a threshold from the absolute value. The `sign` function is shown below:

[0193] ;

[0194] Mutation layer ( , The iterative threshold shrinkage step (representing a specific time point in a time series, where voltage is collected over time) is calculated as follows:

[0195] ;

[0196] in, For the first During the next ISTA iteration, at time point Data values ​​of the mutation layer after cleaning; For the first After the gradient step of the next iteration, at time point Intermediate result values; This represents the soft threshold shrinkage function; This indicates that `sign` takes the sign, ensuring that its sign is not changed. Formula (13) is the explanation of the `sign` function; Represents absolute value; This is the iteration step size.

[0197] Stable layer ( The iterative threshold shrinkage step is calculated as follows:

[0198] ;

[0199] After cleaning, outliers in the data were corrected. For example, in a low-voltage distribution scenario where the voltage is around 220V, a sudden appearance of a 300V voltage would need to be corrected. The cleaned data is then obtained.

[0200] (2) Spatial ISTA adaptive cleaning

[0201] The spatial coordinates have already been divided into high-density and low-density regions using the DBSCAN clustering method, denoted as follows: and .

[0202] Let the observation data be Data after cleaning .

[0203] The second objective function for hierarchical regularization is constructed as follows:

[0204] ;

[0205] in, This is the original spatial layer data; This refers to the cleaned spatial layer data; Represents the data fit term, measuring and Differences between them; The space regularization coefficient; The space weak regularization coefficient; This is the high-density layer data after cleaning; This is the data from the cleaned low-density layer; These represent the coordinates, longitude, and latitude, respectively. For high-density regions, L1 strong regularization is used, and for low-density regions, L1 weak regularization is used.

[0206] The iterative update of spatial layer data using ISTA consists of two steps: the gradient calculation step and the layer threshold shrinkage step.

[0207] The gradient step results are calculated as follows:

[0208] ;

[0209] in, For the first After each ISTA iteration, the intermediate result vector of the spatial layer data is shrunk by thresholding. For the first The estimated value of the cleaned spatial layer data vector obtained after the iteration.

[0210] In ISTA, the sign function is used to ensure that the parameters after shrinkage maintain the direction of the original gradient, while the threshold operation controls the magnitude of shrinkage by subtracting the threshold from the absolute value.

[0211] ;

[0212] The high-density layer iteration threshold shrinkage step is calculated as follows:

[0213] ;

[0214] The iterative threshold shrinkage step for low-density regions is calculated as follows:

[0215] ;

[0216] Step S3 is to perform precise cleaning of the clustered data by executing a differentiated iterative threshold shrinkage algorithm, which uses "strong regularization to remove anomalies and weak regularization to preserve details", thereby obtaining the cleaned data.

[0217] Step S4, specifically.

[0218] By dynamically optimizing cluster boundaries and threshold parameters through a hierarchical residual feedback mechanism, an innovative governance framework of "dynamic hierarchical-adaptive cleaning-closed-loop verification" is formed.

[0219] Calculate the layering residuals of the cleaned abrupt change layer, stable layer, high-density layer, and low-density layer, respectively. , , and ;

[0220] Obtain the final residual ;

[0221] like Output clean data for the cleaned mutation layer, stationary layer, high-density layer, and low-density layer; among them, This is the preset residual threshold.

[0222] After initial cleaning of the classified spatiotemporal data using ISTA, a hierarchical residual feedback mechanism is introduced to diagnose errors and dynamically adjust parameters in the cleaned data, achieving adaptive closed-loop data governance. A customized cleaning strategy is implemented for the hierarchical structure, such as... Figure 4 As shown.

[0223] From a time perspective, the residuals of the abrupt change layer are large, the wavelet energy is high, and the fluctuations are relatively violent. A strong regularization cleaning strategy should be adopted to strengthen the filtering of outliers and suppress the influence of noise. The residuals of the stable layer are small, the autocorrelation is strong, and the fluctuations are relatively gentle. A weak regularization cleaning strategy should be adopted to retain subtle fluctuations and prevent over-smoothing.

[0224] From a spatial perspective, the residuals in high-density layers exhibit strong spatial correlation, and data errors between adjacent or near-neighbor regions show a clustering effect, resulting in dense anomalies. Therefore, a strong regularization cleaning strategy should be adopted to suppress noise using proximity information and to make anomaly clusters easier to identify by increasing the weight of sparse terms. In contrast, the residuals in low-density layers have weak spatial properties, and anomaly points are discretely distributed. Therefore, a regularization cleaning strategy should be adopted to avoid misjudging isolated points as group anomalies.

[0225] Define the cleaned data as , The residual matrix after cleaning the original power data is shown below:

[0226] ;

[0227] in, This is the residual matrix after cleaning.

[0228] Suppose that meter A (with fixed latitude and longitude) collects an initial voltage of 230V at a certain point in time (which may be an abnormal spike). After spatial ISTA cleaning, the voltage of meter A at that point in time is corrected to 220V. Therefore, for meter A at this spatial point, the residual is 230V - 220V = 10V.

[0229] Spatial stratified residuals are the average level of the measurement data residuals of all meters within a region (high-density or low-density layer). Similarly, temporal stratified residuals are the residuals of the measured voltage before and after cleaning at a specific point in time.

[0230] The residuals of the time-abrupt layer, the time-stable layer, the spatially high-density layer, and the spatially low-density layer are shown in equations (22) to (35), respectively. This is equivalent to performing a residual averaging calculation on the residuals of all data within each layer:

[0231] ;

[0232] in, This represents the total number of data in the mutation layer; Indicates the first The L2 norm of the residuals of the mutation layer data at each time point.

[0233] ;

[0234] in, This indicates the total number of data points in the stable layer; Indicates the first The L2 norm of the residuals of the stationary layer data at each time point.

[0235] ;

[0236] in, This indicates the number of devices that are divided into high-density layers; Indicates the first The L2 norm of the high-density layer data residuals at each time point.

[0237] ;

[0238] in, This indicates the number of devices that are classified into the low-density layer; Indicates the first The L2 norm of the low-density layer data residuals at each time point.

[0239] when When the iteration ends, clean data is output.

[0240] The clustering and cleaning control parameters include: the number of clusters in K-means clustering and the neighborhood radius in DBSCAN clustering. With minimum number of points And the strong regularization coefficient and weak regularization coefficient of the iterative threshold shrinkage algorithm.

[0241] Strong regularization coefficients include: time strong regularization coefficients. and spatially strong regularization coefficient ;

[0242] Weak regularization coefficients include: time-based weak regularization coefficients. and spatial weak regularization coefficient .

[0243] The purpose of step S4 is to calculate the hierarchical residuals and construct a feedback loop to achieve dynamic evaluation of the data cleaning effect and autonomous optimization of the core control parameters, thereby ensuring the adaptability and continuous optimization capability of the entire cleaning process.

[0244] To verify the effectiveness of data cleaning, root mean square error (RMSE), mean absolute error (MAE), and coefficient of determination (CDE) were introduced. RMSE and MAE are used as evaluation metrics for data cleaning effectiveness. The smaller the values ​​of RMSE and MAE, the better the data cleaning performance. 2 A higher value indicates better data governance performance of the model. RMSE, MAE, and The calculation is as follows:

[0245] ;

[0246] Where n is the number of original power data points to be cleaned. This is the repair value for the i-th abnormal data. This represents the actual value of the i-th outlier before any outliers were added. This represents the average of the actual data.

[0247] like Figure 4 As shown, the performance and efficiency evaluation and comparison of data cleaning are as follows:

[0248] (1) Data cleaning performance evaluation indicators

[0249] To verify the superior performance of the proposed data cleaning method, this paper selects...

[0250] RMSE, MAE and Three indicators were used as metrics for data cleaning effectiveness. Voltage measurement data from 300 users with location tags in a certain region, spanning three months from January 1st to March 31st, 2025, were selected as the data cleaning object. The proposed hierarchical data cleaning method combining spatiotemporal clustering analysis and ISTA was compared with data cleaning methods based on attention mechanisms (Transformer), graph neural networks (GNN), and long short-term memory networks (LSTM). The RMSE, MAE, and R of the three data cleaning methods were compared. 2 As shown in Table 1.

[0251] Table 1 Data cleaning performance of each model

[0252]

[0253] (2) Data cleaning efficiency evaluation indicators

[0254] To demonstrate the superiority of the proposed data cleaning method in terms of cleaning efficiency, it was used, along with other data cleaning methods, to clean voltage measurement data of 300 users with location tags in a certain region and users with a contamination rate of 4% collected over a three-month period from January 1 to March 31, 2025. The data cleaning efficiency was evaluated as shown in Table 2, using the "dirty data" identification rate (i.e., the ratio of the number of "dirty data" identified to the total number of "dirty data") and calculation time as indicators.

[0255] Table 2 Data cleaning efficiency of each model

[0256]

[0257] (3) Results Comparison and Analysis

[0258] The results from Tables 1 and 2 are compiled into a visual graph, as shown below. Figure 4 As shown, the hierarchical data cleaning method combining spatiotemporal clustering analysis and ISTA proposed in this paper has the lowest RMSE and MAE compared with the other three data cleaning methods, and R... 2It has the largest data cleaning effect. Compared with Transformer, GNN, and LSTM, the RMSE of our method is reduced by 3.40%, 4.67%, and 2.29%, respectively, and the MAE is reduced by 7.91%, 8.42%, and 7.34%, respectively. R 2 The improvements were 1.46%, 1.74%, and 1.63%, respectively. Table 2 shows the cleaning efficiency of the four models; compared to Transformer, GNN, and LSTM, the proposed method improved the "dirty data" identification rate by 3.8%, 5%, and 1.3%, respectively, and reduced computation time by 26.7%, 34.6%, and 24.6%, respectively. Both evaluation metrics indicate that the improved ISTA algorithm optimizes the data cleaning performance better than other algorithms, demonstrating the significant superiority of the proposed method compared to the other three data cleaning methods.

[0259] This invention proposes a data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage, and verifies it through simulation with practical examples. The research results can provide a basis for innovative data cleaning methods. The main conclusions are as follows:

[0260] (a) Low-pressure operation and distribution data has heterogeneous characteristics. The spatiotemporal clustering hierarchical model can effectively identify the hierarchical layers and provide feedback on the hierarchical residuals. It divides the temporal data into abrupt change layer and a stable layer, and the spatial data into a high-density layer and a low-density layer, which helps to improve the data cleaning effect.

[0261] (b) The improved ISTA algorithm has excellent cleaning effect. Compared with deep learning methods such as Transformer, it has higher cleaning performance and "dirty data" identification rate, and less computation time.

[0262] In summary, the data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm of this invention has the following beneficial effects:

[0263] 1. This invention solves the problems of poor adaptability of traditional threshold and rule-based methods to dynamic spatiotemporal data, and the "over-cleaning" and "under-cleaning" of spatiotemporal data due to machine learning methods not fully considering the spatiotemporal hierarchy. It achieves precise and adaptive cleaning by using spatiotemporal clustering to create hierarchical layers (temporally divided into abrupt / stable layers, spatially divided into high / low density layers), thus identifying the inherent structure of the data. Based on this, differentiated ISTA cleaning strategies are implemented for different layers (strong regularization for abrupt / high density layers, and weak regularization for stable / low density layers), achieving a "targeted approach." This avoids the problems of traditional methods' "one-size-fits-all" approach, which erases normal subtle features (over-cleaning) or fails to handle hidden anomalies (under-cleaning), significantly improving the accuracy of cleaning.

[0264] 2. This invention addresses the shortcomings of existing methods in handling dynamic changes in data quality and their lack of self-optimization capabilities. It constructs a closed-loop governance framework of "clustering-cleaning-feedback," enhancing the method's adaptability and robustness. It innovatively introduces a hierarchical residual feedback mechanism. By calculating the residuals after cleaning at each layer, the cleaning effect is dynamically assessed. If the effect is unsatisfactory (residuals not converged), the clustering and cleaning control parameters (such as the number of clusters, neighborhood radius, regularization coefficient, etc.) are automatically optimized, and a new round of iteration is initiated. This enables the method to possess self-verification and continuous optimization capabilities, adapting to changes in data distribution and ensuring stable long-term application results, achieving a leap from "static cleaning" to "dynamic governance."

[0265] 3. This invention addresses the issue that existing machine learning methods suffer from high computational complexity, making it difficult to meet the efficiency requirements of large-scale data cleaning. This invention significantly improves the efficiency of large-scale spatiotemporal data processing while maintaining cleaning accuracy. The solution combines hierarchical processing with the efficient ISTA algorithm. First, hierarchical clustering decomposes the complex problem, reducing the data load and simplifying the problem. Second, the ISTA algorithm itself is an efficient iterative algorithm for solving L1 regularization problems, with low computational complexity and good convergence. The combination of these two approaches enables this solution to significantly reduce computation time while maintaining high cleaning accuracy when dealing with large-scale, multi-source, and heterogeneous spatiotemporal data in low-voltage distribution and operation scenarios. This provides feasible high-quality data support for real-time applications such as distribution network state estimation and demand response.

[0266] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0267] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm, characterized in that, Includes the following steps: Step S1: Obtain the raw power data to be cleaned in the low-voltage operation and distribution integration scenario; wherein, the raw power data includes time layer data and spatial layer data; Step S2: Use a clustering algorithm to subdivide the time layer data into abrupt change layer and a stationary layer, and subdivide the spatial layer data into a high-density layer and a low-density layer. Step S3: Use the iterative threshold shrinkage algorithm to clean the data of the mutation layer and the stable layer, as well as the data of the high-density layer and the low-density layer, to obtain the cleaned data of each layer; wherein, strong regularization constraint is applied to the mutation layer, and weak regularization constraint is applied to the stable layer; strong regularization constraint is applied to the high-density layer, and weak regularization constraint is applied to the low-density layer. Step S4: Calculate the hierarchical residuals of the cleaned mutation layer, stable layer, high-density layer, and low-density layer data respectively to obtain the final residuals; if the final residuals are less than the preset residual threshold, output clean data; otherwise, optimize the clustering and cleaning control parameters and return to step S2 for iteration.

2. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 1, characterized in that, Step S2 includes: Extract the temporal features of the time-layer data, and subdivide the time-layer data into abrupt change layer and stationary layer data based on the K-means clustering algorithm; The spatial features of the spatial layer data are extracted, and the spatial layer data is further subdivided into high-density layer and low-density layer data based on the DBSCAN clustering algorithm.

3. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 1, characterized in that, The time-layer data is further subdivided into a mutation layer and a stationary layer based on the K-means clustering algorithm, including: Extracting wavelet energy from time-layer data for each time window With local outliers , obtained the The feature vectors of each time window are ; The feature vectors of all time windows constitute the input dataset for the K-means clustering algorithm. ;in, Number of time windows; Random initialization Cluster centers ; Calculate the Euclidean distance from the feature vector of each time window to any cluster center; The feature vector of each time window is assigned to the cluster corresponding to the cluster center with the nearest Euclidean distance; Once the feature vectors for all time windows have been assigned, the mean of all feature vectors within each cluster is used as the new cluster center. For multiple randomly initialized Value, calculate each The value corresponds to the DB index, and the number of clusters with the smallest DB index is selected. As the optimal clustering number ; extract The wavelet energy and LOF of each cluster center are used to form the wavelet energy set of the cluster centers and the LOF value set of the center points; Based on the cluster center wavelet energy set and the center point LOF value set, respectively, the wavelet energy threshold and LOF threshold are obtained using the median principle; For each cluster center, if the wavelet energy is greater than the wavelet energy threshold and the LOF value is greater than the LOF threshold, the cluster is determined to be a mutation layer; if the wavelet energy is less than the wavelet energy threshold and the LOF value is less than the LOF threshold, the cluster is determined to be a stationary layer. Based on the defined mutation layer reference point and stationary layer reference point, a cluster center that does not meet the conditions of either mutation layer or stationary layer is determined to be a mutation layer if the distance from the mutation layer reference point to the cluster center is less than the distance to the stationary layer reference point; otherwise, it is a stationary layer.

4. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 3, characterized in that, The reference points for the mutation layer are points formed by the maximum wavelet energy and the maximum LOF value of all cluster centers, as shown below: ； The reference point for the stable layer is a point constructed based on the minimum wavelet energy and the minimum LOF value of all cluster centers, as shown below: ； in, Wavelet energy value for each cluster center; The LOF value for each cluster center.

5. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 2, characterized in that, The spatial layer data is further subdivided into high-density and low-density layers based on the DBSCAN clustering algorithm, including: The spatial layer data includes the location data of multiple devices; Calculate the Haversine distance between any two device points based on location data; Calculate the kernel density estimate of the device points based on the Haversine distance between them; The location data of each device and the kernel density estimate constitute the composite spatial feature vector of each device; Set neighborhood radius and minimum points DBSCAN clustering is performed based on the composite spatial feature vector of each device to obtain the labeling results of whether the device points are core points, boundary points or noise points; The connected set consisting of devices marked as core points and their associated boundary points is determined as high-density layer devices; the data corresponding to high-density layer devices is high-density layer data. The set of devices marked as noise points is identified as low-density layer devices; the data corresponding to low-density layer devices is low-density layer data.

6. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 1, characterized in that, The iterative threshold shrinkage algorithm is used to clean the data of the mutation layer and the stationary layer respectively, resulting in cleaned mutation layer and stationary layer data, including: A first objective function for cleaning the data of the mutation layer and the stationary layer is constructed; iterative cleaning is performed using an iterative threshold shrinkage algorithm until the first objective function converges, and the cleaned mutation layer and stationary layer data are obtained.

7. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 6, characterized in that, The high-density layer and low-density layer data are cleaned using an iterative threshold shrinkage algorithm, resulting in cleaned high-density layer and low-density layer data, including: A second objective function for cleaning high-density and low-density layer data is constructed; iterative cleaning is performed using an iterative threshold shrinkage algorithm until the second objective function converges, resulting in cleaned high-density and low-density layer data.

8. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to claim 1, characterized in that, Calculate the layering residuals of the cleaned abrupt change layer, stable layer, high-density layer, and low-density layer, respectively. , , and ; Obtain the final residual ; like Output clean data for the cleaned mutation layer, stationary layer, high-density layer, and low-density layer; among which This is the preset residual threshold.

9. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to any one of claims 1-8, characterized in that, The clustering and cleaning control parameters include: the number of clusters in K-means clustering and the neighborhood radius in DBSCAN clustering. With minimum number of points And the strong regularization coefficient and weak regularization coefficient of the iterative threshold shrinkage algorithm.

10. The data cleaning method combining spatiotemporal clustering and iterative threshold shrinkage algorithm according to any one of claims 1-8, characterized in that, The time-layer data includes voltage, current, or active power data; The spatial layer data includes the location data of electricity meters and distribution transformers, or the power grid topology diagram.

Images