Data cleaning method, system and equipment combining K-nearest neighbor detection and tensor decomposition, and medium

Abstract

The invention discloses a data cleaning method, system, equipment and medium in combination with K-nearest neighbor detection and tensor decomposition, and belongs to the technical field of power system data processing.The method comprises the steps that operation data of all stations in an intelligent power grid system are obtained, and an original data set is formed based on multiple time sequences and multiple measurement indexes; performing consistency and regularization processing on the original data set to obtain a standard data set; and on the basis of the standard data set, identifying abnormal data and performing missing value completion on data corresponding to an abnormal value by adopting a dual-threshold discrimination method fusing local density estimation and a K-nearest neighbor distance to obtain a cleaned data set. According to the method, hidden abnormal points are identified through dual detection of the local density and the neighbor relation, normal and abnormal modes can be distinguished more accurately even under the conditions of complex data distribution and variable neighborhood structures, the possibility of misjudgment and missing detection is reduced, and a data base is more reliable.

Description

Technical Field

[0001] This invention relates to the field of power system data processing technology, specifically to a method, system, device, and medium for data cleaning that combines K-nearest neighbor detection and tensor decomposition. Background Technology

[0002] With the increasing prevalence of smart grid construction, sensing and edge computing devices are ubiquitous, making the data generated by grid operations exceptionally complex—multi-dimensional, frequently updated, and diverse in data type. This data includes information such as power plant distribution, operational status at different times, and various measurement indicators, forming a crucial foundation for the grid's core tasks of load forecasting, optimized operation, and intelligent dispatching. However, several issues remain unresolved when processing this grid big data, hindering the improvement of data quality and its full potential: first, how to accurately identify anomalies in high-dimensional data; second, how to preserve the multi-dimensional structural information within the data during processing; and third, how to ensure efficient coordination throughout the entire processing workflow.

[0003] Looking at existing technologies, traditional data cleaning methods mostly rely on pre-defined rule templates or fixed numerical standards, which are simply unable to keep up with the constantly changing structure and characteristics of power grid data. Especially when dealing with high-dimensional data from different power plants and at different time scales, their ability to identify anomalies hidden in sparse or boundary regions is particularly weak, easily leading to misjudgments or omissions. Furthermore, current mainstream data compression methods focus on linear dimensionality reduction techniques such as principal component analysis and singular value decomposition. These techniques can only handle two-dimensional tabular data and cannot account for the complex relationships between power grid data in space (e.g., different power plants), time (different time periods), and measurement indicators. This results in the loss of much useful information during data compression, and significant errors when the data is subsequently restored. Summary of the Invention

[0004] To address the aforementioned technical challenges, a data cleaning method combining K-nearest neighbor detection and tensor decomposition is proposed. The method includes: acquiring operational data from various stations within the smart grid system; forming an original dataset based on multiple time series and measurement indicators; performing consistency and normalization processing on the original dataset to obtain a standardized dataset; using a dual-threshold discrimination method integrating local density estimation and K-nearest neighbor distance based on the standardized dataset to identify anomalous data and fill in missing values ​​for the corresponding anomalous values, thus obtaining a cleaned dataset; constructing a three-dimensional tensor structure object set based on station number, time series, and measurement indicator dimensions; performing tensor Tucker decomposition on the three-dimensional tensor structure object set to obtain the core tensor and the truncation factor matrix corresponding to each dimension; and reconstructing the three-dimensional tensor structure object set using the core tensor and truncation factor matrix to output the cleaned power grid data.

[0005] As a preferred embodiment of the data cleaning method combining K-nearest neighbor detection and tensor decomposition described in this invention, the method involves: performing consistency and normalization processing on the original data set to obtain a normalized data set, including: performing data format standardization processing on the original data set to obtain first standardized data; performing imputation operation on missing items in the first standardized data to obtain second imputed data; and performing normalization operation on the second imputed data to obtain a normalized data set.

[0006] As a preferred embodiment of the data cleaning method combining K-nearest neighbor detection and tensor decomposition described in this invention, the method includes: based on the standardized data set, using a dual-threshold discrimination method that integrates local density estimation and K-nearest neighbor distance, identifying abnormal data and imputing missing values ​​in the data corresponding to the abnormal values ​​to obtain a cleaned data set. This includes: calculating the Euclidean distance from each data element in the standardized data set to other elements; determining the K-nearest neighbor set for each element based on this distance; calculating the local density of each element based on its K-nearest neighbor set and simultaneously determining its K-nearest neighbor distance; calculating a local density threshold and a K-nearest neighbor distance threshold based on the local density and K-nearest neighbor distance of all elements; identifying data elements with a local density less than the local density threshold and a K-nearest neighbor distance greater than the K-nearest neighbor distance threshold as abnormal values; and imputing missing values ​​in the data corresponding to the abnormal values ​​to obtain the cleaned data set.

[0007] As a preferred embodiment of the data cleaning method combining K-nearest neighbor detection and tensor decomposition described in this invention, the step of calculating the local density threshold and the K-nearest neighbor distance threshold based on the local density and K-nearest neighbor distance of all elements includes: determining the local density offset and the K-nearest neighbor distance offset based on the data size of the normalized dataset and the distance relationship between outliers and cluster centers; calculating the average local density of all data elements and combining it with the local density offset to obtain the local density threshold; and calculating the average K-nearest neighbor distance of all data elements and combining it with the K-nearest neighbor distance offset to obtain the K-nearest neighbor distance threshold.

[0008] The beneficial effects of this preferred technical solution are as follows: By fusing the dual measures of local density and proximity distance, and adaptively adjusting the average benchmark using offset, the threshold setting is made more closely aligned with the internal structure of the dataset, improving the sensitivity and specificity of identifying outliers from complex distributions, and avoiding misjudgments or missed detections that may be caused by a single criterion. The offset parameter is dynamically determined based on the geometric relationship between the data size and the outlier's deviation from the cluster center, ensuring that the threshold calculation process is not static and can adapt to changes in the dispersion and size of different datasets, thus enhancing the method's generalization ability and robustness when dealing with heterogeneous data.

[0009] As a preferred embodiment of the data cleaning method combining K-nearest neighbor detection and tensor decomposition described in this invention, the step of constructing a three-dimensional tensor structure object set based on site number, time series, and measurement index dimensions includes: interleaving the cleaned data set according to site order, time order, and measurement index order to form an original tensor; performing modular expansion on the original tensor along the site dimension to obtain a first modular expansion matrix; performing modular expansion on the original tensor along the time dimension to obtain a second modular expansion matrix; and performing modular expansion on the original tensor along the measurement index dimension to obtain a third modular expansion matrix.

[0010] As a preferred embodiment of the data cleaning method combining K-nearest neighbor detection and tensor decomposition described in this invention, the method involves: performing tensor Tucker decomposition on the three-dimensional tensor structure object set to obtain the core tensor and the truncation factor matrices corresponding to each dimension, including: performing singular value decomposition transformation on the first modulus expansion matrix to obtain the first truncated left singular vector matrix; performing singular value decomposition transformation on the second modulus expansion matrix to obtain the second truncated left singular vector matrix; performing singular value decomposition transformation on the third modulus expansion matrix to obtain the third truncated left singular vector matrix; and performing Tucker decomposition operation on the original tensor based on the first truncated left singular vector matrix, the second truncated left singular vector matrix, and the third truncated left singular vector matrix to obtain the core tensor.

[0011] As a preferred embodiment of the data cleaning method combining K-nearest neighbor detection and tensor decomposition described in this invention, the method involves: reconstructing a three-dimensional tensor structure object set using the core tensor and truncation factor matrix to output cleaned power grid data. This includes: performing a modal product between the core tensor and a first truncated left singular vector matrix along the station dimension; performing a modal product between the first modal product result and a second truncated left singular vector matrix along the time dimension; performing a modal product between the second modal product result and a third truncated left singular vector matrix along the measurement index dimension to complete the reconstruction; and outputting the reconstructed cleaned power grid data.

[0012] The beneficial effects of this preferred technical solution are as follows: By reconstructing the data through the modal product of the core tensor and the truncation factor matrix, outliers and noise components in the original power grid data are effectively removed, improving data quality. This reconstruction process preserves the inherent structural characteristics of the data, enhancing the reliability of subsequent analysis while avoiding information loss. The combination of tensor decomposition and K-nearest neighbor detection enables accurate identification and cleaning of anomalous patterns in multidimensional spatiotemporal data, improving the adaptability of data processing. This hierarchical reconstruction mechanism optimizes computational efficiency, enabling efficient cleaning while maintaining the spatiotemporal correlation of power grid data.

[0013] As a preferred embodiment of the data cleaning system combining K-nearest neighbor detection and tensor decomposition described in this invention, it is characterized by comprising a raw data acquisition module, a data preprocessing module, an anomaly detection and cleaning module, a tensor construction module, a tensor decomposition and compression module, and a data reconstruction and output module. The raw data acquisition module acquires operational data from various stations in the smart grid system, forming a raw data set based on multiple time series and multiple measurement indicators. The data preprocessing module performs consistency and normalization processing on the raw data set to obtain a normalized data set. The anomaly detection and cleaning module, based on the normalized data set, employs a fusion local... A dual-threshold discrimination method based on density estimation and K-nearest neighbor distance is used to identify abnormal data and fill in missing values ​​for the data corresponding to the abnormal values ​​to obtain a cleaned data set. The tensor construction module is used to construct a three-dimensional tensor structure object set based on the station number, time series, and measurement index dimensions. The tensor decomposition and compression module is used to obtain the core tensor and the truncation factor matrix corresponding to each dimension by performing tensor Tucker decomposition on the three-dimensional tensor structure object set. The data reconstruction output module is used to reconstruct the three-dimensional tensor structure object set using the core tensor and the truncation factor matrix to output the cleaned power grid data.

[0014] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the data cleaning method combining K-nearest neighbor detection and tensor decomposition.

[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the data cleaning method combining K-nearest neighbor detection and tensor decomposition.

[0016] The beneficial effects of this invention are: by detecting both local density and neighbor relationships, hidden anomalies can be identified. Even in situations with complex data distribution and varied neighborhood structures, normal and abnormal patterns can be distinguished more accurately, reducing the possibility of misjudgment and missed detection, and making the data base more reliable.

[0017] The cleaned multi-dimensional data is constructed into tensors and decomposed to capture the interaction between sites, time and indicators. By compressing and reconstructing to filter out residual noise, the information is purified and condensed while maintaining the essential structure of the data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating a data cleaning method combining K-nearest neighbor detection and tensor decomposition, as provided in one embodiment of the present invention. Detailed Implementation

[0020] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0021] Example 1, referring to Figure 1 This is the first embodiment of the present invention, which provides a data cleaning method combining K-nearest neighbor detection and tensor decomposition, comprising: S1. Obtain the operation data of each station in the smart grid system and form a raw data set based on multiple time series and multiple measurement indicators.

[0022] S2. Perform consistency and normalization processing on the original dataset to obtain a normalized dataset.

[0023] S3. Based on the standardized dataset, a dual-threshold discrimination method that integrates local density estimation and K-nearest neighbor distance is used to identify abnormal data and fill in the missing values ​​of the data corresponding to the abnormal values ​​to obtain a cleaned dataset.

[0024] S4. Based on the site number, time series, and measurement index dimensions, the cleaned data set is constructed into a three-dimensional tensor structure object set.

[0025] S5. By performing tensor Tucker decomposition on the three-dimensional tensor structure object set, the core tensor and the truncation factor matrix corresponding to each dimension are obtained.

[0026] S6. Reconstruct the three-dimensional tensor structure object set using the core tensor and truncation factor matrix, and output the cleaned power grid data.

[0027] It should be noted that in traditional smart grid big data cleaning, the common approach is to first identify anomalies in the data by manually setting rule templates or fixed numerical thresholds, and then use linear dimensionality reduction techniques such as principal component analysis and singular value decomposition to compress the data separately. While this approach is clear and easy to operate, it has significant drawbacks: it relies heavily on human experience, is poorly adaptable to constantly changing data structures, and cannot accurately identify hidden outliers in high-dimensional power grid data, leading to a decrease in the accuracy of data cleaning.

[0028] Therefore, to address the aforementioned issues of poor robustness in anomaly detection, significant information loss during compression, and low efficiency due to fragmented processing, steps S1-S6 are used to perform consistency organization and standardization on the original data, laying a reliable foundation for subsequent analysis. By combining a dual-threshold linkage mechanism based on local data density and adjacent data distance, abnormal data is accurately identified and completed, significantly improving the accuracy of identifying hidden and atypical anomalies in high-dimensional power grid data while ensuring data integrity.

[0029] Example 2, refer to Figure 1 This is the second embodiment of the present invention, which differs from the first embodiment in that: a data cleaning method combining K-nearest neighbor detection and tensor decomposition further includes: Furthermore, step S1 includes the following steps: The process involves acquiring operational data from various substations within a smart grid system. This data typically originates from historical operational databases of the power system, real-time monitoring and data acquisition systems, and various intelligent terminal devices. It encompasses recorded sequences of various electrical measurement indicators, such as voltage, current, active power, reactive power, frequency, and power factor, at different sampling times. After data acquisition and aggregation, a raw dataset can be formed based on multiple continuous time series and diverse measurement indicator dimensions. For example, researchers can collect operational data from 200 smart substations under a provincial power grid for a continuous year. Each substation records its outgoing line voltage, total active power, and total reactive power at 15-minute intervals (96 time points per day, totaling 365 days; each substation constitutes a time series containing voltage, active power, and reactive power). The data from all substations is then aligned by time and indicators to form a three-dimensional data structure: the first dimension is the substations (200 substations), the second dimension is time (365 days × 96 points / day = 35040 time points), and the third dimension is the measurement indicators (voltage, active power, and reactive power). This three-dimensional dataset serves as the raw input for subsequent data cleaning and compression, fully preserving the multi-dimensional joint characteristics of smart grid data in space, time, and attributes.

[0030] In this embodiment, the raw data set in step S1 can be a data cube synchronously collected from various stations and organized in time sequence through a unified data interface. Specifically, the data acquisition server deployed in the dispatch center simultaneously initiates data requests to all designated monitoring stations across the network according to a preset global clock and sampling period (e.g., once per minute), receives standardized data packets uploaded by each station, parses out the timestamps, station IDs, and multiple measurement values, and fills these data into a predefined three-dimensional array according to the order of the timestamps and the index order of the station IDs, forming a structurally well-organized raw data set.

[0031] In one alternative implementation, the original dataset can also be a dynamic dataset formed by real-time alignment and integration of asynchronously reported heterogeneous data through a stream processing platform. Specifically, monitoring devices from different manufacturers or of different types may send data to cloud or edge computing nodes at different frequencies and protocols. First, the window functions and timestamp watermarking mechanism of a stream processing engine (such as Apache Flink) are used to time-align and buffer the data streams from different sources. Then, the aligned data is aggregated according to three keys: site, aligned time point, and measurement index, generating and updating a three-dimensional tensor in real time as the original dataset for subsequent consumption.

[0032] In another alternative implementation, the raw dataset can also be an enhanced dataset extracted from massive historical archives and integrated with relevant environmental context for a specific analytical task. Specifically, in addition to the power grid's own operational data, it also retrieves temperature, humidity, wind speed, and light intensity data for the corresponding time period and region from meteorological databases, and retrieves equipment maintenance plans and network topology change records from production management systems. This contextual information is organized together with the electrical quantity data as additional measurement indicator dimensions or independent attribute dimensions to form a raw dataset with richer information and potentially higher dimensions, in order to support more comprehensive situational awareness and in-depth analysis.

[0033] In this embodiment, the operational data in step S1 refers to a set of structured data extracted from the real-time and historical databases of the power grid dispatch automation system and updated according to a fixed scanning cycle. Specifically, the data acquisition service obtains electrical measurements (telemetry) and status signals (telecommunications) from remote terminal units deployed in each substation through periodic inspections or change uploads.

[0034] In one alternative implementation, the operational data specifically refers to a set of multiple time series consisting of high-frame-rate phasor data provided by a wide-area measurement system and absolutely time-synchronized. Specifically, synchronous phasor measurement units distributed across key nodes of the entire network synchronously measure and calculate dynamic quantities such as voltage / current phasors, frequency, and power at a rate of tens to hundreds of frames per second. These data streams are uploaded in real time via a dedicated communication network and, based on their precise time stamps, are rigorously aligned to a unified time axis in the data center. This forms a high-dimensional time series set that is strictly synchronized in time, widely distributed in space, and capable of accurately characterizing the electromechanical dynamic processes of the power grid, providing a high-precision data foundation for analyzing the dynamic behavior of the power grid.

[0035] In another optional implementation, operational data also refers to a comprehensive dataset formed by comprehensively collecting raw waveforms, triggering events, and derived features generated during primary equipment operation, tailored to specific analytical scenarios (such as fault diagnosis or equipment condition assessment). Specifically, for critical equipment (such as important transformers, GIS, etc.), the system synchronously collects and integrates instantaneous voltage / current waveforms recorded by its fault recording device, action events and recording data generated by protection devices, and equipment status parameters (such as gas characteristics, partial discharge, and oil chromatography data) provided by online monitoring devices. These heterogeneous data are aligned and correlated in time based on event triggering, and in terms of indicators, they integrate raw waveforms, event signals, and derived deep state features, together forming a multi-dimensional operational data set that can comprehensively depict the electrical and physical state of the equipment within a specific time period, providing a basis for accurate diagnosis and analysis.

[0036] Furthermore, in step S2, the original dataset undergoes consistency and normalization processing to obtain a normalized dataset, including the following steps A1-A3: A1. Perform data format standardization on the original dataset to obtain the first standardized data.

[0037] A2. Perform imputation on the missing items in the first standardized data to obtain the second imputed data.

[0038] A3. Normalize the data after the second filling to obtain a normalized data set.

[0039] In this embodiment of the application, the data format standardization process in step A1 can be to convert all data items into a unified numerical type and unit system. Specifically, it identifies possible string-type status flags (such as normal, alarm) and power values ​​in different units (such as kW and MW) in the original data, maps them to numerical codes (such as 0 for normal and 1 for alarm) and converts them to a unified standard unit (such as unified to MW), ensuring that all elements in the data matrix are calculable numerical values.

[0040] In one alternative implementation, data format standardization may also include timestamp normalization. Specifically, timestamps from different data sources that may use different time zones or formats (such as 2023-01-01 08:00:00 and 20230101.080000) are all converted into a unified UTC timestamp and standard format, providing an accurate time index for subsequent tensor construction along the time dimension.

[0041] In another alternative implementation, data format standardization can also be a data validity verification and correction for specific measurement indicators. Specifically, for physical quantities such as current and voltage, a threshold is set according to their reasonable value range (e.g., voltage should be between 0.9 and 1.1 times the rated voltage), and abnormal values ​​that significantly exceed this range are marked as missing or invalid items to be processed, providing a target for the filling operation in subsequent step A2.

[0042] Specifically, in step A2, the missing items are filled in. The missing items can be estimated by using linear interpolation based on the data trend of the time period before and after the missing point, or by using the average value of the historical data of the same period at the same time of the site. The specific method can be selected by those skilled in the art based on data continuity and business needs.

[0043] Specifically, the normalization operation in step A3 can use the minimum-maximum normalization method to linearly transform the data of each measurement index to a fixed interval, thereby eliminating the influence of different indicators on subsequent distance calculation and model decomposition due to differences in dimensions and orders of magnitude.

[0044] This invention eliminates data inconsistencies caused by heterogeneous devices, communication anomalies, or differences in units by standardizing the format of the original data, filling in missing values, and normalizing the data. This forms a data foundation with a unified structure and comparable values, providing stable and reliable input for subsequent anomaly detection based on distance metrics and compression reconstruction based on numerical decomposition, thereby ensuring the execution accuracy and effectiveness of the entire method process.

[0045] Furthermore, in step S3, based on the normalized dataset, a dual-threshold discrimination method fusing local density estimation and K-nearest neighbor distance is used to identify outlier data and fill in missing values ​​for the data corresponding to the outliers, resulting in a cleaned dataset, including the following steps B1-B5: B1. Calculate the Euclidean distance from each data element in the canonical dataset to other elements, and determine the K nearest neighbor set for each element based on this distance.

[0046] B2. Calculate the local density of each element based on its K nearest neighbor set, and determine its K nearest neighbor distance.

[0047] B3. Calculate the local density threshold and K-nearest neighbor distance threshold based on the local density and K-nearest neighbor distance of all elements.

[0048] B4. Data elements whose local density is less than the local density threshold and whose K-nearest neighbor distance is greater than the K-nearest neighbor distance threshold are identified as outliers.

[0049] B5. Fill in the missing values ​​in the data corresponding to the outlier to obtain the cleaned data set.

[0050] In this embodiment, when calculating the Euclidean distance in step B1, a data element can refer to a complete multidimensional data point in a standardized dataset. Specifically, for the standardized dataset processed in step S2, it is considered as a matrix containing multiple samples and feature dimensions. The Euclidean distance between the row vectors in the matrix is ​​calculated to determine the K nearest neighbor set of each sample data vector. The distance calculation measures the similarity of the overall power grid state vectors at different time points. The smaller the distance between two row vectors, the more similar their corresponding power grid operating states are. Determining the K nearest neighbor set of each element means finding the K time point data vectors that are most similar to its state.

[0051] In one alternative implementation, when calculating the Euclidean distance, the data element can also be a scalar value for a single measurement indicator at a specific site and time point. Specifically, the three-dimensional specification data is expanded along the site and indicator dimensions to form a time-indexed sequence. The distance of the specific data point at each time point to the same data value at all other time points is calculated, thereby identifying anomalous fluctuations of the data point in the time series. This approach focuses on the time-series variation pattern of a single electrical quantity (such as "the 110kV bus voltage of substation A"). By calculating the Euclidean distance (i.e., the norm of the absolute difference sequence) between the current value and the historical value sequence of the indicator, the K historical moments with similar patterns to the indicator are identified, forming its K nearest neighbor set, which is specifically used to detect isolated anomalies or trend changes at that specific measurement point.

[0052] In another alternative implementation, a weighting mechanism can be introduced when calculating the Euclidean distance to reflect the differences in importance among different indicators. Specifically, when calculating the distance between data element vectors, different weights are assigned to each measurement indicator based on business knowledge, and a weighted Euclidean distance is calculated. This allows indicators more critical to system operation to have a higher weight in anomaly detection. For example, in condition assessment, voltage and frequency stability are usually more critical than the active power of a single line, so voltage and frequency indicators can be given higher weights, while ordinary power measurements can be given lower weights. The weighted distance calculation allows the focus of anomaly detection to be more closely integrated with the actual needs of power grid safe operation.

[0053] Specifically, in step B2, local density is used to quantify the sample density of the neighborhood of each data element. A higher local density indicates that the element is more likely to belong to a mainstream data distribution cluster. The K-nearest neighbor distance reflects the distance between the element and its Kth most similar neighbor. A larger distance indicates that the element is more likely to be located on the edge of its cluster or an isolated point far from other clusters. Local density and K-nearest neighbor distance together characterize the local structural position of data elements in the feature space.

[0054] Specifically, in step B3, the calculation of the local density threshold and the K-nearest neighbor distance threshold may include: determining the local density offset and the K-nearest neighbor distance offset based on the data size of the normalized dataset and the distance relationship between outliers and cluster centers; calculating the average local density of all data elements and combining it with the local density offset to obtain the local density threshold; and calculating the average K-nearest neighbor distance of all data elements and combining it with the K-nearest neighbor distance offset to obtain the K-nearest neighbor distance threshold. This offset mechanism allows for dynamic adjustment of the judgment threshold based on the overall distribution characteristics of the data. For example, for datasets with relatively dispersed data point distribution, the K-nearest neighbor distance offset can be appropriately increased to reduce threshold sensitivity and avoid misjudging too many boundary points as outliers.

[0055] Specifically, in step B4, data elements whose local density is less than the local density threshold and whose K-nearest neighbor distance is greater than the K-nearest neighbor distance threshold are identified as outliers. This criterion means that if a data point simultaneously meets the conditions of "being located in a low-density area" and "being far from its nearest neighbor," it is highly likely to be an outlier that deviates from the normal pattern or belongs to a rare operating condition, such as a power spike caused by a momentary equipment failure or erroneous data caused by a communication error.

[0056] Specifically, in step B5, missing values ​​are filled in for the data corresponding to the outlier, resulting in a cleaned dataset. The completion operation aims to repair the data points identified as anomalous, generating a continuous and complete dataset for subsequent steps. The completion method can be selected based on the context of the outlier. For example, for transient, isolated outliers, linear or spline interpolation based on the preceding and following normal data can be used; for a continuous outlier caused by equipment failure, the mean or regression prediction value based on data from similar equipment during the same period or historical data under the same operating conditions can be used. The final cleaned dataset replaces outliers with reasonable estimates, thereby improving data integrity and consistency.

[0057] This invention uses a dual-threshold discrimination mechanism that integrates local density and K-nearest neighbor distance to identify and complete anomalies. It can effectively identify atypical outliers in high-dimensional smart grid data that are located in sparse regions, have hidden forms, or deviate significantly from the mainstream pattern.

[0058] In step B1, a data element refers to a multidimensional vector in the standardized data set that is jointly determined by the site, time, and measurement indicators.

[0059] In the smart grid dataset to be processed, calculate the data elements. Euclidean distance between the data and other data elements in the dataset If the first corresponding to this dataset... The Euclidean distance is Then data element The k-th nearest neighbor can be represented as: In step B2, the nearest neighbor is used to calculate the data element. The local density is calculated using the following formula: In the formula, N represents the total number of elements in the smart grid dataset; p represents the percentage of N samples; and K is a parameter determined by N and p. Local density The larger the value, the more likely it is that the corresponding data element in the dataset is... The higher the density of the area.

[0060] Based on the above formula, the KNN distance can be defined as follows: The K-nearest neighbor distance for each data element refers to the distance between that element and its Kth nearest neighbor. The distance between them. Based on the formula, the distance of each element in the power grid data can be calculated. Values. In obtaining all elements in the smart grid dataset... After obtaining the values, these values ​​are plotted on a planar coordinate system to form a decision scatter plot of the elements, thereby identifying outliers in the dataset.

[0061] In step B3, within the dataset, if a data element has a high local density and its corresponding K-nearest neighbor distance is also large, then this element can be considered a cluster center, which can then be used to identify outliers. The criteria for determining whether an element with a local density less than that of the cluster center is an outlier are as follows: If the local density of the data to be determined Less than the local density threshold, and its K-nearest neighbor distance is... If the distance to a value is greater than the K-nearest neighbor threshold, the data is considered an outlier in the smart grid dataset. The criteria for this judgment are as follows: In the formula, Indicates the local density threshold; Indicates the K-nearest neighbor distance threshold; It is the total amount of data in the smart grid dataset; and It is an empirical parameter, and its specific value is determined by factors such as the size of the power grid dataset and the distance between outliers and cluster centers.

[0062] After completing the detection of outliers in the smart grid data, the missing data in the dataset will be processed.

[0063] Furthermore, in step S4, based on the site number, time series, and measurement index dimensions, the cleaned data set is constructed into a three-dimensional tensor structure object set, including the following steps C1-C4: C1. The cleaned data set is interleaved and sampled according to the site order, time order, and measurement index order to form the original tensor.

[0064] C2. Perform a modular expansion of the original tensor along the station dimension to obtain the first modular expansion matrix.

[0065] C3. Perform a modular expansion of the original tensor along the time dimension to obtain the second modular expansion matrix.

[0066] C4. Perform a modular expansion of the original tensor along the measurement index dimension to obtain the third modular expansion matrix.

[0067] In this embodiment, the three-dimensional tensor constructed in step C1 can be defined and constructed by prioritizing the arrangement and combination of the three dimensions—site, time, and measurement index—in a preset order to form a structured three-dimensional array. Specifically, the site dimension is fixed as the first dimension, the time dimension as the second dimension, and the measurement index dimension as the third dimension. Following the predetermined order of "site-time-measurement index," all measurement index values ​​for each site at each moment are filled into the corresponding "data cube" of the three-dimensional array, thus forming a structurally regular and dimensionally clear original tensor. This method directly maps to the most common logical structure of smart grid data, facilitating understanding and subsequent processing.

[0068] In one alternative implementation, when constructing a three-dimensional tensor, the definition and construction method of the dimensions can be adjusted according to the focus of the analysis or the characteristics of the data source, adjusting the order of the three dimensions: site, time, and measurement indicator. Specifically, if the analysis focuses on the global evolution pattern of the time series, the time dimension can be defined as the first dimension, the site dimension as the second dimension, and the measurement indicator dimension as the third dimension, constructing the tensor in the order of "time-site-measurement indicator". This construction method makes the first dimension of the tensor directly represent the time series, facilitating holistic analysis along the time direction, such as directly observing the temporal evolution trend of a certain measurement indicator across all sites in the entire network.

[0069] In another alternative implementation, when constructing a three-dimensional tensor, the definition and construction of dimensions can be based on the three core dimensions of site, time, and measurement indicators. An auxiliary fourth dimension is introduced to characterize the data type, source, or batch, thus constructing a four-dimensional or higher-dimensional tensor. Specifically, in addition to the three core three-dimensional tensor (site × time × measurement indicator), a "data category" dimension is added to distinguish data from different data sources (such as SCADA systems, PMU systems), different data batches (such as training sets, test sets), or different operating modes (such as normal state, fault state). This construction method organizes data of different categories but with the same structure within a unified tensor framework, facilitating batch processing, comparative analysis, or cross-category information fusion.

[0070] Specifically, in step C2, expanding the original tensor along the station dimension means arranging all the station slices in the three-dimensional tensor sequentially in the row direction to form a large matrix with the number of rows equal to the total number of stations and the number of columns equal to the product of the number of time points and the number of measurement indicators.

[0071] Specifically, in step C3, expanding the original tensor along the time dimension means arranging all the time slices in the three-dimensional tensor sequentially in the row direction to form a large matrix with the number of rows equal to the total number of time points and the number of columns equal to the product of the number of stations and the number of measurement indicators.

[0072] Specifically, in step C4, performing a modular expansion of the original tensor along the measurement index dimension means arranging all index slices in the three-dimensional tensor sequentially in the row direction to form a large matrix with the number of rows equal to the total number of measurement indexes and the number of columns equal to the product of the number of stations and the number of time points.

[0073] This invention fundamentally changes the traditional method of flattening multidimensional data into matrices for processing by constructing cleaned data into three-dimensional tensor objects based on its inherent business dimensions (site, time, measurement indicators). This tensor representation can completely and accurately preserve the inherent structure and coupling relationships of smart grid data in three orthogonal directions: spatial distribution, temporal evolution, and physical attributes.

[0074] Furthermore, in step S5, tensor Tucker decomposition is performed on the three-dimensional tensor structure object set to obtain the core tensor and the truncation factor matrix corresponding to each dimension, including the following steps D1-D4: D1. Perform singular value decomposition on the first modular expansion matrix to obtain the first truncated left singular vector matrix.

[0075] D2. Perform singular value decomposition on the second modular expansion matrix to obtain the second truncated left singular vector matrix.

[0076] D3. Perform singular value decomposition on the third modular expansion matrix to obtain the third truncated left singular vector matrix.

[0077] D4. Perform Tucker decomposition on the original tensor based on the first truncated left singular vector matrix, the second truncated left singular vector matrix, and the third truncated left singular vector matrix to obtain the core tensor.

[0078] Specifically, in step D1, performing singular value decomposition (SVD) on the first modular expansion matrix refers to linearly reducing the dimensionality of the feature matrix at the site dimension. The rows of the first modular expansion matrix correspond to all sites, and the columns correspond to the joint features of all time points and all measurement indicators. Through singular value decomposition and truncation according to a preset site dimension truncation threshold, the top few largest singular values ​​and their corresponding left singular vectors are selected to form the first truncated left singular vector matrix. The number of rows in this matrix equals the original total number of sites, and the number of columns equals the number of principal components of the sites retained after truncation. Its column vectors represent the most representative spatial feature patterns extracted at the site dimension, thus compressing the data from numerous sites into a linear combination of feature patterns from a few sites.

[0079] Specifically, in step D2, performing singular value decomposition (SVD) on the second modular expansion matrix refers to linearly reducing the dimensionality of the feature matrix in the time dimension. The rows of the second modular expansion matrix correspond to all time points, and the columns correspond to the joint features of all stations and all measurement indicators. Through singular value decomposition and truncation according to a preset time dimension truncation threshold, the top few largest singular values ​​and their corresponding left singular vectors are selected to form the second truncated left singular vector matrix. The number of rows in this matrix equals the total number of original time points, and the number of columns equals the number of principal time components retained after truncation. Its column vectors represent the most prominent dynamic evolution patterns of the data in the time dimension (such as typical daily load curves, seasonal fluctuation patterns, etc.), thereby compressing the lengthy time series into a weighted sum of a few time-series basis functions.

[0080] Specifically, in step D3, performing singular value decomposition (SVD) on the third-mode expansion matrix refers to linearly reducing the dimensionality of the feature matrix of the measurement index dimensions. The rows of the third-mode expansion matrix correspond to all measurement indices, and the columns correspond to the joint features of all stations and all time points. Through singular value decomposition and truncation according to a preset index dimension truncation threshold, the top few largest singular values ​​and their corresponding left singular vectors are selected to form the third truncated left singular vector matrix. The number of rows in this matrix equals the total number of original measurement indices, and the number of columns equals the number of principal components of the indices retained after truncation. Its column vectors reveal the main correlations and collaborative change patterns between different measurement indices (such as voltage, current, and power), thereby compressing multiple physical indices into a few comprehensive index features.

[0081] Specifically, in step D4, the core tensor is obtained by performing Tucker decomposition based on the three truncated left singular vector matrices and the original tensor. This involves performing the following calculations: The original tensor is modally multiplied with the transposes of the first, second, and third truncated left singular vector matrices, respectively, along the corresponding station, time, and measurement indicator dimensions. Essentially, this operation projects the original high-dimensional data onto a low-dimensional feature subspace spanned by the three truncated factor matrices. The result is the core tensor, whose three dimensions are equal to the number of columns in the three truncated factor matrices (i.e., the number of retained principal components). This core tensor condenses the interaction strength and coupling relationships between the compressed station feature patterns, time dynamic patterns, and indicator correlation patterns of the original data, representing a high degree of condensation of the original data in terms of multi-dimensional joint features.

[0082] This invention achieves collaborative dimensionality reduction and feature extraction of high-dimensional data from smart grids by performing Tucker decomposition on the three-dimensional tensor in step S5. The core advantage of this method lies in its ability to simultaneously process data from three dimensions: site, time, and measurement indicators. It also extracts the most representative feature patterns (i.e., truncation factor matrices) in each dimension and the high-order interaction relationships between these patterns (i.e., core tensors) through decomposition.

[0083] Corresponding to steps D1-D4 in S5, the specific decomposition and compression process is supplemented: Based on the tensor calculation principle, after obtaining the three modular expansion matrices, singular value decomposition is performed on them to achieve data compression.

[0084] Based on the principle of tensor computation, smart grid data is sampled in an interleaved manner according to the order of sites, time, and measurement indicators. The grid data tensor is then expanded in three dimensions: site, time, and measurement, resulting in three modular expansion matrices. Singular value decomposition (SVD) is performed on the first, second, and third modular expansion matrices to obtain the following transformation forms: In the formula, It is a tensor Singular value decomposition approximation matrices of three modular expansion matrices; It is the truncated singular value diagonal matrix, and its truncation thresholds are respectively U is the truncated left singular vector matrix, i.e., the truncated orthogonal basis; V is the corresponding truncated right singular vector matrix.

[0085] In step D4, the smart grid data is then decomposed into tensor Tucker components according to the following formula: In the formula, G is the core tensor, which represents the interaction between the three original tensor dimensions in the power grid data; It is the factor matrix of the corresponding modulus tensor, that is, the principal component of this dimension after Tucker decomposition.

[0086] After tensor Tucker decomposition, the core tensor dimension of the smart grid data is compressed. Multiplying the core tensor by the truncated orthogonal basis matrix completes the reconstruction and compression of the grid data. During this process, some approximate tensors in the initial data vector are removed. For the grid data itself, redundant data is eliminated after tensor Tucker decomposition.

[0087] Furthermore, in step S6, the three-dimensional tensor structure object set is reconstructed using the core tensor and truncation factor matrix to output the cleaned power grid data, including the following steps E1-E4: E1. Perform modal product of the core tensor and the first truncated left singular vector matrix along the station dimension.

[0088] E2. Perform a modal product of the first modal product result and the second truncated left singular vector matrix along the time dimension.

[0089] E3. Perform modal product of the second modal product result and the third truncated left singular vector matrix along the measurement index dimension to complete the reconstruction.

[0090] E4. Output the reconstructed cleaning grid data.

[0091] Specifically, in step E1, the core tensor and the first truncated left singular vector matrix are modally multiplied along the station dimension. This process "reconstructs" or "decodes" the information compressed in the core tensor using extracted station feature patterns along the station dimension. This operation expands the size of the core tensor along the station dimension from "the number of retained station principal components" back to "the original total number of stations," generating an intermediate tensor. This intermediate tensor has restored its original structure along the station dimension, but remains compressed along the time and measurement index dimensions (its sizes are "the number of retained time principal components" and "the number of retained index principal components," respectively).

[0092] Specifically, in step E2, the first modal product result is modally multiplied with the second truncated left singular vector matrix along the time dimension, which reconstructs the structure in the time dimension. This operation utilizes the extracted temporal dynamics to expand the size of the intermediate tensor in the time dimension from "the number of retained temporal principal components" back to "the total number of original time points". The second intermediate tensor generated at this time has restored its original structure in both the station and time dimensions, only remaining compressed in the measurement index dimension (size is "the number of retained index principal components").

[0093] Specifically, in step E3, the second modal product result is modally multiplied with the third truncated left singular vector matrix along the measurement index dimension. This process performs the final reconstruction along the measurement index dimension. This operation utilizes the extracted index correlation patterns to expand the tensor's size along the measurement index dimension from "the number of retained principal components" back to "the total number of original measurement indices." This results in a three-dimensional tensor whose three dimensions (number of stations, number of time points, and number of indices) are completely consistent with the dimensions of the original cleaned dataset; this is the reconstructed data.

[0094] Specifically, in step E4, the reconstructed cleaned power grid data is output. This data is a high-quality 3D dataset that has undergone anomaly cleaning, missing data completion, and has had redundant information and noise removed through a tensor decomposition-reconstruction process. Because the reconstruction process is based on patterns extracted from the data that represent its main structural features, the output data is smoother and more consistent while retaining key operational information. It can be converted back to a conventional data format (e.g., a data table arranged by time, site, and index) and directly used for downstream smart grid analysis applications such as load forecasting, condition assessment, and power flow calculation.

[0095] This invention reconstructs the power grid data using the core tensor and truncation factor matrix in step S6, ultimately outputting high-quality, usable cleaned power grid data. This reconstruction process is not only a reverse recovery of compressed data but also an effective data denoising and information enhancement process. Because the feature components representing noise, minor fluctuations, or abnormal interference (corresponding to smaller singular values) in each dimension are discarded during the decomposition stage (step S5), the reconstructed data automatically filters out these undesirable factors, resulting in smoother data curves and more consistent physical meaning.

[0096] Example 3 is the third embodiment of the present invention. It differs from the previous two embodiments in that: the data cleaning method combining K-nearest neighbor detection and tensor decomposition further includes, in order to verify and explain the technical effect of the method, comparing the experimental results with scientific demonstration methods to verify the real effect of the method.

[0097] Based on the above method, actual operating data from five 220kV substations of a power grid were selected over a period of two months, sampled at a frequency of 1Hz. Three types of typical anomalies, including isolated points, jump points, and extreme values, were manually injected to simulate anomaly scenarios in smart grid data. The obtained data includes 11 categories: substation voltage, current, active power, reactive power, power factor, voltage frequency, voltage and current harmonic distortion rates, voltage flicker, transformer temperature, and bus impedance.

[0098] In the experiment, the preprocessed and outlier-inserted data were expanded into a feature matrix, with K set to 15. The Euclidean distance between a sample and its 15 neighboring samples was calculated, and outliers were filtered out using a threshold of 1.2 times the mean sample distance. Subsequently, for the outlier dataset, a tensor decomposition model based on low-rank characteristics was established, and the model was calculated using the alternating direction multiplier method or backward tracking algorithm, combined with the K-nearest neighbor detection method to correct the outliers.

[0099] Meanwhile, to verify the superiority of this patented method, based on the same batch of experimental datasets, and using precision, recall (F1 score), and anomaly false positive rate as the core indicators, this patented method was compared with the PCA-threshold method and the KNN-SVD method according to the above process. The experimental data are shown in Table 1. Table 1 - Experimental Dataset

[0100] This experiment is based on 11 types of electrical data from 5 220kV substations of a power grid over 2 months. After injecting 3 types of anomalies, the three methods were compared. The method of this patent has the best accuracy, recall, and F1 score, and the lowest anomaly false positive rate, which is significantly better than the PCA-threshold method and the KNN-SVD method.

[0101] Example 4, the fourth embodiment of the present invention, differs from the previous three embodiments in that it provides a data cleaning system combining K-nearest neighbor detection and tensor decomposition. The system includes a raw data acquisition module, a data preprocessing module, an anomaly detection and cleaning module, a tensor construction module, a tensor decomposition and compression module, and a data reconstruction and output module. The raw data acquisition module acquires operational data from various stations in the smart grid system, forming a raw data set based on multiple time series and multiple measurement indicators. The data preprocessing module performs consistency and normalization processing on the raw data set to obtain a normalized data set. The anomaly detection and cleaning module, based on the normalized data set, performs... A dual-threshold discrimination method combining local density estimation and K-nearest neighbor distance is used to identify abnormal data and fill in missing values ​​for the data corresponding to the abnormal values ​​to obtain a cleaned data set. The tensor construction module is used to construct a three-dimensional tensor structure object set based on the station number, time series, and measurement index dimensions. The tensor decomposition and compression module is used to obtain the core tensor and the truncation factor matrix corresponding to each dimension by performing tensor Tucker decomposition on the three-dimensional tensor structure object set. The data reconstruction output module is used to reconstruct the three-dimensional tensor structure object set using the core tensor and the truncation factor matrix to output the cleaned power grid data.

[0102] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0104] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0105] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented in combination with any of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0106] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data cleaning method combining K-nearest neighbor detection and tensor decomposition, characterized in that: include, Obtain operational data from various stations in the smart grid system, and form a raw data set based on multiple time series and multiple measurement indicators; The original dataset is subjected to consistency and normalization processing to obtain a normalized dataset. Based on a standardized dataset, a dual-threshold discrimination method that integrates local density estimation and K-nearest neighbor distance is used to identify abnormal data and fill in the missing values ​​of the data corresponding to the abnormal values ​​to obtain a cleaned dataset. Based on the dimensions of site number, time series and measurement indicators, the cleaned data set is constructed into a three-dimensional tensor structure object set; By performing tensor Tucker decomposition on a set of three-dimensional tensor structure objects, the core tensor and the truncation factor matrix corresponding to each dimension are obtained. The three-dimensional tensor structure object set is reconstructed using the core tensor and truncation factor matrix, and the cleaned power grid data is output.

2. The data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in claim 1, characterized in that: The original dataset is subjected to consistency and normalization processing to obtain a normalized dataset, including: The original dataset is subjected to data format standardization processing to obtain the first standardized data; The missing items in the first standardized data are filled to obtain the second filled data. The data after the second imputation is normalized to obtain a normalized dataset.

3. The data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in claim 2, characterized in that: Based on the aforementioned standardized dataset, a dual-threshold discrimination method fusing local density estimation and K-nearest neighbor distance is used to identify outlier data and fill in missing values ​​for the data corresponding to the outliers, thus obtaining a cleaned dataset. include, Calculate the Euclidean distance from each data element in the canonical dataset to other elements, and determine the K nearest neighbor set for each element based on this distance; Calculate the local density of each element based on its K nearest neighbor set, and determine its K nearest neighbor distance; Calculate the local density threshold and the K-nearest neighbor threshold based on the local density and K-nearest neighbor distance of all elements; Data elements whose local density is less than the local density threshold and whose K-nearest neighbor distance is greater than the K-nearest neighbor threshold are identified as outliers; The missing values ​​in the data corresponding to the outlier are filled in to obtain a cleaned data set.

4. The data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in claim 3, characterized in that: The calculation of the local density threshold and K-nearest neighbor distance threshold based on the local density and K-nearest neighbor distance of all elements includes, Based on the data size of the normalized dataset and the distance relationship between outliers and cluster centers, the local density offset and the K-nearest neighbor distance offset are determined. Calculate the average local density of all data elements and combine it with the local density offset to obtain the local density threshold; Calculate the average K-nearest neighbor distance for all data elements and combine it with the K-nearest neighbor distance offset to obtain the K-nearest neighbor distance threshold.

5. The data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in claim 4, characterized in that: The cleaned data set is constructed into a three-dimensional tensor structure object set based on site number, time series, and measurement index dimensions, including: The cleaned dataset was interleaved and sampled according to site order, time order, and measurement index order to form the original tensor. The original tensor is expanded along the station dimension to obtain the first expansion matrix; The original tensor is expanded along the time dimension to obtain the second expansion matrix; The original tensor is expanded along the dimension of the measurement index to obtain the third expansion matrix.

6. The data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in claim 5, characterized in that: Tensor Tucker decomposition is performed on the set of three-dimensional tensor structure objects to obtain the core tensor and the truncation factor matrix corresponding to each dimension, including, Perform singular value decomposition on the first modular expansion matrix to obtain the first truncated left singular vector matrix; Perform singular value decomposition on the second modular expansion matrix to obtain the second truncated left singular vector matrix; Perform singular value decomposition transformation on the third modular expansion matrix to obtain the third truncated left singular vector matrix; The core tensor is obtained by performing Tucker decomposition on the original tensor based on the first truncated left singular vector matrix, the second truncated left singular vector matrix, and the third truncated left singular vector matrix.

7. A data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in claim 6, characterized in that: The three-dimensional tensor structure object set is reconstructed using the core tensor and truncation factor matrix, and the cleaned power grid data is output, including: Modally multiply the core tensor with the first truncated left singular vector matrix along the station dimension; Perform a modal product of the first modal product result and the second truncated left singular vector matrix along the time dimension; The second modal product result is modally multiplied with the third truncated left singular vector matrix along the measurement index dimension to complete the reconstruction; Output the reconstructed cleaning grid data.

8. A data cleaning system combining K-nearest neighbor detection and tensor decomposition, employing the data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in any one of claims 1 to 7, characterized in that, include: The system includes a raw data acquisition module, a data preprocessing module, an anomaly detection and cleaning module, a tensor construction module, a tensor decomposition and compression module, and a data reconstruction and output module. The raw data acquisition module is used to acquire the operating data of each station in the smart grid system and form a raw data set based on multiple time series and multiple measurement indicators; The data preprocessing module is used to perform consistency and normalization processing on the original dataset to obtain a normalized dataset. The anomaly detection and cleaning module is used to identify abnormal data based on a standardized dataset, and to fill in missing values ​​in the data corresponding to the anomaly values ​​to obtain a cleaned dataset. The tensor construction module is used to construct the cleaned data set into a three-dimensional tensor structure object set based on the site number, time series and measurement index dimensions; The tensor decomposition and compression module is used to perform tensor Tucker decomposition on a set of three-dimensional tensor structure objects to obtain the core tensor and the truncation factor matrix corresponding to each dimension. The data reconstruction output module is used to reconstruct the three-dimensional tensor structure object set using the core tensor and the truncation factor matrix, and output the cleaned power grid data.

9. A computer device comprising a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of a data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a data cleaning method combining K-nearest neighbor detection and tensor decomposition as described in any one of claims 1 to 7.

Images