Big data cleaning method and system for intelligent electric energy meter
By employing a three-layer grouping encoding method for smart meter data, a weighted isolated forest algorithm for outlier detection, and an improved K-nearest neighbor decision tree for missing value filling, the problems of inconsistent data formats and missing values were solved, improving data processing efficiency and analysis accuracy, and supporting the efficient application of deep learning models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-04-10
AI Technical Summary
The data generated by smart meters has inconsistent formats, low structure, and contains outliers and missing values, making it difficult for deep learning models to understand and learn effectively, thus affecting the accuracy of operational status assessment and degradation prediction.
A three-layer grouping coding and a weighted isolated forest algorithm based on feature importance are used to detect and remove outliers. An improved K-nearest neighbor decision tree algorithm is used to fill in missing values, and time data is normalized and encoded to form high-quality, highly consistent standardized numerical features.
It enables automated and standardized processing of smart energy meter data, improves data processing efficiency and analysis accuracy, provides ideal input for deep learning models, and enhances the accuracy of operational status assessment and degradation prediction.
Smart Images
Figure CN121833675A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data preprocessing of smart electric energy meters, and particularly relates to a big data cleaning method and system for smart electric energy meters. BACKGROUND
[0002] With the accelerated construction of global energy internet and smart grid, smart electric energy meters have evolved from traditional metering instruments to the core perception unit of grid digitization and intelligence. Modern smart electric energy meters integrate high-precision metering, real-time data acquisition, two-way communication, remote control and other functions, and constitute an important bridge connecting power enterprises and users. These devices generate massive operation data every moment, not only providing a basis for electricity settlement, but also playing an indispensable role in dynamic electricity price implementation, distribution network state estimation, load characteristic analysis, reactive power optimization and electricity theft detection and other advanced applications. Through deep mining of these data, grid operators can accurately master regional electricity consumption rules, optimize grid operation mode, improve energy utilization efficiency, and provide key data support for demand side response. In the field of smart electric energy meter production and quality supervision, its performance reliability is directly related to the fairness of metering and the safety and stability of the grid. Therefore, it is necessary to conduct comprehensive performance testing under various test schemes in accordance with strict national standards and industry specifications. These tests simulate different electrical stress conditions (such as varying current and power factor) and complex environmental stress (such as temperature, humidity, and air pressure fluctuations), thereby generating multi-dimensional, structured time series data containing error performance, environmental parameters and device identification information. These data are valuable resources for evaluating electric metering characteristics, analyzing long-term stability and failure mechanism.
[0003] The introduction of advanced artificial intelligence technologies such as deep learning into this field opens up revolutionary prospects for performance evaluation and predictive maintenance of smart meters. Deep neural network models built based on historical data can automatically learn the combined effects of environmental factors and electrical parameters on meter errors, identify potential early failure patterns, and even predict their remaining useful life. This is expected to achieve a precise portrait of the health status of the meter, shift the maintenance mode from post-maintenance to pre-warning, and significantly improve the fine management level of metering assets and the power supply reliability of the power grid. However, the performance of deep learning models is highly dependent on the quality of input data, and the realization of this technical vision faces serious data foundation challenges. The data format is not unified and the structure is low, the data comes from different manufacturing units and detection bases, and the key identifiers such as 'test scheme','manufacturing unit', and'sample number' are non-numeric text, while electrical stress parameters such as 'current' and 'power factor' are generally represented by industry-specific codes. At the same time, the data is mixed with physical outliers caused by sensor failure or transmission errors, such as environmental readings that exceed the actual possibility, and non-uniform timestamp information. During data collection, data missing due to communication transmission, human operation errors, and other problems also occur. This chaotic data state makes it difficult for deep learning models to effectively understand and learn the features, severely restricting the potential of artificial intelligence technology. Currently, the preprocessing work for the above data problems relies heavily on manual intervention or temporary script processing, which has the outstanding contradictions of fragmented processing flow, low efficiency, uneven quality, and difficulty in large-scale application. SUMMARY
[0004] The technical problem solved by the present application: In view of the above problems of the prior art, the present application provides a big data cleaning method and system for smart meters, which aims to realize automatic and standardized data cleaning and coding, efficiently convert raw and chaotic multi-source data into high-quality, high-consistency, and rich-semantic standardized numerical features, provide ideal input for deep learning models, and improve the accuracy of smart meter operation state evaluation and degradation prediction.
[0005] To solve the above technical problems, the technical scheme adopted by the present application is: The application discloses a big data cleaning method for smart electric energy meters, which comprises the following steps: loading original test data of the smart electric energy meters; grouping and coding the original test data according to three layers of test schemes, manufacturers and sample numbers, and coding the applied electric stress; decomposing the coded input time data into different time dimensions and normalizing the coding; detecting and removing abnormal values by using a weighted isolation forest algorithm based on feature importance for the normalized coded input time data; filling in missing values by using an improved K nearest neighbor decision tree algorithm; and saving the processed data according to the three-layer grouping mode of test schemes, manufacturers and sample numbers.
[0006] Optionally, the original test data comprises test data of multiple samples of different manufacturers or the same manufacturer under multiple test schemes, wherein the test data comprises values of applied electric stress of the smart electric energy meters, measured values obtained by standard metering, and values of environmental stress of the environment where the smart electric energy meters are located recorded by sensors, wherein the electric stress comprises part or all of current and power factor, and the environmental stress comprises part or all of temperature, humidity, air pressure and wind speed.
[0007] Optionally, when the coded input time data is decomposed into different time dimensions and normalized coding, the different time dimensions comprise six time dimensions of year, month, day, hour, minute and second.
[0008] Optionally, the abnormal value detection and removal by using the weighted isolation forest algorithm based on feature importance comprises: S101, acquiring input normalized coded input time data; S102, calculating feature importance of each feature in the normalized coded input time data; S103, constructing and training a weighted isolation forest based on the feature importance of each feature as the weight of the weighted isolation forest; S104, calculating the weighted path length of each data point of the input time data in the weighted isolation forest; S105, calculating the weighted anomaly score of each data point according to the weighted path length; S106, regarding the data points with the weighted anomaly score greater than a preset threshold as abnormal data points, and deleting the samples in which the abnormal data point pollution rate exceeds a preset threshold.
[0009] Optionally, the function expression for calculating the feature importance of each feature in the normalized coded input time data in step S102 is: ; wherein, is the feature importance of the feature . SHAP value of the feature , SHAP value of the feature of the absolute value, the total number of features.
[0010] Optionally, the calculation function expression of the weighted path length in step S104 is: ; wherein, the weighted path length of the data point , the total number of isolated trees in the weighted isolated forest, the total number of features, the feature importance of the feature , the path length of the data point in the first t isolated tree of the weighted isolated forest; the calculation function expression of the weighted anomaly score in step S104 is: ; wherein, the weighted anomaly score of the data point , the average path length of the data point x in all isolated trees of the weighted isolated forest, the average path length of an isolated tree constructed by samples.
[0011] Optionally, the filling of missing values by using the improved K nearest neighbor decision tree algorithm comprises: S201, obtaining the input target data set D={ x 1, x 2,..., x m}, wherein x 1~ x m is the 1st m sample of the target data set D, and any ith sample x i ={ x i1 , x i2 ,..., x in} includes x i1 -x in total n features, and any missing value This represents the missing first element in the target dataset D. i The first sample j One feature; for the target dataset D, the proportion of missing values is calculated according to the following formula: ; in, The proportion of missing values. m The number of samples in the target dataset D. n The number of features of the samples in the target dataset D; if the proportion of missing values in the target dataset D exceeds a preset threshold, the target dataset D is determined to be an invalid dataset, and the process ends and exits; otherwise, proceed to step S202. S202, Calculate each missing value in the target dataset D. Surrounding data density n i : ; in, For the j-th sample, For missing values and the j-th sample The Euclidean distance between them The neighborhood radius; based on missing values Surrounding data density n i Determine missing values K value: ; in, For missing values The K value, To round up, For adjustment factors; S203, based on the missing D values in the target dataset Calculate the missing value from the K nearest neighbor samples Fill value: ; ; in, For missing values The fill value, For missing values The set of K nearest neighbor samples, For the j-th sample The weight, This is to prevent small positive numbers with a denominator of 0; S204, Calculate the missing values in the target dataset D. Subsequent filling error:
[0012] in, To fill the error, The number of data points in the target dataset. For the k-th sample; S205, if the filling error is greater than the preset threshold Then adjust the missing values according to the following formula. After determining the K value, proceed to step S203: ; in, and These are the K values before and after the adjustment, respectively. To adjust the coefficients; otherwise, proceed to step S206. S206, Initialize the feature selection ratio, decision tree depth, and number of decision trees in the decision tree model. ; S207, with the current feature selection ratio, decision tree depth and number of decision trees. As model parameters, a decision tree model is constructed for the target dataset D after imputing missing values, and the feature importance in the decision tree model is calculated according to the following formula: ; in, Selecting data to be filled in the features of the decision tree model j The importance of the features on For the t-th decision tree in feature j The splitting gain is evaluated by the decrease in root mean square error; features whose importance exceeds a preset threshold are considered key features. S208, the training decision tree model is optimized by filling in the gaps according to the following formula: ; in, For missing values The fill optimization value, For the first t The weights of each decision tree For the first t The predicted value of each decision tree; Key features that are not missing; S209, Calculate the imputation error of the target dataset after imputation optimization: ; in, To fill the error, The number of data points in the target dataset D. for the kth sample; S210, if the filling error of the target data set D after filling optimization is greater than the preset threshold RMSE FinalThreshold , return to increase the feature selection ratio and the number of decision trees, and jump to step S207; otherwise, output the target data set D after filling optimization.
[0013] The application also provides a big data cleaning system for a smart electric energy meter, comprising a microprocessor and a memory connected to each other, wherein the microprocessor is programmed or configured to execute the big data cleaning method for the smart electric energy meter.
[0014] The application also provides a computer readable storage medium, wherein a computer program or instructions are stored in the computer readable storage medium, and the computer program or instructions are programmed or configured to execute the big data cleaning method for the smart electric energy meter by a processor.
[0015] The application also provides a computer program product, comprising a computer program or instructions, which are programmed or configured to execute the big data cleaning method for the smart electric energy meter by a processor.
[0016] Compared with the prior art, the application mainly has the following beneficial effects: the method comprises loading original test data of a smart electric energy meter; grouping and coding the original test data according to three layers of test schemes, manufacturers and sample numbers, and coding the applied electrical stress; decomposing and normalizing the coded input time data to different time dimensions; detecting and removing outliers from the normalized input time data by using a weighted isolated forest algorithm based on feature importance; filling missing values by using an improved K nearest neighbor decision tree algorithm; and saving the processed data according to the three-layer grouping mode of test schemes, manufacturers and sample numbers. The application can realize automatic classification, coding, outlier detection, missing value filling and time normalization of original test data of a smart electric energy meter, can efficiently convert original, disordered multi-source data into standardized numerical features with high quality, high consistency and rich semantic information, improves data processing efficiency and analysis accuracy, provides an ideal input for a deep learning model, and improves the accuracy of smart electric energy meter operation state evaluation and degradation prediction. BRIEF DESCRIPTION OF DRAWINGS
[0017] Figure 1 The figure is a basic flowchart of the method of the embodiment of the application.
[0018] Figure 2 The figure is a flowchart of data outlier detection in the embodiment of the application.
[0019] Figure 3 The figure is an implementation effect diagram of outlier detection in the embodiment of the application.
[0020] Figure 4 Flow chart for data missing value filling in the embodiment of the present application.
[0021] Figure 5 Implementation effect diagram for missing value filling in the embodiment of the present application. DETAILED DESCRIPTION
[0022] In order to make the personnel in the technical field better understand the technical solutions of the present application, the technical solutions of the present application will be further described in detail below in combination with the drawings in the embodiments of the present application.
[0023] As shown in Figure 1 the present embodiment for a smart meter big data cleaning method includes the following steps: S1, load the original test data of the smart meter; group and encode the original test data according to the test scheme, manufacturer, sample number in three layers, and encode the applied electrical stress; S2, decompose the encoded input time data to different time dimensions and normalize the encoding; S3, for the normalized encoded input time data, use a weighted isolation forest algorithm based on feature importance for anomaly detection and elimination; S4, use an improved K nearest neighbor decision tree algorithm to fill in the missing values; S5, save the processed data according to the three-layer grouping method of test scheme, manufacturer, and sample number.
[0024] In step S1 of the present embodiment, the original test data includes test data of different manufacturers, multiple samples of the same manufacturer under multiple test schemes, the test data including: the value of the applied electrical stress of the smart meter, the measured value obtained by the standard meter, and the value of the environmental stress of the environment where the smart meter is located recorded by the sensor, the electrical stress including part or all of the current and power factor, and the environmental stress including part or all of the temperature, humidity, air pressure and wind speed.
[0025] In this embodiment, the to-be-tested smart electric energy meter is in real-time operation under the typical natural environment, and the standard source of the test base is used to generate power consumption. The test scheme of each electric energy meter, the applied current, and the power factor are recorded, and the measured value of the to-be-tested smart electric energy meter is measured by the standard meter. The temperature, humidity, air pressure, wind speed, and other sensors are used to record the environmental stress values of the smart electric energy meter. The measurement and collection are performed, the field test data is uploaded to the data acquisition system, and the data file is formed. The test data original file is read, the test scheme, manufacturing unit, sample number, and other key columns are identified and extracted, and the data is structured according to the “test scheme-manufacturing unit-sample number” grouping. The classification structure file is generated and the data is structured. The three-level system of “test scheme-manufacturing unit-sample number” is coded and mapped, and each test scheme is assigned a unique code of 1- n n n Finally, the coding mapping table is generated according to the three-level nested mode, which is used for subsequent data replacement. This system ensures the traceability and consistency of the data.
[0026] For different levels of current stress and power factor stress applied to the smart electric energy meter in the test scheme, the coding method is defined respectively, the text value in the original data is replaced by digital coding, and a unified numerical mapping system is established. The specific coding method is shown in Table 1.
[0027] Table 1: Coding values corresponding to different currents and power factors
[0028] In Table 1, Ib is the reference current of the direct access instrument; Imax is the maximum current that the ammeter can continuously bear and maintain safety and meet the accuracy requirements; L is the power factor of the inductive circuit; C is the power factor of the capacitive circuit.
[0029] In step S2 of the embodiment, the encoded input time data is decomposed into different time dimensions and normalized encoding is performed to meet the input requirements of time series in deep learning methods. Specifically, when the encoded input time data is decomposed into different time dimensions and normalized encoding is performed in the embodiment, the different time dimensions include six time dimensions of year, month, day, hour, minute, and second. For example, for a time field in the data in the format of "%year / %month / %day% hour:%minute:%second", it can be split into six features in the format of year, month, day, hour, minute, and second, mapped to the interval [0, 1] through maximum and minimum normalization, and encoded. The specific normalized encoding mode is shown in Table 2.
[0030] Table 2 Normalized encoding formula and mapping range of each time dimension
[0031] After completing the encoding of the data packet and the electrical stress data, the next step is to detect and remove the abnormal values in the smart meter data. As shown in Figure 2 , the abnormal value detection and removal based on the feature importance weighted isolation forest algorithm in step S3 of the embodiment includes: S101, obtaining the normalized encoded input time data; S102, calculating the feature importance of each feature in the normalized encoded input time data; S103, constructing and training a weighted isolation forest based on the feature importance of each feature as the weight of the weighted isolation forest; S104, calculating the weighted path length of each data point of the input time data in the weighted isolation forest; S105, calculating the weighted anomaly score of each data point according to the weighted path length; S106, regarding the data points with a weighted anomaly score greater than a preset threshold as abnormal data points, and deleting the samples with an abnormal data point pollution rate exceeding a preset threshold in each sample.
[0032] The function expression for calculating the feature importance of each feature in the normalized encoded input time data in step S102 is: ; wherein, is the feature importance of the feature , is the absolute value of the SHAP value of the feature , is the feature importance of the feature . SHAP value The absolute value, The total number of features.
[0033] In step S103, the weighted isolated forest is constructed and trained using the feature importance of each feature as the weights. This is an existing method, and its implementation details will not be elaborated here. When constructing the weighted isolated forest using the feature importance of each feature as the weights, features are randomly selected. x j ,in:
[0034] The above formula means that, j From d Uniform random sampling from each feature ( The index is obtained by calculating features. x j Weights to reflect features x j Selection probability : ; After determining the feature importance weights, they are set as training parameters to construct a weighted isolation forest for training. When the weighted isolation tree reaches its maximum tree depth or contains only a single sample, the training of the weighted isolation forest is considered complete, and the weighted path length of the sample can then be calculated.
[0035] The function expression for calculating the weighted path length in step S104 of this embodiment is as follows: ; in, For data points The weighted path length, The total number of isolated trees in a weighted isolated forest. For the total number of features, Features The importance of features For data points In the weighted isolated forest t The path length in the isolated tree; the expression for the weighted outlier score calculation function in step S104 is: ; in, For data points Weighted outlier scores, For data points x The average path length among all isolated trees in a weighted isolated forest. For the reason The average path length of a sample builds an isolated tree. The data points x The calculation function expression of the average path length in all isolated trees of the weighted isolated forest is: ; When the data points with the weighted anomaly score greater than the preset threshold value are taken as the abnormal data points in step S106, the required threshold value can be selected according to actual needs. In the embodiment, the basis for classifying the original samples into normal value samples and abnormal value samples is that when s weighted ( x )≈1, it indicates that the current data point is an abnormal value data point and is easy to be isolated; when s weighted ( x )≈0.5, it indicates that the current data point is a normal value data point and is not easy to be isolated; and when s weighted ( x )≈0, it indicates that the current data point is in a high-density concentrated area and is a non-abnormal value data point. After the weighted anomaly score s weighted ( x ) is calculated, the samples in which the abnormal data point pollution rate exceeds the preset threshold value are deleted. In the embodiment, the function expression of the condition is:
[0036] wherein, is the abnormal data point pollution rate in the sample , the abnormal data point pollution rate is the ratio of the number of abnormal data points to the total number of data points, represents the 1 - v quantile point of the weighted anomaly score. As shown in Figure 3 , the effect of the abnormal value detection and elimination in the intelligent electric energy meter data in the embodiment can be proved by Figure 4 , which proves that the weighted isolated forest algorithm adopted in the embodiment can effectively complete the abnormal value detection task in the data.
[0037] After the abnormal value detection is completed, the next step is to efficiently fill in the missing values in the data. The improved K nearest neighbor decision tree filling algorithm is adopted. The adaptive K value is adapted to the data density change to improve the filling accuracy. After the KNN filling, preliminary error evaluation is performed to ensure that the filling result does not deviate from the overall distribution of the data. Subsequently, the feature importance analysis is used to select the key variables, the optimized features are selected to improve the decision tree filling effect, and the interference of redundant features is avoided. As shown in Figure 4 , the improved K nearest neighbor decision tree algorithm is adopted to fill in the missing values in step S4 in the embodiment, including: S201, acquiring an input target data set D={x 1, x 2,..., x m},in x 1~ x m For the first to last data point of the target dataset D m n samples, any i-th sample x i ={ x i1 , x i2 ,..., x in} includes x i1 ~x in common n Each feature, with any missing value. This represents the missing first element in the target dataset D. i The first sample j One feature; for the target dataset D, the proportion of missing values is calculated according to the following formula: ; in, The proportion of missing values. m The number of samples in the target dataset D. n The number of features of the samples in the target dataset D; if the proportion of missing values in the target dataset D exceeds a preset threshold, the target dataset D is determined to be an invalid dataset, and the process ends and exits; otherwise, proceed to step S202. S202, Calculate each missing value in the target dataset D. Surrounding data density n i : ; in, For the j-th sample, For missing values and the j-th sample The Euclidean distance between them The neighborhood radius; based on missing values Surrounding data density n i Determine missing values K value: ; in, For missing values The K value, To round up, For adjustment factors; S203, based on the missing D values in the target dataset Calculate the missing value from the K nearest neighbor samples Fill value: ; ; in, For missing values The fill value, For missing values The set of K nearest neighbor samples, For the j-th sample The weight, This is to prevent small positive numbers with a denominator of 0; S204, Calculate the missing values in the target dataset D. Subsequent filling error:
[0038] in, To fill the error, The number of data points in the target dataset. For the k-th sample; S205, if the filling error is greater than the preset threshold Then adjust the missing values according to the following formula. After determining the K value, proceed to step S203: ; in, and These are the K values before and after the adjustment, respectively. To adjust the coefficients; otherwise, proceed to step S206. S206, Initialize the feature selection ratio, decision tree depth, and number of decision trees in the decision tree model. ; S207, with the current feature selection ratio, decision tree depth and number of decision trees. As model parameters, a decision tree model is constructed for the target dataset D after imputing missing values, and the feature importance in the decision tree model is calculated according to the following formula: ; in, Selecting data to be filled in the features of the decision tree model j The importance of the features on For the t-th decision tree in feature j The splitting gain is evaluated by the decrease in root mean square error; features whose importance exceeds a preset threshold are considered key features. S208, the training decision tree model is optimized by filling in the gaps according to the following formula: ; in, For missing values The fill optimization value, For the first t The weights of each decision tree For the first t The predicted value of each decision tree; Key features that are not missing; S209, Calculate the imputation error of the target dataset after imputation optimization: ; in, To fill the error, The number of data points in the target dataset D. For the k-th sample; S210, If the imputation error of the target dataset D after imputation optimization is greater than the preset threshold RMSE FinalThreshold If the result is positive, then return to increase the feature selection ratio and the number of decision trees, and proceed to step S207; otherwise, output the optimized target dataset D. In this embodiment, for a smart energy meter produced by a specific manufacturer, the data missing value imputation effect is as follows: Figure 5 As shown in the diagram, through two magnified local areas, it can be seen that the improved K-Nearest Neighbor Decision Tree (IKNNDT) algorithm in this embodiment can accurately distribute the data points for missing value imputation within the non-missing data region, demonstrating a good imputation effect. To verify the effectiveness of the improved K-Nearest Neighbor Decision Tree (IKNNDT) algorithm for missing value imputation in this embodiment, existing algorithms such as K-Nearest Neighbor (KNN), Decision Tree (DT), Autoregressive Integral Moving Average (ARIMA), and Two-Layer Random Forest (DRF) are compared in this embodiment. Root Mean Square Error (RMSE) and Mean Absolute Residual (MAR) are selected as performance evaluation criteria for the missing value imputation algorithm. The results are shown in Table 3.
[0039] Table 3 Performance Comparison of Different Missing Value Imputation Algorithms
[0040] As can be seen from Table 3, the effectiveness of the improved K nearest neighbor decision tree algorithm (IKNNDT) for filling in missing values in the method of the present embodiment can more effectively fill in missing values. In the intelligent electric energy meter big data cleaning method of the present application, it has high credibility. After filling in the missing data, all the processed effective data is merged, and the data is grouped and saved according to the three-layer nested structure of "test scheme code-manufacturing unit code-sample number code" in the data and stored. Finally, the intelligent electric energy meter data set after big data cleaning suitable for deep learning is obtained.
[0041] In summary, in view of the problems of data format confusion, existence of abnormal values and missing values, and non-uniform coding method in the intelligent electric energy meter big data under the existing circumstances, which makes it difficult for the deep learning algorithm to effectively analyze and process, the present embodiment proposes a big data cleaning method suitable for intelligent electric energy meter. The original data file is grouped and coded according to the three-layer structure of "test scheme-manufacturer-sample number"; then the electrical stress applied in the test is coded; then a weighted isolation forest algorithm based on feature importance is used to detect and eliminate abnormal values in the data; and an improved K nearest neighbor decision tree algorithm is used to fill in the missing values in the data; the time field in the data is split into six dimensions and normalized coding is performed; finally, the data set is saved according to the three-layer grouping coding structure, forming an intelligent electric energy meter big data cleaning method suitable for deep learning. Especially by using the weighted isolation forest algorithm based on feature importance (FIWIF) to detect and eliminate abnormal values for the input time data after normalization coding, and using the improved K nearest neighbor decision tree algorithm (IKNNDT) to fill in the missing values, it can be effectively applied to intelligent electric energy meter field verification and performance evaluation. The present embodiment can be used for electric energy metering devices and other metering devices, and has the advantages of good universality, and can provide data security in intelligent electric energy meter field verification and metering performance evaluation.
[0042] The present embodiment also provides a big data cleaning system for intelligent electric energy meter, comprising a microprocessor and a memory connected to each other, the microprocessor being programmed or configured to execute the big data cleaning method for intelligent electric energy meter.
[0043] The present embodiment also provides a computer readable storage medium having a computer program or instructions stored therein, the computer program or instructions being programmed or configured to execute the big data cleaning method for intelligent electric energy meter by a processor.
[0044] The present embodiment also provides a computer program product comprising a computer program or instructions, the computer program or instructions being programmed or configured to execute the big data cleaning method for intelligent electric energy meter by a processor.
[0045] Those skilled in the art will appreciate that the technology of the present application can be embodied in a method, system, or computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-readable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage devices, etc.) embodying computer usable program code. The specification and drawings are, accordingly, to be regarded as illustrative and not restrictive. The present application is described in terms of flowcharts and / or block diagrams in which each block indicates a set of elements carried out normally by a computer program. Those skilled in the art will appreciate that each block of the flowcharts and / or block diagrams, and combinations thereof, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing machine, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowcharts and / or block diagrams block or blocks. Figure 1 one or more flowcharts and / or block diagrams Figure 1 one or more flowcharts and / or block diagrams Figure 1 one or more flowcharts and / or block diagrams Figure 1 one or more flowcharts and / or block diagrams Figure 1 one or more flowcharts and / or block diagrams Figure 1 one or more flowcharts and / or block diagrams
[0046] The above description is only preferred embodiments of the present application, and the protection scope of the present application is not limited to the above-mentioned embodiments. Any improvement and modification made by those skilled in the art without departing from the principle of the present application should be considered as falling within the protection scope of the present application.
Claims
1. A big data cleaning method for a smart meter, characterized in that, The method comprises the following steps: loading original test data of the smart meter; grouping and encoding the original test data according to a test scheme, a manufacturer and a sample number, and encoding an applied electric stress; The encoded input time data is decomposed into different time dimensions and normalized, and the normalized input time data is subjected to abnormal value detection and elimination by using a weighted isolation forest algorithm based on feature importance; missing values are filled by using an improved K nearest neighbor decision tree algorithm; and the processed data is saved according to the three-layer grouping mode of the test scheme, the manufacturer and the sample number.
2. The method for big data cleaning of smart meters according to claim 1, wherein, The original test data comprises test data of different manufacturers and multiple samples of the same manufacturer under multiple test schemes, and the test data comprises a value of an applied electric stress of the smart meter, a measured value obtained by a standard meter, and a value of an environmental stress of an environment in which the smart meter is located recorded by a sensor, wherein the electric stress comprises part or all of current and power factor, and the environmental stress comprises part or all of temperature, humidity, air pressure and wind speed.
3. The method for big data cleaning of smart meters according to claim 1, wherein, When the encoded input time data is decomposed into different time dimensions and normalized, the different time dimensions comprise six time dimensions of year, month, day, hour, minute and second.
4. The method for big data cleaning of smart meters according to claim 1, wherein, The abnormal value detection and elimination by using the weighted isolation forest algorithm based on feature importance comprises: S101, obtaining input normalized encoded input time data; S102, calculating feature importance of each feature in the normalized encoded input time data; S103, constructing and training a weighted isolation forest based on the feature importance of each feature as a weight of the weighted isolation forest; S104, calculating a weighted path length of each data point of the input time data in the weighted isolation forest; S105, calculating a weighted anomaly score of each data point according to the weighted path length; S106, regarding a data point with a weighted anomaly score greater than a preset threshold as an abnormal data point, and deleting a sample in which a pollution rate of the abnormal data point exceeds a preset threshold.
5. The method for big data cleaning of smart meters according to claim 4, wherein, The function expression for calculating the feature importance of each feature in the normalized encoded input time data in step S102 is: ； wherein, is a feature of the feature importance, is a feature of the SHAP value of the absolute value, is a feature of the SHAP value of the absolute value, is the total number of features.
6. The method for big data cleaning of smart meters according to claim 4, wherein, The function expression for calculating the weighted path length in step S104 is: ； wherein, is the weighted path length of the data point is the total number of isolated trees in the weighted isolation forest, is the total number of features, is the feature is the feature importance of the feature t is the path length of the data point t in the first isolated tree of the weighted isolation forest; the calculation function expression of the weighted anomaly score in step S104 is: ； wherein, is the weighted anomaly score for data point , is the weighted anomaly score for data point x , is the average path length in all isolated trees of the weighted isolation forest, is the average path length for building one isolated tree from samples.
7. The method for big data cleaning of smart meters according to claim 1, wherein, The filling of the missing values by using the improved K nearest neighbor decision tree algorithm comprises: S201, Obtain the input target dataset D={ x 1, x 2,..., x m },in x 1~ x m For the first to last data point of the target dataset D m n samples, any i-th sample x i ={ x i1 , x i2 ,..., x in } includes x i1 ~x in common n Each feature, with any missing value. This represents the missing first element in the target dataset D. i The first sample j One feature; for the target dataset D, the proportion of missing values is calculated according to the following formula: ； wherein, is a proportion of missing values, m is a number of samples of the target dataset D, n is a number of features of the samples of the target dataset D; if the proportion of missing values of the target dataset D exceeds a preset threshold, determining that the target dataset D is an invalid dataset, ending and exiting; otherwise, jumping to step S202; S202, calculating each missing value in the target data set D Density of surrounding data n i : ； in, For the j-th sample, For missing values and the j-th sample The Euclidean distance between them The neighborhood radius; based on missing values Surrounding data density n i Determine missing values K value: ； wherein, is a missing value K value, is a ceiling function, is an adjustment factor; S203, calculate the missing value of the target data set D according to K nearest neighbor samples of the missing value ； ； in, For missing values The fill value, For missing values The set of K nearest neighbor samples, For the j-th sample The weight, This is to prevent small positive numbers with a denominator of 0; S204, calculate the imputation error of the target data set D after imputing the missing values S204, calculate the imputation error of the target data set D after imputing the missing values wherein, to fill in the errors, is the number of data in the target data set, is the kth sample; S205, if the filling error is greater than a preset threshold then adjust the missing value according to the following formula K value after jumping to step S203: ； wherein, and Kpost and Kpre are the K values after and before the adjustment, respectively, is an adjustment coefficient; otherwise, go to step S206; S206, initialize the feature selection ratio of the decision tree model, the decision tree depth and the number of decision trees ; S207, with the current feature selection ratio, decision tree depth and the number of decision trees As a model parameter, a decision tree model is constructed for the target data set D after filling in the missing values, and the feature importance in the decision tree model is calculated according to the following formula: ； wherein, selecting feature importance of the to-be-imputed data on the feature j for a decision tree model, splitting gain of the tth decision tree on the feature j the splitting gain is evaluated by root mean square error reduction; and selecting the feature with feature importance exceeding a preset threshold as a key feature; S208, filling and optimizing the target data set according to the following formula by training a decision tree model: ； wherein, is an imputed value for missing values is an imputed value for missing values, is a weight for the t th decision tree, is a predicted value for the t th decision tree; denotes a key feature that is not missing. S209, calculating a filling error of the target data set after the filling and optimization: ； wherein, to fill in the errors, is the number of data in the target dataset D, is the kth sample; S210, if the imputation error of the target data set D after imputation optimization is greater than the preset threshold RMSE FinalThreshold , return to increase the feature selection ratio and the number of decision trees, and jump to step S207; otherwise, output the target data set D after imputation optimization.
8. A big data cleaning system for smart metering, comprising a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the big data cleaning method for the smart meter according to any one of claims 1-7.
9. A computer-readable storage medium having stored therein a computer program or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the big data cleaning method for the smart meter according to any one of claims 1-7 by the processor.
10. A computer program product comprising computer programs or instructions, characterized in that, The computer program or instructions are programmed or configured to execute the big data cleaning method for the smart meter according to any one of claims 1-7 by the processor.