A processing method of time series data of remote electric energy of a smart meter

Abstract

The application discloses a kind of processing methods of intelligent electric meter remote electric energy time series data.The application provides the missing of history data of time series for intelligent electric meter remote transmission and storage to database, noise data, dirty data problems such as repeated data, carries out data pretreatment.Through the data of database record, corresponding deletion, filling, reasonable modification is carried out to different types of "dirty data", realizes data cleaning, guarantees data overall accuracy and makes data have statistical significance, provides data basis for subsequent data analysis, feature extraction, energy prediction, energy-saving strategy use etc.In the process of energy consumption data analysis, the application processes the data collected to the database, overcomes the data loss, repetition and other problems that may be caused by data collection in hardware layer, forwarding through various networks and devices, so that the cleaned data can be directly analyzed and utilized, and it is an indispensable link in big data engineering.

Description

TECHNICAL FIELD

[0001] The application relates to a data cleaning method for intelligent electric meter data and belongs to the technical field of intelligent electric meter data processing. BACKGROUND

[0002] Building energy saving is an important way of energy saving and emission reduction. In order to promote building energy saving work and strengthen energy saving supervision, China is gradually establishing a perfect energy consumption database and a reasonable benchmark evaluation system and method, using an energy consumption monitoring management system to complete building energy consumption data collection, analysis and forwarding, and providing detailed data basis for evaluation of energy saving and emission reduction effect.

[0003] At present, the monitoring content of the building energy consumption monitoring system in China is generally required to be divided according to energy types, and generally includes water, electricity, gas, central heating, central cooling and other renewable energy and the like. Water consumption is the total amount of water used by the building, gas consumption is mainly gas used in the kitchen, dining room and heat source, and electricity consumption can be divided into socket electricity, air conditioner electricity, power electricity and special electricity and the like. In the building construction stage, the installation position of the design meter is required to be divided according to the building energy consumption analysis, and the building energy consumption sub-metering is closely related to the building daily use behavior. Accurate metering of the electric meter can not only feed back the building energy consumption level, but also is related to the interests of the society and the power grid, and plays an extremely important role in the power grid system. However, in the building electricity consumption data collected by the industry at present, noise data, empty data, destructive data and repeated data exist in the electricity consumption data of almost every building. If these data are directly imported into the data analysis system without processing, not only the relatively complex neural network and machine learning algorithm will output incorrect results, but also the basic descriptive statistical analysis and the same ring analysis will lose significance. Therefore, the data should be thoroughly cleaned before entering the calculation process to improve the overall data quality.

[0004] The dirty data of the electricity data generally appears in four situations: 1. data missing; 2. destructive data far exceeding the meter range; 3. noise data generated due to system fluctuation; and 4. repeated data recorded in the database. It can be seen that the dirty data of the electric meter shows certain characteristics. SUMMARY

[0005] The purpose of the application is to preprocess the dirty data.

[0006] In order to achieve the above purpose, the technical scheme of the application provides a processing method for intelligent electric meter remote electric energy time sequence data, characterized by comprising the following steps:

[0007] Step 1: The historical electric energy data in the database is de-duplicated to ensure that the time sequence data recorded in the database is unique.

[0008] Step two: according to the collection frequency and collection cycle of electric energy data, an empty time series is constructed, and the historical electric energy data obtained in the previous step is filled into the corresponding time series, so as to mark the missing values of the original electric energy time series data;

[0009] Step two is used to construct an empty time series. Because the collection of meter data is often collected after a data is marked as a time point and directly stored in the database, it cannot automatically mark which is the null value. Step two is to actively set the step size to mark the unrecorded null value.

[0010] Step three: calculate the data completeness, use the percentage of missing values in total data to represent the data completeness; if the percentage of null data in total data is greater than the pre-set threshold, the data obtained in the previous step is regarded as worthless data, and the meter continues to collect data, and after new historical data is obtained in the database, it returns to step one again to avoid affecting the overall accuracy due to filling too much data; otherwise, go to the next step.

[0011] Step three is used for data integrity check. Generally, data with more missing values than the pre-set threshold is no longer statistically valuable.

[0012] Step four: process the abnormal large value of electric energy time series due to power failure of the branch where the meter is located, that is, the value obviously exceeding the meter range or the maximum load of the branch when the power is turned on after power failure (the feature of this power consumption data is that a large abnormal value suddenly appears in the time series, and the next value returns to the daily power consumption level), and the normal value is replaced by the abnormal value.

[0013] Step four is to process the abnormal large value of the data. These abnormal values may be due to the large starting current of the DC motor, or the high instantaneous electric energy recorded by the meter, resulting in a large abnormal value. Even some meters have more than 10 7 KWh of daily power consumption, which is obviously not reasonable, so the error data needs to be modified.

[0014] Step five: process the occasional missing values of electric energy time series (there are data before and after): because the power consumption equipment and behavior often change greatly in a long period of time, the mean value of the previous and subsequent time periods is used to fill the missing values.

[0015] Step five is to supplement the occasional missing values. Without this step, step six cannot be performed.

[0016] Step six: mark the noise value in the electric energy time series. This noise value is a characteristic noise data that the electric energy is extremely small at the previous time point and extremely large at the next time point due to sudden power failure and subsequent start of the meter. Replace the noise value.

[0017] Step seven: linear interpolation is used to process the continuous null value of the power time series;

[0018] Step eight: the processed power time series is checked for abnormal data by descriptive statistical analysis, and after verification, subsequent statistical analysis or modeling is carried out.

[0019] Preferably, in step four, the specific method for processing the abnormal large value of power recorded after the power supply of the branch where the electric meter is located is turned on is:

[0020] If the previous time period of the abnormal large value is null, and the abnormal large value is greater than twice the valid power consumption value of the next time period, replace the abnormal value with the next data;

[0021] If the next data is null, traverse the time series data, and if the data at a certain time meets the condition that the data before and after it are null, but the data at the time is greater than twice the 0.75 quantile, replace the data with the mean of all current data.

[0022] Preferably, in step six, the replacement method includes:

[0023] Select the power data of the continuous four time series, V1, V2, V3, and V4 in turn:

[0024] If Then

[0025] If Then let V3 = V4.

[0026] Preferably, in step seven, linear interpolation is used to process the missing values, that is, let the values of the function y = f(x) at two points x0 and x1 be y0 and y1 respectively, then construct a first-order polynomial interpolation

[0027]

[0028] Using the first-order polynomial interpolation Calculate the missing values in the power data to fill in the missing data in the time series data.

[0029] The present application provides a solution to the problem of missing, noisy, repeated and other dirty data in the time series history data transmitted by smart meters and stored in the database, and performs data preprocessing. By traversing the data records in the database, the corresponding deletion, filling and reasonable modification of different types of dirty data are performed to realize data cleaning, ensure the overall accuracy of the data and make the data statistically meaningful, and provide a data basis for subsequent data analysis, feature extraction, energy consumption prediction and energy saving strategy use. In the process of energy consumption data analysis, the present application processes the data collected in the database, avoids data loss and repetition caused by data collection at the hardware layer and data forwarding through various networks and devices, and makes the cleaned data directly available for analysis, which is an indispensable part of big data engineering.

[0030] Compared with the invention patent with the application publication number CN106022640A published on October 12, 2016, the present application does not need to collect additional power information in other systems, but directly processes data missing, abnormality and noise through data cleaning and reasonable replacement. On the one hand, it reduces the increase in workload caused by additional collection of other systems, and on the other hand, it avoids the problem of verification errors caused by inaccurate data in other systems. Compared with the invention patent with the application publication number CN111833589A published on October 27, 2020, the present application does not need to collect additional verification data of the gateway meter, nor does it need to occupy existing network resources. The present application does not need to remotely or manually obtain the meter verification data, which saves the workload of verification and comparison. Compared with the invention patent with the application publication number CN111552685A published on August 18, 2020, the present application fully considers the types of data missing, errors and abnormalities, and defines the recognition rules of abnormal values and noise values according to the physical characteristics of the circuit and possible events, and completes the data cleaning according to a specific processing order. The present application improves the completeness and quality of the data, and the data used for filling and replacement itself has certain physical meaning, thereby ensuring the relative accuracy and providing a basis for subsequent statistical analysis and modeling.

[0031] The application is beneficial to improving the accuracy of data in the whole in the preprocessing process of electric energy data analysis, and specifically shows that: eliminating data outliers, deleting data repeated values, supplementing data missing values, improving the completeness of existing data and filtering noise data, and reducing data errors. The method provided by the application is suitable for different types of equipment recorded by different manufacturers and different types of electric meters, such as electric energy data generated by fans, water pumps, lighting lamps and the like, and is beneficial to improving the measurement accuracy of building energy consumption sub-metering engineering and improving the reliability of energy management platform energy consumption data. Compared with the verification and comparison processing method, the method provided by the application does not need to remotely or manually obtain electric meter verification data, thereby saving the workload of verification and comparison. Compared with the pre-prepared function cleaning method, the application fully considers the types of data missing, errors and abnormalities, defines the identification rules of abnormal values and noise values according to the physical characteristics of the circuit and possible events, and completes the cleaning of the data according to a specific processing order, thereby improving the completeness and quality of the data. The method of the application can be implemented by code and integrated in the background of an energy management system, and the cleaning program is automatically run before data statistical analysis, thereby saving the time cost of manual data processing and improving the reliability of data. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 A flow chart of intelligent electric meter remote electric energy time sequence data processing provided by the application is shown;

[0033] Figure 2 The electric energy original data of an electric meter in a certain project is shown;

[0034] Figure 3 The data missing value after processing the maximum value is shown;

[0035] Figure 4 The data noise of a certain electric meter after step five processing is shown;

[0036] Figure 5 The data after noise cleaning is shown;

[0037] Figure 6 The large range of missing data treated by step seven is shown;

[0038] Figure 7 The all air conditioner annual energy consumption data after data processing is shown. DETAILED DESCRIPTION

[0039] The application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the application after reading the content taught by the application, and these equivalent forms also fall within the scope defined by the appended claims of the application.

[0040] As Figure 1 shown, the processing method of intelligent electric meter remote power time series data disclosed by the application comprises the following steps:

[0041] Step one: remove data duplication to ensure that the recorded power data is unique at the same time.

[0042] Step two: since the data recorded by the remote electric meter cannot record missing data, according to the data collection frequency and collection period, a complete empty time series is constructed, and the historical data is filled into the empty time series, so as to mark the missing values of the original time series data.

[0043] Step two constructs an empty time series, because the collection of electric meter data is often a data collected at a certain time, and the data with time mark is directly stored in the database, and the database cannot automatically mark which is null. Step two is to actively set the step size to mark the unrecorded null values.

[0044] Step three: calculate the data completeness, that is, the percentage of empty data in total data, if the empty data percentage exceeds 20%, it is considered as worthless data, and the electric meter continues to collect data to avoid affecting the overall accuracy due to filling too much data. If the empty data percentage does not exceed 20%, go to the next step.

[0045] Step three is data integrity check, generally, data with more than 20% missing values is no longer statistically valuable.

[0046] Step four: process the abnormal large value of power recorded by the electric meter after the branch power is off and then powered on, that is, the value obviously exceeding the electric meter range or the maximum load of the branch when the power is off and then powered on: the feature of this power consumption data is that an extremely large abnormal value suddenly appears in the time series, and the next value returns to the daily power consumption level. Replace the abnormal value with the normal value.

[0047] Step four is to process the abnormal large value of the data. These abnormal values may be due to the large starting current of the DC motor, or the accumulated large abnormal value due to the high instantaneous power recorded by the electric meter, or even the daily power consumption of some electric meters exceeding 10 7 KWh, which is obviously not reasonable, so the error data needs to be modified. The specific method is: judge whether the power data of the previous time period of the abnormal large value is null, if it is null and the abnormal large value is greater than twice the value of the next valid power consumption, replace the abnormal value with the next data. Process the case that the next data is null: traverse the time series data, if the data at a certain time meets the condition that the data before and after it are null, but the data at that time is greater than 2 times the 0.75 quantile, replace the data with the mean value of all current data.

[0048] Step five: handle the occasional missing values of electric energy data (both before and after time data): because the use of electrical equipment and behavior often have a longer time interval, the larger changes, the use of the mean value of the previous and subsequent time period to fill in the missing values.

[0049] Step five is to supplement the occasional missing values, without this step, the noise processing of step six cannot be performed.

[0050] Step six: mark the noise values in the electric energy time series, which is due to the sudden power failure of the electric meter, and the subsequent start-up of the electric meter, resulting in the characteristic noise data of the minimum value of the previous time point and the maximum value of the next time point. Replace the way to handle.

[0051] In step six, the replacement method includes:

[0052] Select the electric energy data of the next four time series, V1, V2, V3, and V4:

[0053] If Then

[0054] If Then V3 = V4.

[0055] Step seven: use linear interpolation to handle the continuous null values of the electric energy time series, that is, set the values of the function y = f(x) at two points x0, x1 as y0, y1, then construct a polynomial interpolation

[0056]

[0057] Using the first-order polynomial interpolation Calculate the missing values in the electric energy data to fill in the missing data in the time series data.

[0058] Step eight: check whether there are still abnormal data by using descriptive statistical analysis on the processed data. After verification, subsequent statistical analysis or modeling is performed.

[0059] In data processing, generally, the data can be further used and mined for value after processing the repeated values, missing values, abnormal values with extremely large data, and noise values in the data sequence.

[0060] Now select the air conditioning energy consumption data measured by the intelligent electric meter of a certain project. The electric energy data before and after processing by this method is shown in Figures 2-7 As shown in Figure 2 It can be seen that there are obvious missing values and abnormal large values in the original data. Figure 3It is the data situation after step four removes maximum value processing, in which there are data occasional absence, large range absence and data noise. Figure 4 It is the electric meter noise data situation after step four and five. Figure 5 It is the data situation of a certain block of electric meter after step six noise cleaning, and it can be seen that the noise has been processed. Figure 6 It is the large range data absence of a certain block of electric meter using step seven interpolation processing. Figure 7 It is the data summary analysis after cleaning by the method. It basically reflects the energy consumption seasonal characteristics of air conditioner long time use, and the accuracy is greatly improved after data cleaning, and the real situation of power consumption can be fed back.

[0061] Based on the above process, the processing of the intelligent electric meter remote electric energy time series data is completed. The present application is not limited to the above embodiments, and those skilled in the art can make several improvements and refinements without departing from the principles of the present application, and these improvements and refinements are also considered within the protection scope of the present application. The contents not described in detail in the specification belong to the prior art known to those skilled in the art.

Claims

1. A method for processing time series data of remote-transmitted electrical energy from a smart meter, characterized in that, Includes the following steps: Step 1: Remove duplicate historical electricity data from the database to ensure that the time-series data recorded in the database is unique; Step 2: Based on the acquisition frequency and period of the power data, construct an empty time series, fill the corresponding time series with the historical power data obtained in the previous step, and thus mark the missing values ​​of the original power time series data; Step 3: Calculate the data completeness by using the percentage of missing values ​​to represent the data completeness. If the percentage of empty data is greater than a preset threshold, the data obtained in the previous step is considered worthless. Once the meter continues to collect data and new historical data is available in the database, return to Step 1. Otherwise, proceed to the next step. Step 4: Process abnormally large values ​​of electricity recorded in the electricity time series after a power outage and subsequent power restoration in the branch where the meter is located. These are values ​​that significantly exceed the meter's range or the branch's maximum load when power is restored. Replace the abnormal values ​​with normal values. This includes the following steps: Determine if the electricity consumption data for the period preceding the abnormally large value is empty. If it is empty and the abnormally large value is more than twice the value of the next valid electricity consumption, then replace the abnormal value with the next data. Next, handle the case where the next data point is empty: Iterate through the time series data. If the data at a certain moment satisfies the condition that both the time points before and after it are empty, but the data at that moment is greater than twice the 0.75 quantile, replace the data with the mean of all the current data. Step 5: Handle occasional missing values ​​in the energy time series: fill in the missing values ​​with the mean of the preceding and following time periods; Step Six: Mark the noise values ​​in the electricity time series. This noise value is due to the characteristic noise data caused by the sudden power outage and restart of the meter, resulting in a minimum electricity value at one time point and a maximum electricity value at the next time point. The replacement method is used to process this noise data, which specifically includes the following steps: Select electrical energy data from four consecutive time series, namely V1, V2, V3, and V4: If satisfied but If satisfied Let V3 = V4; Step 7: Use linear interpolation to process continuous null values ​​in the power time series; Step 8: Use descriptive statistical analysis to check for any abnormal data in the processed power time series. After verification, perform subsequent statistical analysis or modeling.

2. The method for processing remote-transmitted energy time-series data of a smart meter as described in claim 1, characterized in that, In step seven, linear interpolation is used to handle missing values. That is, let the values ​​of the function y = f(x) at two points x0 and x1 be y0 and y1, respectively, then a first-order polynomial interpolation is constructed. Using first-order polynomial interpolation Calculate the missing values ​​in the electricity data and use them to fill in the missing data in the time series data.

Images