Abnormal line width data cleaning and image effect improving method

Abstract

An abnormal line degree data cleaning and image effect improving method, after converting the data format of the SQL database into the Dataframe format, the line degree data defect type and characteristics are analyzed, and the data cleaning is realized in a functionized and parallelized manner through the data cleaning strategy, the present application can realize relatively complex and universal data cleaning strategy based on python language programming, and also has high adaptability when serving various language data analysis algorithms.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power distribution network control, and particularly relates to an abnormal power flow data cleaning and portrait effect improving method. BACKGROUND

[0002] In the collection, transmission and storage process of power flow data, measurement device failure, data transmission line interruption, database management abnormality and the like can all cause data abnormality, and the abnormal data cannot accurately reflect the general law of load change, and can also affect clustering results, especially the clustering algorithm with poor noise resistance, which can easily produce false results. The abnormal data can be divided into three types of distortion data, missing data and redundant data. The distortion data includes permanent power flow data rise and fall, instantaneous power flow peak value and short-term power flow rise and fall. The missing data is usually caused by measurement unit failure or failure of other related elements in the data collection process. The missing data is generally recorded as null. The redundant data is characterized by multiple power flow values in the same day, but only one data is valid. In addition, the above abnormal data has multiple combination conditions. Such data cannot be cleaned and can only be discarded by results.

[0003] The power company usually cleans the power flow data through SQL language and python self-programming. The data cleaning through SQL language can take advantage of the existing database system of the power company to achieve relatively efficient cleaning. The defect of this method is that the SQL language is simple and cannot realize complex cleaning logic, and cannot cope with most actual data situations. The python self-programming can clean different data abnormal situations and data requirements, and is more closely combined with data analysis programs. The disadvantage is that the cleaning efficiency is often difficult to guarantee, and cannot cope with large-scale data situations. At present, there is still a lot of room for improvement in the cleaning method of power flow data. SUMMARY

[0004] The present application proposes an abnormal power flow data cleaning and portrait effect improving method to solve the above problems in the prior art. The technical innovation of the present application is that the cleaning algorithm is written according to the data characteristics of user power consumption data, which is more suitable for power consumption data cleaning and has better cleaning effect than traditional cleaning algorithms. At the same time, the algorithm is based on python language programming, and has high adaptability when serving various language data analysis algorithms.

[0005] The present application is realized by the following technical solutions:

[0006] The application relates to an abnormal row degree data cleaning and portrait effect improving method.

[0007] The application relates to a system for realizing the above method, which comprises a row degree data cleaning module, an electric quantity data acquisition module and an electric quantity data cleaning module.

[0008] Technical effects

[0009] The application detects abnormal values and curve mutations according to average electric quantity, is more in line with electric characteristics, can identify and cope with most abnormal electric types, can effectively avoid clustering abnormalities caused by abnormal data when performing user portrait in an actual industrial environment, and thus improves the portrait effect; compared with a self-programming python cleaning program, the algorithm can shorten the cleaning time by about 2 / 3, and greatly reduces the time consumption when performing user portrait in an industrial environment. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 It is a flowchart of the application;

[0011] Figure 2 It is a schematic diagram of clustering results of uncleaned data in the embodiment, and k=6 in the figure;

[0012] Figure 3 It is a schematic diagram of clustering results of uncleaned data in the embodiment, and k=15 in the figure;

[0013] Figure 4 It is a schematic diagram of clustering results of uncleaned data in the embodiment, and k=17 in the figure;

[0014] Figure 5 It is a schematic diagram of clustering results of uncleaned data in the embodiment, and k=18 in the figure;

[0015] Figure 6 It is a schematic diagram of clustering results of uncleaned data in the embodiment, and k=10 in the figure;

[0016] Figure 7 It is a schematic diagram of clustering results of cleaned data in the embodiment, and k=14 in the figure. DETAILED DESCRIPTION

[0017] As Figure 1 shown, the present embodiment relates to a pandas-based abnormal row degree data cleaning and portrait effect improvement method. When using pandas to clean abnormal row degree data, a data cleaning process is designed for the distortion, missing, and redundant abnormal conditions of row degree data, which specifically includes:

[0018] Step 1, row degree data is removed, duplicated, and sorted. The specific operation includes:

[0019] 1.1) Write a python program to read data from the database to the memory through the cx_Oracle library and SQL language;

[0020] 1.2) Convert the data format through the pandas.DataFrame function;

[0021] 1.3) Remove empty data through the dropna function;

[0022] 1.4) Remove duplicates through the drop_duplicates function;

[0023] 1.5) Re-sort and fill by timestamp through the reindex and apply(pandas.Series.interpolate) functions.

[0024] Step 2, difference to get power data, and remove empty power data. The specific operation includes:

[0025] 2.1) Difference through the diff function;

[0026] 2.2) Remove empty data through the dropna function.

[0027] Step 3, calculate the mean value E ave of positive power data through the sum function and conditional statements.

[0028] Step 4, compare the first point of power data with E ave , delete data outliers, and achieve through the drop function and conditional statements.

[0029] Step 5, compare the power difference data with E ave , delete data outliers, and achieve through the diff function, the drop function and conditional statements, and then restore

[0030] Step 6, remove negative power data through the drop function and conditional statements.

[0031] Step 7, renumber according to the standard timestamp through the reindex function.

[0032] Step 8, linear filling according to the timestamp, realized by the apply (pandas.Series.interpolate) function.

[0033] This embodiment selects the row degree data of a city in southern China, with a time span from 2019-09-30 to 2020-10-01, a total of 54148362 data, containing 892421 users. First, without cleaning, the row degree data is directly subtracted every 7 days to obtain the weekly electricity consumption curve, 52 points for each user. The weekly electricity consumption data is k-means clustered, and the cluster center number is selected from 1 to 20. The IDBI is used as the cluster effect evaluation standard to obtain the clustering effect of several categories of results, and the cluster center number is 6, 15, 17, 18, and 10. The results of each clustering are shown in Figures 2 to 7

[0034] It can be seen that the clustering results are controlled by several abnormal values, and it is completely impossible to obtain reliable user electricity consumption characteristics. Subsequently, the original row degree data is cleaned by the algorithm of the present application. Compared with the self-programming method with the same cleaning logic, due to the change of data format and the implementation of function parallelization calculation, the time spent by the data cleaning in this paper is only 1 / 4 of the self-programming method. The optimal clustering effect is selected by IDBI, and the optimal cluster center number is finally determined as 14, and the clustering result is shown in Figure 7

[0035] As can be seen from the figure, the method has good cleaning effect and high processing speed for data anomalies of row degree data. In a 20-core 64G setting, the above method is run for about 25 minutes to complete the reading and cleaning of all data.

[0036] Compared with the prior art, the method is helpful to improve the efficiency and effect of large-scale data cleaning in the actual environment, and has important significance for data analysis and application.

[0037] The above specific implementation can be adjusted in different ways by those skilled in the art without departing from the principles and purposes of the present application. The protection scope of the present application is subject to the claims and is not limited by the above specific implementation. Each implementation within the scope is subject to the constraints of the present application.​​

Claims

1. An abnormal row degree data cleaning and image effect improving method, characterized in that, After converting the data format of the SQL database into the Dataframe format, the line degree data defect type and characteristics are analyzed, and the data cleaning strategy is realized in a functionized and parallelized manner, including: Step 1, line degree data emptying, de-duplication, and sorting; Step 2, difference to obtain power data, and power data emptying; Step 3, Calculate the positive power data mean E by sum function and conditional statement ave ; Step 4, compare the first point of the power data with E by drop function and conditional statement ave , delete the data abnormal point; Step 5, compare the power difference data with E ave , delete data anomaly growth point, through the diff function difference, through the drop function and conditional statement screening, then restore; Step 6, removing negative values from power data through the drop function and conditional statements; Step 7, renumbering according to standard timestamps through the reindex function; Step 8, linear filling according to timestamps through the apply(pandas.Series.interpolate) function.

2. The abnormal row degree data cleaning and image effect improving method according to claim 1, characterized in that, The step 1 includes: 1.1) Writing a python program to read data from the database to the memory through the cx_Oracle library and SQL language; 1.2) Converting the data format through the pandas.DataFrame function; 1.3) Emptying through the dropna function; 1.4) De-duplication through the drop_duplicates function; 1.5) Reordering and filling according to timestamps through the reindex and apply(pandas.Series.interpolate) functions.

3. The method for cleaning abnormal row data and improving profiling effect according to claim 1, characterized in that, The step 2 includes: 2.1) Difference through the diff function; 2.2) Emptying through the dropna function.

4. An abnormal row data cleaning and image effect improving system for implementing the method of any one of claims 1-3, characterized in that, Including: Line degree data cleaning module, power data acquisition module, and power data cleaning module, wherein: the line degree data cleaning module performs emptying, de-duplication, and sorting operations on the line degree data in sequence, achieving preliminary data cleaning; the power data acquisition module performs difference based on the above line degree data, and the difference period can be flexibly selected according to requirements; the power data cleaning module calculates the average value for the differentiated power data, filters out excessively large and small values, and sudden increases and decreases, and then linearly fills the missing data to obtain complete power consumption data.

Images