A data cleaning and repair method

Abstract

The present invention relates to a data cleaning and repairing method, the method comprising: setting up an access module to clean and repair data for data errors generated in the collection and data transmission of the distribution network terminal equipment itself; setting up a conversion module to clean and repair data for data errors generated by information equipment in the distribution network information system; setting up a rule base cleaning module for data interaction and data processing between multiple business information systems of the distribution network; setting up an offline module to clean and repair data for the cleaning of historical data that has been stored in the warehouse, and performing a logical investigation on the overall historical data. The present invention can efficiently distinguish data errors and bad data, prevent data leakage and misjudgment, ensure repair efficiency, and meet the data requirements of upper-level business applications.

Description

[Technical field]

[0001] The present invention belongs to the technical field of distributed data processing, and in particular relates to a data cleaning and repair method. [Background technology]

[0002] The quality of data is related to the stable operation of the distribution network. If there is no certain data quality improvement method for massive low-value density data, even the most accurate processing logic and efficient parallel strategy will not get the correct result. At present, many data preprocessing and cleaning systems in the field of information science have been introduced into the field of distribution network big data research, but these systems can only be applied to large-volume data with simple Internet logic or application scenarios with strong regularity such as images, audio and video. They cannot handle the high-speed real-time data of the whole process generated by the continuous and highly automated production process of the power system and the high-dimensional, high-missing and low-correlation historical data accumulated over a long period of time. Therefore, the conventional system architecture is directly used for massive abnormal data processing, resulting in the lack of credibility of the analysis and application results of the distribution network, which in turn affects the accuracy of analysis and calculation decisions. In addition, the distribution network data cleaning and repair system may cause various data quality problems in every stage of the entire life cycle from collection, extraction, transmission, application and storage. The abnormal data types have different mixed characteristics, and different business applications have different requirements for data accuracy and granularity. Therefore, it is very necessary to select an adaptive data cleaning and repair system architecture for the data, and then carry reasonable science and methods to further improve data quality, realize accurate data repair, and improve the business application level in the distribution network field. The current distribution network data has poor quality, different time scales, serious measurement residual pollution, and closed data between systems, which have seriously affected the normal development of current distribution network planning, operation and maintenance, and directly led to inaccurate auxiliary decision-making and advanced application results. A new data cleaning and repair method is urgently needed. The present invention can efficiently identify data errors and bad data, prevent data omissions and misjudgments, ensure repair efficiency, and meet the data requirements of upper-level business applications; it can quickly evaluate quality problems for different data error formation mechanisms and call corresponding repair modules to achieve differentiated and flexible loading of data quality improvement and repair functions, and elastic and adjustable granularity, while ensuring the rapidity and high concurrency of the system. [Summary of the invention]

[0003] In order to solve the above problems in the prior art, the present invention proposes a data cleaning and repair method, which comprises:

[0004] Step 1: For data errors generated in the collection and data transmission of the distribution network terminal equipment itself, an access module is set up to clean and repair the data, so as to realize the cleaning and repair of collection errors, collection errors, remote signaling, remote control and telemetry data transmission errors, etc.;

[0005] Step 2: In view of the data errors generated by the information equipment in the distribution network information system, a conversion module is set up to clean and repair the data, so as to realize the cleaning and repair of the data generated by various communications in the transmission of the network module;

[0006] Step 3: A rule base cleaning module is established to solve the problem of bad data generated during data interaction and data processing between multiple business information systems of the distribution network;

[0007] Step 4: For the cleaning of historical data that has been stored in the database, set up an offline module to clean and repair the data, and perform a logical check on the overall historical data;

[0008] Step 5: Based on the results before and after data cleaning and repair, set up a data quality assessment module to carry out process evaluation before and after cleaning and repair;

[0009] Step 6: Establish a data service module for the cleaned data to provide high-quality data resource services.

[0010] Furthermore, the step 1 is specifically as follows: an access module is established for the collection and data transmission generated by the distribution network terminal equipment itself, and data analysis is performed. Various types of data collected by equipment such as electric meters, automation equipment, charging piles, recording equipment, intelligent indicators, and environmental detectors are transmitted through optical fiber communication, wireless spread spectrum, distribution carrier, Ethernet, and wireless network.

[0011] Furthermore, the protocols relied upon by step 1 include: DLT645-2007, IEC 60870, CAN and network module TCP / UDP protocols for protocol parsing.

[0012] Furthermore, the step 2 is specifically to set up a conversion module for data errors generated by information equipment in the distribution network information system, parse the data packets of the network module, perform error correction through protocol analysis, perform data sampling through a heartbeat sampling mechanism, and perform data analysis sampling at fixed intervals.

[0013] Furthermore, the step 3 is specifically to integrate and fuse multi-source data, establish a rule base to clean the data according to the logical relationship between each business information system, and target the bad data occurring in the data interaction and data processing between multiple business information systems of the distribution network.

[0014] Furthermore, step 4 specifically involves cleaning the historical data that has been stored in the warehouse, including marketing data, operation control data, account data, operation inspection data, planning data, and status monitoring data, setting up an offline module to perform logical investigation on the historical data, and completing the cleaning and repair of massive historical data through distributed parallel cleaning and repair, thereby solving the problem of difficulty in quickly cleaning massive data.

[0015] Furthermore, the step 5 is specifically to establish a data quality assessment module based on the results before and after data cleaning and repair, and to establish an assessment index library including indicators such as data confidence, data missingness, data inconsistency, data logical error, and data duplication rate. Based on these indicators, a process evaluation is carried out before and after cleaning and repair.

[0016] The beneficial effects of the present invention include: being able to efficiently identify data errors and bad data conditions, prevent data omissions and misjudgments, ensure repair efficiency, and meet upper-level business application data requirements; quickly assessing quality issues based on different data error formation mechanisms and calling corresponding repair modules, to achieve differentiated and flexible loading of data quality improvement and repair functions, and elastic and adjustable granularity, while ensuring the system's rapidity and high concurrency.

Brief Description of the Drawings

[0017] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application, but do not constitute an improper limitation of the present invention. In the drawings:

[0018] Figure 1 It is a schematic diagram of the architecture of the data cleaning and repair system of the present invention.

[0019] Figure 2 It is a structural schematic diagram of the data quality assessment system of the present invention.

[0020] Figure 3 It is a schematic diagram of the architecture of the data visualization system of the present invention.

[0021] Figure 4 It is a flow chart of the data cleaning and repairing method of the present invention. [Specific implementation method]

[0022] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments, wherein the illustrative embodiments and descriptions are only used to explain the present invention but are not intended to limit the present invention.

[0023] like Figure 1 As shown, a data cleaning and repair system applied to a data cleaning and repair method of the present invention is described in detail, and the system includes: a physical system, a repair system, a data quality assessment system, a data service system, a business application system, a data visualization system, and a management and control layer system;

[0024] The repair system includes: an access layer module, a conversion layer module, a cleaning layer module and an offline layer module; the repair system is used for data monitoring extraction and data repair; wherein: data monitoring extraction includes extraction of real-time online data and static historical data, and realizes dynamic data capture according to pre-defined rules and metadata; static historical data extraction can be fully extracted or incrementally extracted according to the availability of hardware resources; it is also used to build a full information database based on the public information model of the distribution network; the data repair is a full-process repair, which is used to realize data measurement adjustment, data packet repair, rule base verification and multi-business system data integration for different characteristics and different stages of abnormal data in all modules in the physical system, and at the same time, distributed parallel cleaning is performed for massive static data;

[0025] The physical system includes: a collection layer module, a transmission layer module, a network layer module, a service layer module and a storage layer module; the physical system is used to provide distribution network services, distribution network data transmission, calculation and provision;

[0026] Preferably: the acquisition layer module includes various distribution network terminals and other distribution network secondary equipment, which is used to perform secondary power distribution on various terminal devices and collect various power distribution data;

[0027] Preferably: the transport layer module includes a communication device for real-time data transmission; wherein the communication device includes a wireless communication public network and a communication dedicated line;

[0028] Preferably: the network layer module includes a network layer device for providing network data transmission;

[0029] The network layer equipment is used to connect each distribution network service server; wherein: the network layer equipment includes physical isolation equipment, information bus, channel cabinet, firewall and encryption authentication equipment;

[0030] The business layer module is used to provide services and is the carrier of the business information systems and databases of the distribution network that provide services;

[0031] The storage layer module includes various storage devices and a database built on the storage devices, which is used to provide data storage;

[0032] The access layer module corresponds to the transmission layer module and the collection layer module. Specifically, the access layer module uses a front-end cluster to perform cluster online cleaning on data measurement errors in the collection layer module, data leakage, error transmission, error code, and garbled code problems in the transmission layer module;

[0033] The conversion layer module corresponds to the network layer module, specifically: the conversion layer module performs data packet analysis, verification and error correction based on the application layer gateway for data packet mismatch, misorder, packet loss and other phenomena in the network layer module through the heartbeat sampling mechanism and the network protocol verification and error correction mechanism;

[0034] The cleaning layer module corresponds to the business layer module. Specifically: the cleaning layer module performs rule-based cleaning and data integration on the business logic in the business layer module based on the middleware server;

[0035] The offline layer module corresponds to the storage layer module. Specifically, the offline layer module performs a full scan on the data in the storage layer based on the ETL server to complete the logic investigation and clarification. Preferably, the ETL server is a distributed server, and the full scan is a distributed parallel operation.

[0036] The data quality assessment system is built on a physical system and is used to analyze and classify the causes of possible quality problems in structured data, semi-structured data, and unstructured data in data cleaning and repair methods. For data anomalies, missing data, attribute redundancy, inconsistent precision, and irregular formats, data detection methods based on analysis, rule bases, and statistics, distance, density, and association are used to implement distribution network-oriented outlier detection. Suspected bad data sets are obtained through confidence hypothesis testing, and noise estimation is performed on suspected bad data sets using estimation and identification methods to identify and correct implicit bad data, and implement data variance homogeneity tests and error normality tests.

[0037] Preferably: the structured data is structured data in PMS and GIS; the semi-structured data is semi-structured data in system logs; the unstructured data is unstructured data in video monitoring, customer service audio, etc.;

[0038] The data service system is built on the physical system and is used to provide data services such as self-service data analysis, data retrieval, data market, application interface and data monitoring based on the physical system data;

[0039] The business application system is built on the physical system and is used to provide business applications such as load forecasting, reactive power optimization, grid optimization, economic operation, energy saving and loss reduction;

[0040] As attached Figure 3 As shown, the data visualization system is built on the physical system, business application system, data service system, and data quality assessment system, and is used to visualize data based on dynamic rendering algorithms; it provides a human-computer interaction interface to facilitate user data access;

[0041] The control layer system is built on the physical system, including a data security management module and a data public model module; it is used to provide data security management and modeling for third parties;

[0042] As attached Figure 2 As shown, the data quality assessment system includes a data quality problem module, a basic data assessment module, a deep data assessment module, a distribution network data quality assessment index system construction module, and an index assessment module;

[0043] The data quality assessment system uses the adaptive sliding window online monitoring technology to perform data assessment to obtain the current assessment value. Specifically: obtain the last data assessment time point, the current data assessment time point, and the last assessment value; perform window-by-window assessment on the data at the last data assessment time point and the current data assessment time point based on the sliding window to obtain an assessment value sequence (VALi), where VALi is the assessment value obtained by the i-th sliding window in the assessment value sequence; calculate the current assessment value CurVAL based on the assessment value sequence (VALi) and the last assessment value OldVAL;

[0044] and Where W1 is the weight value of the evaluation value corresponding to the sliding window with the closest time, W n is the weight value of the evaluation value corresponding to the sliding window with the longest time; where:

[0045] Preferably: the sliding window size is set according to the size of the computing resources in the data quality assessment system; when the computing resources are more, the sliding window value is set larger, and vice versa;

[0046] The data quality problem module is used to classify the causes of possible data quality problems, including data anomalies, missing values, redundant attributes, irregular formats, inconsistent precision, different dimensions, missing time scales, data contradictions, etc.

[0047] The basic data evaluation module is used to perform rule base inspection, normative analysis, default inspection, and outlier detection; specifically: using data detection methods based on analysis, rule base, and statistics, distance, density, and association to achieve outlier inspection for distribution network;

[0048] The data depth evaluation module is used to perform variance homogeneity test, error normality test, data noise estimation and confidence hypothesis test; specifically: obtain a suspected bad data set through confidence hypothesis test, use the estimation identification method to perform noise estimation on the suspected bad data set, identify and correct implicit bad data, and implement data variance homogeneity test and error normality test;

[0049] The distribution network data quality evaluation index system construction module is used to quantitatively calculate the integrity I, effectiveness V, reliability R and consistency U of the data to obtain a four-dimensional evaluation value; DQ = {I, V, R, U};

[0050] The index evaluation module is used to perform index expectation calculation to determine whether each of the evaluation values ​​reaches the expected value. If it reaches the expected value, the weight of each evaluation value is determined, and finally the evaluation value itself is evaluated through hierarchical analysis and fuzzy comprehensive evaluation to obtain the index evaluation value; when the four-dimensional evaluation value is presented through the data visualization module, the index evaluation value of the four-dimensional evaluation value itself is presented to the user together;

[0051] As attached Figure 4 As shown, the data cleaning and repair method comprises the following steps:

[0052] Step 1: For data errors generated during the collection and data transmission of the distribution network terminal equipment itself, an access module is set up to clean and repair the data, and realize the cleaning and repair of collection errors, collection errors, remote signaling, remote control, and telemetry data transmission errors; Specifically: For the collection and data transmission generated by the distribution network terminal equipment itself, an access module is set up to analyze the data, and various types of data collected by equipment such as electricity meters, automation equipment, charging piles, recording equipment, intelligent indicators, and environmental detectors are transmitted through optical fiber communication, wireless spread spectrum, distribution carrier, Ethernet, and wireless network. The protocols relied on include: DLT645-2007, IEC 60870, CAN, and network module TCP / UDP protocols for protocol parsing. In the application module, a sliding window is used to receive real-time data volume, and a real-time monitoring and alarm module is set up. When a large amount of missing data occurs, it can promptly alarm the data administrator and remote terminal, and quickly enter the early warning operation state;

[0053] Step 2: For data errors generated by information equipment in the distribution network information system, a conversion module is set up to clean and repair data, so as to realize the cleaning and repair of data generated when various communications are applied in the network module transmission; Specifically: For data errors generated by information equipment in the distribution network information system, a conversion module is set up to parse the data packets of the network module, perform error correction through protocol analysis, perform data sampling through the heartbeat sampling mechanism, and perform data analysis sampling at fixed intervals;

[0054] Step 3: To address the bad data generated during data interaction and data processing between multiple business information systems in the distribution network, a rule base cleaning module is established; Specifically: To address the bad data generated during data interaction and data processing between multiple business information systems in the distribution network, a rule base is established to clean the data by integrating and merging multi-source data based on the logical relationship between each business information system.

[0055] Step 4: For the cleaning of historical data that has been stored in the warehouse, an offline module is set up to clean and repair the data, and the overall historical data is logically checked. Specifically: For the cleaning of historical data that has been stored in the warehouse, including marketing data, operation control data, ledger data, operation inspection data, planning data, and status monitoring data, an offline module is set up to perform a logical check on the historical data, and the cleaning and repair of massive historical data is completed through distributed parallel cleaning and repair, thereby solving the problem of difficulty in quickly cleaning massive data.

[0056] Step 5: According to the results before and after data cleaning and repair, a data quality assessment module is established to carry out a process evaluation before and after cleaning and repair; Specifically: According to the results before and after data cleaning and repair, a data quality assessment module is established, and an evaluation index library including data confidence, data missing degree, data inconsistency, data logical error degree, data duplication rate and other indicators is established. Based on these indicators, a process evaluation is carried out before and after cleaning and repair;

[0057] Step 6: Set up a data service module for the cleaned data to provide high-quality data resource services;

[0058] In the several embodiments provided by the present invention, it should be understood that the disclosed methods and terminals can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

[0059] In addition, the technical solutions in the above embodiments may be combined and replaced with each other without any contradiction.

[0060] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0061] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0062] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any accompanying figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple modules or devices stated in the system claim can also be implemented by one module or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.

Claims

1. A data cleaning and repair method, characterized in that: The method includes: Step 1: For data errors generated in the collection and data transmission of the distribution network terminal equipment itself, an access module is set up to clean and repair the data, so as to realize the cleaning and repair of collection errors, collection errors, remote signaling, remote control and telemetry data transmission errors, etc.; Step 2: In view of the data errors generated by the information equipment in the distribution network information system, a conversion module is set up to clean and repair the data, so as to realize the cleaning and repair of the data generated by various communications in the transmission of the network module; Step 3: A rule base cleaning module is established to solve the problem of bad data generated during data interaction and data processing between multiple business information systems of the distribution network; Step 4: For the cleaning of historical data that has been stored in the database, set up an offline module to clean and repair the data, and perform a logical check on the overall historical data; Step 5: Based on the results before and after data cleaning and repair, set up a data quality assessment module to carry out process evaluation before and after cleaning and repair; Step 6: Set up a data service module for the cleaned data to provide high-quality data resource services; Specifically, step 2 includes setting up a conversion module for data errors generated by information equipment in the distribution network information system, parsing data packets of the network module, performing error correction through protocol parsing, and performing data sampling through a heartbeat sampling mechanism, and performing data parsing sampling at fixed intervals; Specifically, step 5 is to establish a data quality assessment module based on the results before and after data cleaning and repair, and establish an assessment index library including indicators such as data confidence, data missingness, data inconsistency, data logical error, and data duplication rate. Based on these indicators, a process evaluation is carried out before and after cleaning and repair.

2. The data cleaning and repair method according to claim 1, characterized in that: The step 1 is specifically as follows: an access module is established for the collection and data transmission generated by the distribution network terminal equipment itself, and data analysis is performed. Various types of data collected by devices such as electricity meters, automation equipment, charging piles, recording equipment, intelligent indicators, and environmental detectors are transmitted through optical fiber communication, wireless spread spectrum, distribution carrier, Ethernet, and wireless network.

3. The data cleaning and repair method according to claim 2, characterized in that: The protocols relied upon in step 1 include: DLT645-2007, IEC 60870, CAN and network module TCP / UDP protocols for protocol parsing.

4. The data cleaning and repair method according to claim 1, characterized in that: Specifically, step 3 is to integrate and fuse multi-source data to clean data in response to bad data situations that occur during data interaction and data processing among multiple business information systems in the distribution network, and to establish a rule base based on the logical relationship between each business information system.

5. The data cleaning and repair method according to claim 4, characterized in that: Specifically, step 4 includes cleaning the historical data that has been stored in the warehouse, including marketing data, operation control data, ledger data, operation and inspection data, planning data, and status monitoring data. An offline module is set up to perform logical investigation on the historical data, and the cleaning and repair of massive historical data is completed through distributed parallel cleaning and repair methods, thereby solving the problem that massive data is difficult to clean quickly.

Images