Data cleaning and repairing method

Abstract

The invention relates to a data cleaning and repairing method, which comprises the following steps of: for data errors generated in acquisition and data transmission generated by power distribution network terminal equipment, setting an access module to carry out data cleaning and repairing; For data errors generated by information equipment in the power distribution network information system, setting a conversion module for data cleaning and repairing; For data interaction among a plurality of business information systems of the power distribution network, data badness generated in data processing, setting a rule base cleaning module; For cleaning of historical data stored in a warehouse, setting an offline module for data cleaning and repairing, and carrying out logic checking on the whole historical data. According to the method, data errors and bad data conditions can be efficiently judged, data missed judgment and wrong judgment are prevented, the repair efficiency is ensured, and the upper-layer business application data requirement is met.

Description

【Technical field】 [0001] The invention belongs to the technical field of distributed data processing, and in particular relates to a data cleaning and repairing method. 【Background technique】 [0002] The quality of data is related to the stable operation of the distribution network. If there is no certain data quality improvement method for massive low-value density data, even the most accurate processing logic and the most efficient parallel strategy will not obtain correct results. At present, many data preprocessing and cleaning systems in the field of information science have been introduced into the field of distribution network big data research, but these systems can only be applied to large-scale data with simple Internet logic or regular application scenarios such as image audio and video. It cannot handle the high-speed real-time data of the whole process generated by the continuous and highly automated production process of the power system and the high-dimension...

AI Technical Summary

Problems solved by technology

In view of the poor quality of current distribution network data, different time scales, serious measurement residual pollution, and closed data between systems, it has seriously affected the normal development of current distribution network planning, operation and maintenance, and maintenance, directly leading to auxiliary decision-making and advanced For the problem of inaccurate application results, a new data cleaning and repair method is urgently needed. The present invention can efficiently distinguish data errors and bad data, prevent data omissions and misjudgments, ensure repair efficiency, and meet the requirements of upper-level business application data; Different data error formation mechanisms quickly evaluate quality problems and call corresponding repair modules to realize data quality improvement and repair functions. Differentiated and flexible loading, flexible and adjustable granularity, while ensuring the rapidity and high concurrency of the system

Images