Data cleaning method for data files and data files processing method

Abstract

The invention discloses a data cleaning method for data files and a data file processing method. The data cleaning method comprises the following steps of S2: determining a cleaning content and a cleaning rule of each data file according to uniform requirements of a data download platform, on data and the uniform requirements of a class system to the data, determined and predetermined by the data of a source system data sheet, and compiling cleaning process steps; S4: generating a cleaning configuration file corresponding to each data file according to the cleaning content and the cleaning rule of each data file; S6: according to the cleaning configuration file, cleaning the data file according to the cleaning process steps. The data cleaning method disclosed by the invention has the beneficial effects that different data files in source systems are extracted for uniformly cleansed (data cleaning), and the data files are presented and shared in an uniform manner, so that an uniform view is provided for the subsequent data process of all levels of data, and the processing burden of the source system can be reduced.

Description

technical field [0001] The present invention relates to the data file processing field of a data warehouse system or a data analysis system, in particular, to a data cleaning method and a data file processing method for a data file. Background technique [0002] Before the emergence of massive data processing methods, massive data processing was basically carried out in their respective source business processing systems. At this time, the business system not only bears the pressure of daily business processing itself, but also undertakes the processing, query, and analysis of massive data. Wait for a lot of work. With the rapid development of the Bank of China's business, business processing is becoming more and more complex, business processes are constantly being updated, and business systems are more and more closely related. The demand for data statistics and analysis by senior management leaders and department-level managers is increasing. All statistical institutes A...

AI Technical Summary

Problems solved by technology

With the rapid development of the Bank of China's business, business processing is becoming more and more complex, business processes are constantly being updated, and business systems are more and more closely related. The demand for data statistics and analysis by senior management leaders and department-level managers is increasing. All statistical institutes All the required basic data are processed in various source business systems (that is, the source systems in this article), which obviously no longer meet the requirements of Bank of China for statistical analysis of data

Images