Data preprocessing system supporting multiple file formats

Abstract

The invention discloses a data preprocessing system supporting multiple file formats. The data preprocessing system includes a central processing module, a configuration file management module, an analysis mode processing module, a separator mode processing module, an EXCEL mode processing module, a dynamic link mode processing module, an arrangement processing module, a sorting processing module, a text file output module, a mysql output module, a kafka output module and a log management module. Through a scientific and reasonable system design, incomplete, inconsistent and irregular data in data analysis and processing work can be sorted and processed into rule data meeting system requirements, and through sorting processing, specified content can be output to a required storage medium, so that the data analysis and processing quality is improved, and the time and difficulty required by actual processing are reduced; standard data required by a follow-up system is met, the supported source data formats are multiple, output media are rich, the intelligent degree is high, the system can be achieved through configuration, and the system is worthy of application and popularization.

Description

technical field [0001] The invention belongs to the technical field of big data processing, and relates to a data preprocessing system supporting multiple file formats. Background technique [0002] The big data processing process can be summarized into four steps, which are collection, import and preprocessing, statistics and analysis, and mining. This step of import and preprocessing needs to solve the data processing problems caused by a large amount of data and data in different formats. , the source data needs to be processed into relatively regular data. There may be different processing methods for data in different formats, but no matter what kind of data, there are always some steps and methods that are common in the entire data processing process. Yes, the data file preprocessing system can complete these general processes of data processing, reduce the time of data processing and simplify the cumbersome processing process. [0003] Existing data preprocessing sch...

AI Technical Summary

Problems solved by technology

[0003] Existing data preprocessing solutions can only process source files in certain specific formats. When encountering source files in new formats, it is difficult to support them. They cannot flexibly process multiple source files through configuration files, and cannot do dynamic sorting. After processing, it can only be output to a text file

Images