Data preprocessing method, system and device in data mining system

Abstract

The invention discloses a data preprocessing method, a data preprocessing system and a data preprocessing device in a data mining system. The data preprocessing correspondingly has a plurality of preprocessing modes in which the execution sequence is set. The invention adopts a main technical scheme which comprises: determining a current preprocessing mode corresponding to the data preprocessing; and when determining that the processing results obtained according to the current preprocessing mode do not need to be combined and the current preprocessing mode is not the last preprocessing mode of the data preprocessing, processing the data to be processed in the current preprocessing mode at the operating nodes, and controlling the operating nodes to process the processing results in a preprocessing mode next to the current preprocessing mode. According to the technical scheme, the transmission process between different nodes for reading the data to be processed and writing the processing results to be written is not used, the data preprocessing efficiency in the data mining system is improved, and the performance of the entire data mining system is improved.

Description

technical field [0001] The invention relates to the technical field of data mining, in particular to a data preprocessing method, system and device in a data mining system. Background technique [0002] Data mining is the process of extracting potentially useful information and knowledge hidden in it from a large number of incomplete, noisy, fuzzy, random practical application data. The data mining process usually includes: data loading, data preprocessing (ETL), data mining algorithm implementation, and result display. Among them, ETL (Extraction-Transformation-Loading, extraction, transformation and loading) accounts for more than 60% of the workload in the data mining process. [0003] ETL is responsible for extracting data from distributed and heterogeneous data sources, such as relational data and flat data files, to a temporary middle layer for cleaning, conversion, and integration, and finally loading them into data warehouses or data marts to become online analysis ...

AI Technical Summary

Problems solved by technology

Since a complete ETL data processing process usually consists of dozens or even hundreds of data processing functional components to complete data preprocessing, it brings a large number of I / O operations to read and write hard disks, and introduces different data during each read and write. Data transmission between data storage nodes leads to low efficiency of parallel processing of data, thus affecting the performance of the entire data mining system

Images