Python script-based distributed big data cleaning method

Abstract

The invention discloses a python script-based distributed big data cleaning method, which comprises the following steps: firstly, loading to-be-cleaned data, performing fragmentation operation on theloaded to-be-cleaned data, performing distributed scheduling and execution operation on the to-be-cleaned data, and then performing distributed scheduling and execution operation on the to-be-cleaneddata, requesting to-be-cleaned data and backfilling a cleaning result, wherein the step 1 specifically comprises the following sub-steps: data loading: firstly, loading data required to be cleaned from an HBase column storage database, formulating a cleaning strategy and setting a data cleaning strategy; based on the big data technology, conducting data cleaning based on an HBase column storage database, the problem of mass data cleaning is solved, a python engine and scripts are adopted for data cleaning, the problem that traditional SQL cleaning rules are few and the problem of jar package cleaning static coding are solved, a distributed computing engine based on Spark is adopted, the scripts are executed in parallel, and the problem of insufficient big data cleaning computing power is solved.

Description

technical field [0001] The invention relates to the field of data cleaning, in particular to a python script-based distributed big data cleaning method. Background technique [0002] Data cleaning refers to the last procedure for discovering and correcting identifiable errors in data files, including checking data consistency, dealing with invalid and missing values, etc. Unlike questionnaire review, data cleaning after entry is generally done by computer rather than manual Finish. [0003] Existing distributed big data cleaning methods have limited cleaning capabilities and cannot clean a large amount of data, and the existing cleaning methods basically use SQL cleaning rules, which have fewer cleaning rules, resulting in relatively general cleaning effects. The traditional method of cleaning data has insufficient cleaning computing power. Therefore, how to create a distributed big data cleaning method based on python scripts has become an urgent problem to be solved. Co...

AI Technical Summary

Problems solved by technology

[0004]The technical problem to be solved by the present invention is: the existing distributed big data cleaning method has limited cleaning ability and cannot clean a large amount of data, and now Some cleaning methods basically use SQL cleaning rules, and there are fewer cleaning rules, resulting in a general cleaning effect. In addition, the traditional method of cleaning data has insufficient cleaning computing power.

Images