Data extraction method and system based on massive data

Abstract

The invention provides a data extraction method and system based on mass data. The method comprises the steps of according to set conditions, dividing data in an assigned data source list into dynamic source data and static source data; conducting initialization processing on the static source data to obtain static target source data; configuring an extraction assignment for the dynamic source data, executing the extraction assignment and extracting the dynamic source data to obtain dynamic target data; saving the static target data and the dynamic target data as a corresponding static target data file and a corresponding dynamic target data file respectively, and merging the two files into an assigned data warehouse. The system includes a classification module, a statistic data processing module, a dynamic data processing module and a data storage module. According to the data extraction method and system based on the mass data, the quantity of the extraction assignments is reduced, data extraction efficiency is improved, the cost and difficulty of operation and maintenance in the later period are decreased, and the changes and demands of rapid development of services are met.

Description

technical field [0001] The invention relates to the technical field of database data processing, in particular to a data extraction method and system based on massive data. Background technique [0002] Data Warehouse (DW or DWH for short) is a strategic collection that provides all types of data support for enterprise-level decision-making processes. It is a structured data environment for decision support systems (dss) and online analysis application data sources. The data warehouse system is an information providing platform, which obtains data from the business processing system, and provides users with various means to obtain information and knowledge from the data. Therefore, for some enterprises and institutions, they have built their own data warehouses. [0003] The construction of a data warehouse requires multiple steps, from data extraction, storage to use, every step is crucial. Among them, the data extraction as the first step, the efficiency of data extracti...

AI Technical Summary

Problems solved by technology

[0006] In addition, for massive data, according to the existing technology, a data table with a large amount of data can only be extracted simultaneously by creating multi-task segments

This method will lead to many tasks and high maintenance costs in the later period

[0007] Furthermore, the extraction time stamp field in the existing technology is relatively fixed and rigid (usually the time of creation or modification), and cannot be flexibly set according to business conditions or needs. In the end, we can only expect online research and development to carry out structural transformation of the source table

Images