A data extraction method and a data extraction system supporting continuous transmission of breakpoints

Abstract

The invention belongs to the technical field of Internet information processing, and discloses a data extraction method and a data extraction system supporting continuous transmission of breakpoints.The data extraction system comprises a configuration database module, a setting module, a data extraction module, a recording and saving module. Firstly, the data extraction method extracts the database according to the extracted data source information, and selects the incremental field. Then, the start time and the interval time are set to extract the maximum number of records at each time, andthe number of records is extracted at each scheduling page to extract data. Then the process scheduling times ,the total process extraction records, the last process extraction records, the current scheduling extraction records, the number of completed pages, the total number of pages are recorded and saved,. After the server fails, the process extraction starts from the next page which has been completed before. The data extraction method of the invention can avoid the server overload, ensure the stability of the server, can not repeatedly extract the completed data, and improves the efficiency of data extraction.

Description

technical field [0001] The invention belongs to the technical field of Internet information processing, and relates to a data extraction method and system supporting breakpoint resume transmission. Background technique [0002] At present, the existing technologies commonly used in the industry are as follows: [0003] In recent years, with the continuous development of the Internet and the rapid changes in human production and lifestyle, a large amount of business data has been accumulated. It is necessary to analyze, mine and extract a large amount of data into useful knowledge to form reports and charts, thus BI was born. ETL (Extract-Transform-Load) is the most important link in the formation of a data warehouse for BI projects. [0004] Commonly used ETL tools such as Datastage, Powercenter, Automatic, Kettle, NIFI, etc. have their own tools or processors for data extraction, transformation and loading. In the process of data extraction, it is necessary to ensure the...

AI Technical Summary

Problems solved by technology

[0006] After the server fails, the data needs to be re-extracted, which makes the extraction efficiency low and wastes time

[0007] In the existing technology, information such as the initial value, interval value, maximum number of extracted records for each scheduling, and number of query database pages are not set according to the analysis data, resulting in the inability to realize data extraction and extraction in batches and pages low efficiency;

Images