A data extraction method and system supporting resumed data transfer

Abstract

The invention belongs to the technical field of Internet information processing, and discloses a data extraction method and system supporting breakpoint resume transmission. The data extraction system includes a configuration database module, a setting module, an extraction data module, and a recording and storage module; the data extraction method is first based on The extracted data source information configures the extraction database, selects the incremental field; then sets the start time, interval time, maximum number of records extracted each time, number of records extracted by each scheduling page, and extracts data; then records and saves the number of process scheduling and the total number of processes. The number of records extracted, the number of records extracted last time in the process, the number of records extracted by the current schedule, the number of completed pages, and the total number of pages. After the server fails, the extraction will start from the next page that has been completed before. The data extraction method of the present invention can avoid excessive load on the server, ensure the stability of the server, prevent repeated extraction of completed data, and improve the efficiency of data extraction.

Description

technical field [0001] The invention belongs to the technical field of Internet information processing, and relates to a data extraction method and system supporting breakpoint resume transmission. Background technique [0002] At present, the existing technologies commonly used in the industry are as follows: [0003] In recent years, with the continuous development of the Internet and the rapid changes in human production and lifestyle, a large amount of business data has been accumulated. It is necessary to analyze, mine and extract a large amount of data into useful knowledge to form reports and charts, thus BI was born. ETL (Extract-Transform-Load) is the most important link in the formation of a data warehouse for BI projects. [0004] Commonly used ETL tools such as Datastage, Powercenter, Automatic, Kettle, NIFI, etc. have their own tools or processors for data extraction, transformation and loading. In the process of data extraction, it is necessary to ensure the...

AI Technical Summary

Problems solved by technology

[0006] After the server fails, the data needs to be re-extracted, which makes the extraction efficiency low and wastes time

[0007] In the existing technology, information such as the initial value, interval value, maximum number of extracted records for each scheduling, and number of query database pages are not set according to the analysis data, resulting in the inability to realize data extraction and extraction in batches and pages low efficiency;

Images