Data extraction management and control method and device, equipment and storage medium

Abstract

The invention discloses a data extraction control method and device, equipment and a storage medium, and the method comprises the steps: obtaining a data extraction operation instruction, and determining first time and second time corresponding to a target table according to the instruction; the first time is the latest update time in the update time of all data rows in the target table after the data extraction operation is executed on the source table corresponding to the target table last time, and the second time is a future time point after the data extraction operation is executed; performing incremental extraction on data from the first time to the second time of the source table to obtain updated data; and loading the update data to the target table to obtain the target table after incremental update. According to the method, the latest updating time of all the data rows in the target table after the data extraction operation is executed last time is used as the starting time of the data extraction operation for incremental extraction, the method does not depend on professional tools and control programs, extraction errors caused by limitation of the professional tools and the control programs are avoided, and the control operability and the data extraction and updating efficiency are improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data extraction management and control method, device, equipment and storage medium. Background technique [0002] In the era of big data, the explosive growth of data will inevitably lead to extremely extensive and diverse data sources. This complex data environment brings great challenges to the processing of big data. How to efficiently extract data from it on demand is the key to gaining a competitive advantage. Existing technologies basically use professional ETL (abbreviation of Extract-Transform-Load, that is, the process of data extraction, transformation, and loading) tools for data extraction or use programs for control, which leads to over-reliance on ETL tools. It takes time to get familiar with ETL tools. On the other hand, because ETL tools have certain limitations, they cannot completely solve the corresponding problems for specific industries, an...

AI Technical Summary

Problems solved by technology

Existing technologies basically use professional ETL (abbreviation of Extract-Transform-Load, that is, the process of data extraction, transformation, and loading) tools for data extraction or use programs for control, which leads to over-reliance on ETL tools. It takes time to get familiar with ETL tools. On the other hand, because ETL tools have certain limitations, they cannot completely solve the corresponding problems for specific industries, and the operability is poor. In addition, when upgrading or problems occur, it may bring unpredictable wrong results

However, direct control through the program increases the workload of program maintenance, which is costly and error-prone

Images