A system and method for automatically cutting databases for etl operations

Abstract

The invention relates to an automatic base switching system and method for ETL operations. The method comprises the steps of scanning the executing situation of the ETL operations at regular intervals; piecing up manual intervention information according to operation information of an ETL operation reported with errors when the ETL operation reported with errors is found; checking the data synchronization situation of a backup base at regular intervals, judging the working state of the backup base, generating a base switching trigger instrument if the backup base is in the data synchronization completing state, switching an operated object of the ETL operation reported with errors to the backup base from a production base, base switching is completed, modifying operation configuration information according the manual intervention information, and the ETL operation is executed in the backup base. After the errors of the operation are reported, the operated object of the operation is switched from the production base to the backup base, and normal extraction of the operation is ensured so that the stability of data extraction can be ensured. The data timeliness of a downstream system is supported, manpower cost is saved, and the working efficiency is obviously increased, so that the system maintenance is more humanized and intelligentized.

Description

technical field [0001] The invention relates to the field of computer technology, in particular to a system and method for automatically cutting databases for ETL operations. Background technique [0002] ETL job refers to the use of ETL tools to extract data from distributed and heterogeneous data sources (such as relational data, flat data files, etc.) to a temporary intermediate layer for cleaning, conversion, integration, and finally loading to a data warehouse or data mart It becomes the basis of online analytical processing and data mining. [0003] At present, the more important systems will have a production library and a backup library (BCV library). The backup library stores the photographic data at a certain moment of production for emergency use. Most of the ETL tools avoid the peak period of data operation to obtain real-time photo data from the production database. However, in the era of highly developed information, the data operation of the production databa...

AI Technical Summary

Problems solved by technology

Most of the ETL tools avoid the peak period of data operation to obtain real-time photo data from the production database. However, in the era of highly developed information, the data operation of the production database is quite frequent, which will lead to the extraction of long-term ETL jobs due to dirty reading. An error is reported, so the data cannot be extracted, and the only way to manually switch to the backup database is to extract the data

This method consumes a lot of manpower, the implementation efficiency is low, and the stability and reliability of the operation cannot be guaranteed.

Images