Data processing method and system of data library

Abstract

The invention discloses a data handling method and a system in database to solve the problem that the single course treatment requires great amount of time and manpower cost and can not handle abnormities in ETL process. The method includes: arrange data synchronous configuration table and data refreshing configuration table; configuration information of the corresponding data source, start up a plurality of synchronous courses in parallel; refreshing grades, and start up a plurality of refreshing courses to make sequential synchronous and refreshing treatment on a plurality of data sources. The invention also includes parallel monitoring treatment; periodically checking synchronous refreshing information table; any abnormity discovered should be recorded in the alarm information table; the invention also includes parallel alarm treatment: Periodically checking alarm information table; alarm information should be sent immediately for new abnormities. The invention enhances the efficiency of synchronization and refreshing and saves a great deal of time and manpower cost. Real-time monitoring ensures timely handling of problems and safe, reasonable, and stable operation of database.

Description

technical field

[0001] The invention relates to data processing of a data warehouse, in particular to a processing method and system for the process of data extraction, transformation and transfer (ETL) in a data warehouse OLAP system. Background technique

[0002] A data warehouse is a subject-oriented, integrated, time-related, and unmodifiable collection of data in enterprise management and decision-making, that is, for all application systems, such as customer relationship management (CRM, Customer Relationship Management) systems, financial systems, etc. , integrate by topic, and record the entire history of changes. With the continuous improvement of enterprise informatization, a large amount of business data has been accumulated within the enterprise, and the data warehouse system is used to uniformly process these mutually independent and scattered data to meet the needs of high-level decision-making and analysis of the enterprise.

[0003] Referring to Figure 1, it...

Examples

Embodiment Construction

[0050] In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0051] The core idea of ​​the present invention is: by setting the data synchronization configuration table and the data refresh configuration table, the ETL process of the data warehouse OLAP system adopts a multi-process parallel synchronous refresh mode, and after the synchronous processing waits for the data source to recover, multiple data sources are started to be synchronized Process; refresh processing According to the completion of the synchronization process of the same data object and the refresh scheduling relationship of the data object, multiple refresh processes are started for multiple data objects; synchronization and refresh processing are started at the same time, and the same data object is refreshed in time afte...

technical field

[0001] The invention relates to data processing of a data warehouse, in particular to a processing method and system for the process of data extraction, transformation and transfer (ETL) in a data warehouse OLAP system. Background technique

[0002] A data warehouse is a subject-oriented, integrated, time-related, and unmodifiable collection of data in enterprise management and decision-making, that is, for all application systems, such as customer relationship management (CRM, Customer Relationship Management) systems, financial systems, etc. , integrate by topic, and record the entire history of changes. With the continuous improvement of enterprise informatization, a large amount of business data has been accumulated within the enterprise, and the data warehouse system is used to uniformly process these mutually independent and scattered data to meet the needs of high-level decision-making and analysis of the enterprise.

[0003] Referring to Figure 1, it...

Examples

Embodiment Construction

[0050] In order to make the above objects, features and advantages of the present invention more comprehensible, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0051] The core idea of ​​the present invention is: by setting the data synchronization configuration table and the data refresh configuration table, the ETL process of the data warehouse OLAP system adopts a multi-process parallel synchronous refresh mode, and after the synchronous processing waits for the data source to recover, multiple data sources are started to be synchronized Process; refresh processing According to the completion of the synchronization process of the same data object and the refresh scheduling relationship of the data object, multiple refresh processes are started for multiple data objects; synchronization and refresh processing are started at the same time, and the same data object is refreshed in time afte...

Images