Synchronized data optimization and improvement method for data center analysis domain

Abstract

The invention relates to a synchronized data optimization and improvement method for a data center analysis domain. By analyzing data features of correlative service systems, synchronization of history data is designed; by means of logic updating and deletion operations of OGG, the history data which needs to be updated and deleted is quickly located, corresponding operations are conducted on MPPdata, the synchronization efficiency of the history data is greatly improved, the task execution time is shortened, and configuration requirements for synchronization of large table data are met. Meanwhile, by configuring a quality rule for data, the quality of data in the uniform data center analysis domain of the whole services is effectively improved, and a solid foundation is laid for data usage in later analysis-type application based on construction of the analysis domain.

Description

technical field [0001] The invention belongs to the analysis domain of a unified data center for all services, realizes fast positioning and recording of data logic update and deletion operations, improves the synchronization efficiency of historical data, and specifically relates to a method for optimizing and improving data synchronization in the analysis domain of a data center. Background technique [0002] According to the construction requirements of the State Grid's full-service data unified data center analysis domain, the State Grid Fujian Electric Power Company has gradually carried out the access work of 24 sets of business systems. In the process of data access and subsequent transfer, it is necessary to use OGG and ETL tools for data synchronization. It is found that the update and delete operations of historical data cannot be synchronized, and the update operation efficiency is relatively low. [0003] The deletion operation cannot be synchronized: using OGG f...

AI Technical Summary

Problems solved by technology

In the process of data access and subsequent transfer, it is necessary to use OGG and ETL tools for data synchronization. It is found that the update and delete operations of historical data cannot be synchronized, and the update operation efficiency is relatively low.

[0003] The deletion operation cannot be synchronized: using OGG for data synchronization can realize the addition, deletion and modification of data, so that the two ends of the data synchronization can realize data mirroring; however, due to the data mirroring at both ends, that is, the source end is deleted, and the target end is also deleted; subsequent data If the transfer is synchronized by ETL, the deletion operation at the source cannot be obtained, resulting in inconsistent data at both ends

[0004] The update operation cannot be synchronized: According to the data structure of the State Grid analysis domain, the data buffer only retains incremental data

When the business system updates the historical data, the data buffer of the analysis domain has deleted the historical data, and the update operation cannot be obtained, and the subsequent data flow cannot update the historical data.

Images