Spark-based data synchronization method in e-commerce platform

Abstract

The invention provides a spark-based data synchronization method in an e-commerce platform. The method belongs to the technical field of data processing, Mysql is used as source data of a relational database in an e-commerce platform to generate incremental data in real time, a binlog log of mysql is read through an ETL tool to obtain the incremental data, and the incremental data is marked and stored in an hbase. In setting a synchronization rule, the labeled data are regularly synchronized from the hbase to hive through spark. The problem of service data synchronization among multiple systems in an e-commerce platform is solved.

Description

technical field [0001] The invention relates to data processing technology, in particular to a spark-based data synchronization method in an e-commerce platform. Background technique [0002] Multiple systems are bound to be involved in the current Internet e-commerce platform, and data synchronization between systems is particularly important. When various departments have different requirements for business data, such as data analysis or report display, the data needs to be migrated from the current business database to the corresponding target library. For data analysis, it is usually extracted to the target library through ETL tools, but for big data platform data lakes such as hive, it is not friendly to delete operations. [0003] Incremental synchronization of DB2 database is carried out, the synchronization strategy is too simple and there is no elastic scaling design, and the remedial error of data synchronization failure is insufficient. [0004] Data synchroniza...

AI Technical Summary

Problems solved by technology

For data analysis, it is usually extracted to the target library through ETL tools, but for big data platform data lakes such as hive, it is not friendly to delete operations

[0003] Incremental synchronization of DB2 database, the synchronization strategy is too simple and there is no elastic scaling design, and the remedial error of data synchronization failure is insufficient

[0004] Data synchronization from the e-commerce platform to the big data platform, because it involves updating (or deleting) historical data, if you choose HIVE as the only storage component, you need to start the HIVE transaction table mechanism, but the disadvantage is that the HIVE transaction table is updating (or deleting) operations The performance is poor, and SPARK cannot be used to read transaction table data for calculation; if HBASE is selected as the only storage component, the advantage is that HBASE can be used to update (or delete), but the disadvantage is that SPARK reads HBASE table data for calculation, and its performance is far from Lower than HIVE, unable to meet the time requirements for aggregate calculations

Images