Data synchronization method, device and system and computer readable storage medium

Abstract

The invention provides a data synchronization method, a data synchronization device, a data synchronization system and a computer readable storage medium, and relates to the technical field of big data and Internet. The method comprises the steps of: when data synchronization is carried out, obtaining access configuration information corresponding to a source storage component; writing source data corresponding to each service instance in the source storage component into a corresponding theme of a distributed log system Kafka in real time; when it is monitored that the synchronization task to be processed exists, obtaining configuration data of the synchronization task, wherein the configuration data comprises a source storage component, a target theme of Kafka and address information of a target storage component; and executing a synchronization task according to the configuration data. According to the method, the access of multiple service instances is supported, the data synchronization requirements among multiple storage components can be met, the real-time performance of data synchronization is realized, so that the business progress can be ensured, and compared with a traditional data synchronization scheme, the data volume imported at a time is greatly reduced, so that the performance of a target storage component is improved.

Description

technical field [0001] The present invention relates to the technical fields of big data and the Internet, in particular to a data synchronization method, device, system and computer-readable storage medium. Background technique [0002] With the continuous development and expansion of business, the amount of data in each business scenario has increased exponentially. The traditional MySQL (relational database management system) can no longer support the analysis of large amounts of data according to various complex business scenarios. Even With the sub-database and sub-table scheme, it is also difficult to generate decision-making reports on a monthly, quarterly, and annual basis and view them in real time. Therefore, according to the needs of faster and more efficient business decision-making, it is necessary to synchronize the data in MySQL to other corresponding purpose storage components. [0003] The traditional data synchronization solution is to iterate according to...

AI Technical Summary

Problems solved by technology

[0004] However, the traditional data synchronization solution has the following disadvantages: only T(today)+1 data import can be performed, which cannot guarantee the real-time performance of data synchronization, and lags behind the overall business progress; due to the large amount of data imported at a time, the destination storage is seriously affected Component performance

Images