Data real-time synchronization method and device, equipment and storage medium

Abstract

The invention discloses a real-time data synchronization method and device, equipment and a storage medium. The method comprises the steps of creating a plurality of ES indexes based on a synchronization system; the synchronization system creates heterogeneous data based on the updated data; the synchronization system writes the heterogeneous data into the corresponding ES indexes; the synchronization system sends each piece of operation information of the ES to a compensation system; the compensation system determines a compensation target based on the operation information. According to the technical scheme, real-time synchronization of Mysql and ES data is achieved, the data can be directly queried from the ES index without being queried through Mysql full-library full-table scanning, the data query efficiency is improved, the mass data query requirement is met, and service processing and data synchronization are completely decoupled; in addition, data loss caused by various accidents is automatically compensated through timing comparison between heterogeneous data sources, and the data integrity is ensured under the condition that manual intervention is not needed.

Description

technical field [0001] The invention belongs to the technical field of data processing, and in particular relates to a method, device, equipment and storage medium for real-time data synchronization. Background technique [0002] The search engine Elasticsearch (ES for short) can be used as a heterogeneous data source of Mysql to make up for the shortcomings of Mysql sub-database and sub-table query. ES is an open source, distributed, high-performance search engine, each field can be searched as an index field. However, there will be corresponding problems when using ES. For example, how to store the data in Mysql into ES in near real-time synchronously, that is, the problem of data consistency. Due to network fluctuations and other reasons, some synchronous data may be lost. When this scenario occurs, how to compensate for the lost data and ensure data integrity. [0003] At present, the real-time synchronization of Mysql and ES data is realized based on the following two...

AI Technical Summary

Problems solved by technology

However, the first implementation method has a high degree of coupling. Whether the interface caller application or the interface provider application has a system exception, it will affect the real-time business; what's more, it may cause systemic risks; the second implementation method, Although the decoupling between applications is realized; however, if the synchronization between Mysql and ES is realized by relying on business applications to send or consume messages, it will also have a certain impact on real-time business

[0004] In addition, existing heterogeneous data source synchronization schemes cannot automatically detect differences between data sources, nor can they avoid data loss problems caused by accidents

Images