System and method for avoiding data cyclic synchronization among service-based multi-activity data centers

Abstract

The invention discloses a system and a method for avoiding data cyclic synchronization among service multi-active data centers, one service multi-active data center is deployed at each of different geographic positions, and a Kafka cluster in each service multi-active data center is used for realizing data synchronization among the service multi-active data centers. The Import module is used for consuming data of a certain Topic in other service multi-active data centers in real time and storing the data in the Mysql database, the MaxWell component is used for monitoring changes of binlogs in the service multi-active data center in real time and synchronizing incremental data generated in the binlogs to a Kafka cluster, and the ThreadID filtering module is used for consuming data in the Kafka cluster in real time and storing the data in the Mysql database. And filtering data synchronized in a certain external service multi-active data center in the Mysql database, and writing the reserved data into the Topic corresponding to the Kafka cluster in the service multi-active data center. According to the invention, the problem that the data among the service multi-active data centers is cyclically synchronized is solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data synchronization, and in particular, relates to a system and method for avoiding data cyclic synchronization between business multi-active data centers. Background Art

[0002] With the deepening of the Internet and digital transformation, enterprises are increasingly dependent on information systems. Any service interruption may cause significant economic losses and damage to brand reputation. The establishment of a multi-active data center is an important strategic measure for modern enterprises to enhance the resilience of information technology infrastructure, ensure business sustainability and enhance market competitiveness, which is mainly reflected in the following aspects:

[0003] (1) High availability and business continuity of information systems are achieved, ensuring that when a data center's service is interrupted due to natural disasters, technical failures or other unforeseen events, other data centers can seamlessly take over the business, thereby greatly reducing the risk of single point failures and ensuring continuous business operation;

[0004] (2) It can be expanded on demand and flexibly adjust resource allocation according to business growth or changes in regional demand. It can also better cope with sudden traffic and ensure service stability. Especially in global business scenarios, geographically dispersed deployment enhances disaster recovery capabilities and enables multiple active data centers to be closer to users, reducing network latency, improving service response speed, and providing users with a better experience.

[0005] However, the solution for implementing data synchronization between multiple active data centers based on message queues cannot distinguish whether the data is generated by local applications or synchronized from external data centers by default. As a result, these two parts of data will be monitored and collected by the incremental data collection module after being persisted into the warehouse, and then synchronized to other data centers through message queues again, causing data cyclic synchronization problems between multiple active data centers. Summary of the invention

[0006] In response to the problems existing in the prior art, the present invention provides a system and method for avoiding data cyclic synchronization between business multi-active data centers, thereby solving the problem of cyclic synchronization of data between business multi-active data centers when data synchronization between business multi-active data centers is implemented based on message queues.

[0007] To achieve the above-mentioned technical objectives, the present invention adopts the following technical solutions: a system based on avoiding data cyclic synchronization between business multi-active data centers, in which a business multi-active data center is deployed in different geographical locations. When the business data of any business multi-active data center is synchronized using a message queue mechanism, business data changes of other business multi-active data centers are received, or business data changes belonging only to the business multi-active data center are synchronously transmitted to other business multi-active data centers.

[0008] Furthermore, for any business multi-active data center, it includes: Kafka cluster, Mysql database, Import module, MaxWell component and filtering Thread_ID module;

[0009] The Kafka cluster is used to achieve data synchronization between business multi-active data centers;

[0010] The Import module is used to consume the data of a Topic in other business multi-active data centers in real time and store the data in the MySQL database;

[0011] The MaxWell component is used to monitor the changes in binlog in the multi-active data center of this business in real time, and synchronize the incremental data generated in binlog to the Kafka cluster;

[0012] The filtering Thread_ID module is used to consume data in the Kafka cluster in real time, filter out data synchronized from an external business multi-active data center in the Mysql database, and write the retained data into the Topic corresponding to the Kafka cluster in the business multi-active data center.

[0013] Furthermore, the Kafka cluster stores multiple Topics and deploys MirrorMaker 2, which is used to synchronize the data of a Topic in other business multi-active data centers to the Topic corresponding to the current business multi-active data center, or to synchronize the data of a Topic in the current business multi-active data center to the Topic corresponding to other business multi-active data centers.

[0014] Furthermore, the Import module is also used to record the connection thread number Thread_ID for transmitting data to the Mysql database, and write the connection thread number Thread_ID into Redis.

[0015] Furthermore, when the MaxWell component synchronizes the incremental data generated in the binlog to the Kafka cluster, it needs to record the corresponding connection thread number Thread_ID and convert the incremental data into data in Json format.

[0016] Furthermore, the Import modules of other business multi-active data centers only consume the data of Topics synchronized by the Kafka cluster in the multi-active data center of this business.

[0017] Furthermore, the present invention also provides a method for data synchronization based on the system for avoiding data cyclic synchronization between business multi-active data centers, which specifically includes the following steps:

[0018] Step 1: The Import module of a business multi-active data center consumes a Topic 2 in the Kafka cluster of another business multi-active data center in real time through MirrorMaker 2, writes the data in the Topic 2 into the MySQL database, records the connection thread number Thread_ID between the data transmission to the MySQL database, and writes the connection thread number Thread_ID into Redis;

[0019] Step 2: The MaxWell component monitors the binlog changes of the business multi-active data center in real time, generates Json format data from the incremental data generated in the binlog, synchronizes it to the corresponding Topic1 in the Kafka cluster of the business multi-active data center, and records the corresponding connection thread number Thread_ID;

[0020] Step 3, the filtering Thread_ID module of the business multi-active data center consumes Topic1 in the Kafka cluster in real time, compares the connection thread number Thread_ID of the data record in Topic1 with the connection thread number Thread_ID in Redis, and writes the data corresponding to the connection thread number Thread_ID in Redis that is inconsistent with Topic1 into Topic2 of the Kafka cluster of the business multi-active data center;

[0021] Step 4: The Import module of other business multi-active data centers only consumes Topic 2 written by the Import module of the business multi-active data center.

[0022] Compared with the prior art, the present invention has the following beneficial effects: the system and method for avoiding data cyclic synchronization between business multi-active data centers in the present invention consumes data in the Kafka cluster in real time by filtering the Thread_ID module, filters out the data synchronized by an external business multi-active data center in the Mysql database, and writes the retained data into the Topic corresponding to the Kafka cluster in this business multi-active data center, effectively ensuring that the data received by this business multi-active data center from other business multi-active data centers will not be synchronized to other business multi-active data centers again, effectively solving the problem of cyclic synchronization of data between business multi-active data centers, and making it possible to achieve data synchronization using message queues in business multi-active scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A schematic diagram of data flow in a single business multi-active data center in the present invention;

[0024] Figure 2 A schematic diagram of data synchronization based on a system for avoiding data cyclic synchronization between business multi-active data centers according to the present invention;

[0025] Figure 3 This is a relationship diagram between UserID and Thread_ID in a single business multi-active data center in the present invention. DETAILED DESCRIPTION

[0026] The technical solution of the present invention is further explained below in conjunction with the accompanying drawings.

[0027] The present invention provides a system for avoiding data cyclic synchronization between business multi-active data centers. A business multi-active data center is deployed in different geographical locations. When the business data of any business multi-active data center is synchronized using a message queue mechanism, business data changes of other business multi-active data centers are received, and the changed business data comes from the data synchronization service; or business data changes that only belong to this business multi-active data center are synchronously transmitted to other business multi-active data centers. The present invention effectively ensures that the data received by this business multi-active data center from other business multi-active data centers will not be synchronized to other business multi-active data centers again, effectively solving the problem of data being cyclically synchronized between business multi-active data centers, and making it a reality to use message queues to achieve data synchronization in business multi-active scenarios.

[0028] like Figure 1 , for any business multi-active data center, it includes: Kafka cluster, MySQL database, Import module, MaxWell component and filtering Thread_ID module.

[0029] The Kafka cluster is used to achieve data synchronization between business multi-active data centers. Specifically, multiple Topics are stored on the Kafka cluster, and MirrorMaker 2 is deployed. MirrorMaker 2 is used to synchronize the data of a Topic in other business multi-active data centers to the Topic corresponding to the business multi-active data center, or to synchronize the data of a Topic in the business multi-active data center to the Topic corresponding to other business multi-active data centers.

[0030] The Import module is used to consume the data of a Topic in other business multi-active data centers in real time and store the data in the MySQL database. The Import module is also used to record the connection thread number Thread_ID for transferring data to the MySQL database and write the connection thread number Thread_ID into Redis.

[0031] The MaxWell component is used to monitor the changes in binlog in the multi-active data center of this business in real time. Binlog is a binary log file that records all executed DDL and DML statements, which are recorded and saved in binary files in the form of events. The incremental data generated in binlog is converted into data in Json format and synchronized to the Kafka cluster, and the corresponding connection thread number Thread_ID needs to be recorded. The present invention can distinguish the source of business data that needs to be synchronized by recording different connection thread numbers Thread_ID, such as Figure 3 The information_schema.PROCESSLIST table in the MySQL database records in detail the correspondence between the UserID and the connection thread number Thread_ID at that time.

[0032] Traditionally, the data from other business multi-active data centers is marked by adding data field identification, and then filtered by marking, which requires the modification of a large number of applications and table structures to add relevant fields, and will increase the length of the data, greatly increasing the extra overhead of the information system. In the present invention, the filtering Thread_ID module is used to consume the data in the Kafka cluster in real time, filter out the data synchronized by an external business multi-active data center in the Mysql database, and write the retained data into the Topic corresponding to the Kafka cluster in the business multi-active data center, effectively ensuring that the data received by the business multi-active data center from other business multi-active data centers will not be synchronized to other business multi-active data centers again. The method is simpler and easier to implement, and because the connection thread number Thread_ID will not increase sharply in a short period of time, the extra overhead can be basically ignored.

[0033] The Import module of other business multi-active data centers only consumes the data of topics synchronized by the Kafka cluster in the multi-active data center of this business, thereby avoiding data cyclic synchronization.

[0034] like Figure 2 The present invention also provides a method for data synchronization based on a system for avoiding data cyclic synchronization between business multi-active data centers, which specifically includes the following steps:

[0035] Step 1: The Import module of a business multi-active data center consumes a Topic 2 in the Kafka cluster of another business multi-active data center in real time through MirrorMaker 2, writes the data in Topic 2 to the MySQL database, records the connection thread number Thread_ID between the data transmission to the MySQL database, and writes the connection thread number Thread_ID to Redis;

[0036] Step 2: The MaxWell component monitors the binlog changes of the business multi-active data center in real time, generates Json format data from the incremental data generated in the binlog, synchronizes it to the corresponding Topic1 in the Kafka cluster of the business multi-active data center, and records the corresponding connection thread number Thread_ID;

[0037] Step 3: The filtering Thread_ID module of the multi-active data center of the business consumes Topic 1 in the Kafka cluster in real time, compares the connection thread number Thread_ID of the data record in Topic 1 with the connection thread number Thread_ID in Redis, and writes the data corresponding to the connection thread number Thread_ID in Redis that is inconsistent with that in Topic 1 to Topic 2 of the Kafka cluster of the multi-active data center of the business;

[0038] Step 4: The Import module of other business multi-active data centers only consumes Topic 2 written by the Import module of this business multi-active data center.

[0039] The present invention mainly solves the major problem of cyclic synchronization of data between business multi-active data centers, making it possible to achieve data synchronization using a message queue in a business multi-active scenario.

[0040] Example

[0041] Assume that a business needs to deploy active-active data centers in three different geographical locations, named DC-1, DC-2, and DC-3. The data synchronization between the three active-active data centers is as follows: Figure 2As shown in the figure, since the three business multi-active data centers are completely equal in logic, DC-1 is used as an example for specific explanation:

[0042] (1) The Import module on DC-1 consumes Topic 2 of the Kafka cluster in DC-2 and DC-3 in real time through MirrorMaker 2, writes the data in Topic 2 to the MySQL database using MySQL User 2, and records the connection thread number Thread_ID that transfers the data in Topic 2 to the MySQL database, and writes Thread_ID to Redis;

[0043] (2) The MaxWell component on DC-1 turns on the component recording Thread_ID option, monitors the changes in the binlog in DC-1 in real time, generates Json format data from the incremental data generated in the binlog, and synchronizes the Json data to Topic1 of the Kafka cluster in DC-1 as a producer, and records the connection thread number Thread_ID of the incremental data synchronized to Topic1;

[0044] (3) The filtering Thread_ID module on DC-1 consumes Topic 1 of the Kafka cluster in DC-1 in real time, and compares the Thread_ID collected in Topic 1 with the Thread_ID in Redis. If they are the same, no processing is done. Otherwise, the data corresponding to the connection thread number Thread_ID in Redis that is inconsistent with Topic 1 is written to Topic 2 of the Kafka cluster in DC-1.

[0045] (4) The Import modules of DC-2 and DC-3 only consume Topic 2 synchronized by DC-1, so that the entire data synchronization chain forms a closed loop and ensures that the data between the active and active data centers of each business will not be synchronized repeatedly.

[0046] The above are only preferred embodiments of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should be regarded as the protection scope of the present invention.

Claims

1. A system for avoiding data cyclic synchronization between business multi-active data centers, characterized in that: A business multi-active data center is deployed in different geographical locations. When the message queue mechanism is used to synchronize the business data of any business multi-active data center, the business data changes of other business multi-active data centers are received, or the business data changes belonging only to the business multi-active data center are transmitted to other business multi-active data centers synchronously.

2. A system for avoiding data cyclic synchronization between data centers based on business multi-active according to claim 1, characterized in that: For any business multi-active data center, it includes: Kafka cluster, MySQL database, Import module, MaxWell component and filtering Thread_ID module; The Kafka cluster is used to achieve data synchronization between business multi-active data centers; The Import module is used to consume the data of a Topic in other business multi-active data centers in real time and store the data in the MySQL database; The MaxWell component is used to monitor the changes in binlog in the multi-active data center of this business in real time, and synchronize the incremental data generated in binlog to the Kafka cluster; The filtering Thread_ID module is used to consume data in the Kafka cluster in real time, filter out data synchronized from an external business multi-active data center in the Mysql database, and write the retained data into the Topic corresponding to the Kafka cluster in the business multi-active data center.

3. A system for avoiding data cyclic synchronization between data centers based on business multi-active according to claim 2, characterized in that: The Kafka cluster stores multiple Topics and deploys MirrorMaker 2, which is used to synchronize the data of a Topic in other business multi-active data centers to the Topic corresponding to the current business multi-active data center, or to synchronize the data of a Topic in the current business multi-active data center to the Topic corresponding to other business multi-active data centers.

4. A system for avoiding data cyclic synchronization between data centers based on business multi-active according to claim 3, characterized in that: The Import module is also used to record the connection thread number Thread_ID for transferring data to the Mysql database, and write the connection thread number Thread_ID into Redis.

5. A system for avoiding data cyclic synchronization between data centers based on business multi-active according to claim 4, characterized in that: When the MaxWell component synchronizes the incremental data generated in the binlog to the Kafka cluster, it needs to record the corresponding connection thread number Thread_ID and convert the incremental data into data in Json format.

6. A system for avoiding data cyclic synchronization between data centers based on business multi-active according to claim 5, characterized in that: The Import module of other business multi-active data centers only consumes the data of topics synchronized by the Kafka cluster in the multi-active data center of this business.

7. A method for data synchronization based on a system for avoiding data cyclic synchronization between service multi-active data centers as claimed in claim 6, characterized in that: The specific steps include: Step 1: The Import module of a business multi-active data center consumes a Topic 2 in the Kafka cluster of another business multi-active data center in real time through MirrorMaker 2, writes the data in the Topic 2 into the MySQL database, records the connection thread number Thread_ID between the data transmission to the MySQL database, and writes the connection thread number Thread_ID into Redis; Step 2: The MaxWell component monitors the binlog changes of the business multi-active data center in real time, generates Json format data from the incremental data generated in the binlog, synchronizes it to the corresponding Topic1 in the Kafka cluster of the business multi-active data center, and records the corresponding connection thread number Thread_ID; Step 3, the filtering Thread_ID module of the business multi-active data center consumes Topic1 in the Kafka cluster in real time, compares the connection thread number Thread_ID of the data record in Topic1 with the connection thread number Thread_ID in Redis, and writes the data corresponding to the connection thread number Thread_ID in Redis that is inconsistent with Topic1 into Topic2 of the Kafka cluster of the business multi-active data center; Step 4: The Import module of other business multi-active data centers only consumes Topic 2 written by the Import module of the business multi-active data center.

Images