Method and system for mongoDB bidirectional data synchronization based on logical clock

Abstract

The application provides a MongoDB bidirectional data synchronization method and system based on a logical clock, and the method comprises the following steps: changing the received data in a data stream, so that the changed data of the same library table is written into the same partition; an incremental id is maintained in the source end, and the id is assigned to the changed data; the source end periodically performs a checkpoint; the target end consumes the data synchronized, and removes the data according to the incremental id; a dc column is added to the MongoDB table; a write conflict is detected through a logical clock, and the conflict is processed according to a conflict resolution strategy; a master node instance is selected to listen to the changed data stream, and the latest state of the master node instance is obtained. The application realizes the bidirectional data synchronization of MongoDB, guarantees the consistency of data synchronization, solves the write conflict problem, guarantees the reliability of data synchronization, guarantees the real-time performance of data synchronization, guarantees the high availability, and single or few node instance failures will not affect the normal synchronization of data, thereby providing strong support for the construction of business service off-site dual-homing and high availability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, in particular, to the technical field of data synchronization; specifically, to a MongoDB bidirectional data synchronization method and system based on logical clock. BACKGROUND

[0002] With the development of business, the requirement for high availability and high reliability of services is getting higher and higher, and the demand for cross-regional and cross-data center data synchronization of database systems is also increasing.

[0003] MongoDB is one of the most widely used open source NoSQL databases, which stores data in the form of documents. Compared with traditional relational databases, MongoDB has higher scalability and flexibility. However, the support of MongoDB for data synchronization is not perfect, and it only supports one-way data synchronization from the source database to the target database. At present, there is no reliable out-of-box MongoDB bidirectional data synchronization tool.

[0004] The existing MongoDB data synchronization tools mainly have the following two kinds:

[0005] 1. Mongoshake, which is the most commonly used MongoDB data synchronization tool, but Mongoshake only supports bidirectional synchronization of the deeply customized version of MongoDB of Ali, and can only support one-way data synchronization for the open source version of MongoDB;

[0006] 2. Flink CDC, which is the full name of Flink Change Data Capture, is a real-time data synchronization technology realized by using Apache Flink stream processing framework. It can capture and synchronize the incremental changes of the source database in real time, and transmit the changed data to the target system in real time, so as to realize data synchronization. However, Flink CDC can also only support one-way synchronization, and the components are dependent on each other, and the deployment is more complicated. SUMMARY

[0007] Therefore, the purpose of the present application is to develop a MongoDB bidirectional data synchronization method and system based on logical clock, which can support bidirectional data synchronization of MongoDB cluster and realize the ability of off-site dual-homing; the data synchronization platform automatically solves the problem of double-write conflict, and the consistency of data source data and the high availability and reliability of the synchronization platform.

[0008] The present application provides a MongoDB bidirectional data synchronization method based on logical clock, which comprises:

[0009] S1, receiving data from a ChangeStream of a mongodb by a Mongo synchronization service mongo-sync-service, to <database> . <collection>As the key of the kafka cluster (kafka message queue) message, the message is partitioned by the key, so that the change data of the same library table is written to the same topic partition of kafka, ensuring the ordered delivery of the message;

[0010] Specifically, because the data received from the change data stream of mongodb by the mongo synchronization service mongo-sync-service is ordered itself, as long as the order of the message writing to kafka and the order of the message consumption from kafka are guaranteed, and the order of the change data of the same library table is controlled, the ordered delivery of the message can be realized.

[0011] S2, the source end Mongo synchronization service internally maintains an automatic increment id, when listening to the change data stream, every time a data is listened to, the automatic increment id is incremented by 1 (+1), and the automatic increment id is assigned to the change data, and the change data is sent to kafka; The source end Mongo synchronization service performs checkpoint (checkpoint) regularly, sends the latest automatic increment id and resumeToken of the change data stream to the specified topic partition of kafka; The target end Mongo synchronization service consumes the kafka data synchronized from the source end Mongo synchronization service, and removes the duplicate according to the automatic increment id; After the target end Mongo synchronization service completes the writing of a batch of data, the offset is submitted;

[0012] S3, a dc column is added to the Mongo table, and the dc column is maintained by the application end; The Mongo synchronization service judges whether the change data is written by the client or synchronized from the external environment through the value of the dc column;

[0013] S4, write conflict is detected by logical clock, createTime column is set in Mongo table, and the createTime column is used to represent the writing time of a message, and the column value of the createTime column is increasing; Whether there is a conflict is judged by comparing the column value of the createTime column, and the conflict is handled according to the conflict resolution strategy;

[0014] Specifically, there are two conditions to meet the writing conflict:

[0015] 1. The key values written by both ends are the same;

[0016] 2. Both ends write at the same time, that is, when dc1 writes a data a which has not been synchronized to dc2, dc2 also writes data b.

[0017] The first condition is easy to determine, and the second condition is more difficult to determine. As long as it can be determined whether b is written before or after a is synchronized to dc2, it can be determined whether there is a write conflict.

[0018] Therefore, when performing data synchronization, the target end synchronization message carries the message written to the source end, and the source end can receive the latest createTime of the target end. Thus, when the message is synchronized to the target end, it can be determined whether there is a conflict by comparing the createTime;

[0019] S5, using akka cluster to build Mongo synchronization service, using akka cluster singleton (akka cluster singleton) to select an instance as the master node from the akka cluster, the master node listens to the change data stream of Mongo, and pushes to kafka, and the master node also performs checkpoint (checkpoint) regularly, and sends the automatically incremented id, resumeToken and the latest receiving timestamp information to a specified topic of kafka; all other instances outside the master node continuously consume the specified topic to obtain the latest state of the master node instance.

[0020] Specifically, in the source end and the target end, the same is to use the master node instance to consume data, and to perform checkpoint regularly, and to send the automatically incremented id value of each partition received to the specified topic of kafka, and the other instances outside the master node instance continuously consume the specified topic to obtain the latest state of the master node instance.

[0021] The Mongo synchronization service supports high availability and can still run when a single node fails.

[0022] Further, the method for detecting write conflict by logical clock in the S4 step comprises:

[0023] Suppose that the MongoDB cluster dc1 of the source end writes a data a, and the createTime value of the data a is denoted as Ta, after the data Ta is synchronized to the MongoDB cluster dc2 of the target end, dc2 writes data b, and synchronizes to dc1 to overwrite data c; if Tc > Ta, it indicates that there is no conflict; if Tc ≤ Ta, it indicates that there may be a conflict.

[0024] Further, the source end Mongo synchronization service of the S2 step performs checkpoint regularly, and sends the latest automatically incremented id and resumeToken of the change data stream to the specified topic partition of kafka, including:

[0025] If the source end Mongo synchronization service restarts, the state before the restart is obtained, the latest automatic increment id and the resumeToken of the change data stream are obtained from the specified topic partition, and are sent to the specified topic partition of the kafka.

[0026] Further, the de-duplication according to the automatic increment id in the S2 step includes:

[0027] If the id is not incremented, it is repeated data, and the id without increment is directly ignored.

[0028] Further, the S3 step of handing over the dc column to the application end for maintenance includes:

[0029] When the application end writes the dc column data, a fixed value according to the arrangement order number of the application end is written. For example, the value is 1 in the dc1 environment, and the value is 2 in the dc2 environment.

[0030] Further, the S5 step of the other all instances except the master node continuously consuming the specified topic and obtaining the latest state of the master node instance includes:

[0031] When the master node instance is hung up, a new master node instance is selected from the other instances except the master node instance, and service is continued according to the latest checkpoint.

[0032] The application also provides a MongoDB bidirectional data synchronization system based on a logical clock, which is used for implementing the MongoDB bidirectional data synchronization method based on a logical clock, and includes:

[0033] A message passing order module is used for receiving data from a change data stream of a mongodb through a Mongo synchronization service mongo-sync-service, and sending the data to a kafka. <database> . <collection>As the key of Kafka cluster messages, the key is used to partition messages so that the changed data of the same database table are written to the same topic partition of Kafka, ensuring the orderly delivery of messages;

[0034] Message delivery reliability module: The source-side Mongo sync service maintains an auto-incrementing ID internally. When monitoring the change data stream, the auto-incrementing ID is incremented by 1 for each data item monitored, and the auto-incrementing ID is assigned to the change data, which is then sent to Kafka. The source-side Mongo sync service periodically performs checkpoints, sending the latest auto-incrementing ID and the resumeToken of the change data stream to the specified topic partition of Kafka. The target-side Mongo sync service consumes the Kafka data synchronized by the source-side Mongo sync service and deduplicates it based on the auto-incrementing ID. After the target-side Mongo sync service completes writing a batch of data, it commits the offset.

[0035] Service Storm Processing Module: This module is used to add a new DC column to the Mongo table and delegate the DC column to the application for maintenance. The Mongo synchronization service uses the value of the DC column to determine whether the changed data was written by the client or synchronized from the external environment.

[0036] Write conflict handling module: This module detects write conflicts using a logical clock. A createTime column is set in the Mongo table to indicate the time a message was written, with the value of the createTime column increasing. A conflict is determined by comparing the values ​​of the createTime column and handled according to the conflict resolution strategy.

[0037] High-availability module: used to build Mongo synchronization service with Akka cluster. It uses Akka Cluster Singleton to select an instance from the Akka cluster as the master node. The master node monitors Mongo's change data stream and pushes it to Kafka. At the same time, the master node also performs checkpoints regularly and sends the automatically incremented ID, resumeToken and the latest received timestamp information to a specified topic in Kafka. All other instances except the master node continuously consume the specified topic to obtain the latest status of the master node instance.

[0038] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the MongoDB bidirectional data synchronization method based on a logical clock as described above.

[0039] The application further provides a computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor implements the steps of the method for MongoDB bidirectional data synchronization based on a logical clock when executing the program.

[0040] Compared with the prior art, the application has the beneficial effects that:

[0041] The method and system for MongoDB bidirectional data synchronization based on a logical clock provided by the application realize bidirectional data synchronization of a MongoDB cluster, can guarantee consistency of data synchronization, effectively solve the problem of double-write conflict, can guarantee reliability of data synchronization, can guarantee real-time performance of data synchronization, and can guarantee high availability. BRIEF DESCRIPTION OF DRAWINGS

[0042] Various other advantages and benefits will become apparent to those of ordinary skill in the art upon reading the following detailed description of the preferred embodiments. The accompanying drawings are included to provide a description of the preferred embodiments and are not intended to limit the scope of the application.

[0043] In the drawings:

[0044] Figure 1 is a schematic diagram of the overall architecture of the MongoDB bidirectional data synchronization system based on a logical clock of an embodiment of the application;

[0045] Figure 2 is a schematic diagram of the data flow process of the mongo synchronization service of dc1 of an embodiment of the application, which listens to mongo change data streams, processes and filters data, and sends the data to a kafka data stream;

[0046] Figure 3 is a processing logic diagram for realizing high availability of data synchronization of an embodiment of the application;

[0047] Figure 4 is a flowchart of the method for MongoDB bidirectional data synchronization based on a logical clock of the application;

[0048] Figure 5 is a schematic diagram of the composition of a computer device of an embodiment of the application. DETAILED DESCRIPTION

[0049] The detailed description set forth below represents the presently preferred embodiment of the application, examples of which are shown in the drawings. The description below is presented in terms of the following examples, which are not meant to represent the only or even preferred embodiment of the application. Rather, they are provided for explanation purposes to one skilled in the art. The description of the examples is not meant to limit the application to only the described embodiments.

[0050] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used in this disclosure and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0051] It will be understood that, although the terms first, second, third, etc. can be used herein to describe various information, these terms are not intended to denote a temporal sequence. Rather, these terms are used only as distinguishable to refer to the same type of information. For example, a first information can also be termed a second information, and, similarly, a second information can also be termed a first information, without departing from the scope of the present disclosure. Depending on the context, the word "if' as used herein can be interpreted to mean "when" or "in response to determining" or "in response to a determination."

[0052] The embodiments of the present application are further described below.

[0053] The embodiments of the present application provide a MongoDB bidirectional data synchronization method based on a logical clock, as shown in Figure 4 The method comprises the following steps:

[0054] S1, receiving data from a ChangeStream of a mongodb by a mongo synchronization service mongo-sync-service, and sending the data to a local mongo database; <database> . <collection>As the key of the kafka cluster (kafka database) message, the message is partitioned by the key, so that the change data of the same library table is written to the same topic partition of kafka, ensuring the ordered delivery of the message;

[0055] S2, the source end Mongo synchronization service internally maintains an automatic increment id, when listening to the change data stream, every time a data is listened to, the automatic increment id is incremented by 1 (+1), and the automatic increment id is assigned to the change data, and the change data is sent to kafka; the source end Mongo synchronization service periodically performs checkpoint (checkpoint), and sends the latest automatic increment id and the resumeToken of the change data stream to the specified topic partition of kafka; the target end Mongo synchronization service consumes the kafka data synchronized from the source end Mongo synchronization service, and performs deduplication according to the automatic increment id; after the target end Mongo synchronization service completes writing of a batch of data, the offset is submitted;

[0056] The source end Mongo synchronization service periodically performs checkpoint, and sends the latest automatic increment id and the resumeToken of the change data stream to the specified topic partition of kafka, including:

[0057] If the source end Mongo synchronization service restarts, the state before restart is obtained, the latest automatic increment id and the resumeToken of the change data stream are obtained from the specified topic partition, and are sent to the specified topic partition of kafka.

[0058] According to the automatic increment id, the deduplication includes:

[0059] If the id is not incremented, it means that it is repeated data, and the id without increment is directly ignored.

[0060] S3, a dc column is added to the Mongo table, and the dc column is handed over to the application end for maintenance; the Mongo synchronization service judges whether the change data is written by the client or synchronized from the external environment through the value of the dc column;

[0061] The dc column is handed over to the application end for maintenance, including:

[0062] When the application end writes the dc column data, a fixed value according to the arrangement order number of the application end is written. In this embodiment, the value is 1 in the dc1 environment, and the value is 2 in the dc2 environment.

[0063] S4, detecting write conflict through logical clock, setting a createTime column in the Mongo table, using the createTime column to represent the write time of a certain message, the column value of the createTime column tends to increase; judging whether there is a conflict by comparing the column value of the createTime column, and processing according to the conflict resolution strategy;

[0064] The method for detecting write conflict through logical clock comprises the following steps:

[0065] Suppose that the source end MongoDB cluster dc1 writes a data a, the createTime value of the data a is recorded as Ta, after the data Ta is synchronized to the target end MongoDB cluster dc2, the dc2 writes data b and synchronizes to the dc1 to overwrite data c; if Tc > Ta, it indicates that there is no conflict; if Tc ≤ Ta, it indicates that there may be a conflict.

[0066] S5, using the akka cluster to build the Mongo synchronization service, using the akka cluster singleton to select an instance as the master node from the akka cluster, the master node listens to the change data stream of Mongo and pushes to the kafka (as shown by the solid line in Figure 3 ), and the master node also performs checkpoint regularly, and sends the automatically increasing id, resumeToken and the latest receiving timestamp information to a specified topic of the kafka; all the other instances except the master node instance continuously consume the specified topic to obtain the latest state of the master node instance (as shown by the dotted line in Figure 3 ).

[0067] When the master node instance is down, a new master node instance is selected from the other instances except the master node instance, and the new master node instance continues to provide services according to the latest checkpoint.

[0068] In the source end and the target end, the master node instance also consumes data and performs checkpoint regularly, and sends the automatically increasing id value of each partition received to the specified topic of the kafka, and the other instances except the master node instance continuously consume the specified topic to obtain the latest state of the master node instance.

[0069] The Mongo synchronization service supports high availability, and can still guarantee operation when a single node fails.

[0070] The embodiment of the application further provides a MongoDB bidirectional data synchronization system based on a logical clock, which is used for implementing the MongoDB bidirectional data synchronization method based on the logical clock and comprises the following steps of:

[0071] A message passing order module is used for receiving data from a change data stream of the MongoDB through a MongoDB synchronization service mongo-sync-service, and the data is transmitted to the MongoDB through the MongoDB synchronization service mongo-sync-service. <database> . <collection>As the key of Kafka cluster messages, the message is partitioned by key, so that the change data of the same database table are written to the same topic partition of Kafka, ensuring the orderly delivery of messages;

[0072] Message delivery reliability module: The source-side Mongo sync service maintains an auto-incrementing ID internally. When monitoring the change data stream, the auto-incrementing ID is incremented by 1 for each data item monitored, and the auto-incrementing ID is assigned to the change data, which is then sent to Kafka. The source-side Mongo sync service periodically performs checkpoints, sending the latest auto-incrementing ID and the resumeToken of the change data stream to the specified topic partition of Kafka. The target-side Mongo sync service consumes the Kafka data synchronized by the source-side Mongo sync service and deduplicates it based on the auto-incrementing ID. After the target-side Mongo sync service completes writing a batch of data, it commits the offset.

[0073] Service Storm Processing Module: This module is used to add a new DC column to the Mongo table and delegate the DC column to the application for maintenance. The Mongo synchronization service uses the value of the DC column to determine whether the changed data was written by the client or synchronized from the external environment.

[0074] Write conflict handling module: This module detects write conflicts using a logical clock. A createTime column is set in the Mongo table to indicate the time a message was written, with the value of the createTime column increasing. A conflict is determined by comparing the values ​​of the createTime column and handled according to the conflict resolution strategy.

[0075] High-availability module: used to build Mongo synchronization service with Akka cluster. It uses Akka Cluster Singleton to select an instance from the Akka cluster as the master node. The master node monitors Mongo's change data stream and pushes it to Kafka. At the same time, the master node also performs checkpoints regularly and sends the automatically incremented ID, resumeToken and the latest received timestamp information to a specified topic in Kafka. All other instances except the master node continuously consume the specified topic to obtain the latest status of the master node instance.

[0076] The overall architecture of the embodiment of the present invention in a practical application is as follows: Figure 1 As shown, dc1 and dc2 represent two data centers respectively, and a MongoDB cluster is deployed in each data center. The solid arrow represents the flow of data from dc1 mongo to dc2 mongo, and the dashed arrow represents the flow of data from dc2 mongo to dc1 mongo. The components involved include:

[0077] 1. MongoDB cluster, responsible for storing all business data. This component provides the changestream function, which can listen to the change data of the mongo library table in real time through the client API, and can start listening from the specified position of the stream through the resumeToken function.

[0078] 2. Kafka cluster, responsible for caching synchronization data to ensure data transmission reliability and traceability;

[0079] 3. MirrorMaker, a data synchronization component of Kafka, responsible for cross-data center Kafka data synchronization;

[0080] 4. Mongo Sync Service, the core component, has two functions:

[0081] (a) Listen to the change stream in MongoDB, process and filter the data in the data stream according to certain rules, and send it to Kafka;

[0082] (b) Listen to Kafka, receive data sent from the external environment, and synchronize it back to MongoDB, and automatically handle write conflicts in the process;

[0083] The data flow process is as follows:

[0084] 1. The application service writes data to the MongoDB cluster, and the MongoDB persists the data change;

[0085] 2. The Mongo synchronization service captures the change data by listening to the change stream, and sends the data to the Kafka cluster;

[0086] 3. Kafka Mirror Maker synchronizes the change data from the local Kafka cluster to the target data center's Kafka cluster;

[0087] 4. The Mongo synchronization service of the target cluster pulls the new data synchronized from Kafka, executes the conflict resolution strategy, and synchronizes it to the MongoDB cluster of the target cluster.

[0088] Figure 2 The Mongo synchronization service of dcl listens to the mongo change data stream, processes and filters the data, and sends the data stream to the Kafka data flow process. Figure 2 In the figure, there are five messages OPQRS in the mongo change data stream, wherein OQS is a client write, and PR is data synchronized from dcl. The Mongo synchronization service internally maintains an incremental value (indicated by inc) and a timestamp T2, which indicates the maximum value of the createTime of the synchronization message from dcl obtained by the service from the change data stream. Figure 2

[0089] When the Mongo synchronization service receives a message of a client write (i.e., dc=1), the message is added with the incremental value inc and T2, and the message is sent to the Kafka. When a synchronization message from dcl is received (i.e., dc=2), the internally maintained T2 value is refreshed, and the data is filtered out.

[0090] After the Mongo synchronization service of dcl receives the message synchronized from the source, if there is already data with the same primary key in the mongo, the T2 of the message is compared with the createTime of the data inserted into the mongo to determine whether there is a conflict. The following table illustrates the conflict, normal, or abnormal scenarios and the processing logic.

[0091] The "normal" indicates that there is no write conflict problem in data synchronization; the "conflict" indicates that there is a write conflict, and a conflict resolution strategy needs to be performed. A commonly used conflict resolution strategy is to define a priority value for dcl and dcl2 respectively, and the data of the cluster with a high source priority can cover the data with a low priority. The "abnormal" indicates that there is an unexpected problem, which indicates that there may be a configuration error or a serious system failure, and a technician needs to intervene.

[0092]

[0093] In the scheme of the embodiment, it is required that the createTime value in the mongo table is strictly increasing, that is, the createTime of the newly written data is greater than or equal to the old data. However, in actual application, due to the influence of time synchronization, gc, and the like, it is impossible to be strictly increasing, and there is more or less some out-of-order situation. Therefore, a water level mechanism can be introduced, that is, when the synchronization service listens to the change data stream and updates the internal timestamp T value, n seconds are subtracted from the latest createTime, so that the influence of the createTime out-of-order can be effectively avoided, but the possibility of the write conflict alarm is increased to a certain extent.

[0094] ​The logic clock-based MongoDB bidirectional data synchronization method and system of the embodiment realizes bidirectional data synchronization of the MongoDB cluster, guarantees consistency of data synchronization, effectively solves the double-write conflict problem, guarantees reliability of data synchronization, can guarantee that data is not lost no matter network problems or node restarts occur, guarantees real-time performance of data synchronization, synchronization delay is in the order of seconds, guarantees high availability, and single or few node instance failures will not affect normal synchronization of data.

[0095] The embodiment of the application further provides a computer device, Figure 5 is a structural schematic diagram of a computer device provided by the embodiment of the application; as shown in the figure Figure 5 The computer device comprises an input system 23, an output system 24, a memory 22 and a processor 21; the memory 22 is used for storing one or more programs; when the one or more programs are executed by the one or more processors 21, the one or more processors 21 implement the logic clock-based MongoDB bidirectional data synchronization method provided by the above embodiment; wherein the input system 23, the output system 24, the memory 22 and the processor 21 can be connected through a bus or other manners, Figure 5 for example, a bus connection.

[0096] The memory 22 is a readable and writable storage medium of a computer device, and can be used for storing software programs, computer executable programs, such as program instructions of the logic clock-based MongoDB bidirectional data synchronization method described in the embodiment of the application; the memory 22 can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program required by a function; the data storage area can store data created according to use of the device and the like; in addition, the memory 22 can comprise a high-speed random access memory, and can further comprise a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device or other non-volatile solid-state storage device; in some instances, the memory 22 can further comprise a memory remotely arranged relative to the processor 21, and these remote memories can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network and a combination thereof.

[0097] The input system 23 can be used for receiving input digital or character information, and generating key signal input related to user settings and function control of the device; the output system 24 can comprise a display device such as a display screen.

[0098] The processor 21 executes various function applications and data processing of the device by running software programs, instructions and modules stored in the memory 22, that is, implements the above-described logic clock-based MongoDB bidirectional data synchronization method.

[0099] The computer device provided above can be used to execute the method for MongoDB bidirectional data synchronization based on a logical clock provided by the above embodiments, and has corresponding functions and beneficial effects.

[0100] The embodiments of the present application also provide a storage medium containing computer executable instructions, which are used to execute the method for MongoDB bidirectional data synchronization based on a logical clock provided by the above embodiments when executed by a computer processor. The storage medium is any various type of memory device or storage device, and the storage medium includes: installation media such as CD-ROM, floppy disk or tape system; computer system memory or random access memory such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory such as flash memory, magnetic media (such as a hard disk or optical storage); registers or other similar types of memory elements, etc.; the storage medium can also include other types of memory or combinations thereof; in addition, the storage medium can be located in a first computer system in which the program is executed, or can be located in a different second computer system, which is connected to the first computer system through a network (such as the Internet); the second computer system can provide program instructions to the first computer for execution. The storage medium includes two or more storage media that can reside in different locations (for example, in different computer systems connected through a network). The storage medium can store program instructions (for example, specifically implemented as a computer program) executable by one or more processors.

[0101] Of course, the storage medium containing computer executable instructions provided by the embodiments of the present application is not limited to the method for MongoDB bidirectional data synchronization based on a logical clock described in the above embodiments, and can also execute related operations in the method for MongoDB bidirectional data synchronization based on a logical clock provided by any embodiments of the present application.

[0102] So far, the technical solutions of the present application have been described in combination with preferred embodiments, but those skilled in the art can easily understand that the protection scope of the present application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the present application, and the technical solutions after the changes or replacements will fall within the protection scope of the present application.

[0103] The above description is only preferred embodiments of the present application and is not used to limit the present application; for those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.< / collection> < / database> < / collection> < / database> < / collection> < / database> < / collection> < / database>

Claims

1. A method for MongoDB bidirectional data synchronization based on logical clock, characterized in that, Comprise: S1, receiving data from a change data stream of a mongodb by a Mongo synchronization service mongo-sync-service, to <database> . <collection>As the key of the kafka cluster message, the message is partitioned by the key, so that the change data of the same library table is written to the same topic partition of kafka, ensuring the ordered delivery of the message;< / collection> < / database> S2, the source end Mongo synchronization service internally maintains an automatic increment id, when listening to the change data stream, every time a data is listened to, the automatic increment id is incremented by 1, and the automatic increment id is assigned to the change data, and the change data is sent to kafka; The source end Mongo synchronization service periodically executes checkpoint, and sends the latest automatic increment id and resumeToken of the change data stream to the specified topic partition of kafka; The target end Mongo synchronization service consumes the kafka data synchronized from the source end Mongo synchronization service, and removes the duplicate according to the automatic increment id; After the target end Mongo synchronization service completes the writing of a batch of data, the offset is submitted; S3, a dc column is added to the Mongo table, and the dc column is maintained by the application end; The Mongo synchronization service judges whether the change data is written by the client or synchronized from the external environment through the value of the dc column; S4, logical clock detection is used to detect write conflict, a createTime column is set in the Mongo table, the createTime column is used to represent the writing time of a message, and the column value of the createTime column is incrementally increasing; Whether there is a conflict is judged by comparing the column values of the createTime column, and the conflict is handled according to the conflict resolution strategy; S5, the Mongo synchronization service is constructed by using the akka cluster, an akka cluster singleton is used to select an instance as a master node from the akka cluster, the master node listens to the change data stream of Mongo, and pushes it to kafka, and the master node also periodically executes checkpoint, and sends the automatic increment id, resumeToken and the latest receiving timestamp information to a specified topic of kafka; All other instances outside the master node continuously consume the specified topic to obtain the latest state of the master node instance.

2. The logical clock based MongoDB bidirectional data synchronization method according to claim 1, wherein, The method for detecting write conflict by logical clock in the S4 step comprises: Suppose that the MongoDB cluster dc1 of the source end writes a data a, the createTime value of the data a is recorded as Ta, after the data Ta is synchronized to the MongoDB cluster dc2 of the target end, the dc2 writes data b and synchronizes to the dc1 to overwrite data c; If Tc> Ta, it indicates that there is no conflict; If Tc ≤ Ta, it indicates that there may be a conflict.

3. The logical clock based MongoDB bidirectional data synchronization method of claim 1, wherein, The source end Mongo synchronization service of the S2 step periodically executes checkpoint, and sends the latest automatic increment id and resumeToken of the change data stream to the specified topic partition of kafka, which comprises: If the source end Mongo synchronization service restarts, the state before the restart is obtained, the latest automatic increment id and the resumeToken of the change data stream are obtained from the specified topic partition, and are sent to the specified topic partition of the kafka.

4. The logical clock based MongoDB bidirectional data synchronization method of claim 1, wherein, The de-duplication according to the automatic increment id of the S2 step includes: If the id is not incremented, it is repeated data, and the id without increment is directly ignored.

5. The logical clock based MongoDB bidirectional data synchronization method according to claim 1, wherein, The S3 step of handing over the dc column to the application end for maintenance includes: When the application end writes the dc column data, a fixed value according to the arrangement order number of the application end is written.

6. The logical clock based MongoDB bidirectional data synchronization method according to claim 1, wherein, The S5 step of the other all instances outside the master node continuously consuming the specified topic to obtain the latest state of the master node instance includes: When the master node instance hangs up, a new master node instance is selected from the other instances outside the master node instance, and the service is continued according to the latest checkpoint.

7. A MongoDB bidirectional data synchronization system based on logical clocks, characterized in that, The MongoDB bidirectional data synchronization method based on a logical clock comprises the following steps: Message passing order module: for receiving data from the change data stream of the mongodb by the Mongo synchronization service mongo-sync-service, transforming the data into a format suitable for the message passing order module, and sending the data to the message passing order module <database> . <collection>As the key of the kafka cluster message, the message is partitioned by the key, so that the change data of the same database table is written into the same topic partition of the kafka, and the ordered delivery of the message is ensured.< / collection> < / database> The message delivery reliability module: an automatic increment id is internally maintained in the source end Mongo synchronization service, when listening to the change data stream, the automatic increment id is incremented by 1 every time a data is listened to, the automatic increment id is assigned to the change data, and the change data is sent to the kafka; the source end Mongo synchronization service performs a checkpoint regularly, and sends the latest automatic increment id and the resumeToken of the change data stream to the specified topic partition of the kafka; the target end Mongo synchronization service consumes the kafka data synchronized from the source end Mongo synchronization service, and performs de-duplication according to the automatic increment id; the target end Mongo synchronization service submits the offset after completing the writing of a batch of data. The service storm processing module: for adding a dc column to the Mongo table, and handing over the dc column to the application end for maintenance; the Mongo synchronization service judges whether the change data is written through the client or synchronized from the external environment through the value of the dc column; The write conflict processing module: for detecting write conflicts through a logical clock, a createTime column is set in the Mongo table, the createTime column is used to represent the writing time of a message, and the column value of the createTime column is incrementally increasing; whether there is a conflict is judged by comparing the column value of the createTime column, and the conflict is handled according to the conflict resolution strategy; The write conflict processing module: for detecting write conflicts through a logical clock, a createTime column is set in the Mongo table, the createTime column is used to represent the writing time of a message, and the column value of the createTime column is incrementally increasing; whether there is a conflict is judged by comparing the column value of the createTime column, and the conflict is handled according to the conflict resolution strategy; High availability module: used to build Mongo synchronization service with akka cluster, using akka cluster singleton to select one instance as master node from akka cluster, the master node listens to Mongo's change data stream and pushes to kafka, at the same time, the master node also performs checkpoint regularly, sending the information of auto-incremented id, resumeToken and latest receiving timestamp to a specified topic in kafka; all other instances outside the master node continuously consume the specified topic to obtain the latest state of the master node instance.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by a processor, implements the steps of the MongoDB bidirectional data synchronization method based on a logical clock according to any one of claims 1-6.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor, when executing the program, implements the steps of the MongoDB bidirectional data synchronization method based on a logical clock according to any one of claims 1-6.

Images