Ignite-based primary-backup data synchronization method

Abstract

The application provides a kind of ignite-based primary and backup data synchronization method in the technical field of distributed data synchronization, comprising: step S10, two clients start corresponding transaction operation respectively;Step S20, data backup master node generates two transaction managers, and transaction manager receives the activation transaction request sent by two clients respectively;Step S30, data backup master node applies for corresponding number of high water level marks for two clients, and informs two data backup secondary nodes to enter transaction activation state through transaction manager;Step S40, when two clients enter commit phase, data backup master node records wal file first, determines pre-application resource, and then initiates transaction synchronization request to two data backup secondary nodes;Step S50, when node appears down and recovers, primary and backup data synchronization is carried out;Step S60, when data backup master node crashes in commit phase and cannot be pulled up, primary and backup data synchronization is carried out.The application has the advantages that primary and backup data synchronization is realized.

Description

Technical Field

[0001] This invention relates to the field of distributed data synchronization technology, and in particular to a master-slave data synchronization method based on Ignite. Background Technology

[0002] Data replication refers to the redundancy provided for data in a distributed system, that is, persisting the same data on different nodes. When the data stored on one node is lost, it can be reread from the replica data stored on other nodes. Data replication is the only way to solve data loss and anomalies in a distributed system.

[0003] Ignite is a memory-centric distributed database, caching, and processing platform that can handle transactional, analytical, and streaming workloads at memory-level speeds across petabytes of data. Ignite transactions possess Atomicity (A), Consistency (C), Isolation (I), and Durability (D), collectively known as ACID. Atomicity means that all operations within a transaction either complete entirely or not at all, with no intermediate states. Errors occurring during transaction execution are rolled back, making the entire transaction appear as if it had never been executed. Consistency ensures system consistency. Isolation prevents transactions from interfering with each other; the intermediate state of one transaction is not perceptible to other transactions. Durability preserves all data changes made by the transaction within the database, even during power outages or system crashes.

[0004] However, Ignite lacks a solution for transactional caching of data copies. Therefore, three methods have emerged: First, the XA method, which, regardless of whether it's 2PC or 3PC, suffers from data inconsistency due to failures during the commit phase and has poor performance, making it more resource-oriented; second, the TCC method, whose Try-Confirm-Cancel processes are highly intrusive to the code, heavily reliant on high-quality coding, making it more business-oriented; and third, the eventual consistency method, which is only suitable for time-insensitive applications.

[0005] Therefore, how to provide a primary-backup data synchronization method based on Ignite to achieve primary-backup data synchronization has become an urgent technical problem to be solved. Summary of the Invention

[0006] The technical problem to be solved by this invention is to provide a primary-backup data synchronization method based on Ignite to achieve primary-backup data synchronization.

[0007] This invention is implemented as follows: a primary / backup data synchronization method based on Ignite, comprising the following steps:

[0008] Step S10: The first client starts the first transaction operation based on Ignite, and the second client starts the second transaction operation based on Ignite;

[0009] Step S20: The data backup master node generates two transaction managers, which respectively receive activation transaction requests sent by the first client and the second client;

[0010] Step S30: The data backup master node requests a corresponding number of high-water mark for the first client and the second client, and notifies the first data backup slave node and the second data backup slave node to enter the transaction activation state through the transaction manager;

[0011] In step S40, when the first client and the second client enter the commit phase, the data backup master node first records the WAL files of the first transaction operation and the second transaction operation, determines the corresponding pre-requested resources, and then initiates a transaction synchronization request to the first data backup slave node and the second data backup slave node to perform master-slave data synchronization.

[0012] Step S50: When the primary data backup node, the first secondary data backup node, or the second secondary data backup node crashes and recovers, perform primary and backup data synchronization.

[0013] Step S60: When the data backup master node crashes during the commit phase and cannot be restarted, perform master-slave data synchronization.

[0014] Further, step S50 specifically includes:

[0015] Step S51: When the data backup master node, the first data backup slave node, or the second data backup slave node crashes and recovers, rejoin the cluster;

[0016] Step S52: Select a coordinator from the data backup master node, the first data backup slave node, and the second data backup slave node, and collect the transaction counter through the coordinator;

[0017] Step S53: Determine the water level status based on the collected transaction counters. If it is a low water level mark, it means that there is missed data. If it is a high water level mark, it means that there is no missed data. The node corresponding to the low water level mark is the sender, and the node corresponding to the high water level mark is the receiver.

[0018] Step S54: The receiver resets the partition status to migration in order to perform primary and backup data synchronization.

[0019] Further, step S60 specifically includes:

[0020] When the primary data backup node crashes during the commit phase and cannot be restarted, it triggers a baseline change in Ignite, selecting one of the first and second secondary data backup nodes as the primary data backup node and resetting the transaction counter to synchronize the primary and backup data.

[0021] The advantages of this invention are:

[0022] The first and second clients initiate first and second transaction operations based on Ignite. The primary data backup node generates two transaction managers to receive activation transaction requests from the first and second clients, and then requests the corresponding number of high-water mark flags for the first and second clients. The transaction managers notify the first and second secondary data backup nodes to enter the transaction activation state. When the first and second clients enter the commit phase, the primary data backup node first records the WAL file and determines the pre-requested resources, and then initiates transaction synchronization requests to the first and second secondary data backup nodes to perform primary-backup data synchronization. When the primary data backup node, the first secondary data backup node, or the second secondary data backup node crashes and recovers, or when the primary data backup node crashes during the commit phase and cannot be restarted, primary-backup data synchronization is performed. That is, based on the Ignite technical system, the transaction XA scheme is used to solve the original shortcomings of the XA scheme, such as ensuring data consistency after node restart when nodes go offline in each stage; and ensuring primary-backup data consistency when the primary data backup node fails and cannot be restarted during the commit phase, thus achieving primary-backup data synchronization. Attached Figure Description

[0023] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0024] Figure 1 This is a flowchart of a master-slave data synchronization method based on Ignite according to the present invention.

[0025] Figure 2 This is a signaling diagram of a primary / backup data synchronization method based on Ignite according to the present invention.

[0026] Figure 3 This is the signaling diagram of the node crash and recovery process of the present invention to perform primary and backup data synchronization.

[0027] Figure 4 This is a signaling diagram of the primary backup data synchronization process when the primary backup node crashes during the commit phase of this invention. Detailed Implementation

[0028] Please refer to Figures 1 to 4As shown, a preferred embodiment of the primary / backup data synchronization method based on Ignite of the present invention includes the following steps:

[0029] Step S10: The first client (client1) starts a first transaction operation (tx1) based on ignite, including key1, key2 and key3; the second client (client2) starts a second transaction operation (tx2) based on ignite, including key4 and key5.

[0030] Step S20: The data backup primary node generates two transaction managers, which respectively receive activation transaction requests sent by the first client and the second client;

[0031] Step S30: The data backup master node requests a corresponding number of highwatermarks for the first client and the second client, and notifies the first data backup slave node (backup1) and the second data backup slave node (backup2) to enter the transaction activation state through the transaction manager;

[0032] For example, apply for three high-water mark flags for the first client: LWM:0, HWM:3; Seq = []; apply for two high-water mark flags for the second client: LWM:0, HWM:5; Seq = [];

[0033] The primary data backup node, the first secondary data backup node, and the second secondary data backup node are all equipped with transaction counters;

[0034] In step S40, when the first client and the second client enter the commit phase, the data backup master node first records the WAL files of the first transaction operation and the second transaction operation, determines the corresponding pre-requested resources, and then initiates a transaction synchronization request to the first data backup slave node and the second data backup slave node to perform master-slave data synchronization.

[0035] When client2 enters the commit phase, the primary first records the WAL file, determines the update range to be the two pre-requested resources 4 and 5, and marks the transaction as LWM:0, HWM:5; Seq = [(3,2)]. Then, it initiates a transaction synchronization request to backup1 and backup2, and performs the same operation in the same order: first record the WAL file, then record the transaction as LWM:0, HWM:5; Seq = [(3,2)].

[0036] When client1 enters the commit phase, the primary first records the WAL file, determines the update range to be the three pre-requested resources 1, 2, and 3, and checks the transaction markers. There is no gap between the high and low watermarks, so it is marked as LWM:5, HWM:5; Seq = []. Then, it initiates a transaction synchronization request to backup1 and backup2, and performs the same operation in the same order: first record the WAL file, then record the transaction markers as LWM:5, HWM:5; Seq = [].

[0037] The WAL file is used to extend the transaction counter record, which is used for node restart or recovery after failure;

[0038] Step S50: When the primary data backup node, the first secondary data backup node, or the second secondary data backup node crashes and recovers, perform primary and backup data synchronization.

[0039] Step S60: When the data backup master node crashes during the commit phase and cannot be restarted, perform master-slave data synchronization.

[0040] This involves a two-stage transaction counter based on Ignite to achieve primary-backup data synchronization. The high and low watermarks in the transaction data counter can effectively identify committed transactions, uncommitted transactions, and transactions that have not yet started. During primary-backup transaction data synchronization, the entire transaction process is completed in three steps: activating the transaction, reserving resources for the transaction, and committing resources for the transaction.

[0041] Step S50 specifically includes:

[0042] Step S51: When the data backup master node, the first data backup slave node, or the second data backup slave node crashes and recovers, rejoin the cluster;

[0043] Step S52: Select a coordinator from the data backup master node, the first data backup slave node, and the second data backup slave node, and collect the transaction counter through the coordinator;

[0044] Step S53: Determine the water level status based on the collected transaction counters. If it is a low water mark, it means that there is missed data. If it is a high water mark, it means that there is no missed data. The node corresponding to the low water mark is the sender, and the node corresponding to the high water mark is the receiver.

[0045] The lowWaterMark is used to track continuous updates. It only increments when the corresponding update is recorded in persistent storage, there are no lost updates, and the counter value is small. The highWaterMark increments for each update prepared for a transaction and allocates a reserved counter during the transaction preparation phase.

[0046] Step S54: The receiver resets the partition status to migration in order to perform primary and backup data synchronization.

[0047] Based on the XA scheme, if a failure causes the data backup master node to go offline during the commit phase, resulting in data inconsistency, the data backup master node will restart after going offline. It will first find the most recent two-phase transaction data structure in the WAL file and send the transaction counter to the elected coordinator node. After receiving the counter, the coordinator will determine the direction of data synchronization and complete the data migration. This method can quickly identify the direction of cached data synchronization and ensure data consistency.

[0048] Step S60 specifically involves:

[0049] When the primary data backup node crashes during the commit phase and cannot be restarted, it triggers a baseline change in Ignite, selecting one of the first and second secondary data backup nodes as the primary data backup node and resetting the transaction counter to synchronize the primary and backup data.

[0050] Based on the XA scheme, if a failure causes the primary data backup node to crash during the commit phase, resulting in data inconsistency, and the primary data backup node goes offline and can no longer provide services, the baseline version will be changed to ensure data redistribution. At this point in time, the transaction counter is restarted and uses the high-water mark for counting. After the baseline version is changed, the secondary data backup node is promoted to the primary data backup node, still ensuring data consistency.

[0051] For example, two transactions are started on the primary data backup node. Transaction 1 operates on three keys, and transaction 2 operates on two keys. Transaction 1 goes through the preparation phase normally, but the process is relatively long, just to demonstrate the state of unordered transactions. Transaction 2 goes through the preparation phase normally, but after entering the commit phase, the primary data backup node crashes and cannot be recovered. At this time, the transaction counters of the two secondary data backup nodes are both LWM:10, HWM:15, SEQ=[{13,2}]. Because the primary node cannot be restarted, it leaves the cluster, triggering a baseline change. Each node resets its own transaction counter, updating it to LWM:15, HWM:15, SEQ=[]. After the baseline change, the secondary data backup node is promoted to the primary data backup node.

[0052] The baseline is used to maintain safe start-up and shutdown. It maintains the consistency of the order of all nodes and the data storage rules. When the baseline is changed, all operations stop. First, the new location of the data is determined, then the data is migrated, and finally the data route is changed.

[0053] In summary, the advantages of this invention are as follows:

[0054] The first and second clients initiate first and second transaction operations based on Ignite. The primary data backup node generates two transaction managers to receive activation transaction requests from the first and second clients, and then requests the corresponding number of high-water mark flags for the first and second clients. The transaction managers notify the first and second secondary data backup nodes to enter the transaction activation state. When the first and second clients enter the commit phase, the primary data backup node first records the WAL file and determines the pre-requested resources, and then initiates transaction synchronization requests to the first and second secondary data backup nodes to perform primary-backup data synchronization. When the primary data backup node, the first secondary data backup node, or the second secondary data backup node crashes and recovers, or when the primary data backup node crashes during the commit phase and cannot be restarted, primary-backup data synchronization is performed. That is, based on the Ignite technical system, the transaction XA scheme is used to solve the original shortcomings of the XA scheme, such as ensuring data consistency after node restart when nodes go offline in each stage; and ensuring primary-backup data consistency when the primary data backup node fails and cannot be restarted during the commit phase, thus achieving primary-backup data synchronization.

[0055] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A master-slave data synchronization method based on Ignite, characterized in that: Includes the following steps: Step S10: The first client starts the first transaction operation based on Ignite, and the second client starts the second transaction operation based on Ignite; Step S20: The data backup master node generates two transaction managers, which respectively receive activation transaction requests sent by the first client and the second client; Step S30: The data backup master node requests a corresponding number of high-water mark for the first client and the second client, and notifies the first data backup slave node and the second data backup slave node to enter the transaction activation state through the transaction manager; In step S40, when the first client and the second client enter the commit phase, the data backup master node first records the WAL files of the first transaction operation and the second transaction operation, determines the corresponding pre-requested resources, and then initiates a transaction synchronization request to the first data backup slave node and the second data backup slave node to perform master-slave data synchronization. Step S50: When the primary data backup node, the first secondary data backup node, or the second secondary data backup node crashes and recovers, perform primary and backup data synchronization. Step S60: When the data backup master node crashes during the commit phase and cannot be restarted, perform master-slave data synchronization. Step S50 specifically includes: Step S51: When the data backup master node, the first data backup slave node, or the second data backup slave node crashes and recovers, the recovered node is rejoined to the cluster. Step S52: Select a coordinator from the data backup master node, the first data backup slave node, and the second data backup slave node, and collect the transaction counter through the coordinator; Step S53: Determine the water level status based on the collected transaction counters. If it is a low water level mark, it means that there is missed data. If it is a high water level mark, it means that there is no missed data. The node corresponding to the low water level mark is the sender, and the node corresponding to the high water level mark is the receiver. Step S54: The receiver resets the partition status to migration in order to perform primary and backup data synchronization.

2. The primary / backup data synchronization method based on Ignite as described in claim 1, characterized in that: Step S60 specifically involves: When the primary data backup node crashes during the commit phase and cannot be restarted, it triggers a baseline change in Ignite, selecting one of the first and second secondary data backup nodes as the primary data backup node and resetting the transaction counter to synchronize the primary and backup data.

Images