Inter-cluster data synchronization method and system based on CDC

Abstract

The present invention discloses a method and system for synchronizing data between clusters based on CDC, and belongs to the technical field of database. The technical problem to be solved by the present invention is how to achieve high availability between databases and reduce the losses caused by problems in internal components of a database cluster. The technical scheme is as follows: the method uses the CDC function in a ZNBase main cluster to flow data from each table to a receiver, and the receiver adopts a CDC-SINK; after processing, the CDC-SINK plays back the data on a standby cluster, so as to complete the data flow from the ZNBase main cluster to the standby cluster; specifically as follows: data synchronization between the ZNBase main cluster and the standby cluster: after the data flowing out of the ZNBase main cluster is parsed and processed by the CDC-SINK, the temporary data is written into a designated database in the standby cluster; the corresponding data of the standby cluster is written into a table that the standby cluster needs to complete data synchronization with the main cluster; the temporary data in the designated database of the standby cluster is deleted; the consistency between the ZNBase main cluster transaction and the standby cluster transaction.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to an Inspur Yunxi database ZNBase, and more specifically to a CDC-based inter-cluster data synchronization method and system. Background Art

[0002] With the development of the current society, many types of databases have emerged in the field, which can be roughly divided into relational databases, non-relational databases and NEWSQL databases. ZNBase database is one of the NEWSQL databases. As a distributed database, ZNBase database uses the Raft protocol (Raft protocol is a distributed consistency protocol used to ensure the consistency of multiple copies) to achieve high availability and tolerate failures of more than half of the nodes. However, if there is a problem with the internal components of the database cluster, the database cluster will be unusable. Therefore, how to achieve high availability between databases and reduce the losses caused by problems with the internal components of the database cluster is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The technical task of the present invention is to provide a CDC-based inter-cluster data synchronization method and system to solve the problem of how to achieve high availability between databases and reduce the losses caused by problems in internal components of the database cluster.

[0004] The technical task of the present invention is achieved in the following way: a CDC-based inter-cluster data synchronization method, which uses the CDC function in the ZNBase main cluster to flow the data in each table to a receiver, and the receiver uses CDC-SINK; after processing, CDC-SINK plays back the data on the backup cluster, thereby completing the data flow from the ZNBase main cluster to the backup cluster; the details are as follows:

[0005] Data synchronization between the ZNBase master cluster and the standby cluster: After the data flowing out of the ZNBase master cluster is parsed and processed by CDC-SINK, the temporary data is written to a specified database in the standby cluster; then the corresponding data of the standby cluster is written to the table that the standby cluster needs to complete data synchronization with the master cluster; finally, the temporary data in the specified database of the standby cluster is deleted;

[0006] Consistency between ZNBase primary cluster transactions and standby cluster transactions: Data flowing out of the ZNBase primary cluster is processed by CDC-SINK and sent to the standby cluster. The standby cluster determines whether the transaction corresponding to the data is complete based on the timestamp in the data flowing out of the ZNBase primary cluster, and decides whether to apply or roll back based on the integrity of the corresponding transaction.

[0007] Change Data Capture (CDC) feeds efficient, distributed row-level changes into configurable receivers for downstream processing such as reporting, caching, or full-text indexing.

[0008] CDC-SINK is a tool that uses the CDC function to stream data from one ZNBase cluster and then writes it to another ZNBase cluster, thereby synchronizing data between the two clusters.

[0009] As a preferred embodiment, the consistency between ZNBase master cluster transactions and standby cluster transactions is as follows:

[0010] Configure the related tables in the order of primary key table and foreign key table to the config of a CDC-SINK process, and the CDC-SINK process can receive data from all configured tables; and set the inflow order of CDC-SINK to foreign key table-primary key table;

[0011] Put all tables configured by CDC-SINK's config into a pool. When and only when all tables receive the resolved time, send the updated data of all tables before the minimum resolved time, and send them in the order of the tables configured by config to ensure that there are no new updates before this time, thus ensuring the consistency of transactions.

[0012] Among them, the resolved time guarantees that no rows with lower update timestamps (which have not been seen before) will be issued on the partition, that is, all updates before this timestamp have been committed.

[0013] Preferably, the specific requirements for the primary cluster and the backup cluster are as follows:

[0014] ①. Tables in both the ZNBase master and slave clusters must have primary keys and single column families;

[0015] ② All foreign key tables in the standby cluster need to be configured with cascading delete and cascading update;

[0016] ③. The fields of the tables in the ZNBase master and slave clusters cannot have collate fields;

[0017] ④. When the ZNBase master cluster and the standby cluster are synchronized, full synchronization is performed each time.

[0018] Preferably, the CDC-SINK is used as a tool to receive data flowing out from the ZNBase master cluster to the standby cluster and write it to the standby cluster; specifically as follows:

[0019] Receiving data: CDC-SINK is based on the http protocol and the CDC function of the ZNBase main cluster to complete data reception;

[0020] Decoded data: The received data is decoded in CDC-SINK;

[0021] Write data: write to a specified database in the standby cluster;

[0022] Parse data: retrieve data from the standby cluster database and parse it into specific SQL statements;

[0023] Process data: The data is processed by executing it in the corresponding standby cluster table through the driver inside the database.

[0024] Preferably, the CDC-SINK logic processing is as follows:

[0025] Monitor signals;

[0026] Get the command line parameters configured in CDC-SINK;

[0027] Parse the configured command line parameters;

[0028] Connect to the target database according to the configured command line parameters;

[0029] Create the corresponding SINKDB database;

[0030] Create SINK;

[0031] Create a processor;

[0032] Create CDC-SINK service;

[0033] Start the CDC-SINK service.

[0034] Preferably, the internal processing flow of CDC-SINK is as follows:

[0035] (1) Get the URL of the connection, and then proceed to step (2);

[0036] (ii) JSON regular expression matches the URL, and then proceed to step (iii);

[0037] (III) Determine whether the JSON regular expression successfully matches the URL:

[0038] ①. If the match fails, execute step (six);

[0039] ②. If the match is successful, execute step (IV);

[0040] (IV) Find an available sink and execute step (V) next;

[0041] (V) Determine whether the acquisition is successful:

[0042] ①. If yes, complete the processing request;

[0043] ②If not, proceed to step (VIII);

[0044] (VI) The resolved regular expression matches the URL, and the next step is to execute step (VII);

[0045] (VII) Determine whether the resolved regular expression successfully matches the URL:

[0046] ①. If the match fails, execute step (VIII);

[0047] ② If the match is successful, the resolution request is processed;

[0048] (8) Return error.

[0049] Preferably, the specific steps for processing the request are as follows:

[0050] (1) Create a Scanner to continuously obtain row information transmitted from the ZNBase main cluster;

[0051] (2) Determine whether the acquisition is successful:

[0052] ①. If yes, proceed to step (3);

[0053] ②, If not, return an error;

[0054] (3) Process the next line of information and parse it to obtain relevant information;

[0055] (4) Determine whether the analysis is successful:

[0056] ①. If yes, proceed to step (5);

[0057] ②, If not, return an error;

[0058] (5) After parsing, add the record list;

[0059] (6) Determine whether the number of row records is greater than or equal to 1000 rows:

[0060] ①. If yes, proceed to step (7);

[0061] ②If not, jump to step (9);

[0062] (7) Records are written into the temporary table of the standby cluster;

[0063] (8) Clear the record line;

[0064] (9) Continue to process the next line of information until an error occurs or all the line information is processed. Preferably, the specific steps for processing the parsing request are as follows:

[0065] (I) Create a Scanner to continuously obtain row information transmitted from the main database;

[0066] (II) Determine whether the acquisition is successful:

[0067] ①. If yes, proceed to step (III);

[0068] ②, If not, return an error;

[0069] (III) Process the next line and parse it to obtain relevant information;

[0070] (IV) Determine whether the analysis is successful:

[0071] ①. If yes, proceed to step (V);

[0072] ②, If not, return an error;

[0073] (V) Query the resolved record of the last record;

[0074] (VI) Get all sinks currently configured;

[0075] (VII) Execute the update row operation;

[0076] (VIII) Determine whether there are other sinks:

[0077] ①. If yes, jump to step (VII);

[0078] ②, if not, proceed to step (IX);

[0079] (IX) Update the resolved table;

[0080] (Ⅹ) After all sinks have been processed, continue to process the next line.

[0081] Preferably, the update row operation is as follows:

[0082] (i) Get available sinks;

[0083] (ii) Create, update and delete rows;

[0084] (iii) Record the row information that needs to be operated;

[0085] (iv) Get the last row of information recorded;

[0086] (v), decoding line information;

[0087] (vi) After parsing, determine whether the specific behavior of the row is update or delete:

[0088] ①. If it is a deletion operation, execute step (ⅶ);

[0089] ② If you want to perform another operation, proceed to step (ⅶ);

[0090] (vii) Execute corresponding operations in the target database of the standby cluster;

[0091] (ⅷ) Delete the information recorded in resolved.

[0092] A CDC-based inter-cluster data synchronization system, the system is as follows:

[0093] The data synchronization unit is used for data synchronization between the ZNBase main cluster and the standby cluster. That is, after the data flowing out of the ZNBase main cluster is parsed and processed by CDC-SINK, the temporary data is written into a specified database in the standby cluster; then the corresponding data of the standby cluster is written into the table that the standby cluster needs to complete data synchronization with the main cluster; finally, the temporary data in the specified database of the standby cluster is deleted;

[0094] The consistency unit is used for consistency between transactions in the ZNBase main cluster and the standby cluster. That is, the data flowing out of the ZNBase main cluster is processed by CDC-SINK and sent to the standby cluster. The standby cluster determines whether the transaction corresponding to the data is complete based on the timestamp in the data flowing out of the ZNBase main cluster, and decides whether to apply or roll back based on the integrity of the corresponding transaction.

[0095] The CDC-based inter-cluster data synchronization method and system of the present invention have the following advantages:

[0096] (1) The present invention can achieve data synchronization between the primary and standby clusters, achieve high availability between databases, and reduce the losses caused by problems in the internal components of the database cluster;

[0097] (ii) The present invention uses the CDC function to complete the outflow of data in a cluster (the main cluster), which is processed by a tool called CDC-SINK and finally flows to another database cluster (the backup cluster), thereby completing the data synchronization between the main cluster and the backup cluster;

[0098] (III) The present invention is applied to the Inspur Yunxi database ZNBase, providing high availability between clusters and completing data synchronization between clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The present invention is further described below in conjunction with the accompanying drawings.

[0100] Attached Figure 1 It is a schematic diagram of the data synchronization scheme;

[0101] Attached Figure 2 This is a flowchart of CDC-SINK logic processing;

[0102] Attached Figure 3 This is a flowchart of the overall processing within CDC-SINK;

[0103] Attached Figure 4 A flowchart for processing requests;

[0104] Attached Figure 5 A flowchart for processing a parsing request;

[0105] Attached Figure 6 This is a flowchart for the update row operation. DETAILED DESCRIPTION

[0106] The CDC-based inter-cluster data synchronization method and system of the present invention are described in detail below with reference to the accompanying drawings and specific embodiments.

[0107] Embodiment 1:

[0108] The CDC-based inter-cluster data synchronization method of the present invention uses the CDC function in the ZNBase main cluster to flow the data in each table to a receiver, and the receiver uses CDC-SINK; after the CDC-SINK is processed, the data is played back on the standby cluster, and the data flow from the ZNBase main cluster to the standby cluster can be completed; the details are as follows:

[0109] S1. Data synchronization between the ZNBase master cluster and the backup cluster; as shown in the attached Figure 1 As shown, the details are as follows:

[0110] S101, after the data flowing out from the ZNBase main cluster is parsed and processed by CDC-SINK, the temporary data is written into a specified database in the standby cluster;

[0111] S102, writing the corresponding data of the standby cluster into a table that the standby cluster needs to complete data synchronization with the active cluster;

[0112] S103, deleting temporary data in the designated database of the standby cluster;

[0113] S2. Consistency between ZNBase primary cluster transactions and standby cluster transactions: Data flowing out of the ZNBase primary cluster is processed by CDC-SINK and sent to the standby cluster. The standby cluster determines whether the transaction corresponding to the data is complete based on the timestamp in the data flowing out of the ZNBase primary cluster, and decides whether to apply or roll back based on the integrity of the corresponding transaction.

[0114] The consistency between the ZNBase main cluster transaction and the standby cluster transaction in this embodiment is as follows:

[0115] Configure the related tables in the order of primary key table and foreign key table in the config of a CDC-SINK process, and the CDC-SINK process can receive data from all configured tables; and set the inflow order of CDC-SINK to foreign key table-primary key table; theoretically, if a transaction involves primary key table and foreign key table, the order of these tables flowing out from the main cluster is foreign key table-primary key table, but because these data belong to a transaction, their timestamps are the same, and the application order of data cannot be guaranteed to be foreign key table-primary key table in the standby cluster. If the order is reversed, the primary key table cannot be applied due to the constraints of the foreign key table, which will cause inconsistent data in the main and standby clusters. Considering this, there is a solution, using resolved time, and there will be no update submission before this timestamp. Put all tables configured by CDC-SINK config into a pool. When and only when all tables receive resolved time, send the updated data of all tables before the minimum resolved time, and send it in the order of the tables configured in config, to ensure that there is no new update before this time, to ensure the consistency of transactions. Due to the relationship between the primary and foreign key tables, the tables in the standby database need to be configured with cascading delete and cascading update. The reason is that the order of update and insert operations is foreign key table-primary key table, and the order of delete operations is primary key table-foreign key table. The two are exactly opposite. The inflow order of CDC-SINK is set to foreign key table-primary key table. Therefore, if cascading delete and cascading update are not configured, a transaction in the primary cluster deletes the primary key table first and then the foreign key table. In CDC-SINK, the data update order is foreign key table data deletion, primary key table data deletion. In this way, when the foreign key table in the standby cluster is deleted, it cannot be deleted due to the existence of the primary key table, resulting in data inconsistency.

[0116] The specific requirements for the active cluster and standby cluster in this embodiment are as follows:

[0117] ①. Tables in both the ZNBase master and slave clusters must have primary keys and single column families;

[0118] ② All foreign key tables in the standby cluster need to be configured with cascading delete and cascading update;

[0119] ③. The fields of the tables in the ZNBase master and slave clusters cannot have collate fields;

[0120] ④. When the ZNBase master cluster and the standby cluster are synchronized, full synchronization is performed each time.

[0121] The CDC-SINK in this embodiment is a tool for receiving data flowing out from the ZNBase master cluster to the standby cluster and writing it to the standby cluster; the details are as follows:

[0122] Receiving data: CDC-SINK is based on the http protocol and the CDC function of the ZNBase main cluster to complete data reception;

[0123] Decoded data: The received data is decoded in CDC-SINK;

[0124] Write data: write to a specified database in the standby cluster;

[0125] Parse data: retrieve data from the standby cluster database and parse it into specific SQL statements;

[0126] Process data: The data is processed by executing it in the corresponding standby cluster table through the driver inside the database.

[0127] As attached Figure 2 As shown, the CDC-SINK logic processing in this embodiment is specifically as follows:

[0128] ①、Monitoring signal;

[0129] ②、Get the command line parameters configured in CDC-SINK;

[0130] ③、Parse the configured command line parameters;

[0131] ④. Connect to the target database according to the configured command line parameters;

[0132] ⑤. Create the corresponding SINKDB database;

[0133] ⑥. Create SINK;

[0134] ⑦. Create a processor;

[0135] ⑧. Create CDC-SINK service;

[0136] ⑨. Start the CDC-SINK service.

[0137] As attached Figure 3 As shown, the internal processing flow of CDC-SINK in this embodiment is as follows:

[0138] (1) Get the URL of the connection, and then proceed to step (2);

[0139] (ii) JSON regular expression matches the URL, and then proceed to step (iii);

[0140] (III) Determine whether the JSON regular expression successfully matches the URL:

[0141] ①. If the match fails, execute step (six);

[0142] ②. If the match is successful, execute step (IV);

[0143] (IV) Find an available sink and execute step (V) next;

[0144] (V) Determine whether the acquisition is successful:

[0145] ①. If yes, complete the processing request;

[0146] ②If not, proceed to step (VIII);

[0147] (VI) The resolved regular expression matches the URL, and the next step is to execute step (VII);

[0148] (VII) Determine whether the resolved regular expression successfully matches the URL:

[0149] ①. If the match fails, execute step (VIII);

[0150] ② If the match is successful, the resolution request is processed;

[0151] (8) Return error.

[0152] As attached Figure 4 As shown, the specific steps of processing the request in this embodiment are as follows:

[0153] (1) Create a Scanner to continuously obtain row information transmitted from the ZNBase main cluster;

[0154] (2) Determine whether the acquisition is successful:

[0155] ①. If yes, proceed to step (3);

[0156] ②, If not, return an error;

[0157] (3) Process the next line of information and parse it to obtain relevant information;

[0158] (4) Determine whether the analysis is successful:

[0159] ①. If yes, proceed to step (5);

[0160] ②, If not, return an error;

[0161] (5) After parsing, add the record list;

[0162] (6) Determine whether the number of row records is greater than or equal to 1000 rows:

[0163] ①. If yes, proceed to step (7);

[0164] ②If not, jump to step (9);

[0165] (7) Records are written into the temporary table of the standby cluster;

[0166] (8) Clear the record line;

[0167] (9) Continue to process the next line of information until an error occurs or all lines of information are processed. Figure 5 As shown, the specific steps of processing the parsing request in this embodiment are as follows:

[0168] (I) Create a Scanner to continuously obtain row information transmitted from the main database;

[0169] (II) Determine whether the acquisition is successful:

[0170] ①. If yes, proceed to step (III);

[0171] ②, If not, return an error;

[0172] (III) Process the next line and parse it to obtain relevant information;

[0173] (IV) Determine whether the analysis is successful:

[0174] ①. If yes, proceed to step (V);

[0175] ②, If not, return an error;

[0176] (V) Query the resolved record of the last record;

[0177] (VI) Get all sinks currently configured;

[0178] (VII) Execute the update row operation;

[0179] (VIII) Determine whether there are other sinks:

[0180] ①. If yes, jump to step (VII);

[0181] ②, if not, proceed to step (IX);

[0182] (IX) Update the resolved table;

[0183] (Ⅹ) After all sinks have been processed, continue to process the next line.

[0184] As attached Figure 6 As shown, the update row operation in this embodiment is specifically as follows:

[0185] (i) Get available sinks;

[0186] (ii) Create, update and delete rows;

[0187] (iii) Record the row information that needs to be operated;

[0188] (iv) Get the last row of information recorded;

[0189] (v), decoding line information;

[0190] (vi) After parsing, determine whether the specific behavior of the row is update or delete:

[0191] ①. If it is a deletion operation, execute step (ⅶ);

[0192] ② If you want to perform another operation, proceed to step (ⅶ);

[0193] (vii) Execute corresponding operations in the target database of the standby cluster;

[0194] (ⅷ) Delete the information recorded in resolved.

[0195] Embodiment 2:

[0196] The CDC-based inter-cluster data synchronization system of the present invention is specifically as follows:

[0197] The data synchronization unit is used for data synchronization between the ZNBase main cluster and the standby cluster. That is, after the data flowing out of the ZNBase main cluster is parsed and processed by CDC-SINK, the temporary data is written into a specified database in the standby cluster; then the corresponding data of the standby cluster is written into the table that the standby cluster needs to complete data synchronization with the main cluster; finally, the temporary data in the specified database of the standby cluster is deleted;

[0198] The consistency unit is used for consistency between transactions in the ZNBase main cluster and the standby cluster. That is, the data flowing out of the ZNBase main cluster is processed by CDC-SINK and sent to the standby cluster. The standby cluster determines whether the transaction corresponding to the data is complete based on the timestamp in the data flowing out of the ZNBase main cluster, and decides whether to apply or roll back based on the integrity of the corresponding transaction.

[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A CDC-based inter-cluster data synchronization method, characterized in that: This method uses the CDC function in the ZNBase master cluster to flow the data in each table to a receiver, which uses CDC-SINK. After processing, CDC-SINK plays back the data on the standby cluster to complete the data flow from the ZNBase master cluster to the standby cluster. The details are as follows: Data synchronization between the ZNBase master cluster and the standby cluster: After the data flowing out of the ZNBase master cluster is parsed and processed by CDC-SINK, the temporary data is written to a specified database in the standby cluster; then the corresponding data of the standby cluster is written to the table that the standby cluster needs to complete data synchronization with the master cluster; finally, the temporary data in the specified database of the standby cluster is deleted; Consistency between ZNBase master cluster transactions and standby cluster transactions: Data flowing out of the ZNBase master cluster is processed by CDC-SINK and sent to the standby cluster. The standby cluster determines whether the transaction corresponding to the data is complete based on the timestamp in the data flowing out of the ZNBase master cluster, and decides whether to apply or roll back based on the integrity of the corresponding transaction. The consistency between ZNBase master cluster transactions and standby cluster transactions is as follows: Configure the related tables in the order of primary key table and foreign key table to the config of a CDC-SINK process, and the CDC-SINK process can receive data from all configured tables; and set the inflow order of CDC-SINK to foreign key table-primary key table; Put all tables configured by CDC-SINK's config into a pool. When and only when all tables receive resolved time, send the updated data of all tables before the minimum resolved time, and send them in the order of the tables configured by config.

2. The CDC-based inter-cluster data synchronization method according to claim 1, characterized in that: The specific requirements for the active cluster and standby cluster are as follows: ①. Tables in both the ZNBase master and slave clusters must have primary keys and single column families; ② All foreign key tables in the standby cluster need to be configured with cascading delete and cascading update; ③. The fields of the tables in the ZNBase master and slave clusters cannot have collate fields; ④. When the ZNBase master cluster and the standby cluster are synchronized, full synchronization is performed each time.

3. The CDC-based inter-cluster data synchronization method according to claim 1, characterized in that: The CDC-SINK is a tool that receives data flowing from the ZNBase master cluster to the standby cluster and writes it to the standby cluster; the details are as follows: Receiving data: CDC-SINK is based on the http protocol and the CDC function of the ZNBase main cluster to complete data reception; Decoded data: The received data is decoded in CDC-SINK; Write data: write to a specified database in the standby cluster; Parse data: retrieve data from the standby cluster database and parse it into specific SQL statements; Processing data: The data is processed by executing it in the corresponding standby cluster table through the driver inside the database.

4. The CDC-based inter-cluster data synchronization method according to claim 1, characterized in that: The specific CDC-SINK logic processing is as follows: Monitor signals; Get the command line parameters configured in CDC-SINK; Parse the configured command line parameters; Connect to the target database according to the configured command line parameters; Create the corresponding SINKDB database; Create SINK; Create a processor; Create CDC-SINK service; Start the CDC-SINK service.

5. The CDC-based inter-cluster data synchronization method according to claim 1, characterized in that: The internal processing flow of CDC-SINK is as follows: (I) Get the URL of the connection, and then proceed to step (II); (ii) JSON regular expression matches the URL, and then proceed to step (iii); (III) Determine whether the JSON regular expression successfully matches the URL: ①. If the match fails, execute step (six); ② If the match is successful, execute step (IV); (IV) Find an available sink and execute step (V) next; (V) Determine whether the acquisition is successful: ①. If yes, complete the processing request; ②If not, proceed to step (VIII); (VI) The resolved regular expression matches the URL, and the next step is to execute step (VII); (VII) Determine whether the resolved regular expression successfully matches the URL: ①. If the match fails, execute step (8); ②. If the match is successful, process the resolution request; 8. Return error.

6. The CDC-based inter-cluster data synchronization method according to claim 5, characterized in that: The specific steps for processing a request are as follows: (1) Create a Scanner to continuously obtain row information transmitted from the ZNBase main cluster; (2) Determine whether the acquisition is successful: ①. If yes, proceed to step (3); ②, If not, return an error; (3) Process the next line of information and parse it to obtain relevant information; (4) Determine whether the analysis is successful: ①. If yes, proceed to step (5); ②, If not, return an error; (5) After parsing, add the record list; (6) Determine whether the number of row records is greater than or equal to 1000 rows: ①. If yes, proceed to step (7); ②If not, jump to step (9); (7) Records are written into the temporary table of the standby cluster; (8) Clear the record line; (9) Continue processing the next line of information until an error occurs or all lines of information are processed.

7. The CDC-based inter-cluster data synchronization method according to claim 5, characterized in that: The specific steps for processing a parsing request are as follows: (I) Create a Scanner to continuously obtain row information transmitted from the main database; (II) Determine whether the acquisition is successful: ①. If yes, proceed to step (III); ②, If not, return an error; (III) Process the next line and parse it to obtain relevant information; (IV) Determine whether the analysis is successful: ①. If yes, proceed to step (V); ②, If not, return an error; (V) Query the resolved record of the last record; (VI) Get all sinks currently configured; (VII) Execute the update row operation; (VIII) Determine whether there are other sinks: ①. If yes, jump to step (VII); ②, if not, proceed to step (IX); (IX) Update the resolved table; (Ⅹ) After all sinks have been processed, continue to process the next line.

8. The CDC-based inter-cluster data synchronization method according to claim 7, characterized in that: The update row operations are as follows: (i) Get available sinks; (ii) Create, update and delete rows; (iii) Record the row information that needs to be operated; (iv) Get the last row of information in the record; (v), decoding line information; (vi) After parsing, determine whether the specific behavior of the row is update or delete: ①. If it is a delete operation, execute step (ⅶ); ②. If you want to perform another operation, proceed to step (ⅶ); (vii) Execute corresponding operations in the target database of the standby cluster; (ⅷ) Delete the information recorded in resolved.

9. A CDC-based inter-cluster data synchronization system, characterized in that: The system is as follows: The data synchronization unit is used for data synchronization between the ZNBase main cluster and the standby cluster. That is, after the data flowing out of the ZNBase main cluster is parsed and processed by CDC-SINK, the temporary data is written into a specified database in the standby cluster; then the corresponding data of the standby cluster is written into the table that the standby cluster needs to complete data synchronization with the main cluster; finally, the temporary data in the specified database of the standby cluster is deleted; The consistency unit is used for consistency between transactions in the ZNBase master cluster and the standby cluster. That is, the data flowing out of the ZNBase master cluster is processed by CDC-SINK and then sent to the standby cluster. The standby cluster determines whether the transaction corresponding to the data is complete based on the timestamp in the data flowing out of the ZNBase master cluster, and decides whether to apply or roll back based on the integrity of the corresponding transaction. The consistency between ZNBase master cluster transactions and standby cluster transactions is as follows: Configure the related tables in the order of primary key table and foreign key table to the config of a CDC-SINK process, and the CDC-SINK process can receive data from all configured tables; and set the inflow order of CDC-SINK to foreign key table-primary key table; Put all tables configured by CDC-SINK's config into a pool. When and only when all tables receive resolved time, send the updated data of all tables before the minimum resolved time, and send them in the order of the tables configured by config.

Images