Kingbase ES database data synchronization method

Abstract

The invention belongs to the technical field of data synchronization, and particularly discloses a Kingbase ES database data synchronization method which comprises the following steps: firstly, modifying configuration in a Kingbase ES database; then, a Kingbase ES drive program package is used for being connected with a Kingbase ES database through the configured IP address and port; then, whether a stored LSN synchronization site exists in the Debezium platform or not is checked; then, a reset stream of the Kingbase ES is created by using an interface provided by a Kingbase ES official drive program package and an LSN synchronization site, and a thread is started to obtain changed data in real time; then, the data is analyzed; and finally, the analyzed information is sent to a Kafka platform, and data synchronization can be completed. In order to solve the problem that a current Kingbase ES database does not support a Debezium data synchronization component, the WAL of the Kingbase ES is obtained in real time and provided for Debezium for subsequent kafka data synchronization by using a decoder bufs logic decoding plug-in built in the Kingbase ES, so that the problem that the Kingbase ES cannot perform data synchronization with other databases due to the fact that the Kingbase ES database does not support the Debezium data synchronization component is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and in particular to a method for synchronizing Kingbase ES database data. Background Art

[0002] A database is a system for organizing and storing data. It is a widely used computer software. Databases can store various types of data, including numbers, text, images, audio, and video. The main function of a database is to provide persistent storage and management of data. It allows you to easily access and manage data while also ensuring the security and integrity of the data. In a production environment, in order to meet the needs of disaster recovery and testing, it is often necessary to deploy database synchronization software to synchronize data from the source database to the target database in real time.

[0003] Database synchronization means that the interconnected nodes in the system can synchronize the changed data; when the data of one node changes, the other one or more nodes can obtain the changed event and apply the change to the target database to maintain the data consistency between the source database and the target database. This usually involves copying and updating data to ensure that all participating databases can reflect the latest data status. Database synchronization can be performed between databases of the same type or between databases of different types. There are many ways to synchronize, such as trigger-based incremental synchronization, database log-based synchronization, and heterogeneous database synchronization. You can choose the appropriate method according to different needs and scenarios.

[0004] Debezium is an open source distributed platform built on Kafka to capture database data changes. Debezium provides a connector library that supports multiple databases, such as MySQL, Oracle, SQL Server, MongoDB, PostgreSQL, etc., but does not support domestic databases such as Kingbase ES database. In data business development, developers cannot use Debezium to capture data in Kingbase ES database, nor can they use KingbaseES and other non-domestic databases for data synchronization. Therefore, in order to solve the above problems, it is necessary to design a method for Kingbase ES database data synchronization to solve the problem that Kingbase ES database does not support Debezium data synchronization components, resulting in the inability of Kingbase ES database to synchronize data with other databases. Summary of the invention

[0005] The purpose of the present invention is to provide a method for synchronizing Kingbase ES database data, so as to solve the problem that Kingbase ES database does not support Debezium data synchronization component, resulting in Kingbase ES being unable to synchronize data with other databases.

[0006] To achieve the above object, the basic solution provided by the present invention is: a method for synchronizing Kingbase ES database data, comprising the following steps: S1: Enable WAL: Modify the configuration in the Kubernetes database; S2: Connect to Kingbase ES database: Use the Kingbase ES official driver package to connect to the Kingbase ES database through the configured IP address and port; S3: Get synchronization site: Check whether there is a saved LSN synchronization site in the Debezium platform; S4: Start synchronization: Use the interface and LSN synchronization site provided by the Kingbase ES official driver package to create a Kingbase ES replication stream, and start a thread to obtain the changed data in real time; S5: Parse information: Use the decoderbufs logical decoding plug-in to obtain the information of the change record in the protobuf format in the WAL; S6: Send the parsed information to the Kafka platform: Encapsulate the data parsed by the decoderbufs logical decoding plug-in into the Debezium data format, map the information in the Debezium data format to Kafka's SourceRecord, and send it to the Kafka platform through the source interface.

[0007] The beneficial effects of the present invention are as follows: (1) the present invention uses the decoderbufs logical decoding output plug-in built into Kingbase ES to obtain the WAL in the Kingbase ES database in protobuf format in real time, parse it and send it to the Kafka platform, thereby solving the problem that the Kingbase ES database does not support the Debezium data synchronization component, resulting in the Kingbase ES database being unable to synchronize data with other databases; (2) the replication mechanism provided by the Kingbase ES database is used to synchronize data from the Kingbase ES database in real time, thereby ensuring the real-time nature of data synchronization; (3) the interface provided by the Debezium platform is used to save the synchronization site of the data. Even if the data synchronization is interrupted, it can be restored from the saved synchronization site to continue synchronization, so as to be used for Debezium breakpoint resumption.

[0008] Solution 2 is the preferred solution of the basic solution. In S1, change wal_level to logical and change the value of max_wal_senders to a value greater than 1. Enable the logical replication function of the PostgreSQL ES database to synchronize data between different databases.

[0009] Solution 3 is the preferred solution of the basic solution. In S1, create a dynamic link library in the configuration item of shared_preload_libraries and add the decoderbufs logic decoding plug-in. The decoderbufs logic decoding output plug-in can be loaded automatically without manual extension loading. The plug-in can be conveniently used for logical replication operations, reducing the complexity of plug-in management and the possibility of errors, providing convenience for users.

[0010] Solution 4 is the preferred basic solution. In S2, use the jdbc interface to execute the select * from sys_create_logical_replication_slot('debezium','decoderbufs') command to create a replication slot. Debezium's replication slot can prevent WAL from being overwritten and ensure data consistency.

[0011] Solution 5 is the preferred basic solution. In S3, if the LSN synchronization site exists, the saved LSN synchronization site is obtained. If the LSN synchronization site does not exist, the latest LSN synchronization site is obtained using the jdbc interface.

[0012] Solution 6: This is the preferred solution of the basic solution. In S4, the format of the changed data is protobuf.

[0013] Solution 7 is the preferred basic solution. In S5, the information of the change record includes operation type, schema name, table name, change time, old data information, new data information, and field list.

[0014] Solution 8 is the preferred basic solution. In S6, after data synchronization, the LSN synchronization site of the newly acquired data is updated and saved in Debezium. After recording the synchronization site, when an interruption occurs during the data synchronization process, the next time the data is resynchronized, it can be directly continued from the site of the last interruption without starting synchronization from the beginning. DETAILED DESCRIPTION

[0015] The present invention is further described in detail below through specific embodiments: Example A method for synchronizing Kingbase ES database data, comprising the following steps: S1: Enable WAL: Modify the configuration in the KB ES database, change wal_level to logical, change the value of max_wal_senders to a value greater than 1, create a dynamic link library in the shared_preload_libraries configuration item, and add the decoderbufs logical decoding plug-in; S2: Connect to Kingbase ES database: Use the Kingbase ES official driver package to connect to the Kingbase ES database through the IP address and port configured in S1; then use the jdbc interface to execute the select * from sys_create_logical_replication_slot('debezium','decoderbufs') command to create a replication slot named Debezium in the decoderbufs logical decoding plug-in; S3: Get synchronization site: Check whether there is a saved LSN synchronization site in Debezium; if the LSN synchronization site exists, get the saved LSN synchronization site; if the LSN synchronization site does not exist, use the jdbc interface to get the latest LSN synchronization site; S4: Start synchronization: Use the interface and LSN synchronization site provided by the Kingbase ES official driver package to create a Kingbase replication stream, and start a thread to obtain the changed data in real time; the obtained change data is in the format of protobuf; S5: Parsing information: Use the decoderbufs logical decoding plug-in to obtain the information of the change record in the protobuf format in the WAL; the information of the change record includes the operation type, schema name, table name, change time, old data information, new data information and field list; S6: Send the parsed information to the Kafka platform: Encapsulate the data parsed by the decoderbufs logical decoding plug-in into the Debezium data format, map the information in the Debezium data format to the SourceRecord of Kafka (an open source distributed platform for capturing database data changes), and send it to the Kafka platform through the source interface; then update the LSN synchronization site of the newly acquired data and save it in Debezium.

[0016] In summary, when Kingbase ES database synchronizes data with other databases through the above method, the decoderbufs logical decoding output plug-in built in Kingbase ES database is used to obtain the WAL in the protobuf format of Kingbase ES database in real time, and send it to the Kafka platform after parsing. This solves the problem that Kingbase ES database does not support Debezium data synchronization component, resulting in the inability of Kingbase ES database to synchronize data with other databases. At the same time, the interface provided by the Debezium platform is used to save the synchronization point of the data. Even if the data synchronization is interrupted, it can be restored from the saved synchronization point to continue synchronization, realizing breakpoint resumption.

[0017] The above is only an embodiment of the present invention, and the common knowledge such as the known specific structure and characteristics in the scheme is not described in detail here. It should be pointed out that for those skilled in the art, several deformations and improvements can be made without departing from the structure of the present invention, which should also be regarded as the protection scope of the present invention, and these will not affect the effect of the implementation of the present invention and the practicality of the patent. The scope of protection required by this application shall be based on the content of its claims, and the specific implementation methods and other records in the specification can be used to interpret the content of the claims.

Claims

1. A method for synchronizing Kingbase ES database data, characterized in that: The following steps are involved: S1: Enable WAL: Modify the configuration in the Kubernetes database; S2: Connect to Kingbase ES database: Use the Kingbase ES official driver package to connect to the Kingbase ES database through the configured IP address and port; S3: Get synchronization site: Check whether there is a saved LSN synchronization site in the Debezium platform; S4: Start synchronization: Use the interface and LSN synchronization site provided by the Kingbase ES official driver package to create a Kingbase ES replication stream, and start a thread to obtain the changed data in real time; S5: Parse information: Use the decoderbufs logical decoding plug-in to obtain the information of the change record in the protobuf format in the WAL; S6: Send the parsed information to the Kafka platform: Encapsulate the data parsed by the decoderbufs logical decoding plug-in into the Debezium data format, map the information in the Debezium data format to Kafka's SourceRecord, and send it to the Kafka platform through the source interface.

2. A method for synchronizing Kingbase ES database data according to claim 1, characterized in that: In S1, change wal_level to logical and change the value of max_wal_senders to a value greater than 1.

3. A Kingbase ES database data synchronization method according to claim 1, characterized in that: In S1, create a dynamic link library in the shared_preload_libraries configuration item and add the decoderbufs logic decoding plug-in.

4. A method for synchronizing Kingbase ES database data according to claim 1, characterized in that: In S2, use the JDBC interface to execute the select * from sys_create_logical_replication_slot('debezium','decoderbufs') command to create a replication slot.

5. A Kingbase ES database data synchronization method according to claim 1, characterized in that: In S3, if the LSN synchronization site exists, the saved LSN synchronization site is obtained; if the LSN synchronization site does not exist, the latest LSN synchronization site is obtained using the JDBC interface.

6. A Kingbase ES database data synchronization method according to claim 1, characterized in that: In S4, the format of change data is protobuf.

7. A Kingbase ES database data synchronization method according to claim 1, characterized in that: In S5, the information of the change record includes operation type, schema name, table name, change time, old data information, new data information, and field list.

8. A Kingbase ES database data synchronization method according to claim 1, characterized in that: In S6, after data synchronization, the LSN synchronization site of the newly acquired data is updated and saved in Debezium.