Data bidirectional replication component and method based on Kafka Connect

Abstract

The invention discloses a data bidirectional replication component and method based on Kafka Connect, belongs to the technical field of data transmission, and aims to solve the technical problem of how to realize efficient bidirectional data replication between Kafka and KaiwuDB. The KaiwDB Source Connector plug-in is used for carrying out processing and conversion on the read data, and pushing the processed and converted data to a specified Topic of the kafka, and the KaiwDB Source Connector plug-in is used for reading the read data and pushing the processed and converted data to the specified Topic of the kafka; and the KaiwuDB Sink Connector plug-in is used for monitoring data of a specified Topic in kafka, analyzing the data according to a data protocol format requirement specified by a user in a configuration file, processing grammar supported by the KaiwuDB, and writing the data into the KaiwuDB in batch in real time by executing a write-in command.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to a bidirectional data replication component and method based on Kafka Connect. Background Art

[0002] In modern big data applications, data synchronization between databases and message queues is a common requirement. Traditional solutions usually require complex ETL (Extract, Transform, Load) processes, which not only have high development costs but also make it difficult to meet the needs of real-time data synchronization. Apache Kafka, as a high-throughput distributed message queue system, is widely used in real-time data processing and streaming data transmission scenarios. In order to realize data interaction between Kafka and external systems (such as databases, cloud services, etc.), Kafka Connect, as a core component of Kafka, came into being. The existing KafkaConnect only supports integration with some mainstream databases and cannot support data interaction with KaiwuDB. KaiwuDB, as a specific type of database, can be used to store time series data or relational business data.

[0003] How to achieve efficient two-way data replication between Kafka and KaiwuDB is a technical problem that needs to be solved. Summary of the Invention

[0004] The technical task of the present invention is to address the above shortcomings and provide a data bidirectional replication component and method based on Kafka Connect to solve the technical problem of how to achieve efficient bidirectional data replication between Kafka and KaiwuDB.

[0005] In a first aspect, the present invention provides a bidirectional data replication component based on Kafka Connect, which is a KaiwuDB Kafka Connector component including a KaiwuDB Source Connector plug-in and a KaiwuDB Sink Connector plug-in;

[0006] The KaiwuDB Source Connector plug-in is used to read data from the KaiwuDB database and process and convert the read data based on the data protocol format specified by the user in the configuration file. It converts the data into the target protocol format and pushes the processed and converted data to the specified Topic in Kafka.

[0007] The KaiwuDB Sink Connector plug-in is used to monitor data from a specified topic in Kafka. After monitoring the inflow of new data, it parses the data according to the data protocol format specified by the user in the configuration file, processes the syntax supported by KaiwuDB, and writes the data to KaiwuDB in batches and in real time by executing write commands.

[0008] Preferably, the KaiwuDB Kafka Connector component supports KaiwuDB Json protocol format, InfluxDB line protocol format, OpenTSDB Json protocol format and OpenTSDB Telnet line protocol format.

[0009] As a preferred option, the KaiwuDB Source Connector plug-in is used to push all data from a KaiwuDB database after a certain time to Kafka. The KaiwuDB Source Connector reads data from the KaiwuDB database by first pulling historical data in batches, then synchronizing incremental data through a scheduled query strategy, and monitoring changes in tables in the KaiwuDB database, automatically synchronizing newly added tables, and supporting breakpoint resuming. After restarting, synchronization can be resumed from the last interrupted position. Error information recorded during the synchronization process includes data source connection errors, data reading errors, and data writing errors.

[0010] Correspondingly, when the KaiwuDB Source Connector plug-in is working, the workflow is as follows:

[0011] The user configures the configuration file of the KaiwuDB Source Connector plug-in and adds the KaiwuDB Source Connector plug-in to the Kafka service by executing Kafka-related commands;

[0012] After the KaiwuDB Source Connector plug-in is successfully loaded and run, it first pulls historical data from the KaiwuDB database in batches, and then loads the latest data from the KaiwuDB database by periodically executing query statements;

[0013] When executing a query, according to the configuration information in the configuration file and the specified data protocol format, the corresponding data in the KaiwuDB database is queried and the queried data is processed and converted, and the processed and converted data is pushed to Kafka;

[0014] During the data push process, based on the advanced batch processing mechanism, the processed and converted data are grouped according to a certain batch size, and a group of data is pushed to Kafka at one time.

[0015] Preferably, after the Sink Connector plug-in monitors the data of the specified Topic in Kafka, it parses the data according to the data protocol format specified by the user in the configuration file and writes the data into KaiwuDB. The writing process adopts an optimized algorithm to adapt to the characteristics of the KaiwuDB storage engine;

[0016] Correspondingly, the KaiwuDB Sink Connector plug-in performs the following operations when working:

[0017] The processing logic for writing data in different data protocol formats to KaiwuDB is different. For data in InfluxDB line protocol format, users need to create a database instance and data table in KaiwuDB in advance. For data in InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format, the decision on whether to automatically create a data table is based on the specific content of the data.

[0018] The internal implementation logic of KaiwuDB Sink Connector is to process the received data according to the field name, data type and data value, generate SQL and send it to the KaiwuDB database for data writing. When writing, the plug-in will fully consider the storage engine characteristics of KaiwuDB and use an optimized method to write data.

[0019] In a second aspect, the present invention provides a method for bidirectional data replication based on Kafka Connect, which implements bidirectional data replication through a bidirectional data replication component based on Kafka Connect as described in any one of the first aspects, including the following steps:

[0020] The KaiwuDB Source Connector plug-in reads data from the KaiwuDB database and processes and converts the read data based on the data protocol format specified by the user in the configuration file, converting the data to conform to the target protocol format requirements, and pushes the processed and converted data to the specified Topic in Kafka;

[0021] The KaiwuDB Sink Connector plug-in monitors the data of the specified Topic in Kafka. After monitoring the inflow of new data, it parses the data according to the data protocol format requirements specified by the user in the configuration file, processes the syntax supported by KaiwuDB, and executes the write command to write the data to KaiwuDB in batches and in real time.

[0022] Preferably, the KaiwuDB Kafka Connector component supports KaiwuDB Json protocol format, InfluxDB line protocol format, OpenTSDB Json protocol format and OpenTSDB Telnet line protocol format.

[0023] As a preferred option, all data from a certain KaiwuDB database after a certain time is pushed to Kafka through the KaiwuDB Source Connector plug-in. The way KaiwuDB Source Connector reads data from the KaiwuDB database is to first pull historical data in batches, then synchronize incremental data through a scheduled query strategy, and monitor changes in tables in the KaiwuDB database, automatically synchronize newly added tables, and support breakpoint resume function. After restart, synchronization can be continued from the last interruption position. Error information recorded during the synchronization process includes data source connection errors, data reading errors, and data writing errors;

[0024] Correspondingly, when the KaiwuDB Source Connector plug-in is working, the workflow is as follows:

[0025] The user configures the configuration file of the KaiwuDB Source Connector plug-in and adds the KaiwuDB Source Connector plug-in to the Kafka service by executing Kafka-related commands;

[0026] After the KaiwuDB Source Connector plug-in is successfully loaded and run, it first pulls historical data from the KaiwuDB database in batches, and then loads the latest data from the KaiwuDB database by periodically executing query statements;

[0027] When executing a query, according to the configuration information in the configuration file and the specified data protocol format, the corresponding data in the KaiwuDB database is queried and the queried data is processed and converted, and the processed and converted data is pushed to Kafka;

[0028] During the data push process, based on the advanced batch processing mechanism, the processed and converted data are grouped according to a certain batch size, and a group of data is pushed to Kafka at one time.

[0029] As a preferred approach, after listening to the data of the specified Topic in Kafka through the Sink Connector plug-in, it parses the data according to the data protocol format specified by the user in the configuration file and writes the data to KaiwuDB. The writing process uses an optimized algorithm to adapt to the characteristics of the KaiwuDB storage engine;

[0030] Correspondingly, the KaiwuDB Sink Connector plug-in performs the following operations when working:

[0031] The processing logic for writing data in different data protocol formats to KaiwuDB is different. For data in InfluxDB line protocol format, users need to create a database instance and data table in KaiwuDB in advance. For data in InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format, the decision on whether to automatically create a data table is based on the specific content of the data.

[0032] The internal implementation logic of KaiwuDB Sink Connector is to process the received data according to the field name, data type and data value, generate SQL and send it to the KaiwuDB database for data writing. When writing, the plug-in will fully consider the storage engine characteristics of KaiwuDB and use an optimized method to write data.

[0033] The Kafka Connect-based bidirectional data replication method of the present invention has the following advantages:

[0034] 1. Simplified configuration process: Users only need to provide a simple configuration file to achieve complex two-way data synchronization between Kafka and KaiwuDB;

[0035] 2. Improve data transmission efficiency: By adopting an efficient batch processing mechanism, data transmission can be completed quickly, improving overall data processing efficiency;

[0036] 3. Enhanced data interaction flexibility: Support for four data protocol formats can meet the diverse data interaction needs of enterprises;

[0037] 4. Easy to maintain: The architecture design based on Kafka Connect makes the maintenance of the entire system easier. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] The present invention will be further described below with reference to the accompanying drawings.

[0040] Figure 1 This is an architectural diagram of a Kafka Connect-based bidirectional data replication component in Example 1;

[0041] Figure 2 This is a data flow diagram of the KaiwuDB SourceConnector plug-in in a Kafka Connect-based bidirectional data replication component in Example 1;

[0042] Figure 3 This is a data flow diagram of the KaiwuDB SinkConnector plug-in in the Kafka Connect-based data bidirectional replication component in Example 1. DETAILED DESCRIPTION

[0043] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments given are not intended to limit the present invention. Unless there is a conflict, the embodiments of the present invention and the technical features in the embodiments may be combined with each other.

[0044] The embodiments of the present invention provide a data bidirectional replication component and method based on Kafka Connect, which is used to solve the technical problem of how to achieve efficient bidirectional data replication between Kafka and KaiwuDB.

[0045] Example 1:

[0046] The present invention provides a data bidirectional replication component based on Kafka Connect, which is a KaiwuDB Kafka Connector component including a KaiwuDB SourceConnector plug-in and a KaiwuDB Sink Connector plug-in.

[0047] In this embodiment, KaiwuDB Kafka Connector supports four data protocol formats, including: KaiwuDBJson protocol format, InfluxDB line protocol format, OpenTSDB Json protocol format and OpenTSDB Telnet line protocol format. This multi-protocol support feature enables the component to seamlessly connect with a variety of different types of data sources and target libraries, meeting diverse data interaction application scenarios. Taking the system that stores data in the InfluxDB line protocol format as an example, when interacting with the component of the present invention, the user only needs to select the corresponding InfluxDB protocol format in the configuration file, and KaiwuDB Kafka Connector can automatically complete the data format adaptation work without the user having to perform any additional data format conversion operations. It not only saves a lot of time and labor costs, but also avoids data loss or error problems that may be caused by format conversion, ensuring the accuracy and efficiency of data interaction.

[0048] The KaiwuDB Source Connector plug-in is used to read data from the KaiwuDB database, process and convert the read data based on the data protocol format specified by the user in the configuration file, convert the data to conform to the target protocol format requirements, and push the processed and converted data to the specified Topic in Kafka.

[0049] In this embodiment, the KaiwuDB Source Connector plug-in is used to push all data of a KaiwuDB database after a certain moment to Kafka. The way KaiwuDB Source Connector reads data from the KaiwuDB database is: first pull historical data in batches, then synchronize incremental data through a scheduled query strategy, and at the same time monitor changes in tables in the KaiwuDB database, automatically synchronize newly added tables, and support breakpoint resume function. After restart, synchronization can continue from the last interruption position. The error information recorded during the synchronization process includes data source connection errors, data reading errors, and data writing errors.

[0050] Correspondingly, when the KaiwuDB Source Connector plug-in is working, the workflow is as follows:

[0051] (1) The user configures the configuration file of the KaiwuDB Source Connector plug-in and adds the KaiwuDB Source Connector plug-in to the Kafka service by executing Kafka-related commands;

[0052] (2) After the KaiwuDB Source Connector plug-in is successfully loaded and run, it first pulls historical data from the KaiwuDB database in batches, and then loads the latest data from the KaiwuDB database by periodically executing query statements;

[0053] (3) When executing a query, according to the configuration information in the configuration file and the specified data protocol format, the corresponding data in the KaiwuDB database is queried and the queried data is processed and converted, and the processed and converted data is pushed to Kafka;

[0054] (4) During the data push process, based on the advanced batch processing mechanism, the processed and converted data are grouped according to a certain batch size, and a group of data is pushed to Kafka at one time.

[0055] The KaiwuDB Sink Connector plug-in is used to monitor data from a specified topic in Kafka. After monitoring the inflow of new data, it parses the data according to the data protocol format specified by the user in the configuration file, processes the syntax supported by KaiwuDB, and writes the data to KaiwuDB in batches and in real time by executing write commands.

[0056] In this embodiment, after the KaiwuDB Sink Connector plug-in listens to the data of the specified Topic in Kafka, it parses the data according to the data protocol format specified by the user in the configuration file and writes the data to KaiwuDB. The writing process uses an optimized algorithm to adapt to the characteristics of the KaiwuDB storage engine;

[0057] Correspondingly, the KaiwuDB Sink Connector plug-in performs the following operations when working:

[0058] (1) The processing logic for data in different data protocol formats when written to KaiwuDB is different. For data in InfluxDB line protocol format, users need to create a database instance and data table in KaiwuDB in advance. For data in InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format, the decision on whether to automatically create a data table will be made based on the specific content of the data.

[0059] (2) The internal implementation logic of KaiwuDB Sink Connector is to process the received data according to the field name, data type and data value, generate SQL and send it to the KaiwuDB database for data writing. When writing, the plug-in will fully consider the storage engine characteristics of KaiwuDB and use an optimized method to write data.

[0060] In this embodiment, Kafka Connector supports loading configuration files in both Json and Properties formats. Users only need to provide a simple configuration file to complete the data synchronization operation, which can flexibly respond to different data synchronization requirements.

[0061] (1) General configuration

[0062] The following configuration items apply to both KaiwuDB Source Connector and KaiwuDB Sink Connector.

[0063] name: the name of the connector;

[0064] connector.class: the full class name of the connector, for example: com.kaiwudb.kafka.connect.sink.KwdbSinkConnector;

[0065] tasks.max: maximum number of tasks;

[0066] connection.url: KaiwuDB JDBC connection string;

[0067] connection.user: KaiwuDB user name;

[0068] connection.password: KaiwuDB user password;

[0069] connection.attempts: The maximum number of attempts to retrieve a valid JDBC connection;

[0070] connection.backoff.ms: The retry interval for connection failure, in ms;

[0071] key.converter: specifies the converter used to convert Kafka message keys;

[0072] value.converter: specifies the converter used to convert Kafka message values; currently supports String and JSON formats;

[0073] (2) KaiwuDB Source Connector-specific configuration

[0074] connection.database: source database name, no default value;

[0075] poll.interval.ms: The interval for checking whether there are new or deleted tables, in milliseconds;

[0076] topic.prefix: The prefix of the Topic name used when importing data into Kafka;

[0077] timestamp.initial: The initial timestamp used for query. If not specified, all data in the table will be retrieved. The format is yyyy-MM-dd HH:mm:ss;

[0078] topic.delimiter: Topic name delimiter;

[0079] fetch.max.rows: the maximum number of rows to retrieve when searching the database;

[0080] query.interval.ms: The time span for reading data from KaiwuDB. This value should be configured appropriately based on the data characteristics in the table to avoid querying too much or too little data at a time. It is recommended to set an optimal value through testing in specific environments.

[0081] topic.per.stable: If set to true, one hypertable corresponds to one Kafka topic; if set to false, all data in the specified DB goes into one Kafka topic;

[0082] topic.ignore.db: Whether the Topic naming rule includes the database name;

[0083] out.format: result set output format, supports: kaiwudb_json, influxdb_line, opentsdb_json and opentsdb_line;

[0084] read.method: The method of reading data from KaiwuDB, query or subscription.

[0085] (3) KaiwuDB Sink Connector-specific configuration

[0086] opics: Topic list that needs to be synchronized. Multiple topics are separated by ",";

[0087] connection.database: The name of the KaiwuDB database to which the connector will write from the Kafka topic.

[0088] batch.size: Specifies how many records to try to batch together for insertion into the target table;

[0089] max.retries: The maximum number of times to retry an error before failing the task;

[0090] retry.backoff.ms: The time to wait (in milliseconds) before retrying after an error occurs;

[0091] protocol.type: supports the following data formats for writing to KaiwuDB: json_kaiwudb, json_opentsdb, line_opentsdb, and line_influxdb.

[0092] timestamp.precision: The time precision of the KaiwuDB data supported for writing is: seconds, milliseconds, us, and nanoseconds. The KaiwuDB JSON format only supports milliseconds, the InfluxDB Line format only supports milliseconds, us, and nanoseconds, and the OpenTSDB Line and JSON formats only support seconds and milliseconds.

[0093] The components of this embodiment provide an efficient and reliable bidirectional data synchronization mechanism that simplifies the integration between Kafka and KaiwuDB. By supporting multiple data protocol parsing and dynamic data format conversion, and providing a simple configuration file, the flexibility and adaptability of data synchronization are improved. The Kafka Connect architecture is distributed, highly available, and scalable, capable of supporting large-scale data synchronization needs.

[0094] The components of this embodiment are implemented based on the Apache Kafka framework. Kafka Connect is a component of Apache Kafka that enables other systems, such as databases, cloud services, and file systems, to easily connect to Kafka data. Data can flow from other systems to Kafka and vice versa through Kafka Connect.

[0095] Example 2:

[0096] The present invention provides a method for bidirectional data replication based on Kafka Connect, which implements bidirectional data replication through the components disclosed in Example 1, including the following steps:

[0097] Step S100: Read data from the KaiwuDB database through the KaiwuDB Source Connector plug-in, process and convert the read data based on the data protocol format specified by the user in the configuration file, convert the data to meet the target protocol format requirements, and push the processed and converted data to the specified Topic of Kafka;

[0098] Step S200: Listen to the data of the specified Topic in Kafka through the KaiwuDB Sink Connector plug-in. After listening to the new data inflow, parse the data according to the data protocol format requirements specified by the user in the configuration file, process the syntax supported by KaiwuDB, and write the data to KaiwuDB in batches and in real time by executing the write command.

[0099] In this embodiment, in step S100, all data of a certain KaiwuDB database after a certain moment is pushed to Kafka through the KaiwuDB Source Connector plug-in. The way KaiwuDB Source Connector reads data from the KaiwuDB database is: first pull historical data in batches, then synchronize incremental data through a scheduled query strategy, and at the same time monitor changes in tables in the KaiwuDB database, automatically synchronize newly added tables, and support breakpoint resume function. After restart, synchronization can continue from the last interruption position. The error information recorded during the synchronization process includes data source connection errors, data reading errors, and data writing errors.

[0100] Correspondingly, when the KaiwuDB Source Connector plug-in is working, the workflow is as follows:

[0101] (1) The user configures the configuration file of the KaiwuDB Source Connector plug-in and adds the KaiwuDB Source Connector plug-in to the Kafka service by executing Kafka-related commands;

[0102] (2) After the KaiwuDB Source Connector plug-in is successfully loaded and run, it first pulls historical data from the KaiwuDB database in batches, and then loads the latest data from the KaiwuDB database by periodically executing query statements;

[0103] (3) When executing a query, according to the configuration information in the configuration file and the specified data protocol format, the corresponding data in the KaiwuDB database is queried and the queried data is processed and converted, and the processed and converted data is pushed to Kafka;

[0104] (4) During the data push process, based on the advanced batch processing mechanism, the processed and converted data are grouped according to a certain batch size, and a group of data is pushed to Kafka at one time.

[0105] In step S200 of this embodiment, after the KaiwuDB Sink Connector plug-in monitors the data of the specified Topic in Kafka, it parses the data according to the data protocol format specified by the user in the configuration file and writes the data to KaiwuDB. The writing process uses an optimized algorithm to adapt to the characteristics of the KaiwuDB storage engine;

[0106] Correspondingly, the KaiwuDB Sink Connector plug-in performs the following operations when working:

[0107] (1) The processing logic for data in different data protocol formats when written to KaiwuDB is different. For data in InfluxDB line protocol format, users need to create a database instance and data table in KaiwuDB in advance. For data in InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format, the decision on whether to automatically create a data table will be made based on the specific content of the data.

[0108] (2) The internal implementation logic of KaiwuDB Sink Connector is to process the received data according to the field name, data type and data value, generate SQL and send it to the KaiwuDB database for data writing. When writing, the plug-in will fully consider the storage engine characteristics of KaiwuDB and use an optimized method to write data.

[0109] Through the method disclosed in this embodiment, users only need to provide a simple configuration file to easily achieve real-time or batch data synchronization between KaiwuDB and Kafka, and support the selection of multiple data protocol formats. The technical solution of the present invention not only simplifies the integration process of Kafka and KaiwuDB, but also improves the flexibility, stability, and efficiency of data transmission, effectively reducing development and maintenance costs. It can meet the needs of various business scenarios and provide strong technical support for enterprise data management and business development.

[0110] The above is a detailed introduction to the Kafka Connect-based bidirectional data replication component and method provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the ideas of the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A bidirectional data replication component based on Kafka Connect, characterized in that: KaiwuDB Kafka Connector component, which includes KaiwuDB SourceConnector plug-in and KaiwuDB Sink Connector plug-in; The KaiwuDB Source Connector plug-in is used to read data from the KaiwuDB database and process and convert the read data based on the data protocol format specified by the user in the configuration file. It converts the data into the target protocol format and pushes the processed and converted data to the specified Topic in Kafka. The KaiwuDB Sink Connector plug-in is used to monitor data from a specified topic in Kafka. After monitoring the inflow of new data, it parses the data according to the data protocol format specified by the user in the configuration file, processes the syntax supported by KaiwuDB, and writes the data to KaiwuDB in batches and in real time by executing write commands.

2. The Kafka Connect-based bidirectional data replication component according to claim 1, characterized in that: The KaiwuDB Kafka Connector component supports the KaiwuDB Json protocol format, InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format.

3. The Kafka Connect-based bidirectional data replication component according to claim 1, characterized in that: The KaiwuDB Source Connector plug-in is used to push all data from a KaiwuDB database after a certain time to Kafka. The KaiwuDB Source Connector reads data from the KaiwuDB database by first pulling historical data in batches, then synchronizing incremental data through a scheduled query strategy. It also monitors changes to tables in the KaiwuDB database, automatically synchronizing newly added tables, and supports breakpoint-resume transmission. After a restart, synchronization can be resumed from the last interruption point. Error information recorded during the synchronization process includes data source connection errors, data reading errors, and data writing errors. Correspondingly, when the KaiwuDB Source Connector plug-in is working, the workflow is as follows: The user configures the configuration file of the KaiwuDB Source Connector plug-in and adds the KaiwuDB Source Connector plug-in to the Kafka service by executing Kafka-related commands; After the KaiwuDB Source Connector plug-in is successfully loaded and run, it first pulls historical data from the KaiwuDB database in batches, and then loads the latest data from the KaiwuDB database by periodically executing query statements; When executing a query, according to the configuration information in the configuration file and the specified data protocol format, the corresponding data in the KaiwuDB database is queried and the queried data is processed and converted, and the processed and converted data is pushed to Kafka; During the data push process, based on the advanced batch processing mechanism, the processed and converted data are grouped according to a certain batch size, and a group of data is pushed to Kafka at one time.

4. The Kafka Connect-based bidirectional data replication component according to claim 1, characterized in that: After the KaiwuDB Sink Connector plug-in listens to the data of the specified Topic in Kafka, it parses the data according to the data protocol format specified by the user in the configuration file and writes the data to KaiwuDB. The writing process adopts an optimized algorithm to adapt to the characteristics of the KaiwuDB storage engine; Correspondingly, the KaiwuDB Sink Connector plug-in performs the following operations when working: The processing logic for writing data in different data protocol formats to KaiwuDB is different. For data in InfluxDB line protocol format, users need to create a database instance and data table in KaiwuDB in advance. For data in InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format, the decision on whether to automatically create a data table is based on the specific content of the data. The internal implementation logic of KaiwuDB Sink Connector is to process the received data according to the field name, data type and data value, generate SQL and send it to the KaiwuDB database for data writing. When writing, the plug-in will fully consider the storage engine characteristics of KaiwuDB and use an optimized method to write data.

5. A method for bidirectional data replication based on Kafka Connect, characterized in that: Implementing bidirectional data replication by using a Kafka Connect-based bidirectional data replication component according to any one of claims 1 to 4 includes the following steps: The KaiwuDB Source Connector plug-in reads data from the KaiwuDB database and processes and converts the read data based on the data protocol format specified by the user in the configuration file, converting the data to conform to the target protocol format requirements, and pushes the processed and converted data to the specified Topic in Kafka; The KaiwuDB Sink Connector plug-in monitors the data of the specified Topic in Kafka. After monitoring the inflow of new data, it parses the data according to the data protocol format requirements specified by the user in the configuration file, processes the syntax supported by KaiwuDB, and executes the write command to write the data to KaiwuDB in batches and in real time.

6. The method for bidirectional data replication based on Kafka Connect according to claim 5, characterized in that: The KaiwuDB Kafka Connector component supports the KaiwuDB Json protocol format, InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format.

7. The method for bidirectional data replication based on Kafka Connect according to claim 5, characterized in that: The KaiwuDB Source Connector plug-in pushes all data from a KaiwuDB database after a certain point in time to Kafka. The KaiwuDB Source Connector reads data from the KaiwuDB database by first pulling historical data in batches, then synchronizing incremental data through a scheduled query strategy. It also monitors changes in tables in the KaiwuDB database, automatically synchronizing newly added tables, and supports breakpoint-resume transmission. After a restart, synchronization can be resumed from the last interruption point. Error information recorded during the synchronization process includes data source connection errors, data reading errors, and data writing errors. Correspondingly, when the KaiwuDB Source Connector plug-in is working, the workflow is as follows: The user configures the configuration file of the KaiwuDB Source Connector plug-in and adds the KaiwuDB Source Connector plug-in to the Kafka service by executing Kafka-related commands; After the KaiwuDB Source Connector plug-in is successfully loaded and run, it first pulls historical data from the KaiwuDB database in batches, and then loads the latest data from the KaiwuDB database by periodically executing query statements; When executing a query, according to the configuration information in the configuration file and the specified data protocol format, the corresponding data in the KaiwuDB database is queried and the queried data is processed and converted, and the processed and converted data is pushed to Kafka; During the data push process, based on the advanced batch processing mechanism, the processed and converted data are grouped according to a certain batch size, and a group of data is pushed to Kafka at one time.

8. The method for bidirectional data replication based on Kafka Connect according to claim 5, characterized in that: After the Sink Connector plug-in listens to the data of the specified Topic in Kafka, it parses the data according to the data protocol format specified by the user in the configuration file and writes the data to KaiwuDB. The writing process uses an optimized algorithm to adapt to the characteristics of the KaiwuDB storage engine. Correspondingly, the KaiwuDB Sink Connector plug-in performs the following operations when working: The processing logic for writing data in different data protocol formats to KaiwuDB is different. For data in InfluxDB line protocol format, users need to create a database instance and data table in KaiwuDB in advance. For data in InfluxDB line protocol format, OpenTSDB Json protocol format, and OpenTSDB Telnet line protocol format, the decision on whether to automatically create a data table is based on the specific content of the data. The internal implementation logic of KaiwuDB Sink Connector is to process the received data according to the field name, data type and data value, generate SQL and send it to the KaiwuDB database for data writing. When writing, the plug-in will fully consider the storage engine characteristics of KaiwuDB and use an optimized method to write data.

Images