A data synchronization method for MongoDB and ClickHouse

Abstract

The present invention discloses a data synchronization method for MongoDB and ClickHouse, which relates to the technical field of data synchronization. Design and write table-level annotations and field-level annotations required for synchronization, mark the annotations on the MongoDB data table entities and attributes to be synchronized, automatically scan and collect all data entity classes after startup, and generate ClickHouse table creation statements and the data mapping relationship between ClickHouse and MongoDB through the table-level annotations of the data entity classes and all attributes marked with field-level annotations, and cache the parsed data into the ClickHouseManager manager. Finally, an empty ClickHouse data table is generated through a unified scheduler, the MongoDB data is loaded in real time, and the cached information is used for data parsing, conversion, and synchronization.

Description

Technical Field

[0001] The present invention relates to the technical field of data synchronization, and particularly relates to a method for synchronizing data between MongoDB and ClickHouse. Background Art

[0002] In the Internet enterprise architecture, ClickHouse is often used as an intermediate database for processing online analytical processing (OLAP) tasks. In this case, ClickHouse often needs to synchronize a large amount of data from other databases (such as MongoDB databases). In order to synchronize data to ClickHouse, developers used to write data synchronization logic for each table, mainly including reading data from MongoDB, data conversion, writing data to ClickHouse, etc. Although some logic of the program can be modularized for different data tables to share, this processing method ultimately still requires writing a set of data entity classes and corresponding data conversion and mapping logics for each table, which is time-consuming and laborious in the development process and is not conducive to subsequent modification and maintenance.

[0003] To facilitate data synchronization, ClickHouse officially provides a MySQL engine, and data can be automatically synchronized with the help of other commands. However, currently, the official does not provide a synchronization engine for MongoDB, and in actual applications, some data conversions will occur during the data synchronization process. This completely synchronous method does not support customized data conversions, thus returning to the above-mentioned cumbersome development.

[0004] To solve the above problems, it is particularly necessary to develop a method for synchronizing data between MongoDB and ClickHouse. Summary of the Invention

[0005] Aiming at the deficiencies in the prior art, the purpose of the present invention is to provide a method for synchronizing data between MongoDB and ClickHouse. Based on the original MongoDB database operation logic, only a small number of annotations need to be added to achieve automatic ClickHouse data synchronization, with less development effort, high flexibility, and easy to promote and use.

[0006] To achieve the above purpose, the present invention is implemented through the following technical solutions: A method for synchronizing data between MongoDB and ClickHouse, the steps of which are as follows:

[0007] (1) Design and write table-level annotations and field-level annotations required for synchronization;

[0008] (2) Mark the annotations on the MongoDB data table entities and attributes to be synchronized;

[0009] (3) After starting the service, the application automatically scans and collects all data entity classes, and obtains the data entity classes with the ClickHouse table-level annotation marked;

[0010] (4) Through the table-level annotation, obtain the name of the ClickHouse data table to be generated;

[0011] (5) Utilize the Java reflection mechanism to obtain all the attributes of the data entity class marked with field-level annotations, record the target field name, data conversion processor, and the method to obtain the attribute value;

[0012] (6) Through the table-level annotation of the data entity class and all the attributes marked with field-level annotations, parse and generate the ClickHouse table creation statement and the data mapping relationship between ClickHouse and MongoDB, and cache the parsed data into the ClickHouseManager manager;

[0013] (7) Generate an empty ClickHouse data table through a unified scheduler;

[0014] (8) Utilize the original general data operation template of MongoDB to load MongoDB data in real time, and utilize the cached information for data parsing, conversion, and synchronization.

[0015] Preferably, the annotation in step (1) contains the information required to generate the ClickHouse table.

[0016] Preferably, the data processing in step (8) includes:

[0017] ① Call the method to obtain the attribute value to obtain the attribute value;

[0018] ② Customize and convert the data through the data conversion processor. If not configured, this step will not be executed;

[0019] ③ Generate the ClickHouse insert statement and execute it.

[0020] The beneficial effects of the present invention: On the basis of the original MongoDB database operation logic, this method can achieve automatic ClickHouse data synchronization by adding only a small amount of annotations, with less development effort, easy management, high flexibility, and broad application prospects. Detailed implementation manners

[0021] In order to make the technical means, creative features, achieved purposes, and effects of the present invention easy to understand, the present invention will be further described below in conjunction with specific implementation manners.

[0022] The following technical solutions are adopted in this specific embodiment: A data synchronization method between MongoDB and ClickHouse, and its steps are as follows:

[0023] (1) Design and write table-level annotations and field-level annotations required for synchronization. The annotations contain information required to generate ClickHouse tables;

[0024] (2) Mark the annotations on the MongoDB data table entities and attributes to be synchronized;

[0025] (3) After starting the service, the application automatically scans and collects all data entity classes, and obtains the data entity classes with marked ClickHouse table-level annotations;

[0026] (4) Obtain the name of the ClickHouse data table to be generated through the table-level annotation;

[0027] (5) Use the Java reflection mechanism to obtain all the attributes of the data entity class marked with field-level annotations, and record the target field name, data conversion processor, and the method to obtain the attribute value;

[0028] (6) Through the table-level annotation of the data entity class and all the attributes marked with field-level annotations, parse and generate the ClickHouse table creation statement and the data mapping relationship between ClickHouse and MongoDB, and cache the parsed data into the ClickHouseManager manager;

[0029] (7) Generate an empty ClickHouse data table through a unified scheduler;

[0030] (8) Use the original MongoDB general data operation template to load MongoDB data in real time, and use the cached information for data parsing, conversion, and synchronization. The specific data processing includes:

[0031] ① Call the method to obtain the attribute value to obtain the attribute value;

[0032] ② Customize the data conversion through the data conversion processor. If not configured, this step is not executed;

[0033] ③ Generate a ClickHouse insert statement and execute it.

[0034] This specific embodiment introduces the developed and integrated dependent JAR packages, including the required annotations, etc. By marking the annotations on the data tables to be imported, connecting to the ClickHouse database, calling the main program, and passing in the name of the data table to be synchronized, it triggers the data synchronization logic, which highly abstracts and encapsulates the data synchronization from MongoDB to ClickHouse, facilitating data synchronization.

[0035] This specific embodiment solves the problem of synchronizing data from MongoDB to ClickHouse. Through annotation, it reduces the development effort, enables heterogeneous databases to share data entity classes for easy management, and also supports custom data conversion logic, which is very flexible. Developers only need to care about the conversion of specific data, and it has broad market application prospects.

[0036] The above has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. What is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and all these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. A data synchronization method for MongoDB and ClickHouse, characterized in that, The steps are as follows: (1) Design and write table-level annotations and field-level annotations required for synchronization; (2) Mark the table-level annotations and field-level annotations to the MongoDB data table entities and attributes that need to be synchronized; (3) After the service is started, the application automatically scans and collects all data entity classes, and obtains the data entity classes with marked ClickHouse table-level annotations; (4) Through the table-level annotations, obtain the name of the ClickHouse data table to be generated; (5) Utilize the Java reflection mechanism to obtain all the attributes of the data entity class marked with field-level annotations, and record the target field name, data conversion processor, and the method to obtain the attribute value; (6) Through the table-level annotations of the data entity class and all the attributes marked with field-level annotations, parse and generate the ClickHouse table creation statement and the data mapping relationship between ClickHouse and MongoDB, and cache the parsed data into the ClickHouseManager manager; (7) Generate an empty ClickHouse data table through a unified scheduler; (8) Utilize the original MongoDB general data operation template to load MongoDB data in real time, and utilize the cached information for data parsing, conversion, and synchronization.

2. The data synchronization method for MongoDB and ClickHouse according to claim 1, characterized in that The table-level annotations and field-level annotations in step (1) contain the information required to generate the ClickHouse table.

3. A data synchronization method for MongoDB and ClickHouse according to claim 1, characterized in that The data processing in step (8) includes: ① Call the method to obtain the attribute value to obtain the attribute value; ② Customize and convert the data through the data conversion processor. If not configured, this step is not executed; ③ Generate the ClickHouse insert statement and execute it.