Extensible lightweight MongoDB to Elasticsearch data real-time synchronization method

Abstract

The invention discloses an extensible lightweight MongoDB to Elasticsearch data real-time synchronization method, according to the synchronization method, a synchronization tool is developed based on a Go language, parallel MongoDB Channel Stream monitoring, data filtering, data processing and recovery functions after accidental interruption are achieved in the synchronization tool, the synchronization tool supports a configuration mode of a YAML format, and use of the tool is simplified. And a plug-in interface is designed, so that a user is supported to customize processing logic in a configuration file, fragment full-amount extraction of a MongoDB table based on Goroutine is supported, the method is used for initial full-amount synchronization, and incremental updating of the table to be synchronized is monitored in parallel based on Goroutine and MongoDB Change Streams characteristics. According to the method, the Go language is used for development, light weight and high efficiency are achieved, particularly, the memory occupation amount is far lower than that of Flink CDC + Kafka, the framework is simple, only one service is needed, compared with the Flink CDC + Kafka, the cost of the server can be reduced by two thirds or above, and the server is very easy to maintain due to the simple framework.

Description

Technical Field

[0001] The present invention relates to the field of data synchronization, and in particular to an extensible lightweight MongoDB to Elasticsearch data real-time synchronization method. Background Art

[0002] As the most popular document-based database, MongoDB supports a variety of query methods, but its full-text search performance is not high under massive data and complex conditions, especially the full-text search support for Chinese is not friendly enough. Therefore, in actual use, it is usually necessary to synchronize a copy of the data in MongoDB to Elasticsearch, which is better at full-text search, so as to support Chinese full-text search through Elasticsearch. And this synchronization is best in real time, so that the changes in MongoDB can be reflected in the query results of Elasticsearch in real time. In addition, some data conversion operations may be required during the synchronization process, such as filtering out records that do not need to be synchronized (Filter), deleting fields that do not need to be full-text indexed, or replacing foreign key fields with corresponding nested documents (Map), etc. Such conversion operations depend entirely on business needs and may be very complex, so when synchronizing, it is also necessary to support writing code to perform custom processing on the data to be synchronized.

[0003] Currently, the most commonly used data synchronization solution for heterogeneous systems in the industry is Flink CDC. Flink CDC can capture change records of multiple data sources (including MongoDB) and generate incremental event streams. The business code can then process the event streams in real time, such as writing them to Elasticsearch. However, to ensure stability, it is generally necessary to cooperate with stream processing middleware such as Kafka, that is, the incremental events are first written to Kafka by Flink CDC and then read from Kafka by another service and written to Elasticsearch.

[0004] The defects of the above technical solution are:

[0005] 1. The operation and maintenance and server costs of Flink CDC+Kafka are very high and not lightweight enough;

[0006] 2. In order to develop custom conversion logic, you need to learn Flink-related knowledge and tools, which leads to high development costs;

[0007] 3. It is difficult to independently control the synchronization progress of each table, and the actual business scenarios are often complex and changeable. For example, you may need to add a new table to be synchronized. At this time, the new table needs to be fully synchronized first and then transferred to incremental synchronization. The table already in incremental synchronization cannot be affected. A table that is already in incremental synchronization may need to be fully synchronized again and then transferred to incremental synchronization due to table structure adjustment. Other tables that are already in incremental synchronization cannot be affected. Sometimes you may need to roll back the time point of incremental synchronization of a table and restart incremental synchronization from this point. For example, if there is a bug in the newly launched synchronization conversion logic, it needs to be resynchronized from the previous online time after the fix. Although each table can achieve the method of independently controlling the synchronization progress by having an independent set of Flink CDC instances + Kafka Topic, the cost is obviously too high. Summary of the invention

[0008] The purpose of the present invention is to provide an extensible lightweight MongoDB to Elasticsearch data real-time synchronization method to solve the problems raised in the above background technology.

[0009] To achieve the above object, the present invention provides the following technical solution: a scalable lightweight MongoDB to Elasticsearch data real-time synchronization method, comprising the steps of:

[0010] S1: Develop a synchronization tool based on the Go language, and implement parallel monitoring of MongoDBChange Stream, data filtering, data processing, and recovery after unexpected termination in the synchronization tool;

[0011] S2: When the synchronization tool is started or receives a SIGHUP signal, it traverses the configuration file and determines whether there is a record of the corresponding table to be synchronized in the MongoDB progress table. If not, it performs full synchronization, i.e., step S3. If yes, it performs step S4.

[0012] S3: Enter the full synchronization phase, insert a record in the schedule and set it to the current time, that is, the incremental synchronization after the full synchronization needs to start from the start time of the full synchronization. After the full synchronization phase is completed, proceed to step S6;

[0013] S4: further determine the value status. If the value is full, proceed to the full synchronization phase in step S3. If the value is incremental, proceed to step S5.

[0014] S5: Incremental synchronization phase, use the starting position when initializing MongoDB Change Stream, and proceed to step S7 after completion;

[0015] S6: Slice and process the full amount of data in parallel to reduce the time consumption of full synchronization. After completion, proceed to step S7;

[0016] S7: Use the Filter tool to filter and exclude records that do not need to be synchronized;

[0017] S8: poll the OP Channel through the Event Loop, retrieve the data to be synchronized, and determine whether the process method is implemented. If so, execute the method and proceed to step S10; if not, determine the type of OP, i.e., step S9;

[0018] S9: If the OP type is insert or update, the method in the configuration is executed to process the data. If the method is not defined, the default Map logic is used, that is, the data is kept as it is.

[0019] S10: After the data is processed, it is submitted to Elasticsearch Bulk and then synchronized in batches to the Elasticsearch database;

[0020] S11: Check the sum in the MongoDB schedule table at the preset time, and if there is a value, automatically adjust the synchronization stage or incremental synchronization point.

[0021] Preferably, the synchronization tool in step S1 supports configuration in YAML format, is made to support full extraction of shards of MongoDB tables based on Goroutine, supports parallel monitoring of incremental updates of tables to be synchronized based on Goroutine and MongoDB Change Streams features.

[0022] Preferably, for the incremental synchronization in step S5, the latest time point currently processed is updated to a field of the MongoDB schedule table at every predetermined time interval.

[0023] Preferably, the sharding method in step S6 is to call a MongoDB command to split the value range of the field into a specified number of slices.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] 1. The present invention is developed in Go language, which is lightweight and efficient, especially the memory usage is much lower than FlinkCDC+Kafka, and the architecture is simple, requiring only one service (i.e., one process). Compared with Flink CDC+Kafka, the server cost of the present invention can be reduced by more than two-thirds, and due to the simple architecture, it is very easy to maintain.

[0026] 2. When using the present invention, if you need to customize the synchronous conversion logic, you only need to learn three interfaces and write a configuration file, which greatly reduces the development cost.

[0027] 3. The present invention supports conveniently adding tables to be synchronized and modifying the progress of tables being synchronized, without modifying the code or restarting the process, and will not affect the synchronization of unrelated tables. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is an implementation flow chart of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1 The present invention provides a technical solution: a scalable lightweight MongoDB to Elasticsearch data real-time synchronization method, comprising the steps of:

[0031] S1: Develop a synchronization tool based on the Go language, and implement parallel monitoring of MongoDBChange Stream, data filtering, data processing, and recovery after unexpected termination in the synchronization tool;

[0032] S2: When the synchronization tool is started or receives a SIGHUP signal, it traverses the configuration file and determines whether there is a record of the corresponding table to be synchronized in the MongoDB progress table. If not, it performs full synchronization, i.e., step S3. If yes, it performs step S4.

[0033] S3: Enter the full synchronization phase, insert a record in the schedule and set it to the current time, that is, the incremental synchronization after the full synchronization needs to start from the start time of the full synchronization. After the full synchronization phase is completed, proceed to step S6;

[0034] S4: further determine the value status. If the value is full, proceed to the full synchronization phase in step S3. If the value is incremental, proceed to step S5.

[0035] S5: Incremental synchronization phase, use the starting position when initializing MongoDB Change Stream, and proceed to step S7 after completion;

[0036] S6: Slice and process the full amount of data in parallel to reduce the time consumption of full synchronization. After completion, proceed to step S7;

[0037] S7: Use the Filter tool to filter and exclude records that do not need to be synchronized;

[0038] S8: poll the OP Channel through the Event Loop, retrieve the data to be synchronized, and determine whether the process method is implemented. If so, execute the method and proceed to step S10; if not, determine the type of OP, i.e., step S9;

[0039] S9: If the OP type is insert or update, the method in the configuration is executed to process the data. If the method is not defined, the default Map logic is used, that is, the data is kept as it is.

[0040] S10: After the data is processed, it is submitted to Elasticsearch Bulk and then synchronized in batches to the Elasticsearch database;

[0041] S11: Check the sum in the MongoDB schedule table at the preset time, and if there is a value, automatically adjust the synchronization stage or incremental synchronization point.

[0042] In this embodiment, the synchronization tool in step S1 supports the configuration method in YAML format, and is made to support the full extraction of shards of MongoDB tables based on Goroutine, and supports parallel monitoring of incremental updates of the tables to be synchronized based on Goroutine and MongoDB Change Streams features.

[0043] In this embodiment, for the incremental synchronization in step S5, the latest time point currently processed is updated to the field of the MongoDB schedule table at every predetermined time interval, for example, 5 seconds.

[0044] In this embodiment, the sharding method in step S6 is to call a MongoDB command to split the value range of the field into a specified number of pieces.

[0045] The YAML sample configuration of the tool developed based on the GO language is as follows:

[0046]

[0047]

[0048]

[0049] The synchronization progress structure of each table stored in MongoDB is as follows:

[0050]

[0051] It is worth noting that: the present invention also provides definitions of the above terms:

[0052] Filter: The implementation of the Filter method in the plug-in is used to filter data and exclude records that do not need to be synchronized.

[0053] Map: Implementation of the Map method in the plugin, used for simple one-to-one processing of data.

[0054] Process: Implementation of the Process method in the plug-in, used to perform complex data processing and directly operate Elasticsearch.

[0055] OP Channel: A memory data structure that temporarily stores data. OP is the abbreviation of operation, which means the add, delete, and update operations in MongoDB. This naturally corresponds to the incremental changes obtained by listening to the MongoDB Change Stream. At the same time, in order to simplify processing, a set of code supports both full and incremental synchronization. When synchronizing the full data, all data will be encapsulated as incremental operations.

[0056] Event Loop: used to poll the OP Channel to process the data to be synchronized in a timely manner, and also save the synchronization progress to MongoDB at regular intervals.

[0057] Bulk: Elasticsearch BulkAPI, used to perform batch operations on Elasticsearch.

[0058] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0059] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0060] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0061] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0062] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0063] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0064] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0065] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0066] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0067] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A scalable and lightweight real-time data synchronization method from MongoDB to Elasticsearch, characterized in that: Includes steps: S1: Develop a synchronization tool based on the Go language, and implement parallel monitoring of MongoDB ChangeStream, data filtering, data processing, and recovery after unexpected termination in the synchronization tool; S2: When the synchronization tool is started or receives a SIGHUP signal, it traverses the configuration file and determines whether there is a record of the corresponding table to be synchronized in the MongoDB progress table. If not, it performs full synchronization, i.e., step S3. If yes, it performs step S4. S3: Enter the full synchronization phase, insert a record in the schedule and set it to the current time, that is, the incremental synchronization after the full synchronization needs to start from the start time of the full synchronization. After the full synchronization phase is completed, proceed to step S6; S4: further determine the value status. If the value is full, proceed to the full synchronization phase in step S3. If the value is incremental, proceed to step S5. S5: Incremental synchronization phase, use the starting position when initializing MongoDB Change Stream, and proceed to step S7 after completion; S6: Slice and process the full amount of data in parallel to reduce the time consumption of full synchronization. After completion, proceed to step S7; S7: Use the Filter tool to filter and exclude records that do not need to be synchronized; S8: poll the OP Channel through the Event Loop, retrieve the data to be synchronized, and determine whether the process method is implemented. If so, execute the method and proceed to step S10; if not, determine the type of OP, i.e., step S9; S9: If the OP type is insert or update, the method in the configuration is executed to process the data. If the method is not defined, the default Map logic is used, that is, the data is kept as it is. S10: After the data is processed, it is submitted to Elasticsearch Bulk and then synchronized in batches to the Elasticsearch database; S11: Check newStatus and newIncrementalOffset in the MongoDB schedule at the preset time. If there is a value, the synchronization phase or incremental synchronization point is automatically adjusted.

2. According to claim 1, a scalable lightweight MongoDB to Elasticsearch data real-time synchronization method, characterized in that: In step S1, the synchronization tool supports the configuration mode in the YAML format, makes a full shard extraction of the MongoDB table based on Goroutine, supports parallel monitoring of incremental updates of the table to be synchronized based on Goroutine and MongoDB Change Streams features.

3. The scalable lightweight MongoDB to Elasticsearch data real-time synchronization method according to claim 1, characterized in that: For the incremental synchronization in step S5, the latest time point currently processed is updated to the field of the MongoDB schedule table at every predetermined interval.

4. The scalable lightweight MongoDB to Elasticsearch data real-time synchronization method according to claim 1, characterized in that: The sharding method in step S6 is to call a MongoDB command to split the value range of the field into a specified number of slices.

Images