Synchronization method, terminal and storage medium of lightweight multi-target heterogeneous database

Abstract

The application discloses a kind of lightweight multi-target isomeric database synchronization method, terminal and storage medium, method includes: through Reader thread batch polling traversal reads the data of table in source database, the data is pushed to message queue to form message;Corresponding Writer thread is created for downstream database, and the data is written into downstream database by the Writer thread;When the data is successfully written by Writer thread, the position information of the data is updated to shared module;The lightweight multi-target isomeric database synchronization method proposed in the application solves the technical problems of multiple reading source database, increasing management cost and synchronization delay in the conventional isomeric database synchronization method.

Description

Technical Field

[0001] This invention relates to the field of heterogeneous database synchronization technology, and in particular to a lightweight method, terminal, and storage medium for synchronizing multi-target heterogeneous databases. Background Technology

[0002] With the continuous growth of data volume and the constant changes in business needs, the use of heterogeneous databases is becoming increasingly common. Heterogeneous databases refer to database systems of different types, structures, and functions, such as relational databases (e.g., MySQL), distributed databases (e.g., TiDB), column-oriented databases (e.g., Clickhouse), and message queue systems (e.g., Kafka). These database systems differ in performance, data models, and storage methods, thus requiring data synchronization to meet the needs of multiple systems.

[0003] Traditional synchronization methods typically involve exporting data from the source database to data files, using the database's logs, or directly reading table data; however, these methods have some problems:

[0004] 1. Multiple reads of the source database: Traditional synchronization tools require an additional synchronization task for each target database, resulting in multiple reads of the source database. Each read consumes the resources and performance of the source database and can affect the core business of the source database.

[0005] 2. Increased management and maintenance costs: Adding a synchronization task to each target database increases the management and maintenance costs of the system, requiring additional configuration and monitoring.

[0006] 3. Synchronization delay: Due to the need to read the source database multiple times, the synchronization delay is relatively large, making real-time data synchronization impossible and affecting the real-time performance and reliability of the business.

[0007] 4. Data synchronization for sharded tables: When the source database has multiple sharded tables, traditional synchronization methods create a synchronization task for each sharded table, which increases the complexity and risk of synchronization.

[0008] Therefore, existing heterogeneous database synchronization methods still have technical problems such as multiple reads of the source database, increased management costs, and synchronization delays. Summary of the Invention

[0009] The technical problem to be solved by the present invention is to provide a lightweight method, terminal and storage medium for synchronizing multi-target heterogeneous databases, in order to solve the technical problems of multiple readings of the source database, increased management costs and synchronization delays in traditional heterogeneous database synchronization methods.

[0010] The technical solution adopted by this invention to solve the technical problem is as follows:

[0011] In a first aspect, the present invention provides a lightweight method for synchronizing multi-objective heterogeneous databases, comprising:

[0012] The Reader thread reads data from the tables in the source database in batches, and then pushes the data into a message queue.

[0013] A corresponding Writer thread is created for the downstream database, and the data is written to the downstream database through the Writer thread;

[0014] After the Writer thread successfully writes the data, it updates the location information of the data to the shared module.

[0015] In one implementation, the step of reading data from tables in the source database in batches using a Reader thread, and then assembling the data into messages and pushing them to a message queue, includes the following preceding steps:

[0016] Obtain the task file name and the corresponding configuration file. After obtaining the upstream and downstream database connection information and system parameter information from the configuration file, perform system initialization.

[0017] In one implementation, the Reader thread iterates through the database tables in batches, reads data from each table, and then assembles the data into a message and pushes it to a message queue, including:

[0018] The Reader thread reads data from the tables in the source database in batches and uses the read data as the data field of a message.

[0019] The message header is composed of the Reader type corresponding to the message, the current reading position (pos), the name of the library being read, and the name of the table. The message header and the data field are then combined to form a message that is pushed to the message queue.

[0020] In one implementation, creating a corresponding Writer thread for the downstream database and writing the data to the downstream database through the Writer thread includes:

[0021] Create corresponding Writer threads for all downstream databases, and have each Writer thread extract a message from the message queue and parse the message header information.

[0022] In one implementation, parsing the message header information includes:

[0023] When the Reader of the message header information directly reads the upstream data table, and the Writer thread is a relational database or column-store database, the data field of the message is used as a SQL statement template parameter to update the specified database target table.

[0024] When the Reader of the message header information directly reads the upstream data table, and the Writer thread is a message queue system, the data in the data field is converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the message in the message queue system. According to the corresponding format standard, the queue information is formed and pushed to the message queue system.

[0025] In one implementation, parsing the message header information further includes:

[0026] When the Reader of the message header information reads the CDC table and the Writer thread is a relational database, the data in the data field is converted into row records, and the message is converted into the corresponding statement form and pushed to the database according to the type of the row record;

[0027] When the Reader of the message header information reads the CDC table and the Writer thread is a column-store database, the data in the data field is converted into row records, and the message is converted into INSERT statements with different flags according to the type of the row records;

[0028] When the Reader of the message header information reads the CDC table and the Writer thread is a message queue system, the data in the data field is converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the message in the message queue system. According to the corresponding format standard, the queue information is formed and pushed to the message queue system, and the type of the message queue in the message queue system is updated according to the type of the row records.

[0029] In one implementation, updating the data's location information to the shared module after the Writer thread successfully writes the data includes:

[0030] After the Writer thread successfully writes the data, it updates the location information of the data in the variables of the shared module.

[0031] The location-saving thread saves information from shared variables to a file at preset intervals.

[0032] In one implementation, updating the location information of the data to the sharing module further includes:

[0033] A command listening thread connects the client and the server, and the client synchronously modifies key variables in the shared module through the command listening thread.

[0034] Secondly, the present invention also provides a terminal, comprising: a processor and a memory, the memory storing a synchronization program for a lightweight multi-target heterogeneous database, the synchronization program for the lightweight multi-target heterogeneous database being executed by the processor to implement the operation of the synchronization method for the lightweight multi-target heterogeneous database as described in the first aspect.

[0035] Thirdly, the present invention also provides a computer-readable storage medium storing a synchronization program for a lightweight multi-target heterogeneous database, which, when executed by a processor, is used to implement the operation of the synchronization method for a lightweight multi-target heterogeneous database as described in the first aspect.

[0036] The present invention, by employing the above technical solution, has the following effects:

[0037] This invention solves the problem of traditional heterogeneous database synchronization methods, which require creating a synchronization task for each table, by using a Reader thread to iterate through and read data from tables in the source database in batches. This reduces maintenance complexity. After the Reader reads the data and generates a message queue, multiple Writer threads are created to write the data to multiple downstream databases, greatly reducing the number of reads from the source database and lowering its load. The data position is then updated to the shared module. This concurrent batch synchronization improves the efficiency and performance of synchronization, achieving more reliable data synchronization. The lightweight multi-target heterogeneous database synchronization method proposed in this invention solves the technical problems of multiple reads from the source database, increased management costs, and synchronization latency in traditional heterogeneous database synchronization methods. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0039] Figure 1 This is a flowchart of a lightweight multi-target heterogeneous database synchronization method in one implementation of the present invention.

[0040] Figure 2 This is a schematic diagram of the system startup process in one implementation of the present invention.

[0041] Figure 3 This is a schematic diagram of the system initialization process in one implementation of the present invention.

[0042] Figure 4 This is a functional schematic diagram of the terminal in one implementation of the present invention.

[0043] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] This invention provides a lightweight method, terminal, and storage medium for synchronizing multi-target heterogeneous databases. To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the invention.

[0045] Exemplary methods

[0046] Traditional heterogeneous database synchronization methods typically involve exporting data from the source database to data files, using database logs, or directly reading table data. These methods inevitably require multiple reads of the source database, which consumes the source database's performance, impacts its core business logic, and increases management and maintenance costs. Furthermore, using logs or directly reading table data results in significant synchronization delays, hindering real-time data synchronization, affecting business real-time performance and reliability, and increasing the complexity and risk of synchronization.

[0047] To address the aforementioned technical problems, this invention provides a lightweight multi-target heterogeneous database synchronization method. It uses a Reader thread to iterate through the tables in the source database in batches, effectively reading data from the source database in small batches. This solves the problem of traditional heterogeneous database synchronization methods requiring a separate synchronization task for each table, reducing maintenance complexity. Furthermore, after the Reader generates a message queue, multiple Writer threads are created to write the data to multiple downstream databases, significantly reducing the number of reads from the source database and lowering its load. The data location is then updated to the shared module. This concurrent batch synchronization improves synchronization efficiency and performance, achieving more reliable data synchronization. This lightweight multi-target heterogeneous database synchronization method solves the technical problems of multiple reads of the source database, increased management costs, and synchronization latency inherent in traditional heterogeneous database synchronization methods.

[0048] like Figure 1 As shown, this embodiment of the invention provides a lightweight method for synchronizing multi-target heterogeneous databases, including the following steps:

[0049] Step S100: Read the data of the tables in the source database in batches by the Reader thread, and push the data into the message queue.

[0050] In this embodiment, the lightweight multi-objective heterogeneous database synchronization method is applied in a terminal, which includes, but is not limited to, devices such as computers and mobile terminals; the terminal is equipped with a training platform for the lightweight multi-objective heterogeneous database synchronization method.

[0051] In this embodiment, a method for reading data from a source database in small batches and synchronizing it to multiple target databases is proposed. By transforming the data from the source database according to the characteristics and advantages of the target databases, such as storing it in a columnar database, distributing it concurrently to a distributed database, or sending it to a message queue system in the form of messages, the number of reads from the source database is reduced, the load on the source database is reduced, and the performance of core business is improved.

[0052] Specifically, in one implementation of this invention, step S100 includes the following steps:

[0053] Step S101: Obtain the task file name and the configuration file corresponding to the task file. After obtaining the upstream and downstream database connection information and system parameter information from the configuration file, perform system initialization.

[0054] Step S102: The Reader thread traverses the source database tables in batches to read the data, and uses the read data as the data field of a message.

[0055] Step S103: The message header is composed of the Reader type corresponding to the message, the current reading position end pos, the library name and table name being read, and the message header and the data field are combined to push the message to the message queue.

[0056] In this embodiment, a synchronization system that synchronizes SQL Server to MySQL / TiDB / ClickHouse / Kafka is used as an example for explanation.

[0057] In this embodiment, when the synchronization system starts, it first needs to obtain the filename of the task to be started from the startup command, then find the specified configuration file in the task file, and then read the upstream and downstream database connection information and system parameter information from the configuration file before performing system initialization. After initialization is complete, the location saving thread, listening thread, Reader thread, Writer thread, and command listening thread are started in sequence. The system startup process is as follows: Figure 2 As shown.

[0058] In this embodiment, the system initialization process includes reading the task file, reading the configuration file and setting the corresponding logs, creating the corresponding database connection pool, initializing job information, creating the corresponding queue, and starting the command listening thread. The system initialization process is as follows: Figure 3 As shown.

[0059] In this embodiment, the Reader thread iterates through one or more tables in SQL Server (i.e., the source database) in batches to read data. The number of tables is determined by the task file configuration, whether it is one table or multiple tables. The table names can be configured using wildcards. A batch of data obtained from the source database is used as the data field of a message. The Reader type, the current read position (pos), the database name, and the table name are then combined to form a message header. Finally, the message header and the data field are combined to form a message and pushed to the message queue. The format of a message is as follows:

[0060] queue_msg={"reader":reader_name,"pos":pos,"db":db_name,"table":queue_table_name,"data":sql_result}

[0061] In this embodiment, the Reader thread iterates through the partitioned table data, solving the problem that each table requires a separate synchronization task, thus reducing maintenance complexity. Furthermore, by reading data from the source database in small batches (2000 records per batch by default, configurable as needed), for example, if 1 million records need to be synchronized, directly reading 1 million records from the upstream database would likely impact the performance of the upstream database. Therefore, in this invention, the data is partitioned when reading data from the source database, and data is read in small batches each time, further reducing the read pressure on the source database.

[0062] In this embodiment, after synchronizing a batch of data, the Reader thread will sleep for a time slice, with a default time of 20 milliseconds, which can be configured according to requirements. This is to prevent the source database from affecting core business due to long-term read lock occupation when a large amount of data needs to be synchronized.

[0063] like Figure 1 As shown, this embodiment of the invention provides a lightweight method for synchronizing multi-target heterogeneous databases, which further includes the following steps:

[0064] Step S200: Create a corresponding Writer thread for the downstream database, and write the data to the downstream database through the Writer thread.

[0065] In this embodiment, after reading the data from the source database and storing it in the message queue at once through the Reader, the data is written to multiple downstream databases through multiple Writer threads, which greatly reduces the number of times the source database is read and reduces the load on the source database.

[0066] Specifically, in one implementation of this invention, step S200 includes the following steps:

[0067] Step S201: Create corresponding Writer threads for all downstream databases, and extract a message from the message queue through each Writer thread and parse the header information of the message.

[0068] In this embodiment, the system creates a corresponding Writer thread for each downstream database, that is, each downstream database corresponds to one Writer thread; each Writer thread reads a message from the message queue and parses the header information of the message.

[0069] Specifically, in one implementation of this invention, step S201 includes the following steps:

[0070] Step S201a: When the Reader of the message header information directly reads the upstream data table, and the Writer thread is a relational database or a column-store database, the data field of the message is updated to the specified database target table as an SQL statement template parameter.

[0071] Step S201b: When the Reader of the message header information directly reads the upstream data table and the Writer thread is a message queue system, the data of the data field is converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the message in the message queue system. According to the corresponding format standard, the queue information is formed and pushed to the message queue system.

[0072] Step S201c: When the Reader of the message header information reads the CDC table and the Writer thread is a relational database, the data in the data field is converted into row records, and the message is converted into the corresponding statement form and pushed to the database according to the type of the row records.

[0073] Step S201d: When the Reader of the message header information reads the CDC table and the Writer thread is a column storage database, the data in the data field is converted into row records, and the message is converted into INSERT statements with different flags according to the type of the row records.

[0074] In step S201e, when the Reader of the message header information reads the CDC table and the Writer thread is a message queue system, the data of the data field is converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the message in the message queue system. According to the corresponding format standard, the queue information is formed and pushed to the message queue system, and the type of the message queue in the message queue system is updated according to the type of the row records.

[0075] In this embodiment, Reader refers to the module that reads upstream data. This module includes two reading methods: one is MssqlReader, which directly reads the upstream data table, and the other is CDCReader, which reads the CDC table.

[0076] Specifically, when the Reader of the message header information is MssqlReader and the Writer thread is MySQL or ClickHouse, the data field of the message is directly inserted (updated) into the specified MySQL target table as a SQL statement template parameter.

[0077] When the Reader for the message header information is MssqlReader and the Writer thread is Kafka, the data in the data field (a batch of data) is first converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the Kafka message. Then, according to the Maxwell format standard, information such as database and table is added to assemble a Kafka message and pushed to Kafka.

[0078] A Kafka message has the following format:

[0079] kafka_msg={"database":"test1","table":"t","type":"insert","pk":"7","data":{"create_time":"2018-01-01 00:14:38","dept":6,"id":7,"last_login_time":"2018-01-07 20:07:20","name":"user_7"}}

[0080] When the Reader for the message header is CDCReader and the Writer thread is MySQL, the data in the data field (a batch of data) is first converted into row records. If the type of the row record is delete, it is converted into the parameter of the delete statement. If the type of the row record is insert or update, it is converted into the form "INSERTINTO...ON DUPLICATE KEY UPDATE" and pushed to MySQL for execution.

[0081] When the Reader for the message header is CDCReader and the Writer thread is ClickHouse, the data in the data field (a batch of data) is first converted into row records. If the row record type is delete, it is converted into an INSERT statement with a flag of -1; if the row record type is insert, it is converted into an INSERT statement with a flag of 1; if the row record type is update, it is converted into two INSERT statements with flags of -1 and 1 respectively. The record with a flag of -1 is used to offset the old record, and the record with a flag of 1 is the updated data.

[0082] When the Reader for the message header is CDCReader and the Writer thread is Kafka, the data in the data field (a batch of data) is first converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the Kafka message. Then, according to the Maxwell format standard, information such as database and table is added to assemble a Kafka message and pushed to Kafka. The "type" of the Kafka message queue is updated according to the record type. The processing is similar to that when the Reader for the message header is MssqlReader and the Writer is Kafka, the difference is that the "type" of the Kafka message queue is updated according to the record type.

[0083] In this embodiment, by reading data from the source database once, the read data is simultaneously written to multiple downstream databases, which greatly reduces the impact on the source database. Furthermore, since the data is stored in multiple downstream databases, the problem of needing to establish multiple synchronization tasks is solved, thereby reducing synchronization latency and lowering management and maintenance costs.

[0084] like Figure 1 As shown, this embodiment of the invention provides a lightweight method for synchronizing multi-target heterogeneous databases, which further includes the following steps:

[0085] Step S300: After the Writer thread successfully writes the data, it updates the location information of the data to the shared module.

[0086] In this embodiment, after the Writer thread successfully writes the data, it updates the location information of the data to the sharing module.

[0087] Specifically, in one implementation of this invention, step S300 includes the following steps:

[0088] Step S301: After the Writer thread successfully writes the data, it updates the location information of the data in the variables of the shared module.

[0089] Step S302: The location saving thread saves the information in the shared variables to a file at preset intervals;

[0090] Step S303: Connect the client and server through a command listening thread. The client synchronously modifies key variables in the shared module through the command listening thread.

[0091] In this embodiment, after all the Writer threads have successfully written all the data, the location information of the data will be updated in the job variable of the shared module. The location saving thread will save the information in the job variable to the job file at preset intervals.

[0092] In this embodiment, the location saving thread is responsible for saving the job information in the shared variable to the job file every preset time interval, the time being 3 seconds by default, which can be modified by modifying the configuration file or directly online in the synchronization client. When the synchronization instance restarts, the system will start synchronization according to the synchronization position of the job file.

[0093] In this embodiment, the command listening thread implements a server for a synchronization instance. The synchronization client can connect to the server via socket to view the synchronization progress and modify key variables such as tasks, jobs, and configurations shared by the shared module online.

[0094] This embodiment achieves the following technical effects through the above technical solution:

[0095] This invention solves the problem of traditional heterogeneous database synchronization methods, which require creating a synchronization task for each table, by using a Reader thread to iterate through and read data from tables in the source database in batches. This reduces maintenance complexity. After the Reader reads the data and generates a message queue, multiple Writer threads are created to write the data to multiple downstream databases, greatly reducing the number of reads from the source database and lowering its load. The data position is then updated to the shared module. This concurrent batch synchronization improves the efficiency and performance of synchronization, achieving more reliable data synchronization. The lightweight multi-target heterogeneous database synchronization method proposed in this invention solves the technical problems of multiple reads from the source database, increased management costs, and synchronization latency in traditional heterogeneous database synchronization methods.

[0096] Exemplary device

[0097] Based on the above embodiments, the present invention also provides a terminal, comprising: a processor, a memory, an interface, a display screen, and a communication module connected via a system bus; wherein, the processor is used to provide computing and control capabilities; the memory includes a storage medium and internal memory; the storage medium stores an operating system and computer programs; the internal memory provides an environment for the operation of the operating system and computer programs in the storage medium; the interface is used to connect to external devices, such as mobile terminals and computers; the display screen is used to display corresponding information; and the communication module is used to communicate with a cloud server or a mobile terminal.

[0098] When the computer program is executed by the processor, it is used to implement a lightweight method for synchronizing heterogeneous databases with multiple objectives.

[0099] It will be understood by those skilled in the art that Figure 4 The schematic diagram shown is merely a partial structural diagram related to the present invention and does not constitute a limitation on the terminal to which the present invention is applied. A specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0100] In one embodiment, a terminal is provided, comprising: a processor and a memory, the memory storing a synchronization program for a lightweight multi-target heterogeneous database, the synchronization program for the lightweight multi-target heterogeneous database being executed by the processor to implement the operation of the synchronization method for the lightweight multi-target heterogeneous database as described above.

[0101] In one embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a synchronization program for a lightweight multi-target heterogeneous database, the synchronization program for a lightweight multi-target heterogeneous database being executed by the processor to implement the operation of the synchronization method for a lightweight multi-target heterogeneous database as described above.

[0102] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory.

[0103] In summary, this invention provides a lightweight method, terminal, and storage medium for synchronizing multi-target heterogeneous databases. The method includes: using a Reader thread to poll and traverse data from tables in the source database in batches, and pushing the data into a message queue; creating a corresponding Writer thread for the downstream database, and using the Writer thread to write the data into the downstream database; and updating the location information of the data to the shared module after the Writer thread successfully writes the data. The lightweight method for synchronizing multi-target heterogeneous databases proposed in this invention solves the technical problems of multiple reads of the source database, increased management costs, and synchronization delays in traditional heterogeneous database synchronization methods.

[0104] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A lightweight method for synchronizing multi-objective heterogeneous databases, characterized in that, The method for synchronizing lightweight multi-objective heterogeneous databases includes the following steps: The Reader thread reads data from tables in the source database in batches, and pushes the data into a message queue. When reading data from tables in the source database, the data is split and read in small batches each time. After synchronizing a batch of data, the Reader thread sleeps for a time slice, which is preset. A corresponding Writer thread is created for each downstream database, and the data is written to the downstream database through the Writer thread, wherein each downstream database corresponds to one Writer thread; After the Writer thread successfully writes the data, it updates the location information of the data to the shared module. The Reader thread iterates through the database tables in batches, reads data, and assembles the data into messages to push them to the message queue, including: The Reader thread reads data from the tables in the source database in batches and uses the read data as the data field of a message. The message header is composed of the Reader type corresponding to the message, the current reading position (pos), the name of the library being read, and the name of the table. The message header and the data field are then combined to form a message and pushed to the message queue. The step of creating a corresponding Writer thread for the downstream database and writing the data to the downstream database through the Writer thread includes: Create corresponding Writer threads for all downstream databases, and have each Writer thread extract a message from the message queue and parse the message header information.

2. The lightweight multi-objective heterogeneous database synchronization method according to claim 1, characterized in that, The process of reading data from tables in the source database in batches using a Reader thread, assembling the data into messages, and pushing them to the message queue includes the following steps: Obtain the task file name and the corresponding configuration file. After obtaining the upstream and downstream database connection information and system parameter information from the configuration file, perform system initialization.

3. The lightweight multi-objective heterogeneous database synchronization method according to claim 1, characterized in that, The parsing of the message header information includes: When the Reader of the message header information directly reads the upstream data table, and the Writer thread is a relational database or column-store database, the data field of the message is used as a SQL statement template parameter to update the specified database target table. When the Reader of the message header information directly reads the upstream data table, and the Writer thread is a message queue system, the data in the data field is converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the message in the message queue system. According to the corresponding format standard, the queue information is formed and pushed to the message queue system.

4. The lightweight multi-objective heterogeneous database synchronization method according to claim 1, characterized in that, The parsing of the message header information also includes: When the Reader of the message header information reads the CDC table and the Writer thread is a relational database, the data in the data field is converted into row records, and the message is converted into the corresponding statement form and pushed to the database according to the type of the row record; When the Reader of the message header information reads the CDC table and the Writer thread is a column-store database, the data in the data field is converted into row records, and the message is converted into INSERT statements with different flags according to the type of the row records; When the Reader of the message header information reads the CDC table and the Writer thread is a message queue system, the data in the data field is converted into row records, and the row records and field names are serialized together into a JSON format to form the data field of the message in the message queue system. According to the corresponding format standard, the queue information is formed and pushed to the message queue system, and the type of the message queue in the message queue system is updated according to the type of the row records.

5. The lightweight multi-objective heterogeneous database synchronization method according to claim 1, characterized in that, The step of updating the data's location information to the shared module after the Writer thread successfully writes the data includes: After the Writer thread successfully writes the data, it updates the location information of the data in the variables of the shared module. The location-saving thread saves information from shared variables to a file at preset intervals.

6. The lightweight multi-objective heterogeneous database synchronization method according to claim 1, characterized in that, The step of updating the location information of the data to the sharing module further includes: A command listening thread connects the client and the server, and the client synchronously modifies key variables in the shared module through the command listening thread.

7. A terminal, characterized in that, include: The processor and memory, wherein the memory stores a synchronization program for a lightweight multi-objective heterogeneous database, the synchronization program for the lightweight multi-objective heterogeneous database being executed by the processor to implement the synchronization method for a lightweight multi-objective heterogeneous database as described in any one of claims 1-6.

8. A storage medium, characterized in that, The storage medium stores a synchronization program for a lightweight multi-objective heterogeneous database, which, when executed by a processor, is used to implement the synchronization method for a lightweight multi-objective heterogeneous database as described in any one of claims 1-6.

Images