Implementation method and device of data synchronization service

Abstract

The application discloses an implementation method and equipment of a data synchronization service, belongs to the technical field of data synchronization, and is used for solving the technical problem that the existing data synchronization solution supports few database types, has great limitations and cannot expand the application scope while guaranteeing real-time performance. The method comprises the following steps: an application server constructs a corresponding data change set according to a database transaction generated by an application program, and pushes the data change set to a message queue; a data synchronization server receives and analyzes the data change set sent by the message queue; the data synchronization server verifies the transaction state of the database transaction to the application server; and when the transaction state is a commit state, the data synchronization server synchronizes the corresponding data to a target database. While guaranteeing data consistency, the data synchronization efficiency is greatly improved, and any relational database can be adapted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data synchronization, in particular to an implementation method and device of a data synchronization service. BACKGROUND

[0002] Now it is the era of data explosion, both individuals and enterprises will produce massive amounts of data every day. E-commerce, group buying, take-out, new media can push accurately to customers through user behavior habits. Enterprises can also make decisions and plans through data analysis. The original data generated by mobile phones, PCs, smart terminals and other devices have the characteristics of large data volume, poor data quality, and inconsistent data structure, and cannot directly generate data value. Data needs to be synchronized to the query library for data cleaning and processing, and then analyzed and mined to produce real value.

[0003] Massive data will be generated on the Internet every moment, and the real-time requirement of data synchronization is very high in order to truly realize the value of data, because the value of data will become smaller over time. There are three kinds of data synchronization solutions at present: one is active extraction, which is suitable for synchronizing unstructured data such as logs, Excel and other data; two is scheduled task extraction, which is the most popular data synchronization solution in the past; three is real-time extraction, that is, real-time data extraction through capturing database change logs. The real-time performance of the first two solutions is poor, which cannot meet the real-time performance requirements of the current data synchronization solution, and the real-time performance of the third solution is good, but the database it adapts to is less. First of all, this solution depends on whether the database itself supports change data capture technology (CDC), even if the database supports CDC, it also needs to be adapted and developed. The current mainstream open source framework is FlinkCDC, but FlinkCDC only supports Oracle, SqlServer, Mysql, PG and a few other databases, and does not support any domestic database in the new environment, so the scope of application is small. SUMMARY

[0004] The embodiment of the present application provides an implementation method and device of a data synchronization service, which is used to solve the technical problem that the existing data synchronization solution supports few types of databases and has great limitations, and cannot expand the scope of application while ensuring real-time performance.

[0005] The embodiment of the present application adopts the following technical scheme:

[0006] In one aspect, the embodiment of the present application provides an implementation method of a data synchronization service, which comprises: an application server constructs a corresponding data change set according to a database transaction generated by an application program, and pushes the data change set to a message queue; a data synchronization server receives and analyzes the data change set sent by the message queue; the data synchronization server verifies a transaction state of the database transaction to the application server; and when the transaction state is a commit state, the data synchronization server synchronizes the corresponding data parsed out to a target database.

[0007] In one possible implementation, the application server constructs a corresponding data change set according to a database transaction generated by an application program, and pushes the data change set to a message queue, which specifically comprises: the application server listens to the database transaction generated by the application program; and listens to and intercepts JDBC, automatically acquires data generated in the database transaction and automatically organizes the data into the data change set, and caches the organized data change set in a change set log table; wherein the data change set is a data structure for describing data changes; the application server pushes a transaction start message, the data change set and a transaction commit message to the message queue in sequence; and saves transaction states corresponding to the transaction start message and the transaction commit message in a transaction log table at the same time of sending the transaction start message and the transaction commit message.

[0008] In one possible implementation, the organized data change set is cached in the change set log table, which specifically comprises: when the message queue is abnormal and cannot push messages, the application server writes the generated change set into the change set log table first; after the message queue resumes operation, the application server re-sends the data in the change set log table to the message queue and empties the change set log table.

[0009] In one possible implementation, the data synchronization server verifies the transaction state of the database transaction to the application server, which specifically comprises: after receiving the transaction start message and the data change set, if the transaction commit message is not received within a preset time, the data synchronization server looks up the transaction state of the database transaction in the transaction log table; wherein the transaction state at least includes a start state, a rollback state, a suspension state and a commit state.

[0010] In one feasible implementation, after looking up the transaction status of the database transaction in the transaction log table, the method further includes: when the transaction status is in a rollback state, the data synchronization server terminates the processing of the data change set; when the transaction status is in a suspended state, the data synchronization server queries the application server to inquire whether the database transaction is running normally; if it is running normally, it continues to wait for the transaction status to change before proceeding to the next step; if it is running abnormally, it stops the data synchronization service and resumes data synchronization after it recovers; wherein, the suspended state refers to the fact that the transaction status remains unchanged for a preset time after the database transaction begins.

[0011] In one feasible implementation, after pushing the data change set to the message queue, the method further includes: in case of an abnormal situation, the message queue persistently stores the messages that have not yet been pushed in the queue; after the message queue returns to normal, the message queue restores the persistently stored messages and re-pushes them to the data synchronization server.

[0012] In one feasible implementation, the data synchronization server receives and parses the data change set sent by the message queue, specifically including: after receiving the data change set, the data synchronization server determines the target database of the data change set; the data synchronization server parses the data change set according to the type of the target database and generates SQL statements corresponding to the syntax structure of the target database; the data synchronization server processes the SQL statements generated by parsing the change set in parallel to increase the parallelism of data synchronization.

[0013] In one feasible implementation, before synchronizing the parsed corresponding data to the target database, the method further includes: the application server extracting historical data generated in the application to the target database at high speed for data initialization; during the data initialization process, if a new data change set is generated, the new data change set is first stored in the change set log table; after the data initialization is completed, the data change set in the change set log table is sent to the message queue first, and the normal data synchronization process is performed after the change set log table is processed.

[0014] In one feasible implementation, after the data synchronization server parses the data change set and synchronizes the parsed corresponding data to the target database, the method further includes: during the data synchronization process, the data synchronization server periodically creates checkpoints and persistently stores the checkpoint data on the disk to record the current data synchronization progress; if the data synchronization service encounters an anomaly or the target database cannot be connected during the synchronization process, the data synchronization server starts data synchronization from the most recent checkpoint after the data synchronization service recovers or reconnects to the target database.

[0015] On the other hand, embodiments of this application also provide an apparatus for implementing a data synchronization service. The apparatus includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to execute a method for implementing a data synchronization service according to any of the above embodiments.

[0016] This application provides a data synchronization service implementation method and device that can significantly improve data synchronization efficiency while ensuring data consistency. It is compatible with any relational database, including in-memory databases, multidimensional databases, and columnar storage databases. Most importantly, it is compatible with domestically developed databases in the context of domestic IT innovation, as current mainstream real-time synchronization solutions do not support domestic databases. This application, as a data synchronization solution, is not limited by database type, is simple to deploy and maintain, and is independently controllable. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0018] Figure 1 A flowchart illustrating a method for implementing a data synchronization service as provided in this application embodiment;

[0019] Figure 2 This application provides a data synchronization service implementation method data interaction diagram as an embodiment of the present application;

[0020] Figure 3 This is a schematic diagram of the device structure for implementing a data synchronization service according to an embodiment of this application. Detailed Implementation

[0021] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this application.

[0022] This application provides a method for implementing a data synchronization service, such as... Figure 1 As shown, the implementation method of the data synchronization service specifically includes steps S101-S104:

[0023] S101. The application server constructs the corresponding data change set based on the database transaction generated by the application and pushes the data change set to the message queue.

[0024] Specifically, the application server listens to database transactions generated during application execution and intercepts JDBC, automatically obtains the data generated within the database transaction and automatically organizes it into a data change set; then, the organized data change set is cached in the change set log table.

[0025] When an application saves data across multiple database tables, it uses database transactions (hereinafter referred to as transactions) to ensure data consistency, preventing situations where some tables save successfully while others fail due to errors. For example, if an application needs to save data from tables A, B, and C simultaneously, without transactions, if table A saves successfully while tables B and C fail, the data would be incomplete. Therefore, when performing a save operation, the application first starts a transaction, then performs save operations on tables A, B, and C. If all operations succeed, the transaction is committed, and the data is saved. If table A succeeds while tables B and C fail, the transaction is rolled back, and the successfully saved data in table A is reverted.

[0026] As a feasible implementation method, a data change set is used to describe a data structure that has undergone data changes, such as: which data has been added to the application, which fields of which data have been modified, and which data has been deleted. This application provides two ways to organize change sets: First, the data synchronization server listens to and intercepts JDBC, automatically obtains the transaction status, and automatically organizes the data generated within the same transaction into a data change set; second, the application developers organize the data change set manually through coding.

[0027] Furthermore, the application server sequentially pushes the transaction start message, data change set, and transaction commit message to the message queue; and while sending the transaction start and commit messages, it saves the corresponding transaction status in the transaction log table. The message queue is a pipeline connecting the application server and the data synchronization service, supports cluster deployment, and can process messages in parallel for different applications and different database tables, with messages being isolated from each other.

[0028] In one embodiment, the transaction log table also stores the transaction rollback status. There are two ways to obtain the transaction status corresponding to a message: one is that the data synchronization service automatically intercepts and obtains it via JDBC; the other is that developers manually write it to the transaction log table.

[0029] As a feasible implementation method, the application server caches the organized data change sets in the change set log table. Specifically, when the message queue encounters an anomaly and cannot push messages, the application server first writes the generated change sets into the change set log table; after the message queue resumes operation, the application server resends the data in the change set log table to the message queue and clears the change set log table.

[0030] The transaction log table records the database transaction status. During the subsequent data synchronization service, it uses the transaction status in this table to determine whether data synchronization has actually occurred, ensuring data consistency. Application-generated data, change sets, and transaction status logs must all be within the same database transaction for data consistency. A change set log table caches change set data. When the message queue encounters an anomaly and cannot send messages, the application-generated change sets are first written to the change set log table. Once the message queue resumes operation, the data in this table is resent to the message queue, and then the log table is cleared. This is also for data consistency, ensuring that data generated after a message queue anomaly is not lost, preventing inconsistencies between the target database and the application database.

[0031] As a feasible implementation method, in case of an abnormal situation, the message queue will persistently store the messages that have not yet been pushed in the queue; after the message queue recovers, the message queue will restore the persistently stored messages and push them back to the data synchronization server.

[0032] In one embodiment, message push in this application refers to: after the data synchronization server correctly synchronizes the data to the target database, it will send a completion message to the message queue to indicate that the data synchronization is successful. At this time, the message queue will destroy the corresponding message stored in the message queue. Only then is the message push considered complete.

[0033] S102. The data synchronization server receives and parses the data change set sent by the message queue.

[0034] Specifically, after receiving a data change set, the data synchronization server first determines the target database for the change set. Then, based on the type of the target database, it parses the data change set and generates the corresponding SQL statements.

[0035] As a feasible implementation method, the syntax of SQL statements may differ between different databases. The data synchronization service uses the strategy pattern to generate SQL statements with the corresponding target database syntax structure based on the type of the target database when parsing the change set.

[0036] Furthermore, the data synchronization service will parse the SQL statements generated by the change set and process them in parallel, increasing the parallelism of data synchronization and improving synchronization efficiency.

[0037] S103. The data synchronization server verifies the transaction status of the database transaction with the application server.

[0038] Specifically, the data synchronization server listens to the message queue. After receiving the transaction start message and data change set from the message queue in sequence, if it does not receive the transaction commit message within a preset time, it checks the transaction status of the database transaction in the transaction log table. The transaction status includes at least the start status, rollback status, suspended status, and commit status.

[0039] Furthermore, when the transaction status is in rollback state, the data synchronization server terminates the processing of the data change set; when the transaction status is in suspended state, the data synchronization server queries the application server to inquire whether the database transaction is running normally. If it is running normally, it continues to wait for the transaction status to change before proceeding to the next step; if it is running abnormally, it stops the data synchronization service and resumes data synchronization after it recovers. Here, suspended state means that after the database transaction starts, the transaction status does not change for a preset time.

[0040] As a feasible implementation method, after receiving a data changeset, the data synchronization server will not immediately parse the changeset or synchronize the data to the target database. Instead, the server will query the transaction log table for the transaction status of the corresponding database transaction based on the received changeset information. The next step in processing the data changeset will then be based on this transaction status, mainly falling into the following categories:

[0041] a) Transaction Committed Status: The data synchronization service parses the data changeset and synchronizes the parsed data to the target database; b) Transaction Rollback Status: The data synchronization service terminates the processing of the data changeset and does not push the data to the target database; c) Transaction Suspended Status: If a transaction starts but its status remains unchanged for an extended period (neither committed nor rolledback), the transaction is suspended. The data synchronization service will proactively inquire whether the transaction is running normally. If it is running normally, it will continue to wait for the transaction status to change before proceeding to the next step. If the transaction encounters an anomaly, the data synchronization service will stop and resume data synchronization once the application recovers.

[0042] In one embodiment, when an application needs to save data from three tables A, B, and C, in addition to sending the main data changeset to the message queue to notify the data synchronization service, it also sends two messages: The first message: Transaction started; the second message: Data changeset; and the third message: Transaction committed. The data synchronization service receives the messages sequentially. Upon receiving the first message, it starts the synchronization process; upon receiving the second message, it parses the data changeset; and upon receiving the third message, it pushes the parsed data to the target database. For example, a changeset transaction is started, then the changeset is sent to the message queue; after the application successfully saves the data, it records the changeset transaction as committed. In other words, the transaction start, changeset, and transaction commit constitute all the information needed to run a data synchronization service.

[0043] S104. When the transaction status is committed, the data synchronization server will synchronize the parsed corresponding data to the target database.

[0044] After receiving the transaction commit message, or after verifying that the transaction has been committed in the transaction log table, the data synchronization server will execute the SQL statement parsed from the data change set in the target database and synchronize the executed data to the target database.

[0045] As a feasible implementation method, if an application has been running for some time and data synchronization is performed using a data synchronization service, it is necessary to extract historical data generated by the application to the target database at high speed; this process is called data initialization. Real-time synchronization is then performed after data initialization is complete. During the data synchronization service initialization process, the application system is still running. The data generated during this time is handled as follows: If a new data changeset is generated during data initialization, it is first stored in the changeset log table. After data initialization is complete, the data changesets in the changeset log table are prioritized for sending to the message queue. Normal data synchronization then proceeds only after the changeset log table has been processed.

[0046] As a feasible implementation method, during data synchronization, the data synchronization server periodically creates checkpoints and persistently stores the checkpoint data on disk to record the current data synchronization progress. If the data synchronization service encounters an anomaly or the target database becomes unreachable during synchronization, the data synchronization server will resume data synchronization from the most recent checkpoint after the data synchronization service recovers or the target database is reconnected.

[0047] Figure 2 This application provides a data synchronization service implementation method data interaction diagram, as illustrated in the embodiments of this application. Figure 2 As shown, the application server constructs a changeset based on transactions generated in the application, either automatically or manually, through JDBC interception, and saves the changeset to the changeset log. Then, it sequentially pushes three messages to the message queue: "Start Transaction," "Changeset," and "Commit Transaction." The message queue pushes these messages to the data synchronization service. After processing the first and second messages, if the data synchronization service waits for a long time without receiving the third message (commit), it actively checks the transaction log table to see if the transaction for the current changeset has been committed. If not, it queries the application service to check the transaction status. If it's normal, it waits for the commit message; otherwise, it interrupts data synchronization. Another scenario is that the application successfully sends the first and second messages but fails to send the third message. However, the transaction status in the third message is not lost but stored in the transaction log table. If the data synchronization service doesn't receive the third message for a long time, it checks the transaction log table and finds that the transaction has already been committed, indicating that the third message failed and was not received. In this case, the data synchronization service no longer needs to wait for the third message; it can directly determine whether to synchronize the target database based on the commit status in the transaction log table. The parsed change set data is then synchronized to the target database, and checkpoints are created during the synchronization process, with checkpoint data being stored and backed up. Simultaneously, historical data in the application is initialized.

[0048] In addition, embodiments of this application also provide a device for implementing a data synchronization service, such as... Figure 3 As shown, the devices that implement the data synchronization service specifically include:

[0049] At least one processor; and a memory communicatively connected to the at least one processor; wherein,

[0050] The memory stores instructions that can be executed by at least one processor, so that at least one processor can perform the following:

[0051] The application server constructs a corresponding data change set based on the database transactions generated by the application and pushes the data change set to the message queue;

[0052] The data synchronization server receives and parses the data change set sent by the message queue;

[0053] The data synchronization server verifies the transaction status of the database transaction with the application server;

[0054] When the transaction status is committed, the data synchronization server will synchronize the parsed corresponding data to the target database.

[0055] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0056] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0057] The above description is merely an embodiment of this application and is not intended to limit this application. For those skilled in the art, various modifications and variations can be made to the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the embodiments of this application should be included within the scope of the claims of this application.

Claims

1. A method for implementing a data synchronization service, characterized in that, The method includes: The application server constructs a corresponding data change set based on the database transactions generated by the application and pushes the data change set to the message queue; The data synchronization server receives and parses the data change set sent by the message queue, specifically including: After receiving the data change set, the data synchronization server determines the target database of the data change set; according to the type of the target database, the data synchronization server parses the data change set and generates SQL statements corresponding to the syntax structure of the target database; the data synchronization server processes the SQL statements generated by parsing the change set in parallel to increase the parallelism of data synchronization. The application server extracts historical data generated in the application at high speed into the target database for data initialization; If a new data change set is generated during the data initialization process, the new data change set will be stored in the change set log table first. After the data initialization is completed, the data change sets in the change set log table are sent to the message queue first. After the change set log table is processed, the normal data synchronization process is then carried out. The data synchronization server verifies the transaction status of the database transaction with the application server; When the transaction status is committed, the data synchronization server will synchronize the parsed corresponding data to the target database.

2. The method for implementing a data synchronization service according to claim 1, characterized in that, The application server constructs a corresponding data change set based on the database transactions generated by the application and pushes the data change set to a message queue, specifically including: The application server listens to database transactions generated by the application; and listens to and intercepts JDBC, automatically obtains the data generated within the database transaction and automatically organizes it into the data change set, and caches the organized data change set in the change set log table; wherein, the data change set is a data structure used to describe data changes; The application server sequentially pushes the transaction start message, the data change set, and the transaction commit message to the message queue; and while sending the transaction start message and the transaction commit message, it saves the transaction status corresponding to the transaction start message and the transaction commit message in the transaction log table.

3. The method for implementing a data synchronization service according to claim 2, characterized in that, The organization's data change sets are cached in the change set log table, specifically including: When the message queue malfunctions and cannot push messages, the application server will first write the resulting change set into the change set log table. After the message queue resumes operation, the application server resends the data in the change set log table to the message queue and clears the change set log table.

4. The method for implementing a data synchronization service according to claim 2, characterized in that, The data synchronization server verifies the transaction status of the database transaction with the application server, specifically including: After receiving the transaction start message and the data change set, if the data synchronization server does not receive the transaction commit message within a preset time, it searches for the transaction status of the database transaction in the transaction log table; wherein, the transaction status includes at least the start status, rollback status, suspended status, and commit status.

5. The method for implementing a data synchronization service according to claim 4, characterized in that, If not, after checking the transaction status of the database transaction in the transaction log table, the method further includes: When the transaction status is in rollback state, the data synchronization server terminates the processing of the data change set; When the transaction is in a suspended state, the data synchronization server queries the application server to inquire whether the database transaction is running normally. If it is running normally, it continues to wait for the transaction state to change before proceeding to the next step. If it is running abnormally, it stops the data synchronization service and resumes data synchronization after it recovers. The suspended state refers to the fact that the transaction state does not change within the preset time after the database transaction starts.

6. The method for implementing a data synchronization service according to claim 1, characterized in that, After pushing the data change set to the message queue, the method further includes: In abnormal circumstances, the message queue will persistently store messages that have not yet been pushed in the queue. After the message queue returns to normal, it will restore the persistently stored messages and push them back to the data synchronization server.

7. The method for implementing a data synchronization service according to claim 1, characterized in that, After the data synchronization server parses the data change set and synchronizes the parsed corresponding data to the target database, the method further includes: During the data synchronization process, the data synchronization server periodically creates checkpoints and persistently stores the checkpoint data on the disk to record the current data synchronization progress. If the data synchronization service encounters an error or the target database cannot be connected during the synchronization process, the data synchronization server will start data synchronization from the nearest checkpoint after the data synchronization service recovers or the target database is reconnected.

8. A device for implementing a data synchronization service, characterized in that, The device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method for implementing a data synchronization service according to any one of claims 1-7.

Images