A method and device for data synchronization operation numbering

Abstract

The present invention relates to a method and device for numbering data synchronization operations. The method part mainly comprises: establishing a queue of tables to be synchronized at a source end, adding a table to be synchronized to the queue of tables to be synchronized, and setting a log parsing start LSN and a log parsing end LSN of the table; after the source end receives the operation to be parsed, the operation is parsed and numbered by judging whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN. By setting the log parsing start LSN and the log parsing end LSN, the present invention can allow the log parsing service to clearly know after which LSN of the log stream the log operation of the table needs to be parsed, so that after a fault restart, when the parsed log operations are numbered again, the same numbering sequence as last time can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of database data processing, and in particular to a method and device for data synchronization operation numbering. Background Art

[0002] Currently, heterogeneous database replication technology based on database log analysis is widely used. This technology captures the incremental data of the database at the source end and then sends it to the target end. At the target end, the incremental data is applied to the target database through a common database access interface to achieve data replication. Because this technology uses a common database interface, it supports heterogeneous database system replication and heterogeneous operating system environments. The target standby database system is readable and writable, which is a "dual-active" system.

[0003] The synchronization system based on database log analysis generally synchronizes in units of transactions. When analyzing the transaction commit log operation, the corresponding transaction is located by the transaction ID, and then the operations in the corresponding transaction are stored in the target database for processing to complete the synchronization of the transaction.

[0004] During transaction synchronization, there is a partial rollback operation on the source database. Therefore, when managing the operations in the transaction, it is generally necessary to number them in ascending order according to the operation sequence, and then complete the partial rollback operation according to the size of the operation number. In addition, in a synchronization environment where the consistency of the data synchronization operation sequence needs to be checked, the operations on each transaction are numbered in order from small to large, which can easily implement the consistency check of the transaction operation sequence. The difference between the numbers of two adjacent operations in the same transaction is 1. If not, it means that the transaction has lost operations during the transmission process. When the synchronization system is running, it is often necessary to dynamically increase or decrease the tables that need to be synchronized. This action may cause the synchronization system to have inconsistent numbers for each operation of the same transaction after the failure restart and before the failure, which ultimately affects the accuracy of completing the partial rollback operation by numbering or the accuracy of the target end data synchronization service to check the consistency of the transaction operation sequence.

[0005] In the existing data synchronization system of log parsing, when parsing the source database log, on-demand parsing is generally adopted. When the log capture module captures a certain operation log, it first extracts the table ID corresponding to the operation, and then obtains the table definition through the table ID. If it is found that the table does not need to be synchronized, the log operation is directly discarded to avoid wasting CPU resources. The discarded log operation is not numbered because it will affect the continuity of the transaction operation number, thereby affecting the verification of the transaction operation sequence by the target data synchronization service; otherwise, the log is parsed according to the obtained table definition, and then numbered according to the order in which the operation appears in the transaction. It can be seen that the above log parsing data synchronization system parses the log on demand, and then numbers the operations in sequence according to the actual parsed operations. When multiple table operations appear in a transaction, and some of these tables need to be synchronized and some do not, then under different synchronization requirements, the same operation will have inconsistent numbers in the same transaction.

[0006] In view of this, how to overcome the defects of the existing technology, how to solve the technical problem of how to number each operation of the transaction when the data synchronization system is parsing the log, and how to ensure that each operation number of the same transaction is completely consistent with the number before the failure in the event of a failure restart have become important technical problems that the industry needs to solve urgently. Summary of the invention

[0007] In view of the above defects or improvement needs of the prior art, the present invention provides a method and device for numbering data synchronization operations, which utilizes the log LSN and the table to be synchronized to bind, and sets the log parsing start LSN and log parsing end LSN for each table to be synchronized. In this way, when the table to be synchronized is dynamically added or deleted during the data synchronization process, the table can be parsed according to the log parsing start LSN and log parsing end LSN of the table after a fault restart, ensuring that the operation of each transaction can get the same operation number no matter how many times it is parsed, thereby achieving consistency of each operation number in the data synchronization transaction.

[0008] The embodiment of the present invention adopts the following technical solution:

[0009] In a first aspect, the present invention provides a method for data synchronization operation numbering, comprising:

[0010] Create a queue of tables to be synchronized on the source side, add the tables to be synchronized to the queue, and set the log parsing start LSN and log parsing end LSN of the table;

[0011] After the source receives the operation to be parsed, it parses and numbers the operation by determining whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN.

[0012] Furthermore, the process of establishing a queue of tables to be synchronized at the source end, adding the tables to be synchronized to the queue of tables to be synchronized, and setting the log parsing start LSN and log parsing end LSN of the tables specifically includes:

[0013] When adding a table to be synchronized, set the log parsing start LSN and log parsing end LSN of the table according to the maximum log LSN of the current log at the time of adding, and add the table ID to the queue of tables to be synchronized;

[0014] When deleting a table that is being synchronized, the log parsing termination LSN of the table is updated according to the maximum log LSN of the current log at the time of deletion, and the log parsing interval of the table in the queue of tables to be synchronized is modified.

[0015] Further, when adding a table to be synchronized, setting the log parsing start LSN and log parsing end LSN of the table according to the maximum log LSN of the current log at the time of adding, and adding the table ID to the queue of tables to be synchronized specifically includes:

[0016] Get the table ID of the table to be synchronized from the database;

[0017] Get the maximum log LSN of the current log from the database, and set the maximum log LSN as the log parsing start LSN of the table to be synchronized;

[0018] Set the log parsing termination LSN of the table to be synchronized to infinity;

[0019] Add the table ID to the queue of tables to be synchronized.

[0020] Further, when deleting a table being synchronized, updating the log parsing termination LSN of the table according to the maximum log LSN of the current log at the time of deletion, and modifying the log parsing interval of the table in the queue of tables to be synchronized specifically include:

[0021] Get the table ID of the table being synchronized from the database;

[0022] Get the maximum log LSN of the current log from the database, and update the log parsing termination LSN of the table being synchronized to the maximum log LSN;

[0023] Find the corresponding table in the queue of tables to be synchronized based on the table ID and modify its log parsing interval.

[0024] Further, after receiving the log operation to be parsed, the source end completes the parsing of the operation and numbering the operation by judging whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN. Specifically, the parsing of the operation and numbering of the operation include:

[0025] After receiving the log operation to be parsed, the source end extracts the table ID of the log operation to be parsed;

[0026] Table ID judgment: judge whether the table ID to which the operation belongs exists in the queue of tables to be synchronized. If so, perform LSN judgment;

[0027] LSN judgment: judge whether the LSN of the operation is within the start LSN and end LSN interval of the table to which it belongs. If so, parse the log operation and obtain the transaction ID of the operation;

[0028] The operation number of the transaction is incremented by one according to the transaction ID, and the number after the increment is set as the number of the current operation, and the operation is sent to the target end data synchronization service.

[0029] Furthermore, in the table ID judgment, if it is determined that the table ID does not exist in the queue of tables to be synchronized, the log operation is directly discarded without parsing; in the LSN judgment, if it is determined that the LSN of the operation is not within the start LSN and end LSN interval of the table to which it belongs, the log operation is directly discarded without parsing.

[0030] Furthermore, when the source end receives a commit or rollback log, it obtains the corresponding transaction ID, increases the operation number of the transaction by one according to the transaction ID, sets the number after the increase by one as the number of the current operation, and sends the commit or rollback operation to the target end data synchronization service to complete the parsing of the transaction.

[0031] Furthermore, it also includes:

[0032] After receiving the transaction commit message, the target-side data synchronization service processes the transaction to be synchronized and sets the fault recovery start LSN, which refers to the log starting point where the source end restarts analysis after the synchronization service fails and restarts;

[0033] The source end periodically obtains the latest fault recovery start LSN from the target end, and deletes the tables whose log parsing end LSN in the queue of tables to be synchronized is less than the fault recovery start LSN according to the fault recovery start LSN.

[0034] Furthermore, the deployment preparation steps are also included:

[0035] The data synchronization log capture and log parsing services are deployed on the source database to capture and parse log operations and send the parsed operations to the target data synchronization service.

[0036] On the other hand, the present invention provides a device for data synchronization operation numbering, specifically: comprising at least one processor and a memory, wherein the at least one processor and the memory are connected via a data bus, and the memory stores instructions that can be executed by the at least one processor, and after the instructions are executed by the processor, they are used to complete the method for data synchronization operation numbering in the first aspect.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows: first, by setting the log parsing start LSN and log parsing end LSN of the table ID, the log parsing service can clearly know after which LSN of the log stream the log operation of the table needs to be parsed, and when the table no longer needs to be synchronized, the table is not immediately deleted from the queue of tables to be synchronized, but the log parsing of the table is stopped by setting the log parsing end LSN. This ensures that after a fault restart, the parsed log operations can be numbered again in the same order as last time.

[0038] Secondly, the source-side data synchronization service regularly cleans up the queue of tables to be synchronized using the fault recovery start LSN, and removes tables whose log parsing end LSN is smaller than the fault recovery start LSN from the queue of tables to be synchronized, thus solving the problem of invalid synchronization tables in the queue of tables to be synchronized. The combination of the queue of tables to be synchronized and setting the start and end LSN of log parsing for each table in the queue can effectively ensure the consistency and sequential increment of the parsing operation number for multiple log parsings. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 A flow chart of a method for data synchronization operation numbering provided in Example 1 of the present invention;

[0041] Figure 2 A specific flow chart of step 100 provided in Example 1 of the present invention;

[0042] Figure 3 A specific flow chart of step 110 provided in Example 1 of the present invention;

[0043] Figure 4 A specific flow chart of step 120 provided in Example 1 of the present invention;

[0044] Figure 5 A specific flow chart of step 200 provided in Example 1 of the present invention;

[0045] Figure 6 A schematic diagram of the device structure for data synchronization operation numbering provided in Example 4 of the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described in conjunction with the drawings in the embodiment of the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, and the embodiments described herein are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention can be combined with each other arbitrarily to form a feasible technical solution, but it must be based on the ability of ordinary technicians in this field to achieve it. When the combination of technical solutions is contradictory or cannot be achieved, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0047] The present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.

[0048] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0049] Embodiment 1:

[0050] like Figure 1 As shown, an embodiment of the present invention provides a method for data synchronization operation numbering, and the specific steps are as follows.

[0051] Step 100: Create a queue of tables to be synchronized at the source end, add the table to be synchronized to the queue, and set the log parsing start LSN and log parsing end LSN of the table.

[0052] Step 200: After receiving the operation to be parsed, the source end parses and numbers the operation by determining whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN.

[0053] Through the above steps, a queue of tables to be synchronized can be established and the log parsing start LSN and log parsing end LSN of the setting table can be set. Only when the table to which the operation belongs belongs to the queue of tables to be synchronized and the LSN of the operation is within the interval range of the log parsing start LSN and the log parsing end LSN, it will be parsed and numbered. In this way, when the table to be synchronized is dynamically added or deleted during the data synchronization process, it can be parsed according to the log parsing start LSN and log parsing end LSN of the table after a fault restart, ensuring that the operation of each transaction can get the same operation number no matter how many times it is parsed, thereby achieving consistency of each operation number in the data synchronization transaction.

[0054] Specifically, the above solution of this preferred embodiment can be expanded as follows.

[0055] In this preferred embodiment, a deployment preparation step may be performed first: a data synchronization log capture and log parsing service is deployed on the source database to capture and parse log operations and send the parsed operations to the target data synchronization service.

[0056] Further, such as Figure 2 As shown, step 100 of this preferred embodiment (establishing a queue of tables to be synchronized at the source end, adding the table to be synchronized to the queue of tables to be synchronized, and setting the log parsing start LSN and log parsing end LSN of the table) can be expanded into the following two steps:

[0057] Step 110: When adding a table to be synchronized, set the log parsing start LSN and log parsing end LSN of the table according to the maximum log LSN of the current log at the time of adding, and add the table ID to the queue of tables to be synchronized. This step is performed when adding a table to be synchronized, and is mainly used to determine the log parsing start LSN of the table.

[0058] Step 120: When deleting a table being synchronized, update the log parsing termination LSN of the table according to the maximum log LSN of the current log at the time of deletion, and modify the log parsing interval of the table in the queue of tables to be synchronized. This step is performed when deleting a table being synchronized, and is mainly used to determine the log parsing termination LSN of the table.

[0059] Further, if Figure 3 As shown, the above step 110, that is, the operation of adding a table to be synchronized in the source data synchronization service system, can be expanded to:

[0060] Step 111: Obtain the table ID of the table to be synchronized from the database.

[0061] Step 112: Obtain the maximum log LSN of the current log from the database, and set the maximum log LSN as the log parsing start LSN of the table to be synchronized.

[0062] Step 113: Set the log parsing termination LSN of the table to be synchronized to infinity.

[0063] Step 114: Add the table ID to the queue of tables to be synchronized.

[0064] After the above steps are completed, the initial value of the log parsing end LSN is set to infinity, indicating that the newly added table needs to be parsed and synchronized starting from the log parsing start LSN (that is, the maximum log LSN of the current log when this step is completed).

[0065] like Figure 4 As shown, the above step 120, that is, the operation of deleting a table being synchronized in the source data synchronization service system, can be expanded to:

[0066] Step 121: Obtain the table ID of the table being synchronized from the database.

[0067] Step 122: Obtain the maximum log LSN of the current log from the database, and update the log parsing termination LSN of the table being synchronized to the maximum log LSN.

[0068] Step 123: Find the corresponding table in the queue of tables to be synchronized according to the table ID, and modify its log parsing interval. The log parsing interval is the interval from the log parsing start LSN to the log parsing end LSN.

[0069] When deleting a table that is being synchronized, the table ID cannot be deleted from the queue of tables to be synchronized. This is because once it is deleted, the log operations parsed to the table will be directly discarded after the source data synchronization is restarted. However, it is possible that the operation was analyzed during the last analysis and occupied the operation number.

[0070] After the above steps are completed, the logs of the table being synchronized from its log parsing start LSN to the log parsing end LSN (that is, the maximum log LSN of the current log of the database when this step is completed) need to be parsed and synchronized. After the current log parsing is completed, the subsequent logs of the table (that is, the logs after the log parsing end LSN) will no longer be parsed and synchronized.

[0071] like Figure 5 As shown, in this preferred embodiment, step 200 (after the source end receives the log operation to be parsed, the operation is parsed and numbered by judging whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN) specifically includes the following steps:

[0072] Step 201: After receiving the log operation to be parsed, the source end extracts the table ID of the log operation to be parsed.

[0073] Step 202: Table ID determination: Determine whether the table ID to which the operation belongs exists in the queue of tables to be synchronized, and if so, perform LSN determination. If it is determined that the table ID does not exist in the queue of tables to be synchronized, the log operation is directly discarded without being parsed.

[0074] Step 203: LSN determination: Determine whether the LSN of the operation is within the start LSN and end LSN interval of the table to which it belongs. If so, parse the log operation to obtain the transaction ID of the operation. If it is determined that the LSN of the operation is not within the start LSN and end LSN interval of the table to which it belongs, directly discard the log operation without parsing.

[0075] Step 204: According to the transaction ID, the operation number of the transaction is increased by one, and the number after the increase is set as the number of the current operation, and the operation is sent to the target end data synchronization service.

[0076] When the source side receives the log operation to be parsed, there may be multiple types of operations. Among them, when the source side log parsing module receives the commit or rollback log, it obtains the corresponding transaction ID, increases the operation number of the transaction by one according to the transaction ID, and sets the number after the increase as the number of the current operation, and sends the commit or rollback operation to the target side data synchronization service to complete the parsing of the transaction.

[0077] Based on the above-mentioned process of parsing the log operation and numbering the operation, this preferred embodiment further includes the following two steps:

[0078] Step 300: After receiving the transaction commit message, the target end data synchronization service processes the transaction to be synchronized and sets the fault recovery start LSN. The fault recovery start LSN refers to the log starting point where the source end restarts the analysis after the synchronization service fails and restarts. The fault recovery start LSN is continuously advanced as the progress of the data synchronization transaction is entered into the warehouse. The fault recovery start LSN is updated to the progress of the data synchronization transaction entering the warehouse.

[0079] Step 400: The source end periodically obtains the latest fault recovery start LSN from the target end, and deletes the tables whose log parsing end LSN in the queue of tables to be synchronized is less than the fault recovery start LSN according to the fault recovery start LSN.

[0080] The source-side data synchronization service regularly uses the fault recovery start LSN to clear out the tables in the queue of tables to be synchronized that no longer need to be parsed and synchronized. This is because after the fault is restarted, the log LSN will be larger than the log parsing end LSN of these tables. Clearing these tables from the queue of tables to be synchronized helps reduce the size of the queue of tables to be synchronized and release the memory resources they occupy.

[0081] In summary, when parsing the data synchronization log, this preferred embodiment first determines whether to parse the log through the queue of the table to be synchronized, and then determines whether parsing is required through the log parsing LSN interval of the table. This information is determined when the table is added to the synchronization queue or deleted for synchronization. No matter how many times the log parsing service is restarted, it will not affect the order of the operation numbers in the transaction, thereby ensuring that each transaction operation number has the characteristic of sequential increment. On the other hand, the source-end data synchronization service of this preferred embodiment regularly cleans up the queue of the table to be synchronized through the fault recovery start LSN, and deletes those tables whose log parsing end LSN is smaller than the fault recovery start LSN from the queue of the table to be synchronized, solving the problem of invalid synchronization tables in the queue of the table to be synchronized. The combination of the queue of the table to be synchronized and setting the start and end LSN of the log parsing for each table in the queue can effectively ensure the consistency and sequential increment of the parsing operation number each time during multiple log parsing.

[0082] Embodiment 2:

[0083] Based on the method for data synchronization operation numbering provided in Example 1, this Example 2 illustrates the present invention in more detail through a specific application example.

[0084] For example, the source database transaction TRX1 operates the log streams of tables T1 and T2 and adds the table to be synchronized as follows:

[0085] LSN operate Add Synchronization Table 1 INSERT INTO T1 VALUES(10) Add T1 table synchronization 2 INSERT INTO T2 VALUES(20) 3 INSERT INTO T2 VALUES(21) Add T2 table synchronization 4 COMMIT

[0086] After the above operations are completed, according to the rules for adding synchronization tables, when adding table synchronization, the log parsing LSN interval of the table is the log LSN of the current database to infinity, then the log parsing LSN intervals of the T1 table and the T2 table are as follows:

[0087] surface Parsing the starting LSN Parsing end LSN T1 1 infinity T2 3 infinity

[0088] Start log parsing:

[0089] Get the log with log LSN 1. Table T1 exists in the queue to be synchronized, and log LSN 1 is within the parsed LSN range of T1. Therefore, parse the log and set the operation number to 1 according to the transaction ID.

[0090] Get the log with log LSN 2. Table T2 exists in the queue to be synchronized, but log LSN 2 is less than the parsing start LSN of T2, so the operation is discarded and not parsed.

[0091] Get the log with log LSN 3. Table T2 exists in the queue to be synchronized, and log LSN 3 is within the parsed LSN interval of T1. Therefore, parse the log and set the operation number to 2 according to the transaction ID.

[0092] Get the commit log and set the number of this operation to 3 according to the transaction ID.

[0093] After the transaction is parsed, the operation numbers are as follows:

[0094] LSN Operation Number operate 1 1 INSERT INTO T1 VALUES(10) 3 2 INSERT INTO T2 VALUES(21) 4 3 COMMIT

[0095] If the log parsing service fails and restarts at this time, according to the rules of the present invention, the same operation number as the previous one will be obtained.

[0096] However, if the judgment of the start LSN and end LSN of table parsing is omitted and only the judgment of whether the table is in the queue of tables to be synchronized is used, different parsing results will be obtained. The process is as follows:

[0097] Start log parsing:

[0098] Get the log with log LSN 1. Table T1 exists in the queue to be synchronized, so parse the log and set the operation number to 1 according to the transaction ID.

[0099] Get the log with log LSN 2. Table T2 exists in the queue to be synchronized, so parse the log and set the operation number to 2 according to the transaction ID.

[0100] Get the log with log LSN 3. Table T2 exists in the queue to be synchronized, so parse the log and set the operation number to 3 according to the transaction ID.

[0101] Get the commit log and set the number of this operation to 4 according to the transaction ID.

[0102] After the transaction is parsed, the operation numbers are as follows:

[0103] LSN Operation Number operate 1 1 INSERT INTO T1 VALUES(10) 2 2 INSERT INTO T2 VALUES(20) 3 3 INSERT INTO T2 VALUES(21) 4 4 COMMIT

[0104] Compared with the previous analysis result, it can be found that the operation number obtained by judging whether the operation needs to be analyzed only according to the queue of tables to be synchronized is different from the previous one. The second analysis result has an additional insert operation for the T2 table with a log LSN of 2, which causes the subsequent operation numbers of the transaction to change, resulting in inconsistency with the previous analysis result.

[0105] In summary, the present embodiment sets the log parsing start LSN and log parsing end LSN of the table ID, so that the log parsing service can clearly know after which LSN of the log stream the table needs to be parsed for the log operation of the table, and when the table no longer needs to be synchronized, the table is not immediately deleted from the queue of the table to be synchronized, but the log parsing of the table is stopped by setting the log parsing end LSN. In this way, it can be ensured that after the failure restart, the same numbering sequence as the last time can be obtained when the parsed log operations are numbered again. Compared with the traditional technical means of judging whether the operation needs to be parsed only according to the queue of the table to be synchronized, the present embodiment not only determines whether to parse the log by the queue of the table to be synchronized during the data synchronization log parsing, but also judges whether to parse the log by the log parsing LSN interval of the table. This information is determined when the table is added to the synchronization queue or deleted for synchronization. No matter how many times the log parsing service is restarted, the order of the operation numbers in the transaction will not be affected, thereby ensuring that each transaction operation number has the characteristic of increasing in sequence.

[0106] Embodiment 3:

[0107] Based on the method for data synchronization operation numbering provided in Example 1, this Example 3 uses a specific application example to explain in more detail the role of parsing termination LSN involved in the present invention.

[0108] For example, the source database transactions TRX1 and TRX2 serially operate the log streams of the two tables T1 and T2 and the operations of adding the table to be synchronized are as follows:

[0109] LSN operate Add Synchronization Table 1 INSERT INTO T1 VALUES(10) Add T1 table synchronization 2 INSERT INTO T2 VALUES(20) 3 INSERT INTO T2 VALUES(21) Add T2 table synchronization 4 COMMIT 5 INSERT INTO T1 VALUES(11) 6 INSERT INTO T1 VALUES(12) Delete T1 table synchronization 7 INSERT INTO T2 VALUES(22) 8 COMMIT

[0110] After the above operations are completed, according to the rules for adding synchronization tables, when adding table synchronization, the log parsing LSN interval of the table is the log LSN of the current database to infinity, and then the synchronization of the T1 table is deleted when the LSN is 6. The parsing termination LSN of the T1 table is 6, so the log parsing LSN intervals of the T1 table and the T2 table are as follows:

[0111] surface Parsing the starting LSN Parsing end LSN T1 1 6 T2 3 infinity

[0112] Start log parsing the first transaction (TRX1):

[0113] Get the log with log LSN 1. Table T1 exists in the queue to be synchronized, and log LSN 1 is within the parsed LSN range of T1. Therefore, parse the log and set the operation number to 1 according to the transaction ID.

[0114] Get the log with log LSN 2. Table T2 exists in the queue to be synchronized, but log LSN 2 is less than the parsing start LSN of T2, so the operation is discarded and not parsed.

[0115] Get the log with log LSN 3. Table T2 exists in the queue to be synchronized, and log LSN 3 is within the parsed LSN interval of T1. Therefore, parse the log and set the operation number to 2 according to the transaction ID.

[0116] Get the commit log and set the number of this operation to 3 according to the transaction ID.

[0117] After the transaction is parsed, the TRX1 operation number is as follows:

[0118] LSN Operation Number operate 1 1 INSERT INTO T1 VALUES(10) 3 2 INSERT INTO T2 VALUES(21) 4 3 COMMIT

[0119] Next, parse the second (TRX2) transaction:

[0120] Get the log with log LSN 5. Table T1 exists in the queue to be synchronized, and log LSN 5 is within the parsed LSN range of T1. Therefore, parse the log and set the operation number to 1 according to the transaction ID.

[0121] Get the log with log LSN 6. Table T1 exists in the queue to be synchronized, and log LSN 6 is within the parsed LSN range of T1. Therefore, parse the log and set the operation number to 2 according to the transaction ID.

[0122] Get the log with log LSN 7. Table T2 exists in the queue to be synchronized, and log LSN 7 is within the parsed LSN interval of T2. Therefore, parse the log and set the operation number to 3 according to the transaction ID.

[0123] Get the commit log and set the number of this operation to 4 according to the transaction ID.

[0124] After the transaction is parsed, the TRX2 operation number is as follows:

[0125]

[0126]

[0127] If the source end obtains the fault recovery start LSN of 5 from the target end at this time, when using this LSN to clean up the deleted synchronization table, since the resolution end LSN of T1 is 6, which is greater than the fault recovery start LSN, the synchronization of the table cannot be cleaned up.

[0128] If the log parsing service fails and restarts at this time, the source end will re-parse the log with LSN 5, and according to the rules of the present invention, the same operation number as the previous one will be obtained.

[0129] However, if the source side omits the judgment of the table parsing termination LSN when cleaning up the deleted synchronization table, then after the second transaction parsing is completed, the synchronization requirements of the T1 table will be cleared. Then, when the failure is restarted and recovered, a different parsing result will be obtained. The process is as follows:

[0130] Start log parsing:

[0131] Get the log with LSN 5. Table T1 no longer exists in the queue to be synchronized, so the log will be discarded.

[0132] Get the log with LSN 6. Table T1 no longer exists in the queue to be synchronized, and the log will be discarded.

[0133] Get the log with log LSN 7. Table T2 exists in the queue to be synchronized, and log LSN 7 is within the parsed LSN range of T2. Therefore, parse the log and set the operation number to 1 according to the transaction ID.

[0134] Get the commit log and set the number of this operation to 2 according to the transaction ID.

[0135] After the transaction is parsed, the TRX2 operation number is as follows:

[0136] LSN Operation Number operate 7 1 INSERT INTO T2 VALUES(22) 8 2 COMMIT

[0137] Compared with the previous analysis result, it can be found that when cleaning the synchronization table, if the analysis termination LSN is not used for judgment, the TRX2 operation number will be inconsistent with that before the failure when the log is parsed again after the failure restart.

[0138] In summary, the present embodiment sets the log parsing start LSN and log parsing end LSN of the table ID, so that the log parsing service can clearly know after which LSN of the log stream the log operation of the table needs to be parsed, and when the table no longer needs to be synchronized, the table is not immediately deleted from the queue of tables to be synchronized, but the log parsing of the table is stopped by setting the log parsing end LSN. This ensures that after a fault restart, the parsed log operations can be numbered again to obtain the same numbering order as last time.

[0139] Embodiment 4:

[0140] Based on the data synchronization operation numbering method provided in the above-mentioned embodiments 1 to 3, the present invention further provides a device for implementing the data synchronization operation numbering method, such as Figure 6 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The device for data synchronization operation numbering in this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0141] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0142] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the method of data synchronization operation number in Embodiments 1 to 3. The processor 21 executes various functional applications and data processing of the device of the data synchronization operation number by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, the method of data synchronization operation number in Embodiments 1 to 3 is implemented.

[0143] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0144] The program instructions / modules are stored in the memory 22, and when executed by one or more processors 21, the method of data synchronization operation number in the above-mentioned embodiments 1 to 3 is executed, for example, the above-described Figures 1 to 5 The steps shown.

[0145] A person skilled in the art may understand that all or part of the steps in the various methods of the embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0146] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for data synchronization operation numbering, characterized in that: include: Create a queue of tables to be synchronized on the source side, add the tables to be synchronized to the queue, and set the log parsing start LSN and log parsing end LSN of the table; When adding a table to be synchronized, obtain the table ID of the table to be synchronized from the database, obtain the maximum log LSN of the current log from the database, and set the maximum log LSN as the log parsing start LSN of the table to be synchronized, set the log parsing end LSN of the table to be synchronized to infinity, and add the table ID to the queue of tables to be synchronized; when deleting a table being synchronized, obtain the table ID of the table being synchronized from the database, obtain the maximum log LSN of the current log from the database, and update the log parsing end LSN of the table being synchronized to the maximum log LSN, find the corresponding table in the queue of tables to be synchronized according to the table ID, and modify its log parsing interval. When deleting a table being synchronized, the table ID cannot be deleted from the queue of tables to be synchronized; After the source receives the operation to be parsed, it parses and numbers the operation by determining whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN.

2. The method for data synchronization operation numbering according to claim 1, characterized in that: After the source end receives the log operation to be parsed, the parsing and numbering of the operation are completed by judging whether the table to which the operation belongs belongs to the queue of tables to be synchronized and whether the LSN of the operation is between the log parsing start LSN and the log parsing end LSN. Specifically, the following steps are performed: After receiving the log operation to be parsed, the source end extracts the table ID of the log operation to be parsed; Table ID judgment: judge whether the table ID to which the operation belongs exists in the queue of tables to be synchronized. If so, perform LSN judgment; LSN judgment: judge whether the LSN of the operation is within the start LSN and end LSN interval of the table to which it belongs. If so, parse the log operation and obtain the transaction ID of the operation; The operation number of the transaction is incremented by one according to the transaction ID, and the number after the increment is set as the number of the current operation, and the operation is sent to the target end data synchronization service.

3. The method for data synchronization operation numbering according to claim 2, characterized in that: In the table ID judgment, if it is determined that the table ID does not exist in the queue of tables to be synchronized, the log operation is directly discarded without parsing; in the LSN judgment, if it is determined that the LSN of the operation is not within the start LSN and end LSN interval of the table to which it belongs, the log operation is directly discarded without parsing.

4. The method for data synchronization operation numbering according to claim 2, characterized in that: When the source side receives a commit or rollback log, it obtains the corresponding transaction ID, increments the transaction operation number by one according to the transaction ID, sets the incremented number as the number of the current operation, and sends the commit or rollback operation to the target side data synchronization service to complete the parsing of the transaction.

5. The method for data synchronization operation numbering according to any one of claims 1 to 4, characterized in that: Also includes: After receiving the transaction commit message, the target-side data synchronization service processes the transaction to be synchronized and sets the fault recovery start LSN, which refers to the log starting point where the source end restarts analysis after the synchronization service fails and restarts; The source end periodically obtains the latest fault recovery start LSN from the target end, and deletes the tables whose log parsing end LSN in the queue of tables to be synchronized is less than the fault recovery start LSN according to the fault recovery start LSN.

6. The method for data synchronization operation numbering according to any one of claims 1 to 4, characterized in that: It also includes deployment preparation steps: The log capture and log parsing services for data synchronization are deployed on the source database to capture and parse log operations and send the parsed operations to the target data synchronization service.

7. A device for data synchronization operation numbering, characterized in that: It includes at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions that can be executed by the at least one processor, and the instructions, after being executed by the processor, are used to complete the method for data synchronization operation numbering described in any one of claims 1-6.

Images