Fault recovery method, device and system for setting transaction operation dependence

Abstract

The invention relates to a fault recovery method, device and system for setting transaction operation dependence. The method mainly comprises the steps that after a target end data synchronization service receives an operation of a source end, whether the difference between the LSN of the current operation and the LSN value of a last check point is larger than or equal to a preset value N or not is judged, and if yes, the operation within the range from the LSN of the last check point to the current LSN-1 is classified and stored into a check point file according to transactions; after the target end data synchronization service receives the submission operation of the transaction, asynchronous execution storage is performed on the transaction; and after the system is restarted due to a fault, recovering the KEY in the hash table R by utilizing the last check point file. According to the method, seamless connection between the operation received after the fault and the operation conflict detection before the fault can be realized, so that the fault is ensured not to influence the parallel capability of transaction execution before and after the fault.

Description

Technical Field

[0001] The present invention relates to the field of database technology, and in particular to a fault recovery method, device and system for setting transaction operation dependency. Background Art

[0002] In the database real-time synchronization system (supporting heterogeneity) based on the log parsing architecture, the source-side data synchronization service is responsible for capturing the operation log of the source-side database, parsing the log and restoring the corresponding database operations, and then sending them to the target-side data synchronization service. The target-side data synchronization service is responsible for executing these operations in the target-side database, thereby completing the data synchronization process.

[0003] When the target-side data synchronization service synchronizes operations on the target-side database, it often needs to execute in the order in which the transactions are submitted. Because when the previous transaction is not synchronized, the operation of the next transaction cannot be executed concurrently with the previous transaction operation without clarifying whether it conflicts with the previous transaction. It can only wait until all transactions before the current operation are submitted before it can start to execute, which greatly affects the concurrency of transaction operations entering the database. Currently, conflict detection is performed on each operation, and the transaction information on which it depends is set for each operation. During execution, it can be executed concurrently based on this dependent information, thereby achieving the purpose of improving synchronization performance. However, operation conflict detection needs to rely on a continuous conflict detection mechanism. When a failure restart occurs, conflict detection will be interrupted. After the restart, conflict detection will be reset, thereby reducing the synchronization performance after failure recovery.

[0004] In view of this, how to overcome the defects of the existing technology, how to implement the fault recovery mechanism of conflict detection, and how to reduce the impact of faults on synchronization performance have become important technical issues that need to be urgently solved in the industry. Summary of the invention

[0005] In view of the defects or improvement requirements in the prior art: how to implement a fault recovery mechanism for conflict detection and reduce the impact of faults on synchronization performance. The present invention provides a fault recovery method, device and system for setting transaction operation dependencies. In order to ensure the continuity of conflict detection during a fault, the target-side data synchronization service needs to periodically save the received operations in the form of checkpoints. After a fault occurs, the conflict detection environment is restored through the operations saved in the checkpoints to ensure the continuity of conflict detection before and after the fault, thereby reducing the impact of the fault on synchronization performance.

[0006] The present invention adopts the following technical solution:

[0007] In a first aspect, the present invention provides a fault recovery method for setting transaction operation dependency, comprising:

[0008] After receiving the operation from the source, the target data synchronization service determines whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to the preset value N. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1, and the ending LSN is set to the current operation LSN-1;

[0009] After receiving the transaction commit operation, the target-side data synchronization service asynchronously executes the transaction into the database. After the storage is completed, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The judgment is made according to the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file.

[0010] When the system fails and restarts, the last checkpoint file is used to restore the KEY in the hash table R. After the restart is completed, the source end's operations will continue to be received. The received operation LSN will be discarded if it is less than or equal to the checkpoint file's termination LSN. If it is greater, the KEY in the operation will be extracted for conflict detection and judgment.

[0011] In some embodiments, after the target-side data synchronization service is started, a hash table Q for registering transaction information is initialized to save the status of all transactions within the received conflict detection LSN range, including active status, committed status, and rollback status; after the target-side data synchronization service receives the DML operation from the source, the transaction ID of the extraction operation is registered in the hash table Q, and the transaction status is initially active.

[0012] In some embodiments, when the target-side data synchronization service receives a commit operation, it adds the transaction to the queue of pending transactions and modifies the transaction status in the hash table Q from an active status to a committed status; when the target-side data synchronization service receives a rollback operation, it discards the transaction and modifies the transaction status in the hash table Q from an active status to a rollback status, and writes the conflicting TRXID saved in the rolled back operation back to the hash table R.

[0013] In some embodiments, after the target-side data synchronization service is started, a hash table R is initialized for storing and locating received operations, and two memory slices are created for storing the KEY in the hash table R, each memory slice is incremented in the order of creation and the two memory slices are added to the memory linked list;

[0014] After receiving the DML operation from the source, the target data synchronization service extracts the MD5 value calculated by the source as the KEY, and then extracts the transaction ID in the operation as the TRXID. It combines the KEY and TRXID into a record and saves it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice.

[0015] Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection and judgment during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict;

[0016] The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

[0017] In some embodiments, the conflict detection judgment specifically includes:

[0018] Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item, and return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next step of conflict detection and judgment;

[0019] Determine whether the memory chip ID in the address recorded in the hash item is less than the ID value of the first memory chip in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment;

[0020] According to the address recorded in the hash item, the corresponding record Y is loaded from the memory chip, and it is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, and record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item, and the record address in the hash item is set to the next address space of the current detection record X, and the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflict TRXID for return; if not, the detected conflict TRXID is searched in the hash table Q to obtain the status of the conflicting transaction, and the conflicting TRXID is returned according to the conflicting transaction status;

[0021] Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record linked list.

[0022] In some embodiments, searching the detected conflicting TRXID in the hash table Q to obtain the state of the conflicting transaction, and returning the conflicting TRXID according to the conflicting transaction state specifically includes:

[0023] If the conflicting transaction is active: -1 is returned as the conflicting TRXID, indicating that it may conflict with the transaction whose committed LSN is smaller than the transaction to which the current operation belongs;

[0024] If the conflicting transaction is in rollback state: ignore the transaction, continue to search for other conflicting transactions in the current hash item, and if a conflicting transaction is located, continue to classify and process it according to the conflicting transaction status; otherwise, use 0 as the conflicting TRXID to return, indicating that no conflict is detected;

[0025] If the conflicting transaction is in the committed state: the conflicting transaction ID is returned as the conflicting TRXID.

[0026] In some embodiments, when the system fails and restarts, restoring the KEY in the hash table R using the last checkpoint file specifically includes:

[0027] Get the most recent checkpoint file F, get the starting LSN of the checkpoint file F as LSN1, and the ending LSN as LSN2; create an array A of size N;

[0028] Read the transaction ID information of the executed stored transaction in the checkpoint file F for filtering during recovery; read the transaction operation OP saved in the checkpoint file F, extract the transaction ID of the operation, and determine whether the transaction ID has been executed. If so, abandon the operation; otherwise, extract the value of the LSN on the operation OP and store it in the array A according to the algorithm A[LSN-LSN1]=OP;

[0029] Extract the operation OP, determine whether the operation is a DML operation, if so, mark the transaction as active in the hash table Q according to the TRXID on the operation; otherwise, determine whether it is a commit operation, if so, mark the transaction as committed in the hash table Q according to the TRXID on the operation, otherwise, mark it as rollback in the hash table Q according to the TRXID on the operation;

[0030] Read the operations in the order of array A, extract the KEY and transaction ID in the operation to restore the state of hash table R before the failure.

[0031] In a second aspect, the present invention further provides a fault recovery device for setting transaction operation dependency, the device comprising:

[0032] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the processor to execute the fault recovery method for setting transaction operations to be lazy as described in the first aspect.

[0033] In a third aspect, the present invention further provides a fault recovery system for setting transaction operation dependency, applying the fault recovery method for setting transaction operation dependency as described in the first aspect, the system includes a checkpoint setting module, a commit LSN judgment module and a fault restart module, wherein:

[0034] The checkpoint setting module is used to determine whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to a preset value N after the target end data synchronization service receives the operation of the source end. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1, and the ending LSN is set to the current operation LSN-1;

[0035] The commit LSN judgment module is used to asynchronously execute and store the transaction after the target end data synchronization service receives the commit operation of the transaction. After the storage is completed, it is judged whether the commit LSN of the transaction commit operation is in the latest checkpoint file, and the judgment is made according to the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file;

[0036] The fault restart module is used to restore the KEY in the hash table R using the last checkpoint file after the system fails and restarts; after the restart is completed, it continues to receive operations from the source end, discards the operations whose received operation LSN is less than or equal to the checkpoint file termination LSN, and extracts the KEY in the operation for conflict detection and judgment if it is greater than the checkpoint file termination LSN.

[0037] In a fourth aspect, the present invention further provides a non-volatile computer storage medium storing computer executable instructions, which are executed by one or more processors to complete the fault recovery method for setting transaction operation dependency as described in the first aspect.

[0038] Compared with the prior art, the present invention provides a fault recovery method, device and system for setting transaction operation dependency, which has the following beneficial effects:

[0039] First, the target data synchronization service sets an interval LSN value of N and stores the received operations in shards with a maximum of N. When the failure is restarted, only the operation in the most recently saved shard needs to be used to restore the KEY information of the hash table R and the transaction status in the hash table Q before the failure, so that the conflict detection of the operations received after the failure and the operations before the failure can be seamlessly connected, thereby ensuring that the failure will not affect the parallel execution capability of transactions before and after the failure.

[0040] Secondly, after the transaction is executed, if the commit LSN of the executed transaction falls within the range of the start LSN and end LSN of the most recent checkpoint file, the transaction ID of the transaction should be appended to the most recent checkpoint file. When the target data synchronization service uses the most recent checkpoint file to restore the hash table R information before the failure after the failure, it can filter out the transactions that have been executed, speeding up the recovery. Filtering out the transactions that have been executed helps to reduce the size of the KEY information in the hash table R after the recovery, thereby improving the performance of operation conflict detection in the first shard received later. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A flowchart of a fault recovery method for setting transaction operation dependency provided in Example 1 of the present invention;

[0043] Figure 2 A flowchart of conflict detection and judgment provided in Example 1 of the present invention;

[0044] Figure 3 A flow chart of the conflicting TRXID write-back of the rollback operation provided in Embodiment 1 of the present invention;

[0045] Figure 4 A flowchart of recovering the KEY of the hash table R provided in Example 1 of the present invention;

[0046] Figure 5 A schematic diagram of a module of a fault recovery system for setting transaction operation dependency provided in Example 1 of the present invention;

[0047] Figure 6 A schematic diagram of the structure of a fault recovery device for setting transaction operation dependency provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0048] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can also be made. These all belong to the protection scope of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. In addition, although the functional module division may be performed in the device schematic diagram, and the logical order may be shown in the flow chart, in some cases, it may be different from the module division in the device, or the steps shown or described in the order in the flow chart may be performed.

[0049] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0050] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0051] In order to make the purpose, technical scheme and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0052] Embodiment 1:

[0053] like Figure 1 As shown, an embodiment of the present invention provides a fault recovery method for setting transaction operation dependency, and the method includes the following steps.

[0054] Step 100: After receiving the operation from the source, the target data synchronization service determines whether the difference between the LSN of the current operation and the LSN value of the last checkpoint is greater than or equal to the preset value N. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1, and the ending LSN is the current operation LSN-1. It should be noted that in this embodiment, LSN is the log sequence number; TRXID is the id that identifies a transaction; DML is the data insertion, update and deletion operation; MD5 is a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value (hash value) to ensure the integrity and consistency of information transmission; KEY is a key that can be used to determine a specific row of records in a table; ROWID is a pseudo column used to uniquely mark rows in a table.

[0055] Step 200: After the target-side data synchronization service receives the transaction commit operation, it executes the transaction asynchronously and stores it in the database. After the storage is completed, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The judgment is made according to the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file.

[0056] Step 300: When the system fails and restarts, the last checkpoint file is used to restore the KEY in the hash table R; after the restart is completed, the source end operation continues to be received, and the received operation LSN is less than or equal to the checkpoint file end LSN. The operation is discarded, and the operation with a greater LSN is extracted to perform conflict detection and judgment.

[0057] The target-side data synchronization service of the embodiment of the present invention needs to periodically save the received operations in the form of checkpoints. When a failure occurs, the conflict detection environment is restored through the operations saved in the checkpoints to ensure the continuity of conflict detection before and after the failure, thereby reducing the impact of the failure on synchronization performance.

[0058] This embodiment will now explain the above method in more detail.

[0059] In one embodiment, a synchronization system needs to be deployed on the source database and the target database. The source database synchronization system reads logs from the source database and parses them into operations. An MD5 value is jointly generated as the KEY of the operation based on the table ID and primary key, or the table ID and unique index information, or the table ID and the custom selected column in the DML operation. The target database synchronization system is responsible for classifying and collecting the synchronization operations sent from the source by transaction, and using the KEY of each received operation to set its dependency with the conflicting transaction received before it.

[0060] It should be noted that when calculating the KEY of an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must be sorted by column ID first, and then the KEY is calculated according to the fixed order of the primary key or unique index combination columns. In addition, the source-side data synchronization service can detect conflicts for each operation and calculate the conflicting MD5 value more accurately. If the target-side data synchronization service calculates it, if the target database table primary key structure or column order is inconsistent with the source, then the MD5 value calculated by describing the primary key information of the target-side table will have errors.

[0061] In one embodiment, after the target-side data synchronization service is started, a hash table R of size N needs to be initialized, and the record address in the hash item is initialized to empty, which is used to store and locate the received operations; and two memory slices of memory space size M are created to store the KEY in the hash table R, and the version number of each memory slice is incremented in the order of creation, and the two memory slices are added to the memory linked list.

[0062] For hash table R, the structure of the HASH item in its HASH table is a 4-byte version number, set to K; a 4-byte memory offset, set to F, K and F constitute a HASH item in the HASH table, as shown in the following table:

[0063] Version number (K) Memory offset (F)

[0064] It means that the first KEY in the HASH item is stored in the memory slice with version number K, and its offset in the memory slice is F.

[0065] In one embodiment, after the target-side data synchronization service is started, a hash table Q for registering transaction information needs to be initialized to save the commit status of all transactions within the LSN range of the received conflict detection. The hash table Q saves the transaction status in the following three types. Active status: The transaction has not received a commit or rollback operation at the time of the current detection. Commit status: The transaction has received a commit operation at the time of the current detection. Rollback status: The transaction has received a rollback operation at the time of the current detection.

[0066] In one embodiment, after the target-side data synchronization service receives the DML operation from the source side, the transaction ID of the extraction operation is registered in the hash table Q, and the transaction status is active.

[0067] In one embodiment, after receiving the operation from the source, the target data synchronization service first determines whether the difference between the LSN of the current operation and the LSN value of the last checkpoint is greater than or equal to N. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the start LSN value of the checkpoint is set to the last checkpoint LSN+1, and the end LSN is set to the current operation LSN-1. It should be noted that the operations saved in the checkpoint file are classified as transactions to help quickly extract the operation data of the transaction when the transaction is executed.

[0068] In one embodiment, after the source-side data synchronization service captures a DML log operation, it obtains the table ID in the log, and obtains the primary key information and unique index of the table in the source-side database through the table ID. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If so, the table ID and the column value of the primary key or unique index are combined to calculate an MD5 value as the KEY of this operation, which is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when the table has both a primary key and a unique index, the table ID and the primary key are preferably selected to jointly calculate the MD5 value as the KEY; when the table has no primary key but has multiple unique indexes, the table ID and any unique index are selected to jointly calculate the MD5 value. When there is no primary key and unique index, after parsing the current DML log operation, the source-side data synchronization service combines the table ID and the ROWID value of the row data or the custom selected column to calculate an MD5 value as the KEY of this operation, which is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when calculating the KEY for a table without a primary key or unique constraint, if it is known that there are no duplicate values ​​when all the columns of the table are combined, the ROWID of the table can be directly used to calculate the KEY, which can effectively reduce the cost of calculating the KEY; if the table has duplicate rows, it is necessary to use the custom column selection method to calculate the KEY. You can choose which columns or all column values ​​to use to calculate the KEY based on the distribution of column values. In this way, the target-side synchronization service can use the KEY to prevent deadlocks caused by concurrently executed synchronization transactions operating on rows with duplicate values ​​at the same time.

[0069] In one embodiment, after the target-side data synchronization service receives the DML operation from the source, it extracts the MD5 value calculated by the source in the operation as the KEY, and then extracts the transaction ID in the operation as the TRXID, and combines the KEY and TRXID into a record and saves it to the offset F of the memory slice with the current version number K. If the record is the first record of the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice, which is also the conflicting LSN of the operation in the memory slice.

[0070] Since the hash record needs to save the address of the next record, the record saved in the memory chip also needs to reserve the address of the next record, as shown in the following table.

[0071] KEY TRXID Next record memory chip version (K) Next record memory offset (F)

[0072] In one embodiment, the detection record X is constructed using the KEY of the current operation, and the detection record X is merged into the hash table R. During the merging process, conflict detection and judgment are required. If there is a conflicting KEY, the TRXID value corresponding to the conflicting KEY is returned; otherwise, 0 is returned as the conflicting TRXID, indicating no conflict. For details, refer to Figure 2 As shown, the conflict detection and judgment specifically includes the following steps.

[0073] Step 101: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value.

[0074] Step 102: Determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, fill the address of the detection record X back into the hash item, and return 0 as the conflict TRXID to indicate that no conflict is detected; if not, proceed to the next step of conflict detection.

[0075] Step 103: Determine whether the memory slice version number K in the address recorded in the hash item is less than the version number K value of the first memory slice in the memory linked list. If so, it indicates that the record Y has exceeded the inspection range. Set the record address in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection.

[0076] Step 104: Load the corresponding record Y from the memory chip according to the address recorded in the hash item (find the memory chip with the corresponding version number through the recorded version number, and then locate the specific memory address through the offset), and determine whether the KEY of record Y is equal to the KEY of detection record X and the TRXID is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, remove record Y from the record linked list of the hash item to shorten the length of the linked list in the hash item, set the record address in the hash item to the next address space of the current detection record X, fill the address of the detection record X back into the hash item, and return the TRXID stored in record Y as the conflicting TRXID; if not, proceed to the next step of conflict detection.

[0077] Step 105: Search the detected conflicting TRXID in the hash table Q to obtain the status of the conflicting transaction, and return the conflicting TRXID according to the conflicting transaction status. Among them, when the conflicting transaction is in the active state: return -1, indicating that it may conflict with the transaction whose commit LSN is smaller than the transaction to which the current operation belongs. It should be noted that when the conflicting transaction state detected by the operation conflict is the active transaction state, it means that there are duplicate data in the data row of the source database. At this time, the final state of the dependent active transaction is unknown, which may be a rollback transaction or a commit transaction. Then the execution order of the current operation cannot be clear when it is executed, so the operation execution needs to wait for the transaction whose commit LSN is smaller than the transaction to which the current operation belongs to be completed before it can start to execute. When the conflicting transaction is in the rollback state: ignore the transaction, continue to search for other conflicting transactions in the current hash item, if the conflicting transaction is located, continue to classify and process according to the conflicting transaction state; otherwise, use 0 as the conflicting TRXID to return, indicating that no conflict is detected, continue to traverse the next record of record Y in sequence, and perform conflict detection in the above manner until the end of the record linked list. When the conflicting transaction is in the committed state: the conflicting transaction ID is returned as the conflicting TRXID.

[0078] Based on the above process, the next record of record Y is traversed in turn, and conflict detection is performed in the above manner until the record list ends. The condition for the end of the list is: the version number in the next record address of the record is less than the version number of the first memory slice in the current memory list or the version number in the next record address is empty.

[0079] In one embodiment, the TRXID obtained by the conflict check is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN, thereby obtaining the conflict information of this operation.

[0080] In one embodiment, the target end data synchronization service receives the next message and saves it to the memory slice K in the above manner. There are two memory slices in the memory linked list. When the first memory slice is full (the written data exceeds the set size M), it switches to the second memory slice. When the second memory slice is full, the first memory slice in the linked list needs to be eliminated first, and then a new memory slice is created with an increasing version number K and added to the linked list. It should be noted that when saving a record to a new memory slice, if it is the first record in the memory slice, the operation LSN corresponding to the record should be marked as the starting LSN of the memory slice.

[0081] In one embodiment, the target-side data synchronization service receives a commit operation, adds the transaction to the queue of pending transactions, and modifies the transaction status in the hash table Q from the active status to the committed status. It should be noted that since only committed transactions will be executed and stored, when the operation conflict is detected, the transaction that the operation depends on must also be committed. Therefore, after receiving the transaction commit operation, it is necessary to clearly indicate in the hash table Q that the transaction status is the committed status.

[0082] In one embodiment, the target-side data synchronization service receives a rollback operation, discards the transaction, and modifies the transaction status in the hash table Q from the active status to the rollback status. It should be noted that when the received transaction is rolled back, since the rolled back transaction operation will not be executed when executing the storage, then the submitted transaction operation is not allowed to rely on a rolled back transaction, otherwise it will not be able to find the dependent conflicting transaction during execution and it will be considered that there is no conflict, causing the subsequent transaction to be executed in advance, causing an error. For example, there are three transactions A, B and C that all modify the same row of the same table, but transaction B is rolled back. When the target-side data synchronization service receives the operation, the operation conflict detection process will form a dependency order relationship in which B depends on A and C depends on B. Transaction B is directly discarded and not executed because it is rolled back, so C will not be able to find the dependent transaction B during execution and will be executed in advance. Finally, transaction A and transaction C will be executed concurrently, resulting in an error. Therefore, after receiving a rollback operation, the transaction needs to clearly mark the transaction status as rollback in the hash table Q. When a subsequent operation is found to conflict with the rollback transaction during conflict detection, the rollback transaction must be ignored and the search for the conflicting transaction with the committed status must continue.

[0083] In one embodiment, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to write the conflicting TRXID saved in the rolled-back operation back to the conflict detection HASH table, that is, the hash table R. It should be noted that since the transaction dependency information of the rolled-back transaction and the rolled-back operation is restored, the dependency relationship of each transaction received in sequence is clear. These transactions can be committed immediately when they are committed, without waiting for the transaction to be committed with a smaller LSN than their own, thereby further improving the synchronization performance. For the write-back of the conflicting TRXID of the rollback operation, refer to Figure 3 As shown, the specific steps include the following steps.

[0084] Step 201: Locate the transaction that needs to be rolled back by using the transaction ID of the rollback or partial rollback operation, extract the number of rows that need to be rolled back in the partial rollback operation, or extract all rows involved in the transaction in the rollback operation.

[0085] Step 202: Extract the rollback operation from the tail of the transaction operation linked list, and extract the KEY value, conflicting TRXID and conflicting LSN saved in the operation.

[0086] Step 203: Compare the conflicting LSN of the operation with the starting LSN of the first memory slice. If the conflicting LSN is smaller than the starting LSN, terminate the write-back of the partial rollback operation to the hash table R; otherwise, jump to the next step.

[0087] Step 204: Search for conflicting operation records with the same KEY in the HASH table according to the operation KEY. If not found, terminate the writeback of this part of the rollback operation to the hash table R; otherwise, write back the conflicting TRXID to the record in the hash table R, and continue to extract the previous record to be rolled back for the same writeback until the specified number of rows is rolled back.

[0088] By writing back the conflicting TRXID information in the rollback operation, the continuity problem of operation conflict detection can be solved. These rolled back operations can be directly discarded. Otherwise, when the transaction is executed, although the partially rolled back operations do not need to be executed, conflict detection is still required during execution. Otherwise, other transaction operations that depend on it will not be able to find the dependent conflicting transactions and will be executed in advance, resulting in inconsistent data.

[0089] In one embodiment, after receiving the transaction commit operation, the target-side data synchronization service asynchronously executes the transaction into the warehouse. After the completion of the storage, it determines whether the LSN of the transaction commit operation is in the most recent checkpoint file, and makes a judgment according to the condition that the checkpoint file start LSN <= commit LSN <= checkpoint file end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file. It should be noted that during fault recovery, the target-side data synchronization service will only restore the most recent checkpoint file, and record the transactions that have been completed in the warehouse within the scope of the checkpoint file, which can speed up the fault recovery and reduce the size of the conflict detection KEY after the fault recovery.

[0090] In one embodiment, when the system fails and restarts, the last checkpoint file needs to be used to restore the KEY in the conflict detection container, that is, to restore the KEY in the hash table R. For details, refer to Figure 4 As shown, the process of recovering the KEY of the hash table R specifically includes the following steps.

[0091] Step 301: Get the most recent checkpoint file F, get the starting LSN of the checkpoint file F as LSN1, and the ending LSN as LSN2; create an array A of size N.

[0092] Step 302: Read the transaction ID information of the executed stored transaction in the checkpoint file F for filtering during recovery (the transaction that has been executed has lost its meaning of recovery after the failure); read the transaction operation OP stored in the checkpoint file F, extract the transaction ID of the operation, and determine whether the transaction ID has been executed. If so, abandon the operation; otherwise, extract the value of the LSN on the operation OP and store it in the array A according to the algorithm A[LSN-LSN1]=OP. It should be noted that OP refers to the operator, which is used to perform a specific operation. It should also be noted that since the operations stored in the checkpoint file F are classified by transaction, the operations read from the checkpoint file F are also read by transaction classification, and the operation LSNs between transactions are discontinuous, so the transactions recovered from the checkpoint need to be sorted by the LSN of the transaction operations. Since the transaction operations stored in the checkpoint file F have been limited to a limited LSN interval (size N), the array can be used to sort the operations in the checkpoint file.

[0093] Step 303: Extract the operation OP and determine whether the operation is a DML operation. If so, mark the transaction as active in the hash table Q according to the TRXID on the operation. Otherwise, determine whether it is a commit operation. If so, mark the transaction as committed in the hash table Q according to the TRXID on the operation. Otherwise, mark the transaction as rollback in the hash table Q according to the TRXID on the operation.

[0094] Step 304: Read the operations in the order of array A, extract the KEY and transaction ID in the operation to restore the state of hash table R before the failure.

[0095] In one embodiment, after the restart is completed, the source end continues to receive operations, and the received operation LSN is less than or equal to the recovery checkpoint file termination LSN. The received operation LSN is discarded, and the operation LSN greater than the recovery checkpoint file LSN is extracted according to the above method to perform conflict detection.

[0096] The above is the fault recovery method for setting transaction operation dependency. The following describes the workflow of the target-side data synchronization service after setting transaction operation dependency.

[0097] In one embodiment, the target-side data synchronization service creates H working threads, executes transactions in order of the size of the submitted LSN of the received transactions, and stores the transaction operations. It should be noted that the target-side data synchronization service also needs to create a hash table P to register the transaction information (transaction ID) currently being executed, and create a hash table G to register the table information of the current transaction and the start LSN of the transaction.

[0098] In one embodiment, the working thread extracts a transaction from the committed transaction linked list according to the transaction commit order, and registers the transaction ID into the hash table P, and registers the table information involved in the transaction and the start LSN of the transaction into the hash table G.

[0099] In one embodiment, the worker thread extracts an operation of the transaction and determines whether the current operation is a commit operation. If so, the worker thread that is waiting for itself wakes up after committing the current transaction, removes the registration information of the current transaction from the hash table P and the hash table G, and completes the execution of the current transaction; otherwise, the conflicting TRXID and conflicting LSN saved in the operation are obtained. It should be noted that since the transaction that each operation on the transaction depends on has been determined in the conflict detection, the transaction can be directly committed after executing all operations (indicating that the transactions it depends on have been committed), without waiting for the transaction submitted before itself to be committed first, thereby effectively improving the synchronization performance.

[0100] In one embodiment, the working thread uses the conflicting TRXID of the current operation to be executed to search whether there is a transaction with the conflicting TRXID in the hash table P. If there is, the current execution process is suspended, and the conflicting transaction is woken up after being executed and submitted; otherwise, it jumps to the next step. It should be noted that the existence of a transaction with the conflicting TRXID in the hash table P indicates that the data involved in the current operation and the transaction with the conflicting TRXID are in conflict, and the current operation cannot be executed in parallel with the conflicting transaction, and the conflicting transaction needs to be submitted before it can be executed.

[0101] In one embodiment, the working thread uses the information of the current pending operation table to search in the hash table G whether there is a transaction operating the same table, and the transaction commit LSN is smaller than the current transaction, and the transaction start LSN is smaller than the conflict LSN of the current operation. If there is, the current execution process is suspended, and the conflicting transaction is awakened after it is executed and committed; otherwise, it is determined that the conflict detection of the current pending operation of the working thread does not detect a conflict, then the current operation is executed, and then the next operation is continued. It should be noted that this step is mainly to check whether there is a transaction beyond the conflict check LSN range in the current concurrently executed transaction, because the conflict transaction information on the pending operation has a valid interval range, that is, the conflict transaction information is only valid within the interval from the conflict LSN saved in the operation to the current operation LSN, and the operation does not perform conflict detection on the operation before the conflict LSN. That is to say, when there is a transaction whose commit LSN is smaller than the commit LSN of the transaction to which the current pending operation belongs, and the transaction start LSN is smaller than the conflict LSN saved by the current pending operation, it is unknown whether the operation on this transaction conflicts with the current pending operation, so the current pending operation cannot be executed rashly, and it needs to wait until the transaction is committed before it can be executed.

[0102] Based on the above process, this embodiment mainly performs conflict detection on operations within a fixed LSN interval, and saves the detection results and the LSN interval to the corresponding operation. When the working thread executes the operation, it can use the conflict transaction information saved on the operation and its detection LSN interval to determine whether there is a conflict in the current concurrently executed transaction, thereby improving the parallel execution capability.

[0103] refer to Figure 5 As shown, this embodiment also provides a fault recovery system for setting transaction operation dependency, which applies the aforementioned fault recovery method for setting transaction operation dependency, and the system includes a checkpoint setting module, a commit LSN judgment module and a fault restart module, wherein: the checkpoint setting module is used to determine whether the difference between the LSN value of the current operation and the LSN value of the last checkpoint is greater than or equal to a preset value N after the target end data synchronization service receives the operation of the source end, and if so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1, and the ending LSN is set to the LSN-1 of the current operation; the commit LSN judgment module The fault module is used to asynchronously execute and store the transaction after the target end data synchronization service receives the transaction commit operation. After the storage is completed, it is determined whether the commit LSN of the transaction commit operation is in the latest checkpoint file, and the condition of the checkpoint file's start LSN <= commit LSN <= end LSN is used for judgment. If it is satisfied, the transaction ID is appended to the checkpoint file; the fault restart module is used to restore the KEY in the hash table R using the last checkpoint file after the system fault restarts; after the restart is completed, the source end operation is continued to be received, and the operation whose received operation LSN is less than or equal to the checkpoint file end LSN is discarded, and if it is greater than the end LSN, the KEY in the operation is extracted for conflict detection and judgment. The specific functions and principles of the system function module correspond to the specific methods in the method part, and will not be repeated here.

[0104] In summary, this embodiment proposes to provide a fault recovery method and system for setting transaction operation dependency, including the following advantages: First, the target-side data synchronization service sets an interval LSN value of N, and stores the received operations in shards according to the maximum number of N. When the fault is restarted, only the operation in the most recently saved shard needs to be used to restore the KEY information of the hash table R before the fault and the transaction status in the hash table Q, so that the conflict detection of the operations received after the fault and the operations before the fault can be seamlessly connected, thereby ensuring that the fault will not affect the parallel execution capability of transactions before and after the fault.

[0105] Secondly, after the transaction is executed, if the commit LSN of the executed transaction falls within the range of the start LSN and end LSN of the most recent checkpoint file, the transaction ID of the transaction should be appended to the most recent checkpoint file. When the target data synchronization service uses the most recent checkpoint file to restore the hash table R information before the failure after the failure, it can filter out the transactions that have been executed, speeding up the recovery. Filtering out the transactions that have been executed helps to reduce the size of the KEY information in the hash table R after the recovery, thereby improving the performance of operation conflict detection in the first shard received later.

[0106] Embodiment 2:

[0107] Based on the fault recovery method for setting transaction operation dependency provided in the above-mentioned embodiment 1, embodiment 2 of the present invention provides a specific example for illustration.

[0108] For example: The source database has table T (ID INT PRIMARY KEY, C1 INT);

[0109] The source application has three transactions that perform the following operations on table T in the following order:

[0110] TRX1:INSERT INTO T(ID,C1)VALUES(1,1);

[0111] TRX1:COMMIT;

[0112] TRX2:INSERT INTO T(ID,C1)VALUES(2,2);

[0113] TRX2:UPDATE T SET C1=1WHERE ID=1;

[0114] TRX2:COMMIT;

[0115] TRX3:INSERT INTO T(ID, C1)VALUES(3, 3);

[0116] TRX3:COMMIT;

[0117] The order of the above operations represents the order of these operations in the log stream, and the resulting log stream will form the following numbered table:

[0118]

[0119]

[0120] The receiving process is as follows:

[0121] 1. The target data synchronization service sets the preset interval value N of the checkpoint LSN to 5, and creates a hash table R that can accommodate 5*2=10 KEY information. Since it is the first startup, the LSN of the previous checkpoint is set to 0.

[0122] 2. The log receiving thread receives an operation with LSN 1. The difference between LSN 1 and the previous checkpoint LSN 0 is less than 5. Its KEY and TRXID are extracted and stored in the hash table R for conflict detection. The conflict detection result is no conflict.

[0123] 3. The log receiving thread receives an operation with LSN 3. The difference between LSN 3 and the previous checkpoint LSN 0 is less than 5. Its KEY and TRXID are extracted and stored in the hash table R for conflict detection. The conflict detection result is no conflict.

[0124] 4. The log receiving thread receives the operation with LSN 4. The difference between LSN 4 and the previous checkpoint LSN 0 is less than 5. Its KEY and TRXID are extracted and stored in hash table R for conflict detection. The conflict detection result shows that there is a conflict with the transaction with TRXID 1. The conflicting TRXID is written back to the operation.

[0125] 5. The log receiving thread receives a commit operation with LSN 5. If the difference between LSN 5 and the previous checkpoint LSN 0 is greater than or equal to 5, the currently received sharding operation needs to be saved in the checkpoint file F, and the starting LSN of the checkpoint file is marked as 1 (the ending LSN 0+1 of the previous checkpoint) and the ending LSN is 4 (LSN 5-1 of the current operation).

[0126] 6. When the target-side data synchronization service receives a commit operation with LSN 2 and TRXID 1, it will asynchronously execute the transaction. After the transaction is executed, since the commit LSN of the transaction falls within the range of the start LSN and end LSN of the most recent checkpoint file, the TRXID of the transaction will be appended to the most recent checkpoint file F.

[0127] 7. Restart the target data synchronization service and use the last checkpoint file to restore the KEY in the conflict detection container.

[0128] 8. Get the most recent checkpoint file F, whose starting LSN is 1 and ending LSN is 4.

[0129] 9. Create an array A of size 5.

[0130] 10. Read the completed transaction information in the checkpoint file F, including a transaction with transaction ID 1.

[0131] 11. Read the operations in the checkpoint file F by transaction classification. The transaction with TRXID 1 is discarded. The transaction with TRXID 2 and the two operations with LSNs 3 and 4 are subtracted from the checkpoint file start LSN 1 by the operation LSN and stored in array A at positions A[2] and A[3].

[0132] 12. Sequentially read the KEY information of the operations in array A and store it in hash table R to restore the state of hash table R before the failure.

[0133] 13. After the fault recovery is completed, the recovery checkpoint file is received again to terminate the operation after LSN.

[0134] 14. The log receiving thread receives an operation with LSN 6. The difference between LSN 6 and the previous checkpoint LSN 5 is less than 5. Its KEY and TRXID are extracted and stored in hash table R for conflict detection. The conflict detection result is no conflict. In this way, this transaction can be executed in parallel with the transaction TRX2 before the failure. Otherwise, it must wait until the execution of TRX2 before the failure is completed before it can start execution, because it is impossible to confirm whether there is a conflict between this operation and the transaction operation before the failure.

[0135] From the above conflict detection fault recovery process, it can be seen that the operations of TRX1 and TRX2 are saved in the checkpoint file F of the first shard. Since TRX1 has received the commit operation, it will be executed asynchronously. When it is completed, its commit LSN will fall within the start LSN and end LSN range of the most recent checkpoint file F, and the transaction information of TRX1 will be appended to the checkpoint file F. When the fault is recovered, the operation of TRX1 will be abandoned, and the transaction operation of TRX2 will be sorted on array A using LSN. The correctness and integrity of the conflict detection can be ensured by restoring the order of receiving the operations in the checkpoint file. Due to the filtered and executed transaction TRX1, the hash table R after the fault recovery will have one less KEY information than the hash table R before the fault. The hash table R with reduced KEY information will have better performance in conflict detection. At the same time, TRX3 received after the fault can also be executed concurrently with the transaction TRX2 before the fault, and the parallelism will not be reduced due to the fault.

[0136] Embodiment 3:

[0137] It should be noted that for rollback transactions, since the rollback transactions will not be executed, there are three ways to process the rollback transaction operations in the hash table R.

[0138] The first method: do not roll back the impact of the rollback transaction operation on the operation dependency in the hash table R, and do not collect the conflicting TRXIDs of the rollback transaction operation dependency. Directly release all the operations of the rollback transaction. This rollback transaction will be dependent on other operations. The rollback transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependency. When this solution is adopted, the operations of the rollback transaction are directly released as a whole, which can improve the processing performance of the rollback transaction. However, the operations that depend on the rollback transaction must wait until the rollback transaction is released in the pending execution list before they can be executed. Since the conflicting transactions of the operation dependency were not collected before the release of the rollback transaction, it is necessary to wait until all transactions whose commit LSN or rollback LSN is less than its own rollback LSN are executed before they can be released, which will affect the synchronization efficiency. This method corresponds to that when the target-side data synchronization service receives the rollback transaction, all operations on this transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the pending execution queue. For example, when the target-side data synchronization service receives a rollback or partial rollback operation, it needs to locate the transaction based on the transaction ID in the operation, and then mark the transaction to indicate that some operations in the transaction are rolled back. If the operation is a rollback operation, then all operations in the transaction do not need to be executed and are directly discarded. Only the retained transaction information is added to the queue to be executed. The transaction information of the rolled back transaction cannot be discarded.

[0139] The second method: do not roll back the impact of the rollback transaction operation on the operation dependency in the hash table R, but collect the conflicting TRXIDs of the rollback transaction operation dependency, for example, record it in the conflicting TRXID dependency list L. This rollback transaction will be relied upon by other operations, and the rollback transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependency. When this solution is adopted, the conflicting TRXIDs in the operation must be collected before the rollback transaction operation is released. Although the processing performance of the rollback transaction is reduced, the release performance of the rollback transaction will be improved during execution. The operation that depends on the rollback transaction must wait until the rollback transaction is released in the pending execution list before it can be executed. Since the conflicting transactions of the operation dependency were collected before the release operation, the rollback transaction can be released only after these conflicting transactions are executed, without waiting for all transactions whose commit LSN or rollback LSN is less than its own rollback LSN to be executed. Compared with the first solution, this will partially improve the efficiency of synchronization. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, it collects the conflicting TRXIDs saved in the rolled-back operation, and for the rollback operation, the corresponding transaction cannot be discarded, but is added to the queue of pending transactions.

[0140] The third method: roll back the impact of the rolled-back transaction on the operation dependencies in hash table R. Then the rolled-back transaction will not be dependent on other operations, and the rolled-back transaction does not need to be added to the pending execution list. When this solution is adopted, after receiving the rollback operation, the transaction needs to roll back the impact of all operations in the transaction on the operation dependencies in hash table R. Since the transaction scale is uncontrollable, when a large transaction is rolled back, the rollback action will affect the progress of receiving the operation, thereby affecting the efficiency of synchronization. This method corresponds to that when the target-side data synchronization service receives a rollback or partial rollback operation, the conflicting TRXID saved in the rolled-back operation is written back to hash table R.

[0141] The above three solutions for rolling back transactions have their own advantages and disadvantages. The first solution is suitable for large-scale transaction rollback, the second solution is suitable for medium-scale transaction rollback, and the third solution is suitable for small-scale transaction rollback. Therefore, two parameters W1 and W2 are set. W1 represents the minimum number of DML operations in large-scale transactions, and W2 represents the minimum number of DML operations in medium-scale transactions. Based on this, after the target-side data synchronization service receives the DML operation from the source, it is also necessary to manage the received DML operations by transaction classification, and count the scale S of DML operations received by each transaction (S is the number of DML operations received in a single transaction). When the target-side data synchronization service receives a rollback operation, it takes corresponding actions according to the scale S of DML operations in the transaction.

[0142] When the scale S of the rolled-back transaction is greater than or equal to W1, it corresponds to the large-scale transaction rollback in the first solution. In this case, the impact of the rolled-back transaction operation on the operation dependency in the hash table R is not rolled back, and the conflicting TRXIDs of the rolled-back transaction operation dependency are not collected. All operations of the rolled-back transaction are directly released. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution list for execution to ensure the continuity of the operation dependency.

[0143] When the scale S of the rolled-back transaction is greater than or equal to W2 and less than W1, it corresponds to the medium-scale transaction rollback in the second solution. In this case, the impact of the rolled-back transaction operation on the operation dependency in the hash table R is not rolled back, but the conflicting TRXIDs of the rolled-back transaction operation dependency are collected, and it can be selected to be recorded in the conflicting TRXID dependency linked list L. This rolled-back transaction will be dependent on other operations, and the rolled-back transaction must also be added to the pending execution linked list for execution to ensure the continuity of the operation dependency.

[0144] When the scale S of the rolled-back transaction is less than W2, it corresponds to the small-scale transaction rollback in the third solution. In this case, the impact of the rolled-back transaction operation on the operation dependency in the hash table R is rolled back, and the rolled-back transaction does not need to be added to the pending linked list and is directly released.

[0145] Furthermore, for partial rollback operations, due to the particularity of partial rollback, partial rollback itself needs to traverse the operations to be rolled back, so adding the process of collecting conflicting TRXIDs on these operations will not significantly reduce the performance of processing partial rollbacks. Based on this, when the target-side data synchronization service receives a partial rollback operation, it is necessary to count the total number of rolled back rows S1, and perform the following processing according to the size of S1: When the number of rollback operation rows S1 is less than W2, the impact of the rollback operation on the operation dependency in the hash table R is rolled back, that is, the conflicting TRXIDs saved in the rolled back operation are written back to the hash table R. When the number of rollback operation rows S1 is greater than W2, the impact of the rollback transaction operation on the operation dependency in the hash table R is not rolled back, that is, the conflicting TRXIDs saved in the rolled back operation are not written back to the hash table R, but the conflicting TRXIDs of the rollback transaction operation dependency are collected, and the conflicting TRXIDs can be recorded in the conflicting TRXID dependency linked list L.

[0146] Based on the above settings, after the target-side data synchronization service creates a working thread, the working thread extracts a transaction from the committed transaction list in the order in which the transaction is committed, and determines whether the transaction is a rollback transaction. If the current transaction is a rollback transaction, it is necessary to determine whether the rollback transaction has a conflicting TRXID linked list L.

[0147] If the rolled back transaction does not have a conflicting TRXID linked list L, it is necessary to wait for all transactions with a smaller LSN than the current transaction to be committed before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. This is because the operations on the rolled back transaction do not need to be executed and are discarded, so the transaction information they depend on is also unclear. Therefore, when submitting, it is necessary to wait until all transactions submitted earlier than itself are submitted to ensure the correctness of the transaction dependency order.

[0148] If the rolled back transaction has a conflicting TRXID linked list L, it is necessary to wait for all conflicting transactions on the conflicting transaction linked list L to commit before waking up the waiting worker thread, and then remove the registration information of the current transaction from the hash table P and hash table G to complete the execution of the current transaction. Specifically, extract a conflicting TRXID from the current transaction conflicting transaction linked list L in turn, and check whether it exists in the hash table P. If it exists, it is necessary to wait for the transaction to be completed before waking itself up. After waking up, the next conflicting TRXID is extracted from the conflicting transaction linked list L and the same check is performed until all conflicting transactions in the conflicting transaction linked list L are completed. This is because the operations on the rolled back transaction do not need to be executed and are discarded. When the rollback operation is discarded, the conflicting TRXIDs on the operation are collected, so the transaction information they depend on is very clear. Then, when releasing, you only need to wait for these conflicting transactions to be committed to ensure the correctness of the transaction dependency order.

[0149] In summary, since the rollback transaction will not be executed, it will generate three schemes during conflict detection. Each scheme has its own advantages, and appropriate values ​​need to be set to give full play to the advantages of the three schemes, so as to achieve the purpose of improving the overall synchronization performance. In addition, for the partial rollback operation of the committed transaction, the conflicting TRXIDs referenced by them are collected in a targeted manner during the rollback operation, which can effectively reduce the performance loss caused by the partial rollback operation writing back the conflict detection hash table R, so as to achieve the purpose of improving the overall synchronization performance. This embodiment processes the rollback transaction and partial rollback operation in a targeted manner according to the agreed operation scale, and finds the balance point between the cost of operation rollback and rollback transaction execution, thereby improving the overall synchronization performance.

[0150] In addition, in some embodiments, the KEY in the hash table R can be stored in a file or in a memory chip. Specifically, the difference between the two is that when using files for storage, although the storage performance is low and affects the efficiency of conflict detection, after a data synchronization failure, the state of the hash table R between failures can be restored by storing the KEY value in the file, so that when performing conflict detection on the received operations after the failure, there is continuity with the failure, and the synchronization performance will not drop drastically due to the failure; when using memory chips for storage, the performance of conflict detection will be greatly improved compared to the file storage solution, but the data of the previous conflict detection will be lost after the failure, and the recovery detection can only start from the currently received LSN, which will cause a discontinuity between the subsequent conflict detection and the failure, and the synchronization performance will drop drastically at the failure location during synchronization execution. When the storage performance is high (for example, using SSD fixed storage), the performance requirements of conflict detection can be met, and the efficiency of operation reception can be balanced with synchronous execution. Then, the method of storing the KEY in the hash table R through files can effectively avoid the loss of synchronization performance caused by failures; when the storage performance affects the efficiency of conflict detection, the bottleneck of synchronization exists in the operation reception. At this time, the method of storing the KEY in the hash table R through a memory chip can effectively improve the reception performance of the operation, thereby improving the overall synchronization performance. Embodiment 1 of the present application is an example of a method of storing the KEY in the hash table R through memory.

[0151] In some embodiments, you can choose to calculate the MD5 value at the source or target end as the KEY of the operation. The data synchronization service at the source end detects conflicts for each operation and can calculate the MD5 value of the conflict more accurately. If the primary key structure or column order of the target database table is inconsistent with the source end, the MD5 value calculated by the primary key information of the target end table will have errors. Based on this, it is generally chosen to calculate the MD5 value at the source end as the KEY of the operation. After the target end data synchronization service receives the DML operation at the source end, it can directly extract the MD5 value calculated by the source end as the KEY.

[0152] In some embodiments, when calculating the KEY of an operation, since the order of columns in the operation parsed from the log is not fixed, the columns in the operation must first be sorted by column ID, and then the KEY is calculated according to the fixed order of the primary key or unique index combination columns. After the source-side data synchronization service captures a DML log operation, it obtains the table ID in the log, and obtains the primary key information and unique index of the table in the source-side database through the table ID. After parsing the current DML log operation, the source-side data synchronization service determines whether the table has a primary key or unique index. If there is a primary key or unique index, the table ID and the column value of the primary key or unique index are combined to calculate an MD5 value as the KEY of this operation, which is stored in the current DML log operation and sent to the target-side synchronization service. It should be noted that when a table has both a primary key and a unique index, the table ID and the primary key are preferably selected to jointly calculate the MD5 value as the KEY; when a table has no primary key but has multiple unique indexes, the table ID and any unique index are selected to jointly calculate the MD5 value. If there is no primary key or unique index, after parsing the current DML log operation, the source-side data synchronization service combines the table ID and the row data's ROWID value or custom selected column to calculate an MD5 value as the KEY for this operation, stores it in the current DML log operation, and sends it to the target-side synchronization service. It should be noted that when calculating the KEY for a table without a primary key or unique constraint, if it is clear that all columns of the table have no duplicate values ​​when combined, the table's ROWID can be used directly to calculate the KEY, which can effectively reduce the cost of calculating the KEY; if the table has duplicate rows, then it is necessary to use custom selected columns to calculate the KEY, and which columns or all column values ​​to use to calculate the KEY can be selected based on the distribution of column values, so that the target-side synchronization service can use the KEY to prevent deadlocks caused by concurrently executed synchronization transactions operating rows with duplicate values ​​at the same time.

[0153] Embodiment 4:

[0154] Based on the fault recovery method for setting transaction operation dependency provided in the above embodiment 1, the present invention also provides a fault recovery device for setting transaction operation dependency that can be used to implement the above method and system, such as Figure 6 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The fault recovery device for setting transaction operation dependency in this embodiment includes one or more processors 21 and a memory 22. Figure 6 A processor 21 is taken as an example.

[0155] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0156] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the fault recovery method for setting transaction operation dependency in Example 1. The processor 21 executes various functional applications and data processing of the fault recovery device for setting transaction operation dependency by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, implements the fault recovery method for setting transaction operation dependency in Example 1.

[0157] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0158] The program instructions / modules are stored in the memory 22. When executed by one or more processors 21, the fault recovery method for setting transaction operation dependency in the above embodiment 1 is executed. For example, the above described Figure 1-Figure 4 The steps shown.

[0159] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0160] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0161] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault recovery method for setting transaction operation dependency, characterized in that: include: After receiving the operation from the source, the target data synchronization service determines whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to the preset value N. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1, and the ending LSN is set to the current operation LSN-1; After receiving the transaction commit operation, the target-side data synchronization service asynchronously executes the transaction into the database. After the storage is completed, it determines whether the commit LSN of the transaction commit operation is in the most recent checkpoint file. The judgment is made according to the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file. When the system fails and restarts, the last checkpoint file is used to restore the KEY in the hash table R. After the restart is completed, the source end's operations will continue to be received. The received operation LSN will be discarded if it is less than or equal to the checkpoint file's termination LSN. If it is greater, the KEY in the operation will be extracted for conflict detection and judgment.

2. The method for failure recovery of setting transaction operation dependency according to claim 1, characterized in that: After the target-side data synchronization service is started, a hash table Q for registering transaction information is initialized to save the status of all transactions within the scope of the conflict detection LSN, including active status, committed status, and rollback status. After the target-side data synchronization service receives the DML operation from the source, the transaction ID of the extraction operation is registered in the hash table Q, and the transaction status is initially active.

3. The fault recovery method for setting transaction operation dependency according to claim 2, characterized in that: When the target-side data synchronization service receives a commit operation, it adds the transaction to the queue of pending transactions and modifies the transaction status in the hash table Q from the active status to the committed status; when the target-side data synchronization service receives a rollback operation, it discards the transaction and modifies the transaction status in the hash table Q from the active status to the rollback status, and writes the conflicting TRXID saved in the rolled back operation back to the hash table R.

4. The method for failure recovery of setting transaction operation dependency according to claim 1, characterized in that: After the target data synchronization service is started, a hash table R is initialized to store and locate the received operations, and two memory slices are created to store the KEY in the hash table R. The version number of each memory slice is incremented in the order of creation, and the two memory slices are added to the memory linked list; After receiving the DML operation from the source, the target data synchronization service extracts the MD5 value calculated by the source as the KEY, and then extracts the transaction ID in the operation as the TRXID. It combines the KEY and TRXID into a record and saves it to the current memory slice. If the record is the first record in the current memory slice, the log LSN corresponding to the operation is marked on the memory slice as the starting LSN of the memory slice. Use the current operation's KEY to construct a detection record X, merge the detection record X into the hash table R, and perform conflict detection and judgment during the merging process. If there is a conflicting KEY, return the TRXID value corresponding to the conflicting KEY; otherwise, return 0 as the corresponding TRXID, indicating no conflict; The TRXID obtained by the conflict detection is set as the conflict TRXID, and the starting LSN of the first memory slice in the memory linked list corresponding to the current hash table R is written into the current operation and set as the conflict LSN.

5. The method for failure recovery of setting transaction operation dependency according to claim 4, characterized in that: The conflict detection and judgment specifically includes: Calculate the hash value of the KEY of the current detection record X, and locate the hash item through the hash value; determine whether the record address in the hash item is empty. If so, set the record address in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item, and return 0 as the conflict TRXID, indicating that no conflict is detected; if not, proceed to the next step of conflict detection and judgment; Determine whether the memory chip ID in the address recorded in the hash item is less than the ID value of the first memory chip in the memory linked list. If so, it indicates that this record Y has exceeded the inspection range. Set the address recorded in the hash item to the next address space of the current detection record X, and fill the address of the detection record X back into the hash item. Use 0 as the conflict TRXID to return, indicating that no conflict is detected; if not, proceed to the next step of conflict detection judgment; According to the address recorded in the hash item, the corresponding record Y is loaded from the memory chip, and it is determined whether the KEY of record Y is equal to the KEY of the current detection record X and the TRXID of record Y is not equal to the TRXID of the current operation. If so, it means that there is a conflict in the operation, and record Y is removed from the record linked list of the hash item to shorten the length of the linked list in the hash item, and the record address in the hash item is set to the next address space of the current detection record X, and the address of the current detection record X is backfilled into the hash item, and the TRXID stored in record Y is used as the conflict TRXID for return; if not, the detected conflict TRXID is searched in the hash table Q to obtain the status of the conflicting transaction, and the conflicting TRXID is returned according to the conflicting transaction status; Traverse the next record of record Y in sequence and perform conflict detection and judgment until the end of the record linked list.

6. The method for failure recovery of setting transaction operation dependency according to claim 5, characterized in that: The step of searching the detected conflicting TRXID in the hash table Q to obtain the state of the conflicting transaction and returning the conflicting TRXID according to the conflicting transaction state specifically includes: If the conflicting transaction is active: -1 is returned as the conflicting TRXID, indicating that it may conflict with the transaction whose committed LSN is smaller than the transaction to which the current operation belongs; If the conflicting transaction is in rollback state: ignore the transaction, continue to search for other conflicting transactions in the current hash item, and if a conflicting transaction is located, continue to classify and process it according to the conflicting transaction status; otherwise, use 0 as the conflicting TRXID to return, indicating that no conflict is detected; If the conflicting transaction is in the committed state: the conflicting transaction ID is returned as the conflicting TRXID.

7. The method for failure recovery of setting transaction operation dependency according to any one of claims 1 to 6, characterized in that: When the system fails and restarts, the key in the hash table R is restored using the last checkpoint file, which specifically includes: Get the most recent checkpoint file F, get the starting LSN of the checkpoint file F as LSN1, and the ending LSN as LSN2; create an array A of size N; Read the transaction ID information of the executed stored transaction in the checkpoint file F for filtering during recovery; read the transaction operation OP saved in the checkpoint file F, extract the transaction ID of the operation, and determine whether the transaction ID has been executed. If so, abandon the operation; otherwise, extract the value of the LSN on the operation OP and store it in the array A according to the algorithm A[LSN-LSN1]=OP; Extract the operation OP, determine whether the operation is a DML operation, if so, mark the transaction as active in the hash table Q according to the TRXID on the operation; otherwise, determine whether it is a commit operation, if so, mark the transaction as committed in the hash table Q according to the TRXID on the operation, otherwise, mark it as rollback in the hash table Q according to the TRXID on the operation; Read the operations in the order of array A, extract the KEY and transaction ID in the operation to restore the state of hash table R before the failure.

8. A fault recovery device for setting transaction operation dependency, characterized in that: The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to execute the fault recovery method for setting transaction operation dependency as described in any one of claims 1-7.

9. A fault recovery system for setting transaction operation dependency, using the fault recovery method for setting transaction operation dependency as claimed in any one of claims 1 to 7, characterized in that: The system includes a checkpoint setting module, a submission LSN judgment module and a fault restart module, wherein: The checkpoint setting module is used to determine whether the difference between the LSN of the current operation and the LSN of the last checkpoint is greater than or equal to a preset value N after the target end data synchronization service receives the operation of the source end. If so, the operations within the range of the last checkpoint LSN to the current LSN-1 are classified as transactions and saved in the checkpoint file, and the starting LSN value of the checkpoint is set to the last checkpoint LSN+1, and the ending LSN is set to the current operation LSN-1; The commit LSN judgment module is used to asynchronously execute and store the transaction after the target end data synchronization service receives the commit operation of the transaction. After the storage is completed, it is judged whether the commit LSN of the transaction commit operation is in the latest checkpoint file, and the judgment is made according to the condition that the start LSN of the checkpoint file is less than or equal to the commit LSN and less than or equal to the end LSN. If it is satisfied, the transaction ID is appended to the checkpoint file; The fault restart module is used to restore the KEY in the hash table R using the last checkpoint file after the system fails and restarts; after the restart is completed, it continues to receive operations from the source end, discards the operations whose received operation LSN is less than or equal to the checkpoint file termination LSN, and extracts the KEY in the operation for conflict detection and judgment if it is greater than the checkpoint file termination LSN.

10. A non-volatile computer storage medium, characterized in that: The computer storage medium stores computer executable instructions, which are executed by one or more processors to complete the fault recovery method for setting transaction operation dependency as described in any one of claims 1-7.

Images