A method and device for log parsing and synchronous transaction storage

Abstract

The present invention relates to a method and device for log parsing and synchronous transaction storage. The method mainly includes: deploying a data synchronization service at the target end, and when receiving the next log operation sent by the source end, performing corresponding processing according to the operation type; judging whether the memory occupation of all current operations is greater than the set memory occupation value, and if it is greater, triggering a checkpoint saving operation: using the LSN of the last received log operation, setting this LSN as the current checkpoint LSN, and storing the received transaction operations; the data synchronization service at the target end extracts the committed transactions from the checkpoint file, synchronizes them in the order of the committed LSN size of the transactions, and when each checkpoint file is synchronized, cleans up the checkpoint files in the checkpoint file linked list according to the committed LSN of the last synchronized completed transaction. The present invention can avoid a large accumulation of archived logs in the source-end database when the synchronization performance of the target-end database is insufficient, thereby affecting the security of the operation of the source-end database.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for storing log parsing and synchronous transactions. Background Art

[0002] Currently, heterogeneous database replication technology based on database log analysis is widely used. This technology captures the incremental data of the database at the source end and then sends it to the target end. At the target end, the incremental data is applied to the target database through a common database access interface to achieve data replication. Because this technology uses a common database interface, it supports heterogeneous database system replication and heterogeneous operating system environments. The target standby database system is readable and writable, which is a "dual-active" system.

[0003] When running real-time synchronization based on database log parsing, we often encounter the problem that the source database log parsing component is blocked from sending transaction encapsulation message packages due to the slow execution of the destination database transaction. At the same time, when the transaction encapsulation message sending module of the source database log parsing component cannot be sent, the source database archive log cannot be cleaned up, which causes the source database to occupy a large amount of disk space; in the most serious case, the disk space will be full and the source database will be unable to provide services normally. Therefore, how the destination data synchronization service can efficiently store these parsed transaction operations has become an important technical problem that needs to be solved in the industry.

[0004] In view of this, how to overcome the defects of the prior art and solve the above-mentioned technical problems is a difficult problem to be solved in this technical field. Summary of the invention

[0005] In view of the defects or improvement requirements of the existing technology: When data synchronization is performed on the target side, due to the influence of the synchronization performance of the target side database, the synchronization operations received by the target side data synchronization service cannot be stored in time. It is necessary to store these operations that cannot be stored in time to the disk to avoid a large amount of archive logs of the source side database from accumulating when the synchronization performance of the target side database is insufficient, thereby affecting the security of the source side database operation.

[0006] The present invention provides a method and device for log parsing and synchronous transaction storage. In data synchronization, data is synchronized in units of transactions. Therefore, when the target-side data synchronization service stores transactions, it needs to sort the received log operations in units of transactions first. The committed transactions (transactions that have received commit operations) are stored in the committed transaction linked list in the order of the commit LSN of the transactions, and the uncommitted transactions are stored in the active transaction linked list in the order of the start LSN of the transactions. Then, according to the LSN of the checkpoint, the transaction operations and transaction information in the committed transaction linked list with the commit operation LSN less than or equal to the checkpoint LSN are stored in the checkpoint file in units of transactions, and then the transaction operations and transaction information in the active transaction linked list are stored in the checkpoint file, and the reference count of the checkpoint file is marked according to the number of stored active transactions. If there are no active transactions, the maximum transaction commit LSN of the checkpoint file is marked and the storage of the operation is completed. When cleaning up the checkpoint file, according to the maximum LSN of the transactions that have been synchronized and completed, the maximum transaction commit LSN of the checkpoint file is judged from the checkpoint file linked list, and the checkpoint file with the maximum transaction commit LSN less than or equal to the maximum LSN of the transactions that have been synchronized and completed is deleted.

[0007] The embodiments of the present invention adopt the following technical solutions:

[0008] In a first aspect, the present invention provides a method for log parsing and synchronous transaction storage, including:

[0009] Deploy a data synchronization service at the target end. When receiving the next log operation sent by the source end, perform corresponding processing according to the operation type;

[0010] Judge whether the memory occupation of all current operations is greater than the set memory occupation value. If it is greater, trigger the checkpoint saving operation: use the LSN of the most recently received log operation, set this LSN as the current checkpoint LSN, and store the received transaction operations;

[0011] The target-side data synchronization service extracts the committed transactions from the checkpoint file and synchronizes them in the order of the commit LSN of the transactions. When each checkpoint file is synchronized, the checkpoint file in the checkpoint file linked list is cleaned up according to the commit LSN of the last synchronized and completed transaction.

[0012] Further, when deploying the data synchronization service at the target end, the target-side data synchronization service is responsible for receiving the log operations sent by the source end. Each log operation contains the LSN of the operation and the transaction ID information of the operation. The target-side data synchronization service initializes the following information:

[0013] A committed transaction linked list for storing the transactions that have received commit operations, and these transactions are stored in the order of the LSN of the transaction commit operation;

[0014] An active transaction linked list for storing transactions that have not received a commit operation, and these transactions are stored in the order of the LSN of the first operation of the transaction;

[0015] Set a memory occupancy value for triggering the operation to save the checkpoint, which is used to start the checkpoint to save the transaction operations in the current memory when the memory occupied by the received operation reaches this memory occupancy value.

[0016] Further, when receiving the next log operation sent by the source end, the corresponding processing according to the operation type specifically includes:

[0017] If the operation is a commit or rollback operation, check whether the current transaction T has an affiliated checkpoint file. If so, decrement the active transaction reference count of the checkpoint file by 1. When the checkpoint reference count is decremented to 0, update the LSN of the current operation to the maximum transaction commit LSN of the checkpoint file;

[0018] If the operation is a partial rollback operation, check whether the current transaction has an affiliated checkpoint file. If so, it means that the previous operations of this transaction have been cached in the checkpoint file. Delete the operations that need to be rolled back in the checkpoint file, obtain the storage address of the last record of the current transaction in the checkpoint file, traverse forward along the linked list of the records, delete the records whose operation IDs are greater than or equal to the partial rollback operation ID in the linked list, and update the storage address of the last record of the transaction in the checkpoint file to the checkpoint file;

[0019] If it is other operation types, no processing of the checkpoint file is performed.

[0020] Further, for the checkpoint save operation: use the LSN of the most recently received log operation, set this LSN as the current checkpoint LSN, and the storage of the received transaction operations specifically includes:

[0021] Create a checkpoint file A, name the checkpoint file with the value of the checkpoint LSN, initialize the space of file A from 0 to N1 to 0, and mark it as the file header;

[0022] Collect the active transaction set, reserve the active transaction space in the checkpoint file A, that is, the N1 to N2 active transaction reservation area; collect the committed transaction set, reserve the committed transaction space in the checkpoint file A, that is, the N2 to N3 committed transaction reservation area;

[0023] Extract transactions from the committed transaction set in turn and extract transactions from the active transaction set in turn, and store them in the corresponding checkpoint files of the transactions;

[0024] Save the active transaction information to the active transaction reservation area from N1 to N2 in checkpoint file A, and save the committed transaction information to the committed transaction reservation area from N2 to N3 in checkpoint file A;

[0025] Save the checkpoint LSN, the active transaction storage area information from N1 to N2, and the committed transaction storage area information from N2 to N3 to the file header of checkpoint file A;

[0026] Complete the storage action of this checkpoint file A, and release the memory space occupied by the saved transactions;

[0027] After the next checkpoint is triggered, save checkpoint file B in the way of checkpoint save operation, and so on, form a linked list of checkpoint files in the order of the size of checkpoint LSN to complete the storage of transaction information and transaction operations for the target - end data synchronization service.

[0028] Furthermore, the collection of the active transaction set and the reservation of the active transaction space in checkpoint file A; the collection of the committed transaction set and the reservation of the committed transaction space in checkpoint file A specifically include:

[0029] Collect active transactions. Traverse the active transaction linked list using the checkpoint LSN, and collect all transactions whose starting LSN is less than or equal to the checkpoint LSN into the active transaction set; traverse the committed transaction linked list using the checkpoint LSN, and collect all transactions whose starting LSN is less than or equal to the checkpoint LSN and whose commit LSN is greater than the checkpoint LSN into the active transaction set. After collection, deduplicate the transactions in the active transaction set;

[0030] Calculate the space size K bytes required to store the active transaction information, K = the number of bytes occupied by a single active transaction information * the number of active transactions, and reserve space in checkpoint file A. Let N2 = N1 + K, and reserve the part from N1 to N2 as the active transaction space;

[0031] Collect committed transactions. Traverse the committed transaction linked list using the checkpoint LSN, and collect all transactions whose commit LSN is less than or equal to the checkpoint LSN into the committed transaction set;

[0032] Calculate the space size G bytes required to store the committed transaction information, G = the number of bytes occupied by a single transaction information * the number of transactions, and reserve space in checkpoint file A. Let N3 = N2 + G, and reserve the part from N2 to N3 as the committed transaction space.

[0033] Furthermore, the steps of respectively extracting transactions from the committed transaction set in sequence, extracting transactions from the active transaction set in sequence, and storing them into the corresponding checkpoint files of the transactions specifically include:

[0034] Transactions are sequentially extracted from the set of committed transactions, and the checkpoint file X stored by the transaction is determined from the transaction information. If the transaction does not record the checkpoint file information, the current checkpoint file A is used as the checkpoint file X stored by the transaction. The operations of the transaction are appended in the form of a doubly linked list to the space starting from offset N3 in the checkpoint file X, and the storage address of the first operation is recorded in the transaction information;

[0035] Transactions are sequentially extracted from the set of active transactions, and the checkpoint file X stored by the transaction is determined from the transaction information. If the transaction does not record the checkpoint file information, the current checkpoint file A is used as the checkpoint file X stored by the transaction. The operations in the transaction with an LSN less than or equal to the checkpoint LSN are stored in the checkpoint file X in the form of a doubly linked list, and the storage location is appended after the operations of the committed transactions. The storage addresses of the first and last operations and the information of the current checkpoint file X are recorded in the transaction information.

[0036] Further, saving the active transaction information to the N1 - N2 active transaction reserved area in the checkpoint file A and saving the committed transaction information to the N2 - N3 committed transaction reserved area in the checkpoint file A specifically includes:

[0037] Saving the active transaction information to the N1 - N2 active transaction reserved area in the checkpoint file A. If the checkpoint file of the active transaction points to the current checkpoint file A, then for each active transaction saved, the reference count of the active transactions in the checkpoint file A needs to be incremented by 1, and the maximum transaction commit LSN of the checkpoint file is set to infinity;

[0038] Saving the committed transaction information to the N2 - N3 committed transaction reserved area in the checkpoint file A. Determine whether the reference count of the active transactions in the current checkpoint file A is 0. If it is, then set the maximum transaction commit LSN of the checkpoint file to the maximum commit LSN among the committed transactions saved.

[0039] Further, when the target - end data synchronization service fails and restarts, all checkpoint files are loaded, a recovery point is set according to the checkpoint files, and the fault - recovery function is implemented based on the recovery point; specifically:

[0040] When the target - end data synchronization service fails and restarts, all checkpoint files are loaded, and the checkpoint files are sorted according to the LSN value in the file name;

[0041] Read the header of the last checkpoint file in the checkpoint file linked list, extract the saved checkpoint LSN. If the checkpoint LSN is not 0, then use this checkpoint file as the recovery point; otherwise, discard this checkpoint file and use its previous checkpoint file as the recovery point;

[0042] Read the LSN in the checkpoint file used as the recovery point, and restore the transaction information in the active transaction space of checkpoint files N1 to N2 to the active transaction linked list. For each restored active transaction, increment the reference count of the active transaction reference in the checkpoint file pointed to by this transaction, and then continue to receive the transaction operations sent by the source end. Compare the LSN of the operation with the LSN in the recovery point checkpoint file, and only receive the operations whose operation LSN is greater than the LSN in the checkpoint file to implement the fault recovery function.

[0043] Furthermore, when cleaning the checkpoint file, compare this LSN with the maximum transaction commit LSN of each checkpoint file in the checkpoint file linked list according to the maximum commit LSN of the synchronized transaction. When this LSN is greater than or equal to the maximum transaction commit LSN of the checkpoint file, it means that the checkpoint file has been synchronized and can be cleaned up.

[0044] On the other hand, the present invention provides a device for log parsing and synchronous transaction storage, specifically including: at least one processor and a memory, which are connected by a data bus. The memory stores instructions executable by at least one processor. After the instructions are executed by the processor, they are used to complete the method of log parsing and synchronous transaction storage in the first aspect.

[0045] Compared with the prior art, the beneficial effect of the present invention is that when the synchronous operations received by the target end data synchronization service cannot be stored in the database in time, these operations that cannot be stored in time can be efficiently and accurately stored on the disk to avoid a large backlog of archive logs in the source end database when the synchronization performance of the target end database is insufficient, thus affecting the security of the operation of the source end database.

[0046] In addition, after the checkpoint file is created, by collecting the information of active transactions and committed transactions, calculate the space required to store them respectively and reserve it in the checkpoint file. Since the information such as the storage location of the operations in the transaction can only be determined after the operations are stored, this method can avoid modifying the already stored transaction information after the transaction information is stored, and achieve batch writing of transaction information at one time.

[0047] Secondly, save the transaction operations by appending them one by one after the checkpoint file in units of transactions, which can ensure the continuity of IO when storing transaction operations and store transaction operations with higher IO performance.

[0048] Thirdly, when saving transaction operations, it is necessary to first extract the checkpoint file information saved last time for this transaction, and then append the operations on the transaction to the checkpoint file where it was stored last time, avoiding transaction operations spanning multiple checkpoint files and reducing complexity. Description of the Drawings

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other accompanying drawings can be obtained based on these drawings.

[0050] Figure 1 It is a flowchart of a method for log parsing and synchronous transaction storage provided in Embodiment 1 of the present invention;

[0051] Figure 2 It is an expanded flowchart of step 100 provided in Embodiment 1 of the present invention;

[0052] Figure 3 It is a specific flowchart of step 200 provided in Embodiment 1 of the present invention;

[0053] Figure 4 It is a specific flowchart of step 220 provided in Embodiment 1 of the present invention;

[0054] Figure 5 It is a specific flowchart of step 230 provided in Embodiment 1 of the present invention;

[0055] Figure 6 It is a specific flowchart of step 240 provided in Embodiment 1 of the present invention;

[0056] Figure 7 It is an implementation flowchart of fault recovery provided in Embodiment 1 of the present invention;

[0057] Figure 8 It is a schematic diagram of the device structure for log parsing and synchronous transaction storage provided in Embodiment 3 of the present invention. Detailed implementation manners

[0058] In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0059] The present invention is an architecture of a specific functional system. Therefore, in specific embodiments, the functional logical relationships of each structural module are mainly described, and the specific software and hardware implementation manners are not limited.

[0060] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0061] Embodiment 1:

[0062] In the embodiment of the present invention, by comparing the LSN of the transaction operation with the checkpoint LSN, the operations less than or equal to the checkpoint LSN are saved in the corresponding checkpoint file in units of transactions, and when saving, the committed transactions and active transactions need to be distinguished, because the committed transactions are used for synchronous execution, while the active transactions are used for failure recovery.

[0063] As Figure 1 shown, based on the above actual situation, the embodiment of the present invention provides a method for parsing and synchronizing transaction storage of logs, and the specific steps are as follows.

[0064] Step 100: Deploy a data synchronization service at the target end. When receiving the next log operation sent by the source end, perform corresponding processing according to the operation type.

[0065] Step 200: Determine whether the memory occupation of all current operations is greater than the set memory occupation value. If it is greater, trigger the checkpoint saving operation: use the LSN of the most recently received log operation, set this LSN as the current checkpoint LSN, and store the received transaction operations. When judging the memory occupation value in this step, if the memory occupation of all current operations is not greater than the memory occupation value, continue to receive the next log operation.

[0066] Step 300: The data synchronization service at the target end extracts the committed transactions from the checkpoint file and synchronizes them in the order of the commit LSN of the transactions. When each checkpoint file is synchronized, clean the checkpoint file in the checkpoint file linked list according to the commit LSN of the last synchronized transaction.

[0067] Specifically, in an implementation manner of this preferred embodiment, the data synchronization service deployed in step 100 at the target end is responsible for receiving the log operations sent by the source end. Each log operation contains information such as the LSN of the operation and the transaction ID of the operation. The data synchronization service at the target end initializes the following information:

[0068] A committed transaction linked list for storing the transactions that have received the commit operation, and these transactions are stored in the order of the LSN of the transaction commit operation;

[0069] An active transaction linked list for storing the transactions that have not received the commit operation, and these transactions are stored in the order of the LSN of the first operation of the transaction;

[0070] Set a memory occupation value N for triggering the operation to save the checkpoint, which is used to start saving the transaction operations in the current memory when the memory occupation of the received operations reaches this memory occupation value N.

[0071] As Figure 2As shown, in one implementation of this preferred embodiment, when receiving the next log operation sent by the source end in step 100, the corresponding processing according to the operation type specifically includes the following steps (first classify it into the transaction T to which it belongs according to the transaction ID, check the current log operation type, and perform the following processing according to the operation type):

[0072] Step 101: If the operation is a commit or rollback operation, check whether the current transaction T has an affiliated checkpoint file. If so, decrement the active transaction reference count of the checkpoint file by 1. When the checkpoint reference count is decremented to 0, update the LSN of the current operation to the maximum transaction commit LSN of the checkpoint file.

[0073] Step 102: If the operation is a partial rollback operation, check whether the current transaction has an affiliated checkpoint file. If so, it means that the operations before this transaction have been cached in the checkpoint file. It is necessary to delete the operations that need to be rolled back in the checkpoint file, obtain the storage address of the last record of the current transaction in the checkpoint file, traverse forward along the linked list of the records, delete the records whose operation IDs are greater than or equal to the partial rollback operation ID in the linked list, and update the storage address of the last record of the transaction in the checkpoint file to the checkpoint file.

[0074] Step 103: If it is other operation types, no processing of the checkpoint file is performed.

[0075] As Figure 3 shown, in one implementation of this preferred embodiment, the checkpoint save operation in step 200: using the LSN of the last received log operation, setting this LSN as the current checkpoint LSN, and storing the received transaction operations specifically includes the following steps:

[0076] Step 210: Create a checkpoint file A. When naming the checkpoint file, name it with the value of the checkpoint LSN for quick sorting of the checkpoint file, initialize the space of file A from 0 to N1 to 0, and mark it as the file header.

[0077] Step 220: Collect the active transaction set, reserve the active transaction space in the checkpoint file A, that is, the N1 to N2 active transaction reservation area; collect the committed transaction set, reserve the committed transaction space in the checkpoint file A, that is, the N2 to N3 committed transaction reservation area.

[0078] Step 230: Extract transactions from the committed transaction set and the active transaction set in turn, and store them in the corresponding checkpoint files of the transactions.

[0079] Step 240: Save the active transaction information to the reserved area for active transactions from N1 to N2 in checkpoint file A, and save the committed transaction information to the reserved area for committed transactions from N2 to N3 in checkpoint file A.

[0080] Step 250: Save the checkpoint LSN, the information of the active transaction storage area from N1 to N2, and the information of the committed transaction storage area from N2 to N3 to the file header of checkpoint file A.

[0081] Step 260: Complete the storage operation of this checkpoint file A, and release the memory space occupied by the saved transactions.

[0082] Step 270: After the next checkpoint is triggered, save checkpoint file B in the way of checkpoint saving operation (that is, repeat all steps of step 200), and so on, form a linked list of checkpoint files in the order of the size of checkpoint LSN, so as to complete the storage of transaction information and transaction operations for the data synchronization service at the target end.

[0083] As Figure 4 shown, in an implementation manner of this preferred embodiment, for the collection of the active transaction set in step 220, reserve the active transaction space in checkpoint file A; for the collection of the committed transaction set, reserve the committed transaction space in checkpoint file A, which specifically includes the following steps:

[0084] Step 221: Collect active transactions. Traverse the active transaction linked list using the checkpoint LSN, and collect all transactions whose starting LSN is less than or equal to the checkpoint LSN into the active transaction set; traverse the committed transaction linked list using the checkpoint LSN, and collect all transactions whose starting LSN is less than or equal to the checkpoint LSN and whose commit LSN is greater than the checkpoint LSN into the active transaction set. After the collection is completed, deduplicate the transactions in the active transaction set.

[0085] The reason for deduplication in this step is that after collecting the active transaction linked list, these active transactions will be added to the committed transaction linked list after receiving commit messages during subsequent operations, resulting in duplicate collection of the same transactions when collecting active transactions in the committed transaction linked list later. (Active transactions are determined based on the checkpoint LSN. Since after determining the checkpoint LSN, data synchronization at the target end is still collecting operations sent from the source end, transactions that originally existed in the active transaction linked list will be moved to the committed transaction linked list after receiving commit operations. However, relative to the current checkpoint LSN, this transaction still belongs to the active transactions.) Additionally, when collecting, it is necessary to start collecting from the active transaction linked list first. Because if collecting from the committed transaction linked list first, then switching to the active transaction linked list after collecting this linked list, some transactions may be moved to the committed transaction linked list due to receiving commit operations during the process of collecting the active transaction linked list, resulting in omissions.

[0086] Step 222: Calculate the size of the space K bytes required to store active transaction information, where K = the number of bytes occupied by a single active transaction information * the number of active transactions, and reserve space in checkpoint file A. Let N2 = N1 + K, and reserve the part from N1 to N2 as the active transaction space.

[0087] When collecting active transactions, it is not possible to store while collecting because the starting offset where the transaction operations are stored (the starting offset refers to the offset address where this transaction is stored in the checkpoint file, starting from which byte in the checkpoint file) has not been determined yet. Therefore, reserve the space first, and mark the record position of the transaction after the transaction operations are stored, and then save the active transaction information.

[0088] Step 223: Collect committed transactions, traverse the committed transaction linked list using the checkpoint LSN, and collect transactions with a transaction commit LSN less than or equal to the checkpoint LSN into the committed transaction set.

[0089] Step 224: Calculate the size of the space G bytes required to store committed transaction information, where G = the number of bytes occupied by a single transaction information * the number of transactions, and reserve space in checkpoint file A. Let N3 = N2 + G, and reserve the part from N2 to N3 as the committed transaction space.

[0090] As Figure 5 shown, in an implementation manner of this preferred embodiment, the steps of respectively extracting transactions from the committed transaction set in sequence, extracting transactions from the active transaction set in sequence, and storing them into the corresponding checkpoint files in step 230 specifically include the following steps:

[0091] Step 231: Extract transactions from the set of committed transactions in sequence. Determine the checkpoint file X stored by this transaction from the transaction information. If this transaction does not record checkpoint file information, then use the current checkpoint file A as the checkpoint file X stored by this transaction. Append the operations of the transaction to the space starting from offset N3 in the checkpoint file X in the form of a doubly linked list (referring to the space immediately following the space reserved for storing committed transaction information in the checkpoint file), and record the storage address of the first operation in the transaction information so that when extracting this transaction, the first operation of this transaction can be found through this address for traversal.

[0092] If a committed transaction has saved operations in the role of an active transaction in the previous checkpoint, then the checkpoint file information stored last time will be saved in its transaction information. In this case, to ensure that the operations of a single transaction are not stored across files, the subsequent operations of this transaction still need to be saved to the previous checkpoint file during this checkpoint storage.

[0093] Step 232: Extract transactions from the set of active transactions in sequence. Determine the checkpoint file X stored by this transaction from the transaction information. If this transaction does not record checkpoint file information, then use the current checkpoint file A as the checkpoint file X stored by this transaction. Store the operations of the transaction with operation LSN less than or equal to the checkpoint LSN in the form of a doubly linked list in the checkpoint file X, and store them in an appended form after the operations of the committed transactions. Record the storage addresses of the first and last operations and the information of the current checkpoint file X in the transaction information.

[0094] The difference between storing the operations of active transactions and committed transactions lies in: Since an active transaction has not yet received a commit operation, there are uncertain factors in this transaction, and there may be partial rollback actions. When a partial rollback occurs, it is necessary to roll back forward from the last operation. Therefore, the operations of active transactions need to be saved in the form of a doubly linked list, and the address of the last operation is saved in the transaction information; while committed transactions do not have the need to roll back forward, and the address of the last operation does not need to be saved during storage; and recording the information of the checkpoint file X in the transaction information is to ensure that the subsequent newly added operations of this transaction can be recorded in the checkpoint file X during the next checkpoint storage, and to ensure that the operations of a single transaction do not cross files.

[0095] As Figure 6 shown, in an implementation manner of this preferred embodiment, the steps of saving the active transaction information to the N1 - N2 active transaction reserved area in the checkpoint file A and saving the committed transaction information to the N2 - N3 committed transaction reserved area in the checkpoint file A in step 240 specifically include the following steps:

[0096] Step 241: Save the active transaction information to the reserved area for active transactions N1 to N2 in checkpoint file A. If the checkpoint file of the active transaction points to the current checkpoint file A, then for each active transaction saved, the reference count of active transactions in checkpoint file A needs to be incremented by 1, and the maximum transaction commit LSN of the checkpoint file is set to infinity.

[0097] Step 242: Save the committed transaction information to the reserved area for committed transactions N2 to N3 in checkpoint file A. Determine whether the reference count of active transactions in the current checkpoint file A is 0. If it is, then set the maximum transaction commit LSN of the checkpoint file to the maximum commit LSN among the saved committed transactions. If it is not 0, the maximum transaction commit LSN of the checkpoint file is set to infinity, so that it will not be cleaned up.

[0098] In an implementation manner of this preferred embodiment, when the target - end data synchronization service fails and restarts, load all checkpoint files, set the recovery point according to the checkpoint files, and implement the fault - recovery function based on the recovery point. Specifically, as Figure 7 shown, the implementation of the above - mentioned fault recovery specifically includes the following steps:

[0099] Step 401: When the target - end data synchronization service fails and restarts, load all checkpoint files and sort the checkpoint files according to the LSN value in the file name.

[0100] Step 402: Read the header of the last checkpoint file in the checkpoint file linked list, extract the saved checkpoint LSN. If the checkpoint LSN is not 0, then use this checkpoint file as the recovery point; otherwise, discard this checkpoint file and use the previous checkpoint file as the recovery point.

[0101] Step 403: Read the LSN in the checkpoint file used as the recovery point, and restore the transaction information in the active transaction space from N1 to N2 of the checkpoint file to the active transaction linked list. For each active transaction restored, increment the reference count of active transactions in the checkpoint file pointed to by this transaction by 1, and then continue to receive the transaction operations sent by the source end. Compare the LSN of the operation with the LSN in the checkpoint file of the recovery point, and only receive the operations with the LSN of the operation greater than the LSN in the checkpoint file to implement the fault - recovery function.

[0102] In an implementation manner of this preferred embodiment, when cleaning the checkpoint file in step 300, it is necessary to compare the maximum commit LSN of the synchronized transactions with the maximum transaction commit LSN of each checkpoint file in the checkpoint file linked list. When this LSN is greater than or equal to the maximum transaction commit LSN of the checkpoint file, it means that the checkpoint file has been synchronized and can thus be cleaned up.

[0103] When storing transactions through the above solution, the committed transaction information within two adjacent checkpoint LSN intervals is stored in the same checkpoint file. In this way, when synchronizing transactions into the database, as long as the transactions are loaded into the database in the order of the LSN sizes of the checkpoint files, the orderliness of transaction storage and data consistency can be ensured.

[0104] In summary, when the synchronization operations received by the target - end data synchronization service cannot be stored in the database in a timely manner, these operations that cannot be stored in a timely manner can be efficiently and accurately stored on the disk to avoid a large backlog of archived logs in the source - end database when the synchronization performance of the target - end database is insufficient, thus affecting the security of the source - end database operation.

[0105] In addition, after the checkpoint file is created, by collecting information on active and committed transactions, calculating the space required to store them separately, and reserving space in the checkpoint file. Since information such as the storage location of operations in a transaction can only be determined after the operations are stored, this method can avoid modifying the already - stored transaction information after the transaction information is stored, and achieve batch writing of transaction information at one time.

[0106] Secondly, saving transaction operations by appending them one by one after the checkpoint file in units of transactions can ensure the continuity of I / O when storing transaction operations and store transaction operations with relatively high I / O performance.

[0107] Thirdly, when saving transaction operations, it is necessary to first extract the checkpoint file information saved last time for the transaction, and then append the operations on the transaction to the checkpoint file where it was stored last time, avoiding transaction operations spanning multiple checkpoint files and reducing complexity.

[0108] Embodiment 2:

[0109] Based on the method for storing log - parsed synchronous transactions provided in Embodiment 1, this Embodiment 2 further elaborates on the present invention through a specific application scenario.

[0110] The above - mentioned solution is illustrated as follows:

[0111] The source database has a table T1(ID VARCHAR)

[0112] There are three transactions at the source end that perform the following operations on table T1:

[0113] TRX1: INSERT INTO T1(ID) VALUES('TRX1_1');

[0114] TRX1: SAVEPOINT A;

[0115] TRX1: INSERT INTO T1(ID) VALUES('TRX1_11');

[0116] TRX2: INSERT INTO T1(ID) VALUES('TRX2_1');

[0117] TRX2: COMMIT;

[0118] TRX1: ROLLBACK TO SAVEPOINT A;

[0119] TRX1: INSERT INTO T1(ID) VALUES('TRX1_2');

[0120] TRX3: INSERT INTO T1(ID) VALUES('TRX3_1');

[0121] TRX3: COMMIT;

[0122] TRX1: COMMIT;

[0123] The order of the above operations will form the following numbered table when received by the destination end log receiving thread:

[0124]

[0125]

[0126] The process of storing the checkpoint is as follows:

[0127] 1. When the operation with LSN 4 is received and saved, the checkpoint is triggered. At this time, the checkpoint LSN is set to 4, and the transaction {TRX1} exists in the active transaction linked list; the transaction {TRX2} exists in the committed transaction linked list.

[0128] 2. Create checkpoint file A and complete the reservation and initialization of the file header:

[0129] Offset address File space 0-H File header

[0130] 3. Filter out the transactions in the active transaction linked list and the committed transaction linked list whose starting LSN is less than or equal to 4 and store them in the active transaction set. TRX1 meets the conditions. And calculate the space required to reserve for storing the TRX1 transaction information and make the reservation.

[0131] Offset address File space 0 to N1 File header N1 to N2 Active transaction reservation space

[0132] 4. Screen out the transactions in the committed transaction linked list whose transaction commit LSN is less than or equal to 4 and store them in the committed transaction set. TRX2 meets the condition. Calculate the space required to reserve the transaction information of TRX2 and reserve it.

[0133]

[0134]

[0135] 5. Extract transactions from the committed transaction set in sequence, and append the transaction operations of transaction TRX2 to the checkpoint file after N3.

[0136]

[0137] Among them, there is pointer information of the two-way operation linked list between records 1 and 2 in the N3 space.

[0138] 6. Extract transactions from the active transaction set in sequence, and append the transaction operations of transaction TRX1 to the checkpoint file after N3.

[0139]

[0140]

[0141] 7. Save the active transaction information to the N1 to N2 active transaction reservation area in checkpoint file A, and increment the active transaction reference count of checkpoint file A by 1 (increment by the number of active transactions).

[0142]

[0143] 8. Save the committed transaction information to the N2 to N3 committed transaction reservation area in checkpoint file A. Since the active transaction reference count of checkpoint file A is not 0, the maximum transaction commit LSN of this checkpoint file is infinite.

[0144]

[0145]

[0146] 9. Save the checkpoint LSN, the active transaction storage area information N1 to N2, and the committed transaction storage area information N2 to N3 to the file header of checkpoint file A.

[0147]

[0148] 10. When a partial rollback operation with the received LSN of 5 is received currently, according to the rules, the log records saved for TRX1 in checkpoint file A need to be deleted to form the following state:

[0149]

[0150] 11. After receiving and saving the operation with LSN 8, a checkpoint is triggered. At this time, the checkpoint LSN is set to 8, and transaction {TRX1} exists in the active transaction linked list; transaction {TRX3} exists in the committed transaction linked list.

[0151] Create checkpoint file B and form the following two checkpoint files A and B according to the above scheme:

[0152] Checkpoint file A:

[0153]

[0154]

[0155] When the checkpoint LSN is 8, TRX1 is still an active transaction, so its operations will be stored in checkpoint file A pointed to by the transaction information, preventing cross-file storage of transaction operations, and there is a two-way linked list pointer between operation records 3 and 4.

[0156] Checkpoint file B:

[0157]

[0158] When the checkpoint LSN is 8, TRX1 is still an active transaction, so its operations will be stored in checkpoint file A pointed to by the transaction information. The active transaction reference count of checkpoint file A is still 1, and the maximum transaction commit LSN is infinite. While in checkpoint file B, there is no storage of the operations of transaction TRX1 and no active transaction information. At this time, the active transaction reference count of checkpoint file B is 0, and its maximum transaction commit LSN is 8.

[0159] 12. After checkpoint B is completed, restart recovery, load checkpoint files A and B and sort them according to the size of the checkpoint LSN to form a checkpoint file linked list of {A, B}.

[0160] 13. Take the last checkpoint file B in the checkpoint file linked list, read its file header and verify the validity of the checkpoint LSN.

[0161] 14. Load the N1 - N2 active transaction information of checkpoint file B, restore TRX1 to the active transaction linked list, and increment the active transaction reference count of checkpoint file A by 1. It should be noted that when restoring, the program is restarted, and the previously reserved counts will be cleared after restart, so here it is also necessary to increment the active transaction reference count of checkpoint file A by 1.

[0162] 15. Only receive operation logs with the log sequence number (LSN) sent by the source end greater than 8.

[0163] 16. When receiving the commit operation of transaction TRX1 with an operation LSN of 9, since TRX1 points to checkpoint file A, the active transaction reference count of checkpoint file A is decremented by 1. When the reference count becomes 0, the maximum transaction commit LSN of checkpoint file A needs to be updated to the current operation's LSN. At this time, the maximum transaction commit LSN of checkpoint file A is 9, and then transaction TRX1 is moved to the committed transaction linked list.

[0164] 17. The synchronization service sequentially executes TRX2, TRX3, and TRX1 in the order of transaction commit. When TRX3 is executed, its commit LSN 8 is used to clean the checkpoint file. At this time, the maximum transaction commit LSN of checkpoint file A in the checkpoint file linked list is 9, which is greater than the cleaning LSN 8, so it cannot be cleared. While the maximum transaction commit LSN of checkpoint file B is 8, which is less than or equal to LSN 8, so it can be cleared.

[0165] Embodiment 3:

[0166] Based on the method for log parsing and synchronous transaction storage provided in the above Embodiment 1 to Embodiment 2, the present invention further provides a device for log parsing and synchronous transaction storage that can be used to implement the above method, as Figure 8 shown, which is a schematic diagram of the device architecture of an embodiment of the present invention. The device for log parsing and synchronous transaction storage in this embodiment includes one or more processors 21 and a memory 22. Among them, Figure 8 one processor 21 is taken as an example.

[0167] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 8 and taking the connection through a bus as an example.

[0168] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the methods and systems for log parsing and synchronous transaction storage in Embodiment 1 to Embodiment 2. The processor 21 executes various functional applications and data processing of the device for log parsing and synchronous transaction storage by running the non-volatile software programs, instructions, and modules stored in the memory 22, that is, implementing the methods for log parsing and synchronous transaction storage in Embodiment 1 to Embodiment 2.

[0169] The memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 may optionally include a memory remotely disposed relative to the processor 21, and these remote memories may be connected to the processor 21 through a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] Program instructions / modules are stored in the memory 22 and, when executed by one or more processors 21, perform the method for log parsing and synchronous transaction storage in the above-mentioned Embodiments 1 to 2. For example, perform each of the Figures 1 to 7 steps shown above.

[0171] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by instructing relevant hardware through a program, and this program can be stored in a computer-readable storage medium. The storage medium may include: Read Only Memory (ROM), Random Access Memory (RAM), magnetic disk or optical disk, etc.

[0172] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for parsing and synchronizing transaction storage of logs, characterized in that, Including: Deploy a data synchronization service at the target end. When receiving the next log operation sent by the source end, perform corresponding processing according to the operation type, including: If the current operation is a commit or rollback operation, check whether the current transaction T has an affiliated checkpoint file. If so, decrement the active transaction reference count of the checkpoint file by 1. When the checkpoint reference count is decremented to 0, update the LSN of the current operation to the maximum transaction commit LSN of the checkpoint file. If the current operation is a partial rollback operation, check whether the current transaction has an affiliated checkpoint file. If so, it means that the operations before this transaction have been cached in the checkpoint file. Delete the operations that need to be rolled back in the checkpoint file, obtain the storage address of the last record of the current transaction in the checkpoint file, traverse forward along the linked list of records, delete the records whose operation IDs are greater than or equal to the partial rollback operation ID in the linked list, and update the storage address of the last record of the transaction in the checkpoint file to the checkpoint file after deletion. If it is other operation types, no processing of the checkpoint file is performed; Judge whether the memory occupancy of all current operations is greater than the set memory occupancy value. If it is greater, trigger a checkpoint save operation: Use the LSN of the last received log operation, set this LSN as the current checkpoint LSN, and store the received transaction operations; The target end data synchronization service extracts the committed transactions from the checkpoint file and synchronizes them in the order of the commit LSN of the transactions. When each checkpoint file is synchronized, clean up the checkpoint file in the checkpoint file linked list according to the commit LSN of the last synchronized completed transaction.

2. The method for parsing and synchronizing transaction storage of logs according to claim 1, characterized in that, When deploying the data synchronization service at the target end, the target end data synchronization service is responsible for receiving the log operations sent by the source end. Each log operation contains the LSN of the operation and the transaction ID information of the operation. The target end data synchronization service initializes the following information: A committed transaction linked list, which is used to store the transactions that have received commit operations, and these transactions are stored in the order of the commit LSN of the transaction commit operations; An active transaction linked list, which is used to store the transactions that have not received commit operations, and these transactions are stored in the order of the LSN of the first operation of the transaction; Set a memory occupancy value for triggering the operation to save the checkpoint, which is used to start saving the transaction operations in the current memory when the memory occupied by the received operations reaches this memory occupancy value.

3. The method for parsing and synchronizing transaction storage of logs according to claim 2, characterized in that, The checkpoint save operation: Use the LSN of the last received log operation, set this LSN as the current checkpoint LSN, and the specific storage of the received transaction operations includes: Create a checkpoint file A, name the checkpoint file with the value of the checkpoint LSN, initialize the space of file A from 0 to N1 to 0, and mark it as the file header; Collect the active transaction set, reserve the active transaction space in checkpoint file A, that is, the N1 to N2 active transaction reservation area; collect the committed transaction set, reserve the committed transaction space in checkpoint file A, that is, the N2 to N3 committed transaction reservation area; Extract transactions from the committed transaction set and the active transaction set in sequence, and store them in the checkpoint file corresponding to the transaction; Save the active transaction information to the reserved area for active transactions from N1 to N2 in checkpoint file A, and save the committed transaction information to the reserved area for committed transactions from N2 to N3 in checkpoint file A; Save the checkpoint LSN, the information of the active transaction storage area from N1 to N2, and the information of the committed transaction storage area from N2 to N3 to the file header of checkpoint file A; Complete the storage operation of this checkpoint file A, and release the memory space occupied by the saved transactions; After the next checkpoint is triggered, save checkpoint file B in the way of checkpoint save operation, and so on, form a linked list of checkpoint files in the order of the size of checkpoint LSN, so as to complete the storage of transaction information and transaction operations by the data synchronization service at the target end.

4. The method for parsing and synchronizing transaction storage of logs according to claim 3, characterized in that, Reserve space for active transactions in checkpoint file A for the collected active transaction set; Collect the committed transaction set, and reserve space for committed transactions in checkpoint file A specifically including: Collect active transactions, traverse the active transaction linked list using the checkpoint LSN, and collect all transactions whose start LSN of the transaction is less than or equal to the checkpoint LSN into the active transaction set; traverse the committed transaction linked list using the checkpoint LSN, and collect all transactions whose start LSN of the transaction is less than or equal to the checkpoint LSN and the commit LSN of the transaction is greater than the checkpoint LSN into the active transaction set. After the collection is completed, deduplicate the transactions in the active transaction set; Calculate the size of the space required to store the active transaction information as K bytes, K = the number of bytes occupied by a single active transaction information * the number of active transactions, and reserve space in checkpoint file A. Let N2 = N1 + K, and reserve the part from N1 to N2 as the active transaction space; Collect committed transactions, traverse the committed transaction linked list using the checkpoint LSN, and collect all transactions whose commit LSN of the transaction is less than or equal to the checkpoint LSN into the committed transaction set; Calculate the size of the space required to store the committed transaction information as G bytes, G = the number of bytes occupied by a single transaction information * the number of transactions, and reserve space in checkpoint file A. Let N3 = N2 + G, and reserve the part from N2 to N3 as the committed transaction space.

5. The method for parsing and synchronizing transaction storage of logs according to claim 4, characterized in that, The specific process of extracting transactions from the committed transaction set and the active transaction set in sequence and storing them in the checkpoint file corresponding to the transaction includes: Extract transactions from the committed transaction set in sequence, determine the checkpoint file X where the transaction is stored from the transaction information. If the transaction does not record the checkpoint file information, use the current checkpoint file A as the checkpoint file X where the transaction is stored. Append the operations of the transaction to the space starting from the offset N3 in checkpoint file X in the form of a doubly linked list, and record the storage address of the first operation in the transaction information; Extract transactions from the set of active transactions in sequence. Determine the checkpoint file X stored by the transaction from the transaction information. If the transaction does not record checkpoint file information, use the current checkpoint file A as the checkpoint file X stored by the transaction. Store the operations in the transaction whose operation LSN is less than or equal to the checkpoint LSN in the form of a doubly linked list in the checkpoint file X. The storage position is appended after the operations of the committed transactions. Record the storage addresses of the first and last operations and the information of the current checkpoint file X in the transaction information.

6. The method for parsing and synchronizing transaction storage of logs according to claim 5, characterized in that, Saving the active transaction information to the N1 - N2 active transaction reserved area in the checkpoint file A and saving the committed transaction information to the N2 - N3 committed transaction reserved area in the checkpoint file A specifically includes: Save the active transaction information to the N1 - N2 active transaction reserved area in the checkpoint file A. If the checkpoint file of the active transaction points to the current checkpoint file A, then for each active transaction saved, increment the active transaction reference count of the checkpoint file A by 1, and set the maximum transaction commit LSN of the checkpoint file to infinity. Save the committed transaction information to the N2 - N3 committed transaction reserved area in the checkpoint file A. Determine whether the active transaction reference count of the current checkpoint file A is 0. If it is, set the maximum transaction commit LSN of the checkpoint file to the maximum commit LSN among the saved committed transactions.

7. The method for parsing and synchronizing transaction storage of logs according to claim 3, characterized in that, When the target - side data synchronization service fails and restarts, load all checkpoint files, set the recovery point according to the checkpoint files, and implement the failure recovery function based on the recovery point. Specifically: When the target - side data synchronization service fails and restarts, load all checkpoint files and sort the checkpoint files according to the LSN value in the file name. Read the header of the last checkpoint file in the checkpoint file linked list, extract the saved checkpoint LSN. If the checkpoint LSN is not 0, use this checkpoint file as the recovery point; otherwise, discard this checkpoint file and use its previous checkpoint file as the recovery point. Read the LSN in the checkpoint file used as the recovery point, and restore the transaction information in the N1 - N2 active transaction space of the checkpoint file to the active transaction linked list. For each active transaction restored, increment the active transaction reference count of the checkpoint file pointed to by the transaction by 1, and then continue to receive the transaction operations sent by the source side. Compare the LSN of the operation with the LSN in the recovery - point checkpoint file, and only receive the operations whose operation LSN is greater than the LSN in the checkpoint file to implement the failure recovery function.

8. The method for log parsing and synchronous transaction storage according to claim 3, characterized in that, When cleaning up the checkpoint files, compare this LSN with the maximum transaction commit LSN of each checkpoint file in the checkpoint file linked list according to the maximum commit LSN of the synchronized transactions. When this LSN is greater than or equal to the maximum transaction commit LSN of the checkpoint file, it means that the checkpoint file has been synchronized and thus perform the cleanup.

9. A device for log parsing and synchronous transaction storage, characterized in that: Comprising at least one processor and a memory, the at least one processor and the memory are connected via a data bus, the memory stores instructions executable by the at least one processor, and after being executed by the processor, the instructions are used to complete the method for log parsing synchronization transaction storage described in any one of claims 1-8.

Images