Database parallel log replay method, device and equipment and readable storage medium

Abstract

The application provides a database parallel log replay method, device and equipment and a readable storage medium, and is applied to the technical field of computers. The database parallel log replay method comprises the following steps: analyzing a transaction log record stream sent by a master database to obtain starting time of multiple transactions, data operation positions of the multiple transactions and commit time of the multiple transactions; obtaining information that the multiple transactions have been completed by thread replay; determining multiple associated transactions which have intersection with the data operation position of a target transaction according to the data operation positions of the multiple transactions; determining the associated transactions which have starting time earlier than the starting time of the target transaction and commit time earlier than the commit time of the target transaction as preceding transactions of the target transaction; replaying the target transaction in the case that the preceding transactions have been completed by thread replay according to the information that the transactions have been completed by thread replay; and replaying the target transaction in the case that the target transaction has no preceding transaction.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a database parallel log playback method, apparatus, device, and readable storage medium. Background Technology

[0002] When backing up and managing data tables in a database, a primary and secondary setup is typically used. The primary server has a primary database, and the secondary server has a secondary database. The primary server records all data modification operations on the data tables sequentially in the form of log records. The secondary server continuously retrieves these log records and replays the log records on the data tables in its local secondary database, ensuring that the data tables in the secondary database are consistent with those in the primary database of the primary server.

[0003] To improve the efficiency of standby machine log playback, related technologies typically divide the standby machine's data table into multiple partitions in physical storage, with each partition bound to a thread. Each thread is responsible for the log playback of its corresponding partition to achieve parallel log playback. However, this method suffers from low thread resource utilization and inflexible thread scheduling. Summary of the Invention

[0004] In view of this, the present invention provides a database parallel log replay method, apparatus, device and readable storage medium.

[0005] The first aspect of the present invention provides a database parallel log replay method, comprising: parsing a transaction log recording stream sent by a master database to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions, wherein the commit time indicates that the transaction has been completed in the master database; obtaining information that multiple transactions have been replayed by threads; determining multiple related transactions that intersect with the data operation position of a target transaction based on the data operation position of the multiple transactions, wherein the intersection indicates that the data operation position of the target transaction and the data operation position of the related transactions overlap or conflict; determining the related transactions whose start time is earlier than the start time of the target transaction and whose commit time is earlier than the commit time of the target transaction as the prior transactions of the target transaction; replaying the target transaction if the prior transactions have been replayed based on the information that the transaction has been replayed by threads; and replaying the target transaction if there are no prior transactions of the target transaction.

[0006] According to an embodiment of the present invention, determining multiple associated transactions that intersect with the data operation positions of a target transaction based on the data operation positions of multiple transactions includes: determining multiple nodes corresponding to each of the multiple transactions from a transaction dependency graph, wherein the transaction dependency graph is obtained by establishing an association edge between two nodes corresponding to two transactions that intersect in the data operation positions; determining nodes in the transaction dependency graph that have an association edge with the node representing the target transaction as associated nodes; and determining the transactions corresponding to the associated nodes in the transaction dependency graph as associated transactions.

[0007] According to an embodiment of the present invention, the transaction dependency graph is constructed by the following operations: parsing the transaction log record stream sent by the master database to obtain the mapping relationship between transactions and the data operation positions of transactions; determining the data operation position set of multiple transactions according to the mapping relationship; creating multiple nodes corresponding to each of the multiple transactions; constructing an association edge between two nodes corresponding to two transactions whose data operation position sets have an intersection, wherein the association edge represents that there is an overlap and conflict of data operation positions between the two transactions corresponding to the two nodes.

[0008] According to an embodiment of the present invention, determining a set of data operation locations for multiple transactions based on a mapping relationship includes: parsing a transaction log record stream sent to the main database to determine multiple transactions; and mapping the data operation locations of the multiple transactions to the multiple transactions according to the mapping relationship to obtain a set of data operation locations for multiple transactions.

[0009] According to an embodiment of the present invention, overlapping conflicts are determined by the following operations: a preset data operation position comparison unit is set, wherein the data operation position comparison unit indicates the comparison range used to determine whether there is an intersection between the data operation positions of two transactions; if the data operation position comparison unit is determined to be a table unit, and the data operation positions of the two transactions both include the same table, then overlapping conflicts are determined between the data operation positions of the two transactions; if the data operation position comparison unit is determined to be a row unit, and the data operation positions of the two transactions both include the same row, then overlapping conflicts are determined between the data operation positions of the two transactions.

[0010] According to an embodiment of the present invention, replaying a target transaction includes: writing the target transaction as a transaction to be replayed into a cache of a replay execution queue; dynamically adjusting the replay strategy of the transaction to be replayed according to the number of transactions to be replayed in the replay execution queue; wherein the replay strategy indicates the replay order among multiple transactions to be replayed; and generating information to characterize that the target transaction has been replayed in response to determining that the replay of the target transaction has been completed.

[0011] According to an embodiment of the present invention, the replay strategy of the transactions to be replayed is dynamically adjusted according to the number of transactions to be replayed in the execution queue to be replayed, including: in response to determining that the number of transactions to be replayed is less than a predetermined threshold, determining the replay order of multiple transactions to be replayed according to the commit time of the transactions, and calling the target thread to serially replay multiple transactions to be replayed according to the replay order; in response to determining that the number of transactions to be replayed is greater than a predetermined threshold, calling the target thread to replay multiple transactions to be replayed in parallel.

[0012] A second aspect of the present invention provides a database parallel log replay apparatus, comprising:

[0013] The parsing module is used to parse the transaction log record stream sent by the master database to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions. The commit time indicates that the transaction has been completed in the master database.

[0014] The acquisition module is used to obtain information about multiple transactions that have been replayed and completed by threads;

[0015] The module for determining related transactions is used to identify multiple related transactions that intersect with the data operation position of the target transaction based on the data operation position of multiple transactions. The intersection indicates that there is an overlap or conflict between the data operation position of the target transaction and the data operation position of the related transactions.

[0016] The "Identify Prior Transactions" module is used to identify associated transactions whose start time is earlier than the start time of the target transaction and whose commit time is earlier than the commit time of the target transaction as prior transactions of the target transaction.

[0017] The first replay module is used to replay the target transaction based on the information that the transaction has been replayed by a thread, provided that the previous transaction has been replayed.

[0018] The second replay module is used to replay the target transaction if there is no preceding transaction.

[0019] A third aspect of the present invention provides an electronic device comprising:

[0020] One or more processors;

[0021] A memory is used to store one or more programs, wherein when one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the methods described above.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed, are used to implement the methods described above.

[0023] A fifth aspect of the present invention provides a computer program product including computer-executable instructions that, when executed, implement the methods described above.

[0024] According to embodiments of the present invention, the logical execution order of multiple transactions can be determined by parsing the transaction log stream sent by the master database to obtain the start time and commit time of multiple transactions. Based on the data operation positions of the multiple transactions obtained by parsing the transaction log stream sent by the master database, related transactions that intersect with the data operation positions of the target transaction, i.e., those with overlapping and conflicting data operation positions, can be identified. Based on the determined logical execution order of the multiple transactions, the preceding transaction of the target transaction can be determined among the related transactions of the target transaction. The preceding transaction represents the transaction that must be replayed before the target transaction to ensure data consistency after transaction replay. Furthermore, based on the pre-obtained information that multiple transactions have been replayed by threads, the target transaction is replayed if the preceding transaction has been replayed; otherwise, the target transaction is replayed. Since the replay of the target transaction no longer depends on whether other transactions have been replayed, multiple target transactions can be replayed in parallel by multiple threads, which improves the efficiency of transaction replay and increases the resource utilization of threads. Furthermore, since determining whether a target transaction can be replayed is based on whether the target transaction has prior transactions or whether those prior transactions have been replayed, the parallel replay of multiple target transactions can ensure data consistency and avoid data replay errors. Attached Figure Description

[0025] The above and other objects, features and advantages of the present invention will become more apparent from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0026] Figure 1 A diagram illustrating an application scenario of the database parallel log replay method according to an embodiment of the present invention is shown.

[0027] Figure 2 A flowchart of a database parallel log replay method according to an embodiment of the present invention is shown.

[0028] Figure 3 A schematic diagram of the transaction dependency graph of the database parallel log replay method according to an embodiment of the present invention is shown.

[0029] Figure 4 The diagram illustrates the active transaction representation intent of the database parallel log replay method according to an embodiment of the present invention.

[0030] Figure 5 A schematic diagram of the execution queue to be replayed in the database parallel log replay method according to an embodiment of the present invention is shown.

[0031] Figure 6 A structural block diagram of a database parallel log playback device according to an embodiment of the present invention is shown.

[0032] Figure 7 A block diagram of an electronic device suitable for implementing a database parallel log playback method according to an embodiment of the present invention is shown. Detailed Implementation

[0033] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0034] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0035] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0036] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0037] In the embodiments of this invention, the collection, updating, analysis, processing, use, transmission, provision, invention, and storage of data (e.g., including but not limited to user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures have been taken to prevent unauthorized access to user personal information data and to maintain the security of user personal information and network security.

[0038] In the embodiments of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0039] In related technologies, backup management of database tables typically employs a primary / standby architecture. The primary machine has a primary database, and the standby machine has a standby database. The primary machine sequentially records all data modification operations (e.g., insert, update, delete) on the tables as log entries. The standby machine continuously retrieves these log entries and replays them on its local standby database tables, ensuring consistency between the standby database and the primary database. When the standby machine's performance is insufficient to quickly replay the primary machine's logs, data latency occurs, posing the following risks: First, if the primary machine fails and the standby database becomes the primary database, the data changes corresponding to unreplayed log entries will be permanently lost due to the latency in replaying the primary machine's logs. Second, the primary machine's log files are reused cyclically. If the standby machine's replay is too slow, new logs from the primary machine may overwrite older logs that haven't yet been transmitted to the standby machine, causing a break in the synchronization link. To address these issues, related technologies often employ a parallel log replay method based on data sharding and thread binding to improve standby machine replay performance.

[0040] The parallel log replay method based on data sharding and thread binding works as follows: On the standby machine, the backup data table is divided into multiple partitions in physical storage, and multiple replay threads are created. Each thread is responsible for replaying log records from one or a group of partitions. For example, thread 1 is responsible for replaying log records involving partitions 1 and 2, and thread 2 is responsible for replaying log records involving partitions 3 and 4. The log dispatcher routes log records to the corresponding bound threads for replay based on the partition number carried in the log records. However, this parallel log replay method based on data sharding and thread binding suffers from several drawbacks. Since each thread corresponds to one or a group of partitions, and data access is not evenly distributed in actual business scenarios, some threads become very busy because their bound partitions involve many log records, while other threads become relatively idle because their bound partitions involve fewer log records. This results in inefficient use of thread computing resources, with the overall system throughput limited by the busiest thread, significantly reducing the effectiveness of parallel replay. Furthermore, the system's maximum parallel capability is also limited by the number of data partitions. For example, on a standby machine with 32 threads, if the backup data table is only divided into 8 partitions, only 8 threads can participate in the playback, while the remaining 24 threads are idle and cannot contribute to the backup playback. Furthermore, this parallel log playback method based on data sharding and thread binding suffers from poor adaptability because threads are bound to partitions and cannot be dynamically adjusted during peak business periods.

[0041] Therefore, the technical problem with the parallel log replay method based on data sharding and thread binding stems from its approach of performing parallel log replay based on data table sharding. The essential requirement of log replay is to guarantee the logical order of transactions, i.e., temporal causality. When two transactions modify completely unrelated data, they are logically independent, and it is unnecessary to consider which data table partitions the two transactions involve; they can be replayed independently and in parallel without affecting the consistency of the replayed data. However, the parallel log replay method based on data sharding and thread binding in related technologies cannot utilize this independence of transactions. Instead, it uses data sharding of the data table followed by thread binding for parallel log recording and replay. Therefore, due to uneven data access distribution, the parallel replay efficiency of the database in actual business applications is low.

[0042] Embodiments of the present invention provide a database parallel log replay method, apparatus, device, and readable storage medium. Instead of using a parallel log replay method bound to data sharding threads, it performs parallel log replay on a transaction-by-transaction basis. Specifically, it includes: parsing the transaction log stream sent by the master database to identify multiple related transactions whose data operation positions intersect with the target transaction; further, determining the earlier transaction from the related transactions based on the transaction's start and commit times; thus identifying the earlier transaction that intersects with the target transaction's data operation positions and is earlier than the target transaction; and replaying the target transaction based on pre-obtained information that multiple transactions have been replayed by threads, either when the earlier transaction has been replayed or when the target transaction does not have an earlier transaction. Based on the above two scenarios, it can be determined that multiple target transactions do not affect each other during replay. This enables multiple target transactions to be replayed in parallel by multiple threads, with each thread independently responsible for replaying one transaction. This improves the utilization of thread resources, breaks through the limitation of the number of physical partitions on the degree of parallelism, and increases the log replay throughput of the standby machine. Since the earlier transactions that have data operation positions that intersect with the target transactions have been replayed earlier than the target transactions, the data consistency after replay is guaranteed.

[0043] Figure 1 A diagram illustrating an application scenario of the database parallel log replay method according to an embodiment of the present invention is provided. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to embodiments of the present invention, in order to help those skilled in the art understand the technical content of the present invention, but do not mean that embodiments of the present invention cannot be used in other devices, systems, environments or scenarios.

[0044] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a host 101, a device 102, and a standby device 103.

[0045] Among them, host 101 is used to deploy the master database, device 102 is used to deploy the database parallel log replay method of the present invention, and standby 103 is used to deploy the standby database.

[0046] The host 101, device 102 and backup 103 interact through a communication network. The host 101 can be a computer terminal or server, such as a desktop computer or a data storage server; the device 102 can be a computer terminal or server, such as a desktop computer or a server; and the backup 103 can be a computer terminal or server, such as a desktop computer or a backup storage server.

[0047] The primary database deployed in host 101 sends transaction log recording streams to device 102.

[0048] Device 102 parses the transaction log stream sent by the main database to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions.

[0049] After determining the related transactions of the target transaction based on the parsing results, device 102 further determines the preceding transactions of the target transaction from the related transactions, and determines whether to replay the target transaction based on the replay completion status of the preceding transactions.

[0050] The backup database installed in the standby machine 103 of the device 102 can replay multiple target transactions in parallel, realizing the backup and replay of the backup database in the standby machine 103.

[0051] It should be understood that Figure 1 The number of host 101, device 102, and standby 103 shown is merely illustrative. Depending on implementation needs, any number of host 101, device 102, and standby 103 can be included.

[0052] Figure 2 A flowchart of a database parallel log replay method according to an embodiment of the present invention is shown.

[0053] like Figure 2 As shown in the embodiment of the present invention, a database parallel log replay method includes operations S210~S260.

[0054] In operation S210, the transaction log stream sent by the master database is parsed to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions.

[0055] The commit time indicates that the transaction has been completed in the main database.

[0056] The primary database is used to store the database data in use, the backup database backs up the data in the primary database, and the transaction log records all operations of transactions executed in the primary database. Operations include Data Manipulation Language (DML) operations, such as insert, update, and delete operations.

[0057] A transaction log stream is a sequence of transaction log records generated by the primary database. The start time of a transaction indicates the beginning of that transaction. This start time is determined by parsing the timestamp from the transaction log record that contains the start marker. For example, in the transaction log stream, the start time of a transaction is identified by its first data operation record, which includes the transaction ID, timestamp, and other information. The start times of multiple transactions are used to determine the execution order of multiple events.

[0058] The data operation position of a transaction determines where the transaction performs data operations. This can be at the table level or the row level. For example, the data operation position might be where the transaction operates on a target table or a target row. The data operation positions of multiple transactions determine if there is any overlap in their data operation positions. The transaction commit time indicates whether the transaction has been committed in the primary database. The commit times of multiple transactions indicate the order in which they were committed in the primary database.

[0059] For example, parsing the transaction log stream sent by the master database can yield the data operation location of a transaction, such as the set of data partition numbers involved in the operation; the specific DML operation content of the transaction, such as insert, update, or delete operations; the transaction commit flag, such as the commit time of the parsed transaction; and the start time of the transaction, such as the timestamp contained in the transaction start record.

[0060] In one implementation, the transaction log stream sent by the master database is parsed online in real time to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions.

[0061] In another implementation, the transaction logs sent by the master database are saved as local log files and then parsed to obtain the start time of multiple transactions, the data operation positions of multiple transactions, and the commit time of multiple transactions.

[0062] During operation S220, information was obtained that multiple transactions had been replayed and completed by threads.

[0063] Threads are used to replay transactions to achieve data backup; a thread can be a worker thread in the central processing unit. For example, the central processing unit has 4 worker threads, and the 4 worker threads can replay 4 transactions one by one.

[0064] In operation S230, based on the data operation positions of multiple transactions, several related transactions that intersect with the data operation positions of the target transaction are identified. These multiple transactions include the target transaction and the several related transactions whose data operation positions intersect with its own.

[0065] The intersection indicates that there is an overlap or conflict between the data operation locations of the target transaction and the data operation locations of related transactions.

[0066] Intersection is used to determine whether two transactions have operated on the same data location in a data table, thus identifying any overlap or conflict in the data operations performed by the two transactions. For example, intersection can include two transactions operating on the same data table, or two transactions operating on the same row in the same data table.

[0067] Related transactions are used to represent transactions whose data operations intersect with those of the target transaction, i.e., transactions that overlap or conflict with the target transaction. For example, if a transaction operates on the same data table as the target transaction, then this transaction is a related transaction of the target transaction. Related transactions indicate that there is overlap or conflict in the data operation locations of the target transaction and the related transaction, meaning that there is duplicate database access when the target transaction and the related transaction are processed. Therefore, it is necessary to further determine whether the target transaction can be executed and replayed by a thread based on the order of the target transaction and the related transactions.

[0068] In one implementation, based on the data tables involved in multiple transaction data operations, multiple related transactions are identified that overlap with the data tables involved in the target transaction data operation.

[0069] In another implementation, multiple related transactions that overlap with the data rows involved in the target transaction are identified based on the data rows involved in multiple transaction data operations.

[0070] In operation S240, related transactions whose start time is earlier than the start time of the target transaction and whose commit time is earlier than the commit time of the target transaction are identified as the earlier transactions of the target transaction.

[0071] The start time indicates when a transaction begins execution in the primary database, and the commit time indicates when it completes. These times determine the execution order of multiple transactions in the primary database. Based on this order, the replay execution order of these transactions in the backup database is determined, ensuring data consistency after transaction replay. "Prior transactions" identifies related transactions that need to be replayed earlier than the target transaction. Since prior transactions are related transactions whose start and commit times both precede the target transaction's, if there are overlaps or conflicts between the data operations of prior transactions and the target transaction, all prior transactions must be replayed first to guarantee data consistency after replaying the target transaction and prevent data replay errors such as dirty reads.

[0072] In one implementation, based on parsing the transaction log stream sent by the primary database to obtain the start time and commit time of multiple transactions, a transaction execution record table is established locally in the backup database. According to the transaction execution record table, related transactions whose start time is earlier than the start time and commit time is earlier than the commit time of the target transaction are identified as the prior transactions of the target transaction.

[0073] In another implementation, an index is created based on parsing the transaction log stream sent by the master database to obtain the start time and commit time of multiple transactions. The index is then used to identify related transactions whose start time is earlier than the start time and commit time is earlier than the commit time of the target transaction as the prior transactions of the target transaction.

[0074] In operation S250, based on the information that the transaction has been replayed by the thread, the target transaction is replayed if the previous transaction has been replayed.

[0075] In operation S260, if there is no prior transaction to the target transaction, the target transaction is replayed.

[0076] The "Transaction Completed by Thread Replay" information is used to determine which transactions have been replayed and completed by threads in the backup database. This information is generated after a thread finishes replaying the target transaction. Based on this information, it can be determined whether any preceding transactions of the target transaction have been replayed and completed by threads. If a preceding transaction has been replayed and completed, the target transaction can be replayed. This replay logic follows the causal relationship in transaction execution time, thus ensuring data consistency after transaction replay.

[0077] If there are no preceding transactions in the target transaction, it can be determined that the target transaction is an independent transaction. The execution replay of an independent transaction does not depend on the constraint that other transactions must be replayed in advance. Therefore, if there are no preceding transactions in the target transaction, the target transaction can be executed and replayed by a thread.

[0078] In one implementation, there can be multiple target transactions that have been replayed before, and these multiple target transactions can correspond one-to-one with multiple threads, which can then replay them in parallel.

[0079] In another implementation, there can be multiple target transactions that do not have prior transactions, and these multiple target transactions that can be replayed can correspond one-to-one with multiple threads and be replayed in parallel.

[0080] According to embodiments of this application, the logical execution order of multiple transactions can be determined based on the start times and commit times of multiple transactions obtained by parsing the transaction log stream sent to the master database. Furthermore, based on the data operation positions of multiple transactions obtained by parsing the transaction log stream sent to the master database, related transactions that intersect with the data operation positions of the target transaction, i.e., those with overlapping and conflicting data operation positions, can be identified. Based on the determined logical execution order of the multiple transactions, the preceding transaction of the target transaction can be determined among its related transactions; the preceding transaction represents the transaction that must be replayed before the target transaction to ensure data consistency after transaction replay. Based on the pre-obtained information that multiple transactions have been replayed by threads, the target transaction is replayed if the preceding transaction has been replayed; otherwise, the target transaction is replayed. Since the replay of the target transaction does not depend on whether other transactions have been replayed successfully, multiple target transactions can be replayed in parallel by multiple threads. This transaction-based log replay method fully mobilizes each thread to participate in the replay of the target transaction, eliminating the problems of uneven utilization of thread computing resources and the limitation of replay capability to the number of partitions after threads are bound to partitions in related technologies. Furthermore, determining whether a target transaction can be replayed is based on whether the target transaction has prior transactions or whether those prior transactions have been replayed successfully. Therefore, after replaying multiple target transactions in parallel, data consistency can be guaranteed, avoiding data replay errors such as dirty reads.

[0081] According to an embodiment of the present invention, the overlap conflict is determined by the following operation:

[0082] A preset data operation position comparison unit is defined, which indicates the comparison range used to determine whether there is an intersection between the data operation positions of two transactions.

[0083] If the unit of comparison for determining the data operation location is the table unit, and both transactions involve the same table in their data operation locations, then the data operation locations of the two transactions are found to overlap and conflict.

[0084] If the comparison unit for determining the data operation location is the row unit, and both transactions' data operation locations include the same row, then the data operation locations of the two transactions are determined to have an overlap conflict.

[0085] In one implementation, the data operation location comparison unit can be manually preset according to the preset targets for the performance and system overhead of this method.

[0086] In another implementation, the data operation location comparison unit can be adaptively adjusted by monitoring the performance and system overhead of this method. For example, when the system overhead of this method is monitored to be relatively high, the data operation location comparison unit can be selected as the table unit to reduce the amount of computation for comparison and reduce the system overhead of this method.

[0087] According to embodiments of the present invention, different data operation position comparison units can be preset, and different preset data operation position comparison units can have different matching relationships between performance and system overhead. Therefore, preset different data operation position comparison units enable the method to have the ability to balance performance and system overhead, thereby improving the practical application effect of the method.

[0088] Figure 3 A schematic diagram of the transaction dependency graph of the database parallel log replay method according to an embodiment of the present invention is shown.

[0089] According to an embodiment of the present invention, based on the data operation positions of multiple transactions, determining multiple related transactions that intersect with the data operation positions of the target transaction includes:

[0090] Multiple nodes corresponding to each transaction are determined from the transaction dependency graph. The transaction dependency graph is obtained by establishing an association edge between two nodes corresponding to two transactions that have an intersection in the data operation position.

[0091] Nodes in the transaction dependency graph that have associated edges with the node representing the target transaction are identified as associated nodes;

[0092] The transactions corresponding to the associated nodes in the transaction dependency graph are identified as associated transactions.

[0093] like Figure 3 As shown, a transaction dependency graph is a directed acyclic graph (DAG). It is used to represent multiple transactions and the overlapping or conflicting data operations between them, using multiple nodes and the edges that connect them. For example... Figure 3 In this context, nodes T1, T2, and T3 represent transactions T1, T2, and T3, respectively. There is an edge connecting nodes T1 and T3, indicating that transactions T1 and T3 have overlapping and conflicting data operation locations; that is, their data operation locations intersect. The edge also indicates that the start time of transaction T1 is earlier than the start time of transaction T3.

[0094] like Figure 3 As shown, for example, the nodes T1, T2, and T3 corresponding to transactions T1, T2, and T3 are determined from the transaction dependency graph.

[0095] Transaction T1 is the target transaction, and node T3 in the transaction dependency graph that has an associated edge with node T1 is identified as the associated node;

[0096] The transaction T3 corresponding to node T3 in the transaction dependency graph is identified as the associated transaction.

[0097] like Figure 3 As shown, based on the transaction dependency graph, it can be determined that because the data operations of transactions T1 and T3 overlap, and the start time of the transaction at node T1 precedes the start time of the transaction at node T3, transaction T3 can only be replayed after the replay execution of transaction T1. This ensures data consistency after the transaction replay. Furthermore, as... Figure 3 As shown, node T2 has no associated edges with either node T1 or node T3, indicating that node T2 is an independent node. That is, the transaction T2 represented by node T2 has no prior transactions, so transaction T2 can be directly executed and replayed by the thread.

[0098] According to an embodiment of the present invention, by utilizing the nodes corresponding to the transactions indicated in the transaction dependency graph and the associated edges existing between the nodes, the associated nodes of the target transaction can be accurately determined. Furthermore, by using the correspondence between the associated nodes and the transactions, the associated transactions of the target transaction can be accurately determined. Therefore, by utilizing the transaction dependency graph, the associated transactions of the target transaction can be quickly and accurately determined, thereby improving the accuracy and efficiency of determining the associated transactions of the target transaction.

[0099] According to an embodiment of the present invention, the transaction dependency graph is constructed through the following operations:

[0100] By parsing the transaction log stream sent by the main database, the mapping relationship between transactions and the data operation locations of transactions can be obtained.

[0101] The mapping relationship is used to represent the correspondence between the data operation position of a transaction and its parent transaction after parsing the transaction log record stream sent by the main database. The data operation position of a transaction includes the information of the parent transaction. For example, if the data operation position of a transaction is that transaction T1 operates on data table P1, then the data operation position (data table P1) is mapped to transaction T1, and the correspondence between transaction T1 and the data operation position (data table P1) is obtained.

[0102] Based on the mapping relationship, determine the set of data operation locations for multiple transactions;

[0103] For example, based on the mapping relationship between transaction T1 and multiple data operation locations, the multiple data operation locations involved in transaction T1 are determined, and the set of data operation locations is the set of multiple data operation locations involved in transaction T1.

[0104] Create multiple nodes corresponding to each of the multiple transactions.

[0105] In a transaction dependency graph, multiple transactions correspond to multiple nodes, and each transaction corresponds to a unique node. For example, nodes T1, T2, and T3 respectively represent transactions T1, T2, and T3.

[0106] Construct an association edge between two nodes corresponding to two transactions whose data operation locations intersect. The association edge represents the overlap and conflict of data operation locations between the two transactions corresponding to the two nodes.

[0107] An association edge is constructed between two nodes whose data operation positions intersect. For example, transaction T1 operates on data table P1, so the data operation position set of transaction T1 includes data table P1; transaction T3 also operates on data table P1, so the data operation position set of transaction T3 includes data table P1. Therefore, the data operation positions of transaction T1 and transaction T3 intersect. Furthermore, an association edge is established between node T1 corresponding to transaction T1 and node T3 corresponding to transaction T3, indicating that there is an overlap and conflict in the data operation positions of transactions T1 and T3 corresponding to node T1 and node T3.

[0108] In one implementation, after creating multiple nodes corresponding to each of the multiple transactions, each node can be traversed and associated edges can be established one by one.

[0109] In another implementation, after creating multiple nodes corresponding to each of the multiple transactions, an association edge can be established between the two corresponding nodes based on the existence of overlapping conflicts between the two transactions.

[0110] According to an embodiment of this application, based on parsing the transaction log record stream sent by the master database, the mapping relationship between transactions and the data operation positions of transactions is obtained; based on the mapping relationship, a set of data operation positions of multiple transactions is determined; after creating multiple nodes corresponding to each of the multiple transactions, an association edge is constructed between the two nodes corresponding to two transactions that have an intersection in the set of data operation positions of multiple transactions. Therefore, the transaction dependency graph is established based on the parsed multiple transactions and the set of data operation positions of transactions, which has the advantage of being able to accurately represent multiple transactions and the overlapping conflicts of data operation positions between transactions, and facilitates the quick and accurate judgment of overlapping conflicts of data operation positions between transactions.

[0111] Figure 4 The diagram illustrates the active transaction representation intent of the database parallel log replay method according to an embodiment of the present invention.

[0112] Based on the mapping relationship, determine the set of data operation locations for multiple transactions, including:

[0113] Multiple transactions are identified by parsing the transaction log stream sent by the main database;

[0114] For example, parsing the received transaction log records can yield information such as transaction T1, transaction T2, or the transaction ID of transaction T2, as well as the location of data operations involved in the transaction, such as transaction T1 operating on data table P1.

[0115] The parsing determines multiple transactions used to represent the multiple transactions involved.

[0116] The data operation locations of multiple transactions are mapped to multiple transactions according to the mapping relationship, resulting in a set of data operation locations for multiple transactions.

[0117] For example, a transaction can involve multiple data operation locations. The multiple data operation locations obtained by parsing the same transaction can be combined to form the data operation location set for that transaction. For instance, if transaction T1 involves operations on data tables P1 and P2, then the data operation location set for transaction T1 can consist of data tables P1 and P2.

[0118] In one implementation, such as Figure 4 As shown, the active transaction table can be used to record multiple transactions that have been parsed and the locations of data operations within those transactions.

[0119] like Figure 4 As shown, after the start time of transaction T1 is parsed, an entry corresponding to transaction T1 is created in the active transaction table. When transaction T1 involves operations on data table P1, data table P1 is recorded in the data operation position set under the entry corresponding to transaction T1. Similarly, after the start time of transaction T2 is parsed, an entry corresponding to transaction T2 is created in the active transaction table. When transaction T2 involves operations on data table P3, data table P3 is recorded in the data operation position set under the entry corresponding to transaction T2. ​​The same process is repeated after the start time of transaction T3 is parsed, creating an entry corresponding to transaction T3 in the active transaction table. When transaction T3 involves operations on data table P1, data table P1 is recorded in the data operation position set under the entry corresponding to transaction T3. This process continues until an active transaction table containing the data operation position sets of multiple transactions is obtained.

[0120] Based on the active transaction table, identify multiple related transactions that intersect with the data operation locations of the target transaction. Furthermore, create a relationship edge in the transaction dependency graph that points from the target transaction to the related transactions.

[0121] The transaction log stream sent by the master database is parsed to obtain the transaction commit time. The transaction is then marked as committed in the active transaction table and the scheduler is notified. The scheduler uses the transaction dependency graph to determine whether the target transaction can be written into the cache of the execution queue to be replayed.

[0122] According to an embodiment of the present invention, the active transaction table can be used to record multiple parsed transactions and the data operation positions of multiple transactions in real time. Therefore, based on the multiple transactions and the data operation positions of multiple transactions recorded in the active transaction table, multiple related transactions that have an intersection with the data operation positions of the target transaction can be accurately and quickly determined.

[0123] Figure 5 A schematic diagram of the execution queue to be replayed in the database parallel log replay method according to an embodiment of the present invention is shown.

[0124] According to an embodiment of this method, replaying the target transaction includes:

[0125] Write the target transaction as a transaction to be replayed into the cache of the execution queue to be replayed.

[0126] like Figure 5 As shown, target transactions T1, T2, and T3 are all determined to be replayable target transactions, and target transactions T1, T2, and T3 are written into the cache of the execution queue to be replayed in sequence.

[0127] In one implementation, such as Figure 5 As shown, threads 1, 2, and 3 can replay target transactions T1, T2, and T3 respectively, achieving parallel replay. Thread 4 is an idle thread that can queue up and wait to replay transactions that have been newly written into the execution queue by the scheduler.

[0128] The replay strategy for transactions to be replayed is dynamically adjusted based on the number of transactions to be replayed in the execution queue; the replay strategy indicates the replay order among multiple transactions to be replayed.

[0129] The replay strategy is used to determine the replay method for multiple transactions based on the number of transactions in the replay queue. The replay order includes either serial replay or parallel replay.

[0130] In response to determining that the replay of the target transaction has been completed, information is generated to indicate that the target transaction has been replayed successfully.

[0131] The message "Target transaction has been replayed and completed" is used to determine that the corresponding transaction has been replayed and completed in the backup database.

[0132] According to an embodiment of this method, the scheduler obtains information that multiple transactions have been completed by thread replay. At the same time, the scheduler continuously monitors the state of the transaction dependency graph. When the scheduler obtains information that a transaction has been completed by thread replay, it marks the corresponding node as completed in the transaction dependency graph. The scheduler checks whether the nodes corresponding to all prior transactions of the target transaction have been marked as completed. When all the nodes corresponding to prior transactions have been marked as completed, the scheduler writes the target transaction as a transaction to be replayed into the cache of the execution queue to be replayed.

[0133] According to an embodiment of the present invention, the replay strategy of the transactions to be replayed is dynamically adjusted based on the number of transactions to be replayed in the execution queue. This allows for the application of different replay strategies to replay the transactions according to the number of transactions to be replayed, thereby improving the actual effectiveness of the method in different application scenarios.

[0134] According to an embodiment of the present invention, the replay strategy of transactions to be replayed is dynamically adjusted based on the number of transactions to be replayed in the execution queue, including:

[0135] In response to the determination that the number of transactions to be replayed is less than a predetermined threshold, the replay order of multiple transactions to be replayed is determined according to the commit time of the transactions, and the target thread is called to replay multiple transactions to be replayed serially according to the replay order.

[0136] When the number of transactions to be replayed is less than the predetermined threshold, it means that the number of transactions to be replayed is limited. In this case, calling the target thread and replaying multiple transactions to be replayed in sequence according to the replay order can meet the backup quality requirements of the backup database, without having to call too many threads to replay transactions.

[0137] In response to determining that the number of transactions to be replayed is greater than a predetermined threshold, the target thread is invoked to replay multiple transactions to be replayed in parallel.

[0138] For example, the predetermined threshold can be set to 1000 transactions.

[0139] In one implementation, multiple homogeneous threads form an elastic replay worker pool. These threads are not bound to any specific partition. The multiple threads retrieve transactions to be replayed from the cache of the execution queue to be replayed, and then replay all log records of the transactions to be replayed completely.

[0140] When a new transaction is added to the execution queue, the idle thread replays all transaction log records of the new transaction. If a thread fails to replay a transaction, the transaction is put back to the front of the execution queue for other threads to retry.

[0141] When the number of transactions to be replayed exceeds a predetermined threshold, it indicates that the transactions to be replayed may be in a backlog. At this time, multiple target threads are called to replay multiple transactions to be replayed in parallel, ensuring that the backup quality requirements of the backup database are met.

[0142] According to the embodiments of this method, when the number of transactions to be replayed is greater than a predetermined threshold, multiple target threads are invoked to replay multiple transactions in parallel; when the number of transactions to be replayed is less than the predetermined threshold, fewer threads are invoked to replay multiple transactions in serial order, reducing scheduling overhead and achieving a balance between backup quality efficiency and thread resource consumption. This allows the method to achieve a balance between backup efficiency and thread resource consumption in different application scenarios.

[0143] For example, an embodiment of the present invention discloses a database parallel log replay method and compares it with related methods as follows:

[0144] The application scenarios are as follows:

[0145] The database has three partitions: P1, P2, and P3. P1 represents the user table, P2 represents the order table, and P3 represents the product table.

[0146] Transaction T1: Insert a user record into data table P1 and insert the user's initial order into data table P2 (involving data tables P1 and P2).

[0147] Transaction T2: Update the price of a product in data table P3 (product table).

[0148] Transaction T3: Query and update a user record in data table P1 (data table P1 is involved).

[0149] The order in which the three transactions begin is transaction T1, transaction T2, and transaction T3; the order in which they are committed is also transaction T1, transaction T2, and transaction T3.

[0150] The relevant technology employs a method where three threads are bound one-to-one with data tables P1, P2, and P3 respectively to achieve transaction replay, specifically as follows:

[0151] Thread A is bound to data table P1 and needs to serially replay the operations related to data table P1 in transactions T1 and T3.

[0152] Thread B is bound to data table P2 and replays the operations of transaction T1 on data table P2.

[0153] Thread C is bound to data table P3, and replays the operations related to data table P3 in transaction T2.

[0154] Therefore, in related technologies, even if transaction T2 is completely unrelated to transactions T1 and T3, it is still necessary to replay transactions T1, T2, and T3 one by one according to the start and commit order of the three transactions. It is impossible to recognize the independence of transaction T2 and replay transaction T2 in a timely manner.

[0155] An embodiment of the present invention discloses a database parallel log replay method. The replay process is as follows:

[0156] When transaction T1 begins, the active transaction table records transaction T1. Transaction T1 modifies data tables P1 and P2, and the set of data operation locations recorded by transaction T1 is {P1, P2}.

[0157] Transaction T2 begins, and the active transaction table records transaction T2. ​​Transaction T2 modifies data table P3, and the data operation location set of transaction T2 is recorded as {P3}.

[0158] Since there is no conflict between transaction T2 and transaction T1, transaction T2 is an independent node in the transaction dependency graph.

[0159] Transaction T3 begins, and the active transaction table records transaction T3. Transaction T3 modifies data table P1, and the data operation location set of transaction T3 is recorded as {P1}.

[0160] Transaction T3 and transaction T1 have the same data operation location for data table P1, so an association edge is established in the transaction dependency graph from transaction T1 to transaction T3.

[0161] At this point, the state of the transaction dependency graph includes three nodes corresponding to transaction T1, transaction T2, and transaction T3. The node corresponding to transaction T2 has no associated edges, while the node corresponding to transaction T1 has an associated edge pointing to the node corresponding to transaction T3.

[0162] When the primary database commits transaction T1, the scheduler receives the committed information for transaction T1 and records transaction T1 as committed in the active transaction table.

[0163] The scheduler determines, based on the transaction dependency graph, to write transactions T1 and T2 as transactions to be replayed into the cache of the execution queue to be replayed.

[0164] Two idle threads in the worker thread pool can replay transactions T1 and T2 in parallel.

[0165] After transaction T1 and transaction T2 are replayed, a message indicating that the replay has been completed is generated.

[0166] The scheduler writes transaction T3 as a transaction to be replayed into the cache of the execution queue based on the transaction dependency graph.

[0167] An idle thread in the worker thread pool replays transaction T3. After replaying, it generates a message indicating that transaction T3 has been replayed successfully.

[0168] Therefore, the embodiments of the invention disclose a parallel log replay method for databases, which can identify T2 as an independent transaction and execute replay without waiting for the replay of a prior transaction that does not have the same data operation position to be completed. Compared with related technologies, it has better replay execution efficiency.

[0169] Figure 6 A structural block diagram of a database parallel log playback device according to an embodiment of the present invention is shown.

[0170] like Figure 6 As shown, the database parallel log replay device includes a parsing module 610, an acquisition module 620, a related transaction determination module 630, a prior transaction determination module 640, a first replay module 650, and a second replay module 660.

[0171] The parsing module 610 is used to parse the transaction log record stream sent by the master database to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions.

[0172] The commit time indicates that the transaction has been completed in the main database.

[0173] The acquisition module 620 is used to obtain information about multiple transactions that have been replayed and completed by threads.

[0174] The associated transaction determination module 630 is used to determine multiple associated transactions that intersect with the data operation positions of the target transaction based on the data operation positions of multiple transactions.

[0175] The intersection indicates that there is an overlap or conflict between the data operation location of the target transaction and the data operation location of the related transaction.

[0176] The prior transaction determination module 640 is used to determine the associated transactions whose start time is earlier than the start time of the target transaction and whose commit time is earlier than the commit time of the target transaction as prior transactions of the target transaction.

[0177] The first replay module 650 is used to replay the target transaction based on the information that the transaction has been replayed by a thread, provided that the previous transaction has been replayed.

[0178] The second replay module 660 is used to replay the target transaction if there is no prior transaction to the target transaction.

[0179] The module 630 for determining related transactions also includes: a submodule for determining nodes, a submodule for determining related nodes, and a submodule for determining related transactions.

[0180] The node determination submodule is used to determine the multiple nodes corresponding to each of the multiple transactions from the transaction dependency graph.

[0181] The transaction dependency graph is obtained by establishing an association edge between two nodes corresponding to two transactions that have an intersection in the data operation position, with the transaction as the node.

[0182] The "Identify Associated Nodes" submodule is used to identify nodes in the transaction dependency graph that have associated edges with the node representing the target transaction as associated nodes.

[0183] The "Identify Related Transactions" submodule is used to identify the transactions corresponding to the related nodes in the transaction dependency graph as related transactions.

[0184] The database parallel log replay device also includes a building module.

[0185] The construction module is used to parse the transaction log record stream sent to the main database to obtain the mapping relationship between transactions and their data operation positions; determine the data operation position set of multiple transactions based on the mapping relationship; create multiple nodes corresponding to each of the multiple transactions; and construct an association edge between the two nodes corresponding to two transactions whose data operation position sets have an intersection.

[0186] Among them, the associated edge represents the overlapping conflict of data operation positions between the two transactions corresponding to the two nodes.

[0187] The building blocks also include: a transaction determination submodule and a mapping submodule.

[0188] The transaction identification submodule is used to identify multiple transactions by parsing the transaction log stream sent to the main database.

[0189] The mapping submodule is used to map the data operation locations of multiple transactions to multiple transactions according to the mapping relationship, so as to obtain a set of data operation locations of multiple transactions.

[0190] The database parallel log replay device also includes a module for identifying overlapping conflicts.

[0191] The overlapping conflict detection module is used to preset the comparison unit for data operation positions. If the comparison unit for data operation positions is a table unit, and the data operation positions of two transactions both include the same table, then the overlapping conflict for the data operation positions of the two transactions is determined. If the comparison unit for data operation positions is a row unit, and the data operation positions of two transactions both include the same row, then the overlapping conflict for the data operation positions of the two transactions is determined.

[0192] The data operation location comparison unit indicates the comparison range used to determine whether there is an overlap in the data operation locations of two transactions.

[0193] The database parallel log replay device also includes a module for replaying target transactions;

[0194] The target transaction replay module is used to write the target transaction as a transaction to be replayed into the cache of the execution queue to be replayed; dynamically adjust the replay strategy of the transaction to be replayed according to the number of transactions to be replayed in the execution queue; and generate information to indicate that the target transaction has been replayed in response to determining that the replay of the target transaction has been completed.

[0195] The replay strategy specifies the replay order among multiple transactions to be replayed.

[0196] The replay target transaction module includes a dynamic adjustment submodule.

[0197] The dynamic adjustment submodule is used to determine the replay order of multiple transactions based on the transaction commit time when the number of transactions to be replayed is less than a predetermined threshold, and to call the target thread to replay multiple transactions in sequence according to the replay order; and to call the target thread to replay multiple transactions in parallel when the number of transactions to be replayed is greater than a predetermined threshold.

[0198] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention, or at least part of the functions of any one or more of them, can be implemented in a single module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be implemented by being divided into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, and firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present invention can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0199] For example, any and more of the following modules can be implemented in a single module / unit / subunit: parsing module 610, acquisition module 620, related transaction determination module 630, prior transaction determination module 640, first playback module 650, and second playback module 660. Alternatively, any one of these modules / units / subunits can be split into multiple modules / units / subunits. Or, at least some of the functionality of one or more of these modules / units / subunits can be combined with at least some of the functionality of other modules / units / subunits and implemented in a single module / unit / subunit. According to embodiments of the present invention, at least one of the parsing module 610, the acquisition module 620, the associated transaction determination module 630, the prior transaction determination module 640, the first playback module 650, and the second playback module 660 can be at least partially implemented as hardware circuits, such as field-programmable gate arrays (FPGAs), programmable logic arrays (PLAs), systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits (ASICs), or any other reasonable means of integrating or packaging circuits, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the parsing module 610, the acquisition module 620, the associated transaction determination module 630, the prior transaction determination module 640, the first playback module 650, and the second playback module 660 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0200] It should be noted that the data processing system part in the embodiments of the present invention corresponds to the data processing method part in the embodiments of the present invention. The specific description of the data processing system part is referred to in the data processing method part, and will not be repeated here.

[0201] Figure 7 A block diagram of an electronic device suitable for implementing the methods described above, according to an embodiment of the present invention, is shown. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0202] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present invention includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0203] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0204] According to an embodiment of the present invention, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include an input section 706 connected to one or more of the following components: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0205] According to embodiments of the present invention, the method flow according to embodiments of the present invention can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by processor 701, it performs the functions defined in the system of the embodiments of the present invention. According to embodiments of the present invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0206] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0207] For example, according to embodiments of the present invention, a computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0208] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the database parallel log playback method provided in the embodiments of the present invention.

[0209] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this embodiment of the invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0210] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0211] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0212] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A database parallel log replay method, characterized in that, The database parallel log replay method includes: The transaction log stream sent by the master database is parsed to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions, wherein the commit time indicates that the transaction has been completed in the master database; Retrieve information indicating that multiple transactions have been replayed and completed by threads; Based on the data operation positions of multiple transactions, identify multiple related transactions that intersect with the data operation position of the target transaction, wherein the intersection indicates that the data operation position of the target transaction and the data operation position of the related transactions overlap or conflict. The associated transactions whose start time is earlier than the start time of the target transaction and whose commit time is earlier than the commit time of the target transaction are determined as the prior transactions of the target transaction; Based on the information that the transaction has been replayed and completed by the thread, the target transaction is replayed if the prior transaction has been replayed and completed. If the target transaction does not have a preceding transaction, the target transaction is replayed.

2. The database parallel log replay method according to claim 1, characterized in that, The step of determining multiple related transactions that intersect with the data operation positions of the target transaction based on the data operation positions of multiple transactions includes: Multiple nodes corresponding to each of the multiple transactions are determined from the transaction dependency graph, wherein the transaction dependency graph is obtained by establishing an association edge between two nodes corresponding to two transactions that have an intersection at the data operation position, with the transaction as the node; The nodes in the transaction dependency graph that have the associated edge with the node representing the target transaction are identified as associated nodes; The transaction corresponding to the associated node in the transaction dependency graph is identified as the associated transaction.

3. The database parallel log replay method according to claim 2, characterized in that, The transaction dependency graph is constructed through the following operations: Based on parsing the transaction log record stream sent by the main database, the mapping relationship between the transaction and the data operation position of the transaction is obtained; Based on the mapping relationship, a set of data operation locations for multiple transactions is determined; Create multiple nodes corresponding to each of the multiple transactions; An association edge is constructed between the two nodes corresponding to the two transactions whose data operation positions intersect, wherein the association edge indicates that there is an overlap and conflict in the data operation positions between the two transactions corresponding to the two nodes.

4. The database parallel log replay method according to claim 3, characterized in that, The step of determining the set of data operation locations for multiple transactions based on the mapping relationship includes: Based on parsing the transaction log stream sent by the main database, multiple transactions are identified; The data operation locations of the multiple transactions are mapped to the multiple transactions according to the mapping relationship, thereby obtaining a set of data operation locations of the multiple transactions.

5. The database parallel log replay method according to claim 3, characterized in that, The overlapping conflict is determined through the following operation: A preset data operation position comparison unit is defined, wherein the data operation position comparison unit indicates the comparison range used to determine whether there is an intersection between the data operation positions of two transactions; If the data operation location comparison unit is determined to be a table unit, and the data operation locations of the two transactions both include the same table, then it is determined that the data operation locations of the two transactions overlap and conflict. If the data operation location comparison unit is determined to be a row unit, and the data operation locations of the two transactions both include the same row, then it is determined that the data operation locations of the two transactions overlap and conflict.

6. The database parallel log replay method according to claim 1, characterized in that, The replay of the target transaction includes: The target transaction is written as a transaction to be replayed into the cache of the execution queue to be replayed; The replay strategy of the transactions to be replayed is dynamically adjusted based on the number of transactions to be replayed in the execution queue; wherein the replay strategy indicates the replay order among multiple transactions to be replayed. In response to determining that the replay of the target transaction has been completed, information is generated to characterize that the replay of the target transaction has been completed.

7. The database parallel log replay method according to claim 6, characterized in that, The step of dynamically adjusting the replay strategy of the transactions to be replayed based on the number of transactions to be replayed in the execution queue includes: In response to determining that the number of transactions to be replayed is less than a predetermined threshold, the replay order of the multiple transactions to be replayed is determined according to the commit time of the transactions, and the target thread is invoked to serially replay the multiple transactions to be replayed according to the replay order; In response to determining that the number of transactions to be replayed is greater than the predetermined threshold, the target thread is invoked to replay multiple transactions to be replayed in parallel.

8. A database parallel log replay device, comprising: The parsing module is used to parse the transaction log record stream sent by the main database to obtain the start time of multiple transactions, the data operation position of multiple transactions, and the commit time of multiple transactions, wherein the commit time indicates that the transaction has been completed in the main database; The acquisition module is used to obtain information about multiple transactions that have been replayed and completed by threads; The associated transaction determination module is used to determine multiple associated transactions that intersect with the data operation position of the target transaction based on the data operation positions of multiple transactions, wherein the intersection indicates that the data operation position of the target transaction and the data operation position of the associated transaction overlap or conflict. The prior transaction determination module is used to determine the associated transactions whose start time is earlier than the start time of the target transaction and whose commit time is earlier than the commit time of the target transaction as prior transactions of the target transaction; The first replay module is used to replay the target transaction based on the information that the transaction has been replayed and completed by the thread, provided that the prior transaction has been replayed and completed. The second replay module is used to replay the target transaction if the target transaction does not have the preceding transaction.

9. An electronic device, comprising: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the database parallel log replay method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the database parallel log playback method according to any one of claims 1 to 7.

Images